Case History: Real Time A/B split test of video email contentCampaign objective: Identify the winning version of the content and then present this video email content to all recipients, in real time, to maximise sales.
Market leading Hollywood based entertainment content distributor, focused on DVD retail sales.
Purchasing research had indicated that some shoppers purchase DVDs purely for the movie content, while others purchase to access the “extras content” such as directors cuts and deleted scenes and which DVDs have become synonymous with.
The client was seeking to quantify the relative interest in movie content vs. the extras content for a particular DVD release.
Their proposed approach to deliver this insight was to deploy a standard A/B split email campaign to test a version of the email featuring the Movie content vs. the Extra content.
This approach would certainly deliver the required insights that could be employed across future DVD release campaigns, but the insights could not be used to impact the effectiveness of the current campaign. The issue was that these emails were being opened over a prolonged period of a few hours – normal for any email campaign – but a problem when the content you are promoting is live now and relegated to history within a couple of hours at which time the email becomes irrelevant and ineffective.
The Playable platform was used to create a single campaign that enabled the 2 versions of the video content to be tested in real time.
Playable’s feature of real-time A/B testing was configured into 2 segments where 50% of the opens were shown the movie content, and 50% were shown the extras content.
The platform’s real-time analytics calculated and tracked which version of the content was delivering the highest response rate, in this instance click through rate was used. The option to appraise the content versions using downstream data such as number of sales or bascket size, where also considered.
Playable used its machine-learning algorithms to optimise the mix of the audience to receive each version of the content, based on mathematical confidence levels of which version is delivering against the stated metric of clicks, in real time. In machine-learning it’s called “regret minimization”, or in retailing it’s called “incremental sales.”
The Playable machine-learning algorithm identified which version of the video email was driving the most response, and adjusted the mix, in real time, to deliver the winning version to all future openers of the email.
The click-thru rates for the two versions were measured at a differential of 2x, implying that the retailer achieved a doubling of sales by showing the most responsive video content, when compared to the scenario of showing the less responsive version of the video content.