Case History: Real Time A/B split test of video email content

Campaign objective: Identify the winning version of the content and then present this video email content to all recipients, in real time, to maximise sales.

The Client

Market leading Hollywood based entertainment content distributor, focused on DVD retail sales.  

The Challenge

Purchasing research had indicated that some shoppers purchase DVDs purely for the movie content, while others purchase to access the “extras content” such as directors cuts and deleted scenes and which DVDs have become synonymous with.

The client was seeking to quantify the relative interest in movie content vs. the extras content for a particular DVD release.

Their proposed approach to deliver this insight was to deploy a standard A/B split email campaign to test a version of the email featuring the Movie content vs. the Extra content.

This approach would certainly deliver the required insights that could be employed across future DVD release campaigns, but the insights could not be used to impact the effectiveness of the current campaign. The issue was that these emails were being opened over a prolonged period of a few hours – normal for any email campaign – but a problem when the content you are promoting is live now and relegated to history within a couple of hours at which time the email becomes irrelevant and ineffective. 

The Solution 

The Playable platform was used to create a single campaign that enabled the 2 versions of the video content to be tested in real time.

Playable’s feature of real-time A/B testing was configured into 2 segments where 50% of the opens were shown the movie content, and 50% were shown the extras content.

The platform’s real-time analytics calculated and tracked which version of the content was delivering the highest response rate, in this instance click through rate was used. The option to appraise the content versions using downstream data such as number of sales or bascket size, where also considered.

Playable used its machine-learning algorithms to optimise the mix of the audience to receive each version of the content, based on mathematical confidence levels of which version is delivering against the stated metric of clicks, in real time. In machine-learning it’s called “regret minimization”, or in retailing it’s called “incremental sales.”

The Playable machine-learning algorithm identified which version of the video email was driving the most response, and adjusted the mix, in real time, to deliver the winning version to all future openers of the email.   

Results

The click-thru rates for the two versions were measured at a differential of 2x, implying that the retailer achieved a doubling of sales by showing the most responsive video content, when compared to the scenario of showing the less responsive version of the video content.

Seeing is believing

Send yourself a video email campaign from a selection of recent campaigns below.

Playable creates 10 second clips of videos which are configured to play on any device and any connection automatically at the moment the email is opened, and can be sent using any of the popular email service providers.

Get started for free >>How it works >>