Definition of AB Testing or Split-Testing in Email Marketing

AB testing is a technique commonly used in digital marketing for optimizing and improving content performances. In email marketing, AB testing (also called split testing) consists in creating two versions of a message (by variating the subject, the content, the “call to action” buttons, etc.). The two messages are then simultaneously sent to two different samples of recipients in order to determine which one of the two versions gets better results. Once defined the most effective version based on the results of the A and B email campaings, the final campaign is sent to the rest of the email database. This technique allows you to optimize the message before sending it to all of your recipients.

Create an AB Testing email campaign

The option that allows you to create an AB Testing email campaign is available for all paid subscription plans.

1) Login to your account and click on My Campaigns (in the header menu) -> then click All my campaigns in the dropdown menu

2) You will see a list of your email campaigns and a button Create an AB Testing campaign

3) The email campaign creation module consists of 4 steps (1. SETTINGS  2. MESSAGE A, MESSAGE B and  3. VALIDATION)

1) SETTINGS

Settings options:

Define a name for the email campaign for your online backups and also the sender email address. Both versions of the email will be sent with the same sender email address.

You can define 2 different sender names (alias) and different email subjects.

 Settings option - AB testing - Mailkitchen

Select the recipients of your Campaign:

You need to select the mailing lists or segments of the recipients to which you want to send your email campaigns. The software will then be able to create two test samples according to the recipients that you have defined and depending on the options you choose in the section below (A / B Testing Settings).

A/B Testing setting

Under Configuration of A / B Testing options you need to select the percentage of recipients to whom you want to send the test emails A and B. This percentage will be divided into two equal samples (“Test Email A” and “Test Email B”). Once the test finished, the email with the best results can be sent to the rest of your recipients (“Number of emails for the result”).
In the example below, we have defined the test sample to 20% (this represents 30 086 recipients), half of the sample will receive the Test Email A (15 043 recipients) and the other half the Test Email B (15 043 recipients). Once the AB testing email campaign finished, the version with the better results (with more clicks, more opens or less unsubscribes) can be sent to the remaining recipients (“Number of mail for the result: 120 344 recipients).

ab testing settings - email campaign - MailKitchen

Configuration of winning terms allows you to choose between 2 options:
• If you select “Manual validation of the winner” once the AB Testing Email campaign sent, you can manually choose which version (Test Email A or Test Email B) will be sent to the rest of your recipients.
• If you choose “Automatic Validation of the winner ‘, you must define the condition that our software will use for calculating the best of the two emails (A or B) and also how long do you wish to wait for the statistical results of the campaign. Depending on the criteria that you select, the software will automatically send one of two versions to the rest of your recipients.

winning terms - ab testing - Mailkitchen

The Term indicates the criterion that our software uses to determine which one of the two is the “winning” version:

• Open Rate: The software compares the open rates of the two emails and sends automatically the version (A or B) with the higher open rate.
• Click through rate – the software compares the click rate of the two emails and sends automatically the version (A or B) with the higher click rate.
• Unsubscribe rate – the software compares the unsubscribe rates of the two emails and sends automatically the email (A or B) with the lower unsubscribe rate.

term - ab testing - MailKitchen

The Length of test refers to the time period from the moment you’ve sent the AB testing email campaign to the moment you want our software to start comparing the statistical results of the A and B email versions. At the end of this period the software will calculate and compare the results obtained by the two versions of your email. For a more objective comparison of your test results, select a longer test duration.

lenth of test - ab testing - MailKitchen

Options for your dispatch (date and time)

As for a simple email campaign, you can choose to send your AB testing email campaign after the validation step or on a specific date and time.

Steps 2) MESSAGE A and 3) MESSAGE B

The second and the third step allow you to create your Newsletter template, exactly as if you were creating a simple email campaign.

4) VALIDATION

Here you can verify the summary of the settings and click on CHECK AND SEND.

Checking and sending the winning version of your AB testing email campaign

If you selected the option “Automatic Validation of the winner”, the software will automatically send the winning version of the email to the rest of your recipients, according to the criteria you have chosen in your AB Testing campaign settings.

If you selected the option “Manual validation of the winner”, once the campaign sent, it will appear  under the status “A / B Testing: Waiting result.”
1) Click on the campaign and then on the button MANAGE-> Statistics

Manual Validation - AB testing - MailKitchen
2) You will then be redirected to the page comparing the statistics of the two versions of your campaign

Statistics - AB Testing - MailKitchen
3) Scroll down the page and in the section “Choose Winner” you can select the version of the mail that you want to send to the rest of your recipients.

choose winner - AB Testing -  MailKitchen

4) By clicking on SEND, the selected version will be sent to the rest of your recipients