A/B test your email blasts

Automatic email A/B testing is only available to those on the Organization plan or above. To enable this feature, contact support at help@nationbuilder.com. If you're on the Leader plan, here's how to manually split test.

A/B testing sends two versions of your email blast to measure which version receives the most engagement from your recipients. The winning version is then sent to the remaining recipients. Integrating testing into your email campaigns ensures that only your best content is delivered to your recipients.

Create and start an A/B test. Test the sender of the email or the subject line. Select a winner based on the open or click rate of your test.

Note: You cannot clone A/B test emails at this time. However, you can clone a regular email blast.


Create a variant

Begin by creating an email blast. Add recipients, select a theme, and navigate to the Body tab.

Body tab

The Body tab is where you will enter the content of your email blast and set up your A/B test. You can test one of two variables:

  • Email sender (this is who the email is “from”)
  • Subject line

Note that you will test Sender A and Sender B from the same Broadcaster. Both variants will be sent from the same email address but listed with different sender names.

1. Choose a variable to test. Click “+Add variant” above the appropriate text box. You can only test one variable at a time. 

Add variant

2. A second text box will appear. Enter content for the two test variants.

Two test variants

You can remove the variant by clicking “- Remove variant” above the second text box. If you remove the variant you will lose any content entered into the text boxes.

3. Enter content for the body of your email. 

4. Click Save and preview to save your changes. You will be directed to the Preview screen of your email blast. 


Preview your test

Under Preview, both test variants will display. 

Preview screen with two variants

Send yourself a test email by entering an email address in the text box below the preview screens. Pick a variant from the drop-down menu. Click “Send Test Email.” You may send yourself more than one test email.

Send test email

Edit your variants at any time by returning to the “Body” section of your email blast. Remember to save your changes.


Set up your test

1. Select the percentage of recipients you want to use for the test. Sliding the handle right or left will adjust the size of your testing groups. The remainder will receive the winning variant. Recipients are randomly divided into these three groups. You cannot select what group a recipient will fall into.

Handle

2. You can have your blast send automatically when the test completes by checking the appropriate box. Please note, that you can only auto-send if the test can calculate meaningful results. Therefore, you must have a large enough sending population, testing groups of a certain size, and let the test run the full recommended time.    

The ideal total population is upwards of 21,000 — but you can test with fewer recipients, you'll just need a larger test segment and more time. For example, with 13k emails and a 72% test segment, it'll take ~6 hours to automatically select a winner. Alternatively, you can instead opt to pick and send a winner manually at a time of your choosing. 

Automatically send the winning variant

3. Select “Open rate” or “Click rate” from the drop-down menu. This will determine how the winner of your test is selected.

Open or click drop down

You will see a suggestion for how long your test should run. This is the amount of time needed to conclusively declare a winner based on data collected.

Hours for AB test

Factors that affect your test length include:

  • The size of your testing groups. Smaller groups will need a longer test. 
  • Whether you selected open or click rate. Click rate will require a longer test.

4. You may choose to extend the length of your A/B test. Enter a number into the provided textbox.

Extend AB test

This will add a number of hours to the minimum test length. Longer tests are more accurate, so you might consider this option to extend the length of your test without increasing the size of your test groups.

The total length of your test will display in a notification below the settings. 

Length notification


Start your test 

Begin your A/B test by clicking Send AB test now

You may also schedule your A/B test for a future date. 

1. Click Schedule AB test

2. Enter the date and time for the start of your test. 

Schedule AB box

3. Click “Schedule.”

Edit the date of your scheduled test by clicking on the blue link.

Blue link

Adjust your date and time and click “Schedule.” Unschedule the test by clicking “Unschedule.”

Schedule box adjustment

If you selected automatic sending, the blast will send after the scheduled test has completed.  


Your test in progress

Test results will display on the Preview screen of your email blast. 

AB test results

You will be able to track: 

  • When it was sent
  • Email stats
  • The time remaining in the test if you selected automatic sending

Send your blast

There are three ways to send your winning blast.

1. You may elect for automatic sending. Select the appropriate checkbox while setting up your A/B test. When the test completes the winning variant will be sent to the remaining recipients. If the test is inconclusive, Variant A will send. 

Automatically send the winning variant

2. Schedule your blast for a future time and date. Once your A/B test is in progress you can manually judge which variant has the most engagement. Select the chosen variant from the drop-down menu and click “Schedule.”

Schedule blast

Select the date and time for your blast to send. You may edit the date at any time by clicking on the blue link with the date: Blue link and adjusting the information. 

3. Manually send the blast. While the test is in progress, select the variant you want to send from the drop-down menu and click “Send to remaining recipients now.”

Manually send your blast


Final results

After the test is complete and the winning variant sent, the results from the test and final send will remain accessible on the Dashboard of the email. You will also be able to see the stats of the email at [Broadcaster] > Email > Blasts.


A note on meaningful results

You will notice that automatic sending is built around "meaningful" results. A note on how meaningful results are calculated:

There is a standard period of time that a typical recipient needs to make a decision about an email: to open, to click, or to delete. After looking at the aggregate historical data of all email blasts sent in NationBuilder, this standard was calculated. When you set up an A/B test, the specifics of your test are measured against this standard and a recommended running time is produced. Running your test for the full recommended time will lead to meaningful results with a 95% confidence in accuracy.

Keep in mind that you have some control over the recommended running time. The factors listed below will affect the runtime: 

1. Open vs Click rate. More people will open your email then click on a link. Therefore it will take a longer period of time to collect accurate data on the people who clicked in your email. 

2. Size of testing groups. Increasing the size of your testing groups will always decrease the recommended runtime because it means more data is collected before a final decision is made.

3. Size of recipient population. The larger your list, the more data you can collect in a shorter period of time. Note that A/B testing is often not meaningful for lists smaller than 21,000 recipients because of the data needed to accurately declare results.

When sending an A/B test it is important not to make a decision on the data too quickly. Statistics are often misleading and it is hard to be certain that your test has led to a meaningful conclusion. The provided runtime is meant to act as a recommendation to ensure that your results are as meaningful as possible. 

Back to top


How to create an email blast

How to optimize your email deliverability

Understanding email consent

Understanding the results of your email blast

How to maintain a healthy email list

If you’re unclear on how this feature works, please ask a question. If you would like to see changes to this feature, please submit a suggestion.