Use split lists to test your emails

This HOWTO outlines how to manually split test your emails. Automatic email A/B testing is available to those on the Organization plan or above.

You are able to perform simple A/B testing for your NationBuilder email campaigns. In this example, you will test which subject line gets the most people to engage with your email.

Note that A/B testing is most valuable for organizations with lists of tens of thousands of people. There are calculators online to determine the ideal sample size and statistical significance of your results. Generally, with smaller lists, you're going to get the best results by understanding who you're reaching out to and simply matching your messaging to them.

If you're doing a split test start by creating a new list. You may also pick an existing list by going to People > Lists. Select a list from the dropdown by clicking on Edit icon

Once you have selected a list, click on “Split.” 

You may segment the list either by percentages or by equal parts. For A/B testing it is best to split via percentages.

A popular split for subject line testing is dividing your list into percentages of 10%, 10%, and 80%. This allows you to test different subject lines with two small groups of 10%. After observing which test receives the best response you then can send the better subject line to the larger 80%. You can also use this method to test different send times as well as other criteria. 

Enter the correct percentage amounts and click “Start split” to begin the split.

Percentage split

When the split is completed you will see three links to the separate lists next to a count of how many people are in each list. Please note that the people in your original list will be randomly divided only the first time you split the list. Subsequently, every time you split the original list it will split in the same pattern as before.

You can open a list by clicking on the blue names of the newly split lists, or you can locate them at any time under People > Lists.

Post split

Once the lists are finished splitting, create a new email. Under “Recipients” select the “List” radio button and then use the provided drop down menu to select one of the 10% lists you created. 

Recipients version A

After setting up your blast, you will want to clone the email to duplicate the content for your A/B testing. Go to [broadcaster] > Email > Blasts to locate the email you created. Click on Edit icon to open it for editing. Click on "Clone" to copy the email.

Email dashboard

Enter a slug to name the new email. Click “Clone.”

Enter clone slug

After clicking “Clone” you will automatically be taken to the settings of the new blast you just created. Under “Recipients” select the second 10% list you created in the split by typing the name of the list into the drop down menu. Click “Save recipients.” 

Recipient drop down

Edit the criteria you would like to test under “Body.” In this case the subject line is being tested.  

Test subject line B

Click Save and preview to save your changes. 

When you are ready, send both of your test emails. Examine the statistics of your emails to determine whether subject line A or B was more effective before moving forward with sending the email to the 80% segment of your list.


Related HOWTOs

How to use lists and batch actions

How to set up email blasts

Understanding the results of your email blast

How to optimize your email deliverability

If you’re unclear on how this feature works, please ask a question. If you would like to see changes to this feature, please submit a suggestion.