Running an Engage and Retain Experiment

We are big fans of experiments (or A/B tests) at Dataro. Experiments let you test what type of campaigns deliver the best result. In this article we are going to show you how to design an experiment to answer the question:

How many donors can I save using Dataro's RG Churn scores in an "Engage and Retain" campaign?

Step 1: Understand Control and Test

In an "Engage and Retain" experiment, we need to have both a control group and a test group:

• Control Group: This group receives no intervention (e.g. no phone call). Even though they have a high probability of churning, for the purposes of this experiment we must let them churn so we have a good comparison group.
• Test Group: This group receives the engagement phone call in order to reduce churn. For more information about how to conduct these calls, consult this article.

Step 2: Generate your "Engage and Retain" campaign list

In this step you are creating a list of regular givers to target who Dataro predicts is highly likely to churn. There are two steps to this process:

1. Generate your list using CRM-based instructions for either Salesforce or Raiser's Edge
2. Apply the appropriate exclusions for RG Churn as shown in this article. For Engage and Retain campaigns, we recommend excluding regular givers if they:
   • Have a "Do not call" communication preference
   • Have an Invalid phone number
   • Have given less than three successful RG gifts
   • Do not currently have an "Active" regular gift commitment
   • Have not given at least 1 RG gift in the past 3 months
   • Have received a previous phone call communication in the past 3 months
3. Use Dataro's RG Churn Rank to exclude regular givers outside your campaign size. For an Engage and Retain experiment you should normally include at least 1000 regular givers. Do this by adding a filter that selects "RG Churn Rank <= 1000". Keep in mind that this rank does not take into account your exclusions and you will need to increase this number in order to extract a final campaign size of 1000.

Step 3: Randomly allocate your constituents to either the Control or the Test group

Random allocation is central to the scientific method. This ensures there are no underlying differences between our groups that will bias our result before we even start running the experiment. For instance, if your Control group is full of 80 year olds and your Test group is full of 20 year olds, you may conclude that an intervention worked when it was in fact due to the make-up of the groups. This is an "unfair" test. 

To fairly allocate constituents, use a method for generating random numbers. Here is a guide to doing so in Excel. 

• Export your campaign list to Excel, you should include the columns "Constituent ID", "Dataro RG Churn Score", "Age", "Average Gift Size".
• Create two new columns, "Random Number" and "Experiment Group"
• Apply the following formula to every cell in the Random Number column =RAND()
• Sort the entire table by the column, Random Number.
• In the Experimental Group Column, assign the first 50% of your campaign list to group "Control" and the second 50% of the list to the group "Test" (For a campaign of size 1000, there should be 500 in each group).
• Verify the random allocation is successful by counting how many constituent records in each group. Also confirm that each group has the same age, gift size and average Dataro score. 

Step 4: Store your Experimental data in your CRM so you can measure results later

It is important to ensure you have a valid copy of this data before you start the campaign. Dataro recommends storing this Excel file as well as storing the experiment information in your CRM. Regardless of what CRM you are on, you should be able to store your experiment information in additional fields. This can be as simple storing the text "Group A" and "Group B" on each record.

Step 5: Exclude the Control group from Engage & Retain calls for 6 months

In order to validly measure the results, we need to give the Control group enough time to churn, which means missing three consecutive gifts in the six months after the predictions are generated. You therefore need to make sure you exclude these donors from subsequent Engage & Retain campaigns until the end of the experiment period (i.e. for 6 months after the calls). Create an exclusion to make sure these donors aren't called in upcoming Engage & Retain programs.