We are often asked how a DM Appeal list generated using Dataro's scores compares to a DM Appeal List generated using a charity's segments. This article explains how to design and conduct a simple experiment to compare both approaches in a live campaign.
Step 1: Generate Your Own Campaign List Using Your Old Method
- In order to create a fair comparison between your old approach (e.g. RFM analysis) and Dataro's method (Propensity Scoring), you first need to generate a campaign list using your old method (let's call this the 'RFM List'). Typically, this will involve creating a query that selects a number of 'segments' and applies a series of 'exclusions' (e.g. Do Not Mail) in order to arrive at a final list of donors to include.
Tip: For a fair and impartial test, it is critical that you create your campaign list without referring to Dataro's propensity scores and ranks.
Step 2: Generate a Second List Using Dataro's Propensity Scores & Ranks
- Now that you have the the RFM List, you need to generate a second list using Dataro's Propensity Scores and Ranks (the 'Dataro List'). To do this using Raiser's Edge, follow the steps contained in this article. Salesforce instructions are here.
- When it comes to selecting your list size, login in to your Dataro Account and review our current list size recommendation for a DM Appeal. You can also consider these rules of thumb:
- The Dataro List will almost always be smaller than the RFM List (often 10-20% smaller). This is due to the superior targeting using our propensity modelling.
- A DM Appeal Score of 0.01 (i.e. a 1% chance of giving) tends to be a reasonable cut off point.
- You can also make an educated guess using the results from the same appeal last year. At what campaign size did the campaign turn from a positive ROI to a negative ROI?
- Apply the same exclusions to the Dataro List as you applied to the RFM List.
Tip: Remember, you need to apply the exact same exclusions to both the RFM List and the Dataro List. For more help on list size, contact us.
Step 3: Compare your list to the Dataro List
Now that you have created two lists, you can compare them. This allows you to identify:
- Donors that appear in both lists (let's call them the 'Both Lists' segment)
- Donors that only appear in your list only (let's call them the 'Us Only' segment)
- Donors that only appear in the Dataro list only (let's call them the 'Dataro Only' segment)
You can imagine this comparison as a Venn Diagram. The 'Both + Us-Only' segment your List and the 'Both + Dataro-only' segment is the Dataro List.Step 4: Add the extra Dataro donors (the Dataro Only Segment) to the campaign and store results
- In order to measure the results, you need to send the campaign to every donor appearing in either the RFM List or the Dataro List. In essence, this means adding the 'extra' donors from the Dataro List (i.e. the Dataro Only Segment) to the RFM List. This gives you your final campaign data file.
- It is important to store the results correctly for easy analysis later. This means you need to store the Experiment Segment (Both, Dataro Only, Us Only) for every constituent who received the appeal. In Salesforce this can be done with an Experiment field on the Campaign Members Object. For Raiser's Edge this can be done using Comments on the AssignedAppeals. For more information on our recommended data storing practices, see this article.
- If you store the data correctly, after the campaign you will be able to see the following results (and more!)
- Net revenue from the Dataro List vs the RFM List
- Response rate between the Dataro List vs the RFM List
- Accuracy of Dataro's propensity score predictions across the entire campaign.
And that's it! For more help with campaign experiments, reach out to your Dataro representative.