We are often asked how a DM Appeal list generated using Dataro's scores compares to a DM Appeal List generated using a charity's segments. This article explains how to design and conduct a simple experiment to compare both approaches in a live campaign.
Step 1: Generate Your Own Campaign List Using Your Old Method
- In order to create a fair comparison between your old approach (e.g. RFM analysis) and Dataro's method (Propensity Scoring), you first need to generate a campaign list using your old method (let's call this the 'RFM List'). Typically, this will involve creating a query that selects a number of 'segments' and applies a series of 'exclusions' (e.g. Do Not Mail) in order to arrive at a final list of donors to include.
Tip: For a fair and impartial test, it is critical that you create your campaign list without referring to Dataro's propensity scores and ranks.
Step 2: Generate a Second List Using Dataro's Propensity Scores & Ranks
- Now that you have the the RFM List, you need to generate a second list using Dataro's Propensity Scores and Ranks (the 'Dataro List'). To do this using Raiser's Edge, follow the steps contained in this article. Salesforce instructions are here.
- When it comes to selecting your list size, login in to your Dataro Account and review our current list size recommendation for a DM Appeal. You can also consider these rules of thumb:
- The Dataro List will almost always be smaller than the RFM List (often 10-20% smaller). This is due to the superior targeting using our propensity modelling.
- A DM Appeal Score of 0.01 (i.e. a 1% chance of giving) tends to be a reasonable cut off point.
- You can also make an educated guess using the results from the same appeal last year. At what campaign size did the campaign turn from a positive ROI to a negative ROI?
- Apply the same exclusions to the Dataro List as you applied to the RFM List.
Tip: Remember, you need to apply the exact same exclusions to both the RFM List and the Dataro List. For more help on list size, contact us.
Step 3: Compare the RFM List and the Dataro List
- Now that you have created two lists, you can compare them. This allows you to identify:
- Donors that appear in both lists (let's call them the 'Blue' segment)
- Donors that only appear in the RFM List (let's call them the 'Red' segment)
- Donors that only appear in the Dataro List (let's call them the 'Green' segment)
You can imagine this comparison as a Venn Diagram. The 'Blue + Red' segment is the RFM List and the 'Blue + Green' segment is the Dataro List.Step 4: Add the extra Dataro donors (the Green Segment) to the campaign
- In order to measure the results, you need to send the campaign to every donor appearing in either the RFM List or the Dataro List. In essence, this means adding the 'extra' donors from the Dataro List (i.e. the Green Segment) to the RFM List. This gives you your final campaign data file.
- By sending the campaign to donors in both lists, you can then compare:
- Net revenue from the Dataro List vs the RFM List
- Response rate between the Dataro List vs the RFM List
- Accuracy of Dataro's propensity score predictions across the entire campaign.
And that's it! For more help with campaign experiments, reach out to your Dataro representative.