Model features and prediction accuracy
Model Performance insights are available to users with an active Dataro Predict subscription and can be found in your Dataro account under the Predict tab.
The model summary provides information on what type of data your models are trained on, what it is trained to predict and the type of model.
The section on Data Sources Used gives you detailed insights into the data extracted from your CRM that is used to create the predictions. It also covers the Training data size and the total number of Predictive Features used in your specific model.
We also share the top 10 predictors for a donor taking this action.
Model Performance and Incidence Rate
This section details how often the predicted event occurs per 100 donors, relative to their Dataro score and rank.
- Top Ranked Donors (rank 1 - 1,000): The 74.7% incidence rate means that for the top 1,000 donors selected by the model, 747 made an appeal donation in the 90 days after the prediction.
- Low Ranked Donors (rank 10,001-11,000): The 0.1% incident rate means that only 1 out of every 1,000 donors in this group participated in the event.
For new clients, the charts generated will be green and the data will be simulated retroactively to show what Dataro scores/ranks would have been, had they been used to predict historical donor behavior and/or campaigns then.
Once there is enough data and/or time has lapsed, typically 6 months, the chart generated will be in purple and show how your Dataro predictions have performed during this period.
The chart illustrates the relationship between donor ranks, incidence rates, and revenue per donor. It helps you understand how well your predictive model orders donors by their likelihood to donate.
Toggle between the Dataro Ranks and Dataro Scores to see how your predictions performed by Rank or by Score.
We use two key metrics to evaluate our models: PRC AUC and Top-N. These metrics indicate how well a model balances precision and recall. Higher values mean better performance.
-
PRC AUC (Area Under the Precision-Recall Curve): This metric ranges from 0 to 1, with higher values indicating better performance. It evaluates how well the model balances precision (accurate positive predictions) and recall (capturing all true positives), which is particularly important for datasets with imbalanced outcomes, such as having many more non-donors than donors.
-
Top-N: This metric involves counting the number of positive cases within a specific timeframe (eg. donations through direct mail in three months). We rank the predictions and review the top 'n' ranks to find the actual positive cases. For instance, if there were 150 direct mail donors starting from January 1, 2024, and the model accurately identified 120 of them within the top 150 ranks, the Top-N score would be 0.8.