Dataro uses Microsoft's Dataverse API to extract data from and publish data to Dynamics.
In order to integrate Microsoft Dynamics into Dataro’s Machine Learning pipeline, Dataro uses
the Microsoft Dataverse Web API to interpret the underlying schema, pull data, create custom
propensity fields and update the propensity fields.
Schema Understanding
Dataro uses the metadata-related endpoints of the Web API to gather metadata on the system’s
Entities, OptionSets & Relationships. In addition to looking up OptionSet values, this metadata
is used to drive the mapping process to cohere the client’s schema with the Dataro internal
modelling schema.
Pulling Data
Dataro can be configured to handle client-specific schemas and will pull data from relevant
entities, however, generally speaking the modelling process will pull data from the following
entities:
● Contact, Account, Opportunity, DonorCommitment, PaymentSchedule, Transaction, PaymentMethod, Campaign, CampaignActivities, ActivityParties, CampaignItems, CampaignResponses, Lists, ListMembers
Data is requested in bulk, with 5000 records per page. A complete load is performed initially,
then every 30 days, while delta updates (based on records modified in the last 7 days) are run
each day. The schedules for these loads can be configured at the client’s discretion.
Fields that have been identified to contain personally identifiable information are not requested to be returned from the API.
Creating Custom Fields
The output of each propensity model will be stored on the Contact record as two separate fields
-- one recording the calibrated model score (a probability represented as a decimal number from
0.0000..1.0000), then other recording the contacts unique ‘rank’ within the cohort for that
propensity (in integer from 1..length(cohort)).
These fields are created during the onboarding via the EntityDefintion endpoint of the Web API.
Updating Custom Fields
Propensity scores change continuously in response to events occurring in the world and
recorded within the CRM. As such, model scores in the CRM must be regularly updated so that
campaigns generated with them are as optimal as possible.
Currently we update a client’s model scores every 7 days. In Dynamics this is done using batch
operations in the Web API.
System Performance Implications
In terms of pulling data, we have found the Dynamics Web API to be highly performant
compared to equivalent CRM APIs. Actual performance depends on the specific entity, but we
have observed rates in excess of over 1,000 records per second for Contacts (a wide and
dense entity) and over 3,000 records per second for ListMembers (a skinnier entity).
We have not as-yet observed any adverse effects on the client system of extracting large
amounts of data via the Web API. We suspect, based on the API structure and the observed
responses, that the Dynamics backend is simply performing the equivalent of a single full table
scan on the underlying database table as opposed to a costly query involving table joins.
Storing of client access credentials
The client’s Application ID, Directory ID, Client Secret & Service Root URL are stored as AWS
KMS encrypted strings in a Dataro database which has on-disk encryption and runs in a private
cloud, inaccessible from the internet.