Dynamics 365 Marketing provides organizations with the ability to plan and execute customer experiences through which they can engage with their customers. Those journeys can be based on targeted audiences using the Outbound Marketing capabilities or have an engagement based on key moments using Real-Time Marketing. In either scenario, how customers engage with the organization is recorded and tracked and this information provides valuable insights to the organization.
Dynamics 365 Marketing includes visualizations and dashboards to allow organizations to perform the analysis of this insights data, but it also organizations allow the opportunity to create custom reporting by exporting that data. This blog explores how the export of this data can be achieved along with the implications of doing so.
Within the Dynamics 365 Marketing product there are two main modules which group the available functionality: Outbound Marketing (OBM) and Real-Time Marketing (RTM). Each carry their own features but, from a reporting perspective, they p:
- Iin-app insights which cover the more common uses and analysis (in RTM this is provided through PBI Embedded reports while OBM provides Insights controls in the Email, Journey and Contact forms)
- Export approach for building custom reports.
Where the bigger difference occurs is in how the interaction data is stored and exported.
Dynamics 365 Marketing keeps track of the way marketed contacts react to marketing messages and provides detailed analytical views to help organizations understand the impact and analyze the ROI of executed campaigns. Each key marketing table provides statistics, analytics, and key performance indicators (KPIs) based on the marketing results. These insight categories cover a wide range of activities like email interactions, web interactions, form interactions etc.
When Outbound Marketing was designed, it created an architecture which utilized Azure storage to hold the interaction data for the journeys. This provided a faster response times than what was possible in Dataverse at that time. Dataverse has evolved and a managed data lake is now available, providing the faster storage that Dynamics Marketing needs. Real-Time Marketing makes use of this storage to take the place of the previously created Azure storage. The resulting architecture looks like this:
Current Process to Consume Outbound Marketing Insights – Using Azure Blob Connection
Dynamics 365 Marketing provides built-in analytics throughout the application, but you can also create your own custom analytics and reports from your Dynamics 365 Marketing data by using the native Azure Blob Storage Connection feature. To be able to use this feature customers will need to provide an Azure Blob Storage and create a “Shared Access Signature” and use this URL to create a new analytics configuration record. After this is completed, Dynamics 365 Marketing will begin to export interaction data as individual files to your blob storage container, and you can start using this data for your custom reporting requirements.
Step by step instructions to use this feature can be found here.
Real-Time Marketing Insights Data – Using Microsoft Managed Data Lake
Real-Time Marketing utilizes a different approach to store the interaction data which provides enhanced performance as required by the new services that it supports. This data is kept in a Microsoft Managed Data Lake and from there is consumed by the Real-Time Marketing services and the in-app reporting.
While most of the data that is stored in Dynamics 365 Marketing is kept within Dataverse tables, the marketing interaction data requires a different storage to support higher volumes and higher throughput of changes as customers interact with the organization opening emails, clicking on links, etc. With Real-Time Marketing, this data is stored in a Microsoft Managed Data Lake from which it is surfaced to application.
Extracting marketing insights in Real-Time Marketing
As mentioned above, the interaction data collected by Real-Time is stored within the Microsoft-managed data lake. It is however possible to extract this data using, the process described The article shows how to get to the CDS folder but for RTM Marketing interactions, we are interested in the msdynmkt_aria_exporter folder. The detailed process would be :
- Identify the URL for the Dynamics Marketing environment and note that as EnvironmentURL
- Access the https://
- In the resulting JSON, search for data lake folder with name msdynmkt_aria_exporter_folder and then:
- Locate the Container URL property, indicated by containerendpoint and make note of the value as ContainerURL
- Locate the path property and make a note of the GUID provided as PathURL
- Use the values above to obtain a SAS token from https://
/api/data/v9.1/RetrieveAnalyticsStoreAccess(Url=@a,ResourceType=’Folder’,Permissions=’Read,List’)?@a=’ / ‘
- Use the SAS token to construct a SAS URL as https://
/ ? > and access through the desired application, like Azure Storage Explorer. The below screenshot shows the access when using the SAS URL to connect to ADLS Gen 2 Container
Considering the currently available options, one proposed architecture for custom reporting entails copying interactions data from the Microsoft managed storage via the SAS token for Real-Time Marketing interactions, the export to blob feature for Outbound Marketing and use Dataverse’s native Azure Synapse Link feature ( https://docs.microsoft.com/en-us/powerapps/maker/data-platform/azure-synapse-link-synapse) for tables which are stored in Dataverse (like customer journeys and email message).
Once required data set is in place inside customer’s own data lake, then you can set up your Azure Synapse workspace and connect and start processing your Dynamics Marketing data.
For customers who are also using Outbound Marketing and required unified reporting, the existing export to blob storage would continue to be used, transferring that to an Azure Blob Storage and from there to the customer data lake.
In the above sections, it has been described how the interaction data can be exported to create custom analytics. While this covers the current picture, it is useful to consider that there is evolution expected with this architecture. Initially, the interactions captured in outbound marketing will start to become available for export via SAS token along with the Real-Time Marketing interactions, simplifying the architecture as follows.
The interaction data will include a field to make it easier to identify when the source is Outbound Marketing or Real Time Marketing.
Ultimately, Microsoft is aiming to provide sufficient out of the box reports to cover customer needs. This would be complemented by tools like Azure Synapse and PowerBI to facilitate the creation of custom reports within the tools provided and without needing to replicate the data outside of the Dataverse and the Microsoft-managed data lake.
- Outbound Marketing interaction data will be integrated with Real-Time Marketing data in the future and then will become available to extract from the Microsoft Managed Data Lake
- Raw data schema of the Microsoft Managed Data Lake is in private preview and can change with upcoming releases
- As SAS token is valid for 1 hour hence you need to write a tool copy chunk of data which can be copied within that time window or create an automated mechanism to obtain a SAS token as needed
-Emanuel Caraza (Principal Solution Architect)
–Kolappan S (Senior Program Manager)
– Cemal Tevrizci (Senior Solution Architect)