Machine Learning Models: Insights

The Insights tab for a Propensity model provides a set of three Dashboards: Key Insights, Migration, and Model Performance. These Dashboards are accessible using the menu near the top of the screen. 

Key Insights

The Key Insights Dashboard is displayed by default when you select the Insights tab. This Dashboard displays the following information:

• Propensity Score Distribution: This bar chart shows the distribution of your scored customer population, grouping them into "buckets" of five percentage points. This chart is designed to help you view the likelihood of this population performing the selected action for this propensity. The color coding of the bars indicates the Propensity Group (High, Moderate, or Low). If you hover your cursor over a bar, you'll see a tooltip with the exact audience size within that likelihood percentage. 

• Model Fit: This gauge chart is used to indicate how effective this model is at successfully predicting the customer behavior for the selected model. 

• Explainability: This gauge chart is used to indicate how effective the attributes used in this model are at predicting the selected behavior. 

• Predictive Factors: This table lists the factors that affect a customer’s likelihood to engage in the predicted action. The "Importance" column indicates the strength of each factor, and whether it positively or negatively affects the likelihood to engage in the predicted action. The "Explanation" column provides a clear human-readable description of the factor and its importance in the prediction. If you click on a Variable Name, you can drill down into additional insights about that factor. The bar graph shows how that factor differs for the Low, Moderate, and High propensity groups. If you hover your cursor over a bar, you'll see a tooltip with more details.

Migration

The Migration Dashboard is designed to provide insights into how your customer scores are changing over time. This Dashboard displays the following information:

• Previous Period to Current Period: This Sankey diagram shows customer movement by Propensity Group from the previous scoring refresh to the current score. The thickness of the lines is proportional to the number of customers moving between one Group to another. 

• Historical Trend: This stacked bar chart shows how the proportional sizes of the Low / Moderate / High Groups has changed over time. If you hover your cursor over a bar section, you'll see a tooltip with more details.

Model Performance

The Model Performance Dashboard is intended for use by analysts and data scientists who want more insights into the model itself. The information in this Dashboard is highly technical and focused on measuring the performance of the model by comparing the model's predicted behavior against what actually happened. 
For propensity models, a number of metrics are provided:

• Expected Lift: Expected lift by decile. For example, if Decile 1 has an expected lift of 3.4, that would mean that the top 10% of scores (Decile 1) is 3.4 times more likely to engage in the predicted action than the average customer.

• Average Likelihood: Likelihood score, averaged across eligible customers.

• F1 Optimization: The optimal F1 score based on weighing precision and recall. The cutpoint used for the Confusion Matrix (see below) is determined by the intersection of the two lines.

• Model Performance Over Time: A historic view of certain fit metrics to show how model performance may have changed over time.  

• Model Metrics: A table of common model metrics used in classification models. Values for both the training and testing datasets are provided.

• Confusion Matrix: Shows actual performance (i.e. known instances of the outcome) compared to predicted performance.

Note: Send Time Optimization (STO) results are closely integrated with Cheetah Messaging, in order to make this data more actionable. When building an Email Campaign in Messaging, the Campaign screen presents you with the option of using STO for scheduling the Campaign deployment. If you select this option, the platform will automatically identify the optimal time to send messages to each recipient in the Campaign's audience, based on the results of the STO Model scoring. For more details on STO configuration within an Email Campaign, please see the Cheetah Messaging Online Help. 

The Insights tab for a Send Time Optimization model provides a set of three Dashboards: Key Insights, Migration, and Model Performance. These Dashboards are accessible using the menu near the top of the screen. 

Key Insights

The Key Insights Dashboard is displayed by default when you select the Insights tab. This Dashboard displays the following information:

• Observed Engagement Time: Observed behavior (that is, what the recipient actually did) serves as the best resource for Send Time Optimization. This bar chart depicts the distribution of opens / clicks for your engaged customers over a 24-hour period. Customers who are not engaged (i.e., no observed open or click behavior) are not included. If you hover your cursor over a bar section, you'll see a tooltip with more details.

• Time of Day Effect on Engagement: This bar chart depicts the relative impact (positive or negative) of opens or clicks based on the hour of day. 

• ML-Optimized Send Times: This bar chart depicts recommended distribution of optimized send times to those customers not currently engaging. 

Migration

The Migration Dashboard is designed to provide insights into how customers’ optimized send times have changed over time. Please note that the Migration tab is always displayed in Coordinated Universal Time (UTC) regardless of the user's time zone. 

• Previous Period to Current Period: This Sankey diagram shows customer movement by ML Recommended Send Time from the previous scoring refresh period to the current period. The thickness of the lines is proportional to the number of customers moving between one Group to another.

• Historical Trend: This stacked bar chart shows how the proportional sizes of the Time Groups has changed over time. If you hover your cursor over a bar section, you'll see a tooltip with more details.

Model Performance

The Model Performance Dashboard is intended for use by analysts and data scientists who want more insights into the model itself. The information in this Dashboard is highly technical and focused on measuring the performance of the model by comparing the model's predicted behavior against what actually happened. 
For STO, basic model metrics (log intercept, log offset, log decay, R-squared) about the fit of the model are provided. A historical snapshot of certain fit metrics is also provided to show how model performance may have changed over time.

The Insights tab for a Send Time Optimization model provides a set of three Dashboards: Key Insights, Migration, and Model Performance. These Dashboards are accessible using the menu near the top of the screen. 

Key Insights

The Key Insights Dashboard is displayed by default when you select the Insights tab. This Dashboard displays the following information:

• Observed Engagement Time: Observed behavior (that is, what the recipient actually did) serves as the best resource for Send Time Optimization. This bar chart depicts the distribution of opens / clicks for your engaged customers over a 24-hour period. Customers who are not engaged (i.e., no observed open or click behavior) are not included. If you hover your cursor over a bar section, you'll see a tooltip with more details.

• Time of Day Effect on Engagement: This bar chart depicts the relative impact (positive or negative) of opens or clicks based on the hour of day. 

• ML-Optimized Send Times: This bar chart depicts recommended distribution of optimized send times to those customers not currently engaging. 

Migration

The Migration Dashboard is designed to provide insights into how customers’ optimized send times have changed over time. Please note that the Migration tab is always displayed in Coordinated Universal Time (UTC) regardless of the user's time zone. 

• Previous Period to Current Period: This Sankey diagram shows customer movement by ML Recommended Send Time from the previous scoring refresh period to the current period. The thickness of the lines is proportional to the number of customers moving between one Group to another.

• Historical Trend: This stacked bar chart shows how the proportional sizes of the Time Groups has changed over time. If you hover your cursor over a bar section, you'll see a tooltip with more details.

Model Performance

The Model Performance Dashboard is intended for use by analysts and data scientists who want more insights into the model itself. The information in this Dashboard is highly technical and focused on measuring the performance of the model by comparing the model's predicted behavior against what actually happened.

To download the entire contents of a Dashboard:

1. Navigate to the desired Dashboard.

2. Click the Gear icon near the top-right corner of the screen and select either:

• Download as PDF: Enter the filename and click download or open in browser. 

• Download as CSV: The platform creates a .zip file, with a .csv file for each tile in the Dashboard. 
  

Dashboards comprise multiple tiles, each of which contains a graph or chart. The platform allows you to download the data underlying one of these tiles. 

1. Navigate to the desired Dashboard.

2. In the top-right corner of the desired tile, click the three-dot icon, and select Download Data.

3. Select a file format.

4. Next to Results, select whether or not to apply visualization options selected in the interface.

5. Next to Values, select Formatted or Unformatted. 

6. Next to Limits, select whether to limit the data included in the extract file.

7. Enter a filename.

8. Click download or open in browser.

You can email a Dashboard to yourself, or to other recipients in your organization. 

1. Navigate to the desired Dashboard.

2. Click the Gear icon near the top-right corner of the screen and select one of the following: 

• Send: Define and execute a one-off email submission.

• Schedule: Define a schedule that submits the email on a recurring frequency.

3. In the Title field, enter a name for the Dashboard.

4. Next to Where Should this Data Go, click Email. 

5. Enter one or more recipient email addresses (separated by commas).

6. To insert a message into the body of the email, check Include a Custom Message, then enter the message text. 

7. Select a data format: PDF, Visualization, or a .zip file, with a .csv file for each tile in the Dashboard. 

8. If you selected Schedule, select a frequency, and enter or select the frequency interval. 

9. Click send (or save all if you selected Schedule).  

When you want to dig into your data to create your own analysis, EDP provides access to the Explore feature.

This Explore feature is very flexible with many options, and is primarily intended for use by data scientists and analysts who have a strong knowledge of your data model. This Help topic does not describe in detail how to use the Explore features. 

To further explore the contents and data of a standard tile:

1. Navigate to the desired Dashboard.

2. In the top-right corner of the desired tile, click the three-dot icon, and select Explore from Here.

3. The Explorer screen is displayed and populated with the details of that particular tile's contents.