Responsible AI Series: Part 3

Download the Code and Live Script

In this mini-series, we are investigating three different model-agnostic machine learning interpretability techniques offered by the MathWorks product suite. These techniques are:

  1. Local Interpretable Model-Agnostic Explanations (LIME)
  2. Partial Dependence and Individual Conditional Expectations (PDP and ICE)
  3. Shapley Values

In this third piece we shall look at the use of Shapley values.

Interpretability of Machine Learning Models using Shapley Values

Why Interpretable Machine Learning?

Artificial Intelligence has the unique ability of improving its performance over time. This human-like quality has allowed industries to integrate traditional concepts with AI systems (robotics, intelligent and autonomous systems) to improve business efficiency.  With the increased rise in machine learning in many business operations, it is becoming more evident why transparency of machine learning models is important. Some machine learning models are trained and tuned according to what kind of datasets are available and the model decisions may need to be continuously monitored for incorrect predictions and possible improvements. To control model performance and be aware of why particular outcomes are occurring, it is important that the models and systems are correctly understood, and measures are in place to assist in the interpretability of such systems. Model interpretability techniques are useful methods that allow us to better understand the underlying work and identify any misalignments in our models. An example which we explore in this article, uses an interpretability technique to understand the biggest contributing features to a credit rating model for an individual data point (query).  Credit score ratings impact the reputation of a firm; understanding the underlying contributors to a firm’s credit score can help firms influence their credit score.

Shapley Values

So far, we have discussed three interpretable techniques namely: LIMEPDP and ICE. This article particularly focuses on the Shapely Values interpretable technique. The Shapley values technique explains the contribution of a predictor feature to a prediction model, by estimating the deviation of a prediction from the average prediction where the features are averaged and used to create a baseline for comparison, as displayed in Figure 1. This local level technique offers an understanding into the relationship of multiple features on a prediction. Evaluating all combinations of features generally takes a long time. Therefore, in practice, Shapley values are often approximated by applying multiple Monte Carlo Simulations. This interpretability technique is popular in the finance industry as it is derived from game theory and provides a more holistic explanation to help satisfy regulatory requirements. The sum of the Shapley values for all features corresponds to the total deviation of the prediction from the average.

Figure 1: The Shapley values indicate how much each predictor deviates from the average prediction at the point of interest. Adapted from [1].

What Can I Do Next?

  • To explore how the built-in MATLAB Shapley function is used to assist in the interpretability of a machine learning model in more detail, download the interactive Live Script attached to this article. The Live Script walks you through an example of predicting credit score ratings and interpreting local default predictions with the metrics determined by Shapley.
  • Request a trial.
  • Find out more from the team.
  • Visit our Financial Data Science Focus Area page to learn more.

Follow us



What’s to come?

In the final part of the series, we will investigate which model interpretability technique is best suited for your machine learning project given the scenario.


  1. com. 2021. Interpretability. [online] Available at: <> [Accessed 25 March 2021].
  2. com. 2021. Shapley Values for Machine Learning – MATLAB. [online] Available at: <> [Accessed 25 March 2021].
  3. Molnar, C. 2021. Interpretable Machine Learning. [online] Available at < > [Accessed 29 March 2021]
  4. com. 2021. How do you teach AI the value of trust?. [online] Available at: < .pdf> [Accessed 24 March 2021]