Understanding Artificial Intelligence

In the ever-changing world of engineering, a key player has arrived: Artificial Intelligence. With AI changing nearly every area in engineering, the need for more accurate AI models is becoming increasingly necessary. However, the challenge faced when using AI, is whether the results received, and decisions made by these complex AI models can be trusted. 

This is where explainable AI comes in. Explainable AI is a set of methods and processes which allows engineers to understand and trust the results from machine learning algorithms. Put simply: knowing how the results were achieved and understanding how the AI model makes decisions allows us to verify and trust the results.  

Advancements in Artificial Intelligence

Why the push for explainable AI? Models weren’t always this complex. In fact, let’s start with a simple example of a thermostat in winter. The rule-based model is as follows:

• Turn on heater below 65 degrees
• Turn off heater above 68 degrees
  

Is the thermostat working as expected? The variables are current room temperature and whether the heater is working, so it is very easy to verify based on the temperature in the room. 

Certain models, such as temperature control, are inherently explainable due to either the simplicity of the problem, or an inherent, “common sense” understanding of the physical relationships. In general, for applications where black-box models aren’t acceptable, using simple models that are inherently explainable may work and be accepted as valid if they are sufficiently accurate. 

However, moving to more advanced models has advantages:

• Accuracy: In many scenarios, complex models lead to more accurate results. Sometimes results may not be immediately obvious but can get to answers faster.
• Working with more sophisticated data: Engineers may need to work with complex data like streaming signals and images that can be used directly in AI models, saving significant time in model building. 
• Complex application spaces: The complexity of applications is growing and new research is exploring additional areas where deep learning techniques can replace traditional techniques like feature extraction. 

The significance of Explainability

AI models are often referred to as “black-boxes,” with no visibility into what the model learned during training, or how to determine whether the model will work as expected in unknown conditions. The focus on explainable models aims to ask questions about the model to uncover any unknowns and explain their predictions, decisions, and actions.

The Challenge of Balancing Predictive Power with Explainability

For all the positives about moving to more complex models, the ability to understand what is happening inside the model becomes increasingly challenging. Therefore, engineers need to arrive at new approaches to make sure they can maintain confidence in the models as predictive power increases.

Using explainable models can provide the most insight without adding extra steps to the process. For example, using decision trees or linear weights can provide exact evidence as to why the model chose a particular result. 

Engineers who require more insight into their data and models and are driving explainability research for: 

1. Confidence in models: Many stakeholders are interested in the ability to explain a model based on its role and interaction with the application. For example:

• A decision maker wants to trust and understand how the AI model works. 
• A customer wants to feel confident the application will work as expected in all scenarios and the behavior of the system is rational and relatable. 
• A model developer wants insight into model behavior and improvements that can be made to the accuracy by understanding why a model is making certain decisions. 

1. Regulatory requirements: There is an increasing desire to use AI models in safety-critical and governance and compliance applications that may have internal and external regulatory requirements. Although each industry will have specific requirements, providing evidence of robustness of training, fairness, and trustworthiness may be important. 
2. Identifying bias:  Bias can be introduced when models are trained on data that is skewed or unevenly sampled. Bias is especially concerning for models applied to people. It’s important for model developers to understand how bias could implicitly sway results, and to account for it so AI models “generalize” – provide accurate predictions without implicitly favoring groups and subsets.
3. Debugging models: For engineers working on models, explainability can help analyze incorrect model predictions. This can include looking into issues within the model or the data. A few specific explainability techniques which can help with debugging are described in the following section.

Exploring Contemporary Approaches to Explainability

Explainable methods fall into two categories:

• Global methods provide an overview of the most influential variables in the model based on input data and predicted output. 
• Local methods provide an explanation of a single prediction result.