The Challenge of Explainable AI

In recent years, AI technologies have advanced significantly, becoming an integral part of daily life. While these AI tools deliver highly accurate results, their complexity raises concerns. Most of AI systems, and especially Deep Learning or Generative AI, are considered to be “back box” — which means we don’t know how the reasoning was done. Researchers, experts and policymakers are questioning whether we can truly understand how AI works, or if it remains a “black box”, with its potential consequences.

Here is the plan of this newsletter:

• What is Explainable Artificial Intelligence?

• Some context and History

• How does it work?

• What kind of explainability?

• Why is it important and the benefits

• The limitations and risks

• Are all systems requiring explainability?

• Conclusion

• Sources

Explainable AI (XAI) is a terminology that can refer to AI models that are able to provide clear and understandable explanations for their output, or it can refer to the various methods used to achieve this.

As AI becomes more advanced, it challenges our ability to understand how algorithms reach their results. This often turns the calculation process into a "black box" that is difficult to interpret. XAI focuses on making the reasoning behind AI decisions or predictions more transparent and understandable.

It can also overlap with Interpretable AI. Here is a distinction of the terms as well as others as defined by the Royal Society:

• Interpretable ⇒ some sense of understanding how the technology works.

• Explainable ⇒ a wide range of users can understand why or how a conclusion was reached.

• Transparent ⇒ some level of accessibility to the data or algorithm.

• Justifiable ⇒ there is an understanding of the case in support of a particular outcome.

• Contestable ⇒ users have the information they need to argue against a decision or classification.

Explainability issues can also affect symbolic AI (rules-based). When these systems use a complex tangle of rules to make deductions, it can become nearly impossible to trace how inputs are converted into outputs.

If we take a quick step back, the first AI models appeared in the 1950s. But, the first AI models that could explain their reasoning were created between the 1970s to 1990s. Back then, it was symbolic AI systems, like Mycin, because research was more focused on that, at least in the US and Western Europe.

In the early 1990s, neural networks made a comeback, and researchers began to investigate whether it was possible to meaningfully extract the rules automatically generated by opaque trained neural networks.

By the 2010s, the amount of data available exploded, and deep learning emerged as a new trend from researches on neural networks. It was made possible thanks to the increased computing power of GPUs and distributed computing. But with that came the first public concerns about racial and other biases in AI, and the demand for more transparency.

In fact, deep learning is a black box by default, with most validation done empirically (statistically/predictively) rather than theoretically. But, for the most part, AI systems were in the hands of professionals and technicians, so not very visible for the public.

Nowadays, with the emergence of generative AI, which put technology directly into the hands of end users, the need for an AI that can explain its decisions is now stronger than ever.

Modern AI techniques like deep learning and genetic algorithms are often opaque. To improve explainability, methods to have post hoc explications are being developed.

Here are a few examples of XAI techniques:

• Layer wise relevance propagation (LRP) : a technique for identifying which features in an input vector most influence a neural network's output.

• Local interpretability : to explain a specific prediction made by a nonlinear black-box model.

• Example-based explanations : for returning similar examples using techniques like Approximate Nearest Neighbor (ANN).

• Decomposable systems : structure the analysis in several stages, focusing on the most critical steps for decision-making.

Of course, some models are more inherently interpretable, so depending on the use case it might be better to use those. Indeed, some models are self-interpretable ("white box") like linear regression, decision trees and rule-based systems ⇒ interpretability is directly into their design.

AI developers need to consider not just whether an AI is explainable or if one model is more explainable than another, but whether the system offers the right kind of explainability needed for a particular task or user group. So, users and developers have different needs regarding explainability:

• Users ⇒ a "local" approach that explains a specific decision is often the most helpful. Sometimes, it's crucial to allow individuals to contest an outcome, like disputing a rejected loan application.

• Developers ⇒ require a more "global" approach that explain how the entire system operates, to know when it is likely to perform well or poorly.

Also, several insights from psychology and social sciences reveal how human cognitive processes and biases can impact the effectiveness of an explanation in various contexts:

• Individuals like contrastive explanations ⇒ asking why one decision was made over another, rather than just inquiring why a particular outcome occurred.

• Explanations are selective ⇒ we focus on a subset of the total factors that influenced an outcome to understand why it happened.

• Causes over probabilities ⇒ we prefer explanations that refer to the causes of an outcome rather than those that refer to its probabilities.

• Social interactions ⇒ explaining something is often a social interaction — an exchange of information and ideas between people — impacting it is delivered and received.

Thus, a single decision or recommendation might need to be explained in various ways to meet the needs of different audiences and contexts. It is essential to consider the human aspect, as explanations must ultimately be relevant and meaningful to people!

Explainable AI clarifies machine learning and deep learning models, making it easier to address bias and performance issues. This helps to build trust, mitigate risks, and ensure fairness, transparency, and accountability.

More specifically, McKinsey research found that companies achieving significant returns from AI are more likely to be companies that follow best practices for explainability. And, that organizations that build digital trust with AI explainable are experiencing more growth rates (10% and more) in their revenue.

The benefits of explainability:

• Efficiently monitor, maintain, and enhance AI systems ⇒ boost productivity by helping MLOps teams quickly identify errors and improvement areas.

• Building trust in AI outputs, leading to greater adoption of AI tools ⇒ customers, regulators, and the public must trust that AI models make accurate and fair decisions.

• Apply knowledge of why an AI made a prediction ⇒ sometimes, knowing why a prediction was made is more valuable than the prediction itself.

• Assess whether AI applications meet business objectives ⇒ if the technical team explains the AI system, the business team can ensure it meets objectives and identify miscommunications

• Ensure technology and workflows comply with regulations ⇒ legal and risk teams can use technical explanations to ensure the system is compliant. In some sectors, explainability is a requirement.

• Exploitation of the systems:

Organizations must ensure data security. In explainable AI, this means safeguarding against the disclosure of exploitable data. Indeed, revealing important features and influential samples, can also expose vulnerabilities or proprietary information and technologies. Thus, balancing transparency with protection is crucial.

• Technical complexity:

End users often lack the coding skills needed to understand AI systems. Current explanation methods are technical and aimed at ML engineers for debugging, not end users, creating a gap between practical explainability and transparency goals.

• Understanding vs trust:

The aim of making AI systems explainable to end users is to boost trust and address concerns about fairness and discrimination. However, even with a clear understanding, end users might still not trust the system.

• Misinterpretation:

Depending on its implementation, XAI can be too complex or overly simplified, leading to misinterpretation. Information should be concise, transparent, intelligible, and accessible, using clear language. Explanations should also avoid jargon and technical terms, and be neutral to prevent reinforcing biases.

• Over-reliance on the AI systems:

Explanations can make humans more likely to “blindly” accept AI recommendations (automation bias) regardless of their accuracy.

Short answer: no.

Indeed, not all AI systems need the same level of explainability. Everyday tools like recommendation algorithms or autocorrect don't require users to understand their intricate workings. What matters most is their effectiveness and reliability. Excessive transparency in these cases can overwhelm and confuse users. It's like expecting drivers to know the mechanics of their car's engine—most just need it to work reliably.

There is, still today, a trade-off between model explainability and performance ⇒ Models simplify reality, often sacrificing precision for understandability and explanatory power. While simpler models are easier to interpret, they may miss intricate data patterns. In the other hand, more complex models use more data and make better decisions but are harder to explain. So, models may not be perfectly accurate, but they can still effectively predict and understand phenomena.

To conclude, explainability is not always a priority when designing AI systems. Some, like postal code sorting and online recommender systems, function effectively without extensive explainability, particularly when their accuracy is well-validated and there are no significant consequences from errors.

Explainability must be considered within the broader goals of the system, balancing privacy, accuracy, security, and intellectual property concerns. In some cases, high explainability might be needed, alongside mechanisms for user feedback and accountability, especially to empower users in their interactions with AI.

Not all AI systems require complex methods. Simpler and well-understood approaches can perform as well as "black box" methods and might be more suitable for certain challenges. In critical domains like healthcare and justice, human decision-making involves accountability structures. Similarly, AI systems in these areas may need transparency and explainability as a first step toward building trust and accountability.

Find all the sources here.