Problem

The rapid advancement of Artificial Intelligence, especially with complex deep learning models, has led to a critical need for transparency and explainability, particularly in high-stakes applications like medicine and aerospace. Despite the growth of Explainable AI (XAI) methods, there is a significant lack of comprehensive and standardized evaluation protocols. Many new XAI techniques are introduced without adequate assessment of their usefulness or rigorous comparative studies. A major challenge arises because most existing XAI methods are designed for static data environments and fail to provide consistent or accurate explanations in dynamic settings where data distributions and models evolve over time, a phenomenon known as concept drift. Furthermore, evaluations often prioritize the perspective of AI developers over that of end-users, leading to an underutilization of human-centered studies, even though explanations are fundamentally intended for human understanding and trust.

Method

The paper reviews current XAI evaluation taxonomies, distinguishing between application-grounded (real tasks with experts), human-grounded (simpler tasks with lay users), and functionally-grounded (automatic proxy measures). It outlines desiderata for automatic evaluation, including fidelity, identity, stability, feature importance, compactness, and representativeness. Two case studies illustrate evaluation challenges. The first involves using XAI methods like Saliency Maps and Concept-wise Relevance Propagation (CRP) within the DetoxAI system to detect and mitigate bias in image recognition (e.g., gender correlation in the CelebA dataset), followed by concept unlearning. The second case details a human-grounded evaluation study comparing three XAI methods (ACE, ProtoPNet, RISE) for explaining CNN predictions on animal images. This study involved designing an online survey for a diverse audience, standardizing visualizations, and collecting demographic data alongside user preferences and expected feature importance. For evolving data streams, the paper discusses adapting counterfactual explanations by proposing an ensemble of multiple base explainers combined with a multi-criteria approach (Pareto front, Ideal Point Method) to generate a diverse and concise set of explanations. It also explores stream adaptation of prototype explanations, introducing new measures to compare prototype sets over time for detecting concept drift, and examines the evolution of group counterfactual explanations (GCEs) by analyzing changes in cluster centroids and their associated counterfactual vectors to understand drift locality and decision boundary shifts.

Results

In the first case study, the DetoxAI system successfully used XAI to detect and eliminate unwanted biases in image recognition, demonstrating qualitative improvements in saliency maps and quantitative gains in group-level fairness metrics like EqualizedOdds and Demographic Parity after applying concept unlearning techniques. The human-grounded evaluation revealed a clear preference among respondents for the ProtoPNet method, which scored 1,045 points, significantly higher than ACE (763 points) and RISE (643 points). While ProtoPNet generally won across most animal categories, ACE showed stronger performance for specific animals like the bison. Respondents with prior XAI experience exhibited a stronger preference for ProtoPNet, while those without preferred ACE. Crucially, strong correlations were found between the anatomical features respondents deemed important for animal recognition and the explanations provided by the preferred XAI methods (e.g., ACE correlating with an elephant's trunk, ProtoPNet with hippo's mouth or tiger/zebra stripes). For evolving data, the proposed ensemble approach for counterfactuals generated competitive and human-aligned explanations. The introduced measures for prototype evolution effectively supported concept drift detection and, when combined with feature parallel plots, aided in analyzing the causes of drift. Analyzing the evolution of group counterfactuals (GCEs) provided an interpretable proxy for assessing the locality of drifts and examining shifts in decision boundaries, quantified by cosine similarity between evolving counterfactual vectors.

Implications

The paper concludes that a standardized, unified evaluation framework for XAI is critically absent, necessitating more fundamental research into new experimental standards. Current evaluation measures are often highly specific to explanation types and tasks, lack generalization, and primarily serve as functional proxies for complex model behaviors. There is a pressing need for more comparative studies, the application of diverse measures (potentially inspired by multi-criteria decision analysis), and a greater focus on global explanations over the currently dominant local post-hoc ones. The development of robust benchmark datasets with ground-truth explanations is also crucial. The findings strongly advocate for a significant increase in application- and human-grounded evaluations, emphasizing the need for interdisciplinary research that draws from experimental psychology and robust survey design principles, rather than relying on simple user satisfaction questionnaires. This includes careful planning of user interaction sessions, user preparation, validation of responses, and potentially moving towards sequential dialogue systems. Furthermore, the challenges of evaluating XAI in dynamic environments like evolving data streams with concept drift are highlighted as a major open research area, requiring new measures that consider temporal dynamics and interactions between data, model, and explanation changes. The paper suggests that real-world applications in aerospace, earth observation, and medicine can provide valuable inspiration for addressing these complex research challenges.