Problem

Artificial Intelligence (AI) models, particularly deep neural networks, are often treated as "black boxes" due to their complex internal workings and lack of transparency in decision-making. This ambiguity hinders trust and widespread adoption, especially in critical applications such as remote sensing image segmentation, where understanding the rationale behind a model's prediction is essential. While Explainable AI (XAI) has seen success in image classification, extending these methods to image segmentation remains challenging due to the intricate spatial correlations between pixels and regions.

Method

The paper proposes Entropy-Centric, a novel sampling-based XAI method specifically designed for semantic segmentation, leveraging the entropy uncertainty principle. This method adapts concepts from Sobol indices for image segmentation, focusing on how perturbing input patches affects output entropy to identify regions crucial for segmentation confidence. The process involves two primary sampling phases: initializing base sampling matrices (A and B) using quasi-Monte Carlo (QMC) methods, and generating perturbation masks (Ci). Perturbed images are then created, followed by model inference to obtain spatial score maps. These maps are masked for the target object, Softmax probabilities are computed, and finally, importance scores are calculated based on the absolute change in entropy between perturbed and base samples. A larger entropy change indicates a higher attribution score for a region.

Additionally, the authors introduce a new XAI evaluation methodology called the High-Salience Influence Test (H-SIT), which complements existing methods like L-SIT. H-SIT assesses whether regions highlighted as important but lying outside the target object are genuinely influential by measuring the performance degradation (e.g., drop in prediction confidence or IoU) when these regions are removed. This helps in identifying falsely highlighted regions that L-SIT alone might miss.

Results

Qualitative analysis demonstrated that Entropy-Centric and Score-CAM generally produced more effective heatmaps compared to Grad-CAM, successfully highlighting building areas and their surrounding context. Entropy-Centric exhibited more consistent performance, avoiding failure cases (e.g., highlighting distant regions or missing target objects) occasionally observed with Score-CAM. Quantitatively, using both L-SIT and the proposed H-SIT evaluation methodologies with metrics like prediction confidence drop, Intersection over Union (IoU) score drop, and entropy increase, Entropy-Centric consistently outperformed benchmarked methods (Seg-Sobol, Grad-CAM, Score-CAM). Under L-SIT, Entropy-Centric showed minimal performance degradation upon removal of low-importance regions, indicating its precision in identifying irrelevant areas. Under H-SIT, Entropy-Centric consistently yielded the largest performance drops when high-importance regions were removed, confirming its accuracy in isolating decision-critical areas. A comparative analysis also highlighted that Entropy-Centric operates in a fully black-box manner, offering greater robustness and architectural independence compared to Score-CAM, which requires internal model access.

Implications

This work significantly contributes to the field of explainable AI by introducing a robust Entropy-Centric XAI method specifically tailored for semantic segmentation, addressing a critical gap in existing methodologies. Its demonstrated superior performance in identifying both irrelevant and decision-critical regions enhances the interpretability and trustworthiness of AI models in complex domains like remote sensing. Furthermore, the proposed High-Salience Influence Test (H-SIT) provides a valuable and comprehensive framework for evaluating the fidelity of XAI methods, ensuring more reliable assessments of explanations. This research establishes a solid foundation for future advancements in XAI for semantic segmentation, paving the way for multi-dataset validation, the development of hybrid XAI approaches, and support for diverse architectural models.