Problem

The concept of "hallucination" in artificial intelligence, while commonly associated with large language models, also poses a significant concern for medical AI, particularly in medical image processing and computer-assisted interventions. Unlike factual errors in text, identifying hallucinations in biomedical image analysis can be less intuitive. The paper argues that a clear distinction between minor errors and critical "hallucinations" is crucial for the safe integration and regulation of AI in medicine. For instance, a tumor segmentation algorithm with 90% Dice score could be state-of-the-art if errors are minor boundary inaccuracies, or completely unusable if it falsely identifies additional tumors. This highlights the need for a principled way to define, measure, and prevent errors that constitute hallucinations, especially in applications like surgical workflow recognition where incorrect phase predictions can have severe implications.

Method

This article proposes to define a class of "linear topological errors" as hallucinations within surgical workflow analysis and shows how these can be explicitly enforced. These errors violate fundamental ordering rules of surgical phases, which can be expressed as linear temporal logic predicates (e.g., "A PRECEDES B", "A IMPLIES B"). These predicates are often derived from generic surgical process models (gSPMs) created via multi-expert consensus. The core of the proposed method involves integrating these topological constraints directly into the final activation layer of deep learning models using Probabilistic Graphical Models (PGMs). Specifically, sum-product belief propagation replaces traditional softmax, and max-product belief propagation or max-flow replaces argmax. For acyclic graphs, this approach offers mathematical guarantees that the model will not produce predictions violating the encoded topological rules, effectively preventing certain types of hallucinations. This allows the network to learn inherent statistical relationships while ensuring adherence to crucial domain-specific logic.

Results

A simulation of robot-assisted hysterectomy surgical workflow recognition demonstrated the efficacy of the proposed method. By enforcing the topological predicates derived from a simplified gSPM in the application-time (i.e., during inference), the model's accuracy improved by approximately 10%. Crucially, this approach also removed the vast majority of topological errors that would otherwise be classified as hallucinations. Interestingly, incorporating these predicates during the training phase showed minimal additional improvement in accuracy if they were already enforced at application time. This suggests that explicit constraint enforcement at the inference stage is highly effective, potentially allowing for the use of more generalist pre-trained models with guaranteed non-hallucination behavior. The study also noted that enforcing ordering constraints implicitly helped reduce other types of errors, such as uniqueness violations.

Implications

The ability to mathematically guarantee the absence of certain types of errors, identified as hallucinations, has significant implications for the regulation and trustworthiness of medical AI. This method provides a principled way to distinguish and prevent critical errors, bridging the gap between empirical regulatory approaches and more formal verification methods. It suggests a pathway for regulating machine learning in medical image computing and computer-assisted interventions by supplementing empirical validation with mathematical guarantees of correctness. Furthermore, the finding that application-time enforcement of constraints is highly effective opens possibilities for deploying pre-trained foundation models in medical domains, as domain-specific non-hallucination behavior can be ensured without extensive retraining. Challenges remain in defining appropriate and comprehensive sets of predicates, especially given the complexities and variability of real-world surgical procedures and the distinction between normative and descriptive rules.