Problem

The proliferation of deepfakes, synthetic media generated by AI, presents significant societal, political, and financial challenges, including fraud, misinformation, and privacy violations. While deepfake detection techniques have advanced, existing evaluation methods primarily focus on traditional performance metrics like accuracy or AUC on standardized test sets. This narrow focus often overlooks critical aspects such as robustness to perturbations, interpretability of results, and computational feasibility, which are essential for reliable deployment in real-world, uncontrolled environments. The paper highlights the urgent need for a more comprehensive assessment framework to truly understand the trustworthiness of deepfake detectors.

Method

This research introduces a novel reliability assessment framework for deepfake detectors, structured around four key pillars: transferability, robustness, interpretability, and computational efficiency. Each pillar is objectively quantified. Transferability is measured by the average AUC or accuracy in cross-dataset tests, indicating the model's ability to generalize to unseen data generation techniques. Robustness assesses performance against perturbations like compression, noise, and adversarial attacks, calculated as the average performance across these degradation categories. Interpretability is assigned a qualitative score (0-1) based on the depth and integration of explanatory mechanisms. Computational efficiency is estimated using the total number of model parameters, providing an indicator of inference cost and resource usage. The framework allows for the calculation of a global reliability score, facilitating a multi-dimensional comparison of different detection methods beyond mere classification performance. Five state-of-the-art deepfake detectors were analyzed using this proposed framework to identify their strengths and limitations.

Results

The application of the proposed framework to five deepfake detection methods revealed significant progress in areas like transferability, with some models achieving over 88% average accuracy on unseen datasets. However, persistent structural weaknesses were exposed across all pillars. A critical gap identified in robustness evaluation was the almost complete absence of testing against adversarial attacks, making models potentially vulnerable to sophisticated manipulations. Interpretability remained a major bottleneck, with only one method (TruthLens) offering integrated, natural language explanations, while others were limited to basic visualizations like Grad-CAM. Furthermore, a recurring trade-off was observed in computational efficiency: highly explainable or generalizable models often incurred high inference costs due to large foundational architectures. Only one method, CFM, showed a reasonable balance between performance, robustness, and moderate computational efficiency by using an intermediate backbone.

Implications

The findings underscore that achieving high accuracy in controlled laboratory settings is insufficient for the reliable deployment of deepfake detectors in real-world scenarios. The study highlights the critical need for more comprehensive and standardized evaluation protocols that incorporate transferability, robustness, interpretability, and computational efficiency. Developing detectors capable of robustly handling adversarial threats, providing clear and auditable explanations for their decisions, and maintaining efficiency is paramount for building trust and enabling their adoption in sensitive contexts such as digital forensics and legal processes. The paper suggests that future research should focus on expanding the analysis to more models and reproducing experiments in a standardized environment to allow for fairer comparisons and accelerate the development of truly reliable, transparent, and accessible deepfake detection solutions.