Problem

The rapid advancement of generative AI has led to the creation of increasingly realistic and diverse fake facial images, commonly known as deepfakes. These sophisticated forgeries pose significant threats to personal privacy and contribute to the spread of misinformation and malicious attacks. Existing deepfake detection methods, while achieving good performance in controlled intra-domain settings, frequently suffer a substantial decline in accuracy when confronted with unseen forgery patterns or cross-domain scenarios. This limitation often stems from their reliance on complex, hand-crafted modules designed to capture specific forged clues, leading to overfitting on training data and poor generalization capabilities. There is a pressing need for a generalized deepfake detection method that employs a straightforward network structure while effectively addressing the challenge of identifying evolving and novel manipulation techniques.

Method

The proposed Generalized Deepfake Detection framework (GenDF) aims to achieve robust deepfake detection by efficiently transferring the powerful representation learning capabilities of large-scale vision models like ViT to the deepfake domain. GenDF is built upon three core schemes: Deepfake-Specific Representation Learning (DSRL), Feature Space Redistribution (FSR), and Class-Invariant Feature Augmentation (CIFAug).

**Deepfake-Specific Representation Learning (DSRL):** This scheme addresses the fundamental difference between real and fake faces—continuity versus discontinuity of pixel values across patches. It fine-tunes a pre-trained Vision Transformer (ViT) in a low-dimensional space using Low-Rank Adaptation (LoRA). Specifically, LoRA is applied to the Query (Q) and Value (V) matrices within the multi-head self-attention module of ViT, enabling the model to learn deepfake-specific relationships between patches. The Key (K) matrix weights are kept frozen to retain ViT's foundational knowledge. This approach introduces a minimal number of trainable parameters while effectively capturing continuity in real faces and various discontinuity patterns in forged ones.

**Feature Space Redistribution (FSR):** To mitigate the domain mismatch between generic images (on which ViT is pre-trained) and real/fake human facial images, FSR is introduced. It optimizes the distributions of real and fake feature spaces separately by applying a learned scaling factor and noise to the output of the fine-tuned ViT. This process effectively increases the inter-class distance, leading to more discriminative representations without requiring complex, layer-wise transformations or additional computational overhead.

**Class-Invariant Feature Augmentation (CIFAug):** To enhance generalization against diverse real face variations and emerging deepfake techniques, CIFAug actively seeks diversified representations along class-invariant directions. It first calculates the discriminative direction between the centroids of real and fake features. Then, using Gram-Schmidt Orthogonalization, it computes directions orthogonal to this discriminative direction. Features are augmented along these orthogonal directions, expanding the scope of real and fake feature spaces. This augmentation process occurs without introducing any additional trainable parameters, improving robustness and generalization to unseen scenarios while preserving discriminability.

The overall objective function combines a cross-entropy loss (for basic classification), a weighted triplet loss (to encourage tight clustering of real faces and push away fake samples), and a feature augmentation loss (to optimize the CIFAug procedure).

Results

Extensive experiments demonstrate that GenDF achieves state-of-the-art generalization performance across various challenging settings. In cross-domain evaluations, GenDF significantly improves AUC scores on datasets like DFD (1.53% increase), Celeb-DF (0.58% increase), and DFDC (3.11% increase) compared to existing methods. Notably, GenDF fine-tunes only 0.28 million trainable parameters, which is approximately 0.34% of the parameters used by the fully fine-tuned UIA-ViT, yet consistently outperforms it and other baselines.

In cross-manipulation settings, where the model is trained on one forgery type and tested on others, GenDF exhibits superior robustness and maintains consistent performance across different manipulation techniques. For instance, when trained on F2F, FS, or NT, it achieves the highest average generalization performance across all four FF++ forgery types. Ablation studies confirm the crucial contribution of each component: DSRL significantly boosts accuracy and AUC (e.g., +17.58% Acc), FSR enhances discriminative capability (e.g., +4.72% Acc), and CIFAug provides considerable gains in generalization (e.g., +3.29% Acc).

Furthermore, GenDF demonstrates superior robustness against various image quality perturbations such as contrast, saturation, pixelation, and blur, achieving the highest average AUC across these disturbances. Qualitative analysis using Grad-CAM visualizations shows that GenDF accurately identifies continuity in real faces and precise manipulation traces (e.g., boundaries, eye, mouth regions) in different types of deepfakes, unlike the baseline ViT. T-SNE visualizations further illustrate that GenDF learns highly discriminative feature spaces, enabling clear separation between real and fake samples, even in unseen domains.

Implications

The GenDF framework offers a highly effective and efficient solution for the challenging problem of generalized deepfake detection. By leveraging the power of pre-trained Vision Transformers with a parameter-efficient fine-tuning strategy and novel modules for feature learning and augmentation, GenDF achieves state-of-the-art performance with minimal computational overhead. This approach significantly enhances the ability to detect unseen and evolving deepfake patterns, which is critical for combating misinformation and protecting individual privacy in the age of advanced generative AI. The simplicity and strong generalization capabilities of GenDF make it a promising direction for robust and deployable deepfake detection systems, contributing to a more secure digital environment.