September 30, 2026
Transfer learning models for image falsification detection in faces
Transfer Learning Models for Face Image Forgery Detection
Bianca Cavalcante Basso; Jorge Costa Silva Filho
DOI: 10.22167/2675-6528-202602724
Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.
Summary
The advancement of Artificial Intelligence (AI) has consolidated its application in the generation of synthetic media, such as deepfakes, frequently employed in disinformation and fraudulent practices. This work evaluated the effectiveness of pre-trained transfer learning models in distinguishing between real and synthetic images, specifically deepfakes. For this purpose, the Celeb-DF v2 dataset, composed of real and synthetic videos, was used. Preprocessing was performed for frame extraction and image resizing, followed by downsampling to balance the classes, totaling 18,358 images. The ResNet50, VGG16, and MobileNetV2 models were trained for 10 epochs, with their last convolutional layers fine-tuned and the first ones frozen, and subjected to internal validation tests and an independent test set. Internal validation results indicated high performance, with ResNet50 achieving an AUC of 0.8762, MobileNetV2 0.8279, and VGG16 0.7625. However, in independent tests, the models presented a generalization gap, with accuracies stabilizing close to 50%. It was concluded that, although data rebalancing and transfer learning are effective for learning specific features, robust deepfake detection in real-world scenarios requires models that overcome texture dependency and generalize synthesis patterns, going beyond nominal performance in controlled environments.
Keywords: Computer vision; Deepfake; Artificial intelligence; Machine learning; Transfer learning.
1. Introduction
Artificial Intelligence (AI) has been consolidating in various sectors, including the generation of synthetic media such as images, videos, and audios (MONTAÑO, 2023). In this scenario, deepfakes emerge as media generated or altered by deep learning algorithms that simulate real content.
Despite its potential for legitimate uses, this technology is frequently employed in the propagation of disinformation and fraudulent practices (KAUFMAN, 2022). Brazil, for example, ranks third globally in social media consumption (Forbes, 2023), which intensifies the challenges related to verifying the authenticity of digital content and favors the dissemination of deepfakes (JUNIOR, 2023).
Deepfakes are created through deep learning techniques, a branch of AI that relies on multi-layered neural networks (WESTERLUND, 2019). Among these, Convolutional Neural Networks (CNNs) stand out, widely used in image processing. CNNs operate with layers such as Convolution, Rectified Linear Units (ReLU), and Max-Pooling, which extract visual features and reduce data dimensionality (SILVA, 2018; JUNIOR, 2023).
The versatility of these architectures allows for digital manipulations with a high degree of realism, making it difficult to distinguish between real and fake content (WESTERLUND, 2019). Although there are beneficial applications, the use of deepfakes has been diverted to the dissemination of disinformation, generating negative social impacts (JUNIOR, 2023). The production of media of dubious provenance, combined with the speed of sharing, consolidates mistaken perceptions of veracity in the public. A notorious example was the manipulation of a video involving Dr. Drauzio Varella for fraudulent promotion, evidencing risks to moral integrity and reputation (MACHADO, 2023; JUNIOR, 2023).
The availability of free tools for algorithm development, combined with the practicality of Transfer Learning, has democratized access to Artificial Intelligence. However, this accessibility also acts as a catalyst for the increase of deepfakes, as individuals with malicious intentions can employ these resources to create disinformation (JUNIOR, 2023). The literature suggests that the advancement of AI should be accompanied by ethical principles and meticulous inspection of the generated results, aiming to mitigate the risks of disinformation (KAUFMAN, 2022).
Given this scenario, the effective detection of deepfakes becomes crucial for the security and authenticity of digital information. The complexity of high-fidelity manipulations demands robust models capable of distinguishing between real and synthetic media, overcoming generalization challenges. Therefore, this case study aims to compare the performance of three pre-trained models (Transfer Learning), namely ResNet50, VGG16, and MobileNetV2, in identifying deepfakes, using a dataset composed of real and synthetic media.
2. Material and Methods
The present study adopted a quantitative and experimental methodological approach, focused on the implementation and evaluation of machine learning models for deepfake detection. The research consisted of applying Transfer Learning techniques, using pre-trained convolutional neural network architectures to classify images as real or synthetic. The objective was to compare the performance of these models in identifying digital manipulations in faces.
For the development, the Celeb-DF v2 dataset (A Large-scale Challenging Dataset for DeepFake Forensics) was employed, recognized as a high-quality benchmark (LI et al., 2020). This set comprises 5,639 AI-manipulated videos and 590 original videos, collected from the YouTube platform, presenting diversity in ethnicity, age, and gender.
The preprocessing involved extracting frames from videos, converting them into static data for analysis by computer vision models. The MTCNN library was used for facial detection and OpenCV (cv2) for frame manipulation. Sampling consisted of extracting one frame every fifteen seconds, with a margin factor of 20% on the facial crop.
The images were resized to 224 x 224 pixels, a standard compatible with Transfer Learning architectures, converted to RGB, and exported as .jpg. The final dataset, after preprocessing, totaled 98,482 images (9,179 “real” and 89,303 “synthetic”).
To mitigate the class imbalance, the downsampling technique was applied to the majority class. This process resulted in a balanced training set of 18,358 images, with a 1:1 ratio between classes. `shuffle` was applied immediately after downsampling to prevent the model from learning patterns based on the file reading order.
The development environment was the Google Colab platform, operating in a Jupyter Notebook. The cloud-based infrastructure provided graphical processing unit (GPU) and tensor processing unit (TPU) resources, essential for training deep neural networks. File storage and management were carried out via integration with Google Drive.
For the modeling, three pre-trained Convolutional Neural Network (CNN) architectures were selected for Transfer Learning: ResNet50, VGG16, and MobileNetV2. These architectures were chosen for their representativeness and robustness in computer vision tasks (SIMONYAN; ZISSERMAN, 2014; HE et al., 2016; SANDLER et al., 2018). VGG16 evaluated classic models, ResNet50 tested the ability to capture subtle details, and MobileNetV2 verified feasibility in resource-limited scenarios.
The training strategy consisted of freezing the initial convolutional layers of the models (`trainable = False`), leveraging the already optimized low-level feature extractors. The original output layer was replaced by a new binary classification structure, composed of GlobalAveragePooling2D, a Dense layer with 128 neurons and ReLU activation, and a Dropout layer with a rate of 0.5 for regularization. The final output layer used one neuron with a Sigmoid activation function.
The model compilation employed the Adam optimizer and the Binary Crossentropy loss function. After rebalancing, the dataset was split into 80% for training and 20% for validation and testing. Training was performed for 10 epochs. To ensure reproducibility and balanced class distribution in each training batch, the `image_dataset_from_directory` class from TensorFlow was used, configured with a batch size of 32 and shuffle enabled.
To assess the effectiveness of the models, statistical metrics based on the Confusion Matrix were adopted. The metrics included Accuracy, Precision, Recall (Sensitivity), F1-Score, ROC Curve, and Area Under the Curve (AUC), which measured the discrimination capacity between real and synthetic classes (PRATI; BATISTA; MONARD, 2008). The Confusion Matrix was used to visualize the distribution of hits and misses (TP, TN, FP, FN).
3. Results and Discussion
The results obtained in this study reveal a complex panorama regarding the effectiveness of transfer learning models in deepfake detection, evidencing a notable contrast between performance in internal validation and generalization capability in independent tests. Initially, the ResNet50, VGG16, and MobileNetV2 models demonstrated high performance in a controlled environment, after a rigorous process of preprocessing and rebalancing the Celeb-DF v2 dataset. However, when subjected to a more challenging test set, the models presented a significant drop in accuracy, stabilizing around 50%, which points to a crucial generalization gap for the discussion of the robustness of these systems in real-world scenarios.
Data preparation was a fundamental step to mitigate algorithmic bias and optimize the dataset. The MTCNN library was used for facial detection and OpenCV for frame manipulation, extracting one frame every interval proportional to the average duration of the videos, with a 20% margin to preserve facial features. The images were resized to 224 x 224 pixels, a standard compatible with the selected computer vision architectures. In the end, the main dataset contained 98,482 images, with 9,179 real and 89,303 synthetic.
To address the significant imbalance between the “Real” and “Fake” classes in the original dataset, the downsampling technique was applied. This process selectively reduced samples from the dominant class until both categories reached a 1:1 proportion, resulting in a balanced set of 18,358 images for training. This strategy was crucial to ensure that accuracy and precision metrics reflected the models’ true distinguishing capability, rather than just a statistical trend of the dataset, forcing the networks to learn real morphological patterns and manipulation artifacts.
The training strategy involved freezing the initial convolutional layers of the pre-trained ImageNet models (trainable = False), leveraging their optimized weights for low-level feature extraction. The original output layer was replaced by a new binary classification structure, consisting of GlobalAveragePooling2D for dimensionality reduction, a Dense Layer with 128 neurons and ReLU activation function, Dropout of 0.5 for regularization, and an output layer with 1 neuron and Sigmoid activation function. The compilation used the Adam optimizer with a learning rate of 0.001 and the Binary Crossentropy loss function, with the dataset split into 80% for training and 20% for validation and testing.
Performance of Architectures
After applying the downsampling technique and training for 10 epochs, the models exhibited significantly more robust predictive behavior in the internal validation phase. The ResNet50 architecture stood out with the best absolute performance, achieving an AUC of 0.8762. This value indicates a high capacity for correct distinction between real and synthetic classes, surpassing the other evaluated architectures. ResNet50 also obtained an accuracy of 0.7768, precision of 0.8320, recall of 0.7109, and F1-Score of 0.7667, with 506 True Negatives and 526 False Negatives.
The MobileNetV2 architecture, although with a slightly lower AUC (0.8279) than ResNet50, demonstrated a better balance between Precision (0.7443) and Recall (0.7420). This balance suggests that the model is more consistent in handling False Positive and False Negative errors, maintaining stable performance on both fronts. MobileNetV2 achieved an Accuracy of 0.7362, precision of 0.7443, recall of 0.7420, and F1-Score of 0.7431, with 470 True Positives and 478 False Negatives, indicating a higher sensitivity in identifying manipulated content.
In contrast, the VGG16 model presented the most modest performance among the three architectures, with an AUC of 0.7625 and an F1-Score of 0.6901. Its accuracy (0.6907), precision (0.7160), and recall (0.6659) rates were the lowest, suggesting that its structure may require a larger number of epochs or more refined hyperparameter adjustments for this specific dataset. The comparison between the preliminary and current results confirmed that the suppression of the majority class bias was decisive for the stabilization of metrics around 70-80% in accuracy and AUC, validating the effectiveness of Transfer Learning for the task of facial digital forensics.
The evolution of the ResNet50 model’s performance metrics during training, over the 10 epochs, demonstrated progressive convergence. The AUC index, for example, jumped from 0.59 in the first epoch to 0.86 in the tenth epoch. This continuous improvement indicates that the network was able to extract relevant features from the provided sample, learning to distinguish patterns more effectively as training progressed. Intermediate results in the fifth epoch already showed an AUC of 0.8202, confirming the model’s learning capability.
Generalization Gap Analysis
Despite the promising performance in internal validation, submitting the models to an independent test set revealed a sharp drop in performance, with metrics stabilizing around 0.50. This phenomenon, known as generalization gap, suggests a limitation in the models’ ability to apply knowledge acquired on test data that exhibits variations not observed during training. The test accuracy of ResNet50 was approximately 50.4% (920 correct predictions out of 1824 samples).
For the ResNet50 model, the analysis of the confusion matrix on the independent test indicated 506 True Negatives (real images correctly classified) and 526 False Negatives (deepfakes classified as real). The latter is the most critical error in a security scenario, as it means fraudulent media “passed” through the detection filter. The model demonstrated a conservative bias, tending to classify images as real, which can be problematic for threat detection.
The VGG16 model presented a test accuracy of approximately 50.7% (925 correct predictions out of 1824 samples) on the independent dataset. Its confusion matrix showed 469 True Negatives (Real) and 456 True Positives (Fake). Although it demonstrated less bias towards a specific class when compared to ResNet50, VGG16 did not show sufficiently consistent discriminative patterns for a robust separation between real and synthetic data, indicating instability in its classification decisions.
Despite being a lighter and more optimized architecture, MobileNetV2 also presented a challenging performance in the independent test, with an accuracy of approximately 48.5% (884 correct predictions out of 1824 samples). The confusion matrix revealed 414 True Negatives (Real) and 470 True Positives (Fake), but also 462 False Positives (real images classified as fake). This suggests an “alarmist” tendency, where the model easily suspects manipulation, even in the absence of deepfake, which can generate false alerts in practical applications.
Discussions and Analyses
The divergence between high performance in internal validation and low accuracy in independent testing is a widely documented phenomenon in the computer vision literature, known as the generalization gap (Corrêa, 2023). This discrepancy frequently occurs in deepfake detection tasks when the model fails to transfer knowledge acquired from a specific sample to data exhibiting variations in compression and noise not seen during training (Wang et al., 2020). Deep Learning models trained on specific deepfake datasets tend to suffer from over-specialization on the statistical artifacts inherent to the training sample.
The technical explanation for this stagnation of metrics in the test resides in the concept of Shortcut Learning. Convolutional Neural Networks (CNNs) have an intrinsic bias towards textures over the shape or semantics of the object (Geirhos et al., 2019). In the context of this work, it is likely that the Transfer Learning models have identified “shortcuts” in the training images, such as microscopic patterns of digital noise or specific lighting vibrations of the Celeb-DF v2 dataset, which do not represent the essence of genuine facial manipulation. This behavior suggests that the model may have focused on dataset-specific noise patterns, a vulnerability also observed by Junior (2023).
The preprocessing step, including bilinear resizing of images to 224 x 224 pixels, may have functioned as an unintentional smoothing technique. This smoothing might have eliminated synthesis artifacts that would only be noticeable at native resolutions, especially in high-quality deepfakes like those from Celeb-DF v2 (Rössler et al., 2019). The detection of synthetic manipulations is extremely sensitive to image understanding and resizing, and the fixed input resolution of classical architectures imposes a physical limit on the detection of fine details.
Another factor that contributed to the generalization gap was the structure of the training and validation data, performed by the Keras/TensorFlow function `image_dataset_from_directory`. Although this function automates data labeling and separation (80% training, 20% validation), the partition is executed at the individual file (frame) level, ignoring the temporal origin and subject identity contained in the original video. This may have resulted in “face data leakage”, where visually almost identical images were distributed between the training and validation sets, inflating the validation accuracy (88%) and masking the real inability to generalize (Lopes et al., 2026).
The computational infrastructure used, Google Colab, imposed critical limitations on RAM, disk storage, and continuous execution time. These restrictions limited the exploration of hyperparameters and the number of training epochs to only 10. Robust models like ResNet50 and VGG16 often require hundreds of epochs to fine-tune their weights on high-complexity problems (He et al., 2016). The sub-optimization of hyperparameters, due to these limitations, may have prevented the networks from breaking the memorization barrier and achieving true generalization.
The similarity of results between architectures of distinct complexities (ResNet50 and MobileNetV2) in the independent test reinforces the thesis that the obstacle lies not in the computational power of the networks, but rather in the need for data refinement (fine-tuning) and increased diversity of the training sample to break the generalization barrier. The 50% accuracy in the independent test, according to the thesis of Lopes et al. (2026), was a byproduct of identity memorization (overfitting), and not of genuine detection of synthetic artifacts, revealing that automation facilitated by `image_dataset_from_directory`, without rigorous separation by individuals, tends to mask the model’s real inability to generalize when faced with new attacks.
Ultimately, the experimental journey undertaken in this study, from class rebalancing via downsampling to the exhaustive evaluation of Transfer Learning models, reveals that detecting deepfakes in high-fidelity datasets like Celeb-DF v2 transcends simple hyperparameter optimization. The gap observed between the validation accuracy of 88% and the test accuracy of 50% should not be interpreted as a model failure, but rather as a precise technical diagnosis of the limitations of contemporary CNNs. The fallacy of accuracy in controlled environments conceals the fragility of texture artifacts that are often eliminated in preprocessing, as confirmed by the thesis of Lopes, Laroca, and Grégio (2026).
Therefore, the results presented here serve as a methodological alert, indicating that the fight against disinformation requires a paradigm shift, moving the focus from laboratory performance to robustness in real-world scenarios. The reliability of a digital security system lies not in the pursuit of high nominal metrics, but in the model’s ability to generalize synthesis patterns in a way that questions the subject’s identity. This study establishes a solid foundation for future research to incorporate uncertainty analyses and data partitions strictly based on identities, aiming to break generalization barriers and ensure that Artificial Intelligence acts as a truly integral filter in the preservation of biometric truth.
4. Conclusion
This study evaluated the effectiveness of pre-trained transfer learning models in distinguishing between real and synthetic images, specifically deepfakes. It was found that, after preprocessing and rebalancing the Celeb-DF v2 dataset, the ResNet50, MobileNetV2, and VGG16 architectures demonstrated high performance in internal validation, with ResNet50 achieving an AUC of 0.8762. However, when subjected to an independent test set, the models exhibited a significant generalization gap, with accuracies stabilizing around 50%. It was observed that this discrepancy stemmed from shortcut learning, where the models focused on digital noise patterns and specific training dataset textures, rather than genuine manipulation artifacts. Resizing the images to 224 x 224 pixels may also have smoothed or eliminated synthesis artifacts, and partitioning the data at the frame level, rather than by identity, contributed to facial data leakage, inflating validation metrics and masking true generalization capability.
The main contribution of this work lies in methodologically warning that robust deepfake detection in real-world scenarios requires more than hyperparameter optimization or simple weight transfer from generic models. The computational limitations of the development environment restricted the exploration of hyperparameters and the number of training epochs, preventing the networks from breaking the memorization barrier and achieving real generalization. For future studies, data augmentation techniques and fine-tuning of the deeper convolutional layers are suggested. The incorporation of uncertainty analyses and the use of strictly identity-based data partitions are also recommended, aiming to overcome generalization barriers and ensure that Artificial Intelligence acts as an integral filter in preserving biometric truth, effectively combating disinformation through synthetic media.
Bibliographic References
Forbes, 2023 [Referência completa não encontrada no documento original]
Junior, N.; Ediberto do J. 2023. Detecção de deepfakes utilizando redes neurais convolucionais. 46 f. Trabalho de Conclusão de Curso de Graduação em Engenharia de Computação, Campus de Sobral, Universidade Federal do Ceará, Sobral, CE, Brasil. Disponível em: <https://repositorio.ufc.br/handle/riufc/78832>. Acesso em: 31 ago. 2025.
Kaufman, D. 2022. Desmistificando a inteligência artificial. Belo Horizonte, MG, Brasil Autêntica.
MONTAÑO, R. A. N. R.: Inteligência Artificial. 1 ed. Curitiba [PR]: IESDE, 2021.
Silva, R. C. S. 2024. Prevenção contra deepfakes: desenvolvimento de um sistema de reconhecimento facial para diferenciar rostos humanos de rostos gerados por IA. Disponível em: <http://repositorio.ifam.edu.br/jspui/
Westerlund, M. 2019. The Emergence of Deepfake Technology: A Review (Volume 9, Issue 11). Disponível em <https://timreview.ca/article/1282> Acesso 15 out. 2015.
Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq
To learn more about the course, click here and access the MBX Academy platform