Food Detection in Meal Images Using Yolov8 Networks

Luis Felipe de Oliveira Bergamim; Gustavo Dantas Lobo

DOI: 10.22167/2675-6528-202603218

Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

The automatic recognition of food in meal images is a relevant problem in areas such as nutritional monitoring, health, and intelligent systems, but it presents challenges due to the high visual variability of food, overlap between items, and absence of well-defined structures. In this context, the study investigated the feasibility of using computer vision techniques for automatic food detection in meal images. Different variants of the YOLOv8 model were explored with pre-trained weights and progressive adjustments in hyperparameters, architectures, and layer freezing schemes. The FoodRepo dataset was adopted for training, after modifications that reduced and grouped classes, making them more suitable for the scope of the application. Experiments were conducted comparing nano, small, and medium architectures, as well as different optimizers and loss function weights. The results indicated excellent performance in dish detection and moderate performance in food detection, consistent with the visual complexity of the classes and the variability of the dataset. It was observed that reducing classes increased training stability and that partial layer freezing accelerated the process without significant loss of performance. The technical feasibility of using YOLOv8 models for automatic food detection in meal images was demonstrated, contributing to advances in the field of food computing.

Keywords: Food detection; Object detection; Transfer learning; Computer vision; YOLOv8.

1. Introduction

In recent decades, the growing availability of food for a large part of the world’s population has represented a significant advance in the fight against hunger. However, this progress has been accompanied by a significant increase in obesity rates and chronic non-communicable diseases, such as type 2 diabetes and cardiovascular diseases, widely related to the consumption of ultra-processed products (Popkin, 2015; Monteiro et al., 2013). This phenomenon, known as nutritional transition, is characterized by the replacement of traditional diets with eating patterns rich in industrialized and hypercaloric items, posing a global public health challenge.

The United Nations, through the 2030 Agenda, established among its Sustainable Development Goals the commitment to ensure healthy lives and promote well-being for all (UN, 2015). In the Brazilian context, these challenges become even more evident given the growing presence of ultra-processed foods in the population’s diet and the difficulty of accessing simplified tools that help interpret the nutritional quality of meals. In this scenario, technological solutions based on computer vision emerge as promising alternatives to support more conscious food choices.

In this context, automatic food recognition in images has stood out as a promising technology to support different practical applications in the health and nutrition fields. Systems capable of identifying foods from photographs can assist in monitoring food intake, estimating the nutritional composition of meals, tracking patients with chronic diseases, in food education applications, and in epidemiological studies on eating habits. Furthermore, the automation of this process reduces the need for manually recorded food logs, making nutritional assessment faster, more objective, and scalable.

Deep learning models, especially those based on convolutional neural networks, have demonstrated robust performance in image recognition tasks, including applications related to the food sector. Among these approaches, the YOLO (“You Only Look Once”) family of models stands out, widely used for real-time object detection due to the balance between accuracy and speed (Redmon et al., 2016; Varghese & Sambath, 2024). These characteristics make YOLOv8 particularly suitable for applications that demand efficient processing and potential real-time execution, such as automated food recognition systems. The availability of large annotated datasets, like FoodRepo, combined with the advancement of modern deep learning libraries, has facilitated the development of systems capable of automatically identifying food items present in photographs.

The problem that guides this study lies in the difficulty of developing models capable of automatically identifying foods present in meal images with satisfactory precision, given the high visual variability between food items, the overlap between objects, and the large number of possible categories, factors widely recognized as challenging in the literature.

The need for efficient tools for nutritional monitoring and overcoming the challenges inherent in food detection in images justify the relevance of this research. Given this context, the objective of this work is to evaluate the feasibility of using computer vision models based on the YOLOv8 architecture for the automatic detection of food in meal images. To this end, different model configurations were analyzed, including architecture variations, transfer learning strategies, and hyperparameter adjustments, with the aim of identifying the configuration that provided the best balance between performance and computational cost in the food detection task.

2. Material and Methods

The research was conducted with an applied and experimental character, employing quantitative methods for the development of a computer vision approach. The study focused on the automatic detection of food in images, seeking to evaluate the technical feasibility of object detection models applied to the recognition of meal components. The construction and evaluation of a food detection model in images constituted the exclusive focus of this work.

The methodological process was structured in sequential stages, covering from the bibliographic survey and data preparation to the experimental training and evaluation of the food detection models. This systematic approach allowed for an in-depth investigation of the different configurations and parameters that influence the performance of computer vision models.

Initially, a literature review was conducted on automatic food recognition, computational nutrition, and machine learning. The objective was to theoretically ground the research and guide the selection of the techniques employed. The review highlighted the effectiveness of YOLO family models for real-time object detection (Redmon et al., 2016; Jocher et al., 2023) and the relevance of “transfer learning” and “fine-tuning” for training optimization (Ciocca et al., 2021; Varghese & Sambath, 2024).

Subsequently, the dataset was selected and prepared. The public “dataset” FoodRepo was used, containing 54,392 images and 100,256 annotations distributed across 323 food classes. This dataset, already with annotated bounding boxes and segmentations, was adapted to the YOLOv8 format, eliminating the need to create a proprietary “dataset” and following consolidated practices in food classification and segmentation research (Ciocca et al., 2021).

The dataset was inspected and standardized, including resolution adjustments, normalization, and reorganization of the training, validation, and test directories. Additionally, adaptations were made to the “dataset” classes through three progressive grouping schemes (v1, v2, and v3). The objective was to reduce redundancies, minimize class imbalance, and increase training stability.

Each clustering scheme was evaluated in two variants: an automatic one (A), generated by script with predefined rules, and a manual one (M), refined based on visual and nutritional criteria to reduce ambiguities. Six distinct configurations of the “dataset” (v1_A, v1_M, v2_A, v2_M, v3_A, and v3_M) were experimentally evaluated, and the configuration with the best performance was selected for subsequent experiments. “Data augmentation” techniques, such as variations in lighting, rotation, and scaling, were applied according to literature recommendations (Shorten & Khoshgoftaar, 2019).

The transfer learning strategy was implemented by initializing the models with weights previously trained on large object detection datasets. During training, some of the initial layers of the network were kept frozen (“frozen layers”), preventing the update of their weights. This approach aimed to preserve generic visual features, reduce training time, and mitigate the risk of overfitting on specific datasets. Different amounts of frozen layers were evaluated to determine the optimal configuration.

The experiments were carried out in the Google Colab environment, which supports GPU acceleration, allowing the training of models to be executed within the typical computational limitations of accessible environments. Hyperparameter tuning was conducted sequentially and experimentally, changing one factor at a time to isolate the individual impact of each parameter on model performance.

Among the hyperparameters evaluated, the optimizers SGD (“Stochastic Gradient Descent”), Adam, and AdamW were compared. SGD performs gradient-based updates with a fixed learning rate, while Adam and AdamW combine adaptive estimates of gradient moments, with AdamW decoupling weight decay. The automatic configuration of the Ultralytics framework, which corresponds to the use of AdamW, was also considered. The YOLOv8 nano, “small”, and “medium” architectures were compared, differing in the number of parameters, network depth, and computational cost.

The influence of different configurations of the loss function weights (box, cls, and dfl) used by YOLOv8 during training was also evaluated. The “box” parameter controlled the importance of the bounding box localization error, “cls” weighted the classification error of the detected classes, and “dfl” adjusted the regression accuracy of the coordinates. Altering these weights allowed prioritizing specific aspects of training, such as localization or classification accuracy.

The models were trained for 100 epochs, following the default configuration of the YOLOv8 framework. For the food experiments, the total training time was recorded, while for the dishes dataset, due to its lower complexity, the average time per epoch was presented. Model evaluation was performed using widely used metrics in object detection tasks.

The metrics employed included precision, recall, F1-Score, and “mean Average Precision” (mAP). Precision measured the proportion of correct detections relative to the total number of detections, penalizing false positives (Everingham et al., 2015). Recall represented the proportion of actual objects correctly detected, reducing false negatives (Everingham et al., 2015).

The F1-Score, which corresponds to the harmonic mean between precision and recall, was used as a balance metric, being useful in scenarios with unbalanced “datasets” (Goodfellow et al., 2016). The mAP, calculated from individual Average Precisions (AP), evaluated the average performance across all classes. The mAP50 considered correct detections with IoU greater than or equal to 0.50, while mAP50-95 evaluated performance for multiple IoU thresholds, from 0.50 to 0.95, according to the COCO benchmark (Lin et al., 2014; Jocher et al., 2023).

3. Results and Discussion

This section details the experiments conducted with the FoodRepo dataset for the automatic detection of food in meal images, presenting the evaluation of the trained models’ performance. The results obtained provide an in-depth understanding of the viability of the YOLOv8 architecture for this complex task, considering the visual variations and inherent diversity of food items. The analysis covers both quantitative aspects, through performance metrics, and qualitative aspects, by examining the model’s predictions in real-world scenarios, allowing for a comprehensive assessment of the proposed approach’s effectiveness.

Quantitative Results

The quantitative results are presented in accordance with the adopted methodological sequence, allowing for a systematic analysis of the impact of each training strategy on the model’s performance. Initially, the influence of class reduction and clustering of the FoodRepo dataset was evaluated, followed by a comparison between the YOLOv8 architectures, layer freezing, hyperparameter tuning, and finally, the final model configuration. This sequential approach ensures that the effect of each modification can be isolated and understood in relation to the overall study objective.

The first experimental stage focused on evaluating different versions of the FoodRepo dataset reduction, aiming to balance the number of examples per class and minimize redundancy between nutritionally similar categories. Aggregations were performed based on visual and nutritional criteria, grouping foods with similar appearance or low representativeness to increase training stability. Three reduction schemes (v1, v2, and v3) were tested, each with automatic (A) and manual (M) variants, which included grouping items such as grains, cooked vegetables, and other infrequent foods into broader classes, as described in the methodology.

The comparative results of the dataset versions indicated that the v3_M version presented the best overall performance. This configuration achieved an mAP50 of 0.3598 and an mAP50-95 of 0.2705, significantly outperforming the other tested alternatives, which ranged from 0.1366 to 0.2924 for mAP50 and from 0.1048 to 0.2161 for mAP50-95. The superiority of v3_M can be attributed to the higher level of class consolidation, which brought together underrepresented categories into broader food groups, promoting greater stability and representativeness of the data for model training. Due to this performance, the v3_M version was selected for all subsequent training stages, ensuring consistency in the evaluation of other hyperparameters.

After defining the optimized version of the dataset, a comparison was made between the YOLOv8 architectures, specifically the nano, “small”, and “medium” variants. Training was conducted for one hundred epochs, seeking a balance between performance and computational cost for the detection of food and dishes. For food detection, the “small” architecture demonstrated the best compromise, achieving an mAP50 of 0.4043 and an mAP50-95 of 0.3104, with an F1-Score of 0.4361, in a training time of two hours and twelve minutes. Although the “medium” variant presented slightly superior metrics (mAP50 of 0.4350 and mAP50-95 of 0.3360), its training time was more than double, totaling four hours and forty-two minutes, which justified the choice of the “small” architecture for food due to its efficiency.

For dish detection, the nano architecture revealed equivalent or superior performance to larger variants, with an mAP50 of 0.9926, mAP50-95 of 0.8532, and F1-Score of 0.9781, in just thirteen seconds per epoch. The “small” and “medium” architectures presented mAP50-95 of 0.8654 and 0.8209, respectively, with per-epoch times of fifteen and twenty-five seconds. The exceptionally high performance in dish detection is expected, as dishes are large, centralized objects with well-defined edges, which simplifies the detection task. These findings corroborate the literature, which suggests that smaller models tend to generalize well on simpler tasks, while intermediate models are more suitable for more complex multiclass tasks.

Next, the impact of the layer freezing strategy during the transfer learning process was evaluated. For the food dataset, it was observed that mAP50-95 reached 0.309 with one frozen layer and 0.304 with two layers, with the corresponding F1-Score of 0.457 and 0.441. From ten frozen layers onwards, performance began to degrade significantly, reaching 0.224 for mAP50-95 and 0.366 for F1-Score, and dropping to 0.097 and 0.262, respectively, with twenty-two layers. Based on these results, the configuration of two frozen layers was selected as optimal for food detection, as it provided a good balance between model adaptation and the preservation of pre-trained features, avoiding performance loss.

For the dish dataset, layer freezing was also analyzed, revealing that mAP50-95 remained stable around 0.858 with one and two frozen layers, and 0.850 with three layers, with the F1-Score at 0.993, 0.978, and 0.965, respectively. From ten frozen layers onwards, performance began to drop, reaching 0.768 for mAP50-95 and 0.920 for F1-Score, and decreasing to 0.521 and 0.706 with twenty-two layers. The configuration of four frozen layers was chosen for dish detection, as it demonstrated good stability and performance, combined with model simplicity. These results indicate that partial layer freezing is an effective strategy for optimizing training, balancing model specialization with the preservation of pre-existing knowledge.

In the hyperparameter tuning stage, different optimization algorithms were compared. For food detection, the automatic optimizer (AdamW) demonstrated the best performance, with an mAP50 of 0.404, an mAP50-95 of 0.309, and an F1-Score of 0.457. SGD and Adam showed inferior results, with mAP50-95 of 0.290 and 0.279, respectively. For dish detection, the SGD optimizer outperformed the others, achieving an mAP50 of 0.994, an mAP50-95 of 0.864, and an F1-Score of 0.984. The automatic optimizer (AdamW) and Adam obtained an mAP50-95 of 0.858, slightly below SGD. The choice of the appropriate optimizer is crucial for model convergence and performance, and these findings were incorporated into the final configurations.

The impact assessment of altering the weights of the loss functions was also performed. For food detection, modifying the classification loss function weight (cls = 1.75) resulted in an mAP50 of 0.408 and mAP50-95 of 0.310, with an F1-Score of 0.449, indicating a slight improvement in mAP50-95 compared to the default weights. For dish detection, modifying the bounding box loss function weight (box = 9.5) improved the accuracy of the boxes, resulting in mAP50 of 0.993, mAP50-95 of 0.869, and F1-Score of 0.975. These optimizations in loss function weights allowed for refining the training process, prioritizing specific detection aspects to maximize accuracy.

The final configuration selected for the food detection model combined the “small” architecture, the use of two frozen layers, and the automatic optimizer (AdamW), with their respective hyperparameters and loss function weights adjusted. This final model achieved an accuracy of 0.520, recall of 0.425, mAP50 of 0.438, mAP50-95 of 0.344, and F1-Score of 0.467. This performance is considered moderate, reflecting the high visual complexity of the food detection task, characterized by class diversity, item overlap, and lighting variations, factors widely recognized as challenging in the literature (Varghese & Sambath, 2024; Redmon et al., 2016).

For the dish detection model, the final configuration used the nano architecture, with four frozen layers and the SGD optimizer, also with its hyperparameters and weights of the corresponding loss functions. This model achieved an accuracy of 0.975, recall of 1.000, mAP50 of 0.993, mAP50-95 of 0.887, and F1-Score of 0.987. The excellent performance of the dish model demonstrates its high stability and generalization capability for objects with well-defined visual characteristics, which is consistent with the lower intrinsic complexity of the task of identifying a dish compared to detecting multiple foods in a meal.

Qualitative Analysis of Model Predictions

In addition to quantitative metrics, a qualitative evaluation of the predictions from the model trained with the “small” architecture was performed on real meal images. This visual analysis complements the numerical results, allowing observation of the network’s behavior in complex situations and validation of its detection capability in practical scenarios, as recommended by the literature for computer vision models (Ciocca et al., 2021; Kawano & Yanai, 2015). Visual inspection of the predictions offers insights into the model’s strengths and limitations that are not fully captured by the aggregated metrics.

In an example meal consisting of salad, vegetables, and other fresh items, the model correctly identified different components, assigning the classes “salad” and “vegetables” to the corresponding regions. This ability to discriminate multiple food groups in a single dish reinforces the potential of the YOLO-based approach to support automatic food recognition systems in meal images (Varghese & Sambath, 2024; Redmon et al., 2016). In another scenario, with a dish containing salad, meat, and black rice, the model performed satisfactorily by correctly identifying the salad and classifying the black rice as vegetables, consistent with its grouped classes. The model also correctly identified the meat, demonstrating its ability to recognize animal proteins even in situations of visual overlap with other foods.

However, the qualitative analysis also revealed that some minor or partially hidden items received generic classifications. Although this classification is not semantically accurate, it reflects the limitations arising from the class reduction strategy adopted in the dataset, where visually similar foods were grouped to improve frequency per class, as suggested by the literature in scenarios with imbalanced datasets (Goodfellow et al., 2016; Shorten & Khoshgoftaar, 2019). This observation is compatible with the highly variable nature of food and with the challenges described by Bossard et al. (2014) regarding intra-class diversity, which makes it difficult to distinguish specific items within broad categories.

These qualitative analyses reinforce two central points. Firstly, the model demonstrates an adequate generalization capability for broad food groups, exhibiting stability in the detection of salads, vegetables, and protein sources, which is in agreement with results from previous studies (Ciocca et al., 2021). Secondly, the observed limitations, such as the generic classification of specific items, mainly stem from the reduced granularity of classes used during the clustering stage of the dataset, rather than from inherent failures in the YOLOv8 architecture. This highlights the significant impact of data preparation decisions on the model’s final performance, emphasizing the importance of a careful balance between class simplification and specificity preservation for future applications.

In summary, the experiments demonstrated the technical viability of using YOLOv8 models for automatic food detection in meal images. The progressive reduction of dataset classes and the careful selection of the “small” architecture for food and “nano” for plates, along with partial layer freezing and hyperparameter tuning, were crucial for optimizing performance. Although the model achieved moderate performance in food detection due to visual complexity and class diversity, it presented excellent results in plate detection, confirming its ability to identify well-defined objects. The observed limitations are more related to the granularity of the dataset classes than to intrinsic failures of the architecture, indicating that data preparation is a determining factor for the success of the application.

4. Conclusion

This work investigated the feasibility of using computer vision models based on the YOLOv8 architecture for automatic food detection in meal images, given the high visual variability and inherent complexity of the task. Through a systematic experimental evaluation, which explored different variants of the YOLOv8 model, adjustments in hyperparameters, architectures, and layer freezing schemes, the technical feasibility of the approach was verified. The experiments showed excellent performance in dish detection, with mAP50-95 values close to 0.89, and moderate performance in food detection, reflecting the greater complexity of the task. It was observed that the progressive reduction of classes in the FoodRepo dataset contributed to increasing training stability, and partial layer freezing accelerated the process without significant loss of performance. The “small” architecture presented the best balance between performance and computational cost for food detection, while the nano architecture proved sufficient for dish detection. Thus, the study contributes to the field of “food computing” by demonstrating how adequate data preparation and careful selection of hyperparameters are determining factors for optimizing the performance of these models.

Among the identified limitations, the high visual variability of foods and the imbalance between dataset classes stand out, which required grouping poorly represented categories and may reduce the model’s ability to distinguish specific foods. Additionally, the evaluation was performed exclusively with the FoodRepo dataset, and the model’s performance on other datasets or in images obtained under real-world conditions was not investigated. As future work, it is suggested to evolve the developed model into a practical application for automatic meal recognition. This includes developing an application capable of identifying foods in images, estimating nutritional information such as calories and macronutrients, and providing a meal quality assessment, aiming to assist users in dietary monitoring and the promotion of healthier habits.

Bibliographic References

Bossard, L.; Guillaumin, M.; Van Gool, L. “Food-101: Mining Discriminative Components with Random Forests”. In: European Conference on Computer Vision (ECCV), 2014.

Ciocca, G.; Napoletano, P.; Scalfati, S. “Food recognition and leftover estimation for smart diet monitoring”. Pattern Recognition Letters, v. 143, p. 37–43, 2021.

Everingham, M.; Eslami, S. M. A.; Van Gool, L.; Williams, C. K. I.; Winn, J.; Zisserman, A. “The Pascal visual object classes challenge: a retrospective”. International Journal of Computer Vision, v. 111, p. 98–136, 2015.

Goodfellow, I.; Bengio, Y.; Courville, A. “Deep learning”. Cambridge: MIT Press, 2016.

Jocher, G.; Chaurasia, A.; Qiu, J. “Ultralytics YOLOv8”. 2023. Disponível em: <https://github.com/ultralytics/ultralytics>. Acesso em: 4 out. 2025.

Kawano, Y.; Yanai, K. “Food image recognition with deep convolutional features”. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2015.

Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C. L. “Microsoft COCO: Common Objects in Context”. In: Proceedings of the European Conference on Computer Vision (ECCV), 2014.

Monteiro, C. A.; Moubarac, J.-C.; Cannon, G.; Ng, S. W.; Popkin, B. “Ultra-processed products are becoming dominant in the global food system”. Obesity Reviews, v. 14(Suppl. 2), p. 21-28, 2013.

ONU Organização das Nações Unidas. “Agenda 2030 para o desenvolvimento sustentável”. Nova York: ONU, 2015. Disponível em: <https://brasil.un.org/pt-br/sdgs>. Acesso em: 4 out. 2025.

Popkin, B. M. “Nutrition transition and the global diabetes epidemic”. Current Diabetes Reports, v. 15, n. 9, p. 64, 2015.

Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. “You Only Look Once: Unified, Real-Time Object Detection”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

Shorten, C.; Khoshgoftaar, T. M. “A survey on image data augmentation for deep learning”. Journal of Big Data, v. 6, n. 1, p. 60, 2019.

Varghese, R.; Sambath, M. 2024. “YOLOv8: a novel object detection algorithm with enhanced performance and robustness”. In: International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), 2024, Chennai, Índia. Anais… p. 1 6.

Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform