Article

Carlos Eduardo Frantz Manchini; Thiago Gentil Ramires

DOI: 10.22167/2675-6528-2026M15

Article produced by the ResumeAI tool, an artificial intelligence solution developed by the Pecege Institute focused on synthesis and writing

Summary

Automatic recognition of emotions in voice has proven essential for healthcare applications, enabling the identification of emotional states without continuous human supervision. This work aimed to compare traditional machine learning models with a Generative Artificial Intelligence solution, Hume AI, in the task of inferring emotions from the RAVDESS audio database. For the conventional approach, acoustic features were extracted from 1440 audios to train classifiers such as SVM and XGBoost. In the generative approach, the same audios were processed by Hume AI’s Expression Measurement-Prosody API. As main results, it was observed that the generative solution easily identified high-intensity emotions, such as anger, but confused more subtle categories. In contrast, traditional models, despite requiring more preprocessing, achieved superior and more balanced performance across all evaluated classes, with the Support Vector Machine classifier reaching 61.8% accuracy on the test set, while the Hume API obtained 29% global accuracy and 0.26 F1-Macro. It was concluded that supervised models remain more accurate and robust for capturing the emotional spectrum and classifying emotions in audio, highlighting a trade-off between the simplicity of “ready-to-use” solutions and the precision required for detailed emotional analysis.

Keywords: Supervised learning; Audio classification; Prosody; Emotion recognition.

1. Introduction

Recent advances in Data Science and Artificial Intelligence (AI) have transformed the approach to complex problems, especially in the analysis and interpretation of unstructured data. This data, which includes texts, images, and audios, constitutes the majority of information generated today, with eighty percent of new business data being unstructured (Gartner, 2020). Extracting insights from this volume is fundamental for competitiveness, enabled by machine and deep learning (Goodfellow et al., 2016).

The application of these innovations, notably Generative Artificial Intelligence (GenAI) and large language models (LLMs), has generated benefits in various areas. In healthcare, generative models assist in diagnosis and treatment (Topol, 2019), with accuracy superior to ninety-nine percent in cancer (Sheakh et al., 2024). In security, AI enhances facial recognition and fraud detection (Li et al., 2020). In education, GenAI creates virtual assistants and adaptive materials (Luckin et al., 2016).

In audio processing, AI has been employed in voice assistants and automatic translation, enabling more natural interactions between humans and machines (Tan et al., 2021). In this context, automatic speech emotion recognition (SER) is a crucial area, although challenging due to the complexity of data and human emotions. Akçay and Oğuz (2020) indicate that SER is based on distinct acoustic patterns, such as variations in intonation, rhythm, intensity, and vocal quality, modeled by machine learning. With the advancement of GenAI models based on deep neural networks, more sophisticated solutions for processing and interpreting audio signals are being explored (Purwins et al., 2019).

This study focuses on the comparison between traditional Machine Learning methods and generative AI models for the identification of emotions in audio. Initially, the use of the CORAA – Corpus of Annotated Audios (Alvim et al., 2022) dataset, with recordings in Portuguese, was considered. However, CORAA presented significant limitations, such as the absence of detailed labels and the lack of diversity in emotional samples.

Due to these limitations, the RAVDESS dataset – Ryerson Audio-Visual Database of Emotional Speech and Song (Livingstone and Russo, 2018), widely used in emotion detection research, was chosen. This choice allowed for a detailed comparative analysis between traditional approaches (acoustic feature extraction) and generative models (direct processing of recordings). Although deep learning models (foundation of GenAI) outperform conventional techniques in SER (Khare et al., 2024), challenges such as interpretability and the need for large data volumes persist (Latif et al., 2020). This scenario justifies the investigation of the advantages and disadvantages of generative models, considering accuracy, robustness, computational efficiency, and generalization.

The motivation for this research stems from the growing use of generative models in natural language processing and automatic speech recognition, due to their ability to identify emotions via audio. These advances offer practical opportunities, such as improving conversational interfaces, creating emotional support tools, and early clinical diagnosis, particularly in mental health. Subtle changes in vocal tone can indicate early symptoms like stress or anxiety, enabling preventive interventions (Cummins et al., 2015). In customer service, real-time emotion detection optimizes interactions (Poria et al., 2019).

This study seeks to contribute to the academic literature by providing a rigorous comparative analysis between traditional methods and generative models in audio emotion detection. The results can serve as a practical reference for researchers and professionals, driving AI-driven emotional recognition. Thus, the objective of this work is to compare traditional machine learning models with a Generative Artificial Intelligence solution, Hume AI, in the task of inferring emotions from the RAVDESS audio dataset.

2. Material and Methods

This study was characterized as an applied experimental research, with a quantitative approach, which comparatively evaluated the performance of two distinct approaches in audio emotion recognition. The first approach employed traditional Machine Learning techniques, based on feature engineering. The second, more recent, used a Generative Artificial Intelligence solution, based on natural language processing models.

For the research conduction, initially the use of the CORAA dataset – Corpus of Annotated Audios (Alvim et al., 2022), which contains recordings in Brazilian Portuguese, was considered. However, due to limitations such as the absence of detailed labels and the lack of diversity in emotional samples, the RAVDESS dataset – Ryerson Audio-Visual Database of Emotional Speech and Song (Livingstone and Russo, 2018) was chosen. This corpus, widely adopted in emotion detection research, has 1440 audios from 24 people, with 60 variations of speech and song, categorized into eight emotions: anger, calm, disgust, fear, happiness, neutral, sadness, and surprise.

All audios underwent preprocessing to ensure the consistency of the comparative analysis and data quality. Steps included noise reduction, standardization of sound signals, and adjustment of volume variations, aiming for data uniformity. According to Cowie et al. (2001), human speech contains prosodic information, such as pitch, intensity, and rhythm, which are essential for inferring emotions, justifying audio cleaning to optimize the detection of relevant patterns.

In the traditional approach, feature extraction for training Machine Learning and classification algorithms followed the practices described by Akçay and Oğuz (2020). The main acoustic features extracted included Mel-Frequency Cepstral Coefficients (MFCCs), which represent spectral structure; Chroma Features, for voice intonation and resonance; Zero-Crossing Rate (ZCR), which captures signal changes; and intensity patterns, such as RMS energy and fundamental frequencies (pitch F0). Additionally, measures such as Spectral Contrast, spectral descriptors (Centroid, Bandwidth, and Roll-off), and temporal variations of MFCC coefficients (delta MFCCs) were incorporated.

With the extracted features, different supervised learning algorithms were tested for emotion classification, including Random Forest, Support Vector Machines (SVM), and Artificial Neural Networks (Multi-Layer Perceptron – MLP), following the methodology of Schuller et al. (2011). The selection of the best model was based on performance metrics, and k-fold stratified cross-validation was employed to ensure the robustness of the results. The RAVDESS dataset was stratificately divided into training (70%) and testing (30%) sets.

For attribute selection, we sought to maintain the most relevant characteristics and avoid multicollinearity problems. A Pearson correlation analysis indicated a strong correlation between the Chroma variables, which were replaced by three principal components via PCA, preserving 88% of the explained variability. Redundancy was identified among other characteristics, and the final selection was based on explanatory importance by XGBoost, aiming for a parsimonious model.

In the generative approach, the Whisper automatic speech recognition system (OpenAI, 2022) was initially considered. However, this tool was discarded because it was limited to textual transcription, not including crucial intensities and acoustic details for emotional analysis. The Hume AI (2025) model was then adopted, which processed and inferred emotions directly from the recordings. This solution uses proprietary deep learning models, trained on large volumes of multimodal data, without the need for manual feature extraction.

For the processing of the 1440 audios by the Hume Expression Measurement – Prosody API, the files were divided into 15 batches, respecting the upload limitations of the cloud service. The API returned scores associated with different emotional categories, allowing for the evaluation of the relative probability of each affective state. To enable comparison with RAVDESS, which has eight emotional categories, an ontological mapping of Hume AI labels to RAVDESS classes was performed, defining the correspondences: boredom for neutral, amusement for happiness, and excitement for surprise.

The implementation of the experiments was carried out using the Python language, with the aid of specialized libraries. The Librosa library was employed for the extraction of acoustic features, pydub for manipulation and conversion of audio files, and Scikit-learn for classification algorithms, evaluation, and metric calculation. For the evaluation of the generative approach, the Hume Expression Measurement API was used, accessed via HTTP calls integrated with Python. Performance metrics, such as accuracy, precision, recall, and F1-score, were calculated with the support of the scikit-learn library. Visualizations, such as confusion matrices, were generated with Matplotlib and Seaborn. For statistical grounding, significance tests between models, such as Wilcoxon/Mann-Whitney or Friedman, were applied.

3. Results and Discussion

The initial stage of the study focused on defining and selecting the most suitable database for the comparative analysis of emotion recognition in audio. Initially, the use of the CORAA corpus, which contains recordings in Brazilian Portuguese, was considered. However, CORAA presented significant limitations, such as the absence of detailed emotion labels, a restricted variety of emotional categories (neutral, non-neutral male, and non-neutral female), and the lack of a concrete benchmark for quantitatively measuring the assertiveness of classifications. These deficiencies made a robust and refined comparative analysis between different emotions unfeasible, which led to the choice of the RAVDESS database.

The RAVDESS dataset – Ryerson Audio-Visual Database of Emotional Speech and Song (Livingstone and Russo, 2018) was selected for offering a richer and more balanced corpus, with eight categorized emotions and an expert ground truth. The included emotions are angry, calm, disgust, fearful, happy, neutral, sad, and surprised. The database consists of 1440 audios, recorded by 24 individuals, with 60 variations covering narrative, intensity, emotion, and repetition. This choice allowed for a more precise and consistent evaluation of traditional machine learning approaches and the Generative Artificial Intelligence solution.

The exploratory data analysis was fundamental to understanding the behavior of acoustic features and identifying those with greater discriminative capacity between emotions. The first graphical visualizations, using Mel-Frequency Cepstral Coefficients (MFCC) and Log-Mel spectrograms, revealed significant differences, especially between the most extreme emotions, such as sadness and anger. It was observed that, for the emotion of anger, the frequencies were more pronounced and dispersed compared to sadness, indicating distinct acoustic patterns that could be explored by the classification models.

To enrich the set of candidate features and capture additional acoustic information, several measures were included. Among them, RMS energy (Root Mean Square Energy), which quantifies sound intensity in decibels, and Spectral Contrast, which measures the energy difference between frequencies, stand out. Also incorporated were the Zero Crossing Rate (ZCR), which indicates the signal change rate, and spectral descriptors such as Centroid, Bandwidth, and Roll-off. Features related to the fundamental pitch (F0), such as its mean and variance, were calculated to capture the intonation and melody of speech. Finally, temporal variations of MFCC coefficients, the delta MFCCs, were added to record changes in voice intensity and timbre over time, as per the practices described by Akçay and Oğuz (2020).

The feature selection step was crucial to keep only the most relevant variables for emotion classification and mitigate multicollinearity issues. Pearson correlation analysis revealed that Chroma variables were strongly correlated with each other, which was expected due to the cyclical nature of musical notes. To resolve this redundancy, the 12 original Chroma features were replaced by three principal components (PCA), preserving 88% of the explained variability. Furthermore, strong redundancy was identified between other pairs of features, with absolute correlation above 0.8 and positive correlation above 0.9 for two specific pairs. In these cases, we chose to keep two of the four most important variables, based on their explanatory relevance by XGBoost, in order to obtain a more parsimonious model.

The comparison between the full model and the reduced feature model, using the XGBoost algorithm, demonstrated the effectiveness of the selection. The full model, with 81 features, achieved an accuracy of 0.660 and an F1-Macro of 0.648. In contrast, the reduced model, with only 23 features (a reduction of 71.6%), obtained an accuracy of 0.657 and an F1-Macro of 0.635. The difference of only 0.013 in F1-Macro, equivalent to approximately 2%, indicates that the feature selection approach based on XGBoost importance allowed for the construction of a more parsimonious and computationally efficient model, without significantly compromising emotional recognition capability. This evidence supported the decision to proceed with the compact approach for data modeling.

Statistical modeling was performed with three classic supervised learning algorithms: Support Vector Machines (SVM), XGBoost, and Random Forest. The RAVDESS dataset was stratified into training and testing sets, using 70% and 30% of the data, respectively. The performance metrics evaluated were accuracy and F1-Macro, the latter being chosen for its robustness in multiclass and potentially imbalanced problems, ensuring a more balanced evaluation across emotional categories. The results indicated a moderate overall performance of the models, with SVM standing out.

The detailed results of the Machine Learning models revealed that SVM achieved a training accuracy of 0.852 and a test accuracy of 0.618, with a training F1-Macro of 0.848 and a test F1-Macro of 0.607. XGBoost presented a training accuracy of 0.910 and a test accuracy of 0.567, with a training F1-Macro of 0.911 and a test F1-Macro of 0.545. The Random Forest, on the other hand, had a training accuracy of 0.851 and a test accuracy of 0.458, with a training F1-Macro of 0.851 and a test F1-Macro of 0.443. Greater overfitting was observed in the Random Forest and XGBoost models, indicating the need for additional hyperparameter adjustments to improve performance. The superiority of SVM was evident, mainly due to its stability and predictive generalization capability on the test set.

The analysis of the confusion matrix for the SVM model revealed the assertiveness of the predicted classification of emotions in relation to the real emotions. Most of the values were concentrated on the main diagonal, indicating a good number of true-positives for each class. However, the model demonstrated difficulty in distinguishing emotions that share similar acoustic characteristics. High-intensity emotions, such as calm and surprise, were recognized more easily. On the other hand, feelings with closer intensities, such as neutral, calm, and sadness, showed overlap and generated frequent exchanges in classifications, evidencing the complexity of differentiating subtle emotional nuances through acoustic descriptors.

Within the scope of the generative pipeline, the Hume Expression Measurement API was incorporated to process the 1440 audios. To enable sending and processing in the cloud service, the audios were divided into 15 batches, respecting the sending limitations imposed by the API. Initially, the use of the Whisper system (OpenAI, 2022) for automatic speech recognition was considered, but it was discarded because it is limited to textual transcription and does not consider intensities and acoustic details essential for emotional analysis. The strategy of transcribing the audio and processing the text with a Large Language Model (LLM) was considered inadequate, as it could sacrifice prosodic and affective nuances, which are crucial for emotion detection. Thus, Hume AI’s (2025) solution was adopted, which uses proprietary deep learning models trained on large volumes of multimodal data to detect emotional states directly from recordings, without the need for manual feature extraction. The API returns scores associated with different emotional categories, allowing the relative probability of each affective state to be assessed.

A crucial aspect for the comparison between the approaches was the difference in the emotional taxonomies used. While the RAVDESS corpus categorizes emotions into eight well-defined labels, the Hume API returns predictions in a distinct set of labels, including terms such as *boredom*, *amusement*, and *excitement*. To allow for a fair quantitative evaluation, it was necessary to build an ontological mapping, semantically approximating Hume’s labels to RAVDESS’s classes. The established correspondences were: *boredom* to neutral, *amusement* to happiness, and *excitement* to surprise. This harmonization was essential to enable direct comparison of Hume’s performance with the reference set.

After applying the category mapping, a partial improvement was observed in Hume AI’s results. Emotions of higher intensity, such as anger, showed expressive performance, with an F1-score of 0.65. However, subtler classes, such as fear and disgust, obtained F1 values close to zero. This indicates that part of Hume’s initial low accuracy stemmed from misalignment between categories, but also that, even after harmonization, the generative model still has limitations in capturing more discrete emotional nuances. Hume AI’s confusion matrix showed a tendency to concentrate predictions on more neutral classes and a greater difficulty in correctly identifying less intense emotions, such as fear and disgust, which were widely confused with other categories, including closer states like calm and neutral.

Comparatively, the supervised models (SVM, XGBoost, and Random Forest), according to the presented results, achieved an average accuracy of 55% and an F1-Macro of 0.53 on the test set. These values indicate a reasonable capability in recognizing emotions in audio. In contrast, experiments with the Hume API resulted in an overall accuracy of only 29% and an F1-Macro of 0.26, even after the ontological mapping of categories. This data confirms the inferiority of the generative solution when directly applied to the reference corpus, showing that the Hume API correctly classified just over one in five samples, while traditional approaches based on acoustic descriptors achieved between half and two-thirds of correct predictions.

Although supervised classifiers demanded greater effort in data preparation, feature selection, and hyperparameter tuning, they achieved superior metrics for capturing the diversity of emotions. These results reinforce that “off-the-shelf” solutions, such as Hume AI, can be a quick alternative when the interest is limited to intense emotions, but they do not replace the consistency and precision of models adjusted to capture a more detailed emotional spectrum. Additionally, the use of the Hume API incurred a financial cost of approximately US$3.60 for the complete processing of the 1440 audios, a relevant factor to consider in the scalability of the solution.

In summary, the research demonstrated that, although Generative Artificial Intelligence solutions like Hume AI offer simplicity and speed in processing, traditional machine learning models, especially SVM, showed superior and more balanced performance in classifying a broader spectrum of emotions in audio. The generalization capability and accuracy of supervised models, even with the need for preprocessing and feature engineering, make them more robust for applications demanding detailed and reliable emotional analysis, while Hume AI proved more effective for high-intensity emotions but limited for subtle nuances.

4. Conclusion

This study aimed to compare traditional machine learning models with a Generative Artificial Intelligence solution, Hume AI, in the task of inferring emotions from the RAVDESS audio dataset. It was found that supervised models, such as Support Vector Machine (SVM), showed superior and more balanced performance in classifying a broad spectrum of emotions, achieving 61.8% accuracy on the test set. Although they required greater effort in preprocessing and feature engineering, these models demonstrated greater robustness and generalization capability. In contrast, Hume AI’s generative solution, despite offering simplicity and speed, achieved an overall accuracy of only 29% and an F1-Macro of 0.26, highlighting its limitation in capturing subtle emotional nuances, even after ontological mapping of categories. The main contribution of this work lies in the rigorous comparative analysis, which provides a practical reference for researchers and professionals in choosing more effective approaches for emotional recognition in audio, driving advancements in applications such as early clinical diagnosis in mental health.

The research revealed a significant trade-off between the simplicity of “out-of-the-box” solutions and the precision required for detailed emotional analysis. It was observed that Hume AI proved more effective for high-intensity emotions, such as anger, but failed to distinguish more discrete categories, such as fear and disgust. Traditional models, in turn, showed greater stability and predictive capacity. As limitations, the need for additional adjustments to the hyperparameters of the Random Forest and XGBoost models to mitigate overfitting was identified. For future studies, it is suggested to investigate hybrid approaches that combine the agility of generative models with the robustness of supervised classifiers, exploring the use of AI-provided embeddings to train locally adjusted models. Additionally, fine-tuning generative solutions and exploring more heterogeneous Brazilian audio databases are recommended to evaluate generalization capability in real-world scenarios.

Bibliographic References

Akçay, M.B.; Oğuz, K. 2020. Speech emotion recognition: emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication 116: 56-76.

Alvim, G.; Magalhães, L.L.C.; Bigal, R.L.M.; Medeiros, H.F.G.; Souza, S.R.M.; Silva, C.F.M.; Oliveira, E.P.L.; Pardo, S.R.C. 2022. CORAA: a large corpus of spontaneous and prepared speech manually validated for speech recognition in Brazilian Portuguese. In: International Conference on Language Resources and Evaluation, 2022, Marseille, França. Anais… p. 5908-5915.

Cowie, R.; Douglas-Cowie, E.; Tsapatsoulis, N.; Votsis, G.; Kollias, S.; Fellenz, W.; Taylor, J.G. 2001. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18(1): 32-80.

Cummins, N.; Scherer, S.; Krajewski, J.; Schnieder, S.; Epps, J.; Quatieri, T.F. 2015. A review of depression and suicide risk assessment using speech analysis. Speech Communication 71: 10-49.

Gartner [GARTNER]. 2020. Market guide for text analytics. Disponível em: . Acesso em: 24 mar. 2025.

Goodfellow, I.; Bengio, Y.; Courville, A. 2016. Deep Learning. The MIT Press, Cambridge, MA, EUA.

Hume Al. 2025. Expression Measurement Prosody. Disponível em: . Acesso em: 25 set. 2025.

Khare, S.K.; Blanes-Vidal, V.; Nadimi, E.S.; Acharya, U.R. 2024. Emotion recognition and artificial intelligence: a systematic review (2014–2023) and research recommendations. Information Fusion 102: 102019.

Latif, S.; Rana, R.; Qadir, J.; Epps, J.; Schuller, B.W. 2020. Deep representation learning in speech processing: challenges, recent advances, and future trends. Computer Speech & Language 68: 101-178.

Li, Y.; Schuckert, M.; Law, R.; Wang, J. 2020. The impact of artificial intelligence on security and privacy in smart cities. Journal of Urban Technology 27(2): 65-85.

Livingstone, S.R.; Russo, F.A. 2018. The ryerson audio-visual database of emotional speech and song (RAVDESS). PLoS ONE 13(5): e0196391.

Luckin, R.; Holmes, W. 2016. Intelligence Unleashed: An Argument for Al in Education. Pearson, London, UK.

OpenAl. 2022. Whisper: robust speech recognition via large-scale audio training. Disponível em: . Acesso em: 29 mar. 2025.

Poria, S.; Majumder, N.; Mihalcea, R.; Hovy, E. 2019. Emotion recognition in conversation: research challenges, datasets, and recent advances. IEEE Access 7: 100943-100953.

Purwins, H.; Li, B.; Virtanen, T.; Schlüter, J.; Chang, S.Y.; Sainath, T. 2019. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing 13(2): 206-219.

Schuller, B.; Batliner, A.; Steidl, S.; Seppi, D. 2011. Recognizing realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge. Speech Communication 53(9-10): 1062-1087.

Sheakh, M.A.; Azam, S.; Tahosin, M.S.; Karim, A.; Montaha, S.; Fahim, K.U.; De Boer, F. 2024. ECgMLP: a novel gated MLP model for enhanced endometrial cancer diagnosis. Computer Methods and Programs in Biomedicine Update 5: 100181.

Tan, X.; Qin, T.; Soong, F.; Liu, T.-Y. 2021. A survey on neural speech synthesis. Disponível em: . Acesso em: 27 set. 2025.

Topol, E. 2019. Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. Basic Books, New York, NY, EUA.

Article from the Final Course Project of the Specialization in Data Science and Analytics from the MBA USP/Esalq

_{Learn more about the course; click here:}