Comparison of Hyperparameter Optimization Strategies in Deep Neural Networks: Investigations Across Multiple Domains

Cleyner Soares Pereira; Everton Gomede

DOI: 10.22167/2675-6528-202602797

Article derived from a Final Course Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

The growing adoption of deep neural networks in various areas has highlighted the importance of hyperparameter optimization for obtaining efficient, stable models with good generalization capacity. The study aimed to compare three hyperparameter optimization strategies applied to representative problems of image classification, sentiment analysis, and financial time series forecasting. To this end, the research adopted a quantitative, exploratory, and experimental approach, with standardized experiments on three distinct databases, associated with neural architectures suitable for each domain. The methodological procedure comprised the controlled execution of search strategies (Hyperopt with TPE, Hyperband with KerasTuner, and genetic algorithms with DEAP), the final training of the best models, and their evaluation on independent data, considering predictive performance, stability, and computational cost. The results showed that there was no universally superior strategy in all analyzed scenarios, with each method presenting specific advantages according to the domain and the evaluation criterion considered. It was found that the choice of optimization strategy depended on the desired balance between predictive quality, processing time, and consistency of results. It was therefore concluded that hyperparameter optimization should be treated as a contextual decision, directly related to the characteristics of the problem and the practical objectives of the application. The research offered an empirical comparison under a standardized pipeline, highlighting the multidimensional nature of the effectiveness of hyperparameter optimization.

Keywords: Computational efficiency; Stability; Generalization; Hyperparameter optimization; Deep neural networks.

1. Introduction

Artificial Intelligence and Data Science have assumed an increasingly relevant role in research and practical applications, driven by the availability of large volumes of data, the advancement of computational power, and the development of more sophisticated machine learning methods. In this scenario, deep learning has consolidated itself as one of the most expressive approaches in the area, especially in tasks such as computer vision, natural language processing, and pattern recognition. Deep neural networks have stood out for their ability to learn complex and hierarchical representations, which has favored their adoption in different application domains (Goodfellow et al., 2016; Aggarwal, 2023).

Despite this advance, the performance of deep neural models does not depend solely on the chosen architecture or the amount of available data. A decisive factor is the adequate definition of hyperparameters, that is, the values that control aspects of the model’s structure and training, such as learning rate, number of layers, number of units per layer, activation functions, and batch size. The literature highlights that these hyperparameters directly influence the complexity, behavior, training speed, and final performance of machine learning algorithms. Consequently, inadequate choices can compromise model generalization, increase computational cost, and reduce training efficiency (Liao et al., 2022; Agrawal, 2021).

Hyperparameter Optimization (HPO) emerged to address this problem, being a search process for configurations that maximize model performance on validation, reducing the dependence on manual trial and error. This process is challenging, as the search space can be broad, heterogeneous, and composed of continuous, discrete, and conditional variables. Furthermore, each evaluation is usually computationally expensive, as it requires partial or complete training of the neural network. For this reason, HPO is often treated as a black-box optimization problem, for which more efficient strategies have been proposed than traditional approaches, such as grid search and random search (Agrawal, 2021; Bischl et al., 2023).

In recent years, various works have been discussing the fundamentals and applications of hyperparameter optimization. Bischl et al. (2023) present a comprehensive overview of the topic, reviewing from simple methods to more advanced approaches, such as evolutionary strategies, Bayesian optimization, and Hyperband, in addition to practical aspects related to performance evaluation, search space selection, and parallelization. From a more specific perspective, Raiaan et al. (2024) show that hyperparameter optimization in convolutional networks is essential for improving performance, stability, and efficiency, highlighting that manual definition of these configurations is laborious and computationally intensive. The authors also observe that many previous studies focus on isolated algorithms, which reinforces the need for broader comparative analyses. Complementarily, Victoria and Maragatham (2021) emphasize that, in deep networks, hyperparameter tuning plays a central role in training performance and that automated methods, such as Bayesian optimization, become especially useful when evaluations are costly. In the same direction, Gomede et al. (2023) discuss the use of Bayesian optimization as an efficient sampling strategy for the hyperparameter space.

Recent reviews indicate that hyperparameter optimization has evolved from simpler approaches to more sophisticated methods, such as Bayesian optimization, Hyperband, and evolutionary strategies, but the literature still presents fewer controlled comparisons between distinct methods, architectures, and domains (Bischl et al., 2023; Raiaan et al., 2024). This gap is relevant because, in real-world applications, achieving good predictive performance is not enough; search efficiency, computational cost, and result stability also need to be considered. Furthermore, empirical studies show that the effect of hyperparameterization can vary depending on the model architecture, the nature of the data, and the performance properties to be optimized (Liao et al., 2022; Ali et al., 2023).

Given this, this work aimed to compare three hyperparameter optimization strategies in deep neural networks – Bayesian optimization via Hyperopt with TPE (Tree-structured Parzen Estimator), Hyperband with KerasTuner, and genetic algorithms with DEAP in three distinct application domains: image classification with CNN on the CIFAR-10 dataset, sentiment analysis with BILSTM on the IMDB dataset, and financial time series forecasting with GRU on PETR4 stock data. In summary, we sought to answer which hyperparameter optimization strategy showed greater efficiency, robustness, and generalization capacity in different deep learning domains.

2. Material and Methods

The research adopted an exploratory and quantitative approach, outlined as an experimental study. The central objective was to compare three hyperparameter optimization strategies in deep neural networks. The experimental plan was structured in two main stages: initially, a hyperparameter search was performed, generating and evaluating candidate configurations based on validation performance. Subsequently, the best configuration found in each run was used for a more prolonged final training, with evaluation on independent data. This organization allowed differentiating the performance of the optimization process from the predictive quality of the final model.

Hyperparameter optimization was treated as a black-box optimization problem. An objective function, representing the model’s performance on validation, was defined to be minimized. In classification experiments, for example, optimization was operationalized by minimizing the negative of the validation accuracy, aligning accuracy maximization with the library’s minimization interface. The complexity of this problem stemmed from the nonlinear relationship between hyperparameters and performance, the mixed search space, and the computationally expensive evaluation of each configuration.

Three datasets were used, each representing a distinct application domain. The first was CIFAR-10, employed in the image classification task. This dataset consists of 32 × 32 pixel color images, distributed across ten classes. In the project scripts, the dataset was loaded via TensorFlow/Keras, converted to float32, normalized to the [0,1] range, and split into 45,000 samples for training, 5,000 for validation, and an official test set for final evaluation.

The second dataset was IMDB, used in the sentiment analysis task. It contains movie reviews labeled by sentiment polarity, setting up a binary classification problem. The texts were transformed into integer sequences, truncated or padded to a fixed maximum length, and then organized into training, validation, and test sets. Auxiliary decoding and translation procedures for textual examples were included to support qualitative interpretation of the predictions.

The third set was formed by historical data of the PETR4 stock, obtained from Yahoo Finance, for the task of financial time series forecasting. The data were organized chronologically and converted into a supervised temporal forecasting problem through sliding windows. In addition to the price series, derived attributes and transformations such as logarithmic returns and normalized variables were used. Due to the explicit temporal dependence, walk-forward validation was adopted instead of random partitioning.

For the image classification problem with CIFAR-10, a Convolutional Neural Network (CNN) was used. This architecture was chosen for its suitability for data with spatial structure, exploring the local correlation between pixels and allowing automatic learning of filters. The CNN employed local receptive fields and weight sharing, reducing the number of parameters. After convolution, non-linear activation functions such as ReLU, tanh, or sigmoid were applied, followed by max pooling layers with a 2 × 2 window. Training was performed with the Adam optimizer and the sparse_categorical_crossentropy loss function.

For sentiment analysis on the IMDB dataset, a Bidirectional Long Short-Term Memory (BiLSTM) architecture was employed. LSTM networks, belonging to the family of recurrent neural networks, were developed to handle long-range dependencies in sequences, regulating information flow through memory cells and gating mechanisms. The bidirectional version processed the sequence in two directions, combining the hidden states generated by the forward and backward passes, which is advantageous in natural language tasks where context is crucial.

In the problem of financial time series forecasting with the PETR4 stock, a Gated Recurrent Unit (GRU) architecture was used. The GRU, a simplification of LSTM, operates with update and reset gates, which control the preservation and incorporation of new information. This simplified structure reduces the number of parameters and makes the architecture computationally lighter, being efficient in sequential tasks and suitable for balancing modeling capacity and computational cost.

The hyperparameter search spaces were explicitly defined and centralized in the scripts, aiming for methodological consistency, comparability between optimization methods, and computational feasibility. In each domain, a finite set of candidate values was adopted, seeking to balance exploration breadth and computational cost, avoiding configurations that were unfeasible for the experimental environment.

In the CIFAR-10 domain, hyperparameters included the number of convolutional layers at {1,2,3,4}, the number of filters per layer at {8,16,32,64,128}, the convolutional kernel size at {3,5,7}, the activation function at {relu,tanh,sigmoid}, the number of dense layer units at {32,64,128,256}, the batch size at {32,64,128}, and the learning rate at {0.005, 0.001, 0.0005}. For IMDB, the embedding dimension at {32,64,128,256}, the number of recurrent units at {32,64,128,256}, the number of dense layer units at {32,64,128,256}, the activation function at {relu,tanh,sigmoid}, the batch size at {32,64,128}, and the learning rate at {0.005, 0.001, 0.0005} were considered. Parameters such as a maximum vocabulary of 20,000 words and a maximum sequence length of 200 tokens were kept fixed.

For PETR4, the hyperparameters were number of GRU units in {32,64,128,256}, number of GRU layers in {1,2,3,4}, dropout rate in {0.0, 0.1, 0.2, 0.3, 0.4}, number of dense layer units in {16,32,64,128,256}, activation function in {relu,tanh,sigmoid}, batch size in {16,32,64,128}, and learning rate in {0.001, 0.0005, 0.0001, 0.00005}. Fixed temporal parameters included window size equal to 10, training window of 3 years, testing window of 1 year, and temporal advance of 1 year between folds.

Three hyperparameter optimization strategies were evaluated, representing conceptually distinct families. The first was Hyperopt with TPE (Tree-structured Parzen Estimator), framed within Bayesian optimization, which models the relationship between hyperparameters and performance. The second strategy was Hyperband with KerasTuner, based on adaptive resource allocation, evaluating configurations with a small budget and allocating additional resources to the most promising ones. The third strategy used genetic algorithms with DEAP (Distributed Evolutionary Algorithms in Python), where each individual represented a hyperparameter configuration, and evolution was driven by selection, crossover, and mutation operators.

The experimental procedure was standardized across nine scripts, with centralized definition of the HPO algorithm, dataset, architecture, search space, maximum number of evaluations, and training parameters. Pseudo-random seeds were fixed, and determinism mechanisms were employed in TensorFlow to reduce variability. In the experiments with CIFAR-10 and IMDB, the workflow was based on training, validation, and test subsets. In the case of PETR4, walk-forward validation was adopted, respecting the temporal order of observations.

In the HPO stage, each method generated candidate configurations, trained the corresponding model, and evaluated its performance on validation. During the search, CPU usage, RAM memory, GPU utilization, and GPU memory were monitored. Once the search was completed, the best configuration from each run was subjected to a more prolonged final training. For classification problems, this training used EarlyStopping based on val_loss; in PETR4, the procedure was repeated for each fold of the walk-forward scheme. The results were persisted in CSV, TXT, and PNG files.

All scripts were implemented in Python and executed in a Kaggle environment. 50 hyperparameter evaluations were performed per execution and 10 executions per method for stability analysis. The environment used had 2 GHz x86_64 CPUs, 32 GB of RAM, and two Tesla T4 GPUs.

The evaluation of the experiments was conducted on two complementary levels: the performance of the hyperparameter optimization process and the performance of the final selected model. At the optimization level, search efficiency metrics were recorded, such as total search time, number of evaluations performed, best validation accuracy value, and time required to reach a threshold of 95% of the best observed performance for classification. For PETR4, analogous indicators were monitored, such as the best validation MAE value and the time to reach 105% of the best final error. These measures were complemented by statistics on CPU usage, RAM, GPU, and GPU memory.

In the experiments with CIFAR-10 and IMDB, the final model metrics included the final test set loss and the final test accuracy. For CIFAR-10, the evaluation was expanded with macro F1-score, weighted F1-score, per-class F1-score, average macro AUC in a one-vs-rest scheme, absolute and normalized confusion matrix, multiclass ROC curves, and examples of predictions. For IMDB, the evaluation followed similar logic, adapted to binary classification, with macro F1-score, weighted F1-score, classification report, binary ROC curve, AUC, confusion matrices, and interpretable examples of predictions.

In the temporal forecasting problem with PETR4, the evaluation was structured in multiple levels, with error metrics on price (MAE, RMSE, MAPE, R²) and analogous metrics on the return variable, in addition to directional accuracy. The metrics were calculated per temporal fold and then aggregated throughout the walk-forward validation process, resulting in summary measures such as means, medians, variances, and standard deviations. Hyperparameter importance analyses, such as parallel coordinates plot, Random Forest feature importance, permutation importance, SHAP, and hyperparameter landscape, were performed to examine the exploration of the search space and the influence of variables on performance.

3. Results and Discussion

The results obtained in the nine experiments conducted in this research, which compared three hyperparameter optimization strategies (Hyperopt with TPE, Hyperband with KerasTuner, and genetic algorithms with DEAP) in three distinct application domains (image classification with CNN on CIFAR-10, sentiment analysis with BiLSTM on IMDB, and financial time series prediction with GRU on PETR4), revealed that there is no universally superior strategy. Each method demonstrated specific advantages depending on the domain and the evaluation criterion considered, underscoring the contextual nature of hyperparameter optimization. The analysis was performed on two complementary levels: the performance of the optimization process and the final performance of the selected models, allowing for a comprehensive understanding of the effectiveness and computational cost of each approach.

Table 1, which synthesizes the main findings, indicated that Hyperopt presented slightly superior performance in accuracy for CIFAR-10, while Hyperband obtained better results in IMDB and PETR4. Loss values generally followed the accuracy behavior, and in the PETR4 prediction, MAE, as an error metric, was lower for Hyperband, indicating better performance. All algorithms exhibited satisfactory stability, with consistent means and low standard deviations, suggesting that the found solutions were robust. The total search time varied significantly, with DEAP being faster for CIFAR-10 and PETR4, and Hyperband superior for IMDB. This temporal variability partially diverged from literature expectations, which often points to Hyperband as faster, but not always the most accurate. Such findings reinforce the idea that hyperparameter optimization is an intrinsically domain-dependent problem, dependent on the model architecture and the search space structure, with no guarantee of universal superiority for a single method.

Image classification domain results: CIFAR-10 with CNN

In the domain of image classification, the convolutional neural network (CNN) applied to the CIFAR-10 dataset was evaluated using multiclass metrics, including test loss, test accuracy, macro and weighted F1-score, and average macro AUC. The analysis of the results, as illustrated by the training, validation, and test curves of the best model, indicated that the training was consistent and stable, with the use of early stopping preventing overfitting. The absence of significant oscillations during training is an indicator of the robustness of the process, especially considering the influence of hyperparameters such as the number of layers, filters, activation function, batch size, and learning rate on the CNN’s representation capacity and stability.

The quantitative results for CIFAR-10, detailed in Table 2, showed that Hyperopt achieved the best performance across all evaluated metrics, closely followed by DEAP and then Hyperband. This performance hierarchy in terms of effectiveness, although with subtle differences, suggests that the three methods were capable of producing competitive classifiers for this domain. This observation is consistent with the literature, which recognizes that different HPO methods can lead to models with similar final performances, even if the search space exploration strategies are distinct.

The analysis of confusion matrices, both absolute and normalized, revealed that the classes “airplane” and “automobile” showed the highest accuracy rates. In contrast, the classes “bird” and “cat” concentrated the most frequent errors. This difficulty of the classifier in distinguishing between certain categories, such as cats, dogs, birds, and deer, and between birds and airplanes, can be attributed to similar visual characteristics, low image resolution, contour ambiguity, or specific texture patterns. The multiclass ROC curves complemented this analysis, highlighting the model’s discriminative capability among the ten CIFAR-10 categories.

In general, CIFAR-10 proved to be a suitable benchmark for the proposed evaluation, allowing satisfactory solutions to be obtained with relatively little configuration effort, even without reaching the state of the art. This finding reinforces the understanding that, in computer vision tasks, the quality of hyperparameter optimization is intrinsically linked to the method’s ability to effectively explore the structural combinations of the CNN and the parameters of the training regime, highlighting the importance of a well-structured search.

Results in the domain of sentiment analysis: IMDB with BiLSTM

In the domain of sentiment analysis, the Bidirectional Long Short-Term Memory (BiLSTM) architecture was employed on the IMDB dataset for a binary classification problem. The evaluation included metrics such as test loss, accuracy, macro and weighted F1-score, AUC, and binary ROC curve, in addition to confusion matrices and qualitative examples of predictions. The training, validation, and test curves of the best model, presented in Figure 6, indicated rapid convergence in the early epochs, with the best balance between performance and generalization observed in the second epoch. Subsequent epochs showed signs of overfitting, which validated the importance of early stopping in this scenario.

Table 3, which details the quantitative results, suggested that Hyperopt achieved the best performance across all evaluated metrics for IMDB, closely followed by DEAP and then Hyperband. The joint analysis of accuracy, F1-score, and AUC allowed us to conclude that Hyperopt demonstrated greater consistency for this type of problem. The model’s discriminative ability proved adequate, with an AUC substantially higher than the random level in distinguishing between positive and negative outcomes, as illustrated by the binary ROC curve in Figure 7.

The confusion matrices (Figures 8 and 9) indicated that the classifier showed greater robustness in negative class predictions and less robustness in the positive class, keeping false positives and false negatives at relatively low levels. The predominance of failures in this model was associated with positive instances. The qualitative analysis of predictions, exemplified in Figure 10, revealed that the model tends to correctly classify evaluations with more explicit polarity, while errors concentrated in texts containing ambiguities, ironies, negations, or ambiguous words. This qualitative approach is crucial in Natural Language Processing (NLP), as it offers a deeper understanding of the model’s behavior than aggregated metrics.

The results for the sentiment analysis domain, although demonstrating good performance, suggest that there is still room for incremental improvements. Compared to the literature, these findings reinforce the idea that the effect of hyperparameter optimization on textual tasks depends on both the efficiency of the search and the sensitivity of the recurrent architecture to the hyperparameters of representation and sequential memory. The careful choice of hyperparameters, such as embedding dimension, number of recurrent units, and learning rate, is fundamental for the performance of bidirectional architectures in natural language tasks.

Results in the temporal prediction domain: PETR4 with GRU

In the domain of temporal forecasting, experiments were conducted with the historical series of the PETR4 stock, using a Gated Recurrent Unit (GRU) architecture and a walk-forward validation protocol. The evaluation was multifaceted, including metrics on price, return, and directional accuracy, which was essential as the model operated internally on transformed returns, but the final interpretation required the original scale of the series. Table 4 presents the consolidated results, indicating that Hyperband achieved the best performance on most metrics, being the only one to present a slightly positive return R². Hyperopt had the worst performance, and DEAP had intermediate performance.

The interpretation of these results should be done with caution, given the non-stationarity and temporal variability of the PETR4 financial series. However, the findings suggest that Hyperband was the most effective method for modeling time series in this context, reinforced by its superior performance in directional accuracy, which was the only one above 0.50. The training, loss, and MAE curves of the best model, presented in Figure 11, showed rapid convergence in the early epochs and subsequent stabilization, with marginal gains after a few epochs and good generalization.

Figure 12, which displays the results by temporal fold, allowed us to observe the variation in performance over the different windows of the walk-forward scheme. Fold 1 presented the lowest errors, while fold 2 demonstrated greater convergence difficulty, followed by smoother errors in folds 3, 4, and 5. Despite the critical behavior observed in some periods, the analysis indicated that the moments of greatest series volatility generated higher errors in specific periods. The comparison between actual and predicted values over time, illustrated in Figure 13, showed that the predicted curve closely followed the actual trajectory, reproducing the price oscillations well.

The analysis of the residuals, presented in Figure 14, revealed a distribution around zero, suggesting the absence of systematic bias and a relatively controlled error. This confirmed the consistency of Hyperband on the test set. Examples of predictions by date, including actual values, predicted values, and absolute errors, as in Figure 15, showed estimates consistent with the obtained metrics. In interpretive terms, Hyperband proved to be a suitable choice for this type of problem, but the interpretation of performance in financial series must always consider non-stationarity and temporal variability, which makes aggregation by folds essential for a robust analysis.

Hyperparameter search efficiency

Beyond the final performance of the models, the experiments allowed a comparison of the efficiency of the hyperparameter search process. In classification domains, this analysis considered the total search time, the number of evaluations, the best validation score, the best validation loss, and the time to reach 95% of the best final performance. For PETR4, the evaluation was adapted for problem minimization, considering the best validation MAE, the number of valid evaluations, and the time required to reach a threshold equivalent to 105% of the best final error. This approach is aligned with the literature recommendations, which emphasize the importance of efficiency criteria in comparing HPO methods.

The efficiency curves of the search for CIFAR-10, IMDB, and PETR4 (Figures 16, 17, and 18) revealed distinct behaviors. In CIFAR-10, Hyperopt quickly found a promising region, but its evolution was more gradual afterward, presenting the best accuracy, albeit with longer search time and slower convergence. In IMDB, Hyperband demonstrated rapid convergence, but with early stabilization, not allowing significant evolutions, showing a better relationship between final performance and total cost. For PETR4, DEAP exhibited expressive convergence at the beginning, but later attempts were oscillating, with little overall evolution, achieving the best validation MAE with the shortest search time.

Table 5 summarizes the main efficiency indicators for search by domain and method. These results are compatible with the idea that different HPO methods perform distinct trade-offs between exploring the search space, intensifying efforts in promising regions, and computational cost. The analysis of search efficiency, therefore, offers complementary information to the final predictive metrics, providing a more complete understanding of each method’s behavior and its suitability for different scenarios and priorities.

Use of computational resources

The experiments also recorded measures of average and maximum CPU usage, system RAM, process memory, GPU, and GPU memory, as illustrated in Figure 19. These results are crucial for evaluating HPO methods from the perspective of the computational cost effectively incurred, and not just the quality of the solutions found. On CIFAR-10 and IMDB, intensive GPU usage was observed, while on PETR4, GPU usage was light. On CIFAR-10, Hyperopt and Hyperband showed similar average CPU and RAM usage, but differed in GPU. DEAP, in turn, recorded the highest average GPU usage among all methods, but with lower GPU memory, which may indicate less dependence on large data volumes.

In the IMDB domain, the three methods presented quite similar behavior in terms of resource usage. In PETR4, however, the behavior was distinct: Hyperopt and Hyperband showed very low GPU usage and moderate CPU and RAM usage. The increase in GPU usage occurred only in DEAP, accompanied by lower CPU demand. These findings are important for understanding the practical implications of choosing an HPO strategy, especially in environments with limited computational resources, where efficiency and cost are determining factors for the viability of an application.

Stability between runs

The stability of the results was evaluated through multiple independent runs and, in the case of PETR4, also by aggregation across temporal folds. Measures such as mean and standard deviation of the main metrics were considered, allowing for an examination of the methods’ consistency in the face of the inherent variability in neural network training and the search algorithms themselves. Table 6 presents the stability summaries for CIFAR-10, IMDB, and PETR4. The data suggest that the stability of the runs varied according to the method and domain, reinforcing the contextual nature of hyperparameter optimization.

In CIFAR-10, Hyperopt presented the best results, but with higher computational cost and elevated time. Hyperband and DEAP were faster, but with inferior performance. In IMDB, DEAP showed average performance with low dispersion, while Hyperband stood out in temporal terms, and Hyperopt remained competitive in predictive metrics. In PETR4, Hyperopt was clearly superior, both in prediction errors and in lower variability. This result is of great importance, as an isolated high performance does not guarantee practical robustness if it is not accompanied by consistency between executions. Stability, therefore, reinforces that the choice of method must consider quality, variability, and predictability in distinct dimensions of the problem, and not just peak performance.

Importance of hyperparameters and search interpretation

The hyperparameter importance analyses, produced by the scripts using techniques such as parallel coordinates plot, Random Forest feature importance, permutation importance, SHAP, and hyperparameter landscape, allowed us to examine how the search space was explored and which variables exerted the greatest influence on performance. Figures 20a to 20e illustrate these analyses for the different domains, offering a detailed view of the internal dynamics of optimization. In general, it was found that the number of filters and the learning rate exerted the greatest influence in the CIFAR-10 domain, while the batch size and the activation function had a secondary impact, a behavior consistent with the nature of the image classification problem.

In the IMDB domain, greater sensitivity to hyperparameters such as embedding dimension and the number of recurrent units was expected, which are crucial for representation capacity and sequential memory in natural language tasks. In PETR4, parameters related to the recurrent capacity of the GRU and the training regime, such as dropout rate and the number of GRU units, were the most influential. The final interpretation, however, should be made from the patterns actually observed in the experimental graphs, which reveal the complex interactions between hyperparameters and model performance. These visualizations are valuable for shifting the focus of analysis from the simple objective function to the internal dynamics of optimization, allowing for a more complete understanding of the behavior of each HPO method.

Integrated discussion and critical analysis

The joint analysis of the experiments indicated that there was no universally superior method across all evaluated domains. Instead, the results showed that the performance of hyperparameter optimization strategies depends on the interaction between the neural network architecture, the data type, and the adopted evaluation criterion. In general, Hyperopt demonstrated greater robustness in scenarios more sensitive to the final solution quality, such as in image classification. Hyperband presented a better balance between performance and computational cost, standing out in sentiment analysis and temporal prediction. DEAP, in turn, stood out for its speed in some contexts, although with greater oscillation between runs.

The main contribution of this comparison does not lie in pointing out an absolute winner, but in highlighting that the choice of the hyperparameter optimization method should be guided by the problem’s priority. This means that the decision should consider whether the objective is to maximize final performance, reduce search time, or obtain greater stability of results. These conclusions, however, should be interpreted in light of the defined search spaces and the evaluated domains, which recommends caution in generalizations to other scenarios or architectures. The research offered an empirical comparison conducted under a standardized pipeline, with experimental control and evaluation in heterogeneous domains, allowing us to observe how performance, search time, resource usage, and stability compose complementary dimensions in the evaluation of HPO methods. Thus, the work reinforces that the effectiveness of hyperparameter optimization should be understood multidimensionally, considering the inherent trade-offs of each strategy.

4. Conclusion

This study sought to compare three hyperparameter optimization strategies – Hyperopt with TPE, Hyperband with KerasTuner, and genetic algorithms with DEAP – in deep neural networks applied to image classification, sentiment analysis, and financial time series forecasting problems. It was found that no single strategy was universally superior across all evaluated scenarios. The results indicated that Hyperopt demonstrated greater robustness in domains sensitive to the final solution quality, such as in image classification. Hyperband, in turn, presented a better balance between predictive performance and computational cost, standing out in sentiment analysis and time series forecasting. Genetic algorithms with DEAP proved to be faster in some contexts, albeit with greater oscillation between runs. The main contribution of this research lies in highlighting that the choice of hyperparameter optimization method should be guided by the problem’s priority, whether it be maximizing final performance, reducing search time, or achieving greater result stability.

The work’s conclusions should be interpreted in light of the defined search spaces and the specific domains evaluated, which recommends caution in generalizations to other scenarios or architectures. As a suggestion for future studies, it is proposed to expand the investigation with new search spaces, the inclusion of other deep neural network architectures, and evaluation in even more diversified application domains. The study offered an empirical comparison under a standardized pipeline, with experimental control and evaluation in heterogeneous domains, reinforcing that the effectiveness of hyperparameter optimization should be understood multidimensionally, considering the inherent trade-offs of each strategy in terms of performance, search time, resource usage, and stability.

Bibliographic References

Aggarwal, C.C. 2023. Neural Networks and Deep Learning A Textbook. 2ed. Springer Nature Switzerland, Cham, Suíça.

Agrawal, T. 2021. Hyperparameter Optimization in Machine Learning. Apress, Bangalore, Índia.

Ali, Y.A.; Awwad, E.M.; Al-Razgan, M.; Maarouf, A. 2023. Hyperparameter search for machine learning algorithms for optimizing computational complexity. Processes 11: 349.

Bischl, B.; Binder, M.; Lang, M.; Pielok, T.; Richter, J.; Coors, S.; Thomas, J.; Ullmann, T.; Becker, M.; Boulesteix, A.-L.; Deng, D.; Lindauer, M. 2023. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. WIREs Data Mining and Knowledge Discovery 13(2): e1484.

Gomede, E.; Silva, F.; Mendonça, M.; Palácios, R.H.C. 2023. Bayesian Optimization for Sampling the Hyper Parameter Space. Disponível em: <http://dx.doi.org/10.2139/ssrn.4559196>. Acesso em: 31 mar. 2026.

Goodfellow, I.; Bengio, Y.; Courville, A. 2016. Deep Learning. MIT Press, Cambridge, MA, EUA. Disponível em: <http://www.deeplearningbook.org>. Acesso em: 30 mar. 2025.

Liao, L.; Li, H.; Shang, W.; Ma, L. 2022. An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks. ACM Transactions on Software Engineering and Methodology 31(3): Article 53.

Raiaan, M.A.K.; Sakib, S.; Fahad, N.M.; Mamun, A.A.; Rahman, M.A.; Shatabda, S.; Mukta, M.S.H. 2024. A systematic review of hyperparameter optimization techniques in convolutional neural networks. Decision Analytics Journal 11: 100470.

Victoria, A.H.; Maragatham, G. 2021. Automatic tuning of hyperparameters using Bayesian optimization. Evolving Systems 12: 217-223.

Article originating from the Final Course Work of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform