Analytics and Project Management: Performance Optimization in Software Development Teams

Célio Roberto da Silva Costa; Bruno Andrade Costa

DOI: 10.22167/2675-6528-202602781

Article derived from a Final Course Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Abstract

The growing complexity in software development has amplified the challenges of predictability and performance control in agile teams. The study aimed to analyze the predictive capability of traditional metrics and evaluate the use of Machine Learning techniques to optimize deadline estimation in software projects. The research was conducted as a quantitative case study, using real historical data from 2024, extracted from a “DevOps” management tool, which included information on estimated effort, execution time, and sprint dynamics. Initially, an exploratory analysis was performed to identify performance patterns, including the spillover rate and the correlation between story points and delivery time. Subsequently, predictive regression models were tested, incorporating contextual variables such as historical load, team velocity, and relative effort. The results showed a low linear correlation between estimation and actual time (r = 0.1078; p = 0.060), not statistically significant, and a spillover rate of 30.59%, indicating limitations in traditional planning. “Random Forest” showed an improvement of 38.75% compared to naive prediction based on historical average, while Linear Regression achieved the best performance on the temporal test set, more consistently reducing the prediction error. The adoption of multivariate analytical approaches contributed to increasing predictability and supporting more assertive decisions in software project management.

Keywords: analytics; performance; agile management; machine learning; predictability.

1. Introduction

Agile Methods represent a paradigm shift in project management, prioritizing continuous adaptation over rigidly planned approaches. This logic, operationalized by models such as Scrum, seeks to maximize team efficiency and enable value delivery in shorter timeframes (Sutherland, 2014).

Despite the modernization of work tools and the adoption of agile methodologies, the translation of these technologies into consistent gains in operational predictability and real productivity often remains elusive. This phenomenon evokes the classic productivity paradox, formulated by Robert Solow (1987), who observed the difficulty in measuring the return on technology investments in productivity indicators. In contemporary software development, this mismatch is frequently repeated.

Agility, with its iterative nature, introduces unique challenges for schedule estimation, requiring forecasts built abstractly and relatively, typically through story points. Traditional methods like PERT and Three-Point seek to reduce subjectivity but assume simplified relationships and do not capture the complexity of software projects. Planning Poker, although widely used, relies on human judgment, being susceptible to biases and the absence of structured use of historical data (Finco et al., 2021; Poženel et al., 2023; Tissot et al., 2015).

Given these limitations, Machine Learning techniques emerge as more suitable alternatives for dealing with the complexity inherent in software development. Nonlinear learning algorithms stand out for modeling complex patterns and multiple factors simultaneously. However, the model’s suitability depends on the data structure and the analyzed context, which requires empirical comparison of different algorithms such as Linear Regression, Random Forest, Support Vector Machines, and XGBoost (Mahdi et al., 2021; Terlapu et al., 2024).

The absence of linearity between estimation and execution creates an environment prone to planning failures, often manifesting in a broken delivery cadence and the occurrence of spillover, where unfinished tasks overflow into subsequent sprints. To mitigate these delays, the application of Analytics has been consolidated as a strategic differentiator in project management (Zwikael and Meredith, 2018). This evolution marks a significant shift, with artificial intelligence and Machine Learning emerging as global trends to optimize decision-making in dynamic environments and drive data-oriented decisions (Vergara et al., 2025; Nishat et al., 2025). Maturity in project management is crucial, evolving from reactive control to higher stages where performance indicators support strategic forecasts (Kerzner and Saladis, 2020).

Given the gap between planning based on theoretical estimates and the actual performance observed during execution, the limitation of traditional approaches becomes evident. The present research is justified by the need to improve predictability and control in software projects through the integration of advanced analytical approaches. Thus, this work aims to analyze the predictive capacity of conventional agile metrics and evaluate the application of regression models in optimizing deadline estimation in software projects, with an initial focus on the Random Forest algorithm and subsequent comparison with Linear Regression, Support Vector Regression, and XGBoost, investigating whether the incorporation of contextual variables, such as relative effort, historical load, and team velocity, contributes to reducing the prediction error of delivery time, enabling evidence-based management.

2. Material and Methods

This research was characterized as a case study of a quantitative, exploratory, and applied nature, focusing on the analysis of predictability in software development teams. The investigation was conducted using real historical data, with the purpose of comparing traditional planning, based on effort estimates, and a predictive approach based on Machine Learning techniques. This methodological strategy sought to analyze the predictive capacity of conventional agile metrics and evaluate the application of regression models in optimizing deadline estimation in software projects.

The object of analysis focused on the activity execution cycle, with the main variable being the time elapsed between the effective entry of the item into development and its closure, termed “Closed Date”. The initial database comprised 402 work items, extracted from the “Azure DevOps” environment, pertaining to the year 2024, covering sprints 1 to 25. The “data wrangling” process involved cleaning and validation, resulting in a final sample of 274 items for predictive modeling, after excluding support items, administrative tasks, records without estimates or closure dates, and outliers with an execution time exceeding 28 days.

Although the “Azure DevOps” tool does not preserve the static record of the original sprint planning, the complete history of the status updates for each item was maintained. From the extraction and analysis of this history of movements on the “board”, it was possible to identify the exact moment when each task transitioned to the “In Development” stage. This procedure allowed for a more precise measurement of the real execution time of the tasks, eliminating distortions. The entire data extraction and processing procedure was conducted in accordance with confidentiality principles, including the anonymization of developers.

In order to broaden the model’s explanatory capacity and incorporate dimensions not captured by traditional metrics, a structured feature engineering step was performed. This step was aimed at constructing variables representative of the operational and human context of software development. The delivery time, or Lead Time, was defined as the target variable that the predictive model sought to estimate, measuring the actual physical period, in calendar days, from the effective entry of a work item into the “In Development” state until its closing date in “Azure DevOps”.

The relative effort represented a dimensionless and predictive variable, built through feature engineering, formally defined as Erel = SP / Wdev. SP corresponds to the story points assigned to the work item, and Wdev refers to the seniority weight of the responsible developer, with values of 0.5 for intern, 1 for junior, 4 for mid-level, and 7 for senior. The developer’s historical load was the total volume of story points completed by the same professional in the previous sprint, calculated with temporal lag. The team’s recent velocity was the moving average of deliveries from the previous three sprints.

Additionally, a seasonality variable was incorporated, represented by the development task’s entry month. Missing data in essential information were excluded; for initial historical variables, imputation by zero was applied. The scale was not normalized. For predictive modeling, the Random Forest Regressor algorithm was initially selected and implemented via “scikit-learn”, configured with 100 trees and a fixed random state of 42. Subsequently, its performance was compared to Linear Regression, Support Vector Regression, and XGBoost.

The validation strategy adopted was based on temporal division of the data, preserving the chronological order of the sprints, in a procedure equivalent to “Time Series Split”. Approximately 80% of the initial data, corresponding to sprints 1 to 20, were used for model training, while the final 20%, sprints 21 to 25, were allocated for validation. For the comparison between algorithms, the metrics mean absolute error (MAE), root mean squared error (RMSE), and coefficient of determination (R²) were adopted. Shapiro-Wilk normality tests and outlier analysis were also performed.

3. Results and Discussion

The analysis of data extracted from the software development ecosystem throughout 2024 revealed crucial insights into predictability and performance in agile teams. The empirical results challenge the premises of traditional agile planning, highlighting the need for more robust predictive approaches. This study sought to analyze the predictive capacity of conventional agile metrics and evaluate the application of regression models in optimizing deadline estimation, incorporating contextual variables to reduce delivery time prediction error.

Performance and operational efficiency overview

The analyzed database, composed of activities recorded in the 2024 sprints, including development items, improvements, and defect fixes estimated in story points, allowed for the cross-referencing between the team’s theoretical capacity and the effort effectively delivered. It was observed that the team presented significant oscillations in delivery capacity throughout the sprints, with cycles that both exceeded and fell short of the estimated theoretical capacity. These variations indicate an inconsistency in the workflow and adherence to the initial planning.

One of the most relevant findings was the identification of a spillover rate of 30.59%, which means that approximately one-third of the planned tasks were not completed within the originally foreseen period and spilled over into subsequent cycles. This phenomenon directly compromises the predictability of deliveries and the cadence of sprints, evidencing a structural limitation in the traditional planning process. The occurrence of spillover, as observed in sprint 12 of 2024, where it reached 42.86%, was associated with peaks in historical workload among more experienced developers, suggesting that the concentration of effort can negatively impact future delivery capacity.

The predictive capacity of story points as an isolated metric proved to be low. Statistical analysis indicated a weak linear correlation between story points and actual execution time, with a Pearson coefficient of 0.1078 and a p-value of 0.060. The 95% confidence interval between -0.0047 and 0.2177, which includes zero, suggests that there is insufficient statistical evidence to assert a consistent linear association at the 5% significance level. Shapiro-Wilk normality tests rejected the hypothesis of normality for both variables (p < 0.001), and the presence of outliers in “Lead Time” contributed to the instability of the simple linear correlation.

As a complementary verification, Spearman’s correlation, a non-parametric measure, showed a positive and statistically significant association (rho = 0.2000; p < 0.001), although of small magnitude. These results, even after the removal of outliers, reinforce that story points, when used in isolation, do not satisfactorily explain the actual delivery time. This limitation is attributed to multiple operational and contextual factors, which are not captured by estimates based exclusively on subjective perception.

Several factors contribute to the observed low predictability. The inherent subjectivity in the estimation process, influenced by deadline pressure and individual perception, is a critical element. Furthermore, story points do not incorporate contextual variables, such as the developer’s workload, their seniority, or external interferences. The dynamic environment of software projects, with unforeseen events and requirement changes, makes simplified metrics insufficient for exact projections (Rahman et al., 2023). These findings align with literature pointing to the limitations of manual and intuitive planning (Grover and Sharma, 2023).

Traditional estimation approaches, such as PERT and Three-Point, aim to reduce subjectivity by defining optimistic, pessimistic, and most likely scenarios. However, these methods still assume simplified distributions and linear relationships between variables, which limits their ability to model the complexity of software projects, characterized by multiple interdependent factors and high variability. Planning Poker, although collaborative, remains dependent on human judgment and susceptible to biases, without a structured use of historical data (Finco et al., 2021; Poženel et al., 2023; Tissot et al., 2015).

Given these limitations, Machine Learning techniques emerge as more suitable alternatives, capable of capturing non-linear relationships and considering multiple variables simultaneously from historical data (Rahman et al., 2023; Terlapu et al., 2024). The adoption of Machine Learning represents a methodological evolution that allows for greater generalization and adaptation to complex contexts, overcoming the restrictions of simplified heuristic and statistical methods when applied to real-world software development scenarios.

Predictive model performance

To overcome the limitations of traditional estimates, a predictive model based on the Random Forest algorithm was developed, incorporating context variables related to team performance. The model’s performance was evaluated in comparison to a baseline model, which used the historical average execution time. The results demonstrated a significant reduction in the mean absolute error (MAE), which decreased from 6.27 days in the baseline model to 3.84 days in the proposed model, representing a 38.75% improvement in prediction accuracy.

Despite the improvement in accuracy, the model presented a coefficient of determination (R²) close to zero, indicating that a relevant portion of the variability in delivery time was not yet explained. This behavior is expected in complex socio-technical contexts, such as software development, where unobservable factors, such as interruptions, requirement changes, technical dependencies, and individual variability, exert significant influence on performance (Mahdi et al., 2021; Rahman et al., 2023; Terlapu et al., 2024).

The literature on software effort estimation points out that team performance is strongly influenced by contextual and human variables, which are often not captured in structured databases. Consequently, predictive models tend to present limited capacity for global explanation of variability, especially when based on historical operational data. This result reinforces that, in problems with high variability and influence of unobservable factors, the utility of the model should be evaluated primarily by error reduction and not exclusively by global explanation metrics, making MAE more appropriate for evaluation.

Comparison between regression algorithms

To broaden the model validation and evaluate the suitability of different predictive approaches, four regression algorithms were tested: Linear Regression, Random Forest Regressor, SVR, and XGBoost. The comparison was performed on the same temporal test block, with training on sprints 1 to 20 and validation on sprints 21 to 25, using the MAE, RMSE, and R² metrics. The results indicated that Linear Regression showed the best performance among the tested models, with an MAE of 3.7497, RMSE of 6.5616, and R² of 0.1124.

The Random Forest ranked second, with an MAE of 3.9913, RMSE of 6.6235, and R² of 0.0956, while SVR and XGBoost showed higher errors and lower explanatory power. This result suggests that, in the specific context of the analyzed database, the relationship between the explanatory variables and the actual delivery time exhibited predominantly linear behavior. Alternatively, the volume and structure of the available data may not have fully favored the utilization of more sophisticated non-linear models. Thus, although Random Forest demonstrated a significant gain compared to naive forecasting, Linear Regression was the most suitable approach for the scenario evaluated in the temporal test set.

Importance of variables and critical performance factors

The importance analysis of variables in the Random Forest model revealed that task execution time is predominantly influenced by contextual factors, not just estimated complexity. The most relevant variable was relative effort, which considers the relationship between story points and the developer’s seniority level. This finding indicates that the same task can exhibit distinct behaviors depending on the executor, highlighting the importance of adequate resource allocation and consideration of individual productive capacity in modeling.

Next, the developer’s historical load and the team’s recent velocity stood out. The accumulated load, defined as the total volume of story points completed by the same professional in the immediately preceding sprint, proved to be a relevant indicator of future performance, suggesting that high levels of effort in previous periods negatively impact subsequent productivity. The team’s recent velocity, calculated as the moving average of deliveries made in the three previous sprints, served as an indicator of the team’s operational status, reflecting factors such as environmental stability, interruptions, and workflow efficiency.

These findings reinforce the need to consider the operational context in performance analysis, overcoming simplified approaches based exclusively on isolated metrics. Variables such as sprint context (timing), initial estimation (story points), seasonality (start month), feature type (new feature, technical debt, bug), and seniority level also contributed to the prediction, albeit with less relative impact. The incorporation of these software development process dimensions, such as task characteristics and team history, is fundamental for more accurate predictive models (Mahdi et al., 2021; Rahman et al., 2023; Terlapu et al., 2024).

Discussion and managerial implications

The results obtained demonstrate that software project management can significantly benefit from the adoption of analytical and predictive approaches. The high spillover rate and low correlation between estimation and execution indicate that traditional planning presents structural limitations, especially in complex and dynamic environments. The use of predictive models allows a transition from reactive management to data-driven management, in which decisions are based on empirical evidence and not just on experience or intuition.

Considering the potential impact of rework associated with task spillover and structural scope changes, a change cost multiplier factor of 1.6 is adopted as a reference management parameter. This parameter seeks to represent that the impact of delays and changes is not restricted to the effort originally planned for the activity, potentially involving rework, code refactoring, retesting, and developer context switching. Thus, for management analysis purposes, the factor 1.6 represents an increase of 60% over the nominal effort of items affected by these occurrences, allowing for a condition of greater complexity in relation to the originally planned execution flow.

From a practical point of view, the results enable the manager to adopt more assertive strategies, such as balancing the workload and allocating tasks according to seniority level, since predictive optimization in resource allocation significantly reduces idleness and delivery time (Grover and Sharma, 2023). Furthermore, the findings suggest a shift in the manager’s role, who ceases to act solely as a planner or collector of empirical data, to assume the role of a workflow analyst, using the support of analytical tools to guide complex decisions (Asghar and Burria, 2025; Vergara et al., 2025).

The creation of indicators, the identification of critical factors, and the comparison between different regression models demonstrate the viability and potential of applying Analytics to increase organizational predictability and maturity. In summary, the integration of historical data, statistical analysis, and predictive modeling represents a relevant advancement in software project management, contributing to more precise decisions and the continuous improvement of team performance. In the analyzed case, the empirical comparison between algorithms showed that Linear Regression was the most suitable alternative in the temporal test set, reinforcing that the choice of model should be guided by the data and not just by the technical complexity of the algorithm.

From a historical perspective, the findings of this study offer a practical answer to the productivity paradox proposed by Solow (1987), which pointed out the difficulty of observing the tangible return of technological infrastructure in organizations’ efficiency indicators. The present research demonstrates that the accumulation of operational data on DevOps platforms, by itself, does not guarantee greater efficiency or predictability if these records continue to be interpreted in a purely intuitive or manual way. By integrating Machine Learning and multivariate Analytics techniques into the agile workflow, this study shows how the technological apparatus can be converted into predictive information and quantitative subsidies for evidence-based managerial decisions.

In summary, the isolated use of traditional metrics, such as story points, is insufficient to explain and predict performance in software development teams, as evidenced by the low correlation observed and the high spillover rate. On the other hand, the incorporation of contextual variables through predictive models has demonstrated a concrete ability to reduce uncertainty in planning, although it does not eliminate the inherent complexity of the process. Predictability in agile environments should be understood as a multifactorial phenomenon, in which the integration of historical data, statistical analysis, and managerial interpretation represents a consistent path towards increased reliability and maturity in software project management.

4. Conclusion

This study sought to analyze the predictive capacity of conventional agile metrics and evaluate the application of regression models to optimize the estimation of deadlines in software projects, incorporating context variables. It was found that traditional metrics, such as story points, showed low linear correlation with actual execution time, were not statistically significant, and had a high spillover rate of 30.59%, indicating limitations in traditional planning. In contrast, the application of Machine Learning models demonstrated significant gains in predictability. Initially, the Random Forest algorithm reduced the mean absolute error by 38.75% compared to naive forecasting. Subsequently, the comparison between different algorithms revealed that Linear Regression achieved the best performance on the temporal test set, more consistently reducing the prediction error. The analysis of variable importance highlighted relative effort, developer historical load, and team velocity as critical factors, evidencing that performance is a multifactorial phenomenon. This analytical approach contributes to more data-driven software project management, offering a practical response to Solow’s (1987) productivity paradox by converting operational data into quantitative subsidies for managerial decisions.

Despite advances in predictive accuracy, this study has limitations. The analysis was conducted based on data from a single development team, which may introduce selection bias and limit the generalization of findings to other organizational contexts. Furthermore, unobservable factors such as team turnover, interruptions, and requirement changes influenced performance and were not captured in the structured data, resulting in a low coefficient of determination. The research scope focused exclusively on the development cycle, without covering earlier or later project stages. For future studies, it is suggested to apply the model to multiple teams and distinct contexts, expand the dataset to explore more advanced modeling techniques, incorporate unstructured data through natural language processing, and develop integrated tools that use real-time predictive models to enhance management.

Bibliographic References

Asghar, M.Z.; Burria, K.U. 2025. The impact of artificial intelligence on agile project management: enhancing team performance and project success in technology-driven organizations. The Critical Review of Social Sciences Studies 3(4): 1217-1232.

Finco, D. A.; Bastos, L. C.; Neto, A. G. S. S. 2021. Combinando Planning Poker e Aprendizado de Máquina para estimar esforço de software. In: Escola Regional de Engenharia de Software, 2021, Curitiba, PR, Brasil. Anais… p. 1-10.

Grover, H.; Sharma, S. 2023. Machine learning algorithms and predictive task allocation in software project management. International Journal of Open Publication and Exploration 11(1): 34-43.

Kerzner, H.; Saladis, F. P. 2020. Using the Project Management Maturity Model: Strategic Planning for Project Management. 3ed. Wiley, Hoboken, NJ, USA.

Mahdi, M.N. et al. 2021. Software project management using machine learning technique-a review. Applied Sciences 11(11): 5183.

Nishat, H. A.; Tanisha, T. A.; Ali, M. H.; Khondoker, M. A. H. 2025. Applying machine learning for predictive analysis in project-based data: insights into variation orders. Journal of Information Technology in Construction (ITcon), 30: 33–51.

Poženel, M. et al. 2023. Agile effort estimation: comparing the accuracy and efficiency of planning poker, bucket system, and affinity estimation methods. International Journal of Software Engineering and Knowledge Engineering 33(11): 1923-1950.

Rahman, M. et al. 2023. Software effort estimation using machine learning technique. International Journal of Advanced Computer Science and Applications 14(4): 1-10.

Solow, R. M. 1987. We’d better watch out. New York Times Book Review, New York, NY, USA, 12 jul. 1987.

Sutherland, J. 2014. Scrum: A Arte de Fazer o Dobro de Trabalho na Metade do Tempo. Leya, São Paulo, SP, Brasil.

Terlapu, P.V. et al. 2024. Improved software effort estimation through machine learning: challenges, applications, and feature importance analysis. IEEE Access 12(1): 1-15.

Tissot, Α.Α.; Emer, M.C.F.P.; Bastos, L.C. 2015. Influence of the review of executed activities utilizing planning poker. In: 29th Brazilian Symposium on Software Engineering, 2015, Belo Horizonte, MG, Brasil. Anais… p. 170-178.

Vergara, D. et al. 2025. Trends and applications of artificial intelligence in project management. Electronics 14(4): 800.

Zwikael, O.; Meredith, J. R. 2018. Data analytics in project management: A systematic literature review. In: 2018 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Bangkok, Tailândia. Anais… p.6-10.

Article originating from the Final Course Work of the Specialization in Project Management of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform