October 26, 2026
Interpretable machine learning for story points prediction in Scrum projects
Interpretable Machine Learning for Story Point Prediction in Scrum Projects
Matheus de Oliveira Avila Sales; Manoel Flavio Leal
DOI: 10.22167/2675-6528-202601221
Article prepared by the ResumeAI tool, an artificial intelligence solution developed by the Pecege Institute focused on synthesis and writing.
Summary
The estimation of story points in Scrum projects is susceptible to cognitive biases, compromising sprint predictability. Machine Learning (ML) models have been applied to automate these estimates, but they often prioritize accuracy over interpretability, which is essential for adoption by agile managers. The research evaluated the viability of interpretable ML models as support for decision-making in agile planning, combining story point prediction with explainability via SHAP. Three supervised learning algorithms (Linear Regression, Random Forest, and XGBoost) were implemented to predict story points from nine interpretable features, extracted from 23,313 issues from 16 open-source projects. XGBoost achieved the lowest Mean Absolute Error (4.64 story points), but the three models showed similar performance, indicating that the predictive ceiling stemmed from the feature space. The SHAP technique quantified the contribution of each feature, identifying the amount of code blocks, the title length, and the frequency of technical terms as the attributes with the greatest impact. The combination of prediction and explainability configured a decision support tool in agile planning, by making explicit the textual attributes that influence estimates.
Keywords: Supervised learning; Effort estimation; Agile management; Planning poker; SHAP.
1. Introduction
Scrum is one of the most widely adopted agile frameworks in the software development industry (Schwaber and Sutherland, 2020). Within this context, sprint planning, fixed-duration work cycles, crucially depends on the estimation of story points. Story points represent a relative measure of effort that teams assign to backlog tasks, reflecting the complexity, risk, and volume of work involved. The accuracy of these estimates is fundamental to the team’s ability to plan realistic deliveries, calculate its development velocity, and effectively communicate deadlines to stakeholders.
Traditionally, story points estimation is performed through consensus techniques, such as planning poker. This approach, although it promotes egalitarian participation by requiring all members to reveal their estimates simultaneously, is susceptible to cognitive biases (Mahnič and Hovelja, 2012). Group dynamics can, for example, amplify the optimism bias, especially in teams with limited experience, compromising sprint predictability and generating significant deviations between the estimate and the actual effort. In fact, Moløkken and Jørgensen (2003) documented that deviations of 30 to 40% from the planned effort are recurrent in the industry. Pasuksmit et al. (2024) identified information quality and team factors as frequent contributors to inaccuracy in these estimates.
Faced with these challenges, Machine Learning (ML) has been applied to automate the estimation of story points. The evolution of these predictive approaches has demonstrated consistent gains in predictive accuracy. Initially, techniques such as TF-IDF were combined with classifiers (Porru et al., 2016). Subsequently, deep learning models, such as Deep-SE, emerged to capture complex semantic relationships (Choetkiertikul et al., 2019). More recently, pre-trained language architectures, such as GPT2SP (Fu and Tantithamthavorn, 2023) and SBERT (Yalçıner et al., 2024), have been employed, reporting competitive Mean Absolute Errors. However, a persistent limitation of these approaches lies in the fact that they function as “black boxes”. The agile management professional receives a numerical estimate without understanding which characteristics of the issue influenced it, which hinders trust in the prediction and the incorporation of the results into planning discussions.
This scenario reveals a significant gap in the literature: the absence of studies that combine story point prediction with explainability techniques oriented towards agile management. For predictive models to be effectively adopted by Scrum Masters and Product Owners, it is essential that the results are understandable and actionable, allowing them to question and validate the generated predictions (Fernández-Diego et al., 2020). Trust in the tool directly depends on the ability of its users to understand the “why” behind each estimate. The SHapley Additive exPlanations (SHAP) technique, proposed by Lundberg and Lee (2017), offers a promising solution by quantifying the marginal contribution of each feature to the individual prediction, based on cooperative game theory and satisfying axiomatic properties that guarantee a fair and transparent attribution.
Unlike approaches that use numerical text representations with thousands of dimensions, this work prioritized interpretability. Three supervised learning algorithms – Linear Regression, Random Forest, and XGBoost – were implemented using nine structural and interpretable features, extracted directly from the textual fields (title and description) of 23,313 issues from 16 open-source projects of the Deep-SE Story Point Dataset (Choetkiertikul et al., 2019). This choice allowed the SHAP technique to provide accessible explanations, such as “the prediction increased because the issue contains six code blocks,” instead of complex interpretations of high-dimensional vectors. Thus, the objective of this work is to evaluate the viability of interpretable machine learning models as decision support in agile planning, combining story point prediction with explainability via SHAP.
2. Material and Methods
The research was characterized as a study of an applied nature, with a quantitative approach and an experimental-computational design. The central objective was to evaluate the viability of interpretable machine learning models as decision support in agile planning, combining story points prediction with explainability via SHAP. To this end, a data processing and regression model training pipeline was implemented.
The implementation pipeline was structured in four sequential stages: acquisition and exploratory analysis of the dataset, engineering of interpretable features from the textual fields of the issues, preprocessing and partitioning of the data, and training with optimization of the regression models. The entire implementation was conducted in Python, using libraries such as scikit-learn for regression models and cross-validation, XGBoost for gradient boosting, SHAP for calculating Shapley values via TreeExplainer, pandas for tabular data manipulation, matplotlib for figure generation, and textstat for readability indices.
The reproducibility of the analyses was ensured by fixing the seed `random_state=42` in all operations involving randomness. The complete source code of the project was made available in a public repository, ensuring transparency and the possibility of verifying the adopted procedures.
The dataset used was the Deep-SE Story Point Dataset, publicly available from Choetkiertikul et al. (2019). This public repository served as a secondary data source, originating from tracking systems such as Jira and Bugzilla, from 16 open-source software projects. The adoption of this dataset allowed for direct comparability with previous studies (Choetkiertikul et al., 2019; Fu and Tantithamthavorn, 2023).
The dataset gathered 23,313 issues, with each record containing the fields `issuekey`, `title`, `description`, and `storypoint`. The inclusion criteria defined by the original authors of the dataset excluded projects with fewer than 300 estimated issues and removed records with story points equal to zero, negative, or greater than 100. This filtering affected approximately 2.66% of the total records.
Regarding data handling, the `description` field presented 2,231 missing values, corresponding to 9.57% of the total. These cases were handled by replacing them with an empty string, preserving the corresponding records without artificial imputation. The other fields (`issuekey`, `title`, `storypoint`, and `project`) showed no missing values.
Instead of high-dimensional vector representations, nine interpretable features were extracted directly from the textual fields (`title` and `description`) of each issue. The features included: `desc_length` (description length in characters), `title_length` (title length in characters), `is_user_story` (presence of the canonical user story pattern), `has_acceptance_criteria` (presence of acceptance criteria), `code_snippet_count` (code blocks or method calls), `url_count` (URLs referenced in the description), `description_readability` (Flesch-Kincaid Grade Level index), `keyword_complexity` (frequency of technical terms), and `title_desc_ratio` (title length/description length ratio).
The binary features (`is_user_story` and `has_acceptance_criteria`) were detected by regular expressions. The `description_readability` feature employed the Flesch-Kincaid Grade Level index, calculated by the textstat library. The `keyword_complexity` feature counted the occurrence of 20 technical terms associated with high-complexity tasks, such as ‘refactor’, ‘migrate’, ‘integration’, ‘API’, ‘database’, ‘security’, ‘performance’, ‘architecture’, ‘infrastructure’, ‘scalability’, ‘deployment’, ‘authentication’, ‘authorization’, ‘configuration’, ‘optimization’, ‘upgrade’, ‘redesign’, ‘overhaul’, and ‘rewrite’.
The set of 23,313 issues was partitioned into 80% for training (18,650 issues) and 20% for testing (4,663 issues). Random shuffling was controlled by the seed `random_state=42`. Normalization was applied to continuous numerical features by StandardScaler, fitted exclusively on the training set to avoid data leakage. Binary features were kept on the {0, 1} scale, dispensing with normalization.
Three supervised learning algorithms were implemented to predict story points: Linear Regression, Random Forest, and XGBoost. Linear Regression served as the baseline model, with coefficients determined by the ordinary least squares method on the nine features, without hyperparameter optimization.
Random Forest, an ensemble method by bagging, built multiple independent decision trees on random subsamples of instances and features, aggregating individual predictions by averaging. Hyperparameters were selected by grid search with five-fold cross-validation (cv=5), minimizing Mean Absolute Error (MAE) as the criterion. The grid evaluated 36 combinations, totaling 180 adjustments.
XGBoost implemented gradient boosting over decision trees, adjusting each subsequent tree to minimize the residual of the previous iteration by gradient descent, with L1/L2 regularization to control complexity. Grid search evaluated 81 combinations, totaling 405 adjustments with five-fold cross-validation.
The explainability analysis employed the SHapley Additive exPlanations (SHAP) technique, proposed by Lundberg and Lee (2017). This technique assigns a value to each feature that quantifies its marginal contribution to the individual prediction of each observation, based on cooperative game theory. The TreeExplainer, an algorithm optimized for tree-based models, calculated the exact Shapley values for Random Forest and XGBoost without resorting to stochastic approximations.
The results of the SHAP analysis were organized into four types of visualization: global importance (bar plot), directional distribution (beeswarm), local explanation (waterfall), and feature dependency (dependence plot). These visualizations were used to interpret the contribution of each feature to the model predictions, without presenting the empirical findings in this section.
3. Results and Discussion
This research assessed the viability of interpretable machine learning models as support for decision-making in agile planning, combining story point prediction with the explainability provided by the SHAP technique. The initial results of the exploratory analysis of the Deep-SE dataset, composed of 23,313 issues from 16 open-source projects, revealed a distribution of story points with a pronounced positive skew (6.00) and high kurtosis (45.69). This characteristic indicated that the majority of issues were concentrated in the lower values of the Fibonacci scale (1, 2, 3, and 5 story points), while values exceeding 20 story points were rare occurrences, a pattern that directly influenced the predictive behavior of the models.
The variability of story points across the analyzed projects proved to be significant, with the median values ranging from 2 to 8 story points and a substantially different interquartile range among the teams. This pattern reflected the existence of distinct estimation conventions, a finding consistent with what was documented by Tawosi et al. (2022a) in their analysis of the TAWOS dataset, which includes the same projects. This heterogeneity motivated the decision to adopt a cross-project model, trained and evaluated on the entirety of the data, without specific project separation, seeking a generalization of the observed patterns.
The relationship between the length of textual fields and the assigned story points proved to be weak, without a significant linear correlation. Both short and long issues received low and high values, indicating that raw textual characteristics, without further transformation, capture a limited predictive signal. This result corroborates findings by Porru et al. (2016), who reported an analogous pattern when investigating story point estimates from raw term counts. The concentration of the most frequent values in the canonical points of the Fibonacci scale (1, 2, 3, 5, and 8 story points) was confirmed, with the presence of extreme values that produced the heavy tail observed in the distribution.
Feature engineering prioritized interpretability, extracting nine attributes directly from the textual fields of the issues, such as description length, title length, presence of canonical user story, presence of acceptance criteria, code block count, URL count, description readability, keyword complexity, and title/description ratio. This choice was deliberate to ensure that the explanations would be understandable by agile management professionals, without the need for familiarity with language models or advanced algebra. The trade-off between accuracy and interpretability was, therefore, intentional, and the resulting loss of predictive capability was quantified in the model evaluation.
The execution of the feature extraction pipeline on the 23,313 issues revealed that only 0.4% of the issues followed the canonical user story format and only 7.1% contained explicit acceptance criteria. The average description length was 670 characters, with some records exceeding 50,000 characters due to the inclusion of stack traces or full logs. The average Flesch-Kincaid readability score was 11.45, consistent with university-level technical texts. These structural data were pre-processed, with continuous numerical features normalized and binary features kept at their original scale, before partitioning into training and testing sets.
Comparative performance of the models
The three supervised learning algorithms – Linear Regression, Random Forest, and XGBoost – were evaluated on the test set (4,663 issues) using the Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and coefficient of determination (R²) metrics. XGBoost achieved the lowest MAE, with 4.64 story points, closely followed by Random Forest, with 4.66 story points, and Linear Regression, with 4.68 story points. Random Forest presented the highest R² (0.026) and the lowest RMSE (9.15). The absolute differences between the models were marginal, with a variation of only 0.04 story points in MAE and 0.020 in R², indicating a notable performance proximity.
This convergence of performance between algorithms of distinct natures (a linear model, a bagging ensemble, and a gradient boosting ensemble) suggested that the predictive ceiling was imposed by the feature space, rather than by the intrinsic capacity of the algorithms. Linear Regression achieved nearly identical performance to the tree-based models, which implies that most of the predictive signal available in the nine structural features has already been extracted by a weighted linear combination. This finding is relevant for scenarios with computational cost constraints, where a simpler model may be preferable.
The hyperparameters selected by the grid search for the tree-based models were conservative, with a maximum depth (max_depth) of 10 for both, aiming to avoid overfitting. XGBoost used a low learning rate (0.01) and a smaller number of trees (100), while Random Forest compensated for the absence of sequential boosting with a larger number of estimators (500). Column subsampling (colsample_bytree = 0.7) in XGBoost acted as additional regularization. These configurations reinforced the hypothesis that the feature space offered limited predictive signal for the regression task, and that the search for more aggressive parameters resulted in overfitting on the validation set.
Analysis of individual predictions
The three models exhibited a regression to the mean pattern, with predictions concentrated around approximately six story points, regardless of the actual value assigned to the issue. For issues with story points greater than 20, the models systematically underestimated the estimates, rarely exceeding ten story points. This behavior stems from the strong asymmetry of the story point distribution in the dataset, where approximately 75% of issues received up to 8 story points. The models learned to predict values close to this region of higher density, which limited the usefulness of point predictions for high-complexity issues, precisely where decision support is most critical.
The residual distributions (difference between the actual value and the predicted value) of the three models were approximately symmetric around zero, with a high concentration of small errors (in the range of -5 to +5 story points) and a heavy right tail, indicating underestimation in issues with high values. This characteristic is expected in distributions with pronounced positive skewness and high kurtosis, such as that of story points in this study. The proximity of the residual distributions between the algorithms reinforced the evidence that predictive capability was limited by the feature space, not by the algorithm choice, a finding similar to that documented by Tawosi et al. (2022a).
Importance of features in models
The analysis of feature importance by the models’ internal mechanisms (mean decrease in impurity – MDI for Random Forest and gain of information for XGBoost) revealed that, in Random Forest, four features concentrated 90.6% of the total importance: the title/description ratio (29.4%), description readability (24.5%), description length (21.7%), and title length (15.1%). These variables captured complementary dimensions of the issue text, such as the proportion between title and description, linguistic complexity, absolute length, and conciseness of the statement, respectively.
In XGBoost, feature importance was distributed more uniformly. A notable finding was the jump of the `keyword_complexity` feature, which moved from seventh position in Random Forest (4.7%) to second position in XGBoost (17.4%). This difference can be attributed to the gradient boosting mechanism, which adjusts sequential trees on the residuals of the previous iteration, allowing it to capture residual signals in the frequency of technical terms that Random Forest, based on bagging, did not prioritize in its splits. The binary features `is_user_story` and `has_acceptance_criteria` were the least relevant in both models, with importance below 0.1% and 0.2% respectively, due to the extreme imbalance of these variables in the dataset.
Explainability via SHAP
The explainability analysis via SHAP (SHapley Additive exPlanations) revealed significant divergences compared to the MDI/gain importance metrics, especially for the `code_snippet_count` feature. In Random Forest, this feature ranked seventh in MDI (0.022), but rose to first place in average SHAP (0.80). This inversion occurred because SHAP, based on cooperative game theory, captures the marginal effect of each feature on individual prediction, while MDI accounts for usage frequency weighted by impurity reduction. The `code_snippet_count` was used in few tree splits, but with substantial impact on individual predictions, an effect that MDI underestimated. This discrepancy reinforced the need for explainability techniques with axiomatic foundations.
Despite the magnitude differences in SHAP values between Random Forest and XGBoost (Random Forest values were approximately 2.5 times larger, reflecting the model’s higher variance), both algorithms converged on the five most important features: `code_snippet_count`, `title_length`, `desc_length`, `description_readability`, and `title_desc_ratio`. This convergence between models with distinct learning mechanisms provided evidence that these variables possess a genuine predictive signal and are stable indicators for agile management professionals, regardless of the algorithm employed.
Influence Direction
The analysis of the direction of influence, through the beeswarm graph, demonstrated that the `keyword_complexity` feature exhibited a strong positive correlation between its value and the SHAP value. Issues with a high frequency of technical terms, such as “refactor”, “migrate”, “integration”, “API”, and “database”, received high positive SHAP values, indicating that the model increased the story points estimate for these issues. This pattern is interpretable in the context of agile management, as tasks involving these terms tend to be more complex and demand greater effort. For a Scrum Master, this information is actionable, allowing for a re-examination of estimates that were driven by these terms.
In contrast, the `desc_length` feature showed a weak directional pattern. Long descriptions were associated with both positive and negative SHAP values, suggesting that description length, in isolation, is not a reliable indicator of effort. A description can be long due to detailed requirements or, alternatively, by including redundant information or complete logs, which does not necessarily imply greater effective complexity. This result challenges the common intuition that “long description equals complex task,” and the same pattern was observed in Random Forest, indicating that teams relying on this informal heuristic may be basing their decisions on a weak signal.
The `title_length` and `title_desc_ratio` features showed non-linear relationships with SHAP. For `title_length`, short titles were associated with predominantly negative SHAP values, while long titles did not produce a consistent direction. For `title_desc_ratio`, extreme values (very high or very low) generated greater impacts than intermediate values. This suggests that the disproportion between the title and the description, in either direction, is more informative than the proportion itself, indicating that an imbalance in textual communication may signal complexity or lack of clarity.
Local Analysis
The waterfall visualization decomposed the prediction of an individual observation into contributions from each feature, starting from the expected value (average of predictions) and reaching the predicted value. This individual traceability allowed us to identify which features pulled the prediction up or down and by what magnitude. This transparency is crucial for Scrum Masters and Product Owners, as it allows them to compare the model’s estimate with the team’s estimate and identify discrepancies attributable to specific textual characteristics of the issue. For example, if the model estimates a substantially higher value than the team, the waterfall decomposition can reveal that a high count of technical terms or the presence of code blocks inflated the prediction, providing actionable insights to re-examine the human estimate and mitigate anchoring bias, as described by Mahnič and Hovelja (2012).
Interactions between features
The dependency plot revealed an interaction between `keyword_complexity` and `desc_length`. For issues with a high frequency of technical terms, the effect on prediction was amplified when the description was also long. A description combining dense technical vocabulary with considerable length signaled a task involving multiple dimensions of complexity (architectural, integration, refactoring), justifying higher estimates. This interaction would not be captured by a univariate importance analysis (MDI or gain), highlighting the value of SHAP dependency plots for identifying multivariate patterns in the data. The Random Forest results for the four SHAP visualizations were qualitatively similar to those of XGBoost, with the same predominant features and the same directional patterns, and the five features with the highest impact coincided entirely between the two models.
Contextualization with the literature
The MAE values obtained, between 4.64 and 4.68 story points, were higher than those reported by studies that employed high-dimensionality textual representations. Choetkiertikul et al. (2019) achieved an MAE of 2.08 with Long Short-Term Memory (LSTM) networks, and Yalçıner et al. (2024) reported an MAE of 2.15 with Sentence-BERT combined with gradient boosted trees. The difference in predictive accuracy reflected a deliberate design decision, prioritizing the interpretability of the nine features over maximum predictive capability. High-dimensional semantic representations, while capturing complex linguistic nuances, remain opaque to Scrum Masters and Product Owners, hindering the understanding of the “why” behind the estimates.
The robustness of the metrics reported in the literature was questioned by Tawosi et al. (2024), who, upon replicating Fu and Tantithamthavorn’s (2023) GPT2SP, identified an error in the original MAE calculation and demonstrated that a predictor based on the median of story points per project achieved competitive performance with the transformer model after correction. This finding partially explained the R² values close to zero obtained in this work, as the median already constituted a strong predictor, and the residual space for learning was narrow. Even with R² close to zero, the implemented models produced differentiated predictions per issue and allowed, via SHAP, to track which textual attributes influenced each individual prediction, a traceability that a median predictor did not offer. Alexander and Jayadi (2024) reached an analogous conclusion in a direct comparison, in which traditional ML models maintained competitive performance with Large Language Models in story point prediction, reinforcing that the adoption of interpretable models did not imply a significant loss of accuracy compared to state-of-the-art models.
The distribution of story points in Deep-SE followed the Fibonacci scale (1, 2, 3, 5, 8, 13, 21…), used to reflect increasing uncertainty in larger tasks. This scale is ordinal, not interval, and the continuous treatment of the variable, shared by all reviewed studies (Choetkiertikul et al., 2019; Fu and Tantithamthavorn, 2023; Yalçıner et al., 2024), may have contributed to the models’ difficulty with higher values on the scale. Tawosi et al. (2022b) documented that the correlation between story points and actual effort was strong in only 7% of projects, with imprecise estimates in 78% of cases, indicating that story points capture perceptions of relative complexity and not absolute effort. SHAP analysis therefore revealed the textual factors that influenced these perceptions, rather than the actual implementation time.
Fernández-Diego et al. (2020) found that subjective techniques, such as planning poker and expert judgment, remain dominant in the industry, despite evidence of automated approaches. The gap was attributed to the lack of transparency of ML models, as managers resist tools whose predictions cannot be interpreted. Pasuksmit et al. (2024) identified the quality of information about issues as the most frequent reason for inaccuracy in estimates, a dimension that SHAP analysis addressed by quantifying the influence of attributes such as textual length, technical vocabulary, and code blocks. Priya Varshini et al. (2025) validated the Random Forest + SHAP combination in software effort estimation, reporting improved manager confidence, a methodological convergence that reinforced the relevance of the interpretable approach adopted in this work.
Limitations of the study
The study’s central limitation resided in the restricted space of nine structural and textual features. The exclusion of high-dimensional semantic representations was a deliberate choice to ensure the interpretability of SHAP results for Scrum Masters and Product Owners. However, this decision resulted in a limited predictive signal, evidenced by R² values close to zero. In practice, a coefficient of this magnitude implies that point prediction should not be adopted in isolation for critical decisions, as the captured variance was insufficient for capacity planning or resource allocation. The value of the approach resided in the SHAP decomposition, which transformed individually imprecise predictions into actionable diagnostics about the textual attributes that influenced each estimate.
Story points are inherently noisy as a target variable, as documented by Tawosi et al. (2022b), who observed a strong correlation with actual effort in only 7% of projects. This intrinsic noise sets a ceiling on predictability regardless of the model employed. Furthermore, the 16 Deep-SE projects were open-source, with heterogeneous estimation practices and agile maturity. Training in cross-project mode, rather than project-specific, diluted team-specific patterns, which may have limited performance. Generalization to proprietary projects with stable teams and standardized practices remains an open question, depending on access to confidential data.
Treating story points as a continuous variable was a simplification, given that the Fibonacci scale is ordinal and traditional regression did not ideally capture its discrete structure. Khan et al. (2025) proposed a comparative learning framework that infers relative effort, which may be more consistent with the planning poker decision-making process. Another limitation was the absence of temporal features, such as position in the backlog, team historical velocity, or sprint phase. Tawosi et al. (2022b) indicated that these features could complement structural variables and expand the predictive power of the models, as the correlations between story points and actual effort vary throughout the project lifecycle.
Finally, the computational cost of SHAP in production environments represents a limitation. Although TreeExplainer was feasible for the test set of this work (4,663 observations), calculating Shapley values for real-time inference or sets with millions of issues may require strategies such as caching, sampling, or stochastic approximations (KernelSHAP). These considerations are important for the scalability of the solution in high-volume data industrial contexts.
Practical implications for agile management
SHAP analysis translated the predictive results into practical recommendations for Product Owners and Scrum Masters. For Product Owners, the `keyword_complexity` feature emerged as a consistent indicator that issues mentioning terms like “refactor”, “migrate”, “integration”, “API”, and “database” receive systematically higher estimates. This information is crucial for mitigating anchoring bias during planning poker sessions, where superficial familiarity with these terms can mask the task’s real complexity. SHAP decomposition offers a counterpoint to bias, flagging issues that deserve in-depth discussion before voting, allowing for a more informed estimate.
For Scrum Masters, the waterfall visualization functions as an individual diagnostic tool. It allows comparing the model’s estimate with the team’s and identifying the features responsible for the divergence. If the model assigns a high estimate to an issue evaluated with a low value by the team, and the decomposition reveals `code_snippet_count` and `keyword_complexity` as drivers, the Scrum Master can reopen the discussion about technical complexity. Planning poker generates alignment through discussion, and SHAP generates alignment through the transparency of predictive factors, acting as complementary and non-competing approaches, according to the perspective of Moløkken-Østvold and Haugen (2007).
The study’s approach reinforces that the model’s role is to provide an additional perspective, based on patterns extracted from a large volume of data, enriching the discussion without replacing the team’s judgment. The story point scale is a social convention, and SHAP made explicit the structural reasons behind the prediction, allowing the team to assess whether the same factors were considered in their estimate, in line with Schwaber and Sutherland’s (2020) view on the responsibility for sizing by developers. The low prevalence of formal specification practices in the dataset (0.4% of canonical user stories and 7.1% of explicit acceptance criteria) constituted an applicable finding. Teams that systematically adopt acceptance criteria and a standardized user story format can improve both the predictability of estimates and the richness of features available for predictive models, as Pasuksmit et al. (2024) identified information quality as a critical factor for accuracy.
In summary, the research demonstrated that, although predictive models based on interpretable features showed modest performance in terms of point accuracy (MAE of 4.64 story points and R² close to zero), the combination with the SHAP technique transformed them into a valuable decision support tool in agile planning. The ability to explain the “why” of each estimate, identifying the contribution of textual attributes such as the number of code blocks, title length, and frequency of technical terms, allowed Scrum Masters and Product Owners to understand and validate the predictions, mitigating biases and enriching planning discussions, which is fundamental for the effective adoption of Machine Learning models in agile management contexts.
4. Conclusion
This study evaluated the feasibility of interpretable machine learning models as decision support in agile planning, combining story point prediction with explainability via SHAP. Three supervised learning algorithms – Linear Regression, Random Forest, and XGBoost – were implemented to predict story points from nine structural and textual features, extracted from 23,313 issues across 16 open-source projects. It was found that XGBoost achieved the lowest Mean Absolute Error (4.64 story points), but the three models exhibited similar predictive performance, indicating that the predictability ceiling was imposed by the feature space, rather than the algorithms’ intrinsic capabilities. SHAP analysis quantified the contribution of each feature, identifying the number of code blocks, title length, and frequency of technical terms as the attributes with the greatest impact on estimates. The main contribution of this work lies in configuring a decision support tool that, by making explicit the textual attributes influencing estimates, allows Scrum Masters and Product Owners to understand and validate predictions, mitigating biases and enriching agile planning discussions, overcoming the “black box” limitation of previous predictive approaches.
However, the study presented important limitations, such as the restricted space of nine structural and textual features, a deliberate choice that resulted in a limited predictive signal and R² values close to zero. Furthermore, story points are inherently noisy as a target variable, and training in cross-project mode on open-source project data introduced heterogeneity in estimation practices. Treating the Fibonacci scale as a continuous variable and the absence of temporal features also constituted simplifications. For future studies, it is suggested to explore hybrid architectures that combine interpretable features with low-dimensionality semantic representations to increase accuracy without sacrificing transparency. Empirical validation of the approach with Product Owners and Scrum Masters in qualitative studies is also recommended, in order to measure the effect of SHAP visualizations on the calibration of estimates and on the dynamics of planning poker sessions, bringing ML models closer to effective adoption in agile management environments.
Bibliographic References
Choetkiertikul, M.; Dam, H.K.; Tran, T.; Pham, T.; Ghose, A
Mahnič, V.; Hovelja, T. 2012. On using planning poker for estimating user stories. Journal of Systems and Software 85(9): 2086-2095.
Moløkken, K.; Jørgensen, M. 2003. A review of surveys on software effort estimation. IEEE Transactions on Software Engineering 29(10): 1104-1115.
Pasuksmit, J.; Thongtanunam, P.; Karunasekera, S. 2024. A systematic literature review on reasons and approaches for accurate effort estimations in agile. ACM Computing Surveys 56(11): 1-37.
Porru, S.; Murgia, A.; Demeyer, S.; Marchesi, M.; Tonelli, R. 2016. Estimating story points from issue reports. In: International Conference on Predictive Models and Data Analytics in Software Engineering, 12., 2016, Ciudad Real, Espanha. Proceedings… p. 1-10.
Schwaber, K.; Sutherland, J. 2020. The Scrum Guide: the definitive guide to Scrum, the rules of the game. Disponível em: <https://scrumguides.org/>. Acesso em: 30 mar. 2026.
Article originating from the Final Course Work of the Specialization in Software Engineering of the MBA USP/Esalq
To learn more about the course, click here and access the MBX Academy platform