Modeling of Default Probability in Credit Risk: Comparison between Logistic Regression and Xgboost

Berenice Antunes de Oliveira Barbosa; Beatriz Rezzieri Marchezini

DOI: 10.22167/2675-6528-202602713

Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

The probability of default in credit risk was modeled, comparing the performance of a traditional statistical model and a machine learning technique in estimating the Probability of Default (PD), a crucial parameter in credit risk management and Expected Loss calculation. The study aimed to compare logistic regression, a consolidated statistical method, and XGBoost (Extreme Gradient Boosting), a machine learning algorithm recognized for its predictive capacity and ability to capture non-linear relationships. For the analysis, the public dataset “Give Me Some Credit” from the Kaggle platform was used, which contained financial and demographic information of clients, including age, income, indebtedness, and payment delay history. The models were estimated and evaluated based on predictive performance metrics, such as Area Under the ROC Curve (AUC), Gini coefficient, and Kolmogorov-Smirnov (KS) statistic. The results indicated that the XGBoost model showed superior performance to logistic regression, demonstrating a greater discriminatory capacity in ranking clients by default risk. It was concluded that, although machine learning models can provide relevant predictive gains, the choice of the most suitable model should consider not only performance metrics but also aspects such as interpretability, transparency, and practical applicability, contributing to a more accurate measurement of credit risk.

Keywords: Machine Learning; Risk Management; Default; Performance Metrics; Classification Models.

1. Introduction

The credit risk area plays an essential role in financial institutions, being directly linked to the financial health and sustainability of credit operations. Credit risk is considered the main type of risk faced by banks, directly influencing their profitability and solvency (Van Gestel and Baesens, 2009). A well-structured risk model assists in measuring and monitoring nonpayment, contributing to loss reduction, efficient capital allocation, and safer decision-making throughout the credit cycle. Small improvements in nonpayment prediction can generate significant gains in efficiency and profitability, which justifies the growing interest in improving the modeling techniques used.

In the context of credit risk management, the Probability of Default (PD) stands out as one of the main parameters used in the measurement of Expected Loss. PD expresses the probability of a client becoming delinquent within a given time horizon and is a fundamental input for financial institutions in defining provisions and allocating capital. Thus, the precise estimation of PD plays a central role in modern risk management practices, being widely used in both regulatory and internal models.

In recent years, an important transition has been observed from traditional statistical models to approaches based on machine learning. Machine learning can be understood as a set of techniques that allow computers to learn patterns and make predictions from data, without relying on pre-programmed fixed rules (Mitchell, 1997). These techniques stand out for their ability to handle large volumes of data and identify complex relationships that often go unnoticed in conventional models.

Several studies reinforce the potential of these approaches in credit risk. Lessmann et al. (2015) and Moro, Cortez, and Rita (2019) show that machine learning algorithms can capture more subtle patterns of credit behavior, which tends to increase the assertiveness of predictions. Furthermore, Zhang, Xie, and Chen (2022) highlight that models like XGBoost often show superior performance in differentiating between good and bad payers, evidencing the advancement that these techniques represent for the financial sector.

Given this scenario of technological evolution and the pursuit of greater precision in risk measurement, this work focuses on estimating the probability of default, one of the main parameters used in credit risk assessment and a fundamental input for the calculation of Expected Loss. To conduct the analysis, two representative models are considered: logistic regression, a traditional statistical method widely used in credit risk for estimating the probability of default from the linear relationship between explanatory variables (Thomas, 2009; Hand and Henley, 1997), and XGBoost (Extreme Gradient Boosting), a modern machine learning technique that combines multiple decision trees to capture non-linear interactions and complex patterns, recognized for its high performance in classification and default prediction tasks (Li, Zhou and Zhu, 2021; Zhang, Xie and Chen, 2022).

The database used for the analysis corresponds to the public dataset Give Me Some Credit, available on the Kaggle platform, which contains financial and demographic information of clients, such as age, income, indebtedness level, and payment delay history. From this data, the models are estimated and evaluated based on predictive performance metrics, allowing for comparison between approaches and verification of possible performance gains with the use of machine learning techniques compared to the traditional model, contributing to a more accurate measurement of credit risk. This work aims to compare the performance of a traditional statistical model and a machine learning technique in estimating the Probability of Default (PD), one of the main parameters used in credit risk management and in the calculation of Expected Loss.

2. Material and Methods

This research was characterized as applied, with a predominantly quantitative approach and a descriptive-comparative nature. The study was designed to evaluate, through statistical methods and machine learning algorithms, the performance of different models in estimating the probability of nonpayment in the context of credit risk, aligning with the objective of comparing a traditional statistical model and a machine learning technique in estimating the Probability of Default (PD).

The data used in this work were obtained from the public dataset “Give Me Some Credit”, available on the Kaggle platform (Kaggle, 2011). The database gathered financial and demographic information of clients, totaling 150,000 observations. This dataset included variables such as age, monthly income, debt level, number of dependents, and history of payment delays.

The dependent variable, named “Default”, indicated the occurrence of severe nonpayment in the last 24 months. The other variables, such as “Credit Utilization”, “Age”, “Monthly Income”, “Credit Lines”, “90+ Day Delinquency”, “Mortgage”, “30-59 Day Delinquency”, “60-89 Day Delinquency”, and “Dependents”, were used as predictor variables in the logistic regression and machine learning models.

The dataset was used exclusively for academic purposes, without identifying individuals or financial institutions. This approach ensured compliance with the ethical and privacy principles recommended by USP/ESALQ, ensuring that the research respected the confidentiality and anonymity of client data.

Initially, an exploratory data analysis was performed, which included descriptive statistics and checking for the presence of missing or inconsistent values. Subsequently, the data underwent preprocessing steps, which included handling missing values and outliers, as well as preparing the variables for model application.

In the treatment of missing values, it was observed that only the variables “monthly income” and “dependents” presented gaps. For “monthly income”, the missing values were replaced by the median, while for “dependents”, they were filled with zero. Additionally, the presence of extreme values was evaluated, and the “monthly income” variable presented significant outliers.

For the logistic regression modeling, a logarithmic transformation was applied to the “monthly income” variable in order to reduce the impact of outliers. However, for the XGBoost model, the original values of the variable were maintained, considering the algorithm’s intrinsic robustness to extreme values.

The database was organized and segmented into training and test sets. The training data was used for parameter estimation of the models, and the test data, for performance evaluation. This division allowed for the analysis of the models’ generalization capability on data not used during fitting, ensuring comparability between approaches.

In the modeling stage, two models were estimated for the probability of nonpayment. The first was Logistic Regression, a traditional statistical model that estimates the probability of nonpayment occurrence from a linear combination of explanatory variables, related to the probability of the event through the logistic function (Thomas, 2009; Hand and Henley, 1997).

The second model employed was XGBoost (Extreme Gradient Boosting), a machine learning algorithm based on gradient boosting. This algorithm sequentially combines multiple decision trees, capturing non-linear relationships and complex interactions among the explanatory variables (Chen and Guestrin, 2016; Li, Zhou and Zhu, 2021; Zhang, Xie and Chen, 2022).

For the performance comparison of the models, the main metrics used were the Area under the ROC Curve (AUC), the Gini coefficient, and the Kolmogorov-Smirnov (KS) statistic. The choice of these metrics aimed to evaluate the discriminatory capacity of the models in ordering clients according to the risk of default, being relevant in credit risk modeling applications.

The analyses were developed in Python language, using the Google Colab environment. For data manipulation, modeling, and visualization of results, the pandas, scikit-learn, xgboost, and matplotlib libraries were employed. The modeling and analysis techniques applied were studied throughout the MBA in Data Science and Analytics course at ESALQ/USP.

3. Results and Discussion

The data analysis began with an exploratory step, whose objective was to understand the behavior of the variables used in the modeling of Probability of Default (PD). During this phase, the distributions of the variables, the occurrence of extreme values, and the degree of imbalance of the response variable were examined. These insights were crucial for guiding the subsequent steps of data treatment and the construction of predictive models, ensuring that the intrinsic characteristics of the dataset were adequately considered in the modeling process.

The distribution of the response variable, which indicates the occurrence of severe default in the last 24 months, revealed a marked imbalance in the database. Approximately 6.7% of the records were classified as defaulters, while the vast majority, about 93.3%, corresponded to non-defaulters. This pattern is common in real-world credit risk applications and underscores the importance of selecting performance evaluation metrics that are robust to this characteristic, avoiding biases in the interpretation of model results.

The descriptive statistics of the variables used in the probability of default modeling were calculated for a base of 150,000 observations, confirming an average default rate of approximately 7%. It was noted that the monthly income variable exhibited high dispersion and the presence of extreme values, indicating a strong asymmetry in its distribution. Similarly, the variables related to payment delays, such as delay_30_59, delay_60_89, and delay_90, presented a median equal to zero, suggesting a concentration of clients with no recent history of non-payment. These asymmetries and concentrations in the explanatory variables were considered relevant for the preprocessing and modeling stages.

The correlation matrix between the dataset variables allowed for the evaluation of the intensity and direction of linear associations. A high positive correlation was observed between the variables indicating payment delay (delay_30_59, delay_60_89, and delay_90). This finding is expected, as all these variables are intrinsically linked to the deterioration process of payment behavior, differing only by the ranges of days of delay. The other variables in the set generally showed low magnitude correlations among themselves, indicating less linear interdependence.

The presence of a high correlation between the explanatory variables, especially those related to delays, raised the possibility of multicollinearity. Although multicollinearity can affect the stability and interpretability of the estimated coefficients in logistic regression, the decision was made to keep these variables in the modeling. The decision was based on the theoretical relevance of these characteristics for explaining the risk of default and the need for predictive comparability between models, even if this implied coefficients with unexpected signs or magnitudes in one of the models.

In the preparation and treatment phase of the variables, the database underwent crucial steps to ensure quality and suitability for modeling. This included standardizing variable names, checking and treating missing values, and consistency analysis. Specifically, the monthly income variable and the number of dependents were the only ones to present missing values, being treated by substitution with the median for monthly income and filling with zero for dependents, respectively.

Additionally, the presence of extreme values was evaluated, and the monthly income variable was identified as having significant outliers. For the logistic regression model, a logarithmic transformation was applied to monthly income, aiming to mitigate the impact of these extreme values. In contrast, for the XGBoost model, the original monthly income values were maintained, leveraging the algorithm’s intrinsic robustness to outliers. This differentiated approach ensured the comparability between results, adapting the data treatment to the characteristics of each modeling technique.

Default Probability Modeling

Logistic Regression

Logistic Regression was the first model estimated for the probability of default. This statistical model is widely used in credit risk, being suitable for binary response variables, such as the occurrence of default (1 for defaulter, 0 for non-defaulter). Unlike Linear Regression, logistic regression uses a logistic function to relate the explanatory variables to the probability of the event, ensuring that the estimated probabilities remain in the interval between zero and one, which is fundamental for the interpretation of probabilities.

The results of the logistic regression indicated that variables associated with the delay history showed a positive association with the probability of default. In contrast, age and monthly income, after logarithmic transformation, exhibited a negative association with the probability of nonpayment. These findings are in line with the credit risk literature, which frequently points to payment history as a strong predictor of nonpayment and age and income as risk mitigation factors.

When analyzing the coefficients and odds ratios, it was found that variables related to the default history had the greatest impact on the probability of default. Delays exceeding 90 days and delays between 30 and 59 days stood out, with odds ratios of 1.626 and 1.6347, respectively. This means that for each unit increase in these variables, the odds of default increase significantly. However, the variable referring to delays between 60 and 89 days presented a negative coefficient (-0.9456) and an odds ratio of 0.3884, which contradicts theoretical expectations. This behavior may be attributed to the high correlation of this variable with other delay ranges, influencing the estimation of coefficients when considered together in the model.

The confusion matrix of the logistic regression model revealed good performance in identifying non-defaulting customers, but low sensitivity in identifying defaulters. This result is a direct consequence of the imbalance in the response variable and the use of a standard cutoff point of 50%, which tends to favor the majority class. This observation reinforces the inadequacy of a fixed threshold and the need for evaluation metrics that consider the ordering of customers by risk, such as the KS statistic, or specific business criteria for defining cutoff points.

The predictive performance metrics for logistic regression were: Area Under the ROC Curve (AUC) of approximately 0.702, Gini coefficient of about 0.404, and Kolmogorov-Smirnov (KS) statistic of 0.330. These values indicate a moderate discriminatory capacity in ranking customers by default risk. Although not an exceptional performance, these results are compatible with practical credit risk applications and confirm the model’s suitability for PD estimation, serving as a solid basis for comparison with more complex algorithms.

XGBoost

XGBoost (Extreme Gradient Boosting) was the second model implemented, representing a machine learning technique based on gradient boosting. This algorithm builds multiple decision trees sequentially, where each new tree seeks to correct the residual errors of the previous trees. This architecture allows XGBoost to capture non-linear relationships and complex interactions between the explanatory variables, in addition to incorporating regularization mechanisms that help control overfitting, as highlighted by Chen and Guestrin (2016).

The confusion matrix for the XGBoost model demonstrated an improvement in identifying delinquent customers compared to logistic regression. An increase in true positives was observed, indicating greater model sensitivity to the delinquency class. However, this improvement was accompanied by an increase in false positives. This trade-off between sensitivity and specificity is common in classification models and reflects XGBoost’s ability to identify more positive cases, even if it means misclassifying some negative cases.

The performance metrics of the XGBoost model were significantly superior to those of logistic regression. The Area Under the ROC Curve (AUC) was approximately 0.865, the Gini coefficient was 0.73, and the KS statistic was 0.576. These results indicate a good to excellent discriminatory capacity of the model in identifying customers with a higher probability of default. The efficiency of XGBoost in capturing complex patterns and non-linear relationships between the explanatory variables was evidenced by these values, reinforcing its potential in default prediction tasks.

The analysis of the importance of variables in the XGBoost model revealed that delay history is the main factor for predicting default. The variable delay_90, which represents delays greater than 90 days, was the most important, with a value of 0.4063. Credit utilization and other delay indicators (delay_30_59 and delay_60_89) also showed high relevance. Customer profile variables, such as age, monthly income, and number of dependents, although contributing to risk classification, had lower relative importance. These results reinforce that past payment behavior is a crucial determinant in assessing the Probability of Default.

Comparison of Models

The direct comparison between the logistic regression and XGBoost models, based on the performance metrics obtained on the test set, revealed significant predictive gains with the machine learning approach. The XGBoost model presented superior performance in all evaluated metrics: AUC of 0.865, Gini of 0.73, and KS statistic of 0.576. In contrast, logistic regression obtained an AUC of 0.702, Gini of 0.404, and KS of 0.330. This convergence of results across the three metrics confirms XGBoost’s greater discriminatory capacity in ordering customers according to default risk.

The gain observed in the ability to discriminate between clients with different risk levels is relevant for practical applications. A better ordering of individuals by probability of default can provide more accurate information, supporting more effective decisions in credit risk management. This translates into more efficient capital allocation and reduced losses, directly impacting the profitability and sustainability of financial operations, as discussed by Van Gestel and Baesens (2009).

The superior performance of XGBoost can be attributed to its ability to capture complex relationships and non-linear interactions between variables, something that linear statistical models like logistic regression cannot do with the same effectiveness. Furthermore, the algorithm incorporates regularization mechanisms and sequential learning (boosting), which increase its predictive capacity and contribute to the control of overfitting, as pointed out by Chen and Guestrin (2016). This result is aligned with previous studies, such as that of Lessmann et al. (2015), who observed the superior performance of machine learning models compared to traditional methods in credit risk classification problems.

Despite the inferior performance in this study, logistic regression maintains its relevance. Its main advantage lies in the high interpretability and transparency of the results, fundamental aspects in credit decisions and regulatory environments, where the estimation of PD is used to calculate expected loss (Hand and Henley, 1997; Thomas, 2009). The ability to understand the contribution of each variable to the probability of default is a differential that often outweighs the pursuit of marginal gains in predictive performance in specific contexts.

In general, the results obtained indicate that variables related to the delay history play a central role in PD estimation, which is in line with the literature in the area. Both models demonstrated discrimination capacity, but XGBoost stood out in ordering the default risk. This study reinforces the importance of considering not only predictive power, but also interpretability and practical applicability in choosing the most suitable model for credit risk management, especially in a corporate environment.

As a strong point of this study, the adoption of different modeling approaches applied to credit risk stands out, as well as the use of robust evaluation metrics, such as AUC, Gini, and KS. This methodology allowed for a consistent analysis of the models’ discriminatory capacity and provided greater reliability in comparing the results. The incorporation of these metrics aligns the evaluation carried out in this study with the practices used in credit risk modeling and validation, expanding the practical applicability of the comparison between models.

The limitations of this study include the imbalance of the database and the possible absence of additional variables that could improve the explanation of the default risk. It is important to note that, in real credit risk applications, maintaining the original data distribution is often adopted, as balancing techniques can distort the PD estimate. Furthermore, the use of a specific database can limit the generalization of the results, which requires that the findings be interpreted with caution when applied to other contexts.

In summary, the research demonstrated that, although logistic regression is a robust and interpretable model, the XGBoost algorithm presented superior predictive performance in the estimation of Probability of Default, evidencing its greater discriminatory capacity in ordering clients according to their default risk. The findings underline the importance of delay history as a central predictor and reinforce that the choice of the ideal model should balance predictive power with interpretability and practical applicability in the context of credit risk management.

4. Conclusion

This study aimed to compare the performance of logistic regression and the XGBoost algorithm in estimating the Probability of Default (PD), a fundamental parameter in credit risk management. It was found that the XGBoost model demonstrated superior predictive capability compared to logistic regression, evidenced by metrics such as an Area Under the ROC Curve (AUC) of 0.865, a Gini coefficient of 0.73, and a Kolmogorov-Smirnov (KS) statistic of 0.576. In contrast, logistic regression achieved AUC values of 0.702, Gini of 0.404, and KS of 0.330, indicating a moderate discriminatory power. The findings revealed that payment delay history constitutes the main predictor of default in both models, particularly delays exceeding 90 days, while customer profile variables, such as age and monthly income, showed lower relative importance. The superiority of XGBoost was attributed to its ability to capture non-linear relationships and complex interactions among the predictor variables.

The main contribution of this work lies in demonstrating the predictive gains that machine learning techniques, such as XGBoost, can offer in credit risk measurement, promoting a more accurate assessment of the probability of default. However, it is emphasized that the choice of the most suitable model in a corporate environment must consider not only predictive performance but also interpretability, transparency, and practical applicability, attributes in which logistic regression still excels. The study’s limitations include the imbalance of the database and the use of a specific dataset, which may restrict the generalization of the results. For future studies, it is suggested to explore other machine learning algorithms and include additional predictor variables, aiming to further enhance the predictive capability and applicability of the models in various credit risk contexts.

Bibliographic References

CHEN, T.; GUESTRIN, C. XGBoost: A Scalable Tree Boosting System. 2016.

HAND, D. J.; HENLEY, W. E. Statistical classification methods in consumer credit scoring: a review. Journal of the Royal Statistical Society: Series A (Statistics in Society), v. 160, n. 3, p. 523-541, 1997.

KAGGLE. Give Me Some Credit Dataset. Disponível em: https://www.kaggle.com/c/GiveMeSomeCredit. Acesso em: 04 nov. 2025.

LESSMANN, S.; BAESENS, B.; SEOW, H. V.; THOMAS, L. C. Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research. European Journal of Operational Research, v. 247, n. 1, p. 124-136, 2015.

LI, J.; ZHOU, Z.; ZHU, L. Credit risk assessment based on XGBoost algorithm. Procedia Computer Science, v. 199, p. 274–281, 2021.

MITCHELL, T. M. Machine Learning. New York: McGraw-Hill, 1997.

MORO, S.; CORTEZ, P.; RITA, P. Business intelligence in banking: A literature analysis from 2002 to 2013 using text mining and latent Dirichlet allocation. Expert Systems with Applications, v. 42, n. 3, p. 1314–1324, 2015.

THOMAS, L. C. Consumer Credit Models: Pricing, Profit and Portfolios. Oxford: Oxford University Press, 2009.

VAN GESTEL, T.; BAESENS, B. Credit Risk Management: Basic Concepts: Financial Risk Components, Rating Analysis, Models, Economic and Regulatory Capital. Oxford: Oxford University Press, 2009.

ZHANG, Y.; XIE, J.; CHEN, Y. Credit risk prediction using machine learning methods: Evidence from China’s peer-to-peer lending market. Journal of Finance and Data Science, v. 8, n. 1, p. 1-13, 2022.

Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform