Comparison between Logistic Regression and Boosting in Default Prediction

Bruno Nogueira Pereira; Alexandre Mendes da Silva

DOI: 10.22167/2675-6528-202602747

Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

A comparative analysis between logistic regression, a traditional method in credit risk assessment, and “ensemble” techniques such as LightGBM and XGBoost was presented. The main objective was to verify the effectiveness of these models in predicting default, using the “Home Credit Default Risk” “dataset”. A quantitative and applied methodology of an experimental nature was adopted, with data preprocessing and feature engineering. The performance of the models was evaluated using metrics such as AUC-ROC, Gini index, and confusion matrix. The SHAP technique was applied to interpret the influence of variables on the “ensemble” models. The results obtained indicated that the “ensemble” methods showed slight superiority in terms of predictive capacity compared to logistic regression, better capturing the complexities of the “dataset”. The SHAP analysis highlighted the importance of variables such as external “score” sources and the client’s financial characteristics. It was concluded that the “ensemble” models offered slightly superior performance, reinforcing the need to consider interpretability and the imbalance of the target variable in the evaluation of predictive models for default.

Keywords: Nonpayment; LightGBM; Logistic Regression; Credit Risk; XGBoost.

1. Introduction

Consumer credit plays a fundamental role in the economy, as it facilitates the acquisition of goods and services, boosting consumption. However, the expansion of credit brings with it the challenge of risk management, which requires effective methodologies for its measurement and control.

Credit risk is understood as the measurement of uncertainty regarding the future receipt of an agreed amount, to be settled by the borrower (Gonçalves et al., 2013). Within this context, the risk of default/ non-payment, which refers to the non-fulfillment of the payment obligation of a credit operation, represents one of the main concerns for financial institutions (Çallı and Coşkun, 2021; Duffie and Singleton, 2003).

Effective credit risk management is essential for the financial health of institutions, allowing for the judicious allocation of resources and the prevention of losses. Default/ Non-payment is influenced by a variety of factors, including borrower characteristics, loan conditions, and macroeconomic variables (Camargos et al., 2010). Understanding these elements is crucial for the development of accurate predictive models.

In this scenario, binary classification models are widely employed to predict the probability of a customer becoming delinquent. Logistic Regression, for example, is a traditional and popular statistical technique in binary event modeling, valued for its interpretability and ability to identify the contribution of each factor to the risk (Hosmer et al., 2013; Araújo and Carmona, 2009; Gonçalves et al., 2013).

However, logistic regression can present limitations when dealing with complex non-linear relationships and significant interactions between variables (Coelho et al., 2021). In contrast, “ensemble” methods, such as “boosting” (Friedman, 2001; Zhou, 2012), which include algorithms like LightGBM and XGBoost (Chen and Guestrin, 2016; Ponsam et al., 2021), have demonstrated high predictive performance by capturing complex patterns in the data. The literature points, however, to a gap in the connection between high-performance models and those that offer good interpretability (Baesens et al., 2016; Lessmann et al., 2015).

Faced with this problem, the demand for models that combine good predictive performance with the ability to explain the factors influencing default arises. Interpretability is crucial to justify credit decisions to regulators, managers, and clients. To mitigate the “black box” nature of “ensemble” models, the SHAP (SHapley Additive exPlanations) technique offers a unified approach to interpret predictions, assigning to each feature a value that represents its contribution (Lundberg and Lee, 2017).

Thus, the present research is justified by the need to deepen the understanding of the balance between the predictive capacity and interpretability of models in the context of credit risk, using a real database of defaults. In this way, the main objective of this work is to investigate and compare the effectiveness of logistic regression and “ensemble” techniques, specifically “boosting” models such as LightGBM and XGBoost, in predicting defaults, and to analyze the interpretability of their predictions through the SHAP technique.

2. Material and Methods

In this study, a quantitative and applied methodology was adopted, focusing on the comparative analysis of predictive performance between different statistical and machine learning models. The research is experimental in nature, as practical tests were carried out with a real dataset, evaluating the models’ behavior across various performance metrics.

The methodological approach employed was comparative, contrasting logistic regression, a traditional method in credit risk modeling, with the models of

group

LightGBM and XGBoost. These models were selected for their ability to capture non-linear relationships and complex interactions between variables, allowing us to evaluate whether the superior predictive performance of the models of

increasing

justifies its greater complexity in relation to logistic regression.

The empirical object of the research was the “Home Credit Default Risk” dataset, made publicly available by the “Home Credit Group” company. This

data set

contains detailed information about loans and customers’ payment history, covering socioeconomic variables, data from

office

of credit and previous payment history.

The dependent variable, representing default, is binary, indicating whether the client defaulted (value 1) or paid normally (value 0). It was observed that the

data set

is unbalanced, with approximately 8% of defaulting clients, which was considered an additional challenge for predictive modeling.

The data preprocessing was a crucial step to ensure the quality and consistency of the models. Initially, the original column names were translated to Portuguese and converted to lowercase, aiming to facilitate interpretation and standardization in the work.

Next, feature engineering was performed, creating new variables to enhance the predictive capability and interpretability of the models. Variables such as age_years, calculated from birth_days, and several financial ratios, including credit_to_income, annual_fee_to_income, credit_to_annual_fee, and credit_to_asset_value, were generated. Anomalous values in the employed_days variable were treated as missing.

For the treatment of missing values, a strategy of removing variables that presented a proportion of missing data greater than 20% was adopted. This approach sought to avoid the inclusion of attributes with low information availability, which could compromise the quality of subsequent analyses.

The remaining categorical variables were transformed into numerical variables through the technique of

one-hot encoding

. Subsequently, the numerical variables were standardized using

StandardScaler

, ensuring that all were on the same scale and contributing to the stability of the models.

Three predictive models were implemented and compared: Logistic Regression, LightGBM, and XGBoost. Logistic Regression was implemented with default parameters and class balancing (

class weight = “balanced”

), without hyperparameter optimization, to serve as a baseline for comparison.

LightGBM was implemented using the LightGBM library, with class balancing and hyperparameter optimization

n_estimators

(100 and 200) and

number of leaves

(31 and 50) through the

GridSearchCV

. XGBoost was implemented with the XGBoost library, using class balancing via

scale_pos_weight

and hyperparameter optimization

n_estimators

(100 and 200) and

maximum depth

(3 and 5) also through the

GridSearchCV

.

For the optimization of LightGBM and XGBoost models, stratified cross-validation with 3

creases

, maintaining the proportion between classes in each split. The best combination of hyperparameters was selected based on the AUC-ROC metric. The data were divided into training (80%) and testing (20%) sets in a stratified manner, preserving the proportion of the target variable.

Logistic Regression is a statistical technique widely used for modeling of

credit scoring

, suitable for binary classification problems, such as default prediction (Araújo and Carmona, 2009). The model estimates the probability of an event occurring, being valued for its interpretability (Gonçalves et al., 2013; Hosmer et al., 2013).

The methods of

group

, such as LightGBM and XGBoost, combine multiple models, usually regression or classification trees, to achieve superior predictive performance (Zhou, 2012). The

increasing

, in particular, builds models sequentially, where each new model is trained to correct the errors of previous models (Friedman, 2001; Chen and Guestrin, 2016; Ponsam et al., 2021).

To evaluate the models’ performance on the test set, the following metrics were used: AUC-ROC (

Area Under the Receiver Operating Characteristic Curve

), which measures the ability to distinguish between classes; the Gini Index, derived from the ROC curve; the Confusion Matrix, which presents true positives, false positives, true negatives, and false negatives; and the Classification Report, which includes precision,

recollection

and F1-score for each class.

Considering the imbalance of the target variable, with a predominance of the non-default class, the performance of the models was evaluated not only by global metrics, such as AUC-ROC, but also by metrics sensitive to the minority class, such as

recollection

, precision and F1-score, which are crucial for the identification of defaulters.

To interpret the predictions of the models of

group

(LightGBM and XGBoost), the SHAP technique was applied (

SHapley Additive exPlanations

). This unified approach explains the output of any model

machine-based learning

, assigning to each feature a value that represents its contribution to the final prediction (Lundberg and Lee, 2017).

The application of the SHAP technique allowed the generation of graphs such as the SHAP Summary Plot and the SHAP Feature Importance Plot (Bar), which illustrate the importance and impact of each feature on the model. These graphs are fundamental to understanding how variables influence credit decisions, mitigating the “black box” nature of the models.

group

.

3. Results and Discussion

This section details the research findings, comparing the performance of logistic regression with the LightGBM and XGBoost “ensemble” models in predicting default, and exploring the interpretability of “boosting” models using the SHAP technique. The results obtained confirm the superior predictive capability of “ensemble” methods compared to logistic regression, albeit with marginal differences, and highlight the importance of specific variables in determining credit risk, according to the SHAP analysis. The discussion addresses the balance between predictive performance and interpretability, a central challenge in credit risk modeling.

Comparative Performance of Models

The performance evaluation of the models on the test set revealed that the LightGBM and XGBoost “ensemble” models showed slightly superior discriminatory capacity compared to Logistic Regression. The Logistic Regression model achieved an AUC-ROC value of 0.746 and a Gini Index of 0.491. In contrast, LightGBM achieved an AUC-ROC of 0.762 and a Gini Index of 0.524, while XGBoost demonstrated marginally better performance, with an AUC-ROC of 0.764 and a Gini Index of 0.528. These values indicate that both “boosting” models are more effective in distinguishing between performing and non-performing customers.

This advantage, although modest, of “ensemble” models can be attributed to their ability to identify more complex patterns and non-linear relationships present in the data, a characteristic that logistic regression, due to its linear nature, cannot capture with the same effectiveness. However, it is important to note that the practical gain observed was limited, as the performance metrics of the three models remained relatively close. Furthermore, the evaluation was based on a single data split for testing, which prevents the assertion of statistical significance of the differences observed between the models.

Analysis of Confusion Matrices

The analysis of the confusion matrices provided a detailed view of each model’s classification errors. Logistic Regression correctly identified 38,940 non-defaulting customers (true negatives) and 3,324 defaulting customers (true positives). However, it erroneously classified 1,641 defaulting customers as non-defaulting (false negatives) and 17,598 non-defaulting customers as defaulting (false positives).

Ensemble models demonstrated superior performance in identifying defaulting customers. LightGBM presented 40,094 true negatives and 3,364 true positives, with 1,601 false negatives and 16,444 false positives. XGBoost achieved the highest number of true negatives (40,358) and true positives (3,371), and the lowest number of false negatives (1,594) and false positives (16,180) among the three models. Although all models classified a considerable number of performing customers as defaulting (false positives), this behavior is expected in scenarios with imbalanced target variables and the use of a standard decision threshold of 0.5.

Classification Reports

When examining classification metrics such as precision, recall, and F1-score, similar behavior was observed among the models, especially for the predominant class of non-defaulters (class 0). For this class, all models showed very high precision, around 0.96, indicating that predictions of the event not occurring are frequently correct. The recall for class 0 ranged between 0.69 and 0.71, and the F1-score remained high, approximately between 0.80 and 0.82.

For the minority class of defaulters (class 1), the precision was considerably low, ranging between 0.16 and 0.17, which points to a significant number of false positives. However, the recall for class 1 reached more reasonable levels, between 0.67 and 0.68, indicating that the algorithms managed to identify a good portion of true positives, even at the cost of an increase in false positives, a common behavior in imbalanced databases. The resulting F1-score for class 1, between 0.26 and 0.28, directly reflects this imbalance between precision and recall.

The XGBoost model presented the best F1-score in class 0 (0.82) and a slightly higher recall in class 1 (0.68) compared to the others. LightGBM was very close, with an F1-score of 0.81 for class 0 and 0.27 for class 1, and a recall of 0.68 for class 1. The accuracy of both models was 71%. Logistic Regression, on the other hand, lagged slightly behind, with an F1-score of 0.80 for class 0 and 0.26 for class 1, and a recall of 0.67 for class 1, resulting in an accuracy of 69%.

The differences between the models are small, but logistic regression demonstrated a more contained performance, associated with its lower capacity to capture non-linear relationships present in the data. It is crucial to note that, in credit applications, the choice of metric must consider the consequences of errors. A false negative, which classifies a defaulter as a non-defaulter, can result in financial losses. Thus, the recall of class 1 is vital for identifying potential defaulters, while precision assesses the correctness of the default classifications.

Interpretability with SHAP

SHAP analysis was applied to the LightGBM and XGBoost models to understand the contribution of the variables in the predictions. Both models identified the variables `fonte_ext_3` and `fonte_ext_2` as having the greatest impact, indicating that these external “score” sources possess a strong predictive capacity in the analyzed context. This consistency in variable importance between the “ensemble” models reinforces the robustness of the findings.

Other important variables included feature engineering characteristics, such as `annual_fee_credit` and `asset_value_credit`, in addition to `days_employed`, `days_birth` (age), gender, education level, and car ownership. The direction of the impact, according to the SHAP plots, revealed that lower values of `ext_source_3` and `ext_source_2` tend to increase the default prediction. On the other hand, higher values of `days_employed` (indicating longer employment duration) tend to decrease the default prediction, while higher values of `days_birth` (younger age) tend to increase it.

SHAP analysis offers transparency to “ensemble” models, which are often seen as “black boxes”. By assigning a value to each feature that represents its contribution to the final prediction (Lundberg and Lee, 2017), the technique allows us to understand which factors contribute most to a client’s risk classification. This interpretability is essential for model validation and for making credit decisions, justifying the choices to regulators, managers, and clients.

Balance between Interpretability and Predictive Performance

The results of this study highlight an important balance between the interpretability and predictive performance of the models. Logistic Regression stands out for its ability to allow a direct reading of the estimated coefficients, which facilitates understanding the meaning and intensity of the relationship between the explanatory variables and the probability of default. This characteristic makes Logistic Regression particularly useful in contexts where the explanation of decisions is as relevant as predictive capability, as pointed out by Hosmer et al. (2013).

On the other hand, “ensemble” models, such as LightGBM and XGBoost, showed slightly superior performance in the discriminatory metrics, suggesting a greater ability to capture non-linear relationships and more complex interactions between variables. This behavior is consistent with the “credit scoring” literature, which indicates that more flexible techniques tend to achieve marginal performance gains on complex databases, although generally at the cost of lower interpretative transparency (Coelho et al., 2021).

In this context, the use of the SHAP technique represents an important resource for reducing the opacity of these models, allowing the estimation of the individual contribution of each variable to the predictions. In addition to providing global explanations about the importance of variables, the technique also contributes to the local interpretation of decisions, which is especially relevant in credit risk applications, where the justification of the result is an essential part of the analytical process. The literature already highlighted the relevance of this tension between performance and interpretability (Baesens et al., 2016; Lessmann et al., 2015), and this study reinforces the need for explanatory methods for more complex models.

Study Limitations

The evaluation of the models in this study was performed using a single split between the training and testing sets, which may limit the robustness of the comparison, although the optimization of the “ensemble” models was conducted with three-fold stratified cross-validation. Additionally, logistic regression did not undergo a hyperparameter optimization process, and the default decision threshold of 0.5 was used for all models, which may have influenced the classification metrics, especially in a context of class imbalance. Finally, the analysis was based on a single database, which restricts the generalization of the results to other credit contexts or different customer profiles.

In summary, the results confirm that the LightGBM and XGBoost “ensemble” models offer slightly superior predictive performance to Logistic Regression in predicting default, better capturing the complexities of the “Home Credit Default Risk” dataset. SHAP analysis was fundamental in interpreting the contribution of variables, highlighting the relevance of external “score” sources and financial characteristics. This study reinforces the importance of considering predictive capacity, interpretability, and the treatment of target variable imbalance in the evaluation of credit risk models, seeking the best compromise for the decision-making context.

4. Conclusion

This study investigated the effectiveness of logistic regression and “ensemble” techniques, such as LightGBM and XGBoost, in predicting default, and analyzed the interpretability of their predictions. It was found that the “ensemble” models demonstrated a slightly superior predictive capability compared to logistic regression, as indicated by metrics like AUC-ROC and Gini Index. XGBoost, in particular, achieved marginally better performance, with an AUC-ROC of 0.764. Although the differences in discriminatory metrics were modest, the “boosting” methods revealed a greater aptitude for capturing complex patterns in the data. SHAP analysis was fundamental in uncovering the contribution of variables in the “ensemble” models, highlighting the importance of `fonte_ext_3`, `fonte_ext_2`, `credito_por_anuidade`, and `dias_empregado` as crucial predictive factors. This interpretability is essential for model validation and for credit decision-making, offering transparency to models often considered “black boxes”.

The main contribution of this work lies in demonstrating the balance between predictive performance and interpretability in the context of credit risk, reinforcing the need for explanatory methods for more complex models. However, the study presented limitations, such as the evaluation of models from a single split between training and testing sets, the absence of hyperparameter optimization for logistic regression, and the use of a standard decision threshold of 0.5 for all models, which may have influenced the classification metrics in a class imbalance scenario. Furthermore, the analysis was based on a single database, restricting the generalization of the results. For future studies, it is suggested to evaluate the models with different strategies for handling imbalance, including “oversampling” techniques and adjustment of the decision threshold, as well as the application of statistical procedures to assess the significance of performance differences across various credit databases.

Bibliographic References

Araújo, E.A.; Carmona, C.U.M. 2009. Desenvolvimento de modelos credit scoring com abordagem de regressão logística para a gestão da inadimplência de uma instituição de microcrédito. Contabilidade Vista e Revista 18(3): 107-131.

Baesens, B.; Roesch, D.; Scheule, H. 2016. Credit Risk Analytics: Measurement techniques, applications, and examples in SAS. Wiley, Hoboken, NJ, EUA.

Camargos, M.A.; Camargos, M.C.S.; Coutinho, E.S. 2010. Fatores condicionantes de inadimplência em processos de concessão de crédito: uma análise com dados em painel. Revista de Administração Mackenzie 11(5): 103-131.

Chen, T.; Guestrin, C. 2016. XGBoost: a scalable tree boosting system. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 22, 2016, San Francisco, CA, EUA. Anais… p. 785-794.

Coelho, F.F.; Amorim, D.P.L.; Camargos, M.A. 2021. Analisando métodos de machine learning e avaliação do risco de crédito. Revista Gestão & Tecnologia 21(1): 89-116.

Duffie, D.; Singleton, K.J. 2003. Credit Risk: Pricing, measurement, and management. Princeton University Press, Princeton, NJ, EUA.

Friedman, J.H. 2001. Greedy function approximation: a gradient boosting machine. The Annals of Statistics 29(5): 1189-1232.

Gonçalves, E.B.; Gouvêa, M.A.; Mantovani, D.M.N. 2013. Análise de risco de crédito com o uso de regressão logística. Revista Contemporânea de Contabilidade 10(20): 139-160.

Hosmer, D.W.; Lemeshow, S.; Sturdivant, R.X. 2013. Applied Logistic Regression. 3ed. Wiley, Hoboken, NJ, EUA.

Lessmann, S.; Baesens, B.; Seow, H.-V.; Thomas, L.C. 2015. Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research. European Journal of Operational Research 247(1): 124-136.

Lundberg, S.M.; Lee, S.I. 2017. A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems (NIPS), 30, 2017, Long Beach, CA, EUA. Anais… p. 4765-4774.

Ponsam, J.G.; Gracia, S.V.J.B.; Geetha, G.; Karpagaselvi, S. 2021. Credit risk analysis using LightGBM and a comparative study of popular algorithms. In: International Conference on Computing and Communication Technologies (ICCCT), 4, 2021.

Zhou, Z.-H. 2012. Ensemble Methods: Foundations and algorithms. Chapman & Hall/CRC, Boca Raton, FL, EUA.

Çallı, B.A.; Coşkun, E. 2021. A longitudinal systematic review of credit risk assessment and credit default predictors. Sage Open 11: 21582440211061333.

Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform