February 09, 2026
Application of artificial intelligence in the classification of ISO 9001:2015 audits
Oziris Teixeira de Menezes; Charles Gomes da Silva
Summary prepared by the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege focused on synthesis and writing.
The objective of this study is to evaluate the performance of the combination of Term Frequency-Inverse Document Frequency (TF-IDF) vectorization with the Random Forest classifier in the automatic classification of textual responses from ISO 9001:2015 standard audits. The investigation seeks to determine the capability of this machine learning approach to accurately and consistently assign the classes “Conform”, “Non-Conform”, and “Inconclusive” to textual evidence, verifying its generalization on unseen data and its viability as a support tool. The work proposes an automated alternative to manual analysis processes, which are costly, time-consuming, and susceptible to auditor subjectivity. The research fills a gap by adapting and validating Artificial Intelligence (AI) and Natural Language Processing (NLP) techniques for the normative domain of quality management, historically dependent on human judgment.
Quality management, governed by ISO 9001:2015, faces the challenge of the growing volume of textual information generated in audits. With approximately 1.26 million valid certificates globally, according to the ISO Survey 2022, millions of audits are conducted every triennial cycle. These processes, crucial for competitiveness and market access, involve significant costs, with initial certifications estimated between US$4,000 and US$6,000, especially for small and medium-sized enterprises. Manual analysis of evidence consumes time, resources, and introduces an interpretive variability that can compromise the consistency of results. The dependence on auditor experience and bias represents a bottleneck for the scalability and standardization of the certification process.
In this context, AI emerges as a solution to optimize audit governance. Although deep language models like BERT (Devlin et al., 2019) represent the state of the art in NLP, their implementation in regulated environments is limited by high computational costs, the need for vast datasets, and lower interpretability, a critical factor in domains where decision traceability is mandatory. In contrast, approaches like TF-IDF vectorization (Manning, Raghavan, and Schütze, 2008) and the Random Forest classifier (Breiman, 2001) offer a balance between performance, robustness, and explainability. TF-IDF provides a numerical representation of term relevance, while Random Forest handles high-dimensional data well, resists overfitting, and allows for the extraction of variable importance, facilitating communication of results (Bishop, 2006).
The relevance of this investigation lies in the empirical verification that consolidated methods with low computational cost can offer competitive results when evaluated with statistical rigor. The study addresses a gap in the literature, which lacks a body of knowledge on the automation of compliance screening in ISO 9001:2015 audits. By quantifying the model’s performance with standardized metrics, cross-validation, and confidence intervals, the work demonstrates the technical viability of the solution and offers a replicable protocol. The main practical contribution is the demonstration of a tool capable of supporting organizations with faster, more consistent, and less subjective analyses, strengthening the reliability of the certification process and guiding corrective actions.
The central research question is whether the combination of TF-IDF and Random Forest can adequately generalize in the classification of audit responses, maintaining a balance between accuracy, interpretability, and implementation feasibility. The underlying hypothesis is that, even in the face of advances in deep neural networks, this approach proves sufficient and effective for the problem. Rigorous analysis of overfitting, result variability, and the importance of textual features aims to provide a quantitative and qualitative answer to this question, contributing to evidence-based decision-making in quality auditing.
The study was conducted as an applied and quantitative research, with an experimental design to evaluate the model in simulated audit scenarios. Data collection, carried out over two months, began with a checklist of 14 questions aligned with the standard, modeled after a certification body’s roadmap. This instrument was submitted to 12 professionals qualified in ISO 9001:2015 audits, instructed to generate textual responses for three scenarios: “Conform”, “Non-Conform”, and “Inconclusive”. After an initial round of low adherence, the checklist was reformulated, resulting in a final consolidated and balanced database of 504 textual responses.
The data preparation followed an NLP preprocessing pipeline in Python (version 3.10) with the Pandas, Scikit-learn, and NLTK libraries. The steps included text normalization (conversion to lowercase, removal of punctuation and stopwords), followed by tokenization. Numerical representation was performed with the TF-IDF model, which weighs the importance of a term in a document based on its local frequency, inversely proportional to its frequency across the entire corpus, highlighting distinctive terms for each class (Manning, Raghavan, and Schütze, 2008).
For training and evaluation, the Random Forest algorithm was selected, a method that builds multiple decision trees and defines the final class by majority vote (Breiman, 2001). The choice is justified by its robustness with the high dimensionality of textual data, lower propensity for overfitting, and ability to provide an importance measure for each variable, fundamental for interpretability. For an unbiased evaluation of generalization, the dataset was divided in a stratified manner into training (80%) and testing (20%), preserving the original class proportion.
The quantitative results demonstrate robust performance. 5-fold cross-validation revealed an average accuracy of 0.887, with a standard deviation of 0.023 and a 95% confidence interval between 0.866 and 0.907. Macro precision reached an average of 0.891 (95% CI: 0.870–0.912), and macro recall obtained an average of 0.887 (95% CI: 0.866–0.907). The macro F1-score, the harmonic mean between precision and recall, registered an average value of 0.885 (95% CI: 0.865–0.906). The low variability, indicated by the reduced standard deviations (around 0.024), and the narrow confidence intervals confirm the model’s stability.
The overfitting analysis, comparing performance on the training and testing sets, revealed an expected trend. The model achieved 100% on all metrics in the training set, while on the testing set, performance stabilized at 91.1%. This nine-percentage-point difference characterizes a moderate generalization error. The learning curves corroborated this finding: the training curve remained at 1.00, while the cross-validation curve converged around 0.91. Although the gap indicates that the model memorized part of the training data, the high performance on the test set demonstrates its excellent generalization capability for unseen data.
The analysis of the confusion matrix, generated on the test set, reinforced the model’s ability to discriminate between the three classes. The values on the main diagonal (correct classifications) were high, while the off-diagonal values (errors) were low and evenly distributed, without a systematic bias of confusion between a specific pair of classes. The ROC curves for each class also showed exemplary behavior, with an average macro area under the curve (AUC) of approximately 0.91. This value indicates that, 91% of the time, the model assigns a higher score to a random positive instance than to a negative one, confirming its high discriminative ability.
Interpretability was investigated by analyzing the feature importance of the Random Forest. Terms associated with negation and uncertainty were the most influential. The word ‘não’ (no) was the most important variable, contributing 17.6% to the predictive power. Next, ‘mas’ (but) (14.1%) and the expression ‘resposta não’ (response no) (5.8%) appeared. Other relevant terms included conjunctions such as ‘se’ (if) (5.5%), ‘mas não’ (but not) (5.2%), and ‘ou’ (or) (4.8%). This finding is semantically coherent with the audit domain; negations or caveats are strong indicators of non-compliance or inconclusiveness.
To deepen the understanding, agnostic interpretability techniques such as SHAP (SHapley Additive Explanations), Partial Dependence Plots (PDP), and Individual Conditional Expectation (ICE) were applied. SHAP plots confirmed the global importance of terms like ‘no’ and ‘but’, showing how their presence consistently directed the prediction towards the “Non-Conforming” or “Inconclusive” classes. PDPs illustrated the average marginal effect of these terms, demonstrating that an increase in the frequency of ‘no’ drastically reduces the probability of a “Conforming” classification. ICE plots revealed the heterogeneity of this effect in individual instances, showing that context can mitigate or intensify the impact of a single word.
The hyperparameter optimization process, via GridSearchCV with 5-fold cross-validation, identified the optimal configuration for the Random Forest. The combination with the best average performance (99.95% accuracy) was: 400 trees (nestimators=400), unlimited maximum depth (maxdepth=None), minimum of 5 samples for splitting a node (minsamplessplit=5), and minimum of 1 sample per leaf node (minsamplesleaf=1). The analysis showed that the model is robust to variations in hyperparameters, but systematic optimization ensured that the final configuration balances complexity and generalization.
In summary, the results demonstrate that the proposed approach is effective for classifying audit responses. The model achieves high levels of accuracy, precision, and recall stably, as validated by cross-validation and confidence intervals. The generalization capability, despite moderate overfitting, is sufficient for practical applications. Interpretability allows auditors and managers to understand the factors behind automatic classifications, generating trust in the system and providing insights into linguistic patterns associated with compliance, aligning with principles of explainable AI (Fawcett and Provost, 2013; Géron, 2019).
This study evaluated the application of the TF-IDF model with Random Forest for the automatic classification of responses in ISO 9001:2015 audits. The results were consistent, with the model presenting an average accuracy of 0.887 and a macro F1-score of 0.885, both with narrow 95% confidence intervals, indicating stability. The overfitting analysis revealed a moderate generalization error of nine percentage points, which does not compromise the model’s predictive capability on unseen data. The classifier’s interpretability, evidenced by the importance of terms such as ‘no’ and ‘but’, reinforces the model’s validity and its suitability for the audit context; justifying decisions is fundamental. The methodological protocol guarantees the reproducibility and reliability of the findings. It is concluded that the objective was achieved.
References:
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. New York: Springer.
Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32.
Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of NAACL-HLT 2019, p. 4171–4186. Minneapolis, MN: ACL.
Fávero, L. P.; Belfiore, P. (2017). Manual de análise de dados: estatística e modelagem multivariada com Excel, SPSS e Stata. 1. ed. Rio de Janeiro: Elsevier.
Fawcett, T.; Provost, F. (2013). Data Science and its Relationship to Big Data and Data-Driven Decision Making. Big Data, 1(1), 51–59.
Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. 2. ed. Sebastopol: O’Reilly Media.
Joachims, T. (1998). Text categorization with Support Vector Machines: Learning with many relevant features. In: ECML-98 — European Conference on Machine Learning, LNCS 1398, p. 137–142. Berlin/Heidelberg: Springer.
Jurafsky, D.; Martin, J. H. (2021). Speech and Language Processing. 3. ed. Draft. Stanford: Prentice Hall.
LawInnovation. (2024). AI and Machine Learning make internal auditing more assertive. Available at: https://lawinnovation. com. br. Accessed on: 1 Apr. 2025.
Manning, C. D.; Raghavan, P.; Schütze, H. (2008). Introduction to Information Retrieval. Cambridge: Cambridge University Press.
McCallum, A.; Nigam, K. (1998). A comparison of event models for Naive Bayes text classification. In: AAAI-98 Workshop on Learning for Text Categorization, p. 41–48. Madison, WI: AAAI Press.
Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781.
Müller, A. C.; Guido, S. (2016). Introduction to Machine Learning with Python: A Guide for Data Scientists. Sebastopol: O’Reilly Media.
Recima21. (2023). The impact of artificial intelligence on the automation of audits. Available at: https://recima21. com. br. Accessed on: 1 Apr. 2025.
Executive summary from the Final Course Work of the Specialization in Data Science and Analytics of the MBA USP/Esalq
Learn more about the course; click here:
