September 30, 2026
Improvement of the Goiano IPM through automatic classification of items in fiscal documents
Improvement of the Goiano IPM Through Automatic Classification of Items in Fiscal Documents
Allan da Costa Andrade; Abdoulaye Aboubacari Mohamed
DOI: 10.22167/2675-6528-202602655
Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.
Summary
The automation of fiscal data classification is crucial for improving tax management. The study developed, implemented, and evaluated machine learning models to automate the processing and classification of fiscal data used in the calculation of the Municipal Participation Index (IPM) in the State of Goiás, focusing on the identification of horticultural products traded at CEASA-Goiás. The research was based on the analysis of secondary data from electronic invoices from 2024, characterized by large volume and complexity. A quantitative, descriptive, and explanatory approach was adopted. The methodological process included data extraction, processing, standardization, and organization, as well as the construction of a supervised dataset for training, validation, and testing. Natural Language Processing techniques and supervised learning algorithms were used to classify fiscal item descriptions. Performance was evaluated by metrics such as accuracy, precision, recall, and consistency. The results achieved an accuracy of over 95% in identifying horticultural products. Compared to the manual process, a significant reduction in processing time, a decrease in inconsistencies, and an improvement in the integrity of the data used in the IPM calculation were observed. The automated solution contributed to greater transparency, reliability, and accuracy of fiscal information, optimizing the use of human resources. It was concluded that the application of data science and artificial intelligence techniques is an effective alternative for improving state tax management.
Keywords: Active Learning; Data Science; Tax Management; Municipal Participation Index; NF-e.
1. Introduction
The application of data science techniques has been widely expanded in the context of public management. The adoption of data science and machine learning methods is fundamental to extracting value and knowledge from large volumes of data, an imperative that extends to the optimization of public policies and fiscal processes (Fávero and Belfiore, 2021).
The Municipal Participation Index (IPM) is an instrument that proportionally distributes resources collected by the state among municipalities, considering economic and fiscal factors. In the State of Goiás, the methodology adopts total transparency, using granular data from all electronic fiscal documents. This approach generates a volume of approximately six billion invoice items, which, while ensuring transparency, imposes technical processing barriers.
The task of automatic classification of fiscal item descriptions can be understood as a classic text classification problem in Natural Language Processing. In these cases, unstructured data, often short and noisy, must be categorized automatically. This type of problem presents specific challenges related to high linguistic variability, semantic ambiguity, and low textual standardization (Manning et al., 2008).
Additionally, real fiscal bases, as those used in this study, tend to present class imbalance, in which certain categories have many more examples than others. This phenomenon is widely discussed in the machine learning literature, as it can lead models to present bias in favor of the majority classes, reducing generalization capacity (Fávero and Belfiore, 2021).
In fiscal contexts, the processing and correct categorization of electronic documents constitute relevant challenges, especially given textual variability and the complexity of tax rules. Laruccia (2022) demonstrated the feasibility of using Natural Language Processing models and machine learning for the automatic classification of electronic invoices. Interian et al. (2024), when analyzing Brazilian fiscal data for anomaly detection, highlight the importance of robust data mining techniques to ensure the integrity and compliance of information.
Despite the advances presented in the literature, a gap is observed in the integrated application of these approaches in real Brazilian fiscal contexts, especially in scenarios involving massive data, short and noisy descriptions, as well as significant class imbalance. There is a scarcity of studies that simultaneously explore these challenges in the context of fiscal item classification, also considering sectoral particularities, such as the fruit and vegetable segment and its influence on IPM calculation. Thus, the present work is inserted in this context, focusing on the classification of fruit and vegetable items from CEASA Goiás, aiming to avoid duplication in the IPM calculation base and contribute to greater accuracy and fairness in resource distribution. The objective of this work is to develop, implement, and evaluate machine learning models applied to the automation of the processing and classification of fiscal data used in the calculation of the Municipal Participation Index (IPM) in the State of Goiás, focusing on the identification of fruit and vegetable items traded at CEASA-Goiás.
2. Material and Methods
The methodological approach adopted in this study was based on consolidated data science and machine learning techniques. The research was characterized as descriptive and explanatory, seeking to determine the impact of automated classification on the accuracy of the Municipal Participation Index (IPM). The main design consisted of the implementation and evaluation of algorithms of
machine learning
, aligning with the research lines of automation and
strategic analytics
(Fávero and Belfiore, 2021; Manning et al., 2008).
The research used secondary data, originating from internal databases of the Goiás State Tax Administration. The information sources were the granular databases of electronic invoices (NF-e) from CEASA/CEARANA-Goiás. The temporal scope of the study focused on the year 2024, analyzing the massive volume of fiscal items for which the manual classification of fruits and vegetables demanded significant effort.
The initial methodological process included stages of data extraction, processing, standardization, and organization. Subsequently, a supervised base was built for training, validation, and testing of the classification models. The preliminary analysis involved a complete file with over 344,000 records, which would be classified binarily as produce or not produce.
The automatic classification of textual descriptions in fiscal documents falls within the context of classic short text classification problems. Such descriptions are characterized by high lexical variability, low semantic context, and the presence of noise, requiring specific preprocessing and modeling techniques (Manning et al., 2008; Salton and Buckley, 1988). Real fiscal databases often present class imbalance, which can compromise the predictive performance of models (Fávero and Belfiore, 2021).
The data preprocessing stage consisted of applying textual cleaning and normalization techniques (Manning et al., 2008). This included character standardization, accent removal, conversion of numbers to “NUM”, and elimination of duplicate spaces. Were used
stop words
such as “of”, “from”, “to”, “for”, “with”, “in”, “on”, “kg”, “un”, “und”, “cx”, “sc”, “gr”, “ml”, “gtin”, and “cm”, and a regular expression to remove isolated words.
For the representation of textual data, the choice was made for feature extraction based on n-grams, including unigrams, bigrams, and trigrams. This approach was chosen to capture semantic context in short descriptions, mitigating the ambiguity of isolated terms (Manning et al., 2008; Fávero and Belfiore, 2021). The TF-IDF technique was employed to transform textual data into numerical representations, reflecting the relative importance of terms in the corpus (Salton and Buckley, 1988).
The choice of classification algorithms considered their computational efficiency and suitability for textual classification problems in high-dimensional spaces. The Naive Bayes algorithm and the Passive-Aggressive Classifier, an online learning algorithm based on linear models, were used. Both are recognized in the literature for their good performance and generalization capability in scenarios with large data volumes and high textual variability (Manning et al., 2008; Ng and Jordan, 2002).
For the incremental and supervised construction of the training and validation bases, the technique of
engaged learning
(Settles, 2012; Marcacini and Rezende, 2012). This strategy aimed to prioritize the selection of more informative examples for the learning process, reducing supervision effort. Initially, a minimal supervised base of 700 records was built. The process followed an iterative flow of training, prediction, identification of the most uncertain cases, manual validation, and database update.
The dataset was divided into training (80%) and validation (20%) subsets, using stratified sampling to preserve the class proportion. This approach is relevant in scenarios with class imbalance, contributing to a robust evaluation of model performance (He and Garcia, 2009). Model evaluation was performed using quantitative classification metrics, including accuracy, precision, recall (
memory
), F1-score and confusion matrix analysis.
3. Results and Discussion
The research had as its main objective to develop, implement, and evaluate machine learning models to automate the classification of fiscal data, focusing on the identification of fruit and vegetable items traded at CEASA-Goiás, aiming to optimize the calculation of the Municipal Participation Index (IPM) in the State of Goiás. The results obtained demonstrate the viability and effectiveness of applying these techniques, culminating in an automated solution with high accuracy and significant improvements in tax management processes.
Initially, the information gathering involved the analysis of a complete file with over 344,000 electronic invoice records from 2024. The central task consisted of the binary classification of these items, determining whether or not they belonged to the fruit and vegetable category. This massive volume and the complexity of the data underscored the need for an automated approach to handle the technical processing and avoid barriers in the IPM calculation.
In the exploratory analysis of textual data, visualization tools such as word clouds were employed. The initial word cloud, based on 344,863 records, revealed the predominance of terms like “tomato” and “potato”, following Zipf’s Law. This frequency distribution, where a few terms concentrate most of the probability mass, indicated a bias challenge for machine learning, as the model might learn transaction frequency instead of discriminative semantic features.
The transition to a word cloud with 31,746 distinct records, obtained by removing duplicates, significantly altered the distribution of terms. This strategy reduced the influence of products with higher sales volume, highlighting a broader lexical diversity in the base and allowing less frequent items to become more visible. This approach was crucial for mitigating the bias caused by record repetition and identifying noise unrelated to the fruit and vegetable domain.
The identified scientific challenge was characterized as a problem of classifying short texts in noisy environments, where fiscal descriptions suffer from data scarcity and lack of syntactic context (Aggarwal & Zhai, 2012). The presence of non-standard prefixes and abbreviations required the vectorizer and the model to be robust to high lexical variance. The use of the TF-IDF technique was justified to penalize noisy terms with low discriminability and value specific technical terms, mitigating the impact of low semantic context (Salton & Buckley, 1988; Rossi, 2015).
The statistical analysis of the corpus revealed that the average number of words per record was 2.62, with 78% of the descriptions containing up to three terms. This behavior is characteristic of fiscal databases with short and unstructured descriptions, which hinders the extraction of meaning. To capture the semantic context in such concise descriptions, the choice was made for feature extraction based on n-grams (unigrams, bigrams, and trigrams), as recommended by Fávero and Belfiore (2021) and Manning et al. (2008).
The representation of textual data in numerical format, essential for machine learning algorithms, was performed using the TF-IDF technique. This configuration allowed capturing not only isolated words but also frequent composite patterns in fiscal descriptions, increasing the model’s discriminatory capacity. The choice of classification algorithms considered their computational efficiency and suitability for textual classification problems in high-dimensional spaces.
Dois algoritmos foram selecionados para o treinamento inicial: o Naive Bayes e o Passive-Aggressive Classifier. O Naive Bayes, baseado no Teorema de Bayes, é conhecido por sua simplicidade, baixo custo computacional e rápida convergência, sendo frequentemente utilizado como modelo de referência (Ng & Jordan, 2002). O Passive-Aggressive Classifier, por sua vez, é um algoritmo de aprendizado online baseado em modelos lineares, projetado para lidar eficientemente com grandes volumes de dados e descrições textuais de alta dimensionalidade (Manning et al., 2008).
For the training and validation of the models, the dataset was divided into training (80%) and validation (20%) subsets, using stratified sampling to preserve the proportion between classes. This approach is crucial in scenarios with class imbalance, contributing to a more robust and representative evaluation of model performance (He & Garcia, 2009). Accuracy, precision, recall, and F1-score metrics were used to measure performance.
In the first training, the Multinomial Naive Bayes model achieved an accuracy of 79.29%. Its classification report indicated a precision of 0.76 and recall of 0.95 for class 0, and a precision of 0.89 and recall of 0.57 for class 1. The confusion matrix revealed 78 correct predictions and 4 errors for class 0, and 25 errors and 33 correct predictions for class 1, showing greater difficulty in identifying the minority class.
In contrast, the Passive-Aggressive Classifier demonstrated superior performance in initial training, reaching an accuracy of 82.86%. For class 0, it achieved a precision of 0.80 and recall of 0.94, while for class 1, it reached a precision of 0.89 and recall of 0.67. The confusion matrix showed 77 correct predictions and 5 errors for class 0, and 19 errors and 39 correct predictions for class 1, indicating a greater capacity to correctly identify the minority class compared to Naive Bayes.
This superiority of the Passive-Aggressive Classifier can be attributed to its structural differences, especially its capacity to adapt to scenarios with high textual variability and dimensionality, as discussed by Manning, Raghavan, and Schütze (2008). However, it is important to note that these initial comparisons were based on point metrics from a single partitioning, without cross-validation or formal statistical tests, which requires caution in interpreting the differences.
Given the absence of a previously labeled and validated dataset, and the high volume of records, manual labeling of the entire dataset was unfeasible. Therefore, the “active learning” technique was adopted to prioritize the selection of the most informative examples for the learning process, reducing the human effort required in the supervision stage, as advocated by Settles (2012) and Marcacini and Rezende (2012).
The active learning strategy was based on the combination of two uncertainty criteria: the probability estimated by the Multinomial Naive Bayes model and the decision margin provided by the Passive-Aggressive model. A combined uncertainty score was calculated to prioritize samples with lower classification reliability. This process generated spreadsheets with product descriptions associated with the highest uncertainty values, indicating instances where the model had low decision confidence.
It was observed that descriptions related to mechanical components, for example, were correctly classified as non-horticultural products (Class 0), but with high uncertainty values. This suggested that the model made such classifications based on unrepresentative patterns, approaching the decision threshold. For Class 1 (horticultural products), products with composite or atypical descriptions, such as less common varieties, also showed lower prediction confidence.
The careful selection of these examples, located in the boundary region between classes, for manual review and subsequent incorporation into the training base, represented an efficient data enrichment strategy. By including instances from the uncertainty region, the classifier began to recognize previously ignored patterns, refining decision boundaries and reducing the error rate in subsequent iterations, according to Settles (2012) principles.
After adding 112 items to the training base through “active learning”, a new training was performed. The results indicated that the accuracy of Multinomial Naive Bayes was 78.53%, and that of Passive-Aggressive was 80.37%. This improvement was not significant, and in some cases, there was a slight decrease in accuracy, suggesting that the simple addition of data did not guarantee a linear improvement of the model.
In a subsequent step, 189 new items pointed out by “active learning” were included, using only the error probability of Multinomial Naive Bayes. With this new training, the accuracy of Multinomial Naive Bayes rose to 86.00%, and that of Passive-Aggressive to 87.00%. Although there was improvement, the performance evolution over the training cycles did not show to be linear, with more expressive initial gains and subsequent stabilization, indicating that the quality of the added data is more impactful than the quantity.
For the final evaluation of the models’ performance in a scenario close to the production environment, an independent validation base was built, composed of more than 30,000 records duly verified by the IPM management team. This base was used to ensure more representative metrics, excluding records selected for training to avoid overlap, as per the recommendations of Fávero and Belfiore (2021).
In the validation set, the Multinomial Naive Bayes model presented an accuracy of 88.10% for Class 1, while the Passive-Aggressive Classifier achieved 77.42% accuracy for the same class. The difference between training and validation performance was attributed to the greater diversity of descriptions in the validation set and the imbalance between classes, which reduced the model’s sensitivity to Class 1 in unseen data (Fávero & Belfiore, 2021).
The behavior of the Passive-Aggressive Classifier proved to be more stable between training and validation compared to Naive Bayes, which is expected for linear models with online learning on high-dimensional textual data (Manning, Raghavan & Schütze, 2008). The drop in Naive Bayes performance in validation reinforced the limitation of the conditional independence hypothesis between attributes, especially when the vocabulary of the validation set diverges from the training set.
The analysis of data preprocessing revealed that many values that did not distinguish the product were entering the classifier, generating confusion. To optimize the results, more rigorous preprocessing rules were implemented, including the removal of accents, the conversion of all numbers to the word “NUM”, the removal of duplicate spaces, and the inclusion of several keywords such as “kg”, “un”, “cx”, “sc”, “gr”, “ml”, “gtin”, “cm” in the stop words list, as well as a regex to remove standalone words.
Even with these treatments, the 95% classification barrier was not initially altered. A more in-depth analysis found that some items in the training set had only one example, which confused the classifier. To resolve this, the set was quadrupled, ensuring sufficient examples for the classifier in all training and testing divisions, which resulted in significant performance improvements.
After refining the preprocessing and expanding the training base, Multinomial Naive Bayes achieved an accuracy of 98.21% in training and 96.05% in validation, with 28,814 correct predictions and 1,186 errors in the validation base. This result demonstrated a substantial advance in the model’s ability to classify fiscal items for fruits and vegetables.
The Passive-Aggressive Classifier, with the same preprocessing and database expansion adjustments, achieved even higher accuracy, 98.81% in training and 96.54% in validation. In validation, 28,963 correct predictions and 1,037 errors were recorded. This superior performance of Passive-Aggressive reinforces its suitability for scenarios with high textual variability and large data volume, surpassing Naive Bayes in terms of final accuracy.
The final accuracy of 96.54% obtained by the Passive-Aggressive Classifier on the validation set is in line with the results reported in the literature on Brazilian fiscal document classification. Laruccia (2022), for example, achieved 96.7% accuracy with an ensemble classifier for binary classification of electronic invoices. This validates the effectiveness of the approach proposed and implemented in this study to improve state tax management.
In summary, the research demonstrated that the combination of Natural Language Processing techniques, supervised learning algorithms, and active learning strategies is an effective alternative for automating the classification of fiscal data. The accuracy exceeding 95% in identifying fruit and vegetable products, the reduction in processing time, and the improvement in data integrity confirm the contribution of the automated solution to greater transparency, reliability, and accuracy of fiscal information in IPM calculation, optimizing the use of human resources.
4. Conclusion
The present study sought to develop, implement, and evaluate machine learning models to automate the processing and classification of fiscal data, focusing on the identification of fruits and vegetables marketed at CEASA-Goiás, aiming to optimize the calculation of the Municipal Participation Index (IPM) in the State of Goiás. The feasibility and effectiveness of applying these techniques were verified, culminating in an automated solution that achieved over 95% accuracy in identifying fruits and vegetables. A significant reduction in processing time was observed compared to the manual method, along with a decrease in inconsistencies and an improvement in the integrity of the data used for IPM calculation. The combination of Natural Language Processing techniques, supervised learning algorithms, and active learning strategies proved to be an effective alternative for automating the classification of fiscal data. This automated solution contributed to greater transparency, reliability, and accuracy of fiscal information, optimizing the use of human resources and improving state tax management.
However, the study presented important limitations, such as the dependence on the quality of the original descriptions of fiscal documents and the restriction of the analysis to a single institutional context. Additionally, the comparisons between the models were based on punctual metrics from a single train and test partitioning, without the application of formal statistical significance tests, which requires caution in interpreting performance differences. For future studies, it is suggested to explore the application of cross-validation (k-fold) and formal statistical tests for a more robust comparison of classifiers. It is also recommended to use neural network-based models and class balancing techniques to further enhance generalization capability and sensitivity to minority classes in imbalanced datasets.
Bibliographic References
FÁVERO, L.P.; BELFIORE, P. 2021. Data Science para Negócios: Conceitos, técnicas e aplicações. 1ed. Elsevier, Rio de Janeiro, RJ, Brasil.
INTERIAN, R.; CARPANESE, I.; MELLO, B.; RIBEIRO, C. C. Red flag algorithms for Brazilian electronic invoices: outlier detection and price risk classification. International Transactions in Operational Research. Disponível em https://repositorio.usp.br/item/003154046 acesso em 12/10/2025: 1533-1553
LARUCCIA, J. B. Classificação automática de notas fiscais eletrônicas para utilização de créditos tributários de PIS/PASEP e COFINS. 2022. Trabalho de Conclusão de Curso (MBA) – Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos, 2022. Acessado em https://bdta.abcd.usp.br/item/003125927 dia 01/10/2025
MANNING, C. D.; RAGHAVAN, P.; SCHÜTZE, H. Introduction to Information Retrieval. Cambridge: Cambridge University Press, 2008.
SALTON, G.; BUCKLEY, C. 1988. Term-weighting approaches in automatic text retrieval. Information Processing
Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq
To learn more about the course, click here and access the MBX Academy platform