October 26, 2026
Application of the CRISP-DM methodology in mapping the data flow in the finance area
Alexandre Barbosa de Souza; Elaine Barbosa de Figueiredo
DOI: 10.22167/2675-6528-202600480
Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.
Summary
The CRISP-DM methodology was applied to map the data flow in the financial planning and budgeting area of an industrial company. The study addressed the challenge of manual processes and intensive use of spreadsheets in consolidating financial data, which compromised productivity, accuracy, governance, and strategies. The objective was to develop a data solution capable of automating the information flow, reducing human interference, improving corporate governance and data quality, and streamlining business strategies. The methodology followed the six phases of the CRISP-DM model: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. The technical architecture involved extracting data from the management system to a database, using Python to process and enrich records, and focused on data engineering, manipulation, and integration, as well as producing visualizations in Power BI, without machine learning algorithm modeling. Preliminary results indicated a significant improvement in operational efficiency, with a reduction in total processing time from three hours and forty minutes to an interval between ten and seventeen minutes. Greater agility was observed in identifying missing classifications and a reduction in classification errors from 14% to less than 0.05%. Interactive dashboards were implemented to facilitate visualization and monitoring. The application of the CRISP-DM method enabled the agile and secure development of the solution, providing a comprehensive business overview and indicators for strategic decisions and continuous monitoring.
Keywords: Automation; Operational Efficiency; Data Engineering; Data Governance; Financial Planning.
1. Introduction
The contemporary business scenario is marked by a growing dependence on data for strategic decision-making, especially in critical areas such as financial and budgetary planning. The ability to efficiently collect, process, and analyze information has become a competitive differentiator, driving the search for solutions that optimize data flow and ensure information integrity. Companies in the industrial sector, in particular, face the challenge of managing large volumes of data from various sources, requiring systematic approaches to transform this data into actionable intelligence.
In many organizations, financial data management still relies on manual processes and intensive use of electronic spreadsheets for information consolidation. This dependence on traditional methods, as observed in the context of a growing medium-sized industrial company, results in significant negative impacts on productivity, data accuracy, scalability, and corporate governance. Manual data manipulation and the creation of links between multiple files and spreadsheets increase the risk of errors, compromise reliability, and hinder auditing, factors crucial for the management and monitoring of business strategy, as pointed out by Carvalho et al. (2020) when discussing the importance of compliance with institutional laws and regulations.
The absence of a structured methodology for data projects exacerbates this problem. Research indicates that most data projects do not employ a methodology, leading to failure and wasted resources, as highlighted by Shimaoka et al. (2024). Furthermore, understanding the need for robust data engineering is fundamental, as explained by Reis and Housley (2024). Data engineering encompasses the preparation of raw data from various sources through a process known as ETL (Extract, Transform and Load), which includes treatment and transformation for consumption in analysis flows, also called a data pipeline, essential for decision-making (Inmo, 2005; Escovedo and Kalinowski, 2023).
In this context, the CRISP-DM (Cross Industry Standard Process for Data Mining) methodology emerges as a consolidated alternative. It is recognized for its robustness, flexibility, and effectiveness in various corporate environments, including the financial sector (Shearer, 2000; Plotnikova et al., 2022). Defined as a structured systematization that guides from general phases to specific process instances (Chapman et al., 2000), CRISP-DM offers a framework for the agile and secure development of data solutions. Its application is adaptable to different segments, such as equipment engineering or infrastructure, and can be integrated with quality control tools, such as Six Sigma, for continuous improvement and defect reduction (Huber et al., 2018; Araújo and Silva, 2019).
Data intelligence, or Business Intelligence, as described by Oliveira and Rondina (2023), is an essential component for transforming large volumes of data into graphical visualizations that support strategic decision-making. In the financial scope, this translates into a clear view of the impacts on cash flow (financial view) and on the business’s performance and profitability (budgetary view), as elucidated by Padoveze (2012). The ability to categorize operational expenses (Opex) and capital investments (Capex) (Assaf Neto, 2021; Gitman, 2017) is crucial for detailed and strategic financial analysis.
The gap identified in financial data management, characterized by the inefficiency of manual processes and the absence of a standardized methodology, justifies the search for innovative solutions. The application of the CRISP-DM methodology represents an opportunity to mitigate data manipulation risks, optimize productivity, and improve governance. Given this scenario, the present work aimed to apply the CRISP-DM methodology to map the data flow in the finance area, seeking to systematize project phases, develop a data solution to increase productivity, preserve data, automate the ETL process, enhance reliability in engineering steps, reduce human interference, improve auditing and governance mechanisms, unify primary databases, and elevate data classification quality, enabling more assertive analyses to guide business strategies.
2. Material and Methods
This study was characterized as an applied research, descriptive in nature, with a methodological approach based on the application of the Cross-Industry Standard Process for Data Mining (CRISP-DM). The central objective was to map the data flow of the financial and budgetary planning area of an industrial company, aiming to systematize the project phases and develop a data solution to optimize processes and improve information management.
The research was conducted in a medium-sized industrial company, headquartered in Brazil, which has shown consistent growth in recent years. To preserve confidentiality and privacy, the organization’s name and specific operational data were kept anonymous. The unit of analysis focused on the processes and data flows of the finance and budgeting area.
The CRISP-DM methodology was selected for its robustness and adaptability, being widely recognized in the market for data projects (Shearer, 2000; Plotnikova et al., 2022). The process was structured in six interconnected phases, as proposed by Chapman et al. (2000) and the Data Science Process Alliance (2024): Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment.
In the Business Understanding phase, meetings were held with finance managers to determine the project objectives, which included automating the data classification process and utilizing existing company software. The current context was evaluated, identifying risks associated with manual data control in spreadsheets and multiple information sources, which could impact productivity and accuracy.
The data mining objectives were defined to build a database enriched with variables such as income statements, profit centers, division, product NCM, responsible parties, and record type classification (Opex or Capex), in addition to a governance indicator. A work plan was developed, spanning seven weeks, with activities including mapping, solution design, unit and integrated testing, and requesting access to systems and tools.
The Data Understanding phase involved the analysis of the data dictionary together with the business area. The necessary data were obtained from multiple sources, including the management system (ERP) and Excel spreadsheets received by email or in shared folders. A detailed description of data from various areas was carried out, such as payments, billing, purchasing, sales, controlling, human resources, and accounting.
During data exploration, a relationship was observed between metallic inputs and treatment services. In the quality assessment, it was identified that approximately 11% of the purchase order variable data were excluded, about 99.8% of the profit center data were missing, and 99.9% of the sales orders also showed absence. Approximately 40% of the general ledger account data were incorrectly assigned, according to log records.
The Data Preparation phase, considered the most time-intensive (Jung, 2024), focused on the selection, cleaning, and construction of new attributes. The data were selected from the ERP, covering 19 variables, such as paying/receiving company, vendor and customer codes, accounting account, purchase order, sales order, clearing document, source accounting document, division, profit center, document type, block code, document dates, posting and due date, payment term, item code, currency code, and debit/credit code (Iudicibus, 2019).
The data were cleaned, correcting missing or incorrect information, and new attributes were constructed, such as the company name linked to the customer/supplier code and the description of the financial record type (Opex or Capex). The integration of other sources was suppressed, as the management system became the primary source, and the final dataset did not require reformatting, being compliant with the specified characteristics.
For the technical architecture, the data were stored in a relational SQL database. Application programming interfaces (APIs) were developed to extract data directly from the ERP (Saudate, 2017). The transformation process was carried out in Python, using the Pandas, Numpy, Polars, Requests, and OpenPyXL libraries, to process records, analyze null and duplicate data, and enrich the output table with new variables. The result was an Excel file, subsequently integrated into Power BI for interactive visualizations.
In the Modeling phase, there was no need to apply complex machine learning algorithms. The initial model was created based on the scope and guidelines of the previous CRISP-DM phases. During testing, errors related to the identification of signal values, character size in specific fields, and descriptions linked to the financial record type were identified and corrected. The model was built by the development team and validated with a historical database of up to three years.
The Evaluation phase focused on the quality of the output data, verifying the absence of losses, record duplication, and consistency with the database schema. A reduction of approximately 92% in processing time for product delivery was observed. The solution logs were implemented to allow for auditing and traceability of financial classification. The process was reviewed at each phase, and no failures requiring rework were identified, allowing the transition to the deployment phase.
Finally, the Deployment phase involved planning the solution’s release into production, together with the technology and finance teams of the company and the consultancy. A thirty-day assisted operation was carried out, with support from the development team. Alert mechanisms for failures and maintenance periods were developed, and the project documentation was continuously drafted and reviewed. Challenges and successes were recorded to form a knowledge base for future projects.
3. Results and Discussion
The application of the CRISP-DM methodology in the context of financial and budgetary planning for the industrial company revealed a series of significant findings, which unfolded throughout the six phases of the process, from business understanding to solution implementation. The results demonstrated a substantial transformation in data flow management, addressing the initial challenges of manual processes and spreadsheet dependency, which compromised productivity, accuracy, and corporate governance. The developed solution not only automated the information flow but also enhanced data quality and agility in strategic decision-making, as per the central objective of the study.
In the initial Business Understanding phase, the study clearly identified the need to automate the process of classifying records for cash flow and budgets. The activity, previously performed manually, relied on multiple data sources and intensive use of Excel spreadsheets, which served as a rudimentary “database” and ETL (Extract, Transform and Load) tool, with macro-based automation. This approach, as pointed out by Shimaoka et al. (2024), significantly increases the risk of failure in data projects, exceeding 80% when there is no structured methodological basis. Direct human intervention in data entry and the creation of links between files and spreadsheets was a critical risk factor for data integrity and productivity.
The structuring of a seven-week activity schedule for the solution delivery was a fundamental step to mitigate the risks associated with the absence of a methodology. The clarity of the objectives, focused on increasing productivity, preserving data, systematizing the ETL process, enhancing data engineering reliability, reducing human interference, improving audit and governance, unifying databases, and enhancing classification, guided all subsequent stages. This systematic approach, inherent to CRISP-DM, was crucial for the effective direction of the project and for the optimized allocation of resources, avoiding the waste often associated with data projects without a methodology.
The Data Understanding phase allowed a detailed mapping of the multiple data sources and the creation of a comprehensive data dictionary. In-depth analysis revealed that the primary source of the data was the management system (ERP), which enabled the exploration of internal improvements. However, significant inconsistencies in data quality were identified. Approximately 11% of the data population for the “purchase order” variable had been excluded, and the absence of data was alarming, with about 99.8% in the “profit center” variable and 99.9% in “sales orders”. Furthermore, 40% of the “accounting accounts” variable population presented incorrect data, not resulting from manual alteration, but from failures recorded in the system logs, according to Machado’s (2020) analysis of event logs.
These findings on data quality highlighted governance problems, not only related to system access, but also to the assignments and validations of access to specific ERP areas, which allowed undue alterations and deletions, as discussed by Carvalho et al. (2020). The “health” of the data, a fundamental concept for Inmo (2005), was compromised, requiring the construction of APIs (Application Programming Interfaces) to request data directly from the management system. The need to correct undue user actions or failures in the ERP module integrations, which were likely not adequately validated during the project phase, as per Escovedo and Kalinowski (2023), became a critical point for ensuring data integrity and completeness.
In the Data Preparation stage, data were directly selected from the primary source, the ERP, and APIs executed the requests to correctly populate records that previously contained missing or incorrect data. Additional variables were created, such as the company name linked to the customer or supplier code, the type of financial record (Opex or Capex), and the record description, replacing manual entry via spreadsheets. This automation eliminated the risk of outdated or inconsistent information, which could lead to misinterpretations and ineffective strategies, as warned by Inmo (2005) and Carvalho et al. (2020).
The solution was developed using the Python language and libraries such as Pandas, Numpy, Polars, Requests, and OpenPyXL, which allowed for the processing and enrichment of records, including the analysis of null and duplicate data. The result of this processing was an Excel file made available in a specific company directory, which was subsequently integrated into Power BI for the creation of interactive visualizations. This approach not only systematized the ETL process but also ensured that the data was prepared for more assertive analyses, fundamental for directing business strategies.
The Modeling phase, although the shortest in the project, was crucial for the materialization of the solution. There was no need to apply complex machine learning algorithms, as the focus was on data engineering and manipulation for automation. The initial model, built from the scope and guidelines of the previous CRISP-DM phases, underwent rigorous testing. During these unit tests, errors related to the interpretation of business rules were identified and corrected, such as the identification of value signals (positive or negative), the character size of fields like company name, order number, and profit center, and the description linked to the type of financial record. The correction of these errors and the execution of new tests were essential to ensure the correct execution and delivery of the solution, in line with the importance of unit and automated testing to mitigate operational risks, as highlighted by Shimaoka (2024).
The model validation was carried out by the development team, which subjected the solution to tests with a historical database of up to three years of retroactivity. This comprehensive validation ensured the reliability of the solution and its adherence to business requirements. The ability to stress tasks and routines, as mentioned by Domingos (2023), ensures fairness, consistency, and the absence of failures in integrations, which was fundamental for the robustness of the developed model. The absence of a complex machine learning model simplified the phase, allowing a direct focus on the quality and integrity of the data pipeline.
In the Evaluation phase, the results confirmed the high quality of the output data, with no loss or duplication of records, and with consistency and respect for the structure of the relational database schema. These aspects are of extreme relevance for the finance area, which deals with sensitive data that directly impacts business strategy. One of the most impactful findings was the reduction of approximately 92% in processing time for product delivery. Additionally, the implementation of logs in the solution allowed for the auditing and traceability of the classification applied to financial records, mechanisms that, according to Carvalho et al. (2024), are essential for data governance and auditing.
Despite the advances, a persistence of 0.05% of classification failures was observed, attributed to manual entry of records in the ERP, where users did not fill in essential fields or the system did not perform adequate validation. Although the root cause was outside the project’s scope, this limitation highlights the importance of training and validations in ERP project phases, or the use of document reading tools for data entry, aiming to preserve the original data and mitigate risks. Continuous review of the process in each phase of CRISP-DM did not identify failures requiring rework, and the collected results presented no impediments for the deployment phase, ensuring project progression.
The final Production Deployment phase was executed in collaboration with the company’s technology and finance teams, as well as the consulting firm responsible for the delivery. The assisted operation, lasting thirty days with support from the development team, ensured a smooth transition. Alert mechanisms for failures and maintenance periods in the code were developed to guarantee continuous execution and solution delivery. Project documentation was prioritized from the beginning, undergoing revisions and fine-tuning until the pre-deployment phase was completed, ensuring the company possessed complete project knowledge. Challenges and success points were recorded, contributing to the knowledge base for future projects, a crucial aspect for continuous improvement and organizational knowledge management.
In summary, the use of Excel for data classification activity consumed more than three hours daily of the professional’s time, negatively impacting productivity. With the implementation of the API-based solution, the total processing time was drastically reduced from three hours and forty minutes to a range between ten and seventeen minutes, representing a saving of approximately 94% in the activity’s delivery time. The solution also enabled a significant reduction in classification errors, from 14% of the population to less than 0.05%, and increased agility in identifying missing classifications. These operational and technical gains, combined with the implementation of interactive dashboards for visualization and monitoring, and the structuring of audit and governance processes, resulted in more precise and agile indicators and analyses for management, fully meeting the objective of increasing productivity, preserving data, and improving corporate governance.
4. Conclusion
This study applied the CRISP-DM methodology to map the data flow of the financial and budgetary planning area of an industrial company, with the purpose of systematizing project phases, developing a data solution to increase productivity, preserve data, automate the ETL process, increase reliability in engineering steps, reduce human interference, improve auditing and governance mechanisms, unify primary databases, and elevate data classification quality, enabling more assertive analyses to guide business strategies. It was verified that the systematic application of CRISP-DM was fundamental for the transformation of data management, resulting in a significant improvement in operational efficiency. A reduction in total processing time was observed from three hours and forty minutes to an interval between ten and seventeen minutes, representing a saving of approximately 94% in activity delivery time. Additionally, a significant decrease in classification errors was identified, from 14% to less than 0.05%, and greater agility in identifying missing classifications. The developed solution, using Python and APIs for data extraction and treatment from the ERP, allowed for the unification of databases and the creation of interactive dashboards in Power BI, which facilitate visualization and monitoring, consolidating a substantial practical contribution to data governance and strategic decision-making.
Despite the advances achieved, the study identified a persistent limitation of 0.05% of classification failures, attributed to manual entry of records in the ERP system, where the absence of essential fields or lack of adequate system validation compromised data integrity at the source. Although the root cause was outside the direct scope of this project, this observation highlights the importance of future actions. For future studies, it is suggested to investigate more robust validation mechanisms in the ERP, implement continuous user training, or adopt document reading tools to automate data entry, aiming to completely mitigate these risks and preserve the quality of the original data. The detailed project documentation and the record of lessons learned contribute to the company’s knowledge base, supporting continuous improvement and the replication of successes in future data engineering initiatives.
Bibliographic References
Araújo, L. C.; Silva, M. G. 2019. Lean Six Sigma: Guia prático para redução de desperdícios e melhoria de processos. Atlas, São Paulo, SP, Brasil.
Assaf Neto, Alexandre. 2021. Estrutura e Análise de Balanços: Um enfoque econômico-financeiro. 12. ed. Atlas, São Paulo, SP, Brasil.
Carvalho, André Castro; Panochia, Patricia; Capp, Ricardo T. 2020. Gestão de risco e compliance. Editora Senac, São Paulo, SP, Brasil.
Chapman, P. et al. 2000. CRISP-DM 1.0: step-by-step data mining guide. Disponível em: <https://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/14.2/es/CRISP-DM.pdf>. Acesso em: 21 abril 2026.
Data Science Process Alliance. 2024. What is CRISP-DM? Disponível em: <https://www.datascience-pm.com/crisp-dm-2/>. Acesso em: 21 abril 2026.
Domingos, 2023 [Referência completa não encontrada no documento original]
Escovedo, Tatiana; Kalinowski, Marcos et al. 2023. Engenharia de Software para Ciência de Dados: Um guia de boas práticas com ênfase na construção de sistemas de Machine Learning em Python. Casa do Código, São Paulo, SP, Brasil.
Gitman, Lawrence J. 2017. Princípios de Administração Financeira. 14. ed. Pearson Education do Brasil, São Paulo, SP, Brasil.
Huber et al., 2018 [Referência completa não encontrada no documento original]
Inmon, William H. 2005. Building the Data Warehouse. 4. ed. Wiley, Indianapolis, IN, United States of America.
Iudicibus, 2019 [Referência completa não encontrada no documento original]
Jung, Dominik. 2024. The Modern Business Data Analyst: A case study introduction into business data analytics with CRISP-DM and R. Springer, Cham, Switzerland.
Machado, Felipe Nery Rodrigues. 2020. Arquitetura de Sistemas de Informação. 4. ed. Érica, São Paulo, SP, Brasil.
Oliveira, Roberto; Rondina, João Marcelo. 2023. Business Intelligence: Ferramentas e métodos. Editora FGV, Rio de Janeiro, RJ, Brasil.
Padoveze, Clovis Luís. 2012. Controladoria Estratégica e Operacional: Conceitos, estrutura, aplicação. 3. ed. Cengage Learning, São Paulo, SP, Brasil.
Plotnikova, V., Dumas, M., Nolte, A.; Milani, F. 2023. Designing a data mining process for the financial services domain. Journal of Business Analytics. Disponível em: <https://doi.org/10.1080/2573234X.2022.2088412>. Acesso em: 21 abril 2026.
Reis, Joe; Housley, Matt. 2024. Fundamentos de Engenharia de Dados. Novatec, 2024. São Paulo, SP, Brasil.
Saudate, Alexandre. 2017. APIs REST: Seus serviços prontos para o mundo real. Casa do Código, São Paulo, SP, Brasil.
Shearer, C. 2000. The CRISP-DM model: the new blueprint for data mining. Journal of Data Warehousing. v. 5, n. 4, p. 13-22.
Shimaoka, André M.; Ferreira, Renato C.; Goldman, Alfredo. 2024 The evolution of CRISP-DM for Data Science: Methods, Processes and Frameworks. SBC Reviews on Computer Science. Disponível em: <https://www.doi.org/10.5753/reviews.2024.3757>. Acesso em: 21 abril 2026.
Article originating from the Final Course Work of the Specialization in Software Engineering of the MBA USP/Esalq
To learn more about the course, click here and access the MBX Academy platform