Executive Summary

_{Author: Beatriz Carvalho Almeida — Advisor: Renato Godoi Da Cruz}

^{Summary prepared by the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege focused on synthesis and writing.}

This research analyzed the patterns of errors and successes in the Chemistry questions of the National High School Exam (ENEM) to reflect on the skills in Chemistry and Natural Sciences developed in Basic Education. Based on this analysis, it was discussed how the competencies of the National Common Curricular Base (BNCC) are mobilized and what reflections this brings to teaching. The investigation diagnosed, through this important evaluative instrument, the proficiency of high school graduates not only in conceptual content, but in the ability to apply scientific knowledge critically and contextually, a pillar for the formation of scientifically literate citizenship.

Recent educational and curricular documents, both national and international, converge on an education that transcends the transmission of concepts, focusing on the understanding of interrelationships between science, technology, society, and the environment. This approach aims to empower students for public debates and informed decisions on socio-scientific issues (NRC, 2012). In Brazil, the BNCC formalizes this perspective, establishing scientific literacy as a central objective of Basic Education. The Base advocates for the development of competencies that articulate conceptual knowledge with the social, historical, and environmental contextualization of science, in addition to fostering research practices and the mastery of scientific languages (Brasil, 2018).

Large-scale assessments, such as the Programme for International Student Assessment (PISA), reflect this trend by evaluating how students apply knowledge to analyze, reason, and communicate when solving problems in diverse contexts (OECD, 2006). Similarly, ENEM, since its reformulation in 2009, has adopted an approach based on competencies and skills. The exam has been structured into knowledge areas to assess students’ ability to mobilize their knowledge to solve complex problems, many of which require reflection on the social and technological impacts of scientific development (Costa et al., 2016).

The ENEM reference matrix for the area of Natural Sciences and their Technologies highlights this orientation. Area competence 7, which covers Chemistry, is not limited to testing the mastery of theories, but requires candidates to evaluate the social, environmental, and economic impacts associated with the production and use of resources, as well as the risks and benefits of human interventions in the environment (Brazil, 2024). This structure positions ENEM as a potential thermometer of the success of educational policies aimed at a more integral and critical scientific education.

Given the relevance of ENEM as the main access route to higher education and as an indicator of the quality of Basic Education, it is crucial to investigate whether its evaluative design and results mirror the development of the advocated skills. This study deepens this issue with a robust quantitative analysis of the exam’s microdata. Focusing specifically on Chemistry questions, the research seeks to offer a detailed diagnosis of which competencies are solidly developed and which represent bottlenecks in the training of young people, providing subsidies for reflection on pedagogical practices and curricular policies.

This is a quantitative study of an exploratory-descriptive nature, which employed multivariate statistics to analyze secondary data. The source was the microdata from ENEM 2023, made available by the National Institute for Educational Studies and Research Anísio Teixeira (INEP). The original database was filtered, selecting exclusively the records of participants who responded to the test booklets of blue, yellow, pink, and gray colors, ensuring that everyone in the sample had been exposed to the same set of questions. The final consolidated sample for the analysis was composed of 8,152 participants.

The subsequent step was the identification of Chemistry questions. The ENEM Natural Sciences exam is interdisciplinary, with 45 items of Chemistry, Physics, and Biology. A qualitative analysis was conducted by the researcher to identify the 15 questions of predominantly chemical content. Once selected, the questions were mapped to each exam booklet for the correct extraction of answers. The central variable, ‘Result’, was created from the comparison between the participant’s answer and the official answer key, being categorized as ‘Correct’ or ‘Incorrect’ for each of the 15 questions.

The crucial step of the methodology was the categorization of each question according to the skills of the ENEM reference matrix. Each question was classified into two dimensions: ‘Chemistry Skill’ and ‘Natural Sciences Skill’. The first dimension focused on area competence 7, concerning the appropriation of Chemistry knowledge. The second covered a broader spectrum, incorporating skills from area competences 1, 2, 3, and 5, related to scientific languages, quality of life, and environmental interventions. This dual categorization recognizes the interdisciplinary nature of ENEM questions, where solving a chemical problem often demands more general scientific competencies. For example, a question about chemical equilibrium in the context of water contamination was classified both by the specific chemical skill (H27 – evaluate risks and benefits) and by the general science skill (H7 – use criteria to evaluate products).

For the data analysis, the chosen technique was Multiple Correspondence Analysis (MCA), implemented in Python with the Prince library. MCA is suitable due to its ability to analyze the interrelationship structure between multiple categorical variables and to graphically represent these associations, facilitating the interpretation of complex patterns (Bertoncelo, 2022). The process included the generation of contingency tables, the application of the chi-squared test to verify statistically significant associations, and the calculation of adjusted standardized residuals to identify the strength of specific associations (Fávero and Belfiore, 2015). Finally, MCA was executed to extract two main dimensions, and the model’s adequacy was verified by analyzing the eigenvalues. The interpretation of the results was based on the analysis of the perceptual map and the contributions of each category, with a significance level of 5% for all tests.

The initial exploratory analysis revealed a challenging panorama: 74.8% of the analyzed responses were incorrect, while only 25.2% were correct. This result suggests a widespread difficulty in solving Chemistry items. Deepening the analysis, the success rate was calculated for each skill. Skills H26 (evaluate impacts of energy resource production/consumption) and H4 (evaluate environmental intervention proposals aiming at sustainability) showed the highest success rates. In contrast, skills H27 (evaluate risks and benefits of environmental interventions based on chemical knowledge) and H10 (analyze environmental disturbances and their impacts) registered the worst performances.

The chi-squared test confirmed that the associations between the outcome (correct/incorrect) and the skill categories were statistically significant (p < 0.001), validating that performance is not random. The analysis of adjusted standardized residuals corroborated the exploratory analysis, showing a positive and significant association between skill H26 and correct answers, and a strong negative association between skills H27 and H24 (use Chemistry codes and nomenclatures) and correct answers, i.e., a strong association with errors. Similarly, for the general Natural Sciences skills, H4 and H18 (relate material properties to their uses) were positively associated with correct answers, while H7 (use criteria to compare products) and H10 were significantly associated with errors.

Multiple Correspondence Analysis (MCA) resulted in a two-dimensional model that explained 41.20% of the total data inertia, a value considered robust for studies in social and educational sciences. The interpretation of the perceptual map revealed a clear structure. The first dimension was predominantly defined by the opposition between skills H27, H10, and H7, which were located close to the ‘Incorrect’ category. The second dimension was defined by the opposition between skills H4, H26, and H24, with the first two (H4 and H26) positioning themselves close to the ‘Correct’ category. This configuration suggests that Dimension 1 can be interpreted as an axis of “analytical-chemical difficulty”, while Dimension 2 represents an axis of “contextual socio-environmental understanding”.

The discussion of these results allows us to infer that students demonstrate greater ease in mobilizing knowledge to analyze environmental issues of a more general nature, such as sustainability (H4) and energy consumption impacts (H26). These are skills that can be developed through a discursive understanding of the topics, frequently addressed in the media. Satisfactory performance in these areas may indicate that the contextualized approach to science teaching, advocated by the BNCC, has had some success in promoting awareness of major socio-environmental issues.

In stark contrast, performance drops drastically when questions require a more technical and specific application of chemical knowledge. The worst-performing skills (H27, H10, H7) demand that the student not only recognize a problem but also use chemical concepts (such as equilibrium, reactivity, concentration) to analyze data and evaluate risks quantitatively or semi-quantitatively. This result suggests that the bridge between conceptual knowledge of Chemistry and its practical application in complex contexts is not being solidly built during Basic Education.

One of the most significant findings was the strong association of skills H17 (relating information presented in different forms of language) and H24 (using codes and nomenclature of Chemistry) with incorrect answers. These instrumental skills represent the mastery of the “language of science”: the ability to read and interpret graphs, tables, and chemical equations. Systematic difficulty in these competencies functions as a fundamental bottleneck, preventing students from processing the information needed to solve problems, even if they possess some conceptual knowledge. This finding corroborates previous studies pointing to deficiencies in graphic and symbolic literacy (Veras et al., 2021) and reinforces that full scientific literacy, which includes fluency in the forms of scientific representation, remains a distant goal.

The joint analysis points to a paradox in Science teaching and assessment. Although ENEM and curricular documents promote a contextualized approach, success in the exam still seems to be determined by proficiency in more traditional disciplinary skills, especially those linked to the mathematical and symbolic language of Chemistry. This suggests that pedagogical approaches may not be effectively integrating the development of conceptual, contextual, and instrumental competencies. The difficulty in transferring chemical knowledge to problem-solving situations, as identified by Xavier et al. (2021), appears to be a symptom of this disarticulation. The results indicate that the assessment may end up measuring proficiency in decoding scientific representations more than the capacity for critical reasoning about socio-scientific issues, as also pointed out by Rosa et al. (2019).

This study revealed a significant mismatch between the pedagogical objectives formalized in the guiding educational documents and the effective development of scientific skills demonstrated by Basic Education graduates in the ENEM. The analysis of performance patterns in Chemistry indicates that, although students show some ability to articulate knowledge with broad environmental themes, they face systematic difficulties in competencies that are the basis of scientific literacy: the interpretation of languages and representations specific to science, such as graphs, tables, and nomenclature. This instrumental gap compromises young people’s ability to mobilize scientific knowledge autonomously and critically for informed decision-making. The results suggest that Chemistry teaching may be prioritizing the memorization of concepts in isolation, to the detriment of contextualized application that requires fluency in scientific language.

The persistence of these difficulties points to the need for a reformulation of pedagogical practices. It is imperative that Science teaching integrates, inseparably, the scientific concept, its application in social and environmental contexts, and the explicit development of data representation and analysis skills. Only through an approach that treats the language of science as a central part of scientific knowledge itself will it be possible to achieve the scientific literacy objectives advocated by the BNCC. It is concluded that the research objective was fully achieved: it was demonstrated that the performance patterns in Chemistry in the ENEM reveal an unequal development of skills, with proficiency in contextual environmental themes, but with critical gaps in instrumental scientific literacy, especially in the interpretation of scientific languages and representations.

References:
^{Bertoncelo, E. 2022. Building relational spaces with multiple correspondence analysis: applications in social sciences. ENAP, Brasília, DF, Brazil.
Brazil. Ministry of Education [MEC]. 2018. National Common Curricular Base.
Catena, D.; Testa, I. 2024. Exploring students’ attitudes towards physics and their association with gender. Journal of Physics: Conference Series 2727: 012017.
Costa, E. S. C.; Santos, M. L.; Silva, E. L. 2016. Approach to chemistry in the new ENEM: an analysis of interdisciplinarity. Química Nova na Escola 38(2): 112-120.
Fávero, L. P.; Belfiore, P. 2015. Data analysis: exploratory multivariate techniques with SPSS and STATA. 1st ed. Elsevier, Rio de Janeiro, RJ, Brazil.
Giuliana, G.; Marzoli, I.; Testa, I.; Scotti di Uccio, U. 2024. Exploring students’ views about basic concepts in introductory quantum mechanics. Journal of Physics: Conference Series 2750: 012021.
National Institute for Educational Studies and Research Anísio Teixeira [INEP]. 2024. Reference matrix for ENEM 2023.
Koyuncu, I. 2021. TIMSS International Benchmarks of Eighth Graders in Mathematics: a correspondence analysis study. International Electronic Journal of Elementary Education 14(2): 179-194.
National Research Council [NRC]. 2012. A Framework for K-12 Science Education: practices, crosscutting concepts, and core ideas. The National Academies Press, Washington, DC, USA.
Organization for Economic Cooperation and Development [OECD]. 2006. Assessing Scientific, Reading and Mathematical Literacy: A Framework for PISA 2006. OECD Publishing, Paris, France.
Prada Núñez, R.; Rabelo Procopio, M. V.; Fernández-Cézar, R.; Solano-Pinto, N. 2024. Affective domain and mathematics achievement of Colombian students under multiple correspondence analysis. Frontiers in Education 8: 1261829.
Rosa, T.; Lorenzetti, L.; Lambach, M. 2019. Levels of scientific and technological literacy in the chemistry assessment of the National High School Exam. Educação Química en Punto de Vista 3(1): 1730.
Veras, W. S.; Ferreira, L. A.; Sousa, P. S. A.; Carvalho, H. A.; Rodrigues, M. G.; Vieira, T. B. S.; Oliveira, R. E. G.; Alves, M. J. P.; Ayala, K. N. R.; Silva, T. B. L.; Araujo, J. C.; Siqueira, F. J. G.; Vieira Junior, R. P. M. 2021. National High School Exam (ENEM): Most addressed mathematical content and its difficulties. Research, Society and Development 10(6): 16041.
Xavier, R. P. S.; Quintella, S. A.; Jacinto Junior, S. G.; Pires, A. M. L. 2021. Analysis of student difficulties in the chemistry curriculum component based on the content addressed in ENEM. Research, Society and Development 10(15): 23523.}

---

Executive summary from the Final Project of the Specialization in Data Science and Analytics from the MBA USP/Esalq

_{Learn more about the course; click here:}