September 15, 2026
Between the Artificial and the Human: Identification of Humanized Texts in Enem by Linguistic Features
Lucas Lopes Ribeiro; Ana Julia Righetto
DOI: 10.22167/2675-6528-202602261
Article derived from a Final Course Work (TCC), with content based on the student’s original work and adapted to the editorial format of the Revista E&S with the support of the ResumeAI tool, an artificial intelligence solution developed by the Pecege Institute for textual synthesis and organization.
Summary
The growing use of language models in academic text production has made the development of methods to identify authorship in textual productions relevant, especially in the face of humanization tools that mask synthetic origin. The objective was to investigate the feasibility of detecting texts generated by artificial intelligence, even after automatic rewriting, through interpretable linguistic attributes. For this purpose, a corpus of argumentative-dissertative essays in the ENEM standard was built, encompassing human texts, texts generated by a language model, and versions humanized by a paraphrasing tool. Stylometric metrics of lexical diversity, syntactic complexity, readability, and textual predictability were extracted and used in the modeling of a supervised binary classification problem. Three machine learning algorithms (Logistic Regression, Random Forest, and XGBoost) were trained and evaluated by internal validation, on an independent dataset, and in comparison with a commercial tool. The models showed high performance in distinguishing between human and synthetic texts, maintaining consistency even with humanized texts, while the commercial tool showed greater difficulty. It was concluded that the approach based on stylometric attributes proved to be effective and robust for the detection of textual authorship in the analyzed context.
Keywords: Linguistic analysis; Machine learning; Text authorship detection; Text humanization; Natural language processing.
1. Introduction
Artificial Intelligence (AI) is a field of study focused on building systems capable of perceiving, reasoning, and acting upon the world (Nilsson, 1998). Among its subfields, Natural Language Processing (NLP) stands out, responsible for enabling machines to understand and produce human language. The recent advancement of large language models (LLMs), such as GPT-3 (Brown et al., 2020), has revolutionized the way texts are automatically generated, making AI-produced writing increasingly fluent, coherent, and difficult to distinguish from human writing. Consequently, not only artificial text detectors have emerged, but also tools called “humanizers,” which rewrite AI texts through paraphrasing and stylistic alterations to make them harder to identify (Masrour et al., 2025).
As researchers delve into the literature, the detection of texts generated by artificial intelligence has become a very active research area. However, studies show that well-founded detectors can lose effectiveness when texts are modified, and commercial or research detectors exhibit high false positives or negatives, especially in academic texts (Weber-Wulff et al., 2023). Regarding humanization tools, studies such as Masrour et al. (2025) analyzed nineteen commercial tools and demonstrated that they significantly reduce the accuracy of established detectors, reinforcing the need for more robust methods, especially in educational contexts, where artificially modified texts can affect learning.
Given these mentioned studies, despite presenting significant advances, they share some limitations: many focus on general scenarios, such as news or online texts, and not on school texts or genres like ENEM essays. Furthermore, there is a scarcity of research exploring explainable linguistic features, such as syntax, cohesion, connectors, and lexical variation as a central part of the model. A recent example is the work by Yadagiri et al. (2024), which proposes the detection of AI-generated texts from linguistic and stylistic features, such as lexical diversity, readability, and the use of passive voice, demonstrating that this information can improve both the accuracy and interpretation of the results. The use of these features can offer two main advantages: improve the interpretability and transparency of the model, and capture more subtle marks left by rewriting or humanization processes, which purely probabilistic or perplexity-based detectors might miss.
With this in mind, in the educational context, the discussion about the use of artificial intelligence tools in writing becomes increasingly relevant. According to the Regional Center for Studies for the Development of the Information Society (Cetic, 2023), seven out of ten high school students already use generative AI resources in school activities, and Brazil ranks as the third country that most uses ChatGPT, with over 140 million messages exchanged daily (CNN Brasil, 2024). Although these tools can assist in text elaboration, indiscriminate use can compromise the development of essential skills, such as argumentation, cohesion, and creativity (Costa; Von Rondon, 2024).
Given this scenario, the present work is justified by the need to develop effective methods for identifying texts produced by artificial intelligence, even after humanization processes, in the educational environment. The present work aimed to develop a classification model capable of identifying texts generated by artificial intelligence, even after the application of humanization tools, focusing on argumentative-dissertative essays in the standard of the National High School Exam (ENEM).
2. Material and Methods
The research adopted an experimental approach of an applied nature, with quasi-experimental elements. It sought to empirically analyze the differences between texts produced under conditions of human, artificial, and humanized textual generation, focusing on measurable linguistic characteristics. The data were processed under control and repeatability, with pre-existing human texts not having control over external variables, while artificial productions were generated under conditions controlled by language models and defined prompts, and humanization tools.
The corpus was constituted by 1,800 argumentative-dissertative essays in the ENEM standard, divided into three categories of 600 texts each: (i) human essays, produced by high school students between January 2020 and December 2022, with a minimum length of 150 words and a score above 700 points, provided by a Brazilian educational technology startup; (ii) artificial essays, generated by the language model GPT-4o mini (OpenAI) from the same themes as the human essays and standardized prompts to simulate ENEM instructions; and (iii) humanized essays, obtained by automatic rewriting of the artificial versions by the paraphrasing tool QuillBot (Standard mode), with the objective of softening traces of synthetic writing. Each dataset was organized into tabular spreadsheets, with columns for ID, full text, writing prompt, theme, and, for human essays, assigned score; for artificial essays, AI model; and for humanized essays, humanization tool.
The computational processing of the texts was carried out in the Python language, using the spaCy library (Honnibal & Montani, 2017) for sentence segmentation, part-of-speech tagging (POS Tagging), and dependency parsing; the Transformers library (Wolf et al., 2020) for the GPT-2 model used in perplexity calculation; and Pandas (McKinney, 2010) for the tabular structuring of the corpus. A conservative preprocessing approach was adopted, restricting ourselves to tokenization to preserve stylometric markers.
For the quantitative characterization of writing styles, 26 linguistic features were extracted, organized into four main analytical dimensions: a) Lexical Diversity and Repetition (Type-Token Ratio – TTR, Hapax Legomena, Shannon Entropy); b) Complexity and Readability (Flesch and Gunning Fog indices adapted for Portuguese (Martins et al., 1996), average sentence and line length, deep syntactic complexity); c) Morphosyntactic Analysis (distribution of word classes via POS Tagging); d) Argumentative Structure and Cohesion (adversative and conclusive connectives, Conclusive Logical Reason, incidence of analytical passive voice); e) Rhythm and Predictability (Burstiness, perplexity with GPT-2 Small Portuguese).
The problem was formally modeled as a supervised binary classification task. The three original corpus categories were recoded into two mutually exclusive classes: Human (0) and Artificial/Humanized (1), grouping the texts generated by GPT-4o mini and the versions rewritten by QuillBot. This methodological decision aimed to distinguish genuinely human authorship from any form of synthetic production. The selection of the final feature vector was guided by comparative exploratory analysis, resulting in a definitive set of 10 high-separability stylometric features, which included Perplexity, Burstiness, Lexical Diversity (TTR), Flesch Reading Ease Score, Passive Voice Ratio, Conclusive Logic Ratio, and syntactic densities of Adjectives, Adverbs, Pronouns, and Subordinating Conjunctions. Variables without robust discriminative power were excluded. Performing this step prior to the train-test split may introduce feature selection bias, which was considered in the interpretation of the results.
For the internal validation experiments, the complete corpus of 1,800 essays was divided by holdout. 1,500 instances were allocated for training and model optimization, and 300 instances for the internal test set. The division was performed with stratified sampling, using the `stratify` parameter of the `train_test_split` function from the Scikit-Learn library, resulting in 100 human essays and 200 synthetic essays (100 artificial and 100 humanized) in the test set. The random seed `random_state=42` was fixed to ensure reproducibility.
Three supervised algorithms were trained: Logistic Regression, Random Forest, and XGBoost. Logistic Regression, adopted as a linear baseline, had its training data standardized with Scikit-Learn’s `StandardScaler` and was instantiated with `max_iter=2000` and `random_state=42`. Random Forest was configured with 100 decision trees (`n_estimators=100`), automatic class balancing (`class_weight=’balanced’`), and `random_state=42`. XGBoost was configured with 100 estimators (`n_estimators=100`), a learning rate of 0.1 (`learning_rate=0.1`), a maximum tree depth of 5 (`max_depth=5`), an internal evaluation metric `eval_metric=’logloss’`, and `random_state=42`. Upon completion of training, the models and the fitted `StandardScaler` were serialized and persisted to disk using the `joblib` library.
External validation was conducted by applying the trained models to a new independent dataset, composed of 300 unpublished essays, which were not part of the original corpus. This external set was constructed following the same methodological criteria as the main corpus: 100 human essays (score above 700 on the ENEM standard), 100 essays generated by the GPT-4o mini model from new themes and prompts, and 100 essays humanized using the QuillBot tool (Standard mode) applied to the synthetic texts. The models were loaded and applied directly, without adjustment or recalibration, with the `StandardScaler` used for Logistic Regression.
The performance of the models was evaluated using a standardized set of binary classification metrics (Sokolova & Lapalme, 2009; Fawcett, 2006): Global Accuracy, Precision, Recall, and F1-Score. Complementarily, Confusion Matrices were analyzed to examine error patterns, distinguishing false positives (human text classified as synthetic) and false negatives (synthetic text, including humanized text, classified as human).
To contextualize the models’ performance, a comparative evaluation was conducted with GPTZero, a commercial tool for detecting texts generated by language models. The evaluation was performed on the same set of 300 essays from the external validation. Each essay was individually submitted to the GPTZero web interface, and the classifications were manually recorded. The GPTZero outputs were converted to a binary classification task (human texts vs. synthetic texts, including humanized ones), and the same evaluation metrics were calculated. This evaluation was conducted from a single sample and through manual interaction with the tool, which may introduce limitations regarding the reproducibility and standardization of the process.
3. Results and Discussion
This section presents and discusses the results obtained throughout the research, structured into two main analytical fronts. The first part is dedicated to exploratory data analysis, detailing the stylometric metrics evaluated and the results of the statistical tests that supported the selection of the most relevant attributes for the binary classification problem. The second part focuses on evaluating the predictive performance of the trained machine learning architectures, including internal and external validation, and the benchmarking carried out against a commercial tool.
Exploratory analysis and selection of stylometric features
The initial stage of data analysis consisted of validating the feature vector through a comparative exploratory analysis among the three text categories: human, artificial, and humanized. The objective was to verify the existence of statistical patterns that would allow distinguishing between genuine human production and texts generated by language models, as well as to assess the impact of obfuscation (humanization) tools on these metrics.
The generated graphs revealed distinct behaviors among the classes, especially regarding the deep structure and lexical variability of the texts. It was observed that certain metrics showed a clear separation between human and artificial authorship, while others demonstrated the limitation of humanization tools in altering the “digital signature” of the base models.
The analysis of adjective density, presented in Figure 1, revealed one of the most pronounced distinctions between authorship classes. It was observed, through the box plot, that human authorship essays exhibit a significantly lower median of adjectivation, with 0.08 compared to texts generated by artificial intelligence and humanized, with approximately 0.11.
Figure 1. Adjective analysis – boxplot.
Source: Original research results
This result suggests that adjective density constitutes a strong stylometric marker for synthetic authorship detection, as generative models tend to overuse adjectives to emulate assertiveness and formality, distancing themselves from the sobriety characteristic of natural student writing.
The analysis of lexical diversity, operationalized by the Type-Token Ratio (TTR) (Figure 2), revealed a robust statistical divisor between the categories. While the human group presented a median of 0.59, with an interquartile range extending to 0.62, the artificial productions (AI and Humanized) demonstrated a significantly lower diversity ceiling, with medians for both situated at 0.52. This reduction of approximately 13% in the lexical richness of AIs suggests a dependence on a restricted lexical core, a characteristic that humanization tools failed to mitigate, keeping synthetic productions statistically segregated from the natural variability of students.
Figure 2. TTR Analysis – boxplot
Source: Original research results
Regarding perplexity, this metric, originating from Information Theory, measures the degree of “surprise” of a language model when processing a sequence. It was observed (Figure 3) that the human group exhibits the highest level of entropy, with a median of 42.48, which highlights the non-linear and less predictable nature of student writing. In contrast, the AI productions registered only 14.78, reflecting the tendency of generative models to select statistically high-probability tokens, resulting in mechanically “expected” texts. The humanized version presented a median of 22.03, indicating that paraphrasing introduces lexical variations that increase perplexity, but still fails to reach the informational complexity levels of human authorship.
Figure 3. Perplexity analysis – boxplot
Source: Original research results
Regarding Burstiness (Figure 4), writing rhythm, operationalized by the standard deviation of sentence length, revealed the machine’s syntactic monotony. The human group’s median of 12.58 stands out for its wide variability, indicating that students dynamically alternate between short sentences and long subordinate clauses. In artificial and humanized productions, the medians were 8.50 and 9.53, respectively. This compression in Burstiness values indicates that AI tends to generate paragraphs with nearly uniform length structures to ensure cohesion, resulting in a standardized reading rhythm that the paraphrasing tool cannot make organic.
Figure 4. Burstiness Analysis – BoxPlot
Source: Original research results
The incidence of analytical passive voice acted as an unexpected indicator of human authorship in the context of ENEM. The observed median (Figure 5) in the human group was 0.18, a value substantially higher than the 0.11 of the artificial group and 0.10 of the humanized group. This finding suggests that, in the pursuit of meeting the exam’s competencies that require impersonality, the human student makes intensive and, at times, exaggerated use of the passive voice. AIs, being optimized for clarity and readability (active voice), fail to emulate this stylistic behavior specific to the textual genre investigated, creating a usable statistical gap for detection.
Figure 5. Passive voice analysis – boxplot
Source: Original research results
Regarding the Flesch Index, this metric assesses text complexity based on word and sentence length, where lower scores indicate denser and more complex texts. It was observed (Figure 6) that human essays present a median of 39.84, placing them in a “difficult” reading zone, but still higher than the levels recorded by artificial productions. The Artificial group presented a median of 30.12, and the Humanized group 31.28, which places them in the “very difficult” reading category. This difference of almost 10 points suggests that artificial intelligence generates texts that are significantly denser and with longer words (polysyllables) than the average student. Furthermore, the human group’s variability is notable: the standard deviation of 10.08 for humans, versus only 6.03 for AI, demonstrates that while students exhibit different proficiency levels and writing styles, AI maintains a “standardized density” and excessive complexity, which humanization fails to effectively smooth out.
Figure 6. Flesch index analysis – boxplot
Source: Original research results
The analysis of adverb density (Figure 7) revealed an inverse behavior to that observed in adjectivation (Figure 1), consolidating itself as a marker of human authorship. Students presented a median of 0.051, a value significantly higher than the 0.038 recorded by the artificial group. This finding suggests that human writing is richer in adverbial adjuncts that confer nuances of manner, time, and intensity to the argumentation. Notably, the humanization process resulted in a further decline of this metric to 0.034, indicating that automated paraphrasing tends to neutralize the text, eliminating circumstantial markers that are essential for the fluidity and subjectivity of human discourse.
Figure 7. Adverb analysis – boxplot
Source: Original research results
Regarding the analysis of the conclusive logic ratio (Figure 8), this metric, which calculates the proportion between conclusive and adversative connectives, revealed a discrepancy between the groups. While human students presented a median of 0.87, AI and humanized texts registered values of 0.40 and 0.41, respectively. This data indicates that human writing uses almost double the logical closing structures in relation to dialectical oppositions, suggesting an argumentative progression more directed towards the conclusion, while AI maintains an artificial balance that results in more neutral and less assertive argumentation.
Figure 8. Conclusive logical ratio analysis – boxplot
Source: Original research results
The analysis of conjunctions (Figure 9 and Figure 10) revealed a profound structural difference in sentence architecture. The human group used significantly more subordinate conjunctions (median of 0.036) than AI (0.023) and the humanized group (0.025). In contrast, AI demonstrated a preference for coordinate conjunctions (median of 0.049 vs. 0.033 for humans). This inversion proves that human writing is inherently more complex and nested, while language models generate more linear and independent structures, facilitating reading but losing the syntactic density typical of high school.
Figure 9. Analysis of subordinating conjunctions – boxplot
Source: Original research results
Figure 10. Analysis of coordinative conjunctions – boxplot
Source: Original research results
The other linguistic metrics extracted did not show robust statistical variations or satisfactory discriminative power for distinguishing between the studied classes, justifying their exclusion from the final attribute vector. Thus, the exploratory analysis consolidated a definitive vector composed of 10 stylometric characteristics with high separability power, which formed the exclusive database for predictive modeling. This final set includes the following metrics: Perplexity, Burstiness, Lexical Diversity (TTR), Flesch Reading Ease Score, Passive Voice Ratio, and Conclusive Logical Ratio, in addition to the syntactic densities of Adjectives, Adverbs, Pronouns, and Subordinating Conjunctions.
Performance in internal validation and baseline establishment
Once the final vector composed of ten selected stylometric features was defined, the predictive modeling step was carried out, with the training of three supervised algorithms: Logistic Regression, Random Forest, and XGBoost. Internal validation was conducted on a test sample composed of 300 essays, partitioned by stratified holdout. The results indicated high predictive performance. As observed in Table 1, Logistic Regression, adopted as a linear baseline, achieved 100% (1.00) global accuracy, with precision, recall, and F1-Score close to 1.00 for both classes. The tree-based models, Random Forest and XGBoost, also presented high performance, with global accuracy of approximately 99% and balanced metrics between the Human and Artificial/Humanized classes.
Table 1. Predictive performance of the models in Internal Validation
| Model | Class | Precision | Revocation (Recall) | F1-Score | Global Accuracy |
|---|---|---|---|---|---|
| Logistic Regression | Human (0) | 1 | 0,99 | 0,99 | 1 |
| Synthetic/Obfuscated (1) | 1 | 1 | 1 | ||
| Random Forest | Human (0) | 0,98 | 0,99 | 0,99 | 0,99 |
| Synthetic/Obfuscated (1) | 0,99 | 0,99 | 0,99 | ||
| XGBoost | Human (0) | 0,98 | 0,98 | 0,98 | 0,99 |
| Synthetic/Obfuscated (1) | 0,99 | 0,99 | 0,99 |
Source: Original research results.
These results suggest that the selected set of stylometric attributes has relevant discriminatory capacity between classes in the context of the analyzed sample. The performance of Logistic Regression, in particular, indicates that part of this separation can be captured by linear decision boundaries in the feature space. However, it is important to consider that the feature selection was performed prior to the train-test split, which may introduce bias in the internal performance estimation.
Performance in External Validation and the Structural Failure of Humanization
External validation was conducted with the objective of evaluating the generalization capacity of the models on data not used in any previous stage of the process. For this purpose, a set composed of 300 unpublished essays was constructed, with 100 human-generated, 100 AI-generated, and 100 humanized through QuillBot. The results obtained remained high, although slightly lower than those observed in the internal validation. According to Table 2, the Random Forest and XGBoost models achieved a global accuracy of approximately 99.33%, while Logistic Regression presented an accuracy of 99.67%. The precision, recall, and F1-Score metrics remained balanced among the classes, indicating consistency in the classifiers’ performance.
Table 2. Predictive performance of the models in External Validation
| Model | Class | Precision | Revocation (Recall) | F1-Score | Global Accuracy |
|---|---|---|---|---|---|
| Logistic Regression | Human (0) | 1 | 0,99 | 0,99 | 0,9967 |
| Logistic Regression | Synthetic/Obfuscated (1) | 1 | 1 | 1 | 0,9967 |
| Random Forest | Human (0) | 0,99 | 0,99 | 0,99 | 0,9933 |
| Random Forest | Synthetic/Obfuscated (1) | 0,99 | 0,99 | 0,99 | 0,9933 |
| XGBoost | Human (0) | 0,99 | 0,99 | 0,99 | 0,9933 |
| XGBoost | Synthetic/Obfuscated (1) | 0,99 | 0,99 | 0,99 | 0,9933 |
Source: Original research results.
It is noteworthy that Logistic Regression showed a revocation equal to 1.00 for the Artificial/Humanized class, indicating that, in the evaluated set, all synthetic texts, including those subjected to the humanization process, were correctly identified. Although this result is significant, its interpretation must be contextualized: it refers to a performance observed in a specific dataset, limited to a single generator model (GPT-40 mini) and a single paraphrasing tool (QuillBot, Standard mode). The proximity between the results of internal and external validations suggests that the models did not suffer significant performance degradation when applied to new data. This behavior indicates that, at least in the adopted experimental context, the classifiers were able to capture relatively stable patterns between the classes.
Generalization and Mitigation of Attribute Selection Bias
The consistency observed between the results of internal and external validations suggests that the selected set of stylometric attributes captures relevant aspects of the differentiation between human and synthetic texts in the analyzed context. Metrics such as perplexity, lexical diversity, burstiness, and morphosyntactic indicators proved useful for representing structural and stylistic patterns associated with different forms of textual production. However, it is important to emphasize that the selection of these features was based on the analysis of the complete original dataset, which may introduce attribute selection bias. The use of an external validation set helps to partially mitigate this effect, but does not eliminate it completely. Therefore, the results should be interpreted as initial evidence of the utility of these metrics, rather than as definitive proof of their generalization to other contexts. Furthermore, the corpus used consists exclusively of essays of the argumentative-dissertative genre in the ENEM style, which implies that the identified patterns may be partly associated with the specific characteristics of this textual genre.
Benchmarking with Commercial Tool and Domain Specialization
The comparison with the GPTZero tool was performed on the same set of 300 essays used in the external validation, allowing for a direct comparative analysis between the models. In general, the classifiers developed in this study showed superior performance to the commercial tool, as indicated in Table 3. In terms of error pattern, it is observed that the proposed models predominantly committed false positives and, to a lesser extent, false negatives, maintaining balanced performance between classes. In contrast, GPTZero showed distinct behavior: it did not register false positives, but concentrated its errors on false negatives.
Table 3. Predictive performance and error analysis in comparative benchmarking
| Model | Global Accuracy | Precision | Revocation (Recall) | F1-Score | False Positive (Human Error) | False Negative (AI/Humanized Error) |
|---|---|---|---|---|---|---|
| Logistic Regression | 0.9967 | 0.995 | 1 | 0.9975 | 1 | 0 |
| Random Forest | 0.9933 | 0.995 | 0.995 | 0.995 | 1 | 1 |
| XGBoost | 0.9933 | 0.995 | 0.995 | 0.995 | 1 | 1 |
| GPTZero | 0.9833 | 1 | 0.975 | 0.9873 | 0 | 5 |
Source: Original research results.
The analysis of these cases reveals a particularly relevant aspect: GPTZero’s errors are mostly associated with texts that underwent a humanization process via QuillBot. This indicates that the tool has greater difficulty in identifying as synthetic texts that have undergone alterations in lexical and structural surface through automated paraphrasing. In contrast, the models developed in this study proved to be more consistent in detecting this type of text, even after the humanization process. This result suggests that the use of stylometric features can contribute to capturing more global writing patterns, less sensitive to superficial modifications. Nevertheless, this comparison should be interpreted with caution. GPTZero is designed to operate in multiple textual domains, while the proposed model was developed in a specific context – dissertative-argumentative essays in the ENEM style. Furthermore, the evaluation was conducted on a single sample, which limits the generalization of the results.
Confusion Matrix Analysis
The analysis of confusion matrices was performed with the objective of examining not only the aggregated performance of the models, but also the nature of the errors committed, distinguishing between false positives and false negatives. In the context of this research, a false positive is defined as the case in which a human text is classified as synthetic, and a false negative is the case in which a synthetic text — including humanized texts — is classified as human. These two error categories have distinct implications: while false positives can lead to the undue classification of human productions as artificial, false negatives indicate failures in detecting texts generated by artificial intelligence.
In the external validation, a low incidence of errors was observed in the developed models, as can be seen in Figure 11. Logistic Regression presented a single false positive and no false negatives, while the Random Forest and XGBoost models presented one false positive and one false negative each. These results indicate that, in the analyzed set, the classifiers were able to maintain consistent performance in identifying both human and synthetic texts.
Figure 11. Confusion matrix of the models
Source: Original research results.
In contrast, the GPTZero tool presented a higher number of false negatives, totaling five occurrences. The analysis of these cases reveals a consistent pattern: all errors correspond to synthetic texts that were previously submitted to humanization processes through QuillBot and were classified as human by the tool. In the case of GPTZero, the recurrence of false negatives in humanized texts suggests that this type of transformation may impact the tool’s detection capability, at least within the analyzed set. This result does not allow for broad generalizations, but it points to the relevance of considering humanization scenarios in the evaluation of authorship detection systems. In general, the analysis of confusion matrices complements the aggregated metrics by evidencing that, despite the high overall performance, there are specific cases where the distinction between human and synthetic texts becomes less clear, especially when rewriting interventions are applied.
Parsimony and model interpretability
One of the findings of this study refers to the performance of Logistic Regression compared to more complex models. Despite being a linear classifier, with low computational cost and high interpretability, its performance was comparable and, in some cases, slightly superior to that of tree-based models. This result suggests that, for the problem investigated, a reduced set of stylometric features may be sufficient to capture relevant patterns for distinguishing between human and synthetic texts. In educational contexts, this characteristic is particularly important, as the transparency of automated decisions can be a relevant requirement. Nevertheless, this finding should be interpreted with caution. The performance of simple models depends directly on the quality of the features and the domain analyzed, making it impossible to generalize this observation to other contexts or data types.
The results obtained indicate that the approach based on stylometric features was effective in discriminating human texts from synthetic texts in the analyzed context. The use of structural linguistic metrics proved sufficient to capture relevant differences between writing patterns, allowing the construction of models with high predictive performance. Additionally, it was found that texts subjected to automatic humanization processes remain, to some extent, distinguishable from genuinely human texts when analyzed from the perspective of stylometric characteristics. This result reinforces the relevance of structural language analysis as an alternative to exclusive reliance on complex and poorly interpretable models. The choice of less complex supervised models also proved adequate for the problem, demonstrating that simpler approaches, when theoretically well-founded, can achieve competitive performance.
In this sense, the importance of model interpretability in the educational context stands out, especially in applications involving automated decision-making on textual productions. Despite the results obtained, some limitations must be considered, such as the specific domain of argumentative-dissertative texts from ENEM and the use of specific tools for generation and humanization, which restricts the generalization of the findings. Future investigations could explore the application of these same features in other domains to evaluate their robustness, as well as the expansion of the corpus to include different textual genres, proficiency levels, and generation and rewriting tools, in addition to investigating hybrid approaches that combine interpretable linguistic attributes with more complex representations. This work contributes by demonstrating that stylometric analysis constitutes a promising approach for authorship detection in texts, offering an interpretable and potentially applicable alternative in real educational contexts.
4. Conclusion
The present work aimed to develop a classification model capable of identifying texts generated by artificial intelligence, even after the application of humanization tools, focusing on argumentative-dissertative essays in the standard of the National High School Exam (ENEM). It was found that the approach based on stylometric features was effective in discriminating human texts from synthetic texts. Structural linguistic metrics, such as adjective density, lexical diversity (TTR), perplexity, and Burstiness, revealed distinctive patterns between human and artificial writing, even after rewriting by humanization tools. The trained machine learning models, including Logistic Regression, Random Forest, and XGBoost, demonstrated high predictive performance in detecting these texts, outperforming a commercial tool in humanization scenarios. The main contribution lies in demonstrating that stylometric analysis offers a robust and interpretable alternative for detecting textual authorship, essential for the educational context.
Despite the promising results, the study presented some limitations. The research was conducted in a specific domain of ENEM argumentative-dissertative essays, which restricts the generalization of findings to other textual genres. Furthermore, the generation of synthetic and humanized texts used specific tools (GPT-40 mini and QuillBot), not encompassing the diversity of existing models and humanization strategies. For future studies, it is suggested to expand the corpus to include different textual genres and proficiency levels, as well as to investigate other generation and rewriting tools. It is also recommended to explore hybrid approaches that combine interpretable linguistic attributes with more complex representations, in order to improve the robustness and applicability of the models in broader scenarios.
Bibliographic References
BROWN, T. et al. Language Models Are Few-Shot Learners. Advances in Neural Information Processing Systems, v. 33, p. 1877-1901, 2020. Disponível em: https://arxiv.org/abs/2005.14165. Acesso em: 21 set. 2025.
MASROUR, E.; EMI, B. N.; SPERO, M. DAMAGE: Detecting Adversarially Modified Al Generated Text. In: Proceedings of the 1st Workshop on GenAl Content Detection (GenAIDetect), Abu Dhabi, 2025. Disponível em: https://arxiv.org/abs/2501.03437. Acesso em: 21 set. 2025.
NILSSON, N. J. Artificial Intelligence: A New Synthesis. San Francisco: Morgan Kaufmann, 1998.
WEBER-WULFF, D. et al. Testing of Detection Tools for Al-Generated Text. International Journal for Educational Integrity, v. 19, art. 26, 2023. Disponível em: https://edintegrity.biomedcentral.com/articles/10.1007/s40979-023-00146-z. Acesso em: 22 set. 2025.
Article originating from the Course Conclusion Work of the Specialization in Data Science and Analytics of the MBA USP/Esalq
To learn more about the course, click here and access the MBX Academy platform