February 09, 2026
Comparative evaluation of RAG architectures for hallucination mitigation in LLMs
Ighor Felipe de Melo Queiroz; Adriana Camargo de Brito
Summary prepared by the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege focused on synthesis and writing.
This research comparatively evaluated the performance of three Generative Augmented Retrieval (RAG) architectures — Traditional RAG, Tree-RAG, and MultiQuery RAG — with the LangChain framework, to identify the most effective approach in reducing hallucinations and increasing the accuracy of Large Language Models (LLMs) on textual and complex knowledge bases. The study sought to fill a gap in the literature, which lacks systematic comparisons of the performance of different RAG architectures on a technical corpus with strong contextual dependence, using a combination of quantitative and qualitative metrics.
Despite the proliferation of Artificial Intelligence (AI) tools (Sichman, 2021), a critical limitation of LLMs is “hallucination”, the generation of factually incorrect information. This phenomenon occurs because models generate text based on statistical patterns, without access to external sources for real-time verification (Rapp et al., 2025). The plausibility of these responses compromises the reliability of the technology in critical applications (Sun et al., 2025).
To mitigate this risk, the Retrieval-Augmented Generation (RAG) technique, introduced by Lewis et al. (2020), integrates an information retrieval mechanism into the generation process. Before responding, the RAG system consults an external and specific database, providing the most relevant excerpts to the LLM as additional context. The model is instructed to base its response exclusively on this information, which increases factual accuracy and ensures that outputs are auditable and evidence-based (Arslan et al., 2024; Zhao et al., 2024).
The versatility of RAG has driven variations for specific domains. In healthcare, Ye (2024) applied RAG to electronic records with learning-to-rank, while Ozmen & Mathur (2025) highlight its potential for clinical decision support. In mental health, the combination of RAG with human feedback reduced hallucinations (Soman et al., 2025a). The technique was also applied in engineering, such as in Failure Mode and Effects Analysis (FMEA), using knowledge graphs to improve data retrieval (Bahr et al., 2025).
More sophisticated architectures have emerged to optimize retrieval. Tree-RAG (Fatehkia et al., 2024) uses the hierarchical structure of documents to generate contextual descriptions, achieving high precision in factual queries. DMQR-RAG (Li et al., 2024) employs a multi-query rewriting strategy to broaden search coverage. Despite these advances, the literature lacks a comparative analysis of these architectures on the same technical corpus. This study performs this evaluation using the Dungeons & Dragons 5th Edition Player’s Handbook as the corpus, chosen for its normative complexity, conceptual interdependence, and formal structure, which challenge AI systems.
The methodology ensured a rigorous comparison by keeping the processing pipeline constant, varying only the retrieval strategy. The corpus was the Dungeons & Dragons 5th Edition Player’s Handbook, a 315-page manual with approximately 212,919 words, chosen for its structure, informational density, and contextual dependency. The content was extracted from the PDF using the PyMuPDF library, handling column formatting, and the hierarchical structure was extracted with regular expressions and stored in JSON.
After extraction, the text was segmented into smaller units, or chunks, using the RecursiveCharacterTextSplitter method, with a maximum length of 850 characters and an overlap of 150 characters. This configuration balances information granularity and the preservation of semantic context. Subsequently, the chunks were converted into vector representations (embeddings) using the all-MiniLM-L6-v2 model from the Sentence Transformers library. The resulting vectors were indexed in the FAISS (Facebook AI Similarity Search) library, optimized for high-dimensional similarity searches, enabling efficient information retrieval.
The pipeline was orchestrated by the LangChain framework, using the openai/gpt-oss-120b language model, accessed via Groq infrastructure, for response generation. For each query, the system retrieved the 10 most semantically similar chunks from the FAISS index. These were inserted into a prompt that instructed the LLM to generate a response based exclusively on the provided context, with the directive to answer “I don’t know based on the available information” if the answer could not be found. The three architectures differed in their use of context: Traditional RAG used direct concatenation of retrieved chunks; Tree-RAG explored the hierarchical structure of the document to perform progressive summarizations; and MultiQuery RAG generated multiple reformulations of the question for parallel searches.
The models’ performance was evaluated through a hybrid approach. Five questions of varied complexity (factual, conceptual, procedural, analytical, and exploratory) were developed, each with a reference answer. For quantitative evaluation, ROUGE-1, ROUGE-2, and ROUGE-L metrics (Lin, 2004) were used to measure lexical overlap, and BERTScore F1 (Mirtaheri et al., 2025) to assess semantic similarity. Qualitative evaluation was conducted in two stages: a human analysis assigning scores from 1 to 5 on a Likert scale for four criteria (fidelity, completeness, clarity, and relevance), followed by a cross-validation in which an external LLM (Gemini 2.5) performed the same evaluation to reduce individual biases.
The evaluation results revealed that the effectiveness of each architecture is linked to the nature of the query. For the factual question “What is the life data of a warrior?”, Tree-RAG was the only one to provide the correct and direct answer (“d10”), demonstrating the superiority of its hierarchical approach for locating specific information, which corroborates the observations of Fatehkia et al. (2024). In contrast, Traditional RAG and MultiQuery RAG failed, stating that the information was not available in the context.
In conceptual and procedural questions, such as “What are stress tests?” and “What are the steps to create a persona?”, Traditional RAG presented the most balanced performance. Its answers were consistently more complete and well-structured. For example, in the explanation about stress tests, Traditional RAG defined the concept and detailed the mechanics, receiving top marks in all qualitative criteria. Tree-RAG and MultiQuery RAG also provided correct answers, but they were less detailed, indicating that the direct concatenation of fragments from Traditional RAG is more effective for preserving semantic chaining.
The analysis of more complex questions, such as “How does advantage and disadvantage work?” (analytical) and “Explain how a wizard’s spell progression works” (exploratory), exposed the limitations of more sophisticated architectures. Tree-RAG failed in both, indicating that its hierarchical structure may fragment context when the answer requires integrating dispersed information. MultiQuery RAG also failed on the question about spell progression and was less detailed on the other. Traditional RAG, although also unable to fully explain spell progression, provided a partial and relevant answer, demonstrating greater robustness.
The qualitative analysis, both human and performed by the LLM Gemini, consolidated these observations. Traditional RAG obtained the highest global score in both evaluations. Tree-RAG stood out for factual accuracy, but received low scores in completeness and relevance for the questions it failed. MultiQuery RAG showed intermediate performance, but the human evaluation pointed out that the diversity of queries sometimes introduced noise and fragmented the context. Gemini’s evaluation confirmed the overall superiority of Traditional RAG.
The quantitative analysis reinforced the qualitative findings. Tree-RAG achieved the highest ROUGE-1 (0.421) and ROUGE-2 (0.235) scores on the factual question, reflecting its high lexical fidelity. On the other hand, Traditional RAG consistently obtained the highest BERTScore F1 values on conceptual and procedural questions (e.g., 0.838 on the advantage/disadvantage question), indicating better preservation of semantic similarity. MultiQuery RAG showed greater variability in scores, suggesting that its reformulation strategy may lead to context retrieval that is less focused on a structured corpus.
The discussion of the results, when confronted with the literature, offers insights. The superiority of Tree-RAG in factual queries aligns with the findings of Fatehkia et al. (2024), but this study also highlights its fragility in queries that demand the synthesis of dispersed information. The inconsistent performance of MultiQuery RAG contrasts with the accuracy gains reported by Li et al. (2024) in more diverse corpora, suggesting that question reformulation may introduce more noise than signal in a technical corpus with well-defined terminology.
This study successfully compared the Traditional RAG, Tree-RAG, and MultiQuery RAG architectures, evaluating their ability to mitigate hallucinations in a technical corpus. The results indicate that Traditional RAG offers the best overall balance between fidelity, clarity, and completeness, being particularly effective in conceptual and procedural responses. Tree-RAG demonstrated superior accuracy in factual and objective queries. MultiQuery RAG, in turn, presented greater variability and introduced noise into the retrieved context, which reduced response consistency in this scenario. The conclusions reinforce that the ideal RAG architecture is application-dependent. For systems dealing with direct fact retrieval, Tree-RAG is the indicated choice. For applications requiring detailed explanations, Traditional RAG proves more robust. The observed limitations, such as difficulty in processing tabular content, point to the need for developing hybrid approaches. It is concluded that the objective was achieved: it was demonstrated that the effectiveness of each RAG architecture intrinsically depends on the nature of the query, with Traditional RAG presenting the best overall balance, Tree-RAG excelling in factual accuracy, and MultiQuery RAG showing limited potential in highly structured corpora. For future work, it is recommended to expand the analysis to unstructured corpora, explore strategies that integrate tabular and textual data, and evaluate the feasibility of implementing these architectures in corporate contexts.
References:
Arslan, M., Ghanem, H., Munawar, S., & Cruz, C. (2024). A Survey on RAG with LLMs. Procedia Computer Science, 246(C), 3781–3790. https://doi. org/10.1016/j. procs.2024.09.178
Bahr, L., Wehner, C., Wewerka, J., Bittencourt, J., Schmid, U., & Daub, R. (2025). Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysis. Journal of Industrial Information Integration, 45, 100807. https://doi. org/10.1016/J. JII.2025.100807
Fatehkia, M., Lucas, J. K., & Chawla, S. (2024). TREE-RAG: Lessons from the LLM Trenches. http://arxiv. org/abs/2402.19473
Heredia Álvaro, J. A., & Barreda, J. G. (2025). An advanced retrieval-augmented generation system for manufacturing quality control. Advanced Engineering Informatics, 64, 103007. https://doi. org/10.1016/J. AEI.2024.103007
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. https://github. com/huggingface/transformers/blob/master/
Li, Z., Wang, J., Jiang, Z., Mao, H., Chen, Z., Du, J., Zhang, Y., Zhang, F., Zhang, D., & Liu, Y. (2024). DMQR-RAG: Diverse Multi-Query Rewriting for RAG. http://arxiv. org/abs/2411.13154
Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries.
Mirtaheri, S. L., Pugliese, A., Movahed, N., & Shahbazian, R. (2025). A comparative analysis on using GPT and BERT for automated vulnerability scoring. Intelligent Systems with Applications, 26, 200515. https://doi. org/10.1016/J. ISWA.2025.200515
Ozmen, B. B., & Mathur, P. (2025). Evidence-based artificial intelligence: Implementing retrieval-augmented generation models to enhance clinical decision support in plastic surgery. Journal of Plastic, Reconstructive & Aesthetic Surgery, 104, 414–416. https://doi. org/10.1016/J. BJPS.2025.03.053
Rapp, A., Di Lodovico, C., & Di Caro, L. (2025). How do people react to ChatGPT’s unpredictable behavior? Anthropomorphism, uncanniness, and fear of AI: A qualitative study on individuals’ perceptions and understandings of LLMs’ nonsensical hallucinations. International Journal of Human-Computer Studies, 198, 103471. https://doi. org/10.1016/J. IJHCS.2025.103471
Shang, Y., Ke, Z., Lin, P., Ren, Q., Zhang, W., Wang, X., Li, X., Gong, F., Wang, S., Wang, B., Xu, Z., Sun, M., & Tan, S. (2025). Empowering knowledge graphs with hybrid retrieval-augmented generation for the intelligent mix scheme of mass concrete. Case Studies in Construction Materials, 23, e04979. https://doi. org/10.1016/J. CSCM.2025. E04979
Sichman, J. S. (2021). Artificial Intelligence and society: advances and risks. Estudos Avancados, 35(101), 37–49. https://doi. org/10.1590/s0103-4014.2021.35101.004
Soman, G., Judy, M. V., & Abou, A. M. (2025a). Human guided empathetic AI agent for mental health support leveraging reinforcement learning-enhanced retrieval-augmented generation. Cognitive Systems Research, 90, 101337. https://doi. org/10.1016/J. COGSYS.2025.101337
Sun, S., Lin, Z., & Wu, X. (2025). Hallucinations of large multimodal models: Problem and countermeasures. Information Fusion, 118, 102970. https://doi. org/10.1016/J. INFFUS.2025.102970
Vidivelli, S., Ramachandran, M., & Dharunbalaji, A. (2024). Efficiency-Driven Custom Chatbot Development: Unleashing LangChain, RAG, and Performance-Optimized LLM Fusion. Computers, Materials and Continua, 80(2), 2423–2442. https://doi. org/10.32604/CMC.2024.054360
Ye, C. (2024). Exploring a learning-to-rank approach to enhance the Retrieval Augmented Generation (RAG)-based electronic medical records search engines. Informatics and Health, 1(2), 93–99. https://doi. org/10.1016/J. INFOH.2024.07.001
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024). Retrieval-Augmented Generation for AI-Generated Content: A Survey. http://arxiv. org/abs/2402.19473
Executive summary from the Final Course Work of the Specialization in Data Science and Analytics of the MBA USP/Esalq
Learn more about the course; click here:
