Reproducibility of LLM Inference under Load

Cláudio Rafael da Silva; Anaximandro Anderson Pereira Melo De Souza

DOI: 10.22167/2675-6528-202602788

Article derived from a Final Course Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

The incorporation of Large Language Models [LLMs] into corporate systems has amplified the need to understand their reliability, especially in contexts where automated responses needed to be reproduced, audited, and validated. The objective was to empirically analyze the reproducibility of LLM inference under different levels of concurrent load, investigating whether the increase in dynamic batching compromised the semantic stability of responses even with a temperature of zero, a fixed seed, and other controlled parameters. The research, of an applied, exploratory, descriptive, and quantitative nature, employed an empirical-experimental design. The experiment was conducted in a controlled computational environment, using GPUs, the “vLLM” inference engine, “bfloat16” precision, and scaled simulation of concurrent requests. 600 inferences were generated, distributed across 30 load tiers with 20 repetitions each, and the responses were evaluated for internal consistency and semantic similarity. The results indicated that response stability decreased with increasing concurrent load, showing that controlling generation hyperparameters did not ensure absolute determinism. The degradation manifested as subtle semantic variation between formally coherent responses, rather than explicit system failure. It was concluded that the reproducibility of LLMs at scale depends not only on the model and the prompt, but also on the physical and computational conditions of the inference infrastructure.

Keywords: “Dynamic Batching”; Software Reliability; Prompt Engineering; “GPU Computing”; Generative Artificial Intelligence.

1. Introduction

The growing incorporation of Large Language Models (LLMs) in organizational environments has transformed how digital systems produce, interpret, and automate textual information. Although these advances enhance operational efficiency, they also introduce technical challenges associated with the reliability, traceability, and predictability of generated responses. In artificial intelligence-based systems, especially when integrated into critical processes, linguistic coherence is not enough; it is necessary for the model’s behavior to be testable, auditable, and reproducible under controlled conditions (O’Brien et al., 2024; Yuan et al., 2025).

In current architectures, LLMs run on high-performance computational infrastructures, generally supported by graphics processing units (GPUs), tensor libraries, and optimized inference engines. This infrastructure allows processing large volumes of simultaneous requests, but also introduces physical and mathematical factors that can interfere with the logical behavior perceived by the user. Recent literature has shown that LLM inference depends not only on the prompt and developer-defined hyperparameters, but also on numerical precision, memory allocation, the order of matrix operations, and dynamic request batching mechanisms (Kalamkar et al., 2019; Kwon et al., 2023). Thus, the reliability of these systems needs to be analyzed in an integrated manner, considering both the algorithmic layer and the infrastructure layer.

Under ideal conditions, a model configured with deterministic parameters, such as a temperature of zero and a fixed seed, is expected to produce the same response to the same prompt. This expectation is relevant for regression testing, system debugging, security filter validation, and automated response auditing. However, recent research has indicated that this stability is not fully guaranteed in real-world inference environments. Even when logical parameters remain constant, small numerical variations can occur during model execution and, under certain conditions, may be amplified throughout the decoding process, generating semantically distinct responses (O’Brien et al., 2024; Yuan et al., 2025). The technical explanation for this phenomenon is partly related to floating-point arithmetic, where numerical operations performed on computers, especially with reduced precision formats like bfloat16, do not perfectly adhere to abstract mathematical properties. In matrix calculations executed on GPUs, the order of sums and multiplications can vary depending on the load distribution across processing cores, leading to minor differences in intermediate results (Kalamkar et al., 2019).

Beyond numerical precision, memory management also plays a relevant role in the stability of inferences. Modern inference engines, such as vLLM, use optimization techniques to increase response throughput in scenarios with multiple simultaneous requests. Among these techniques, PagedAttention stands out, which reorganizes the use of the key-value cache to reduce memory waste and allow a greater number of concurrent connections (Kwon et al., 2023). Although these solutions are essential to enable the operation of LLMs at scale, they also modify the way calculations are distributed and executed on hardware, especially when the system uses dynamic batching to group requests of different sizes. The computational efficiency of models, as demonstrated by FlashAttention (Dao et al., 2022), depends heavily on how data is moved between the GPU’s memory levels, highlighting that LLM inference is a computational phenomenon conditioned by physical architecture, numerical precision, parallelism, and memory management.

The numerical instability observed in graphics accelerators is not a recent phenomenon, but its manifestation in LLM inference directly affects the end-user. In software engineering contexts, where LLMs produce program snippets, corrections, or infrastructure configurations, silent variation between executions ceases to be mere statistical noise and becomes a software supply chain risk, as distinct artifacts can be incorporated into repositories from identical requests. This risk is exacerbated by the fact that the differences tend to preserve syntactic validity, evading usual automated checks. Recent literature on the incompatibility between training and inference precision reinforces how the choice of numerical format directly influences the stability of the observed behavior (Li et al., 2024; Yuan et al., 2025). The justification for this work stemmed from the gap between the practical expectation of determinism and the behavior effectively observed in production systems, where seemingly small variations can compromise automated tests, audits, security validations, and decision-making processes based on LLM responses, without manifesting as an obvious error, but as a subtle semantic alteration.

The operational premise widely assumed in professional practice, according to which setting the temperature to zero would be sufficient to ensure identical outputs, lacks formal guarantees from inference engines in production under dynamic request batching. Understanding and quantifying the load threshold from which this premise no longer holds and characterizing the corresponding failure mode is crucial for software engineering and artificial intelligence governance, as reproducibility at scale is a systemic property, dependent on the interaction between the model, inference engine, numerical precision, batching policy, and hardware infrastructure. In this context, the objective of this work was to empirically analyze the reproducibility of LLM inference under different levels of concurrent load, investigating whether increasing the size of dynamic batches in a GPU environment with bfloat16 precision produced semantic variations in the responses, even with temperature set to zero, a fixed seed, and other inference parameters controlled.

2. Material and Methods

The research was characterized, in terms of its nature, as applied, as it sought to understand a concrete technical problem related to the reproducibility of inference in Large Language Models (LLMs) under high computational load. The study was not limited to the theoretical discussion of non-determinism, but sought to produce empirical evidence to support practical decisions in software engineering, artificial intelligence governance, and validation of systems based on generative models (Gil, 2019; Vergara, 2016).

Regarding the objectives, the study assumed an exploratory and descriptive character. It was exploratory because it investigated a phenomenon still recent in the applied literature, related to the possibility of semantic variation in LLM inferences even when the logical generation parameters remained controlled. It was descriptive because it recorded, measured, and analyzed the behavior of the generated responses under different levels of concurrent load, allowing for the observation of patterns of semantic stability and degradation (Gil, 2019; Vergara, 2016).

The methodological approach adopted was quantitative, with qualitative interpretive support in the analysis of the generated responses. The quantitative dimension occurred through the measurement of internal consistency, semantic similarity, and response variation across different load levels. The qualitative dimension was used to interpret the meaning of the observed textual changes, especially when the responses remained formally coherent but presented a relevant change in meaning.

The methodological design corresponded to a controlled computational experiment, structured by a stepped load simulation. This design was chosen to directly manipulate the independent variable, represented by the size of the concurrent request batch, and observe its effects on the dependent variable, which was the degradation of semantic reproducibility. The data were obtained directly through experimental execution in a controlled computational environment, without involving human respondents or participant sampling.

The experiment was conducted in a cloud computing environment, using “Google Colab Pro” as the operational platform and an “NVIDIA A100” graphics processing unit (GPU) as the main accelerator. This infrastructure allowed for the simulation of high-load inference conditions and observation of the model’s behavior in a computational concurrency scenario. The inference engine used was “vLLM”, with the “Qwen2.5-7B-Instruct” model, executed in “bfloat16” precision.

The experimental setup reproduced a typical efficient LLM serving architecture, where memory optimization and dynamic request batching are employed to increase processing throughput. This choice was supported by studies discussing the effects of reduced precision, efficient memory management, and numerical sources of non-determinism in inference (Kalamkar et al., 2019; Kwon et al., 2023; O’Brien et al., 2024).

The experimental architecture was composed of several layers. In the physical layer, the 40 GB “NVIDIA A100” GPU was used, whose tensor cores performed matrix operations in reduced precision. The “CUDA” execution environment and the “PyTorch” library operated on top of it, responsible for memory allocation and dispatching execution cores.

The next layer corresponded to the “vLLM” inference engine (version 0.19.1), which employed three main mechanisms: “PagedAttention”, which fragments the key and value cache into fixed-size blocks; “chunked prefill”, which limits the amount of “tokens” processed in each scheduling step to 8,192; and asynchronous scaling, which overlaps the preparation of the next batch on the CPU with the execution of the current batch on the GPU (Kwon et al., 2023).

These mechanisms were relevant because the effective composition of each processed batch depended on the state of the scheduler at the instant of execution, and not just on the set of submitted requests. Above the engine, the “Qwen2.5-7B-Instruct” model was positioned, executed in “bfloat16” precision. In a separate layer, the telemetry instrumentation and the semantic evaluation subsystem operated, the latter based on the “all-MiniLM-L6-v2” model from the “Sentence Transformers” library.

The engine instantiation parameters included “bfloat16” precision, a context window limited to 1,024 “tokens”, a reserved video memory fraction of 0.8, single-process execution, and enabled execution graph capture. The model sampling parameters were configured with a temperature of zero, “Top-P” of one, a fixed seed of 42, and a generation limit of ten “tokens”.

The choice of “bfloat16” precision was a technical decision directly implicated in the phenomenon investigated. This format preserves the eight exponent bits of “fp32”, maintaining the same dynamic range of representation, but reduces the mantissa to seven explicit bits. In terms of relative resolution, “bfloat16” proved to be approximately eight times coarser than “fp16” and about sixty-five thousand times coarser than “fp32” (Kalamkar et al., 2019; Li et al., 2024).

This characteristic did not represent a fragility of the design, but a deliberate condition of observation, as the “bfloat16” format is standard in production environments. The adoption of this format preserved the ecological validity of the experiment and placed it in a configuration where numerical microvariations are more likely to become observable. The use of “fp32” would be unfeasible due to the memory budget available for the seven-billion-parameter model.

Control variables were defined to reduce external interferences and isolate the effect of concurrent load. The main generation hyperparameters, the model, the control prompt, the execution environment, and the numerical precision format were kept constant. The independent variable was the dynamic batch size of requests, varying in different load levels up to the limit of 250 concurrent requests.

The dependent variable was the loss of reproducibility, observed through the dispersion of responses and the semantic similarity between the outputs obtained. Data collection was carried out using a Python script developed specifically to automate the experiment. In each round, a control prompt was submitted to the model along with additional fill requests, intended to occupy the batch and increase effective concurrency on the inference engine.

The auxiliary requests were built from the same base text, replicated to compose the volume corresponding to each load level, with equivalent length between them. The generation limit was set at ten “tokens” for all requests. In total, 600 inferences were generated, distributed across 30 load levels, with 20 independent repetitions at each level.

The “notebook” used in the collection, containing the engine instantiation, the control “prompt”, the load scheduling loop, and the semantic evaluation routines, was made available in a public repository under an Apache 2.0 license and archived under a persistent digital identifier, allowing for full replication of the procedure (Silva, 2026).

To reduce contamination risks between executions, the allocator’s cache release routine was invoked at the end of each iteration. Continuous persistence of results was implemented, storing each generated response with its operational metadata, including batch size, iteration number, and execution timestamp. This procedure strengthened the experiment’s traceability and allowed for subsequent verification of the raw data.

The data analysis was conducted in two stages. In the first, the internal consistency of the responses at each load level was evaluated, checking if the model produced identical or semantically divergent responses. In the second stage, the responses were transformed into vector representations using the “all-MiniLM-L6-v2” model from the “SentenceTransformers” library, allowing the calculation of cosine similarity between the generated texts.

Cosine similarity was used because it is a widely applied metric in Natural Language Processing to compare semantic proximity between textual vectors. The interpretation of the results considered that the reproducibility break should not be analyzed solely as a literal difference between character strings, but also as a semantic alteration with potential operational impact.

The analysis combined the frequency of distinct responses, semantic similarity, and qualitative inspection of the main variations observed. This analytical triangulation allowed us to assess whether the instability generated under load represented only superficial variation in wording or an actual semantic alteration with potential impact on corporate systems.

In this way, the methodological path adopted allowed for empirically testing the hypothesis that the increase in concurrent load, mediated by dynamic “batching” and the reorganization of operations in GPU, could compromise the reproducibility of inferences even at zero temperature. The experimental structure proved coherent with the work’s objective, as it kept the logical generation parameters controlled and focused the analysis on the influence of computational infrastructure on the stability of the responses.

3. Results and Discussion

The analysis of the results was conducted from a base of 600 inferences, generated in 30 distinct concurrent load levels, with 20 independent repetitions for each level. The study focused on four main dimensions: the textual stability of the responses, semantic similarity evaluated by cosine, average inference latency, and video memory (VRAM) usage. The central objective was to verify if the deterministic inference logical configuration, which included a temperature of zero and a fixed seed, would be sufficient to guarantee reproducibility under different load levels. The findings indicated that initial stability was maintained only in the first few levels, while increased concurrency raised the probability of semantic divergences. It is important to note that the dimension related to video memory usage did not prove informative due to the measurement method employed, so the conclusions were based on the first three dimensions.

The consolidated data revealed that the first six load tiers, corresponding to an interval of 1 to 43 simultaneous requests, presented 100% average similarity and produced a single textual response. This period was characterized as initial stability, where the model’s behavior was fully deterministic. However, from 52 requests onwards, the first consistency rupture was observed, with 5% divergence in the initial tier of this new load range, and an abrupt increase to 45% at 61 requests. This transition did not follow a simple linear pattern, indicating a specific sensitivity of the system to concurrent load.

Between 86 and 112 requests, the system demonstrated a temporary readjustment, returning to a stable state where no divergence was recorded. However, instability became persistent from 121 requests onwards, with the average proportion of divergences reaching 30.9% and peaking at 50% with a maximum of 250 simultaneous requests. This dynamic suggests that the loss of reproducibility did not stem from a single deterministic failure, but rather from a zone of instability sensitive to how the dynamic batch was organized by the inference engine, as discussed by Kwon et al. (2023).

The evolution of the rate of semantic divergences as a function of increasing concurrent load revealed a pattern of intermittent instability up to 112 requests, which became persistent after 121 requests. The absence of divergences at the initial levels confirmed that the model, under low load, reproduced the same output. However, the presence of divergent responses at intermediate and high loads indicated that reproducibility was compromised by the systemic pressure of concurrent processing. The Pearson correlation between load and the percentage of divergences was 0.741, a positive and high value, suggesting a strong association between the increase in simultaneous requests and the growth of semantic instability.

This finding is in line with the technical literature on inference in Large Language Models, which points out how the use of dynamic batching and paged attention can modify the operational ordering of GPU calculations. This is particularly relevant when there are low-precision tensors and scheduling redistributes requests across successive steps (Kwon et al., 2023; O’Brien et al., 2024). Divergence was operationalized as the deviation from the modal response of the plateau itself, i.e., the most frequent formulation among the twenty repetitions of each load level. This definition imposed a structural ceiling of 50% on the metric, as the modal response, by construction, represents at least half of the occurrences for each plateau. Thus, in plateaus where the rate approached this limit, the effective instability may have been higher than recorded, and the observed value should be interpreted as a conservative estimate.

The average inference latency showed a practically monotonic growth as a function of concurrent load, increasing from 0.165 seconds in the unit batch to 0.371 seconds at the level of 250 simultaneous requests. The correlation between load and average latency was extremely high, with a Pearson coefficient of 0.996. This analysis was crucial because latency served as an operational indicator of the pressure exerted on the inference engine, demonstrating that the experimental scaling effectively imposed greater operational pressure on the system. However, latency, by itself, did not represent the direct cause of semantic divergence, but rather a symptom of computational overload.

Regarding video memory usage, telemetry recorded a practically constant value, around 32.58 GB, throughout the entire experiment. However, this result did not allow for a substantive interpretation, as the inference engine pre-allocates the memory set for the key-value cache, according to the configured fraction of 0.8 of the device’s capacity. The measurement procedure recorded the occupied memory at the instant of reading, not the effective peak of internal utilization. The observed constancy therefore stemmed from the allocation design itself, rather than the system’s behavior under load, which limits the ability to sustain or reject hypotheses related to memory usage.

The textual analysis of the 600 generated responses reinforced the interpretation that the degradation did not produce random, incoherent, or syntactically flawed responses. Only two final formulations were observed: “Digital innovation for corporate processes.”, with 445 occurrences, and “Digital innovation for transformed businesses”, with 155 occurrences. The system oscillated between these two plausible and well-formed outputs, a phenomenon called semantic bistability. The cosine similarity was 100% for the response identical to the reference and 76.15% for the divergent formulation, indicating considerable semantic proximity, but with a relevant change in meaning.

From a software engineering perspective, this result is particularly relevant, as a failure of this type may not be detected by superficial textual quality inspections or by conventional unit tests based on literal string comparison. The divergent response, while maintaining linguistic fluency, altered the semantic focus from “corporate processes” to “transformed businesses,” which can modify a business rule in applications that depend on reproducible outputs. The overall average similarity of 95.43% aggregated two distinct regimes: one fully reproducible and another with recurrent semantic divergence, evidencing that evaluations based solely on aggregated averages can hide localized reproducibility breaks.

The results allowed to clearly delimit what was observed and what remained in the realm of conjecture. It was observed that the increase in concurrent load was associated with the growth of semantic divergences, that latency monotonically accompanied this increase, and that degradation took the form of alternation between two well-formed outputs. The internal mechanisms responsible for this association were not directly observed, as they were not instrumented. The most compatible explanation with the documented configuration involves the variation in the effective composition of the processed batches, resulting from the fractionation of initial processing and asynchronous scaling.

These factors alter the order of reduction operations executed in parallel and, under the reduced mantissa of the “bfloat16” format, can accumulate sufficient microvariations to invert the ordering between close candidates in the decoding stage, which are amplified by autoregressive generation (Kalamkar et al., 2019; Kwon et al., 2023; O’Brien et al., 2024; Yashwanth, 2025). This is a theoretically grounded hypothesis and compatible with the literature, but not a mechanism directly demonstrated by this study, which did not include ablation trials or instrumentation at the execution core level. For Software Engineering and Artificial Intelligence governance, the observed behavior imposes the practical consequence that the execution infrastructure cannot be treated as a layer transparent to the model’s logical behavior, especially in critical enterprise architectures.

In summary, the research empirically demonstrated that setting logical generation hyperparameters, such as zero temperature and a fixed seed, does not guarantee the reproducibility of LLM inference under concurrent load. Stability was observed only at low load levels, while increased concurrency led to subtle but significant semantic variations that can impact the reliability of enterprise systems. The degradation manifested as semantic bistability, alternating between formally coherent responses but with distinct operational meanings. This finding reinforces that reproducibility at scale is a systemic property, dependent on the complex interaction between the model, the inference engine, numerical precision, the batching policy, and the hardware infrastructure.

4. Conclusion

The present study aimed to empirically analyze the reproducibility of Large Language Model (LLM) inference under different levels of concurrent load, investigating the impact of increased dynamic batching on the semantic stability of responses, even with controlled generation hyperparameters. It was found that response stability was maintained only at low load levels. With increased concurrency, a degradation in reproducibility was observed, manifested by subtle semantic variations between formally coherent responses, a phenomenon termed semantic bistability. Average inference latency grew monotonically with load, indicating operational pressure on the system. The main contribution of this work lies in demonstrating that fixing logical generation parameters, such as zero temperature and a fixed seed, does not ensure absolute determinism in LLM inference environments under load. This finding is crucial for software engineering and artificial intelligence governance, as it reveals that reproducibility at scale is a systemic property, dependent on the complex interaction between the model, the inference engine, numerical precision, the batching policy, and the hardware infrastructure, and not solely on the application code or the model in isolation.

However, the design presented limitations, such as video memory telemetry, which did not prove informative due to the measurement method, and the absence of ablation tests, which prevented isolating the individual contribution of each inference engine mechanism. The divergence metric, defined in relation to the modal response, imposed a structural ceiling of 50%, suggesting that the observed values may be conservative estimates. Furthermore, the experimental environment logging did not preserve all software versions, and the random selection of auxiliary request lengths did not use a fixed seed, affecting the experiment’s full reproducibility. The study was also restricted to a single model, accelerator architecture, numerical format, and prompt. For future studies, it is recommended to conduct ablation tests with execution core-level instrumentation and replication under other quantization schemes and silicon architectures. As long as the industry does not provide inference cores strictly invariant to batch composition, mitigating this risk will require development teams to adopt semantic redundancy layers and continuous auditing in client applications.

Bibliographic References

DAO, T.; FU, D. Y.; ERMON, S.; RUDRA, A.; RÉ, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv preprint arXiv:2205.14135, 2022.

GIL, A. C. Métodos e Técnicas de Pesquisa Social. 7. ed. São Paulo: Atlas, 2019.

KALAMKAR, D.; MUDIGERE, D.; MELLEMPUDI, N.; GEORGANAS, E. et al. A Study of BFLOAT16 for Deep Learning Training. arXiv preprint arXiv:1905.12322, 2019.

KWON, W.; LI, Z.; SHENG, Y.; YU, C.H.; GONZALEZ, J.E. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv preprint arXiv:2309.06180, 2023.

LI, B.; LIN, J.; YANG, P.; YAO, J.; ZHANG, Y.; ZHOU, J. Defeating the Training-Inference Mismatch via FP16. arXiv preprint arXiv:2407.13527, 2024.

O’BRIEN, P. D.; SRIDHAR, R.; ANASTASOPOULOS, A.; KUMAR, V. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. arXiv preprint arXiv:2407.03112, 2024.

SILVA, C. R. nondeterminism_llm_inference: rotina de coleta e análise de dados do experimento de reprodutibilidade da inferência de LLMs sob carga. Versão v1.0-tcc. Zenodo, 2026. DOI: 10.5281/zenodo.22740542. Disponível em: https://doi.org/10.5281/zenodo.22740542. Acesso em: 14 set. 2026.

VERGARA, S. C. Projetos e Relatórios de Pesquisa em Administração. 16. ed. São Paulo: Atlas, 2016.

YASHWANTH, T. S. On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication. arXiv preprint arXiv:2511.00025, 2025.

YUAN, J.; LI, H.; DING, X.; XIE, W.; LI, Y. J. et al. Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning. arXiv preprint arXiv:2506.09501, 2025.

Article originating from the Final Course Work of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform