Why the Same Prompt Succeeded This Time but Failed Last Time: Non-Determinism in LLM Inference and Engineering Countermeasures

In production environments, the most tormenting bugs are often not error messages, but "intermittent reliability": the same prompt sent five times yields two correct answers and three incorrect ones. This is not metaphysics; it stems from several quantifiable mechanisms within LLM inference.

Temperature and Top-p: Where Does Randomness Come From?

Sampling parameters directly determine the randomness of the output. Setting `temperature=0` essentially turns argmax into a deterministic selection path. However, due to floating-point rounding, changes in batch size, or different hardware backends, subtle differences in underlying logits can still cause output drift. Top-p truncation allows the "second-best option" to rise to the top; when the probabilities of the first and second choices are very close, a single change in sampling can flip the result. The first engineering countermeasure is to include `temperature` and `top_p` in configuration files under version control, rather than scattering them throughout invocation code. Before attempting to reproduce an issue, first confirm that "the parameters for these two requests were indeed identical."

Looking deeper, it is the difference in logits before softmax that determines the probability of a flip. Suppose the top choice has a 51% probability and the second has 49%; regardless of temperature adjustments, flipping between samples is a common occurrence. Conversely, if the top choice is 95% and the second is 5%, the result is largely stable. In other words, "the same prompt sometimes works and sometimes fails" often indicates that the task lies at the boundary of the model's capabilities. The prompt itself needs stronger constraints (examples, format, criteria), rather than simply tuning sampling parameters. Tuning parameters treats the symptoms; redefining the task addresses the root cause.

Perturbations Introduced by Batching and Hardware

In inference engines, continuous batching combines requests from the same or different sources into a single batch, altering the order of floating-point accumulation on the GPU. Consequently, the same model, on the same machine, with the same prompt, may produce different outputs at different times. These differences are usually slight variations in wording, but if downstream processes rely on regex or string matching to parse the output, minor phrasing changes can lead to parsing failures. The countermeasure involves two layers: do not rely on "exactly this word" in the parsing layer; instead, use structured outputs (JSON schema, function calling) with validation and retries. If strict reproducibility is required for business logic, you must either fix batch behavior (e.g., exclusive batches) or design fault tolerance based on the premise that "output will vary."

Inconsistent Cache Hit Paths

Acceleration mechanisms such as KV cache reuse, prefix caching, and speculative decoding cause "the same input to follow different computational paths." While most implementations guarantee numerically approximate consistency, approximation does not equal equality. A common pitfall during A/B testing or offline evaluation is that evaluations run on a cold-start path without cache, while production runs on a hot-cache path, leading to different result distributions and biased conclusions. The minimum viable approach is to use the same inference configuration for both evaluation and production environments and to note the cache status in reports.

Retries Are Not a Panacea

"Retry three times if it fails" is the lowest-cost yet most dangerous countermeasure. It transforms sporadic errors into cost volatility: with a 5% error rate, retrying three times raises the single-request success rate to 98.75%, but the average cost increases to nearly 1.15x, and each retry incurs the full token cost of the prompt again. A more robust approach is to add "memory" to retries: include the previous failed output in the prompt for the second retry ("The previous answer was wrong because of X, please correct it"). This type of self-correction is effective for tasks with clear criteria but offers limited benefits for open-ended tasks. The clearer the criteria (executable tests, schema validation, keyword checks), the more worthwhile the retry strategy becomes.

An Actionable Checklist

1. Fix and version-control `temperature`, `top_p`, and `seed` (if supported by the backend);

2. Use structured output + validation in the parsing layer, avoiding reliance on string matching;

3. Ensure evaluation and production environments share the same configuration, and report cache status;

4. Include failure context in retries and set a limit (recommended: 2 retries);

5. Keep complete request/response logs for "intermittently reliable" requests; reproduce first, then optimize.

Non-determinism cannot be eliminated, but it can be caged: allow it to fluctuate in wording, but not in correctness.