01 logo

Why Your AI Still Gives Different Answers at Temperature Zero

Floating-point math, thread scheduling, and batch size explain the reproducibility problem nobody warned you about.

By JinPublished 20 days ago • 9 min read

A common intuition says that if you send the same prompt to the same large language model repeatedly, you should get the same answer, especially when temperature is set to zero. Temperature zero turns sampling into greedy decoding. At every step, the model picks the token with the highest logit. No randomness. No dice rolls. No variation.

In practice, the answer is often no. Even with temperature set to zero, the same prompt can produce different outputs across repeated requests. The usual explanation is concurrency plus floating-point precision. That explanation is on the right track, but it is incomplete. Temperature zero removes randomness in the sampling stage. It does not remove randomness in the computation that produces the logits. Once a tiny difference appears in the logits, autoregressive decoding can amplify it step by step until two runs diverge.

This article explains three things. First, what temperature zero actually rules out and what it leaves untouched. Second, how floating-point non-associativity combines with nondeterministic reduction order to create different outputs from identical inputs. Third, why real inference services have an even more subtle source of nondeterminism: the lack of batch invariance.

Temperature Zero Only Removes Sampling Randomness

Token generation in an LLM has two stages. In the first stage, the model runs a forward pass and produces a vector of logits for the next token. In the second stage, a decoding rule selects a token from those logits.

Parameters such as temperature, top-k, and top-p affect the second stage. When temperature is set to zero, the second stage becomes deterministic argmax. The largest logit wins. Sampling randomness is gone.

But the first stage is still floating-point computation. Matrix multiplications, RMSNorm, attention, and softmax are all floating-point operations. Their results depend on the order of additions, the reduction strategy, the kernel implementation, the batch size, and thread scheduling. Temperature does not control any of those.

So the real question is not whether sampling is random. The real question is whether two requests with the same prompt produce bitwise-identical logits. If they do not, even when the difference sits in the least significant bit, argmax can flip at some decoding step. Each generated token becomes part of the context for the next step. A single flipped token can send the entire trajectory down a different path.

That is why the phenomenon is not merely “the model is stochastic.” It is a system-level reproducibility problem.

The First Layer: Floating-Point Addition Is Not Associative

IEEE-754 floating-point numbers use a finite number of binary bits to represent real numbers. Most real numbers cannot be represented exactly. Every operation can introduce rounding error. The key point is that floating-point addition does not satisfy the associative law: (a+b)+c does not equal a+(b+c).

This is not mathematical addition. It is computer addition. A classic example is adding 0.1 to 10^20 and then subtracting 10^20. If you add 0.1 first and then subtract the large number, the 0.1 is swallowed and the result is zero. Change the grouping and the result changes.

Thinking Machines Lab calls this the “original sin” of floating-point computation in its article “Defeating Nondeterminism in LLM Inference.” The behavior is counterintuitive, but it follows from finite precision. Rounding error exists. Different orders of operations can produce different results.

The “original sin” is only a necessary condition. It does not by itself answer why floating-point values would be added in different orders, when that happens, or how to avoid it. To answer those questions, we need to look at the execution layer.

The Second Layer: Nondeterministic Reduction Order Under Concurrency

A large fraction of LLM inference is reduction. Matrix multiplication computes dot products. Multiple threads compute partial sums. Multiple thread blocks accumulate partial results on a GPU. Reduction means adding many numbers together. If the addition order is fixed, the result can be bitwise deterministic. If the addition order depends on thread scheduling, the result can differ from run to run.

A precise example comes from debugging PowerInfer, which is based on llama.cpp. Even with temperature set to zero, which degenerates to deterministic greedy sampling through argmax over logits, the same prompt still produced different text. By hashing layer outputs and diffing multiple runs, the investigation tracked the problem to a custom axpy operator.

The implementation worked roughly like this. Each thread repeatedly used atomic_fetch_add on a global atomic counter to claim an available chunk. For each row in that chunk, it computed an inner product and accumulated the result into its own private buffer, vy[]. That phase was lock-free. Then, under a lock, each thread added its local vy vector into the global result res.

The problem was in that final step. Different threads acquired chunks and acquired the lock in an order determined by hardware execution. The N local vy vectors were reduced in a random order: res = 0 + (p[0] + p[1] + (p[2] + p[3])) + ... Because IEEE-754 single-precision floating-point addition is not associative, different orders produced different results. The difference might be tiny, but it was real, reproducible in principle, and determined by scheduling.

The tiny deviation from an early decoding step was amplified step by step. Each hidden state was built on a context that already contained a small numerical deviation. After many layers and many decoding steps, the accumulated error could flip the argmax at some step. Once that happened, the later trajectory became increasingly different.

The fix was straightforward. The spinlock was replaced with a round-robin lock that enforced a fixed order, and the order of chunk acquisition and reduction was made deterministic. After that, repeated runs of the operator produced identical outputs.

That experiment directly validates the diagnosis. The nondeterminism came from reduction order.

There was also an important footnote. Even after fixing that operator, the final response sometimes still changed. The suspicion was that layers offloaded to the GPU had similar behavior, but there was no time to investigate every other operator. That detail matters because nondeterminism can be distributed across multiple operators. Fixing one operator does not necessarily fix the whole pipeline.

The Same Problem Exists on GPUs

The PowerInfer example is a CPU multithreading case, but the same class of problem exists on GPUs. cuBLAS split-K GEMM kernels use atomicAdd to accumulate partial sums from different thread blocks into the output. Floating-point atomicAdd is not order-stable. The GPU scheduler determines which block completes its atomic addition first, and the accumulation order can differ from call to call. vLLM has specific fixes to disable such kernels because they can cause argmax flips even at temperature zero.

One experiment matters here. On a GPU, running the same matrix multiplication, such as torch.mm(A, B), a thousand times can produce bitwise-identical results. This shows that concurrent floating-point computation does not automatically imply nondeterminism. For the same kernel, same batch size, and same launch configuration, the internal reduction tree is fixed. The physical timing of concurrent execution does not necessarily change the logical order of additions.

The PowerInfer operator was nondeterministic because it explicitly made the reduction order depend on thread scheduling. Whichever thread acquired the lock first added first. A well-designed kernel with a fixed reduction tree can remain bitwise deterministic for the same shape.

The more accurate statement is this: concurrency alone does not necessarily cause nondeterminism. What matters is whether the reduction order depends on scheduling.

The Third Layer: The Lack of Batch Invariance

Even if every kernel is made bitwise deterministic for the same input and the same shape, real inference services have another, more subtle source of nondeterminism: the lack of batch invariance.

Thinking Machines Lab points out that kernels such as matrix multiplication, RMSNorm, and attention can have “run-to-run determinism.” The same input produces the same result across repeated runs. Those kernels can still lack “batch invariance.” When the batch size changes, the computation result for an individual batch element can change.

High-performance kernels may dynamically choose different algorithms depending on the current batch size. They may decide whether to use split-K, what tile shape to use, what reduction strategy to apply, and which kernel version to launch. When your request is packed with other requests into a batch of size 7, the kernel may take path A. The next time, when it is packed into a batch of size 31, the kernel may take path B. Path A and path B use different floating-point accumulation orders for the same sample, so the least significant bits of the result differ.

The inference server’s load, and therefore the batch size, is a random variable from the user’s perspective. During peak hours the batch size might be 128. During low-traffic periods it might be 8. The same request is placed into different batch sizes and therefore different computation paths. The output changes, even though the prompt and temperature are unchanged.

This problem is not limited to GPUs. It exists on CPUs and TPUs as well. The root cause is not a hardware-specific quirk. The root cause is that the computation graph shape changes with batch size.

Why “Same Shape, Repeated Runs Are Identical” Is Not a Contradiction

Putting these layers together explains a set of seemingly contradictory observations.

The same kernel, with the same batch size, may produce bitwise-identical results across repeated runs. The same prompt, under different server loads, may produce different outputs. Temperature zero makes sampling deterministic, but logits can still differ. Fixing one operator may still leave the final response variable, because other operators or GPU-offloaded layers may have similar issues.

Reproducibility is not a property of a single kernel. It is a property of the entire pipeline, from operator implementation to scheduling, from batch assembly to kernel selection.

One way to organize the sources is by layer. At the sampling layer, temperature, top-k, and top-p introduce randomness, and temperature zero removes it. At the floating-point layer, IEEE-754 rounding and non-associativity create least-significant-bit differences. Temperature zero does nothing about them. At the concurrent reduction layer, reduction order may depend on thread scheduling or atomic operations. That can produce different results from the same input across runs. Temperature zero does nothing about that either. At the batch layer, changes in batch size can change kernel paths. The same request under different loads can produce different outputs. Temperature zero again does nothing.

The apparent contradiction disappears once you see that these are different layers of the stack.

How to Get More Reproducible Output

If the goal is debugging, evaluation, or auditing rather than maximum throughput, there are two main directions.

The first is to fix the reduction order. In your own code paths, avoid making reduction order depend on lock contention or atomic operation timing. Use fixed reduction trees, round-robin locks, and deterministic reductions. On the GPU side, pay attention to split-K, atomicAdd reductions, and whether the framework provides determinism switches. Many determinism switches only guarantee reproducibility for the same shape and the same batch. They do not guarantee batch invariance.

The second is to eliminate the effect of batch size. If the goal is server-side reproducibility, fixing a single kernel’s reduction order is not enough. You need batch-invariant kernels, so that the computation path for one sample does not depend on how many other requests are in the same batch. Thinking Machines Lab proposed batch-invariant RMSNorm, batch-invariant matrix multiplication, and batch-invariant attention. vLLM also has a batch-invariant mode. In engineering practice, you can also fix the batch size, fix padding, or serve one request per batch. The cost is throughput.

A practical debugging sequence looks like this. First, determine whether the problem only appears on the server or whether it also reproduces locally with a single request. Second, fix the batch size and kernel configuration and see whether the output becomes stable. Third, hash or diff each layer’s output to locate which operator introduces the difference. Fourth, check whether that operator has nondeterministic reductions, atomicAdd, split-K, or dynamic kernel selection. Fifth, after fixing the operator, check the batch assembly and scheduling layers.

Conclusion: Reproducibility Is a System Property

The same prompt producing different outputs at temperature zero is not mysticism. It is not simply “the model is random.” It involves at least three layers.

Floating-point addition is not associative. That single fact creates the opening for everything else. Concurrency can make reduction order depend on thread scheduling. Batch size changes can alter kernel computation paths and therefore the order of floating-point work.

The PowerInfer debugging story shows that a tiny deviation can be amplified step by step until it changes the argmax. The Thinking Machines Lab article further shows that even if a single kernel is deterministic for the same shape, the lack of batch invariance can still make the same request produce different results under different loads.

Reproducibility is a system property. It requires determinism across the entire pipeline: floating-point reduction, concurrent scheduling, kernel selection, and batch assembly. Temperature zero only turns off one light in the room. Other lights are still on.

A concise way to put it is this. Temperature zero guarantees that the rule for choosing a token is deterministic. It does not guarantee that the logits themselves are bitwise identical. Autoregressive decoding is extremely good at turning a difference in the least significant bit into two completely different paths.

tech newsfact or fictionthought leaders

About the Creator

Jin

Writer of reamstories

https://reamstories.com/jin

Enjoyed the story? Support the Creator.

Subscribe for free to receive all their stories in your feed. You could also become a paid subscriber, letting them know you appreciate their work.

Subscribe For Free

Reader insights

Comments

There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment
    Written by Jin