Temperature

If you set temperature to 0, do you get the same answer every time?

A dial on a probability distribution.

Ask a language model for the next word and it does not hand you a word. It hands you a score for every token in its vocabulary, plus a rule for turning those scores into a draw.

Temperature is that rule’s dial. It divides the scores before they become probabilities. Turn it down and the distribution sharpens: likely tokens get likelier, the tail fades. Turn it up and the distribution flattens: unlikely tokens become plausible.

The two values everyone argues about:

• T = 0: the model always takes the highest-scoring token. There is no draw at all. This is greedy decoding, and most APIs special-case 0 exactly this way.
• T = 1: the model samples from the distribution it actually learned. This is not maximum randomness. A common misconception says that 1 is “wild mode”, when it is really the trained distribution, which is why it is the default almost everywhere. Providers differ on the ceiling: OpenAI and Google allow up to 2, Anthropic stops at 1.

At T = 0 every run returns “three”. Switch on a little arithmetic noise and the same prompt starts alternating between “three” and “four”. That is the short answer to can it be absolutely deterministic? Temperature removes one kind of randomness, not all of it.

Divide the logits, not the probabilities.

The mechanism is one division. For logits z and temperature T, the probability of token i is softmax applied to the scaled logits:

p_i(T) = exp(z_i / T) / Σ_j exp(z_j / T)

Two limits matter. As T → 0, the largest logit takes everything: the distribution becomes a one-hot vector at the argmax. Greedy decoding is that limit case, not a separate algorithm. As T → ∞, all tokens become equally likely, which is uniform noise.

The quietest property is the important one. For any T > 0, dividing by T is monotonic, so temperature preserves the order of the tokens. It can never change which token is most likely, only how often the sampler takes it.

And it cannot separate a tie. The ratio between the top two candidates is:

p₁ / p₂ = exp((z₁ − z₂) / T)

In the lab the gap is 0.005. At T = 0.1 the ratio is about 1.05, a coin flip at a temperature most people would call deterministic. A near-tie is a property of the weights and the input, and no temperature removes it.

Where the dial is turned.

Look at that pipeline and ask which parts temperature controls. Only step 3. Everything else is still in the room when you set T = 0:

• Batch composition. The same prompt processed in a different batch can produce different logits, because GPU kernels reduce in different orders depending on shape. vLLM’s own docs say the engine “does not guarantee the reproducibility of the results by default, for the sake of performance”. Batch invariance is an opt-in beta flag, and even then reproducibility holds only “on the same hardware and the same vLLM version”.
• Hardware and version drift. New GPU, new driver, new kernel, new serving stack, and slightly different arithmetic. OpenAI’s docs are blunt: seed is a “best effort”, “determinism is not guaranteed”, and the system_fingerprint changes when they update the numerical configuration, a few times a year.
• Mixture-of-experts routing. Tiny numeric differences can change which experts fire, and that changes the output far more than the last decimal.
• Distributed inference. Tensor parallelism adds cross-GPU reductions; custom all-reduce is fast and not order-stable.
• Near-ties. The one your model produces every day: if the top two logits are 0.005 apart, the noise from any of the above is bigger than the gap.

The determinism checklist.

• T = 0 removes sampling randomness, not arithmetic randomness. It guarantees “always the top token” only if the logits are bit-identical, which the stack does not promise.
• Seeds don’t fix it. A seed makes the sampler repeatable; it cannot make the forward pass repeatable. Two different logit vectors, one seed, two different answers.
• Test your own stack. Run the same prompt 20 times, hash the outputs, count the distinct ones. That number, your actual determinism rate, is the only one that matters, and it varies by model, provider and week.
• Pin what you can. Model version, provider, parameters; log the fingerprint where one is offered; re-run the determinism check after every upgrade.
• If you need bit-exact reproducibility, own the whole stack. Self-hosted engines can get close (deterministic kernels, batch invariance, frozen hardware, frozen version) at a measurable performance cost. Even then the guarantee is “same hardware, same version”, not “same request, same answer”.
• Design for reproducible-enough. Most systems don’t need bit-exact; they need trustworthy: schema validation, retries, caching keyed on prompt + parameters, and evals that measure variance instead of pretending it away.
• Temperature 0 is still the right default for extraction, classification and anything with one correct answer. Just don’t confuse greedy with deterministic.

Determinism is a property of the whole stack: weights, kernels, hardware, version, scheduler, sampler. Temperature is the only part of it you can change with a number.