Inference and sampling
If you submit the exact same prompt to a language model twice, you will often receive two distinct answers. This variability is not a glitch—it is an intentional feature introduced during generation by a configurable sampling layer outside the neural network itself.
In machine learning terminology, Inferencerunning a trained model to produce an output. The weights do not change and no learning happens.Full glossary entryIntroduced in Inference and sampling refers to running an existing, trained model to evaluate inputs and generate predictions. During inference, model weights remain entirely static: the neural network doesn't learn new information or adjust its parameters. It simply maps input numbers to output numbers.
What a single forward pass produces
When input tokens pass through a model's weights, a single forward pass calculates a raw numerical score—called a logit—for every single token in the tokenizer's vocabulary. If a model operates with a vocabulary of 128,000 tokens, it outputs 128,000 individual logits.
These raw logits are normalized through a softmax function to produce a Next-token distributionthe probability a model assigns to every token in its vocabulary as the next one, produced by applying softmax to the raw scores from one forward pass.Full glossary entryIntroduced in Inference and sampling: a complete list of 128,000 probabilities that sum to 1.0. For an incomplete sentence like "The capital of France is", the resulting distribution might look like this:
| Candidate | Probability |
|---|---|
Paris | 0.92 |
located | 0.03 |
a | 0.02 |
the | 0.01 |
| All other tokens combined | 0.02 |
The neural network's internal calculation concludes here. It has assigned a likelihood to every candidate in its vocabulary, but it hasn't actually selected which token will appear next.
Two phases: prefill and decode
Turning an initial prompt into a completed response involves two computationally distinct stages:
Prefillthe phase that processes the whole prompt in one parallel pass, before any output token exists. Its cost grows with prompt length, which is why a large prompt is slow to start.Full glossary entryIntroduced in Inference and sampling processes the incoming prompt all at once. Because all input tokens are known from the start, GPUs can process them concurrently across parallel cores, calculating attention matrices and populating the key-value (KV) cache. Prefill duration scales primarily with prompt length: a prompt containing 100,000 tokens will take noticeably longer to digest than a 50-token query.
Decodethe phase that generates output tokens one at a time, each one depending on the last. It is serial by construction and accounts for most of the wall-clock time of a long answer.Full glossary entryIntroduced in Inference and sampling generates the response sequentially, one token at a time. Each generation step evaluates the accumulated context, computes a new probability distribution, samples a token, appends that token to the context, and feeds the updated sequence into the next pass. Because token 200 cannot be determined until token 199 has been generated, decode is inherently serial. The vast majority of the wall-clock time spent waiting for a long response occurs during this decode phase.
This architectural difference explains user-perceived latency. Time to first tokenthe delay before the first output token reaches the caller, driven mostly by prompt length and queueing rather than by how long the answer will be.Full glossary entryIntroduced in Inference and sampling (TTFT) reflects prompt prefill plus network transmission and server queueing; it increases when you send larger prompts. Once generation begins, subsequent tokens appear at a steady pace determined primarily by hardware memory bandwidth and model size. In other words, a massive prompt makes a completion slow to start, while a lengthy response makes it slow to finish.
Streamingsending each output token to the caller as it is decoded, rather than holding the whole response. Total generation time is unchanged; the wait is just spent reading.Full glossary entryIntroduced in Inference and sampling bridges this gap for interactive applications. Rather than buffering the entire completion until generation completes, the serving infrastructure streams individual tokens over a connection as they decode. While streaming does not change the total generation time, it allows users to read output immediately, dramatically improving perceived responsiveness.
Where the randomness enters
The model outputs a probability distribution, but the downstream Samplingthe step outside the model that picks one token from the next-token distribution. It lives in the serving stack rather than in the weights, and it is where the visible randomness comes from.Full glossary entryIntroduced in Inference and sampling algorithm decides which token actually gets appended to the sequence. This sampling step runs outside the neural network within the model serving stack, and it is where output variability originates.
The simplest selection strategy is Greedy decodingalways taking the highest-probability token. It removes the sampler's randomness but not the provider's, so a hosted call is still not guaranteed to repeat.Full glossary entryIntroduced in Inference and sampling, which always chooses whichever token holds the highest probability. While predictable, greedy generation often leads to repetitive or unnatural phrasing. Most production applications use parameterized sampling to introduce controlled diversity.
Temperature
Temperaturea sampler setting that flattens or sharpens the whole next-token distribution before a token is picked. Below 1 the leading candidate pulls further ahead; above 1 the also-rans get a real share.Full glossary entryIntroduced in Inference and sampling scales logits before the softmax normalization occurs. Setting temperature below 1.0 sharpens the distribution, exaggerating the gaps between candidates so the highest-scoring token becomes overwhelmingly dominant. Raising temperature above 1.0 flattens the distribution, giving lower-ranked alternatives a higher likelihood of selection.
Consider how temperature alters candidate probabilities following the phrase "The report was":
| Candidate | Temperature 0.2 | Temperature 1.0 | Temperature 1.5 |
|---|---|---|---|
clear | 0.98 | 0.60 | 0.42 |
thorough | 0.02 | 0.25 | 0.27 |
late | 0.00 | 0.10 | 0.18 |
damning | 0.00 | 0.05 | 0.13 |
These numbers illustrate the underlying shift: at temperature 0.2, the model selects "clear" almost every time. At temperature 1.5, "damning" gets chosen roughly one in eight runs. Adjusting temperature does not make a model more capable or imaginative; it simply controls how frequently the algorithm selects lower-probability tokens.
Top-p (nucleus sampling)
Top-pa sampler setting, also called nucleus sampling, that keeps only the smallest set of candidates whose probabilities sum to p and discards the rest before picking.Full glossary entryIntroduced in Inference and sampling (or nucleus sampling) controls the selection pool by probability mass rather than mathematical scaling. The sampler sorts tokens by descending probability, calculates a cumulative sum, and cuts off the list as soon as the total reaches threshold $p$. Tokens falling outside this cumulative threshold are completely discarded, and the remaining subset is re-normalized.
For example, with top-p set to 0.9 on the temperature 1.0 column above, the sampler includes clear (0.60), thorough (0.85 cumulative), and late (0.95 cumulative, crossing the 0.9 mark). The tail candidate damning is excluded entirely.
Combining both parameters provides fine-grained control: temperature controls the relative flatness of the candidate distribution, while top-p truncates the long tail of implausible tokens. Raising temperature while holding top-p at 0.85 or 0.90 encourages stylistic variety without letting nonsensical or ungrammatical words slip through. Most providers recommend tuning one parameter while keeping the other at its default to avoid unpredictable interactions.
Sampler controls on newer models
Before designing custom user interfaces around temperature and top-p sliders, verify whether your target model supports them. Several modern reasoning models reject non-default temperature settings or lock sampling parameters entirely. Providers often find that deviating from default sampling values degrades performance on reasoning-tuned models, sometimes causing repetitive loops or broken logic. Check vendor-specific parameters before relying on legacy sampling knobs.
Why greedy decoding is not fully deterministic
If you set temperature to 0, the sampling algorithm becomes deterministic: it will always choose the single highest-probability token. However, sending the identical prompt to a commercial hosted API can still produce different outputs across calls.
This discrepancy occurs because the underlying logits themselves are rarely bit-identical across runs, due to several operational factors:
- Dynamic server-side batching: Cloud providers bundle concurrent user requests into shared matrix multiplications on GPUs to maximize throughput. Because floating-point addition is non-associative—
(a + b) + cdoes not strictly equala + (b + c)in hardware arithmetic—differing batch compositions alter rounding in the final decimal places. When two top candidates have nearly identical probabilities (such as 0.4999 vs 0.5001), minor rounding differences can flip the winning token, causing the entire subsequent response to diverge. - Heterogeneous hardware and kernels: Providers route traffic across clusters containing mixed accelerator generations and optimized compute kernels. Internal matrix reduction orders vary across hardware revisions, introducing slight numerical drift.
- Mixture-of-experts (MoE) routing: In MoE architectures where only a subset of specialized sub-networks activate per token, cross-request batching dynamics can subtly influence expert routing decisions.
- Model aliases and rolling updates: Pointing requests to generic aliases like
latestor unversioned model names means your calls resolve to whichever build the vendor is currently serving. Providers frequently roll out minor updates, optimizations, or safety patches without changing base model identifiers.
Reproducibility with random seeds
Some model APIs support an optional seed parameter to initialize the sampler's pseudo-random number generator. While setting a seed stabilizes random token selection, it cannot eliminate variability caused by server-side batching, kernel differences, or backend infrastructure updates. Most providers document seed support as best-effort rather than an absolute guarantee of bit-level determinism.
When running local models on dedicated hardware where you control batch size, precision, and GPU environment, setting a seed does produce strictly reproducible completions, as discussed in local and cloud inference.
Pin every part of the call that you can control. Accept that the output is still not guaranteed byte-identical.
- Name an exact model version. Do not call an alias such as
latest. - Set temperature to 0 if the model still accepts the parameter.
- Set the seed if the provider offers one.
- Record the model version and the fingerprint the provider returns with each response.
- Assert on facts in the output. Do not assert on the exact string.
- Run the assertion several times before you trust a pass.
- Re-run your evaluation set when the provider announces any model change.
Unit tests that compare model responses against static string snapshots are notoriously brittle, leading teams to dismiss test failures as noise. Instead, assert against semantic criteria: verify that output conforms to valid JSON, that mandatory schema fields exist, that extracted numerical values fall within acceptable tolerances, and that classifications match expectations.
Terms introduced
- Inference: running a trained model to produce an output. No weights change.
- Prefill: the phase that processes the whole prompt in parallel, before any output token exists.
- Decode: the phase that generates output tokens one at a time, each one depending on the last.
- Next-token distribution: the probability the model assigns to every token in its vocabulary as the next one.
- Sampling: the step outside the model that picks one token from that distribution.
- Temperature: a sampler setting that flattens or sharpens the distribution before a token is picked.
- Top-p: a sampler setting that keeps only the smallest set of candidates whose probabilities sum to p, and discards the rest.
- Greedy decoding: always taking the highest-probability token, which removes the sampler's randomness but not the provider's.
- Time to first token: the delay before the first output token arrives, driven mostly by prompt length and queueing.
- Streaming: sending each output token to the caller as it is decoded, rather than holding the whole response.
How providers do it
The direction of travel is the story here. All three vendors are taking the sampling knobs away on their newest models rather than adding more.
| Question | Anthropic | OpenAI | |
|---|---|---|---|
| Temperature range and default | 0.0 to 1.0, default 1.0 | 0 to 2, no default stated | 0.0 to 2.0, default 1.0 |
| Top-p | top_p | top_p | topP, default 0.95 |
| Top-k | top_k | Not offered | topK, default 40 |
| Still settable on the newest models? | No. Non-default values return a 400 | Not on reasoning models; migration says remove them | Migration says strip them from generation configs |
| Seed | Not offered | Yes, "best effort", with system_fingerprint | Yes, seed in generationConfig |
| Determinism guarantee | Explicitly disclaimed, first-party and third-party alike | "Determinism is not guaranteed" | Not confirmed on a page that would load |
| Streaming | "stream": true, server-sent events | stream=True, semantic events | streamGenerateContent, or stream=True |
Statuses differ per row and are marked in the tabs. Google's determinism hedge, OpenAI's seed support on the Responses API and OpenAI's per-model temperature rules are the open ones.
Two vendors ship a seed and neither promises it works. If a test needs the same bytes twice, it needs a model running on hardware you control, not an API.
- Anthropic
- OpenAI
What this maps to: the Messages API. The notable thing here is that Anthropic has removed the sampling knobs on its newest models rather than exposing more of them.
| Question | Answer | Status |
|---|---|---|
| Sampling parameters exposed | temperature, top_p and top_k, all three now marked deprecated in the API reference | confirmed |
| Documented range and default | temperature "defaults to 1.0. Ranges from 0.0 to 1.0". top_k samples only from the top K options | confirmed |
| Are they still accepted? | Not on current models. "Models released after Claude Opus 4.6 do not support setting temperature. A value of 1.0 will be accepted for backwards compatibility, all other values will be rejected with a 400 error." top_p below 0.99 and any top_k are rejected the same way | confirmed |
| Which models reject them | Fable 5.1, Mythos 5.1, Fable 5, Mythos 5, Mythos Preview, Opus 5, Opus 4.8, Opus 4.7 and Sonnet 5, "regardless of whether thinking is used" | confirmed |
| Guidance on combining temperature and top-p | None given; both are deprecated instead. On older models with thinking enabled, temperature and top_k are incompatible with thinking and top_p is allowed only between 0.95 and 1 | confirmed |
| Is a seed offered? | No. The Create a Message body has no seed field | confirmed |
| Determinism guarantee | Explicitly disclaimed. "Even with temperature set to 0, the results will not be fully deterministic and identical inputs may produce different outputs across API calls. This applies both to Anthropic's first-party inference service and to inference through third-party cloud providers" | confirmed |
| Streaming | Yes, "stream": true, over server-sent events. The sequence is message_start, then content_block_start, content_block_delta and content_block_stop per block, then message_delta and message_stop | confirmed |
| Delta types | text_delta, input_json_delta, thinking_delta and signature_delta. usage on message_delta is cumulative | confirmed |
| Errors mid-stream | Arrive as error events after a 200 has already been sent | confirmed |
| Pinning a build | Dated model IDs exist, such as claude-haiku-4-5-20251001, alongside undated aliases such as claude-haiku-4-5 | confirmed |
Their vocabulary
| Standard term | Their term |
|---|---|
| Sampling | Sampling, via temperature, top_p and top_k, all deprecated |
| Streaming | Streaming Messages, over server-sent events |
| Greedy decoding | Not named. Temperature 0 was the route to it and is no longer settable on current models |
Where to look
The Create a Message reference lists the full accepted request body, which is the quickest way to see what a given model still takes. Mid-stream errors are visible only if you read the event stream rather than the final assembled message.
Last verified: 2026-09-09 against the Anthropic platform docs: Create a Message, streaming, thinking, and the glossary (platform.claude.com/docs/en/api/messages/create, /build-with-claude/streaming, /build-with-claude/thinking, /about-claude/glossary).
What this maps to: the Responses API, with Chat Completions still documented beside it. OpenAI is the one of the three that documents a seed and a fingerprint to go with it.
| Question | Answer | Status |
|---|---|---|
| Sampling parameters exposed | temperature and top_p | confirmed |
| Documented range | temperature "between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic". No default stated | confirmed |
| What top-p does, in their words | "the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered" | confirmed |
| Guidance on combining them | "We generally recommend altering this or temperature but not both" | confirmed |
| Reasoning models | Do not support temperature, top_p or top_logprobs. The GPT-6 Astra migration guidance says to remove all three from every request type | confirmed |
Whether that applies to every gpt-5.6 variant | The prohibition is stated on the model-guidance page and framed around GPT-6 Astra. The individual gpt-5.6-* model pages say nothing either way | unconfirmed |
| Is a seed offered? | Yes, and hedged. "Our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result." Followed by "Determinism is not guaranteed" | confirmed |
| Seed on the Responses API | Confirmed only on the Completions and Chat Completions references | unconfirmed |
| Fingerprint | system_fingerprint "represents the backend configuration that the model runs with" and is meant to be read alongside seed to spot backend changes that affect determinism | confirmed |
| Streaming | Yes. stream=True on Responses, with semantic events response.created, response.output_text.delta, response.completed and error. Chat Completions streams data-only server-sent events terminated by data: [DONE] | confirmed |
stream_options and usage reporting during a stream | Not checked today | unconfirmed |
Their vocabulary
| Standard term | Their term |
|---|---|
| Top-p | top_p, described as nucleus sampling |
| Streaming | Streaming responses, as semantic events on Responses and token deltas on Chat Completions |
| Provider-side build identity | system_fingerprint |
Where to look
Send the same seed twice and compare system_fingerprint on the two responses. A change there is the documented signal that the backend moved under you.
Last verified: 2026-09-09 against the OpenAI developer docs: the Completions create reference, advanced usage, streaming responses, and the latest-model guidance (developers.openai.com/api/reference/resources/completions/methods/create, /api/docs/guides/advanced-usage, /api/docs/guides/streaming-responses, /api/docs/guides/latest-model).
What this maps to: generationConfig on the Gemini API. Google publishes explicit numeric defaults, and then tells you not to change them on its newest models.
| Question | Answer | Status |
|---|---|---|
| Sampling parameters exposed | temperature, topP and topK in generationConfig, alongside maxOutputTokens, stopSequences, candidateCount and seed | confirmed |
| Documented ranges and defaults | temperature 0.0 to 2.0, default 1.0. topP 0.0 to 1.0, default 0.95. topK default 40 | confirmed |
| Per-model defaults | The reference notes defaults vary by model, and the per-model pages read today do not publish their own figures | unconfirmed |
| Are they still accepted? | Not on the newest models. The gemini-3.8-flash migration guidance says to "Strip temperature, top_p, and top_k from generation configs" and to remove candidate_count | confirmed |
| Guidance on changing them | For Gemini 3, "we strongly recommend keeping the temperature parameter at its default value of 1.0", because lowering it "may lead to unexpected behavior, such as looping or degraded performance, particularly in complex mathematical or reasoning tasks" | confirmed |
| Guidance on combining temperature, topP and topK | None found on any page read today | unconfirmed |
| Is a seed offered? | Yes. seed in generationConfig: "When set, provides deterministic output for the same input and seed value across generations" | confirmed |
| Is that a guarantee? | The hedged wording appeared only in a search snippet, not in a page body that would load. Google's own developer forum carries open reports of non-determinism with a fixed seed on gemini-2.5-pro | unconfirmed |
| Streaming | Yes. Legacy tree: streamGenerateContent over REST with alt=sse, or generateContentStream in the SDKs. Interactions API: stream=True, or POST /v1beta/interactions?alt=sse, with text arriving in event.delta.text on step.delta events | confirmed |
Their vocabulary
| Standard term | Their term |
|---|---|
| Sampling settings | generationConfig |
| Top-p | topP |
| Streaming | streamGenerateContent (legacy) or stream=True on Interactions |
Where to look
Check which docs tree you are reading before copying a field name. The Interactions API and the legacy generateContent tree use different names for the same things.
Last verified: 2026-09-09 against the Gemini API docs: the generateContent API reference, the Gemini 3 guide, text generation, latest-model migration, and text streaming (ai.google.dev/api/generate-content, /gemini-api/docs/gemini-3, /gemini-api/docs/text-generation, /gemini-api/docs/latest-model).