How to Tune LLM Sampling Parameters: Temperature, Top-p, and Top-k

Most API consoles display these three parameters with a default value of 0.7, and many users leave them untouched for months. However, they determine one specific thing: the rule by which the model selects a token from the candidate pool during each generation step. This step directly impacts output stability and even the actual cost of batch tasks. This article breaks down these three parameters and clarifies which ones to adjust for different scenarios.

What Each Parameter Actually Changes

LLM generation proceeds token by token. For each token, the model first calculates probabilities across the entire vocabulary (tens of thousands of tokens), then narrows the candidate pool down to a small subset based on specific rules, and finally samples from this subset according to their probabilities.

- **Temperature**: Divides the logits by $T$ before calculating probabilities. The smaller $T$ is, the more prominent high-probability tokens become; as $T$ approaches 0, it degenerates into greedy decoding, always selecting the highest-probability token, resulting in completely deterministic output. The larger $T$ is, the flatter the distribution becomes, giving low-probability tokens a higher chance of being selected.

- **Top-k**: Retains only the top $k$ candidates by probability, discarding the rest.

- **Top-p (Nucleus Sampling)**: After sorting by probability, selects the smallest set of top tokens whose cumulative probability just reaches $p$, discarding the rest. The difference from top-k is that the candidate pool size floats with the context—when the model is confident, the pool automatically shrinks.

Practical Feel for Three Scenarios

**JSON Extraction, Classification Tagging**: Use a temperature of 0 or 0.1, and compress top-p to 0.1–0.5. These tasks require the same input to always produce the same set of fields. Any fluctuation in word stability might result in an extra comma or a missing field, forcing you to rely on retries for the failed cases, which doubles the cost.

**Creative Copywriting, Brainstorming Drafts**: Use a temperature of 0.9–1.2 and top-p of 0.9–0.95. The value in these scenarios lies precisely in occasional unexpected word choices. Repeated calls should yield different drafts; excessive stability merely wastes parallel computing costs.

**Code Generation, Long-form Writing**: A temperature of 0.2–0.4 is a common balance point. The main body needs to be stable, but retaining slight lexical variation avoids generating the exact same code six times in a row.

Two engineering details are easy pitfalls. First, do not push all three parameters to their extremes simultaneously. Setting both top-k and top-p too small truncates the candidate pool twice, making behavior hard to predict. A common practice is to manage temperature and top-p together while leaving top-k at its default. Second, investigate prompts before tweaking sampling when facing "instability." Vague answers are mostly due to ambiguous system prompts, insufficient few-shot examples, or tasks that are too granular. Sampling parameters only change "which word is picked within the model's capabilities"; they do not expand the capabilities themselves. If a model cannot answer well, increasing the temperature just makes it err in different ways.

How to Test Quickly in Production

Fix two things: output format constraints (JSON Schema or few-shot examples) and a fixed evaluation set (20–30 representative inputs). Then, adjust only one parameter at a time, score using the same batch of samples, and monitor two metrics: structural error rate (whether JSON can be parsed, whether fields are complete) and consistency rate across multiple outputs for the same input. If the structural error rate does not drop even when temperature is compressed to 0.1, the problem likely lies not in sampling but in the prompt or task granularity—go back and refine those.

Add another frequently ignored parameter: **seed**. Many inference services support specifying a random seed. When temperature is not 0, different tokens are sampled each time; with a fixed seed, the same input and parameters will yield the same result. Its correct usage is not to "fix production output," but for regression testing: run the same batch of samples (with the same seed) before and after changing the prompt, then diff the results to see exactly which outputs were skewed by the change. This is far more reliable than judging by feel. Note that different services do not guarantee identical seed implementations, so regression results cannot be directly carried over when switching services.

You can directly copy these common combinations:

| Scenario | Temperature | Top-p | Top-k |

|---|---|---|---|

| Extraction / Classification | 0–0.1 | 0.1–0.5 | Default |

| Runtime Code Generation | 0.2–0.4 | 0.8–1 | Default |

| Creative Drafts | 0.9–1.2 | 0.9–0.95 | Default |

Summary: Although these three parameters look like a trio, they actually do only one thing—narrow the candidate pool for word selection. Narrow it to the limit for tasks requiring stability, and leave some gap for tasks requiring novelty. When output is unstable, modify prompts and format constraints first, then adjust sampling; reversing this order means spending days tuning temperature only to make the answer accuracy even harder to discern.