SEARCH THE LAB

What are you looking for?

SkillsReview information not provided

How to distinguish first-token latency from generation speed using two measurements

Review information not providedAdded to the lab: 19 November 2026

First-token latency and generation speed are two different things. The first is the time from sending a request to seeing the first content chunk; the second…

Added to the lab
Downloads
0
Human review
Review information not provided

First-token latency and generation speed are two different things. The first is the time from sending a request to seeing the first content chunk; the second is the rate at which content chunks keep arriving. If you merge the two into a single "response time," you neither know why users find it laggy nor which segment you should optimize. A workable approach is to run two controlled measurements and use the difference to separate fixed overhead from per-token cost.

First, the measurement protocol. Send a streaming request from the client and record three moments: t0 when the request is sent, t1 when the first chunk containing body text arrives, and t2 when the last chunk containing body text arrives; at the same time, take the actual number of tokens generated, n, from the returned result. Do not count chunks — one chunk may contain several tokens, or only half a token; and do not count the trailing chunk that carries only a finish reason and no body text, or the speed will be dragged down.

The design of the two measurements is: fix the prompt, fix the sampling parameters, use the same machine and the same network, and change only the output length. For example, first have the model write about 100 tokens, then about 500 tokens. If first-token latency is fixed, total time should be approximately a straight line against token count: total time = first-token latency + n × per-token time. Use the two results to compute the slope; the slope is the per-token time. Substitute the slope back to compute the intercept; the intercept is the first-token latency.

Here is an arithmetic example — the numbers are only meant to illustrate the method: in the first run, the first chunk arrives at 0.9 seconds, the end at 3.9 seconds, with 100 tokens; in the second run, the first chunk arrives at 0.92 seconds, the end at 15.9 seconds, with 500 tokens. The total-time difference is 12 seconds and the token difference is 400, giving a slope of 0.03 seconds/token, or about 33 tokens per second; the intercept is 3.9 − 100×0.03 = 0.9 seconds. So you can conclude: under this configuration, first-token latency is about 0.9 seconds and generation speed is about 33 tokens/s. The intercept should be close to the measured first-chunk time; if it is clearly larger, that means the first few tokens are slower, or that pauses in the middle have been averaged away. Conversely, if the second run's first chunk jumps to 2.5 seconds while the finish time still grows proportionally, then what has slowed down is queueing or prefill, not decoding.

A few pitfalls. First, first-token latency includes DNS, TLS, proxy buffering, and server-side queueing, and it also grows with prompt length, so the prompts in the two runs must be exactly identical, and it is best to measure in the same region and during the same time window. Second, with only two points, noise turns directly into slope error; in practice, using three to five lengths, repeating each several times, and taking the median is far more reliable than a single pair of numbers. Third, decoding speed is not a constant: the longer the context, the heavier the attention computation, and models often slow down as they write more, so the slope is an average over that interval and cannot be extrapolated to longer outputs. Fourth, pauses can occur in the middle — safety filtering, tool calls, and rejected speculative decoding can all break the stream for a moment; the average will not show it, so you need to look at the distribution of chunk intervals or at tail latency. Fifth, reasoning models count chain-of-thought tokens in usage but may not stream them; in that case completion_tokens is far larger than the body tokens you actually see, and using it to compute speed yields absurdly high values.

One more point that is often overlooked: Chinese readers perceive characters, not tokens. A single Chinese character may correspond to a multi-byte token, or several characters may make up one token; only when converted into "characters per second" does it come close to the reading experience.

The value of the two measurements is not in producing a precise number, but in splitting "slow" into two attributable segments: if first-token latency is high while the slope is normal, you should look at queueing, the network, and prefill; if the first token is fast but the slope is steep, you should look at decoding efficiency, batching, and output-length policy. Before measuring, think clearly about which question you want to answer — that is more useful than explaining a pile of numbers afterwards.