The Model Isn't Slow - So Why Does It Feel Slow? Three Entries in the Latency Perception Ledger
The Model Isn't Slow - So Why Does It Feel Slow? Three Entries in the Latency Perception Ledger A previous piece covered the basic latency ledger for AI …
The Model Isn't Slow - So Why Does It Feel Slow? Three Entries in the Latency Perception Ledger
A previous piece covered the basic latency ledger for AI systems: time to first token, generation speed, and end-to-end completion time. This one zooms in on a subtler question: with identical technical numbers, why can two products feel several times different in speed? The answer isn't in the model - it's in the perception ledger.
Entry one: the quality of waiting matters more than the quantity
The human brain never estimates waiting with a stopwatch. A classic finding in psychology: waits filled with information gain feel short, empty waits feel long. A three-second load that keeps giving feedback - a typewriter effect, step indicators - may feel like 1.5 seconds. A spinner frozen for three seconds can feel like five, and comes with did it crash? anxiety for free.
For AI apps this means TTFT (time to first token) isn't just an engineering metric; it's an experience lever. Cutting TTFT from 800 ms to 300 ms buys more than shaving total generation from 10 s to 8 s, because the first removes the uncertainty of whether there is any response at all, while the second merely shortens a wait whose existence was already certain.
Entry two: predictability beats raw speed
Bus stops say it best: knowing the bus arrives in two minutes makes waiting comfortable; not knowing makes one minute irritating. AI products work the same way. The greatest trick of streaming isn't delivering answers sooner - it converts an unpredictable black-box wait into a predictable, token-by-token promise being kept.
So when prioritizing engineering work, ask in this order:
- Can the first visible feedback come sooner? (Route to a small model first, cache hits, pre-warmed connections.)
- Is the feedback rhythm stable? (Avoid a stall followed by a flood; variance in tok/s hurts experience more than the mean.)
- Only then: compress total duration.
Entry three: cognitive load disguised as latency
Much of it feels slow is not the system at all - it is that users have nothing to do while waiting: the answer is too long, the structure too messy, and the reading burden gets misfiled as waiting cost. That is why the same 2,000-token output feels faster with clear subheadings and lists: users skim, grab the conclusion early, and their cognitive assembly line clocks out ahead of schedule.
Show the system your ledger
In practice, instrument three perception metrics: time to first visible byte, P95 variance of tok/s, and the share of generations users stop midway. The third is the most honest - it is users rating their latency satisfaction with their feet. Technical metrics tell you whether the system is fast; perception metrics tell you whether users think it is. Only when both line up does the ledger balance.
Next time you optimize an AI experience, do not stare at dashboard averages. The latency ledger in your users heads records impressions, not milliseconds.