Cache Hit Rate and User Wait Time: What Averages Hide
Cache hit rate is often presented as a performance metric, but it describes a proportion at the request level, whereas what users actually experience is how …

Cache hit rate is often presented as a performance metric, but it describes a proportion at the request level, whereas what users actually experience is how long their own handful of operations had to wait. The relationship between the two is not a simple linear one; request counts, the cost of a miss, and the latency distribution sit in between.
Let a hit take h to respond, a miss take m to go back to the origin, and the hit rate be r. Then the average latency per request is r·h + (1−r)·m. For example: h = 5 ms, m = 500 ms, r = 99%, which gives an average of about 9.95 ms—that looks excellent. But 1 out of every 100 requests still makes a user wait half a second; the median may be only 5 ms, while p99 is 500 ms. If a page issues 20 requests, the probability that a user encounters at least one miss is 1 − 0.99²⁰ ≈ 18%. In other words, nearly one in five page loads will be slowed down by that 1%.
This shows that hit rate has to be considered together with the number of requests per user session. The more requests there are, the greater the chance that a small number of misses gets amplified. This is especially visible on mobile devices with weak connectivity and in scenarios where the first screen fires many parallel requests; conversely, for a single-query API, even a somewhat lower hit rate has limited user-perceived impact.
A second issue is that the cost of a miss is not uniform. Some key misses only involve looking up a small table, tens of milliseconds; other misses require aggregating across services, scanning large tables, or even triggering a cache stampede that puts pressure on the database, with latency possibly reaching seconds. Product detail pages and recommendation lists are two typical cases: the former is heavy to fetch from the origin, the latter relatively light. The same hit rate can have very different effects on wait time under different key distributions. When a hot key expires, a burst of concurrent requests may all go back to the origin at once, forming a stampede; here a short-term dip in hit rate is amplified by concurrency.
A third issue is how hit rate is measured. It is usually counted by requests, whereas user experience is counted by session; it is also often averaged globally, which masks differences by region, device, and endpoint. Some endpoints may have a 99.9% hit rate while others have only 80%; the global number looks good, yet specific users may still be consistently slow. Looking at it stratified by endpoint and user group is more useful than a single total.
One more easily overlooked point: a hit does not mean correct. If the cache returns stale data, users see the wrong price or an outdated status; although the wait is short, they have to act again, so the actual time is longer. For time-sensitive data, hit rate and freshness must be weighed together; expiry times cannot be extended indefinitely just to make the numbers look good.
In practice, rather than fixating on a single hit rate, it is better to look at several things at once: tail latency counted by session, such as p95 and p99; the latency distribution of miss requests; the correlation between requests per page and misses; and whether the cache invalidation strategy causes concentrated origin fetches. Request coalescing, short-lived stale-while-revalidate, adding random jitter to TTLs, and setting timeouts and fallbacks can be used to reduce the worst case. The goal of all these measures is to reduce users' actual waiting, not to push the hit rate up another percentage point.
The limitations should also be stated clearly: these metrics depend on instrumentation, and sampling and definitions affect the conclusions; different businesses define "waiting" differently—first byte versus interactivity; network, rendering, and third-party scripts outside the cache layer also take time, and no matter how high the hit rate is, they cannot be saved. In addition, raising the hit rate usually means caching more content and longer expiry times, which increases memory and consistency costs.
Conclusion: hit rate is a means, not a goal. Only by breaking user wait time into the baseline latency on a hit, plus the miss probability multiplied by the miss cost, and then multiplied by the session request count, can you see where to actually optimize. For ordinary technical readers, one point is enough to remember: users never encounter an "average request"; they only encounter their own.