MoE Routing Is Not a Cost-Saving Switch: Why Expert Models Fear Load Imbalance

Mixture of Experts (MoE) models appear to offer a straightforward optimization: only a small subset of experts is activated per request, allowing the parameter scale to grow without a proportional increase in computational cost. However, in production, the challenge of MoE rarely lies in "how many experts there are," but rather in "who decides where each token goes."

This decision is made by the router. It computes a score for each token and assigns it to one or more experts. Ideally, the load across experts remains nearly uniform, data movement between GPUs stays manageable, and the latency curve remains smooth. In reality, if certain experts are repeatedly selected, the system immediately suffers from queuing, increased cross-GPU communication, and elevated tail latency.

Therefore, MoE routing is primarily a systems engineering problem. During model training, a load balancing loss is often added to prevent the router from funneling all tokens to a few experts. On the inference serving side, a capacity factor must be set to limit the maximum number of tokens a single expert can receive. If the capacity is too tight, tokens may be dropped or the system may degrade to a fallback path; if it is too loose, it wastes VRAM and scheduling headroom.

The second issue is shape variation within a batch. In standard dense models, computation at each layer is relatively regular. In MoE layers, however, tokens must be regrouped by expert, processed, and then restored to their original order. This process introduces additional all-to-all communication overhead. As models grow larger and experts become more distributed, communication costs are increasingly likely to erode the benefits gained from activating only a subset of experts.

The third issue is hot-spot inputs. Different types of traffic cause the router to exhibit varying preferences. Code requests, math queries, and casual chat may naturally favor different experts. If the structure of online traffic shifts suddenly, the previously balanced expert distribution can become skewed. Consequently, MoE services cannot rely solely on average tokens per second; they must also monitor token distribution per expert, overflow rates, tail latency, and cross-GPU communication time.

For application teams, the correct approach to MoE is not to treat it as a "cheaper large model." A more prudent strategy is to view it as a cluster system requiring observability and capacity planning: first, replay real requests to analyze routing distribution, then determine batch strategies, expert parallelism methods, and degradation paths.

Without these metrics, MoE is prone to a common misjudgment: offline benchmarks look impressive, but performance jitters under peak online load. The model hasn't suddenly degraded; rather, the router has turned certain experts into bottlenecks.

The value of MoE is real. It enables large-parameter models to run at acceptable costs. However, it is not free magic. The router, load balancing, communication topology, and capacity control are the key factors determining whether MoE can be deployed stably.