Offloading

When weights exceed GPU memory the remainder lives in system RAM, and every token crosses both legs in sequence. System RAM is roughly an order of magnitude slower, so the spilled fraction dominates the clock long before it dominates the byte count. MoE models get an exception worth taking: pin the experts to system RAM, keep attention and shared tensors on the GPU, and the fast leg carries the traffic that repeats.

Formula

t_{token} = \frac{f _{gpu} B}{MBU _{g} \cdot BW _{g}} + \frac{f _{cpu} B}{MBU _{c} \cdot BW _{c}} tps = 1/ t_{token}

Worked example: Expert offload on 12 GB of VRAM

Qwen3-35B-A3B on a 12 GB card runs at 17 tokens per second with ordinary layer offload. Pin the experts to system RAM and keep attention on the GPU and it doubles to 34, with no change in how much spilled. Which bytes cross the slow leg matters more than how many.

ordinary offload: 17 tps expert offload: 34 tps (2.0 \times)

Try a large MoE model on 12 GB of VRAM