Offloading
Offloading lets a model run when its weights do not all fit in GPU memory, but each token then crosses a fast GPU leg and a slower system-RAM leg in sequence. The slow leg can dominate. MoE models have a useful exception: expert tensors can stay in system RAM while attention and shared tensors remain on the GPU.
Formula
time/token = GPU fraction × bytes ÷ (GPU MBU × GPU BW) + CPU fraction × bytes ÷ (CPU MBU × RAM BW) tps = 1 ÷ time/token
Worked example: Expert offload on 12 GB VRAM
For Qwen3-35B-A3B on a 12 GB GPU, the research artifact reports 17 tps with ordinary offload and 34 tps when attention stays on GPU and experts are pinned to CPU memory.
ordinary offload: 17 tps MoE expert offload: 34 tps