KV cache and context
The KV cache stores attention state for tokens already in the conversation. It grows with context length, layers, KV heads, and head dimension. That extra memory must fit, and reading it makes decode slower as the conversation grows. Q8 KV halves its memory in the research model; Q4 quarters it but can carry a long-context speed and quality cost.
Formula
KV bytes = 2 × KV heads × head dimension × layers × bytes per value × context
Worked example: Llama 3 70B at 128k context
The artifact gives Llama 3 70B a BF16 KV cost of 0.31 MB per token. At 128k context, the KV cache alone is about 42 GB, before the model weights and workspace.
0.31 MB/token × 128,000 tokens ≈ 42 GB