KV cache and context

The KV cache holds attention state for every token already in the conversation, growing with context, layers, KV heads, and head dimension. It competes with the weights for memory and adds to the bytes each new token must read, so a long conversation is slower than a short one on identical hardware. Q8 KV halves it. Q4 quarters it and can cost accuracy exactly where long context was the point.

Formula

KV bytes = 2 \times n_{kv heads} \times d_{head} \times L \times b \times context

Worked example: Llama 3 70B at 128k context

Llama 3 70B costs 0.31 MB of KV per token at BF16. Fill 128k of context and the cache alone is 39.7 GB, before a single weight is loaded. That is a second model's worth of memory bought entirely with conversation length.

0.31 MB \times 128, 000 = 39, 680 MB = 39.7 GB of KV alone

Try a 120B model at 128k context