Attention

Why does doubling the context length quadruple the work?

Everyone reads everyone.

A token does not carry its meaning alone. To decide what it means in this sentence, the model lets it look at the other tokens and mix their information. That mixing operation is attention, and it is the reason transformers work at all.

The cost is social. In a meeting where each person reads everyone else’s notes, ten people produce ninety reading pairs. A hundred people produce nearly five thousand. A thousand people produce about half a million. Every token looks at every previous token, so the work grows with the square of the context length, not with the context length.

The matrix shows three things at once. It stays almost empty at four tokens and becomes a dense triangle at a thousand. The cache that stores the keys and values grows linearly, but with a large constant: about 128 KB per token for a common 8B model. And every row must sum to one, so as the row grows, each position gets a smaller share. The model does not get more attention when you give it more context. It spreads the same amount thinner.

Softmax over a growing row.

Each token produces three vectors: a query A vector that represents what this token is looking for. It is compared against the keys of other tokens to decide how much attention to pay them. , a key A vector that represents what this token offers. Queries and keys are compared to produce attention scores. and a value A vector that carries the information this token will pass on if it is attended to. The output is a weighted average of value vectors. vector. Attention compares the query of one token against the keys of the others, turns those comparisons into weights, and returns a weighted average of the values:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

Three details carry most of the meaning:

• The dot product measures relevance. A query and a key that point in similar directions score high.
• The √d scaling keeps softmax usable. Without it, dot products grow with the vector dimension, softmax saturates into a one-hot vector, and the gradients that train the model become tiny.
• Softmax makes each row a distribution that sums to 1. That constraint is the whole dilution story: a fixed budget per token, divided across however many positions it can see.

Where the cost lands in serving.

The practical consequences are easy to observe:

• Time to first token grows with prompt length, not with the length of the answer. A huge retrieved context is paid before the model says anything.
• Tokens per second depends on the cache size, because each generated token re-reads it. Longer contexts decode more slowly.
• Throughput is a memory problem. Long requests reduce how many others fit on the same GPU, which is why providers charge more for long context and why context limits are enforced so strictly.
• Techniques attack different terms. Grouped-query attention and quantized caches shrink the memory term, FlashAttention shrinks the memory traffic, paged attention reduces fragmentation, and prefix caching avoids redoing prefill for a shared prefix.
• Retrieval exists partly because of this. Sending five relevant chunks instead of a million tokens is the cheapest attention optimization available: it removes tokens from the square. The same budget logic appears in embedding dimensions and in tokenization, where every token you add is paid for twice, once in compute and once in memory.

The fine print.

• FlashAttention made attention fast, not linear. The arithmetic is still quadratic in context length. What changed is how efficiently the hardware executes it.
• Quadratic compute is not always the bottleneck. In decode, attention is limited by memory bandwidth: reading the cache costs more than the arithmetic on it. Optimizing FLOPs can leave latency unchanged.
• More context is not more attention. The softmax budget per token is fixed, so long prompts dilute every weight. Quality can fall even when the answer is technically in the context.
• The KV cache number is a product decision. About 128 KB per token means 8k tokens cost roughly 1 GB per request and 100k tokens cost more than 12 GB. That is memory taken away from concurrency, before the model weights are counted.
• Do not compare context lengths across models. KV cache size depends on layers, KV heads, head dimension and precision, not on the advertised token limit. Two models with the same limit can differ by a factor of ten in memory.
• The measurement that matters is yours. Watch time to first token, tokens per second, and memory per request as context grows. The square is visible in real numbers long before it is visible in the bill.