EXHIBIT 03 / MEMORY & GENERATION
Why do longer answers keep consuming memory?
During autoregressive generation, each new position attends to earlier positions. A KV cache keeps earlier key and value vectors so their projections do not have to be rebuilt at every step.
THE SHORT ANSWERReuse old projections. Retain more state.
6–10 MIN · STEP-BY-STEP · NO ACCOUNTA cache changes one part of the account—not the whole model.
- 01 / POSITIONOne token enters
A position corresponds to a token, not necessarily a word or character.
- 02 / PROJECTEach layer makes K and V
Without a cache, earlier K/V projections are rebuilt when the prefix is processed again.
- 03 / RETAINKeep the earlier vectors
With a cache, each new position adds its own K and V while earlier values are reused.
- 04 / ATTENDHistory still has a cost
The new query still attends across retained history. Caching removes repeated projection work; it does not make long context free.
The linear storage statement applies to a fixed structure that retains the full history. Sliding windows, eviction, quantisation, grouped-query attention, and other designs change the curve.
Follow one new position through the trade-off.
Advance the record yourself. The sequence is an editorial teaching trace; the numerical account below is computed in your browser.
WITHOUT CACHE
Rebuild the prefix
To produce the next position, the teaching baseline sends the whole prefix through every layer again.
Earlier K/V projections are recomputed.
Keyboard: focus this record and use ← or →. Nothing advances automatically.
Move sequence length. Watch compute and retained state separate.
This exact-integer browser account counts only K and V projection multiply-accumulates under a square-projection teaching model.
- Cached / recomputed K/V MAC count
- 0.39%
- Values retained in the cache
- 503,316,480
- Logical tensor storage at 2 bytes/value
- 960.0 MiB
At 512 positions, caching removes repeated K/V projection work in this teaching account, while 503,316,480 retained values remain in the full-history cache.
This account counts only the K and V projection multiply-accumulates. It is not an end-to-end speed measurement and not a GPU-memory reading.
Account A: cached K/V projections = 2LD²N; recomputed K/V projections = 2LD²N(N+1)/2; retained values = 2LDN. One MAC means one multiplication plus one accumulation, not a declared FLOP count.
A real toy model tests a broader account.
350,720 / 6,507,520 MACs; Q/K/V projections plus attention, 2 layers × dimension 32 × 40 positions.
K/V projections only. It is deliberately different from Account B and cannot replace it.
Tensor element count in the toy model—not Python memory and not a GPU-memory measurement.
Two equivalent paths
The Python script runs recompute and cache paths for the same tiny attention calculation and checks their outputs.
Counts and local timing
MAC counts and cache elements are deterministic for the fixed configuration. Milliseconds belong only to the recorded machine and run.
Whole-model speed
Neither browser Account A nor the toy Python Account B measures a production model, GPU memory, or end-to-end serving speed.
RECORDED RUN / EXECUTED
Inspect the result behind the interpretation.
On the recorded 2-layer, dimension-32, 40-position CPU run, caching used 350,720 of 6,507,520 counted MACs. Timing varied by machine; the operation counts did not.
Tracked content revision: dac090c15fd670a6b115fc2b8a9f552bd3cb4297a8511378104d05d1d0f9261d
Show exact lines from the original stdout
The program’s original labels are in Simplified Chinese; the lines and numbers below are copied without translation.
位置 不缓存·本步 [最小, 最大] 缓存·本步 [最小, 最大] 中位数比 缓存占用
40 14.48 ms [ 14.10, 16.21] 0.45 ms [ 0.44, 0.50] 32.0× 5,120 个数
总乘加次数 不缓存 6,507,520 缓存 350,720
缓存这一路是不缓存的 5.4%PRIMARY & PRACTICE SOURCES