Block-level memory scheduling
A little space. A lot of context.
Watch requests compete for KV cache. Keep hot blocks on GPU, swap cold ones to CPU, and recompute what memory pressure leaves behind.
Simulation live
0 s / tick
0running requests
0 / 25completed
0swap-outs
0recomputes
01 / GPU
Resident · decode-ready
Fast tier · least recently touched leaves first
02 / CPU
Swapped · context preserved
Swap tier · most recently touched returns first
03 / Evicted
Dropped · recompute required
Dashed blocks are lost context, not allocated memory
Request trace / 25 sequences
GPUCPUEvicted
| ID | Request / arrival | Tokens | State | Residency · last 40s |
|---|