Skip to content
SIGPULSE
Papers Batch 2 2 min read raw .md ↗

What Actually Makes LLM Serving Expensive? VRAM — and vToken Brings OS-Style Virtual Memory to the KV Cache

Chinese original 2026-08-15 · 「大模型贵在哪？显存。有人给它装了套"虚拟内存"」 · translated to English 2026-08-30

Ask why large models are expensive to serve and most people answer “compute.” The August 13, 2026 preprint our Chinese-language column dissected targets the real bottleneck in many deployments: VRAM — and its fix is an idea operating systems perfected fifty years ago. The column’s title: “What are big models expensive in? VRAM. Someone installed ‘virtual memory’ for it.”

The claims

  • The KV-cache mechanism: per-token intermediate results retained for later lookback; long conversations mean more parked notes, and VRAM admits no new users once full — a several-thousand-dollar GPU idling half the time waiting for memory to clear (the column’s framing)
  • The legacy gap: coarse-grained eviction (whole shelves, not books), recomputation when evicted entries are needed again, and — per the paper — token-level policies that logically reclaim memory physical block managers never actually free
  • vToken (arXiv:2608.13263): a token-level virtualization layer decoupling logical token liveness from physical block placement, with prefetch/recompute for evicted tokens and cross-request sharing

How the Chinese column reads it

The essay’s chain is deliberately pedestrian: mechanism (KV cache as meeting notes) → accounting (services are VRAM-bound, not FLOP-bound) → prior art and why it falls short (the library that can only reshelve by the shelf) → the transplant (OS paging, down to the single book). Its two landing points for readers: pricing (membership and per-token API floors track VRAM occupancy, so cache management is where the cost curve actually bends), and long context (whole-book, whole-contract use cases become affordable when the cache stops squatting). The closing provocation sets the two roads against each other: hardware vendors stack more VRAM — buy bigger cards; systems people squeeze the existing cards — which do you back? And if VRAM bottlenecks do get ground down by work like this, where does the price war burn first?

What to watch

Adoption inside serving stacks (the interesting question is always whether a thin layer survives contact with vLLM-class schedulers); the recomputation-vs-prefetch cost line as context lengths grow; and the second-order effect the column hints at — if serving cost decouples from VRAM capacity, the hardware bull case loses one of its props.

Provenance & disclosure. Originally published in Chinese on our WeChat channel on 2026-08-15 (“大模型贵在哪？显存。有人给它装了套’虚拟内存’”); drafted with AI assistance under human editorial direction. Translated to English on 2026-08-30 (AI-assisted, human-reviewed). The paper ID was located and cross-confirmed across three independent mirrors (papers.cool, alphaXiv, Semantic Scholar); claims follow the abstract as relayed by the column. Preprint caveat inherited. Translated commentary — not a SigPulse measurement; first-party numbers live in the dispatches and the /data/ ledger.

Papers covered in this digest (machine-readable in /papers.json)

  • 2608.13263 — vToken: Token-Level Virtualization for Reclaimable KV Caches

FAQ — Direct Answers

Why is VRAM, not compute, often the binding constraint?
LLM inference requires the KV cache: for every token the model reads, intermediate results are stored for later tokens to look back at — meeting notes that must be kept or the conversation stops making sense. A long conversation parks hundreds or thousands of tokens' notes in VRAM without moving; when VRAM fills, no new users fit. The column's phrasing: the expensive GPU may spend half its time waiting for VRAM to clear.
Why wasn't existing KV-cache management enough?
Mainstream policies evict at coarse granularity — whole blocks or sequences get dropped (a library that can only clear entire shelves, hot books included), and an evicted entry that becomes needed again is simply recomputed, spending the compute you saved. Worse, token-level eviction policies meet runtimes that manage memory in physical blocks: evicting a few tokens reclaims nothing until their whole block goes.
What does vToken do?
It inserts a lightweight token-level virtualization layer between the eviction policy and the block manager, decoupling logical token liveness from physical placement — the fifty-year-old virtual-memory idea transplanted into inference. Idle tokens' space is reclaimed individually; needed tokens are prefetched or recomputed on demand; cross-request KV sharing rides along. The model believes all its notes are present; the system schedules the pages underneath.
Why does this matter to a normal user's wallet?
AI pricing bottoms out in GPU cost, and a large share of that is VRAM occupancy. The finer the KV management, the more users and longer conversations fit per card — and long-context applications (a book, a few-hundred-page contract) move from 'prohibitively expensive or over-limit' to usable. Price cuts come from engineering scalpel-work like this, not from keynote specs.