synodic-ai ML SYSTEMS
ML Systems Desk · Inference Systems

The Long-Context Bill Has Three Line Items: Cache Memory, Draft Quality, and Tokenization

2026-10-15

Long contexts and long-running agent sessions are usually discussed as an attention-cost problem, but the cost shows up in at least three separate places in a serving stack: the size of the key-value cache, the quality of speculative drafts on long inputs, and the CPU front end that turns text into tokens. Three preprints from 2026-10-14 each attack one of those line items, and each does it without retraining the target model.

Where long context actually costs

Long contexts enlarge key-value (KV) caches, which raises memory use and attention cost during generation. Speculative decoding, which accelerates generation with a lightweight drafter and parallel verification, should help most on long inputs, where the cost of reading the KV cache can be amortized over several accepted tokens, yet most drafters are trained on short sequences and degrade on long ones. And in agent workloads, where the same conversation is sent again with a little text appended each turn, inference engines reuse the KV cache of the shared prefix but still re-tokenize the whole request from its first character.

The three papers below take these in turn. Latent Compaction learns a much smaller cache per document. FOLD changes how a drafter attends to distant context without training it. Delta Tokenizer tokenizes only what changed. All three leave the target model's weights untouched.

Latent Compaction: optimize the cache, not the model

Yingfa Chen, Xingyu Shen, Chaojun Xiao, and Xu Han treat a document's KV cache as something to be trained. Latent Compaction learns a compact, document-specific cache through test-time training, under the constraint that compression must preserve information for questions that have not yet been asked. Cache entries are initialized from reconstruction importance, teacher attention, and key-space diversity. Training first fits the compact cache's attention outputs and log-normalizers to those of the full cache, holding the reconstruction-selected entries fixed, and then optimizes the whole cache jointly by end-to-end distillation from the full-cache model. Model weights stay frozen, and training uses only generic prompts about the document, with no access to downstream questions or answers.

On RULER, LongBench, and Phonebook Lookup, the authors report substantial gains over budget-matched compression baselines; on RULER, for example, the method reaches 96.7% accuracy at 32x compression, against 40.0% for KVZip+. They also report that it transfers across model scales and families, with retrieval gains on two model families, and they have released the code. Their conclusion is that treating the KV cache as a neural network and optimizing it with test-time training can preserve retrieval accuracy under aggressive compression.

FOLD: let the drafter read everything, at one shared position

Existing training-free approaches to long-context drafting restrict the drafter's context window, rescale positions, or add retrieved distant entries to the recent context. The FOLD authors argue that this is more restriction than drafters need: a drafter reads the target model's hidden states, which already encode contextual and positional information, and its lightweight architecture makes attention over the full history affordable.

FOLD is training-free. It preserves exact positions for local entries and treats every distant entry as far memory placed at a single shared position. Because attention over that memory dilutes as it grows, FOLD sharpens attention within the far memory to emphasize relevant entries and separately controls the memory's total weight relative to the local window. Across six target-drafter pairs and benchmarks of up to 256K tokens, the authors report that FOLD predominantly delivers the highest speedup; in the best case it more than doubles the speedup over autoregressive decoding, from 2.08x to 4.22x.

Delta Tokenizer: the front end that starts from the first character

Zhenyu Zhang and Zhichao Cao point at a cost that prefix caching leaves in place. An agent sends the model the same conversation again and again, each turn adding a little text at the end; the engine reuses the KV cache of the shared prefix so only new tokens pass through the model, but the CPU front end converts every request back into token IDs from the first character. That lengthens time to first token (TTFT) and consumes CPU on every turn.

Delta Tokenizer tokenizes only the difference between a request and the previous request in the same session. It keeps the earlier token sequence, re-encodes only the appended text plus the short tail whose tokens the append could affect, and joins the two at a position that the tokenizer's own splitting rules cannot move, which keeps the result exact. For long requests with no prior state, it cuts the text where practical tokenizers already keep neighboring pieces independent, encodes the pieces on several cores, and concatenates them into the reference encoding. A shared CPU pool schedules these chunks together with short incremental updates, so an update runs between two chunks instead of waiting behind a whole long job.

The incremental path shortens tokenization of a session turn by roughly 40x relative to the reference tokenizer the authors compare against. On 16 cores the chunk-parallel path encodes a four-million-character request 8.2x faster than a serial encode and 1.6 to 2.1x faster than LoPT. Inside SGLang, the authors report a 59% reduction in median TTFT through the chat interface at 100K tokens, and P99 TTFT for agent sessions down 9.5x under load and 23x when long requests tie up the front end. Every output they checked matched the reference, including 41.8 billion documents encoded by the fast encoder.

Reading the three results together

All three are preprints, and the figures are the authors' own, on their chosen benchmarks, hardware, and baselines. Latent Compaction's 32x figure is one RULER setting and requires a test-time training pass per document, which suits documents that are queried repeatedly more than one-off prompts. FOLD's 4.22x is the best case across its target-drafter pairs. Delta Tokenizer's largest TTFT gains are measured under load and with long requests competing for the front end.

The common point is that long-context serving cost is not one number. A cache that is too large, a drafter that degrades with distance, and a tokenizer that repeats work every turn are separate bottlenecks with separate fixes, and an improvement in one does not show up in the others. A team that has adopted prefix caching, for example, may still find its agent-session tail latency set by the CPU front end.

What serving teams can take from this

Each paper suggests a measurement worth adding before choosing an optimization.

Sources