synodic-ai ML SYSTEMS
ML Systems Desk · Inference Systems

Spend Compute Where the Answer Changes: Joint KV-Cache Compaction, Deferred Video Caches, and Load-Aware Sampling

2026-10-20

Three papers posted to arXiv on 2026-10-18 and 2026-10-19 replace fixed inference rules with decisions priced against what they buy: which KV-cache entries to keep, how much of a video stream to precompute, and when to stop sampling answers. Each gains by modeling an interaction the fixed rule ignored, and each says where the gain narrows.

Deleting cache entries together, not one at a time

Offline KV-cache compaction lowers the cost of repeatedly querying a fixed context: a smaller cache is built once and reused. The hard part is choosing which entries to delete. Moritz Wagner, Sebastian Pokutta and colleagues point out that existing methods score each deletion candidate on its own while holding the rest of the cache fixed, but softmax renormalization couples those decisions nonlinearly, because removing one entry changes the attention weights of every remaining one.

KV-Surgeon uses calibration data to account for those interactions tractably. The authors derive a quadratic surrogate with closed-form expressions for jointly making the optimal greedy removal decision and compensating within the remaining entries. On QuALITY, LongHealth and Long Code Arena, at every compaction ratio evaluated, the authors report that it substantially advances the cost-accuracy Pareto frontier for dense and mixture-of-experts models from 4B to 30B parameters. At 50x compaction it improves question-answering accuracy by up to four points over attention matching, which the authors cite as the state of the art, at comparable cost, and a faster variant also beats the state of the art at two fifths of its cost.

Precomputing only as far as a query will need

Haoyang Fan, Viktor Prasanna and colleagues study real-time streaming video question answering, where queries arrive at any time and need low latency while the system keeps up with the video. Existing systems build KV caches for the entire video as it arrives, even though a query typically uses only a few preceding frames as evidence. That exposes a trade-off: computing more on arrival cuts query-time work but can increase processing lag, while deferring work raises the video-processing rate but adds query-time computation for retrieved content.

StreamAhead handles it with adaptive precomputation. Segments are encoded independently and each can sit at a different representation level, from inexpensive visual features up to costly KV caches; KV caches are completed only for the segments a query retrieves. The cost of advancing a segment to each level is profiled offline, and the runtime uses those profiles to schedule segment computation concurrently with query execution, pipelined across heterogeneous CPU and GPU resources while holding the required video rate.

The authors report accuracy competitive with state-of-the-art methods at up to 10.4x lower time to first token, and up to 5.9x lower than state-of-the-art streaming systems. It sustains 30 frames per second with P95 time to first token under 0.5 s, and up to 57 frames per second with P95 under 2 s.

Stopping the sampling when the GPU gets expensive

Test-time scaling improves reasoning by sampling several answers, but Mobina Kashaniyan and Ali Jannesari note that fixed sampling budgets ignore both query difficulty and the changing energy cost of inference under GPU contention. They measured up to a 4.1-fold increase in energy for the same four-sample query as GPU load rose.

ATEST weighs the expected value of another sample against its current GPU cost. Its default mode, ATEST-Online, needs no offline calibration: it generates samples incrementally, estimates the stability of the plurality answer, monitors live GPU conditions, and learns per-step energy cost online to decide when to stop. ATEST-Calibrated instead picks a per-query budget from offline accuracy and cost models, which suits deployments with stable load.

The evaluation covers four reasoning and question-answering datasets with a small model on an NVIDIA A100 under controlled, varying load, with transfer tests on an NVIDIA L40S and a second small model. On the four A100 datasets, ATEST-Online answered every query without exceeding the energy budget, reached the highest or statistically tied accuracy, and sampled less as load increased. On GSM8K at high load it reached 60.0 percent accuracy with no deferrals, against 44.0 percent accuracy and 42.4 percent deferrals for the calibrated mode. Ablations attribute the adaptation to online cost learning, and including offline profiling, the online mode used 42 times less GPU time and 48 times less energy than the calibrated mode for the evaluated workload. The trade-off is higher tail latency and lower throughput on short tasks; offline calibration stays competitive when conditions are stable.

Pricing the decision instead of fixing it

The three systems share a structure. Each had been using a rule that treated a decision as independent of its context: per-entry deletion scores that ignore softmax coupling, full KV caches built regardless of which frames a query will use, and sample counts set without regard to live GPU cost. Each replaces the rule with a model of what the decision actually costs and buys, from calibration data, offline profiles or online measurement.

Each also states where that model can mislead. KV-Surgeon's gains are measured on specific long-context benchmarks and model sizes. StreamAhead depends on offline profiles of per-level cost. ATEST-Online buys robustness under varying load with worse tail latency on short tasks, and the calibrated mode remains the better choice when load is stable.

Engineering takeaways

Each result comes from the authors' own models, benchmarks and hardware, so treat these as hypotheses to validate on your stack.

Sources