synodic-ai ML SYSTEMS
ML Systems Desk · Inference Systems · Adaptive Budgets

From Fixed Settings to Measured Ones: Adaptive Draft Configuration, KV Precision Allocation, and Head-Wise Cache Budgets

2026-10-14

Three preprints posted on 2026-10-13 take a serving parameter that is usually fixed in advance, the draft configuration for self-speculative decoding, the split of a KV-cache quantization budget, and each attention head's share of a compressed cache, and set it from measurements instead. Each reports gains over the fixed version, and each is specific about where those gains are smaller.

Uniform settings leave throughput and accuracy unclaimed

All three papers begin by naming a setting that existing methods treat as fixed. For self-speculative decoding of vision-language models, the SP2ec authors note that efficiency depends on choosing a suitable draft configuration for each prompt. For KV-cache quantization, the MultiAlloc authors argue that the practical problem is not compression alone but how to allocate limited quantization resources among different precision-preservation strategies under a fixed memory budget. For cache compression that selects tokens from an observation window, the ChampionKV authors point out that budgets are allocated uniformly or by a predefined scheme across attention heads, overlooking input-dependent differences in how much each head needs.

The three answers differ in when the measurement is taken: during generation for each query, offline before deployment, or from the prompt itself at the start of each request. That difference shapes what each method costs and what it can adapt to.

SP2ec: a bandit over draft configurations, scored in wall time

Huang, Li, Hou, Zhang, Pan, and Tan start from a limitation of speculative decoding for vision-language models: the diversity of input modalities makes it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding avoids a separate drafter by reusing parts of the target model, which keeps the model's multimodal capability, but its efficiency then depends on picking the right draft configuration for each prompt.

SP2ec, Single-Peak Speculative decoding, is a training-free bandit algorithm that adapts the draft configuration during generation. It exploits a single-peak throughput pattern the authors observe in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and their own simple cosSim method. The algorithm is designed to minimize what the authors call wall-time regret for each query, a quantity that corresponds to the empirical wall-time latency. Their theoretical guarantees on wall-time regret show that SP2ec can exploit the optimal draft configuration with negligible exploration cost. In experiments on two 7B-class vision-language models across visual and textual tasks, they report consistent improvements of up to 11 percent over the original self-speculative methods.

MultiAlloc: splitting a fixed KV memory budget between layers and tokens

Zhou, Wu, Jiang, Ren, Han, and Zhao note that the KV cache is a dominant source of memory consumption during inference, particularly in long-context and complex reasoning workloads. Existing quantization methods work at different granularities, adjusting precision layer by layer or preserving information for selected tokens. Under a fixed memory budget, the question becomes how to divide limited quantization resources between those strategies.

MultiAlloc is a profiling-guided precision allocation framework that distributes resources between layer-wise precision enhancement and token-wise preservation. Lightweight offline profiling characterizes layer sensitivity and token importance. Guided by the profiled memory-accuracy trade-off, the framework derives an allocation strategy automatically, with no online allocation search, and applies the resulting policy during inference. On reasoning and long-context benchmarks, the authors report consistent improvements over layer-wise, token-wise, and recent multi-granularity KV quantization methods under matched memory budgets, across models from four families.

ChampionKV: giving dispersed attention heads more of the cache

Chen, Guo, Liu, Ruan, Xu, and Wang build an input-adaptive, head-wise budget allocator for long-context KV-cache compression. ChampionKV uses the maximum aggregated observation-window attention score as a lightweight empirical proxy for how concentrated a head's attention is. Heads with dispersed attention receive larger budgets and heads with concentrated attention receive smaller ones. An abnormal-head mask and a floor budget improve robustness when attention responses are weak. The allocator plugs into compressors such as SnapKV and PyramidKV without changing how they select tokens.

On LongBench and RULER, both ChampionKV variants consistently improve over their SnapKV and PyramidKV base compressors. The improvements are largest, and statistically significant, on the largest evaluated model, a 70B-class instruction-tuned model, especially under tight budgets; on the tested 7B and 8B models the results are competitive but mixed. The authors also report that ChampionKV does not uniformly outperform Ada-KV, whose global cross-head Top-K selection keeps an advantage in several settings. A controlled analysis shows more attention mass retained than with the corresponding base compressors, at negligible overhead.

When the measurement is taken matters

SP2ec measures online. It spends a small amount of exploration within each query to find the best draft configuration and, by the authors' guarantee, that exploration cost is negligible. The benefit is that it adapts to each prompt, including the mix of visual and textual inputs that motivated the work.

MultiAlloc measures offline. Profiling happens before deployment and no allocation search runs during inference, so the runtime cost of adaptation is zero. The trade-off is that the policy reflects the profiled layer sensitivity and token importance; how well it transfers depends on how closely production traffic resembles the profiling workload.

ChampionKV measures per request from the prompt's own attention, using a cheap proxy computed over the observation window. It adapts to each input without a search, but the proxy is a heuristic, and the authors' own results show where it falls short: mixed gains on smaller models and an advantage retained by a method that selects tokens globally across heads.

What serving teams should take from the three papers

All three are preprints, and the figures are the authors' own, measured on their chosen models, benchmarks, budgets, and baselines.

Sources