synodic-ai ML SYSTEMS
ML Systems Desk · Decode-Time Memory Traffic

Moving Fewer Bytes per Token: Replayed SSM State, Load-Adaptive KV Residency, and Calibration-Free 4-Bit Scales

2026-10-13

Three arXiv preprints posted on 2026-10-09 treat LLM decoding as a memory-traffic problem and attack it at three different stores: the recurrent state of state-space layers, the key-value cache of long-context attention, and the weights and activations themselves. Each changes what has to be read, written, or kept resident on a decoding step, and each reports gains against a named prior method.

Decoding is bounded by what moves, not by what is computed

All three papers start from the same observation in different words. The ReplaySSM authors note that state-space models keep a fixed-size recurrent state, but reading and writing that state at every decoding step produces large state I/O with low arithmetic intensity and can bottleneck hybrid models. The Adaptive DHiSparse authors note that long-context serving runs out of KV memory before the hardware reaches its useful throughput. The DualQuant authors note that accelerators with native 4-bit support deliver their full throughput gains only when both weights and activations are quantized, and that activations are the harder half because existing methods typically need calibration data.

In each case the constraint is a store that must be read, written, or kept resident on every step, and the gain comes from changing what lives in that store rather than from faster arithmetic. The third removes the calibration step that activation quantization typically requires.

ReplaySSM: cache the inputs, replay the recurrence

Liou, Nover, Naik, Skliar, Shovkun, and Shahaf identify three costs of eager state management, in which the SSM state is read and written at every decoding step. The first is the state I/O itself, which has low arithmetic intensity. The second is that speculative decoding, which must discard rejected draft tokens, has no cheap way to roll the state back. The third is that when the state is quantized, the quantization error is fed into every subsequent recurrent update.

ReplaySSM unrolls the recurrence so that the state is represented as a checkpoint plus a buffer of recent SSM inputs. This separates output computation from checkpoint updates, and the checkpoint is updated only periodically. Because ReplaySSM avoids state writes on most steps, the authors report that the dominant state memory traffic roughly halves.

The same representation changes speculative decoding. With recent inputs retained, rolling back rejected tokens becomes a buffer-pointer operation, and verification can compute outputs without materializing intermediate states. It also permits two precisions: a low-precision checkpoint serves output computation on most steps, while a higher-precision checkpoint serves the recurrent updates, so rounding error in the low-precision copy never enters later updates.

The implementation is in vLLM, for two SSM layer types used in hybrid models. At a serving batch size of 256 the authors report standard decoding speedups of 1.20 to 1.47 times over a tuned vLLM baseline across four hybrid models, and about 1.8 times the throughput of vLLM's speculative decoding on the two 120B-class models. Keeping separate state precisions also reduced output KL divergence and downstream accuracy losses compared with a single shared low-precision checkpoint.

Adaptive DHiSparse: let KV residency follow load

Naji Alazhar and Wassel Alazhar build on HiSparse, which stores attention history in host memory and fetches only the selected entries. They identify two limits in HiSparse: it uses one cache budget at every load level, and it does not handle draft-model speculative decoding schemes such as DSpark and DFlash2.

Adaptive DHiSparse addresses both. Residency adapts to load: the required working set is protected, additional history is retained while GPU memory is spare, and the rest is recovered through exact fused retrieval. DSpark and DFlash2 keep their own proposal and acceptance rules; the integration work is in the memory path. Queries share missing-row retrieval, the draft model's KV uses a window-bounded ring, and verification staging is reused across layer groups.

At 100K context and concurrency 128, the first evaluated model served with DFlash2 delivers 793 output tokens per second per decode GPU, 1.45 times HiSparse and 1.27 times non-speculative adaptive serving. Held to the same latency bounds as HiSparse, the system sustains up to 4 times the concurrency and 2.3 to 3.7 times the output, depending on the per-token bound. On the second evaluated model at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85 times the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2 percent to 1.4 percent of selected rows at low load and releases memory under pressure.

DualQuant: one solve for weight error and activation scales

Salmani, Sharify, and Mostafa address the weights and activations themselves. On accelerators with native 4-bit support, realizing the full throughput gain requires quantizing both. Post-training weight quantization has been studied extensively, but activation quantization remains considerably more challenging and typically requires calibration data.

DualQuant is a dual-scale, calibration-free post-training quantization framework that directly minimizes weight-reconstruction mean squared error. The key observation is that the scaling factors obtained by minimizing weight-reconstruction error also serve as effective per-channel activation rescalings, applied before activation quantization. Competitive activation quantization is therefore a by-product of the weight problem and needs no calibration set. The authors formulate the problem as a non-differentiable weight reconstruction and solve it with variable splitting and block coordinate descent, in which each subproblem has a closed-form update.

The method supports the full MX family of formats and NVFP4, and the authors state that, to their knowledge, it is the first dual-scaling method to formally support MX and NV block formats. Across two model families evaluated on WikiText-2, C4, and zero-shot downstream tasks, DualQuant matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats.

Speculative decoding changes what the memory system has to support

Two of the three papers are as much about speculative decoding as about memory. In both, the obstacle to combining speculation with a memory-saving layout is bookkeeping: rejected draft tokens must be undone, and verification would otherwise materialize intermediate state or repeat cache fetches.

ReplaySSM handles this by construction: rollback is a pointer move over the input buffer, and verification computes outputs only. Adaptive DHiSparse handles it by sharing work: queries share retrieval of missing rows, the draft model's cache lives in a window-bounded ring, and verification staging is reused across layer groups. Both report results against a speculative baseline, about 1.8 times vLLM's speculative decoding for ReplaySSM on the 120B-class models, and 1.85 times resident serving with DSpark for Adaptive DHiSparse at 500K context.

The practical reading is that speculative decoding is not a free addition to a memory-optimized serving stack. Eager SSM state leaves speculation without cheap rollback, and HiSparse as published does not handle draft-model speculation at all. Both papers show layouts in which the memory saving and the speculative speedup compose instead of competing.

What serving teams should take from the three papers

All three are preprints posted on 2026-10-09, and the figures above are the authors' own, measured on their chosen models, batch sizes, context lengths, and baselines. They are not drop-in results for a production stack.

Sources