synodic-ai ML SYSTEMS
ML Systems Desk · Memory-Bound Inference and Training

Shrinking What the Accelerator Reads: Preconditioner-Space Rounding, Value-Cache Folding, and Decode-Aware Pruning

2026-10-11

Three arXiv preprints posted on 2026-10-08 attack the same constraint from three sides: the bytes a GPU must read or keep resident per step. One redesigns 4-bit AdamW optimizer-state quantization around the space in which rounding decisions are made, one compresses the value cache by exploiting cross-layer symmetry, and one recalibrates pruning on the activations a model actually sees during decoding. Each shows that where you measure error matters as much as how much error you tolerate.

Memory traffic is the shared bottleneck

Large-model training and serving are increasingly limited not by arithmetic but by what must be stored and moved. During pretraining, AdamW keeps two moment tensors per parameter, so optimizer state is a persistent multiple of the model's own size. During long-context decoding, the key-value cache grows with sequence length and can dominate memory. During every decode step, the entire set of nonzero weights is read from memory to produce a single token, which makes decoding memory-bound rather than compute-bound.

Three papers posted the same day each target one of these consumers. What makes them worth reading together is a shared methodological move: each identifies a mismatch between where a prior technique measured its error and where that error actually affects the computation, then redesigns the technique around the correct coordinate.

Rounding space matters: ZIP-SR and ZE-EDEN for 4-bit AdamW states

Li, Tang, Braithwaite, Dexter, Neves, and Gupta start from a known problem: quantizing AdamW's optimizer states cuts persistent storage, but quantization errors propagate through the moment recurrences and perturb every subsequent adaptive update. Their contribution is to reframe the design question in terms of rounding space, defined as the coordinate in which a quantizer chooses between adjacent reconstruction levels. The second moment is not used directly; it enters the update through the preconditioner, and the paper asks whether rounding should be decided in state space or in preconditioner space.

A local analysis of the quantization cell adjacent to zero shows that a small mean error in the stored second-moment state need not imply a small mean error in the preconditioner at the next step, because the preconditioner transformation is highly sensitive near zero. A one-dimensional quadratic construction then shows that state-space rounding and preconditioner-space rounding produce qualitatively different optimization dynamics, not merely different error magnitudes.

Two designs follow. Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR) retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space, so the expected preconditioner, rather than the expected state, is preserved. Zero-Excluding EDEN calibration (ZE-EDEN) takes the complementary route: it uses a zero-excluding second-moment codebook and rescales the quantized second-moment block to offset the preconditioner distortion introduced by a positive quantization floor. Both configurations use 4-bit NormalFloat (NF4) for the first moment and apply targeted stochastic rounding to the LM-head first moment during the final 10% of training. Across GPT- and Llama-style pretraining runs from 130M to 2.7B parameters, both methods improve on the mean validation loss of the TorchAO 4-bit AdamW baseline.

VFold: symmetry in the value cache across layers

Verma, Kim, Murray, and Duh target the KV cache, which accelerates decoding by storing per-layer key and value states but can dominate memory at long context lengths. One line of prior work compresses the cache by exploiting similarities between layers, but most existing techniques require architectural changes to the model and incur substantial overhead, which limits their use on deployed checkpoints.

VFold proposes a symmetry-aware value-cache merging strategy that reduces cache memory while avoiding both harmful performance degradation and architectural overhead during decoding. The method operates on the value cache specifically, and the authors argue that this is a major source of underutilized capacity: value states across layers carry enough shared structure that merging them costs little in output quality.

The result that matters for deployment is composability. The authors show that value-cache merging can be applied alongside existing cache compression techniques, composing with high-ratio quantization or with key-cache pruning to reach compression ratios that neither method achieves alone, at minimal additional cost. Because the technique is applied at decode time without retraining or architectural modification, it slots into a serving stack as one more stage in the cache-compression pipeline rather than as a replacement for the stages already there.

SparseDecoding: calibrate the Hessian on what the model actually sees

Wang, Niu, Su, Zhao, Zhu, and Wang address the memory-bound decode stage through pruning. Layer-wise, training-free pruning guided by the Hessian is an established approach because fewer nonzero parameters means fewer bytes read per decode step. The paper identifies two problems with how that line of work is typically practiced.

The first is a distribution shift in calibration. Typical methods compute the Hessian from pre-collected natural sequences, but during decoding the model is fed its own generated tokens. The Hessian calculated on natural text differs from the one calculated on generated text, and the authors observe that this discrepancy causes the activation distribution during generation to deviate from the distribution the pruning was optimized for, which further degrades the pruned model. The second problem is at the kernel level: most LLM pruning methods that deliver real speedup target sparse matrix-matrix (SpMM) multiplication, which is the shape of prefill, and offer limited support for the sparse matrix-vector (SpMV) operations that dominate decoding.

SparseDecoding responds on both axes. Algorithmically, it constructs calibration matrices from layer-wise activations collected during the dense model's own autoregressive generation, explicitly excluding prefill, so the pruning objective is aligned with the activations present at decode time. On the system axis, the paper targets the SpMV path that the authors identify as the dominant operation during decoding, so that the sparsity achieved by the algorithm translates into reduced memory traffic where it is actually incurred.

The common thread: measure error where it lands

Each paper's gain comes from the same correction. ZIP-SR and ZE-EDEN move the rounding decision from the stored second moment to the preconditioner, because that is where the error perturbs the update. SparseDecoding moves Hessian calibration from a natural-text corpus to the model's own decode-time activations, because that is the distribution the pruned weights will actually see. VFold identifies the value cache, rather than the KV cache as a whole, as the component with redundant capacity, and compresses there without touching the architecture.

The practical pattern for systems engineers is to ask, for any compression technique, two questions: in which coordinate is the error being minimized, and in which coordinate does it actually affect the output? When the two differ, the technique is optimizing a proxy, and the three papers show that closing that gap is worth more than tightening the proxy.

Deployment considerations

None of the three techniques should be adopted on the strength of its paper alone, but each is cheap to evaluate. For training teams already running 4-bit optimizer states, ZIP-SR or ZE-EDEN is a drop-in change to the quantizer, and the reported gains across 130M to 2.7B models give a clear expectation to test against at larger scale. For serving teams, VFold's composability claim is the one to verify: measure the compression ratio and quality with the existing quantization or key-pruning stage left in place, since that is the configuration that would ship. For anyone pruning for decode throughput, SparseDecoding's calibration recipe is a one-line change in data collection, and the kernel question, whether the deployed runtime actually accelerates SpMV, should be settled before any sparsity ratio is chosen.

The broader lesson is that the memory wall in modern LLM systems is being pushed back less by new hardware than by more careful accounting of where error and bytes actually accumulate. All three papers are, at root, measurement corrections. That they arrive together on one day is a reasonable signal of where the field's attention is.

Sources