synodic-ai ML SYSTEMS
ML Systems Desk · Inference Systems

Calling the Target Model Only When Needed: Trajectory Triggers, Bit-Shared Drafts, and Flash Beside HBM

2026-10-19

Three papers posted to arXiv on 2026-10-16 attack the cost of LLM decoding at three layers: when a large target model is called, what a draft model is made of, and where the bytes that do not fit in HBM should live. Each replaces a fixed rule with a measured one, and each reports where its gains stop.

Calling the target only when the trajectory says so

Speculative decoding and similar dense-assistance methods repeatedly verify draft tokens with a target model, which keeps target computation coupled to generation. Amit Ranjan Trivedi and colleagues study a sparser arrangement: a draft model generates most tokens, and the target is invoked only for short corrective segments. The whole design then depends on deciding when to call the target.

Their first result is negative. Local uncertainty signals are weak triggers: entropy, negative log-likelihood, logit margin, top-k mass and velocity features predict draft failure at near chance, with AUC between 0.50 and 0.53. Aggregate features describing the shape of the draft's confidence trajectory do much better, reaching AUC 0.86 with a lightweight logistic predictor trained only from draft correctness labels.

That creates a timing tradeoff. Early calls have more leverage to redirect the trajectory but rest on weaker evidence; later calls have stronger evidence but less ability to change the outcome. Used online, the learned trigger improves on the best open-loop schedule by 5.3 to 20.0 accuracy points across five GSM8K draft-target pairs, and by 5.0 to 8.3 points on StrategyQA, showing transfer beyond arithmetic. On MATH-500 it gains 7.0 to 13.5 points when early prediction and target competence align, and loses when either is weak. The paper was accepted at the EMNLP 2026 uncertainty workshop.

Drafts carved out of the target's own bits

Jiwoo Kim and colleagues start from the serving side. Workloads span a wide range of batch sizes, context lengths and performance requirements. Conventional speculative decoding needs a separate draft model and KV cache, and existing self-speculative methods suffer from draft configurations that ignore the workload and from inefficient mixed-precision execution.

BS-Spec stores the target in block floating point (BFP) and derives several draft representations from that single model by partially loading the most significant bits of the weights and the KV cache while preserving the shared exponents, so no separate draft model has to be stored. Offline calibration picks, for each serving regime, the weight and KV-cache precision pair that maximizes estimated speedup, and a precision lookup table applies the chosen configuration at runtime. To execute those varying precisions efficiently, the authors design a mixed-precision BFP accelerator with a lookup-table-based variable-precision datapath used for both draft and target computation.

Reported results are up to a 1.9x speedup over conventional autoregressive decoding, and an average 67.2 percent improvement in speedup over the bit-sharing self-speculative baselines evaluated. The authors present the result as evidence for co-designing bit sharing, precision adaptation and specialized hardware together.

Flash next to HBM, and what it costs in endurance

Jinheng Li and colleagues look at capacity. High Bandwidth Flash (HBF) offers terabyte-scale, accelerator-local capacity at bandwidths approaching HBM, with a potential cost-per-bit advantage. Its NAND foundation, however, brings long latency, out-of-place updates and finite write endurance directly into the accelerator memory hierarchy, which makes its effect on performance and device lifetime hard to predict.

HBFSim is an LLM-aware, event-driven simulator for heterogeneous HBM plus HBF systems. It preserves application-level memory semantics while modeling flash-internal behavior, and it simulates HBM, HBF with host-managed flash translation, and conventional capacity tiers together, driven by a calibrated serving frontend. The authors use it to study decoding for a 70-billion-parameter model and a 235-billion-parameter model with 22 billion active parameters, under three workloads that differ in input and output length.

Three findings stand out. HBF serves data that does not fit in HBM 11.0 to 19.0 times faster than host DRAM. HBM is better spent holding the running sequences' KV cache than acting as a cache in front of HBF. And at a published peak load for the 70-billion-parameter model, HBF lasts 2.0 to 6.9 years, in proportion to the number of stacks that share the KV writes. The simulator artifact is public and is meant to track HBF device data as it becomes available.

Measured policies, stated limits

The three papers share a method. The sparse-decoding work replaces token-level uncertainty thresholds with a predictor learned from the whole confidence trajectory. BS-Spec replaces one fixed draft with a precision pair calibrated per serving regime and looked up at runtime. HBFSim replaces intuition about tiering with simulation that includes flash wear, and its answer, keep the running KV cache in HBM rather than using HBM as a cache for flash, is a concrete placement rule.

Each also reports where its gains end. Sparse correction loses on MATH-500 when early prediction or target competence is weak. BS-Spec's headline speedup is a maximum, and the design includes a purpose-built accelerator. HBF lifetime scales with how many stacks share the KV writes, so concentrating KV writes on fewer stacks shortens it. None of the three is a drop-in default; each is a measurement method that a serving team can apply to its own models and traffic.

Engineering takeaways

Each result comes from the authors' own models, workloads, hardware design or simulator, so treat these as hypotheses to validate on your stack.

Sources