Production Inference Architecture: Latency, Speculative Decoding, and Quantization Tradeoffs at Scale

Serving large language models in enterprise production requires navigating fundamental hardware tradeoffs between compute utilization and memory bandwidth. While the prefill phase is compute-bound and highly parallelizable, autoregressive decoding is strictly memory-bandwidth-bound, requiring specialized systems techniques to achieve low Time-to-First-Token (TTFT) and high token generation throughput.

The Memory Bandwidth Bottleneck in Autoregressive Generation

During single-batch token generation, a transformer model must read every model weight and the entire Key-Value (KV) cache from High Bandwidth Memory (HBM) into SRAM for every single generated token. With arithmetic intensity dropping below 1 FLOP/byte during decoding, GPU tensor cores spend the vast majority of execution cycles stalled on memory access.

To overcome this bottleneck, production serving runtimes rely on three interconnected architectural optimizations: KV-cache memory management, speculative decoding, and post-training quantization.

1. KV-Cache Memory Management: PagedAttention and MLA

Standard contiguous KV-cache allocation suffers from severe fragmentation (up to 60–80% memory waste). PagedAttention (Kwon et al.) partitions the KV cache into non-contiguous virtual memory blocks, enabling dynamic allocation and near-zero memory waste. In advanced architectures, Multi-Head Latent Attention (MLA) compresses the KV cache by projecting key and value representations into a low-dimensional latent space, reducing per-token KV memory footprint by up to 85% compared to standard Multi-Head Attention (MHA).

2. Speculative Decoding: Modified Rejection Sampling

Speculative decoding (formalized by Leviathan et al.) accelerates decoding by pairing a large target model with a fast, lightweight draft model (e.g., an 8B draft for a 70B target model). The draft model speculatively generates a sequence of $K$ candidate tokens in a fraction of the time. The target model then verifies all $K$ tokens simultaneously in a single compute-bound forward pass.

To guarantee that the output distribution remains strictly mathematically identical to the target model's native distribution, each draft token $x$ is accepted with probability:

$lpha = \min\left(1, rac{p(x)}{q(x)}ight)$ $ ext{where } p(x) ext{ is the target probability and } q(x) ext{ is the draft probability.}$

When a draft token is rejected, the target model samples a replacement token from the residual distribution $\max(0, p(x) - q(x))$, discarding subsequent speculative tokens. In high-acceptance regimes ($lpha > 0.75$), speculative decoding delivers 2.0–2.8x end-to-end latency reductions without altering model output quality.

3. Quantization Dynamics: FP8 vs. INT4 / AWQ

System Synthesis

Modern production inference clusters achieve optimal cost-performance by combining continuous batching (which dynamically interleaves prefill and decode phases), FP8 weight and KV-cache quantization, and speculative verification pipelines. This unified architecture maximizes GPU memory bus saturation while maintaining sub-20ms inter-token latencies under heavy multi-tenant load.