AI模型协同设计：使用推测解码加速LLM推理

Developer Tools & Techniques Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference Sep 02, 2026 By Tiyasa Mitra, Maximilian Golub, Maor Ashkenazi, Samkit Jain, Nidhi Bhatia and Bita Darvish Rouhani Like L T F R E AI-Generated Summary Speculative decoding accelerates LLM inference by having a small draft model propose multiple tokens that a larger target model verifies in parallel, reducing decoding iterations while preserving output accuracy. Five guidelines help select the optimal draft length and mechanism across the Pareto frontier: push GEMMs into the compute-bound region without increasing KV cache pressure; when attention dominates, set draft length to 128/G - 1; align larger draft lengths with attention kernel tile boundaries of 128; at very low latency, increase draft length only while acceptance gains justify the added draft cost; and choose the draft mechanism by balancing acceptance length, draft overhead, and training and deployment cost. Draft mechanisms compared include external draft models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods, each with different tradeoffs in training cost, serve-time memory, and speculation cost per draft tokens. SPEED-Bench, a speculative decoding benchmark developed by NVIDIA, provides realistic workloads for measuring acceptance length across task domains such as coding and summarization. Ready-to-run training examples for EAGLE-3, DFlash, and DSpark are available in NVIDIA/Model-Optimizer, including fine-tuning and quantization workflows demonstrated on Nemotron 3.5 Lightning. Next Steps Explore the NVIDIA/Model-Optimizer repository for training examples and quantization workflows. Benchmark acceptance length on SPEED-Bench with realistic prompts and diverse task domains. Quantify draft overhead using NVIDIA TensorRT LLM for high-performance inference. Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding and offers five guidelines for selecting draft length and draft mechanism across the Pareto frontier. For a discussion of how model design choices impact both throughput and interactivity without sacrificing accuracy, see AI Model Co-Design: Hardware-Friendly LLM Design (Part 1). For an explanation of how group size (query heads per KV head), head dimension, and sequence length shape the performance of dense attention, see Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference (Part 2). What is speculative decoding? Speculative decoding is a technique for accelerating the autoregressive decoding phase of LLM inference by predicting multiple tokens per iteration. A small draft model first predicts several likely next tokens. These tokens are then verified in parallel with a single pass through the larger target model. Figure 1. Speculative decoding moves LLM decoding toward a more compute-bound region, shifting the throughput-interactivity Pareto frontier up and to the right. As interactivity increases toward the right of the curve, batch size decreases, and GEMMs account for a larger share of runtime than attention This approach reduces the total number of decoding iterations while increasing the arithmetic intensity of the target model, without requiring higher concurrency. The target model accepts the proposed tokens in sequence until it encounters the first mismatch. The next prediction cycle then resumes from that position. Because only tokens accepted by the target model are retained, speculative decoding produces the same output sequence as standard decoding, unless the acceptance criteria are deliberately relaxed. Figure 2 illustrates this decoding flow. The draft length (\(D\)) is the number of proposed tokens per target iteration. The acceptance length (\(AL\)) is the number of tokens produced (that is, accepted) per target iteration. \(AL\) ranges from \(1\) to (\(1 + D\)) because the target can always produce one new ground-truth token in addition to the accepted draft tokens. The speedup from speculative decoding can be quantified as the ratio of time taken by the target model to sequentially generate \(AL\) tokens to the time taken to verify D tokens in parallel, while also accounting for the latency overhead of drafting \(D\) tokens: \(\mathrm{speedup} = \frac{T_{\mathrm{verif}}(B) \times \mathrm{AL}}{T_{\mathrm{verif}}\left(B \times (1 + D)\right) + T_{\mathrm{draft}}(B, D)}\) where \(B\) is the batch size, \(T_{\mathrm{verif}}(x)\) is the target verification time for \(x\) tokens, and \(T_{\mathrm{draft}}(b, y)\) is the time needed to generate a draft of length \(y\) for batch size \(b\). It is evident that to maximize speedup, it’s necessary to find the optimal \(\left(D, AL, T_{\mathrm{draft}}\right)\) tuple. Figure 2. The draft-then-verify flow of speculative decoding. A smaller draft model proposes \(D\) tokens, and the target model verifies them together in one pass Selecting the optimal draft length For simplicity, if we ignore draft model latency, speculation provides a speedup when: \(\frac{T_{\mathrm{verif}}\left(B \times (1 + D)\right)}{T_{\mathrm{verif}}(B)} \frac{128}{G} – 1\), prefer values where \(G \times (1 + D)\) is a multiple of 128 to avoid tile underutilization. The relative weight of Guideline 1 compared to Guidelines 2 and 3 depends on the percentage of runtime spent in FFNs compared to attention at the preferred operating point. Communication also adds verification cost as D increases, although compute and communication overlap can mitigate the overhead. Draft length at the far right of the Pareto curve Moving towards the far right of the Pareto curve, \(B\) becomes very small. Here, fixed kernel setup and post-processing overheads dominate for both compute and communication kernels. These fixed costs don’t grow significantly with the number of verified tokens, keeping verification overhead largely invariant with the number of draft tokens. While MoEs do see more experts activated with increasing draft length, a combination of model sharding strategies and efficient kernels such as Grouped GEMMs can keep this overhead low. Larger draft lengths can therefore help in the low latency region, as long as acceptance remains high. At very low latency, the number of sequential kernel launches determines workload latency. Because kernel launches scale linearly with layer count, for an autoregressive draft model with a similar layer structure to the target, the speedup can be approximated as: \(\mathrm{speedup} = \frac{L_{\mathrm{target}} \times \mathrm{AL}}{L_{\mathrm{target}} + D \times L_{\mathrm{draft}}}\) where \(L_{\mathrm{target}}\) and \(L_{\mathrm{draft}}\) are the layer counts in the target and draft models,  respectively. Defining a constant draft_depth_ratio, \(\rho = \frac{L_{\mathrm{draft}}}{L_{\mathrm{target}}}\): \(\mathrm{speedup} = \frac{\mathrm{AL}}{1 + \rho D}\) and draft overhead, \(O_d = \rho D\) In other words, increasing \(D\) helps only while the gain in \(AL\) is large enough to offset the draft overhead. Guideline 4: At very low latency, increase \(D\) only while the gain in \(AL\) justifies the added draft cost. Selecting a draft mechanism Selecting \(D\) tells you how many tokens to speculate. Next, decide how to generate these tokens to maximize speedup. A multitude of techniques have been proposed over the years, with different tradeoffs in training, parameter, and runtime costs. External drafting suggests using a small standalone LLM, while MTP, EAGLE-3, DFlash, and DSpark use auxiliary layers combined with information from the target model to predict tokens. Suffix and n-gram methods are model-free and instead reuse patterns already seen in the token stream. Table 1 compares key drafting methods, including how they generate tokens, their serve-time memory, and their draft overhead. External draft model EAGLE-3 MTP DFlash DSpark Suffix / n-gram Draft architecture Small LLM Decoder layer(s) + linear projection Decoder layer(s) + linear projection Decoder layer(s) + linear KV fusion Decoder layer(s) + linear KV fusion + lightweight Markov head String matching, no model Inputs Token IDs Previous-token embedding + target hidden states Target final hidden state Fused target hidden states used as draft KV Fused target hidden states used as draft KV Token stream Generation method \(D\) autoregressive forward passes Runs the module \(D\) times Runs the module \(D\) times Generates \(D\) tokens in one parallel step Generates \(D\) tokens in one parallel step, followed by a lightweight sequential correction One lookup Release method Separate model release Separate post-training head Part of the base model checkpoint Separate post-training head Separate post-training head Independent of target Training cost From scratch: 1T-10T+ tokens Distilled: 100B-400B Adapted: 10M-1B 1-10B additional tokens after target training Usually trained with the target during pretraining 1-10B additional tokens after target training 1-10B additional tokens after target training None Activation memory cost Draft weights + full KV cache Weights + small KV cache Weights + small KV cache Weights + small KV cache built from target Weights + small KV cache built from target None Speculation cost per \(D\) tokens Full draft model forward 1 layer, serial 1 layer, serial ~5 layers, once ~5 layers once + lightweight Markov head, serial O(1) lookup Best for LPU and LPU+GPU Not recommended for GPUs  Not recommended – lower \(AL\) than MTP  Best for larger models on GPUs Smaller models, batch size 1 Smaller models, batch size 1 Workloads with high repetition Table 1. Comparison of common draft mechanisms across inputs, generation method, training cost, serve-time memory, and draft cost To quantify the tradeoff between \(AL\) and \(O_d\), begin by observing how \(AL\) scales with \(D\). Figure 6 shows how AL changes as D increases on SPEED-Bench, with Qwen 3.5 122B A10B as the target. SPEED-Bench is a speculative decoding benchmark developed by NVIDIA with the goal of representing realistic production workloads. It covers multiple task domains such as coding and summarization and has a variety of splits at different input sequence lengths. We recommend SPEED-Bench for \(AL\) comparisons. On the 32K split, Qwen 3.5 35B A3B reaches an \(AL\) of 6 at \(D = 9\), while the 4B draft reaches an AL above 5. MTP and DFlash ALs level off as \(D\) grows. N-gram has lower acceptance on this workload and is better suited to workloads with repeated token patterns. Figure 6. \(AL\) versus \(D\) for different draft mechanisms on SPEED-Bench, throughput 32K split, with Qwen 3.5 122B A10B as the target Higher \(AL\) does not equal higher speedup. You also need to consider how much it costs to generate the draft. As shown in Figure 6, all external drafts reach higher \(AL\) than other draft methods at \(D > 3\). The Qwen 3.5 122B MTP is much smaller, with 2.5B total parameters and fewer than 150M active parameters. At \(D = 3\), a larger external draft may not be worth the added cost. At \(D = 21\), its higher AL may make that cost worthwhile, assuming an efficient serving strategy. Meanwhile, DFlash \(AL\)s level off relatively quickly. However, both DFlash and DSpark generate \(D\) draft tokens in parallel, lowering \(O_d\). For comparison, at minimum latency and with \(D=11\), a one-layer MTP head takes 11 steps, leading to \(O_d^{\mathrm{MTP}} = \frac{11}{L_{\mathrm{target}}}\), while a five-layer DFlash head generates the draft in one pass, giving \(O_d^{\mathrm{DFlash}} = \frac{5}{L_{\mathrm{target}}}\). For a large target model with many layers, both overheads are negligible, but as \(L_\mathrm{target}\) decreases, the draft overhead becomes significant. For small models at low latency, therefore, DFlash or DSpark can be the optimal choice even with lower \(AL\). To navigate the tradeoff between \(AL\) and draft latency, it is crucial to benchmark both accurately. In the context of speculative decoding, this means benchmarking with realistic prompts and covering a wide set of task domains. We recommend SPEED-Bench for \(AL\) measurements and high performance inference frameworks such as NVIDIA TensorRT LLM to quantify draft overhead. Beyond inference performance, draft training scope and costs deserve consideration.  While MTP must be co-trained with the target model, EAGLE, DFlash, and DSpark can be added to the final model checkpoint. Similarly, a continuum of options exists for external draft model creation. Training from scratch and distillation from the target yield the highest ALs, while adapting an existing model to the desired target through cross-model adaptation techniques can significantly reduce training costs at the expense of \(AL\). Fine-tuning the target model can change its output distribution and hidden representations. Learned drafters trained for a particular target checkpoint can therefore lose acceptance even when the target model improves. After changing the target, remeasure \(AL\) on representative workloads. The required adaptation depends on the draft mechanism. MTP is part of the target model and should continue to be trained during fine-tuning or be realigned in a dedicated follow-up stage. Target-attached drafters such as EAGLE-3, DFlash, and DSpark use the target’s hidden states and will need to be adapted to the updated checkpoint with additional training. External draft models do not consume target hidden states, but they still approximate the target’s output distribution and may require fine-tuning or distillation. Suffix and n-gram methods have no learned drafter and therefore require no retraining, although their effectiveness still depends on how much repetition appears in the deployed workload. Guideline 5: Choose the draft mechanism that gives the best decode speedup for your workload and hardware, considering \(AL\), draft latency, and training and deployment cost. Get started co-designing speculative decoding Use the five guidelines summarized below as a checklist for selecting \(D\) and draft mechanism across the Pareto frontier. Increase speculative decoding draft length to push GEMMs into a compute-bound region without increasing KV cache capacity pressure. When attention dominates decode time, choose \(D = \frac{128}{G} – 1\) as a starting point. For larger draft lengths, prefer values where \(G \times (1 + D)\) is a multiple of 128 to align with the attention kernel tile size. At very low latency, use a fast draft mechanism and increase \(D\) only while the gain in \(AL\) justifies the added draft cost. Choose the draft mechanism by balancing \(AL\) and draft overhead for your workload and hardware. Benchmark both under realistic serving conditions and consider training and deployment cost. For post-trained drafters, ready-to-run training examples are available in NVIDIA/Model-Optimizer for EAGLE-3, DFlash, and DSpark. See how we did this for NVIDIA Nemotron 3.5 Lightning: fine-tune DSpark, then quantize it to FP8 or NVFP4. Use the examples as starting points, then validate AL and end-to-end speed on your own model, workload, and hardware. Acknowledgments This post is a cross-team NVIDIA effort. We are grateful to Bhargava Gopireddy, Ritika Borkar, Dor Tsur, Andrii Skliar, Benjamin Chislett, Yaniv Galron, Talor Abramovich, Yoav Miron, Rabeeh Karimi Mahabadi, Roger Waleffe, Udi Karpas, Ran Zilberstein, Brian Pharris, and Eduardo Alvarez for their help. Like Tags Agentic AI / Generative AI | Data Center / Cloud | Developer Tools & Techniques | Hardware / Semiconductor | General | Intermediate Technical | Best practice | Deep dive | AI Inference | Inference Performance | LLMs | NVFP4 | Training AI Models About the Authors About Tiyasa Mitra Tiyasa Mitra is a senior AI and machine learning engineer at NVIDIA, where she leads work on cutting-edge technologies to improve generative AI efficiency through algorithm-hardware co-design. She has over a decade of experience in data-center optimization and networking, with a strong track record of building high-performance AI systems at scale. Tiyasa holds a master’s degree in electrical engineering from UCLA and a bachelor’s degree from IIT Kharagpur in India. View all posts by Tiyasa Mitra About Maximilian Golub Maximilian Golub is a senior ML engineer at NVIDIA, focusing on advancing speculative decoding algorithms for generative AI through thoughtful co-design. He has previously contributed to research on MoE scaling, structured sparsity, and the MXFP quantization formats. Maximilian holds a Master of Applied Science in Computer Engineering from the University of British Columbia, and a bachelor’s degree in Electrical Engineering from the University of Washington. View all posts by Maximilian Golub About Maor Ashkenazi Maor Ashkenazi is a team leader and senior researcher at NVIDIA, focusing on LLM inference algorithms. His recent work centers on speculative decoding, bridging model research and inference systems to deliver practical end-to-end speedups. Maor has completed his doctoral studies in Computer Science at Ben-Gurion University of the Negev. He also holds a master's degree in Computer Science and a bachelor’s degree in Software Engineering, awarded summa cum laude. View all posts by Maor Ashkenazi About Samkit Jain Samkit Jain is a senior deep learning performance architect at NVIDIA, focusing on optimizing the performance and energy efficiency of generative AI inference and systems at scale. Prior to NVIDIA, Samkit was a software research scientist at Intel, where he led hardware/software co-design for sparse AI and graph analytics accelerators. Over his career, he has worked across multiple layers of the stack, spanning architecture pathfinding, performance modeling, and HPC system software. He holds a master’s degree in Electrical Engineering from the University of Minnesota, Twin Cities. View all posts by Samkit Jain About Nidhi Bhatia Nidhi Bhatia is a senior AI/ML engineer at NVIDIA, driving model-hardware co-design to improve the efficiency of serving generative AI inference. She has worked across multiple layers of the stack, from GPU architecture and performance modeling to distributed inference algorithms, with her work shaping how large-scale AI inference systems are built and optimized. Nidhi holds a master's in electrical and computer engineering from Carnegie Mellon University and a bachelor's in electronics and instrumentation from BITS Pilani, India. View all posts by Nidhi Bhatia About Bita Darvish Rouhani Bita Darvish Rouhani is a senior distinguished engineer and manager at NVIDIA, leading algorithms, software, and hardware co-design initiatives for cost-optimized generative AI inference. Prior to joining Nvidia, Bita was a partner group manager at Microsoft, where she co-founded and led the OCP MX consortium. The consortium standardized the first set of 4- and 6-bit data types for AI training and inference for nearly all mainstream AI chips. Bita holds a Ph.D. in computer engineering from UC San Diego. View all posts by Bita Darvish Rouhani