A speculative decoding method called 'JetSpec' has been developed that can speed up AI by up to 9.64 times.



Hao AI Lab , an AI lab at the University of California, San Diego, has developed a speculative decoding method called ' JetSpec '.

JetSpec: Parallel Tree Drafting
https://jetspec-project.github.io/jetspec-web/

JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting | Hao AI Lab @ UCSD
https://haoailab.com/blogs/parallel-tree-decoding/

Mainstream large-scale language models output long sentences by 'continuously predicting the next token.' Speculative decoding is a speed-up technique that uses a small drafting AI model to predict multiple next words and then selects one of them, allowing for speed improvements while maintaining the quality of the AI model.

Existing speculative decoding methods can be broadly divided into two types: 'autoregressive' and 'block-diffusion.' According to Hao AI Lab, autoregressive methods suffer from the problem of 'wasteful predictions due to long forecasts,' while block-diffusion methods suffer from the problem of 'wasteful predictions due to contradictory prediction trees.' JetSpec is a method that solves these problems and enables even faster processing compared to existing methods.

The graph below compares the speed improvement rates of different speculative decoding methods when running Qwen3-8B on an NVIDIA H100. JetSpec achieved a 9.64x speedup in the mathematical inference benchmark MATH-500, resulting in faster processing than existing methods. It also achieved a 4.58x speedup in MT-Bench, which measures complex chat capabilities.



Hao AI Lab has

developed a version of its vLLM AI inference engine that adds JetSpec execution capabilities, and is running Qwen3-8B using an NVIDIA B200. As a result, they achieved extremely fast output of over 1000 tokens per second.



Clicking the images below will allow you to see the inference speeds of 'Standard Qwen3-8B,' 'Qwen3-8B accelerated with DFlash,' and 'Qwen3-8B accelerated with JetSpec.'



Hao AI Lab has released JetSpec draft models for 'Qwen3-8B', 'Qwen3 30B A3B', 'Qwen3.6 35B A3B', 'gpt-oss-20b', 'Gemma 4 26B A4B IT', and 'Step 3.7 Flash' at the following link.

JetSpec
https://huggingface.co/JetSpec



Additionally, papers and related code concerning JetSpec have been made publicly available.

[2606.18394] JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
https://arxiv.org/abs/2606.18394

GitHub - hao-ai-lab/JetSpec: JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Causal Parallel Tree Drafting · GitHub
https://github.com/hao-ai-lab/JetSpec

Related Posts:

in AI, Posted by log1o_hf