Speculative decoding's effectiveness is governed by two quantities: the acceptance rate of drafted tokens and the cost of producing them.
We show the two objectives can be co-optimized with algorithm-system co-design, and present JetSpec++, which retains a strong learned draft model while hiding its cost almost entirely.
JetSpec++ combines three components:
- intra-block self refinement, which trains the draft model to iteratively correct its own parallel predictions within a block of tokens;
- inter-block foresight drafting, which begins the next block before the current one is verified;
- and pipelined drafting and verification, which overlaps the two stages so drafting's marginal wall-clock cost approaches zero.
In comparison with state-of-the-art speculative decoding methods, JetSpec++ attains a wall-clock speedup of up to $7.45\times$ over autoregressive (AR) decoding and $1.54\times$ over DSpark, with up to 1,361 tokens per second for a single request on GB200, and an average acceptance length of 8.94 across seven benchmarks, while preserving the target model's output distribution.
Our code and models are available at https://github.com/hao-ai-lab/JetSpec.