Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding
Abstract: Modern autoregressive speech synthesis models leveraging language models have demonstrated remarkable performance. However, the sequential nature of next token prediction in these models leads to significant latency, hindering their deployment in scenarios where inference speed is critical. In this work, we propose Speech Speculative Decoding (SSD), a novel framework for autoregressive speech synthesis acceleration. Specifically, our method employs a lightweight draft model to generate candidate token sequences, which are subsequently verified in parallel by the target model using the proposed SSD framework. Experimental results demonstrate that SSD achieves a significant speedup of 1.4x compared with conventional autoregressive decoding, while maintaining high fidelity and naturalness. Subjective evaluations further validates the effectiveness of SSD in preserving the perceptual quality of the target model while accelerating inference.
Figure 1: Speech Speculative Decoding (SSD) framework. The dashed line depicts autoregressive decoding in the draft model, followed by parallel verification by the target model. The prefix comprises all prompt tokens and previously generated tokens during inference. Green indicates accepted tokens, while red represents rejected tokens.
Speech Samples
Prompt Text & Speech
Target Text
Draft Model
CosyVoice 2
CosyVoice 2 + SSD
I get wet to my knees when I go down to pick cherries.
"So you must," cried Jasper, enchanted at his success in amusing; "and I'll go with you."
The case is not entirely devoid of interest.
For an instant I imagined that Bannister had taken the unpardonable liberty of examining my papers.
born but to smile and fall. Ah!
Dearest, teach me so To pour out gratitude, as thou dost, good!
A great saint, saint Francis Xavier!
The bell rang. The master marked the sums and cuts to be done for the next lesson and went out.
I say, "whether color be gay or sad."
Every plant in the grass is set formally, grows perfectly, and may be realized completely.
I want it to bring my Patchwork Girl to life.
"I never knew I was called the Unlucky," said Ojo, soberly; "but it is really a good name for me."
"She was ten years older," said her husband.
This I took for a sign that he had himself something to produce and that we should only have to wait.
"Cut his bands," said Hawkeye to David, who just then approached them.
At last the scout spoke in English, and at once explained the embarrassment of their situation.
Up in the sick room Zora lay on the little white bed.
The Sun, the Swamp? Then finding all well, she closed her eyes and slept.
The sail stretches tight like a bubble ready to burst.
What a very bad notion that was of his, I thought to myself, to take soundings just here!
