DISENTANGLING CONTENT AND FINE-GRAINED PROSODY INFORMATION VIA HYBRID ASR BOTTLENECK FEATURES FOR VOICE CONVERSION - Demo

ABSTRACT

Non-parallel data voice conversion (VC) have accomplished considerable breakthroughs recently through introducing bottleneck features (BNFs) extracted by an automatic speech recognition(ASR) model. However, selction of BNFs have a significant impact on VC result. For example, when extracting BNFs from ASR trained with Cross Entropy loss (CE-BNFs) and feeding into neural network to train a VC system, the timbre similarity of converted speech is significantly degraded. If BNFs are extracted from ASR trained using Connectionist Temporal Classification loss (CTC-BNFs), the naturalness of the converted speech may decrease. This phenomenon is caused by the difference of information contained in BNFs. In this paper, we proposed an any-to-one VC method using hybrid bottleneck features extracted from CTC-BNFs and CE-BNFs to complement each other advantages. Gradient reverse layer and instance normalization were used to extract prosody information from CE-BNFs and content information from CTC-BNFs. Auto-regressive decoder and Hifi-GAN vocoder were used to generate high-quality waveform. Experimental results show that our proposed method achieves higher similarity, naturalness, quality than baseline method and reveals the differences between the information contained in CE-BNFs and CTC-BNFs as well as the influence they have on the converted speech.

Conversion Tasks

Our goal is to convete the timbre to target speaker while preserve the fine-grained prosody information contaioned in source waveform, which lead to high naturalness.

We will generate waveforms with high timbre similarity, naturalness and quality.

• Proposed - the proposed hybrid BNFs VC system.
• Baseline - the original Recognition-synthesis VC system using BNFs or PPGs.

Below are a few audio samples from a private spkr.

Below are a few audios samples from a public spkr.

Below are a few demo audios.

CE-BNFs And CE-PPGs Baseline model

BNFs or PPGs extracted from ASR model trained with Cross-Entropy loss are fed into Baseline VC model.

CTC-BNFs And CTC-PPGs Baseline model

BNFs or PPGs extracted from ASR model trained with CTC loss are fed into Baseline VC model.

Proposed Method

Proposed hybrid ASR BNFs VC model and Baseline model using CE-BNFs or CTC-BNFs as input.