Introduction

It is for the dataset from https://en.data-baker.com/datasets/freeDatasets/

The dataset contains 10000 Chinese sentences of a native Chinese female speaker, which is about 12 hours.

Note: The dataset is for non-commercial use only.

matcha

./matcha contains the code for training Matcha-TTS

Checkpoints and training logs can be found here. The pull-request for this recipe can be found at https://github.com/k2-fsa/icefall/pull/1849

The training command is given below:

python3 ./matcha/train.py \
  --exp-dir ./matcha/exp-1/ \
  --num-workers 4 \
  --world-size 1 \
  --num-epochs 2000 \
  --max-duration 1200 \
  --bucketing-sampler 1 \
  --start-epoch 1

To inference, use:

# Download Hifigan vocoder. We use Hifigan v2 below. You can select from v1, v2, or v3

wget https://github.com/csukuangfj/models/raw/refs/heads/master/hifigan/generator_v2

python3 ./matcha/infer.py \
  --epoch 2000 \
  --exp-dir ./matcha/exp-1 \
  --vocoder ./generator_v2 \
  --tokens ./data/tokens.txt \
  --cmvn ./data/fbank/cmvn.json \
  --input-text "当夜幕降临，星光点点，伴随着微风拂面，我在静谧中感受着时光的流转，思念如涟漪荡漾，梦境如画卷展开，我与自然融为一体，沉静在这片宁静的美丽之中，感受着生命的奇迹与温柔。" \
  --output-wav ./generated.wav

soxi ./generated.wav

prints:

Input File     : './generated.wav'
Channels       : 1
Sample Rate    : 22050
Precision      : 16-bit
Duration       : 00:00:17.31 = 381696 samples ~ 1298.29 CDDA sectors
File Size      : 763k
Bit Rate       : 353k
Sample Encoding: 16-bit Signed Integer PCM

https://github.com/user-attachments/assets/88d4e88f-ebc4-4f32-b216-16d46b966024

To export the checkpoint to onnx:

python3 ./matcha/export_onnx.py \
  --exp-dir ./matcha/exp-1 \
  --epoch 2000 \
  --tokens ./data/tokens.txt \
  --cmvn ./data/fbank/cmvn.json

The above command generates the following files:

-rw-r--r-- 1 kuangfangjun root 72M Dec 27 18:53 model-steps-2.onnx
-rw-r--r-- 1 kuangfangjun root 73M Dec 27 18:54 model-steps-3.onnx
-rw-r--r-- 1 kuangfangjun root 73M Dec 27 18:54 model-steps-4.onnx
-rw-r--r-- 1 kuangfangjun root 74M Dec 27 18:55 model-steps-5.onnx
-rw-r--r-- 1 kuangfangjun root 74M Dec 27 18:57 model-steps-6.onnx

where the 2 in model-steps-2.onnx means it uses 2 steps for the ODE solver.

HINT: If you get the following error while running export_onnx.py:

torch.onnx.errors.UnsupportedOperatorError: Exporting the operator
'aten::scaled_dot_product_attention' to ONNX opset version 14 is not supported.

please use torch>=2.2.0.

To export the Hifigan vocoder to onnx, please use:

wget https://github.com/csukuangfj/models/raw/refs/heads/master/hifigan/generator_v1
wget https://github.com/csukuangfj/models/raw/refs/heads/master/hifigan/generator_v2
wget https://github.com/csukuangfj/models/raw/refs/heads/master/hifigan/generator_v3

python3 ./matcha/export_onnx_hifigan.py

The above command generates 3 files:

hifigan_v1.onnx
hifigan_v2.onnx
hifigan_v3.onnx

HINT: You can download pre-exported hifigan ONNX models from https://github.com/k2-fsa/sherpa-onnx/releases/tag/vocoder-models

To use the generated onnx files to generate speech from text, please run:


# First, generate ./lexicon.txt
python3 ./matcha/generate_lexicon.py

python3 ./matcha/onnx_pretrained.py \
  --acoustic-model ./model-steps-4.onnx \
  --vocoder ./hifigan_v2.onnx \
  --tokens ./data/tokens.txt \
  --lexicon ./lexicon.txt \
  --input-text "在一个阳光明媚的夏天，小马、小羊和小狗它们一块儿在广阔的草地上，嬉戏玩耍，这时小猴来了，还带着它心爱的足球活蹦乱跳地跑前、跑后教小马、小羊、小狗踢足球。" \
  --output-wav ./1.wav

soxi ./1.wav

Input File     : './1.wav'
Channels       : 1
Sample Rate    : 22050
Precision      : 16-bit
Duration       : 00:00:16.37 = 360960 samples ~ 1227.76 CDDA sectors
File Size      : 722k
Bit Rate       : 353k
Sample Encoding: 16-bit Signed Integer PCM

https://github.com/user-attachments/assets/578d04bb-fee8-47e5-9984-a868dcce610e

4.3 KiB Raw Blame History

Introduction

matcha

4.3 KiB

Raw Blame History