marcoyang1998 0aee07fb4c change the valid/test sets; only do simple normalization in the dataloader, i.e only replace full-width symbol, replace double hyphen with space

2023-07-19 11:00:07 +08:00

..

local

add char coverage option to avoid having a lot of rarely used tokens in the BPE; add the option to use byte-fallback in training BPE

2023-07-19 10:55:57 +08:00

zipformer

change the valid/test sets; only do simple normalization in the dataloader, i.e only replace full-width symbol, replace double hyphen with space

2023-07-19 11:00:07 +08:00

prepare.sh

only use medium text to train the BPE as the whole corpus is tooooo large

2023-07-18 10:06:01 +08:00

shared

initial commit for libriheavy

2023-07-14 23:50:27 +08:00