11 Commits

Author	SHA1	Message	Date
marcoyang1998	5532bb1683	add files for decoding	2023-07-19 22:05:53 +08:00
marcoyang1998	4f3a6606ad	add necessary files for training	2023-07-19 22:04:11 +08:00
marcoyang1998	88a311734d	add script to prepare validation and test sets	2023-07-19 11:01:07 +08:00
marcoyang1998	0aee07fb4c	change the valid/test sets; only do simple normalization in the dataloader, i.e only replace full-width symbol, replace double hyphen with space	2023-07-19 11:00:07 +08:00
marcoyang1998	0d1cd4f595	add char coverage option to avoid having a lot of rarely used tokens in the BPE; add the option to use byte-fallback in training BPE	2023-07-19 10:55:57 +08:00
marcoyang1998	b53c0d1e5f	initial commit for zipformer recipe	2023-07-18 11:42:19 +08:00
marcoyang1998	6939b3d6aa	minor fixes	2023-07-18 11:14:06 +08:00
marcoyang	0e7df7c5c4	add necessary utility files	2023-07-18 10:06:22 +08:00
marcoyang	189d424b25	only use medium text to train the BPE as the whole corpus is tooooo large	2023-07-18 10:06:01 +08:00
marcoyang	fef229e024	add necessary files to compute features	2023-07-17 10:36:25 +08:00
marcoyang	44d01195c0	initial commit for libriheavy	2023-07-14 23:50:27 +08:00