6 Commits

Author	SHA1	Message	Date
marcoyang1998	88a311734d	add script to prepare validation and test sets	2023-07-19 11:01:07 +08:00
marcoyang1998	0d1cd4f595	add char coverage option to avoid having a lot of rarely used tokens in the BPE; add the option to use byte-fallback in training BPE	2023-07-19 10:55:57 +08:00
marcoyang1998	6939b3d6aa	minor fixes	2023-07-18 11:14:06 +08:00
marcoyang	0e7df7c5c4	add necessary utility files	2023-07-18 10:06:22 +08:00
marcoyang	189d424b25	only use medium text to train the BPE as the whole corpus is tooooo large	2023-07-18 10:06:01 +08:00
marcoyang	fef229e024	add necessary files to compute features	2023-07-17 10:36:25 +08:00