3 Commits

Author	SHA1	Message	Date
marcoyang1998	0d1cd4f595	add char coverage option to avoid having a lot of rarely used tokens in the BPE; add the option to use byte-fallback in training BPE	2023-07-19 10:55:57 +08:00
marcoyang	189d424b25	only use medium text to train the BPE as the whole corpus is tooooo large	2023-07-18 10:06:01 +08:00
marcoyang	fef229e024	add necessary files to compute features	2023-07-17 10:36:25 +08:00