14 Commits

Author	SHA1	Message	Date
Yuekai Zhang	07cefa82a7	change scaleadam to adamw	2024-01-15 19:50:55 +08:00
Yuekai Zhang	bb1c4466e3	rename train, train2, add support to fine-tune embedding table	2024-01-15 19:49:26 +08:00
Yuekai Zhang	d926585b10	fix loading	2024-01-15 19:49:26 +08:00
Yuekai Zhang	2a288fb9bf	add custom tokenizer	2024-01-15 19:49:26 +08:00
Yuekai Zhang	22ee287312	add token files	2024-01-15 19:49:26 +08:00
Yuekai Zhang	7e387dd54b	change vocab table	2024-01-15 19:49:26 +08:00
Yuekai Zhang	72e9a436b8	fix typo	2024-01-15 19:49:26 +08:00
Yuekai Zhang	cc6432443d	add decoding with avg model	2024-01-15 19:49:26 +08:00
Yuekai Zhang	5f399dc780	load checkpoint to decode	2024-01-15 19:49:26 +08:00
Yuekai Zhang	e81545714a	update decoding from checkpoint	2024-01-15 19:49:26 +08:00
Yuekai Zhang	cbc3852876	add fairseq2 require	2024-01-15 19:49:26 +08:00
Yuekai Zhang	3a7ad277ad	add requirements	2024-01-15 19:49:26 +08:00
Yuekai Zhang	363c3f1f82	update finetuning codes	2024-01-15 19:49:26 +08:00
Yuekai Zhang	f99f4d7c92	add decode seamlessm4t	2024-01-15 19:49:26 +08:00