13 Commits

Author	SHA1	Message	Date
marcoyang1998	1e31fbcd7d	Add clamping operation in Eve optimizer for all scalar weights to avoid (#550) ... non stable training in some scenarios. The clamping range is set to (-10,2). Note that this change may cause unexpected effect if you resume training from a model that is trained without clamping.	2022-08-25 12:12:50 +08:00
Mingshuang Luo	93c60a9d30	Code style check for librispeech pruned transducer stateless2 (#308)	2022-04-11 22:15:18 +08:00
Daniel Povey	03c7c2613d	Fix docs in optim.py	2022-04-11 15:13:42 +08:00
Daniel Povey	d1e4ae788d	Refactor how learning rate is set.	2022-04-10 15:25:27 +08:00
Daniel Povey	4d41ee0caa	Implement 2o schedule	2022-04-09 18:37:03 +08:00
Daniel Povey	a41e93437c	Change some defaults in LR-setting rule.	2022-04-06 12:36:58 +08:00
Daniel Povey	25724b5ce9	Bug-fix RE sign of target_rms	2022-04-05 13:49:35 +08:00
Daniel Povey	c3169222ae	Simplified optimizer, rework somet things..	2022-04-05 13:23:02 +08:00
Daniel Povey	47d49f29d7	Fix weight decay formula by adding 1/1-beta	2022-04-05 00:31:55 +08:00
Daniel Povey	2b0727a355	Fix weight decay formula by adding 1/1-beta	2022-04-05 00:31:28 +08:00
Daniel Povey	179d0605ea	Change initialization to 0.25	2022-04-04 23:34:39 +08:00
Daniel Povey	d1f2f93460	Some fixes..	2022-04-04 22:40:18 +08:00
Daniel Povey	72f4a673b1	First draft of new approach to learning rates + init	2022-04-04 20:21:34 +08:00