17 Commits

Author	SHA1	Message	Date
Daniel Povey	a9a172aa69	Multiply lr by 10; simplify Cain.	2022-06-04 15:48:33 +08:00
Daniel Povey	679972b905	Fix bug; make epsilon work both ways (small+large); increase epsilon to 0.1	2022-06-03 19:37:48 +08:00
Daniel Povey	8085ed6ef9	Turn off natural gradient update for biases.	2022-06-03 18:40:14 +08:00
Daniel Povey	3fff0c75bb	Code cleanup	2022-06-03 11:54:12 +08:00
Daniel Povey	d6e65a0e7f	Remove decompose=True	2022-06-03 11:48:45 +08:00
Daniel Povey	a66a0d84d5	Natural gradient, with power -0.5 (halfway; -1 would be NG)	2022-06-02 14:01:03 +08:00
Daniel Povey	b1f6797af1	Remove some rebalancing code that I am now not going to use.	2022-06-01 22:19:28 +08:00
Daniel Povey	0c73664aef	Reduce threshold to 1024	2022-06-01 14:42:56 +08:00
Daniel Povey	ca09b9798f	Remove decomposition code from checkpoint.py; restore double precision model_avg	2022-06-01 14:01:58 +08:00
Daniel Povey	03e07e80ce	More drafts for rebalancing code	2022-06-01 13:58:42 +08:00
Daniel Povey	9c9bf4f1e3	Some drafts of rebalancing code in optim.py	2022-06-01 11:34:19 +08:00
Daniel Povey	bc5c782294	Limit magnitude of linear_pos	2022-06-01 10:40:54 +08:00
Daniel Povey	61619c031e	Add activation balancer to stop activations in self_attn from getting too large	2022-06-01 00:40:45 +08:00
Daniel Povey	b2259184b5	Use single precision for model average; increase average-period to 200.	2022-05-31 14:31:46 +08:00
Daniel Povey	ab9eb0d52c	Use decompose=True arg for model averaging	2022-05-31 14:28:53 +08:00
Daniel Povey	1651fe0d42	Merge changes from pruned_transducer_stateless4->5	2022-05-31 13:00:11 +08:00
Daniel Povey	741dcd1d6d	Move pruned_transducer_stateless4 to pruned_transducer_stateless7	2022-05-31 12:45:28 +08:00