160 Commits

Author	SHA1	Message	Date
Daniel Povey	008dbaf745	Use 4 times the normal grad_scale for BasicNorm if max_rms violated.	2022-12-31 23:42:38 +08:00
Daniel Povey	8952b69d42	Reduce BasicNorm.eps default from 0.25 to 0.1	2022-12-30 16:28:44 +08:00
Daniel Povey	d604284f16	Change initial log_scale back to 1.0 and initial eps to 0.1	2022-12-30 15:48:12 +08:00
Daniel Povey	c4101c7873	Change initial log_scale from 2 to 0. (was 1.0 in previous expt	2022-12-30 15:31:38 +08:00
Daniel Povey	851912c581	Remove bias from BasicNorm, add an eps instead.	2022-12-30 15:13:00 +08:00
Daniel Povey	e164393e91	Increase default grad_scale of Balancer from 0.02 to 0.04.	2022-12-29 21:38:19 +08:00
Daniel Povey	59be36181c	Replace ActivationBalancer with Balancer	2022-12-29 20:34:46 +08:00
Daniel Povey	71d7843654	Re-introduce bias into BasicNorm and replace eps with log_scale.	2022-12-26 21:22:00 +08:00
Daniel Povey	3d6ee443e3	Revert some recent changes that may not have been helpful.	2022-12-24 21:17:43 +08:00
Daniel Povey	2b50ce2247	Change eps range from -3..3 to -2 .. 2	2022-12-24 00:09:39 +08:00
Daniel Povey	2e0f4de8ff	Apply limit on BasicNorm.eps more effectively using limit_param_value; add final norm to Zipformer.	2022-12-23 15:59:51 +08:00
Daniel Povey	049174722f	Change BasicNorm by adding 1+eps denominator; fix to (unused) DoubleSwish, revert to old status.	2022-12-23 13:16:51 +08:00
Daniel Povey	ade7db54e3	Revert BasicNorm to its previous status, without the bias	2022-12-22 23:47:21 +08:00
Daniel Povey	56fcb14e18	Merge branch 'scaled_adam_exp758' into scaled_adam_exp759 ... # Conflicts: # egs/librispeech/ASR/pruned_transducer_stateless7/scaling.py	2022-12-22 17:37:22 +08:00
Daniel Povey	1dbe1e4086	Bug fix regarding bias	2022-12-22 17:35:29 +08:00
Daniel Povey	dd7257f01b	Replace 1st ConvNorm2d with BasicNorm, remove the 2nd.	2022-12-22 16:50:52 +08:00
Daniel Povey	a0b2276f68	Subtract bias after scaling	2022-12-22 15:45:45 +08:00
Daniel Povey	d31e2e12c6	Change for memory efficiency	2022-12-22 15:20:58 +08:00
Daniel Povey	903955f5d9	Add bias to BasicNorm	2022-12-22 15:14:49 +08:00
Daniel Povey	b39cde85c8	Implement bias in BasicNorm	2022-12-22 14:59:29 +08:00
Daniel Povey	11f68afa1f	Revert "Remove memory-cutoff from ActivationBalancer." ... This reverts commit 5afe0e78556e2e76750cae64008c9dd5c1931c5c.	2022-12-21 18:39:16 +08:00
Daniel Povey	788c4d97f1	Remove memory-cutoff from ActivationBalancer.	2022-12-21 15:09:26 +08:00
Daniel Povey	96d167a2ec	Reduce floor on conv_min	2022-12-21 15:08:59 +08:00
Daniel Povey	05bcfd3b07	Make Whiten module update its prob every time	2022-12-21 12:56:37 +08:00
Daniel Povey	c097c13720	Change memory cutoff for ActivationBalancer; remove it for Whiten	2022-12-21 11:25:17 +08:00
Daniel Povey	244633660d	Implement ConvNorm2d and use it in frontend after convnext	2022-12-20 20:28:03 +08:00
Daniel Povey	71880409cc	Bug fix; also make the final norm of Conv2dSubsampling a ConvNorm1d	2022-12-20 19:44:04 +08:00
Daniel Povey	3b4b33af58	Avoid infinities in padding frames	2022-12-20 19:19:45 +08:00
Daniel Povey	494139d27a	Replace BasicNorm of encoder layers with ConvNorm1d	2022-12-20 19:15:14 +08:00
Daniel Povey	5e1bf8b8ec	Add BasicNorm to ConvNeXt; increase prob given to CutoffEstimator; adjust default probs of ActivationBalancer.	2022-12-18 14:14:15 +08:00
Daniel Povey	dfeafd6aa8	Remove print statement in CutoffEstimator	2022-12-17 16:28:45 +08:00
Daniel Povey	29df07ba2c	Add memory cutoff on ActivationBalancer and Whiten	2022-12-17 16:20:15 +08:00
Daniel Povey	744dca1c9b	Merge branch 'scaled_adam_exp724' into scaled_adam_exp726	2022-12-17 15:46:57 +08:00
Daniel Povey	b9326e1ef2	Fix to print statement	2022-12-16 18:07:43 +08:00
Daniel Povey	8e6c7ef3e2	Adjust default prob of ActivationBalancer.	2022-12-16 15:08:46 +08:00
Daniel Povey	56ac7354df	Remove LinearWithAuxLoss; simplify schedule of prob in ActivationBalancer.	2022-12-16 15:07:42 +08:00
Daniel Povey	083e5474c4	Reduce ConvNeXt parameters.	2022-12-16 00:21:04 +08:00
Daniel Povey	8d9301e225	Remove potentially wrong typing info	2022-12-15 23:47:41 +08:00
Daniel Povey	6caaa4e9c6	Bug fix in caching_eval, may make no difference.	2022-12-15 23:32:29 +08:00
Daniel Povey	f5d4fb092d	Bug fix in caching_eval	2022-12-15 23:24:36 +08:00
Daniel Povey	d26ee2bf81	Try to implement caching evaluation for memory efficient training	2022-12-15 23:06:40 +08:00
Daniel Povey	f66c1600f4	Bug fix to printing code	2022-12-15 21:55:23 +08:00
Daniel Povey	2d0fe7637c	Memory fix in WithLoss	2022-12-11 17:20:26 +08:00
Daniel Povey	0fc646f281	Merge branch 'scaled_adam_exp663' into scaled_adam_exp665	2022-12-10 00:07:37 +08:00
Daniel Povey	d35eb7a3a6	Add cosmetic/diagnostics changes from scaled_adam_exp656.	2022-12-09 22:02:42 +08:00
Daniel Povey	5c0957d950	Fix memory issue in ActivationBalancer	2022-12-09 18:11:27 +08:00
Daniel Povey	2ef0228db0	Make the ActivationBalancer relative to the mean, limited to -min_abs..max_abs	2022-12-09 17:59:00 +08:00
Daniel Povey	3f82ee0783	Merge dropout schedule, 0.3 ... 0.1 over 20k batches	2022-12-08 18:18:46 +08:00
Daniel Povey	22617da725	Make dropout a schedule starting at 0.3.	2022-12-05 23:39:24 +08:00
Daniel Povey	178eca1c0e	Revert scaling, scale only grad.	2022-12-05 17:53:23 +08:00