632 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Daniel Povey	d301f8ac6c	Merge Decorrelate work, and simplification to RandomCombine, into pruned_transducer_stateless7	2022-06-11 11:07:07 +08:00
Daniel Povey	30379bb35d	Merge branch 'cain3d_merge_supersimple' into merge_decorr_cain	2022-06-11 10:51:30 +08:00
Daniel Povey	a3c97554e8	Reduce scale from 0.1 to 0.05	2022-06-10 23:34:46 +08:00
Daniel Povey	2bbc63a2f5	Change first1k to decay1k	2022-06-10 23:33:16 +08:00
Daniel Povey	41cdbae753	Increase scale from 0.05 to 0.1	2022-06-10 19:07:48 +08:00
Daniel Povey	5fb64a59b8	Change beta from 0.8 to 0.95	2022-06-10 19:05:04 +08:00
Daniel Povey	6ed181595b	Scale by grad norm	2022-06-10 18:34:42 +08:00
Daniel Povey	6a47bf1178	Decrease scale to 0.05	2022-06-10 18:33:50 +08:00
Daniel Povey	9d4633facf	Simplified gradient scaling [no scaling]; only use 1k first iters; beta =0.8	2022-06-10 17:16:05 +08:00
Daniel Povey	cecd52155c	Apply x row scaling with grad	2022-06-10 16:42:52 +08:00
Daniel Povey	86c2d0fcc0	Make it run only for first 3k steps; larger scale; remove limit of 1.0	2022-06-10 16:33:22 +08:00
Daniel Povey	eeb95ed502	Fix issue with cov scale	2022-06-10 16:25:45 +08:00
Daniel Povey	c671e213fc	Increase beta from 0.95 to 0.98	2022-06-10 14:39:58 +08:00
Daniel Povey	c6cfb1e5fa	Remove logging that was excessive	2022-06-10 14:25:23 +08:00
Daniel Povey	ff0309947a	Do scaling a different way, so loss function is more consistent; accum stats in backward pass	2022-06-10 14:16:44 +08:00
Daniel Povey	58cbc3d961	Move PseudoNormalizeFunction to a different place.	2022-06-10 14:01:13 +08:00
Daniel Povey	77357cdaa8	Change scale from 0.1 to 0.05	2022-06-10 12:42:43 +08:00
Daniel Povey	173c8e6ca3	Make decorrelate scale really 0.1	2022-06-10 12:39:01 +08:00
Daniel Povey	42667aacf9	Remove linear layers from RandomCombine	2022-06-10 11:38:50 +08:00
Daniel Povey	c92d9d72aa	Fix inf issue	2022-06-10 11:20:47 +08:00
Daniel Povey	950cd4a3e8	Introduce normalization..	2022-06-10 10:47:18 +08:00
Daniel Povey	e891a65735	Move Decorrelate module to after encoder, with scale 0.02->0.1	2022-06-10 10:15:28 +08:00
Daniel Povey	4a5143e548	Increase decay to 1k	2022-06-10 10:09:46 +08:00
Daniel Povey	e2ef8732d1	Increase beta to 0.95	2022-06-10 10:05:28 +08:00
Daniel Povey	a61e21ac85	Change beta to 0.9	2022-06-09 23:33:05 +08:00
Daniel Povey	2c5ebc065e	Change eps to 1e-20	2022-06-09 23:24:33 +08:00
Daniel Povey	c533f91fa2	Remove one line..	2022-06-09 23:13:16 +08:00
Daniel Povey	0fd2cb141f	Code cleanup and refactoring	2022-06-09 22:54:56 +08:00
Daniel Povey	2621cb7f54	Change beta to 0.8	2022-06-09 20:17:12 +08:00
Daniel Povey	082a890635	Fix apply_prob_decay to 500	2022-06-09 19:20:03 +08:00
Daniel Povey	fca844d80c	Make it really have 2k decay and revert to 0.02 scale	2022-06-09 17:45:11 +08:00
Daniel Povey	e99344f15e	Increase scale to 0.04	2022-06-09 13:24:31 +08:00
Daniel Povey	bfcd288afd	Decrease scale on decorrelate component from 0.02 to 0.01	2022-06-09 12:07:19 +08:00
Daniel Povey	56d6dd55ae	Bug fixes	2022-06-09 12:06:35 +08:00
Daniel Povey	1669e21c0c	Use decorrelation in conformer layers also	2022-06-09 11:31:52 +08:00
Daniel Povey	b9a476c7bb	Remove loss factor from decorr_loss_scale	2022-06-08 20:19:17 +08:00
Daniel Povey	8e56445c70	Try to resolve graph-freed problem	2022-06-08 20:07:35 +08:00
Daniel Povey	46ca1cd4c4	Add Decorrelate module that adds something to gradients in backward pass	2022-06-08 19:44:58 +08:00
Daniel Povey	9fb8645168	Implement JoinDropout	2022-06-08 16:17:42 +08:00
Daniel Povey	e7886d49a9	Bug fix	2022-06-08 11:05:29 +08:00
Daniel Povey	a83bde1372	Simplify implementation as current idea was not working to decorrelate	2022-06-08 10:24:41 +08:00
Daniel Povey	135be1e19c	Change dropout_rate from 0.2 to 0.1; fix logging statement; fix assignment to rand_scales, nonrand_scales to use [:]	2022-06-08 00:42:04 +08:00
Daniel Povey	a6050cb2de	Implement new, more principled but maybe slower version.	2022-06-07 23:38:38 +08:00
Daniel Povey	75c822c7e9	Pre and post-multiply by inv_sqrt_stddev,stddev	2022-06-07 20:32:18 +08:00
Daniel Povey	a270973b69	Add gaussian version of decorrelation	2022-06-07 18:55:48 +08:00
Daniel Povey	5d24489752	Have 2 scales on dropout	2022-06-07 18:31:42 +08:00
Daniel Povey	53ca61db7a	Reduce scale on decorrelation by 5, to 0.01	2022-06-07 17:10:54 +08:00
Daniel Povey	7c6d923d3f	Add decorrelation to joiner	2022-06-07 16:47:54 +08:00
Daniel Povey	cd6b707e2b	Various bug fixes	2022-06-07 16:45:32 +08:00
Daniel Povey	40a0934b4e	Implement GaussProjDrop	2022-06-07 11:51:24 +08:00