mirrors/icefall
1
0
Fork 0
mirror of https://github.com/k2-fsa/icefall.git synced 2025-12-11 06:55:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,704 Commits 5 Branches 3 Tags
Commit Graph

174 Commits

Author SHA1 Message Date
Daniel Povey
1580c1c1cc Fix MulForDropout3 2023-01-11 12:26:41 +08:00
Daniel Povey
8bbcd81604 Memory efficient backprop for dropout3 2023-01-10 17:46:32 +08:00
Daniel Povey
4033000730 Share dropout masks across time in ff modules 2023-01-10 17:12:32 +08:00
Daniel Povey
1a0155fcb5 Merge branch 'scaled_adam_exp863' into scaled_adam_exp870 
# Conflicts:
#	egs/librispeech/ASR/pruned_transducer_stateless7/scaling.py
 2023-01-08 23:36:29 +08:00
Daniel Povey
b3527fe4ac Implement caching evaluation for ConvNeXt 2023-01-07 17:31:20 +08:00
Daniel Povey
b973929d7c Bug fixes to ScheduledFloat 2023-01-04 20:54:05 +08:00
Daniel Povey
ae73469b7e Refactor ScheduledFloat to include PiecewiseLinear 2023-01-04 20:46:42 +08:00
Daniel Povey
3a5b3f640d Remove eps from BasicNorm and reintroduce bias 2023-01-02 00:02:31 +08:00
Daniel Povey
a2227a07fc Revert some changes to Balancer. 2023-01-01 23:02:52 +08:00
Daniel Povey
dadeb3feec Fixes for jit scripting and osmetic improvements 2023-01-01 14:35:51 +08:00
Daniel Povey
287bd120be Reduce min_abs of zipformer balancer1; constraints on eps of Conv2dSubsampling.out_norm 2023-01-01 14:28:18 +08:00
Daniel Povey
1797d0ec6d Fix bugs in how max_rms/min_rms constraint were applied, which had the effect of making min_rms dominate over mean. 2023-01-01 13:05:41 +08:00
Daniel Povey
8db0636f1d Fix to Balancer to treat max-rms and min-rms losses separately, only max-rms loss scaled up 2023-01-01 00:38:07 +08:00
Daniel Povey
907d28ca2a Make RMS loss dominate mean loss in Balancer if both are active; remove the 4x scale introduced in 814. 2023-01-01 00:09:14 +08:00
Daniel Povey
008dbaf745 Use 4 times the normal grad_scale for BasicNorm if max_rms violated. 2022-12-31 23:42:38 +08:00
Daniel Povey
8952b69d42 Reduce BasicNorm.eps default from 0.25 to 0.1 2022-12-30 16:28:44 +08:00
Daniel Povey
d604284f16 Change initial log_scale back to 1.0 and initial eps to 0.1 2022-12-30 15:48:12 +08:00
Daniel Povey
c4101c7873 Change initial log_scale from 2 to 0. (was 1.0 in previous expt 2022-12-30 15:31:38 +08:00
Daniel Povey
851912c581 Remove bias from BasicNorm, add an eps instead. 2022-12-30 15:13:00 +08:00
Daniel Povey
e164393e91 Increase default grad_scale of Balancer from 0.02 to 0.04. 2022-12-29 21:38:19 +08:00
Daniel Povey
59be36181c Replace ActivationBalancer with Balancer 2022-12-29 20:34:46 +08:00
Daniel Povey
71d7843654 Re-introduce bias into BasicNorm and replace eps with log_scale. 2022-12-26 21:22:00 +08:00
Daniel Povey
3d6ee443e3 Revert some recent changes that may not have been helpful. 2022-12-24 21:17:43 +08:00
Daniel Povey
2b50ce2247 Change eps range from -3..3 to -2 .. 2 2022-12-24 00:09:39 +08:00
Daniel Povey
2e0f4de8ff Apply limit on BasicNorm.eps more effectively using limit_param_value; add final norm to Zipformer. 2022-12-23 15:59:51 +08:00
Daniel Povey
049174722f Change BasicNorm by adding 1+eps denominator; fix to (unused) DoubleSwish, revert to old status. 2022-12-23 13:16:51 +08:00
Daniel Povey
ade7db54e3 Revert BasicNorm to its previous status, without the bias 2022-12-22 23:47:21 +08:00
Daniel Povey
56fcb14e18 Merge branch 'scaled_adam_exp758' into scaled_adam_exp759 
# Conflicts:
#	egs/librispeech/ASR/pruned_transducer_stateless7/scaling.py
 2022-12-22 17:37:22 +08:00
Daniel Povey
1dbe1e4086 Bug fix regarding bias 2022-12-22 17:35:29 +08:00
Daniel Povey
dd7257f01b Replace 1st ConvNorm2d with BasicNorm, remove the 2nd. 2022-12-22 16:50:52 +08:00
Daniel Povey
a0b2276f68 Subtract bias after scaling 2022-12-22 15:45:45 +08:00
Daniel Povey
d31e2e12c6 Change for memory efficiency 2022-12-22 15:20:58 +08:00
Daniel Povey
903955f5d9 Add bias to BasicNorm 2022-12-22 15:14:49 +08:00
Daniel Povey
b39cde85c8 Implement bias in BasicNorm 2022-12-22 14:59:29 +08:00
Daniel Povey
11f68afa1f Revert "Remove memory-cutoff from ActivationBalancer." 
This reverts commit 5afe0e78556e2e76750cae64008c9dd5c1931c5c.
 2022-12-21 18:39:16 +08:00
Daniel Povey
788c4d97f1 Remove memory-cutoff from ActivationBalancer. 2022-12-21 15:09:26 +08:00
Daniel Povey
96d167a2ec Reduce floor on conv_min 2022-12-21 15:08:59 +08:00
Daniel Povey
05bcfd3b07 Make Whiten module update its prob every time 2022-12-21 12:56:37 +08:00
Daniel Povey
c097c13720 Change memory cutoff for ActivationBalancer; remove it for Whiten 2022-12-21 11:25:17 +08:00
Daniel Povey
244633660d Implement ConvNorm2d and use it in frontend after convnext 2022-12-20 20:28:03 +08:00
Daniel Povey
71880409cc Bug fix; also make the final norm of Conv2dSubsampling a ConvNorm1d 2022-12-20 19:44:04 +08:00
Daniel Povey
3b4b33af58 Avoid infinities in padding frames 2022-12-20 19:19:45 +08:00
Daniel Povey
494139d27a Replace BasicNorm of encoder layers with ConvNorm1d 2022-12-20 19:15:14 +08:00
Daniel Povey
5e1bf8b8ec Add BasicNorm to ConvNeXt; increase prob given to CutoffEstimator; adjust default probs of ActivationBalancer. 2022-12-18 14:14:15 +08:00
Daniel Povey
dfeafd6aa8 Remove print statement in CutoffEstimator 2022-12-17 16:28:45 +08:00
Daniel Povey
29df07ba2c Add memory cutoff on ActivationBalancer and Whiten 2022-12-17 16:20:15 +08:00
Daniel Povey
744dca1c9b Merge branch 'scaled_adam_exp724' into scaled_adam_exp726 2022-12-17 15:46:57 +08:00
Daniel Povey
b9326e1ef2 Fix to print statement 2022-12-16 18:07:43 +08:00
Daniel Povey
8e6c7ef3e2 Adjust default prob of ActivationBalancer. 2022-12-16 15:08:46 +08:00
Daniel Povey
56ac7354df Remove LinearWithAuxLoss; simplify schedule of prob in ActivationBalancer. 2022-12-16 15:07:42 +08:00