Daniel Povey
|
137ac513bf
|
Some changes to try to reduce mem consumption; decrease batch size
|
2023-05-28 21:50:34 +08:00 |
|
Daniel Povey
|
625e39fd1a
|
Avoid penalize_abs_values_gt when memory usage high
|
2023-05-28 20:40:47 +08:00 |
|
Daniel Povey
|
bcc9971ebe
|
Add clip_grad
|
2023-05-23 14:00:56 +08:00 |
|
Daniel Povey
|
bfeeddda81
|
Reduce mem consumption of softmax backward
|
2023-05-16 12:18:09 +08:00 |
|
Daniel Povey
|
6f5c4688ef
|
Add (back) straight_through_rate, with rate 0.025; try to handle memory allocation failures in backprop better.
|
2023-04-30 15:19:34 +08:00 |
|
Daniel Povey
|
6c26754628
|
Fix tests, make SwooshL and SwooshR more efficient in forward pass.
|
2023-04-27 22:37:19 +08:00 |
|
yaozengwei
|
0b0732ae28
|
use ActivationDropoutAndLinearFunction and swoosh kernel functions
|
2023-04-12 19:11:26 +08:00 |
|
Daniel Povey
|
f59da65d82
|
Remove some more unused code; rename BasicNorm->BiasNorm, Zipformer->Zipformer2
|
2023-03-06 14:27:11 +08:00 |
|
Daniel Povey
|
3424b60d8f
|
Remove some unused code
|
2023-03-06 14:18:01 +08:00 |
|
Daniel Povey
|
0191e8f3e4
|
Simplify how dim changes are dealt with; see also scaled_adam_exp977
|
2023-02-22 11:40:33 +08:00 |
|
Daniel Povey
|
49627c6251
|
Cosmetic fix to formula
|
2023-02-11 15:38:03 +08:00 |
|
Daniel Povey
|
329175c897
|
Change how chunk-size is specified
|
2023-02-11 14:35:31 +08:00 |
|
Daniel Povey
|
e7e7560bba
|
Implement chunking
|
2023-02-10 15:02:29 +08:00 |
|
Daniel Povey
|
1774853bdf
|
Remove caching eval
|
2023-01-11 13:12:25 +08:00 |
|
Daniel Povey
|
1580c1c1cc
|
Fix MulForDropout3
|
2023-01-11 12:26:41 +08:00 |
|
Daniel Povey
|
8bbcd81604
|
Memory efficient backprop for dropout3
|
2023-01-10 17:46:32 +08:00 |
|
Daniel Povey
|
4033000730
|
Share dropout masks across time in ff modules
|
2023-01-10 17:12:32 +08:00 |
|
Daniel Povey
|
1a0155fcb5
|
Merge branch 'scaled_adam_exp863' into scaled_adam_exp870
# Conflicts:
# egs/librispeech/ASR/pruned_transducer_stateless7/scaling.py
|
2023-01-08 23:36:29 +08:00 |
|
Daniel Povey
|
b3527fe4ac
|
Implement caching evaluation for ConvNeXt
|
2023-01-07 17:31:20 +08:00 |
|
Daniel Povey
|
b973929d7c
|
Bug fixes to ScheduledFloat
|
2023-01-04 20:54:05 +08:00 |
|
Daniel Povey
|
ae73469b7e
|
Refactor ScheduledFloat to include PiecewiseLinear
|
2023-01-04 20:46:42 +08:00 |
|
Daniel Povey
|
3a5b3f640d
|
Remove eps from BasicNorm and reintroduce bias
|
2023-01-02 00:02:31 +08:00 |
|
Daniel Povey
|
a2227a07fc
|
Revert some changes to Balancer.
|
2023-01-01 23:02:52 +08:00 |
|
Daniel Povey
|
dadeb3feec
|
Fixes for jit scripting and osmetic improvements
|
2023-01-01 14:35:51 +08:00 |
|
Daniel Povey
|
287bd120be
|
Reduce min_abs of zipformer balancer1; constraints on eps of Conv2dSubsampling.out_norm
|
2023-01-01 14:28:18 +08:00 |
|
Daniel Povey
|
1797d0ec6d
|
Fix bugs in how max_rms/min_rms constraint were applied, which had the effect of making min_rms dominate over mean.
|
2023-01-01 13:05:41 +08:00 |
|
Daniel Povey
|
8db0636f1d
|
Fix to Balancer to treat max-rms and min-rms losses separately, only max-rms loss scaled up
|
2023-01-01 00:38:07 +08:00 |
|
Daniel Povey
|
907d28ca2a
|
Make RMS loss dominate mean loss in Balancer if both are active; remove the 4x scale introduced in 814.
|
2023-01-01 00:09:14 +08:00 |
|
Daniel Povey
|
008dbaf745
|
Use 4 times the normal grad_scale for BasicNorm if max_rms violated.
|
2022-12-31 23:42:38 +08:00 |
|
Daniel Povey
|
8952b69d42
|
Reduce BasicNorm.eps default from 0.25 to 0.1
|
2022-12-30 16:28:44 +08:00 |
|
Daniel Povey
|
d604284f16
|
Change initial log_scale back to 1.0 and initial eps to 0.1
|
2022-12-30 15:48:12 +08:00 |
|
Daniel Povey
|
c4101c7873
|
Change initial log_scale from 2 to 0. (was 1.0 in previous expt
|
2022-12-30 15:31:38 +08:00 |
|
Daniel Povey
|
851912c581
|
Remove bias from BasicNorm, add an eps instead.
|
2022-12-30 15:13:00 +08:00 |
|
Daniel Povey
|
e164393e91
|
Increase default grad_scale of Balancer from 0.02 to 0.04.
|
2022-12-29 21:38:19 +08:00 |
|
Daniel Povey
|
59be36181c
|
Replace ActivationBalancer with Balancer
|
2022-12-29 20:34:46 +08:00 |
|
Daniel Povey
|
71d7843654
|
Re-introduce bias into BasicNorm and replace eps with log_scale.
|
2022-12-26 21:22:00 +08:00 |
|
Daniel Povey
|
3d6ee443e3
|
Revert some recent changes that may not have been helpful.
|
2022-12-24 21:17:43 +08:00 |
|
Daniel Povey
|
2b50ce2247
|
Change eps range from -3..3 to -2 .. 2
|
2022-12-24 00:09:39 +08:00 |
|
Daniel Povey
|
2e0f4de8ff
|
Apply limit on BasicNorm.eps more effectively using limit_param_value; add final norm to Zipformer.
|
2022-12-23 15:59:51 +08:00 |
|
Daniel Povey
|
049174722f
|
Change BasicNorm by adding 1+eps denominator; fix to (unused) DoubleSwish, revert to old status.
|
2022-12-23 13:16:51 +08:00 |
|
Daniel Povey
|
ade7db54e3
|
Revert BasicNorm to its previous status, without the bias
|
2022-12-22 23:47:21 +08:00 |
|
Daniel Povey
|
56fcb14e18
|
Merge branch 'scaled_adam_exp758' into scaled_adam_exp759
# Conflicts:
# egs/librispeech/ASR/pruned_transducer_stateless7/scaling.py
|
2022-12-22 17:37:22 +08:00 |
|
Daniel Povey
|
1dbe1e4086
|
Bug fix regarding bias
|
2022-12-22 17:35:29 +08:00 |
|
Daniel Povey
|
dd7257f01b
|
Replace 1st ConvNorm2d with BasicNorm, remove the 2nd.
|
2022-12-22 16:50:52 +08:00 |
|
Daniel Povey
|
a0b2276f68
|
Subtract bias after scaling
|
2022-12-22 15:45:45 +08:00 |
|
Daniel Povey
|
d31e2e12c6
|
Change for memory efficiency
|
2022-12-22 15:20:58 +08:00 |
|
Daniel Povey
|
903955f5d9
|
Add bias to BasicNorm
|
2022-12-22 15:14:49 +08:00 |
|
Daniel Povey
|
b39cde85c8
|
Implement bias in BasicNorm
|
2022-12-22 14:59:29 +08:00 |
|
Daniel Povey
|
11f68afa1f
|
Revert "Remove memory-cutoff from ActivationBalancer."
This reverts commit 5afe0e78556e2e76750cae64008c9dd5c1931c5c.
|
2022-12-21 18:39:16 +08:00 |
|
Daniel Povey
|
788c4d97f1
|
Remove memory-cutoff from ActivationBalancer.
|
2022-12-21 15:09:26 +08:00 |
|