mirrors/icefall
1
0
Fork 0
mirror of https://github.com/k2-fsa/icefall.git synced 2025-12-11 06:55:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make out_proj of feedforward modules be a LinearWithAuxLoss, with nonzero final value at 0.01.

Browse Source
This commit is contained in:
Daniel Povey 2022-11-25 18:00:46 +08:00
parent 8f1ef60951
commit 2368968114
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
egs/librispeech/ASR/pruned_transducer_stateless7/zipformer.py
View File

@ -1390,8 +1390,10 @@ class FeedforwardModule(nn.Module):
min_prob=0.25)
self.activation = DoubleSwish()
self.dropout = nn.Dropout(dropout)
self.out_proj = ScaledLinear(feedforward_dim, embed_dim,
initial_scale=0.01)
self.out_proj = LinearWithAuxLoss(feedforward_dim, embed_dim,
initial_scale=0.01,
aux_grad_scale=ScheduledFloat((0.0, 0.1), (1000.0, 0.01)),
)
self.out_whiten = Whiten(num_groups=1,
whitening_limit=_whitening_schedule(7.5),
prob=(0.025, 0.25),