mirrors/icefall
1
0
Fork 0
mirror of https://github.com/k2-fsa/icefall.git synced 2025-12-11 06:55:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Increase penalty cutoff in NonlinAttention to 40.

Browse Source
This commit is contained in:
Daniel Povey 2023-05-29 23:02:59 +08:00
parent 09294c0b51
commit d0309c3f3d
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
egs/libriheavy/LM/zipformer1/subformer.py
View File

@ -1758,7 +1758,7 @@ class NonlinAttention(nn.Module):
# ensure the activations after multiplication don't get too large. # ensure the activations after multiplication don't get too large.
self.hidden_penalty = AbsValuePenalizer( self.hidden_penalty = AbsValuePenalizer(
limit=10.0, penalty=1.0e-04, prob=0.1) limit=40.0, penalty=1.0e-04, prob=0.1)
self.out_proj = ScaledLinear(hidden_channels, channels, self.out_proj = ScaledLinear(hidden_channels, channels,
bias=True, bias=True,