mirrors/icefall
1
0
Fork 0
mirror of https://github.com/k2-fsa/icefall.git synced 2025-12-11 06:55:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Reduce the limit on attention weights from 50 to 25.

Browse Source
This commit is contained in:
Daniel Povey 2022-10-21 12:13:23 +08:00
parent c5cb52fed1
commit 9f68b5717c
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
egs/librispeech/ASR/pruned_transducer_stateless7/conformer.py
View File

@ -1116,7 +1116,7 @@ class RelPositionMultiheadAttention(nn.Module):
# this mechanism instead of, say, a limit on entropy, because once the entropy # this mechanism instead of, say, a limit on entropy, because once the entropy
# gets very small gradients through the softmax can become very small, and # gets very small gradients through the softmax can become very small, and
# some mechanisms like that become ineffective. # some mechanisms like that become ineffective.
attn_weights_limit = 50.0 attn_weights_limit = 25.0
# caution: this penalty will be affected by grad-scaling in amp. # caution: this penalty will be affected by grad-scaling in amp.
# It's OK; this is just an emergency brake, and under normal # It's OK; this is just an emergency brake, and under normal
# conditions it shouldn't be active # conditions it shouldn't be active