zr_jin
a81396b482
Use tokens.txt to replace bpe.model ( #1162 )
2023-08-12 16:53:59 +08:00
Desh Raj
a4402b88e6
SURT multi-talker ASR recipe ( #1126 )
...
* merge upstream
* add SURT model and training
* add libricss decoding
* add chunk width randomization
* decode SURT with libricss
* initial commit for zipformer_ctc
* remove unwanted changes
* remove changes to other recipe
* fix zipformer softlink
* fix for JIT export
* add missing file
* fix symbolic links
* update results
* clean commit for SURT recipe
* training libricss surt model
* remove unwanted files
* remove unwanted changes
* remove changes in librispeech
* change some files to symlinks
* remove unwanted changes in utils
* add export script
* add README
* minor fix in README
* add assets for README
* replace some files with symlinks
* remove unused decoding methods
* fix symlink
* address comments from @csukuangfj
2023-07-04 19:25:58 +08:00
Nickolay V. Shmyrev
eca0202632
Add start-batch option for RNNLM training ( #1161 )
...
* Add start-batch option for RNNLM training
* Also set epoch
* Skip batches on load
2023-07-04 10:13:25 +08:00
Peter Ross
b4c38d7547
Use symlinks for best epochs ( #1123 )
...
* utils: add symlink_or_copyfile
* pruned_transducer_stateless7: use symlinks (when possible) to output best epochs
* Rename function
---------
Co-authored-by: Yifan Yang <64255737+yfyeung@users.noreply.github.com>
2023-06-12 13:51:46 +08:00
Wei Kang
ba257efbcd
Add Context biasing ( #1038 )
...
* Add context biasing for librispeech
* Add context biasing for wenetspeech
* fix bugs
* Implement Aho-Corasick context graph
* fix some bugs
* Fixes to forward_one_step; add draw to context graph
* add output arc; fix black
* Fix wenetspeech tokenizer
* Minor fixes to the decode.py
2023-06-03 21:28:49 +08:00
Zengwei Yao
7a604057f9
update diagnostics, print limits in Balancer, merge changes from Dan's branch zlm59 ( #1109 )
2023-06-01 14:24:19 +08:00
Zengwei Yao
6826b076d4
add flops profiler, support for Zipformer encoder and Conformer encoder ( #1093 )
...
* add flops profiler, support for Zipformer encoder and Conformer encoder
* support for reworked conformer and old zipformer
* skip black check
2023-05-24 19:10:45 +08:00
Fangjun Kuang
dbcf0b41db
Fix stateless7 training error ( #1082 )
2023-05-23 12:52:02 +08:00
Zengwei Yao
a7e142b7ff
Support long audios recognition ( #980 )
...
* support long file transcription
* rename recipe as long_file_recog
* add docs
* support multi-gpu decoding
* style fix
2023-05-19 20:27:55 +08:00
Zengwei Yao
f18b539fbc
Add the upgraded Zipformer model ( #1058 )
...
* add the zipformer codes, copied from branch from_dan_scaled_adam_exp1119
* support model export with torch.jit.script
* update RESULTS.md
* support exporting streaming model with torch.jit.script
* add results of streaming models, with some minor changes
* update README.md
* add CI test
* update k2 version in requirements-ci.txt
* update pyproject.toml
2023-05-19 16:47:59 +08:00
Wei Kang
bccd20d978
Traning with byte level BPE (TAL_CSASR) ( #1033 )
...
* Add byte level bpe tal_csasr recipe
* Minor fixes to decoding and exporting
* Fix prepare.sh
* Update results
2023-05-16 12:44:52 +08:00
arbs-gpu
30bde4b788
fix rnn_lm/train.py usage ( #1055 )
2023-05-11 17:37:47 +08:00
PF Luo
44d016e4a7
export score_token interface for onnx-runtime ( #1050 )
2023-05-10 22:41:07 +08:00
Wei Kang
80156dda09
Training with byte level BPE (AIShell) ( #986 )
...
* copy files from zipformer librispeech
* Add byte bpe training for aishell
* compile LG graph
* Support LG decoding
* Minor fixes
* black
* Minor fixes
* export & fix pretrain.py
* fix black
* Update RESULTS.md
* Fix export.py
2023-05-04 19:16:17 +08:00
PF Luo
61ec3a7a8f
fix export RNNLM onnx model typo ( #1029 )
2023-04-28 19:53:06 +08:00
PF Luo
298ed4520f
add meta-data embedding_dim to RNNLM onnx-model ( #1026 )
2023-04-28 16:33:46 +08:00
Fangjun Kuang
2767b9ff11
Support exporting RNNLM to ONNX. ( #1014 )
...
* Support exporting RNNLM to ONNX.
* add int8 models
* fix style issues
* Fix EOS padding
* support exporting for streaming ASR
2023-04-27 14:36:36 +08:00
Yifan Yang
81d386ef3e
Add compute_ppl.py and ngram_entropy_pruning.py ( #1013 )
2023-04-20 12:27:43 +08:00
marcoyang1998
34d1b07c3d
Modified beam search with RNNLM rescoring ( #1002 )
...
* add RNNLM rescore
* add shallow fusion and lm rescore for streaming zipformer
* minor fix
* update RESULTS.md
* fix yesno workflow, change from ubuntu-18.04 to ubuntu-latest
2023-04-17 16:43:00 +08:00
marcoyang1998
d337398d29
Shallow fusion for Aishell ( #954 )
...
* add shallow fusion and LODR for aishell
* update RESULTS
* add save by iterations
2023-04-03 16:20:29 +08:00
Zengwei Yao
bcc5923ab9
Support batch-wise forced-alignment ( #970 )
...
* support batch-wise forced-alignment based on beam search
* add length_norm to HypothesisList.topk()
* Use Hypothesis and HypothesisList instead
2023-03-28 23:24:24 +08:00
PF Luo
15d48e3a6a
fix rnn_lm && transformer_lm import problem ( #971 )
2023-03-28 19:14:08 +08:00
Yifan Yang
cad6735e07
Modify make_pad_mask to support TensorRT ( #943 )
...
* Modify make_pad_mask to support TensorRT
* Fix for test
2023-03-10 19:28:59 +08:00
emilyluj
59ac8bfc70
fix mmi graph compiler bug. ( #895 )
2023-02-09 18:32:03 +08:00
Zengwei Yao
d12e6f098c
Get (start, end) timestamps for CTC models ( #876 )
...
* parse timestamps and texts for BPE-based models
* parse timestamps (frame indexes) and texts for other cases
* add test functions
* add parse_fsa_timestamps_and_texts function, test in conformer_ctc3/decode.py
* calculate symbol delay for (start, end) timestamps
2023-02-07 21:43:16 +08:00
Zengwei Yao
2a463a420d
Filter uneven-sized batch ( #843 )
...
* add filter_uneven_sized_batch fucntion
* set --filter-uneven-sized-batch=True as default
2023-01-16 20:15:35 +08:00
Fangjun Kuang
8642dbc0bd
Fix setup_dist ( #806 )
2023-01-04 12:21:19 +08:00
Daniil
2fd970b682
not removing result_dir in tedlium conformer ctc2 + add lm stem to compile_hlg_using_openfst.py + add MASTER_ADDR to be prvided to setup_dist ( #801 )
2023-01-02 08:08:32 +08:00
marcoyang1998
1f0408b103
Support Transformer LM ( #750 )
...
* support transformer LM
* show number of parameters during training
* update docstring
* testing files for ppl calculation
* add lm wrampper for rnn and transformer LM
* apply lm wrapper in lm shallow fusion
* small updates
* update decode.py to support LM fusion and LODR
* add export.py
* update CI and workflow
* update decoding results
* fix CI
* remove transformer LM from CI test
2022-12-29 10:53:36 +08:00
Daniil
b293db4baf
Tedlium3 conformer ctc2 ( #696 )
...
* modify preparation
* small refacor
* add tedlium3 conformer_ctc2
* modify decode
* filter unk in decode
* add scaling converter
* address comments
* fix lambda function lhotse
* add implicit manifest shuffle
* refactor ctc_greedy_search
* import model arguments from train.py
* style fix
* fix ci test and last style issues
* update RESULTS
* fix RESULTS numbers
* fix label smoothing loss
* update model parameters number in RESULTS
2022-12-13 16:13:26 +08:00
Zengwei Yao
b25c234c51
Add Zipformer-MMI ( #746 )
...
* Minor fix to conformer-mmi
* Minor fixes
* Fix decode.py
* add training files
* train with ctc warmup
* add pruned_transducer_stateless7_mmi
* add zipformer_mmi/mmi_decode.py, using HP as decoding graph
* add mmi_decode.py
* remove pruned_transducer_stateless7_mmi
* rename zipformer_mmi/train_with_ctc.py as zipformer_mmi/train.py
* remove unused method
* rename mmi_decode.py
* add export.py pretrained.py jit_pretrained.py ...
* add RESULTS.md
* add CI test
* add docs
* add README.md
Co-authored-by: pkufool <wkang.pku@gmail.com>
2022-12-11 21:30:39 +08:00
Fangjun Kuang
4501821fd9
Support using OpenFst to compile HLG. ( #606 )
...
* Support using OpenFst to compile HLG.
* Fix style issues
2022-12-09 16:46:44 +08:00
huangruizhe
0e325c8782
Fixed rnn_lm model.py ( #738 )
2022-12-07 15:43:26 +08:00
Wei Kang
c25c8c6ad1
Add need_repeat_flag in phone based ctc graph compiler ( #727 )
...
* Fix is_repeat_token in icefall
* Fix phone based recipe
* Update egs/librispeech/ASR/conformer_ctc3/train.py
Co-authored-by: Fangjun Kuang <csukuangfj@gmail.com>
* Fix black
Co-authored-by: Fangjun Kuang <csukuangfj@gmail.com>
2022-12-04 17:20:17 +08:00
Zengwei Yao
8eb4b9d96d
Combining rnnt loss and k2-ctc loss for Dan's Zipformer ( #683 )
...
* init files
* add ctc as auxiliary loss and ctc_decode.py
* tuning the scalar of HLG score for 1best, nbest and nbest-oracle
* rename to pruned_transducer_stateless7_ctc
* fix doc
* fix bug, recover the hlg scores
* modify ctc_decode.py, move out the hlg scale
* fix hlg_scale
* add export.py and pretrained.py, and so on
* upload files, update README.md and RESULTS.md
* add CI test
2022-12-03 19:01:10 +08:00
Amir Hussein
6f71981667
MGB2 ( #396 )
...
* mgb2
* mgb2
* adding pruned transducer stateless to mgb2
* update display_manifest_statistics.py
* .
* stateless transducer MGB-2
* Update README.md
* Update RESULTS.md
* Update prepare_lang_bpe.py
* Update asr_datamodule.py
* .nfs removed
* Adding symlink
* .
* resolving conflicts
* Update .gitignore
* black formatting
* Update compile_hlg.py
* Update compute_fbank_musan.py
* Update convert_transcript_words_to_tokens.py
* Update download_lm.py
* Update generate_unique_lexicon.py
* adding simlinks
* fixing symbolic links
2022-12-02 10:58:34 +08:00
Fangjun Kuang
04c9fc9c9f
Fix for older versions of k2 ( #725 )
2022-12-02 09:18:28 +08:00
Fangjun Kuang
2bca7032af
Update RNNLM training scripts ( #720 )
...
* Update RNNLM training scripts
* Fix a typo
* Fix CI
2022-12-01 15:57:43 +08:00
Zengwei Yao
ece728d895
Apply delay penalty on k2 ctc loss ( #669 )
...
* add init files
* fix bug, apply delay penalty
* fix decoding code and getting timestamps
* add option applying delay penalty on ctc log-prob
* fix bug of streaming decoding
* minor change for bpe-based case
* add test_model.py
* add README.md
* add CI
2022-11-28 22:34:02 +08:00
abb128
61032e70e0
Fix exception in find_checkpoints ( #668 )
2022-11-26 10:10:37 +08:00
Desh Raj
d31db01037
manual correction of black formatting
2022-11-17 14:18:05 -05:00
Desh Raj
107df3b115
apply black on all files
2022-11-17 09:42:17 -05:00
Fangjun Kuang
60317120ca
Revert "Apply new Black style changes"
2022-11-17 20:19:32 +08:00
Desh Raj
cad8f6aca4
merge upstream
2022-11-16 19:50:43 -05:00
Daniil
fca796cc2c
Small code refactoring ( #687 )
2022-11-17 06:55:53 +08:00
Desh Raj
d110b04ad3
apply new black formatting to all files
2022-11-16 13:06:43 -05:00
Fangjun Kuang
7e82f87126
Add Zipformer from Dan ( #672 )
2022-11-12 18:11:19 +08:00
Yuekai Zhang
2f43e4508b
fix mask errors when padding audios ( #670 )
2022-11-10 22:28:04 +08:00
Zengwei Yao
32de2766d5
Refactor getting timestamps in fsa-based decoding ( #660 )
...
* refactor getting timestamps for fsa-based decoding
* fix doc
* fix bug
2022-11-05 22:36:06 +08:00
marcoyang
a2d7095c1c
resolve conflicts
2022-11-04 11:37:42 +08:00
