mirrors/icefall
1
0
Fork 0
mirror of https://github.com/k2-fsa/icefall.git synced 2025-08-10 02:22:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
803 Commits 5 Branches 3 Tags
Commit Graph

53 Commits

Author SHA1 Message Date
Yifan Yang
d67a49afe4
Add multidataset (#1010) 
* Add Common Voice for multidataset

* Add prepare_multidataset.sh

* Add dataset mixing


* Update prepare_multidataset.sh

* Update prepare_giga_speech.sh

* update comments

* Add split and shuffle mechanism

* Add multi-dataset train

* Fix for deleting

* Fix for modifying

* Add comments

* Change type for perturb_speed

* Fix for style check

* Small fix

* Add filter

* Remove warning
 2023-04-21 18:09:41 +08:00
Yifan Yang
33578cca48
Fix filter_cuts in compute_fbank_librispeech.py (#993) 2023-04-11 11:12:05 +08:00
Yifan Yang
28af269e5e
Fix for workflow (#934) 2023-03-09 17:38:15 +08:00
Karel Vesely
35e5a2475c
Librispeech, validate_manifest.py (#890) 2023-02-09 07:57:02 +08:00
Daniil
2fd970b682
not removing result_dir in tedlium conformer ctc2 + add lm stem to compile_hlg_using_openfst.py + add MASTER_ADDR to be prvided to setup_dist (#801) 2023-01-02 08:08:32 +08:00
Daniil
b293db4baf
Tedlium3 conformer ctc2 (#696) 
* modify preparation

* small refacor

* add tedlium3 conformer_ctc2

* modify decode

* filter unk in decode

* add scaling converter

* address comments

* fix lambda function lhotse

* add implicit manifest shuffle

* refactor ctc_greedy_search

* import model arguments from train.py

* style fix

* fix ci test and last style issues

* update RESULTS

* fix RESULTS numbers

* fix label smoothing loss

* update model parameters number in RESULTS
 2022-12-13 16:13:26 +08:00
Fangjun Kuang
4501821fd9
Support using OpenFst to compile HLG. (#606) 
* Support using OpenFst to compile HLG.

* Fix style issues
 2022-12-09 16:46:44 +08:00
Fangjun Kuang
6533f359c9
Fix CI (#726) 
* Fix CI

* Disable shuffle for yesno.

See https://github.com/k2-fsa/icefall/issues/197
 2022-12-02 10:53:06 +08:00
Fangjun Kuang
2bca7032af
Update RNNLM training scripts (#720) 
* Update RNNLM training scripts

* Fix a typo

* Fix CI
 2022-12-01 15:57:43 +08:00
Desh Raj
d31db01037 manual correction of black formatting 2022-11-17 14:18:05 -05:00
Desh Raj
107df3b115 apply black on all files 2022-11-17 09:42:17 -05:00
Fangjun Kuang
60317120ca
Revert "Apply new Black style changes" 2022-11-17 20:19:32 +08:00
Desh Raj
cad8f6aca4 merge upstream 2022-11-16 19:50:43 -05:00
Daniil
fca796cc2c
Small code refactoring (#687) 2022-11-17 06:55:53 +08:00
Desh Raj
d110b04ad3 apply new black formatting to all files 2022-11-16 13:06:43 -05:00
Desh Raj
c8ce243255
Zipformer output length (#686) 
* add assertion for output length

* add comment in filter_cuts

* add length filter to Zipformer recipes
 2022-11-16 11:29:45 +08:00
Tiance Wang
952a7b3fcc
Fix typo (#681) 
* Update add_alignment_librispeech.py

* Update scaling_converter.py
 2022-11-15 10:45:48 +08:00
Fangjun Kuang
e334e570d8
Filter utterances with number_tokens > number_feature_frames. (#604) 2022-11-12 07:57:58 +08:00
Zengwei Yao
03668771d7
Get timestamps during decoding (#598) 
* print out timestamps during decoding

* add word-level alignments

* support to compute mean symbol delay with word-level alignments

* print variance of symbol delay

* update doc

* support to compute delay for pruned_transducer_stateless4

* fix bug

* add doc
 2022-11-01 10:24:00 +08:00
Fangjun Kuang
e18fa78c3a
Check that read_manifests_if_cached returns a non-empty dict. (#555) 2022-08-28 11:50:11 +08:00
Fangjun Kuang
ec69967584
Set overwrite=True when extracting features in batches. (#487) 2022-07-29 11:17:19 +08:00
ezerhouni
0475d75d15
[Ready to be merged] Add RNN-LM to Conformer-CTC decoding (#439) 2022-06-23 19:37:03 +08:00
Fangjun Kuang
ed66877694
Replace ChunkedLilcomHdf5Writer with LilcomChunkyWriter. (#411) 2022-06-09 11:18:52 +08:00
Fangjun Kuang
1094a3cb37
Replace LilcomChunkyWriter with ChunkedLilcomHdf5Writer. (#404) 2022-06-07 18:14:25 +08:00
Fangjun Kuang
f1abce72f8
Use jsonl for CutSet in the LibriSpeech recipe. (#397) 
* Use jsonl for cutsets in the librispeech recipe.

* Use lazy cutset for all recipes.

* More fixes to use lazy CutSet.

* Remove force=True from logging to support Python < 3.8

* Minor fixes.

* Fix style issues.
 2022-06-06 10:19:16 +08:00
Ewald Enzinger
8c5722de8c
[egs] Add prefix when reading manifests due to recent lhotse changes (#382) 
* [egs] Add prefix when reading manifests due to recent lhotse changes

* Fix wenetspeech

* Fix style issues
 2022-05-23 23:37:35 +08:00
Fangjun Kuang
0f180b3ce2
Validate that there are no OOV tokens in BPE-based lexicons. (#359) 
* Validate that there are no OOV tokens in BPE-based lexicons.

* Typo fixes.
 2022-05-13 14:00:35 +08:00
Fangjun Kuang
6af15914fa
Validate generated manifest files. (#338) 2022-05-03 07:02:54 +08:00
Fangjun Kuang
ac84220de9
Modified conformer with multi datasets (#312) 
* Copy files for editing.

* Use librispeech + gigaspeech with modified conformer.

* Support specifying number of workers for on-the-fly feature extraction.

* Feature extraction code for GigaSpeech.

* Combine XL splits lazily during training.

* Fix warnings in decoding.

* Add decoding code for GigaSpeech.

* Fix decoding the gigaspeech dataset.

We have to use the decoder/joiner networks for the GigaSpeech dataset.

* Disable speed perturbe for XL subset.

* Compute the Nbest oracle WER for RNN-T decoding.

* Minor fixes.

* Minor fixes.

* Add results.

* Update results.

* Update CI.

* Update results.

* Fix style issues.

* Update results.

* Fix style issues.
 2022-04-29 15:40:30 +08:00
Wei Kang
021c79824e
Add LG decoding (#277) 
* Add LG decoding

* Add log weight pushing

* Minor fixes
 2022-04-19 17:23:46 +08:00
Fangjun Kuang
2332ba312d
Begin to use multiple datasets in training (#213) 
* Begin to use multiple datasets.

* Finish preparing training datasets.

* Minor fixes

* Copy files.

* Finish training code.

* Display losses for gigaspeech and librispeech separately.

* Fix decode.py

* Make the probability to select a batch from GigaSpeech configurable.

* Update results.

* Minor fixes.
 2022-02-21 15:27:27 +08:00
Fangjun Kuang
1d44da845b
RNN-T Conformer training for LibriSpeech (#143) 
* Begin to add RNN-T training for librispeech.

* Copy files from conformer_ctc.

Will edit it.

* Use conformer/transformer model as encoder.

* Begin to add training script.

* Add training code.

* Remove long utterances to avoid OOM when a large max_duraiton is used.

* Begin to add decoding script.

* Add decoding script.

* Minor fixes.

* Add beam search.

* Use LSTM layers for the encoder.

Need more tunings.

* Use stateless decoder.

* Minor fixes to make it ready for merge.

* Fix README.

* Update RESULT.md to include RNN-T Conformer.

* Minor fixes.

* Fix tests.

* Minor fixes.

* Minor fixes.

* Fix tests.
 2021-12-18 07:42:51 +08:00
Fangjun Kuang
68506609ad
Set fsa.properties to None after changing its labels in-place. (#121) 2021-11-16 23:11:30 +08:00
Fangjun Kuang
53b79fafa7
Add MMI training with word pieces as modelling unit. (#6) 
* Fix an error in TDNN-LSTM training.

* WIP: Refactoring

* Refactor transformer.py

* Remove unused code.

* Minor fixes.

* Fix decoder padding mask.

* Add MMI training with word pieces.

* Remove unused files.

* Minor fixes.

* Refactoring.

* Minor fixes.

* Use pre-computed alignments in LF-MMI training.

* Minor fixes.

* Update decoding script.

* Add doc about how to check and use extracted alignments.

* Fix style issues.

* Fix typos.

* Fix style issues.

* Disable macOS tests for now.
 2021-10-18 15:20:32 +08:00
Wei Kang
9a6e0489c8
update api for RaggedTensor (#45) 
* Fix code style

* update k2 version in CI

* fix compile hlg
 2021-09-14 16:39:56 +08:00
Fangjun Kuang
abadc71415
Use new APIs with k2.RaggedTensor (#38) 
* Use new APIs with k2.RaggedTensor

* Fix style issues.

* Update the installation doc, saying it requires at least k2 v1.7

* Use k2 v1.7
 2021-09-08 14:55:30 +08:00
Fangjun Kuang
6c2c9b9d74
Add recipe for the yes_no dataset. (#16) 
* Add recipe for the yes_no dataset.

* Refactoring: Remove unused code.

* Add Colab notebook for the yesno dataset.

* Add GitHub actions to run yesno.

* Fix a typo.

* Minor fixes.

* Train more epochs for GitHub actions.

* Minor fixes.

* Minor fixes.

* Fix style issues.
 2021-08-23 11:36:29 +08:00
pkufool
19c4214958
Fix code style and add copyright. (#18) 
* Fix style and add copyright

* Minor fix

* Remove duplicate lines

* Reformat conformer.py by black

* Reformat code style with black.

* Fix github workflows

* Fix lhotse installation

* Install icefall requirements

* Update k2 version, remove lhotse from test workflow
 2021-08-23 10:43:59 +08:00
Fangjun Kuang
1c3b13c7eb
Minor fixes. (#9) 2021-08-16 19:01:25 +08:00
Fangjun Kuang
5a0b9bcb23
Refactoring (#4) 
* Fix an error in TDNN-LSTM training.

* WIP: Refactoring

* Refactor transformer.py

* Remove unused code.

* Minor fixes.
 2021-08-04 14:53:02 +08:00
Fangjun Kuang
acc63a9172 WIP: Add BPE training code. 2021-07-29 20:23:52 +08:00
Fangjun Kuang
bd69e4be32 Use attention decoder for rescoring. 2021-07-28 12:22:09 +08:00
Fangjun Kuang
f65854cca5 Add BPE decoding results. 2021-07-27 17:38:47 +08:00
Fangjun Kuang
4ccae509d3 WIP: Begin to add BPE decoding 2021-07-26 20:06:58 +08:00
Fangjun Kuang
2e33e24348 Add CI test. 2021-07-24 17:47:41 +08:00
Fangjun Kuang
f3542c7793 Add CTC training. 2021-07-24 17:13:20 +08:00
Fangjun Kuang
a01d08f73c Add self-loops to propagate disambiguation symbols. 2021-07-21 13:12:20 +08:00
Fangjun Kuang
8a72901f3a Minor fixes. 2021-07-20 19:54:12 +08:00
Fangjun Kuang
d5e0408698 Add prepare_lang.py based on prepare_lang.sh 2021-07-20 19:41:21 +08:00
Fangjun Kuang
e005ea062c Minor fixes after review. 2021-07-20 10:02:20 +08:00