mirror of
https://github.com/k2-fsa/icefall.git
synced 2025-08-09 01:52:41 +00:00
Update installation doc. (#1188)
This commit is contained in:
Fangjun Kuang
GitHub
parent
3fb0a43170
commit
19b942c958
@ -90,4 +90,9 @@ rst_epilog = """
|
|||||||
.. _musan: http://www.openslr.org/17/
|
.. _musan: http://www.openslr.org/17/
|
||||||
.. _ONNX: https://github.com/onnx/onnx
|
.. _ONNX: https://github.com/onnx/onnx
|
||||||
.. _onnxruntime: https://github.com/microsoft/onnxruntime
|
.. _onnxruntime: https://github.com/microsoft/onnxruntime
|
||||||
|
.. _torch: https://github.com/pytorch/pytorch
|
||||||
|
.. _torchaudio: https://github.com/pytorch/audio
|
||||||
|
.. _k2: https://github.com/k2-fsa/k2
|
||||||
|
.. _lhotse: https://github.com/lhotse-speech/lhotse
|
||||||
|
.. _yesno: https://www.openslr.org/1/
|
||||||
"""
|
"""
|
||||||
|
|||||||
@ -3,40 +3,23 @@
|
|||||||
Installation
|
Installation
|
||||||
============
|
============
|
||||||
|
|
||||||
|
.. hint::
|
||||||
|
|
||||||
|
We have a colab notebook guiding you step by step to setup the environment.
|
||||||
|
|
||||||
``icefall`` depends on `k2 <https://github.com/k2-fsa/k2>`_ and
|
|yesno colab notebook|
|
||||||
`lhotse <https://github.com/lhotse-speech/lhotse>`_.
|
|
||||||
|
.. |yesno colab notebook| image:: https://colab.research.google.com/assets/colab-badge.svg
|
||||||
|
:target: https://colab.research.google.com/drive/1tIjjzaJc3IvGyKiMCDWO-TSnBgkcuN3B?usp=sharing
|
||||||
|
|
||||||
|
`icefall`_ depends on `k2`_ and `lhotse`_.
|
||||||
|
|
||||||
We recommend that you use the following steps to install the dependencies.
|
We recommend that you use the following steps to install the dependencies.
|
||||||
|
|
||||||
- (0) Install CUDA toolkit and cuDNN
|
- (0) Install CUDA toolkit and cuDNN
|
||||||
- (1) Install PyTorch and torchaudio
|
- (1) Install `torch`_ and `torchaudio`_
|
||||||
- (2) Install k2
|
- (2) Install `k2`_
|
||||||
- (3) Install lhotse
|
- (3) Install `lhotse`_
|
||||||
|
|
||||||
.. caution::
|
|
||||||
|
|
||||||
99% users who have issues about the installation are using conda.
|
|
||||||
|
|
||||||
.. caution::
|
|
||||||
|
|
||||||
99% users who have issues about the installation are using conda.
|
|
||||||
|
|
||||||
.. caution::
|
|
||||||
|
|
||||||
99% users who have issues about the installation are using conda.
|
|
||||||
|
|
||||||
.. hint::
|
|
||||||
|
|
||||||
We suggest that you use ``pip install`` to install PyTorch.
|
|
||||||
|
|
||||||
You can use the following command to create a virutal environment in Python:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
python3 -m venv ./my_env
|
|
||||||
source ./my_env/bin/activate
|
|
||||||
|
|
||||||
.. caution::
|
.. caution::
|
||||||
|
|
||||||
@ -50,27 +33,20 @@ Please refer to
|
|||||||
to install CUDA and cuDNN.
|
to install CUDA and cuDNN.
|
||||||
|
|
||||||
|
|
||||||
(1) Install PyTorch and torchaudio
|
(1) Install torch and torchaudio
|
||||||
----------------------------------
|
--------------------------------
|
||||||
|
|
||||||
Please refer `<https://pytorch.org/>`_ to install PyTorch
|
Please refer `<https://pytorch.org/>`_ to install `torch`_ and `torchaudio`_.
|
||||||
and torchaudio.
|
|
||||||
|
|
||||||
.. hint::
|
|
||||||
|
|
||||||
You can also go to `<https://download.pytorch.org/whl/torch_stable.html>`_
|
|
||||||
to download pre-compiled wheels and install them.
|
|
||||||
|
|
||||||
.. caution::
|
.. caution::
|
||||||
|
|
||||||
Please install torch and torchaudio at the same time.
|
Please install torch and torchaudio at the same time.
|
||||||
|
|
||||||
|
|
||||||
(2) Install k2
|
(2) Install k2
|
||||||
--------------
|
--------------
|
||||||
|
|
||||||
Please refer to `<https://k2-fsa.github.io/k2/installation/index.html>`_
|
Please refer to `<https://k2-fsa.github.io/k2/installation/index.html>`_
|
||||||
to install ``k2``.
|
to install `k2`_.
|
||||||
|
|
||||||
.. caution::
|
.. caution::
|
||||||
|
|
||||||
@ -78,21 +54,18 @@ to install ``k2``.
|
|||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
We suggest that you install k2 from source by following
|
We suggest that you install k2 from pre-compiled wheels by following
|
||||||
`<https://k2-fsa.github.io/k2/installation/from_source.html>`_
|
`<https://k2-fsa.github.io/k2/installation/from_wheels.html>`_
|
||||||
or
|
|
||||||
`<https://k2-fsa.github.io/k2/installation/for_developers.html>`_.
|
|
||||||
|
|
||||||
.. hint::
|
.. hint::
|
||||||
|
|
||||||
Please always install the latest version of k2.
|
Please always install the latest version of `k2`_.
|
||||||
|
|
||||||
(3) Install lhotse
|
(3) Install lhotse
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
Please refer to `<https://lhotse.readthedocs.io/en/latest/getting-started.html#installation>`_
|
Please refer to `<https://lhotse.readthedocs.io/en/latest/getting-started.html#installation>`_
|
||||||
to install ``lhotse``.
|
to install `lhotse`_.
|
||||||
|
|
||||||
|
|
||||||
.. hint::
|
.. hint::
|
||||||
|
|
||||||
@ -100,17 +73,16 @@ to install ``lhotse``.
|
|||||||
|
|
||||||
pip install git+https://github.com/lhotse-speech/lhotse
|
pip install git+https://github.com/lhotse-speech/lhotse
|
||||||
|
|
||||||
to install the latest version of lhotse.
|
to install the latest version of `lhotse`_.
|
||||||
|
|
||||||
(4) Download icefall
|
(4) Download icefall
|
||||||
--------------------
|
--------------------
|
||||||
|
|
||||||
``icefall`` is a collection of Python scripts; what you need is to download it
|
`icefall`_ is a collection of Python scripts; what you need is to download it
|
||||||
and set the environment variable ``PYTHONPATH`` to point to it.
|
and set the environment variable ``PYTHONPATH`` to point to it.
|
||||||
|
|
||||||
Assume you want to place ``icefall`` in the folder ``/tmp``. The
|
Assume you want to place `icefall`_ in the folder ``/tmp``. The
|
||||||
following commands show you how to setup ``icefall``:
|
following commands show you how to setup `icefall`_:
|
||||||
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
@ -122,285 +94,334 @@ following commands show you how to setup ``icefall``:
|
|||||||
|
|
||||||
.. HINT::
|
.. HINT::
|
||||||
|
|
||||||
You can put several versions of ``icefall`` in the same virtual environment.
|
You can put several versions of `icefall`_ in the same virtual environment.
|
||||||
To switch among different versions of ``icefall``, just set ``PYTHONPATH``
|
To switch among different versions of `icefall`_, just set ``PYTHONPATH``
|
||||||
to point to the version you want.
|
to point to the version you want.
|
||||||
|
|
||||||
|
|
||||||
Installation example
|
Installation example
|
||||||
--------------------
|
--------------------
|
||||||
|
|
||||||
The following shows an example about setting up the environment.
|
The following shows an example about setting up the environment.
|
||||||
|
|
||||||
|
|
||||||
(1) Create a virtual environment
|
(1) Create a virtual environment
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
$ virtualenv -p python3.8 test-icefall
|
kuangfangjun:~$ virtualenv -p python3.8 test-icefall
|
||||||
|
created virtual environment CPython3.8.0.final.0-64 in 9422ms
|
||||||
|
creator CPython3Posix(dest=/star-fj/fangjun/test-icefall, clear=False, no_vcs_ignore=False, global=False)
|
||||||
|
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/star-fj/fangjun/.local/share/virtualenv)
|
||||||
|
added seed packages: pip==22.3.1, setuptools==65.6.3, wheel==0.38.4
|
||||||
|
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
|
||||||
|
|
||||||
created virtual environment CPython3.8.6.final.0-64 in 1540ms
|
kuangfangjun:~$ source test-icefall/bin/activate
|
||||||
creator CPython3Posix(dest=/ceph-fj/fangjun/test-icefall, clear=False, no_vcs_ignore=False, global=False)
|
|
||||||
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/fangjun/.local/share/v
|
|
||||||
irtualenv)
|
|
||||||
added seed packages: pip==21.1.3, setuptools==57.4.0, wheel==0.36.2
|
|
||||||
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
|
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$
|
||||||
|
|
||||||
(2) Activate your virtual environment
|
(2) Install CUDA toolkit and cuDNN
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
You need to determine the version of CUDA toolkit to install.
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
$ source test-icefall/bin/activate
|
(test-icefall) kuangfangjun:~$ nvidia-smi | head -n 4
|
||||||
|
|
||||||
(3) Install k2
|
Wed Jul 26 21:57:49 2023
|
||||||
|
+-----------------------------------------------------------------------------+
|
||||||
|
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|
||||||
|
|-------------------------------+----------------------+----------------------+
|
||||||
|
|
||||||
|
You can choose any CUDA version that is ``not`` greater than the version printed by ``nvidia-smi``.
|
||||||
|
In our case, we can choose any version ``<= 11.6``.
|
||||||
|
|
||||||
|
We will use ``CUDA 11.6`` in this example. Please follow
|
||||||
|
`<https://k2-fsa.github.io/k2/installation/cuda-cudnn.html#cuda-11-6>`_
|
||||||
|
to install CUDA toolkit and cuDNN if you have not done that before.
|
||||||
|
|
||||||
|
After installing CUDA toolkit, you can use the following command to verify it:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$ nvcc --version
|
||||||
|
|
||||||
|
nvcc: NVIDIA (R) Cuda compiler driver
|
||||||
|
Copyright (c) 2005-2019 NVIDIA Corporation
|
||||||
|
Built on Wed_Oct_23_19:24:38_PDT_2019
|
||||||
|
Cuda compilation tools, release 10.2, V10.2.89
|
||||||
|
|
||||||
|
(3) Install torch and torchaudio
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Since we have selected CUDA toolkit ``11.6``, we have to install a version of `torch`_
|
||||||
|
that is compiled against CUDA ``11.6``. We select ``torch 1.13.0+cu116`` in this
|
||||||
|
example.
|
||||||
|
|
||||||
|
After selecting the version of `torch`_ to install, we need to also install
|
||||||
|
a compatible version of `torchaudio`_, which is ``0.13.0+cu116`` in our case.
|
||||||
|
|
||||||
|
Please refer to `<https://pytorch.org/audio/stable/installation.html#compatibility-matrix>`_
|
||||||
|
to select an appropriate version of `torchaudio`_ to install if you use a different
|
||||||
|
version of `torch`_.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$ pip install torch==1.13.0+cu116 torchaudio==0.13.0+cu116 -f https://download.pytorch.org/whl/torch_stable.html
|
||||||
|
|
||||||
|
Looking in links: https://download.pytorch.org/whl/torch_stable.html
|
||||||
|
Collecting torch==1.13.0+cu116
|
||||||
|
Downloading https://download.pytorch.org/whl/cu116/torch-1.13.0%2Bcu116-cp38-cp38-linux_x86_64.whl (1983.0 MB)
|
||||||
|
________________________________________ 2.0/2.0 GB 764.4 kB/s eta 0:00:00
|
||||||
|
Collecting torchaudio==0.13.0+cu116
|
||||||
|
Downloading https://download.pytorch.org/whl/cu116/torchaudio-0.13.0%2Bcu116-cp38-cp38-linux_x86_64.whl (4.2 MB)
|
||||||
|
________________________________________ 4.2/4.2 MB 1.3 MB/s eta 0:00:00
|
||||||
|
Requirement already satisfied: typing-extensions in /star-fj/fangjun/test-icefall/lib/python3.8/site-packages (from torch==1.13.0+cu116) (4.7.1)
|
||||||
|
Installing collected packages: torch, torchaudio
|
||||||
|
Successfully installed torch-1.13.0+cu116 torchaudio-0.13.0+cu116
|
||||||
|
|
||||||
|
Verify that `torch`_ and `torchaudio`_ are successfully installed:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$ python3 -c "import torch; print(torch.__version__)"
|
||||||
|
|
||||||
|
1.13.0+cu116
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$ python3 -c "import torchaudio; print(torchaudio.__version__)"
|
||||||
|
|
||||||
|
0.13.0+cu116
|
||||||
|
|
||||||
|
(4) Install k2
|
||||||
~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
We will install `k2`_ from pre-compiled wheels by following
|
||||||
|
`<https://k2-fsa.github.io/k2/installation/from_wheels.html>`_
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
$ pip install k2==1.4.dev20210822+cpu.torch1.9.0 -f https://k2-fsa.org/nightly/index.html
|
(test-icefall) kuangfangjun:~$ pip install k2==1.24.3.dev20230725+cuda11.6.torch1.13.0 -f https://k2-fsa.github.io/k2/cuda.html
|
||||||
|
|
||||||
Looking in links: https://k2-fsa.org/nightly/index.html
|
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
|
||||||
Collecting k2==1.4.dev20210822+cpu.torch1.9.0
|
Looking in links: https://k2-fsa.github.io/k2/cuda.html
|
||||||
Downloading https://k2-fsa.org/nightly/whl/k2-1.4.dev20210822%2Bcpu.torch1.9.0-cp38-cp38-linux_x86_64.whl (1.6 MB)
|
Collecting k2==1.24.3.dev20230725+cuda11.6.torch1.13.0
|
||||||
|________________________________| 1.6 MB 185 kB/s
|
Downloading https://huggingface.co/csukuangfj/k2/resolve/main/ubuntu-cuda/k2-1.24.3.dev20230725%2Bcuda11.6.torch1.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (104.3 MB)
|
||||||
|
________________________________________ 104.3/104.3 MB 5.1 MB/s eta 0:00:00
|
||||||
|
Requirement already satisfied: torch==1.13.0 in /star-fj/fangjun/test-icefall/lib/python3.8/site-packages (from k2==1.24.3.dev20230725+cuda11.6.torch1.13.0) (1.13.0+cu116)
|
||||||
Collecting graphviz
|
Collecting graphviz
|
||||||
Downloading graphviz-0.17-py3-none-any.whl (18 kB)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/de/5e/fcbb22c68208d39edff467809d06c9d81d7d27426460ebc598e55130c1aa/graphviz-0.20.1-py3-none-any.whl (47 kB)
|
||||||
Collecting torch==1.9.0
|
Requirement already satisfied: typing-extensions in /star-fj/fangjun/test-icefall/lib/python3.8/site-packages (from torch==1.13.0->k2==1.24.3.dev20230725+cuda11.6.torch1.13.0) (4.7.1)
|
||||||
Using cached torch-1.9.0-cp38-cp38-manylinux1_x86_64.whl (831.4 MB)
|
Installing collected packages: graphviz, k2
|
||||||
Collecting typing-extensions
|
Successfully installed graphviz-0.20.1 k2-1.24.3.dev20230725+cuda11.6.torch1.13.0
|
||||||
Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
|
|
||||||
Installing collected packages: typing-extensions, torch, graphviz, k2
|
|
||||||
Successfully installed graphviz-0.17 k2-1.4.dev20210822+cpu.torch1.9.0 torch-1.9.0 typing-extensions-3.10.0.0
|
|
||||||
|
|
||||||
.. WARNING::
|
.. hint::
|
||||||
|
|
||||||
We choose to install a CPU version of k2 for testing. You would probably want to install
|
Please refer to `<https://k2-fsa.github.io/k2/cuda.html>`_ for the available
|
||||||
a CUDA version of k2.
|
pre-compiled wheels about `k2`_.
|
||||||
|
|
||||||
|
Verify that `k2`_ has been installed successfully:
|
||||||
|
|
||||||
(4) Install lhotse
|
.. code-block:: bash
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$ python3 -m k2.version
|
||||||
|
|
||||||
|
Collecting environment information...
|
||||||
|
|
||||||
|
k2 version: 1.24.3
|
||||||
|
Build type: Release
|
||||||
|
Git SHA1: 4c05309499a08454997adf500b56dcc629e35ae5
|
||||||
|
Git date: Tue Jul 25 16:23:36 2023
|
||||||
|
Cuda used to build k2: 11.6
|
||||||
|
cuDNN used to build k2: 8.3.2
|
||||||
|
Python version used to build k2: 3.8
|
||||||
|
OS used to build k2: CentOS Linux release 7.9.2009 (Core)
|
||||||
|
CMake version: 3.27.0
|
||||||
|
GCC version: 9.3.1
|
||||||
|
CMAKE_CUDA_FLAGS: -Wno-deprecated-gpu-targets -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_35,code=sm_35 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_50,code=sm_50 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_60,code=sm_60 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_61,code=sm_61 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_75,code=sm_75 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_80,code=sm_80 -lineinfo --expt-extended-lambda -use_fast_math -Xptxas=-w --expt-extended-lambda -gencode arch=compute_86,code=sm_86 -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas
|
||||||
|
CMAKE_CXX_FLAGS: -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable -Wno-strict-overflow
|
||||||
|
PyTorch version used to build k2: 1.13.0+cu116
|
||||||
|
PyTorch is using Cuda: 11.6
|
||||||
|
NVTX enabled: True
|
||||||
|
With CUDA: True
|
||||||
|
Disable debug: True
|
||||||
|
Sync kernels : False
|
||||||
|
Disable checks: False
|
||||||
|
Max cpu memory allocate: 214748364800 bytes (or 200.0 GB)
|
||||||
|
k2 abort: False
|
||||||
|
__file__: /star-fj/fangjun/test-icefall/lib/python3.8/site-packages/k2/version/version.py
|
||||||
|
_k2.__file__: /star-fj/fangjun/test-icefall/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so
|
||||||
|
|
||||||
|
(5) Install lhotse
|
||||||
~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
.. code-block::
|
.. code-block:: bash
|
||||||
|
|
||||||
$ pip install git+https://github.com/lhotse-speech/lhotse
|
(test-icefall) kuangfangjun:~$ pip install git+https://github.com/lhotse-speech/lhotse
|
||||||
|
|
||||||
Collecting git+https://github.com/lhotse-speech/lhotse
|
Collecting git+https://github.com/lhotse-speech/lhotse
|
||||||
Cloning https://github.com/lhotse-speech/lhotse to /tmp/pip-req-build-7b1b76ge
|
Cloning https://github.com/lhotse-speech/lhotse to /tmp/pip-req-build-vq12fd5i
|
||||||
Running command git clone -q https://github.com/lhotse-speech/lhotse /tmp/pip-req-build-7b1b76ge
|
Running command git clone --filter=blob:none --quiet https://github.com/lhotse-speech/lhotse /tmp/pip-req-build-vq12fd5i
|
||||||
Collecting audioread>=2.1.9
|
Resolved https://github.com/lhotse-speech/lhotse to commit 7640d663469b22cd0b36f3246ee9b849cd25e3b7
|
||||||
Using cached audioread-2.1.9-py3-none-any.whl
|
Installing build dependencies ... done
|
||||||
Collecting SoundFile>=0.10
|
Getting requirements to build wheel ... done
|
||||||
Using cached SoundFile-0.10.3.post1-py2.py3-none-any.whl (21 kB)
|
Preparing metadata (pyproject.toml) ... done
|
||||||
Collecting click>=7.1.1
|
|
||||||
Using cached click-8.0.1-py3-none-any.whl (97 kB)
|
|
||||||
Collecting cytoolz>=0.10.1
|
Collecting cytoolz>=0.10.1
|
||||||
Using cached cytoolz-0.11.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.9 MB)
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1e/3b/a7828d575aa17fb7acaf1ced49a3655aa36dad7e16eb7e6a2e4df0dda76f/cytoolz-0.12.2-cp38-cp38-
|
||||||
Collecting dataclasses
|
manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
|
||||||
Using cached dataclasses-0.6-py3-none-any.whl (14 kB)
|
________________________________________ 2.0/2.0 MB 33.2 MB/s eta 0:00:00
|
||||||
Collecting h5py>=2.10.0
|
|
||||||
Downloading h5py-3.4.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
|
|
||||||
|________________________________| 4.5 MB 684 kB/s
|
|
||||||
Collecting intervaltree>=3.1.0
|
|
||||||
Using cached intervaltree-3.1.0-py2.py3-none-any.whl
|
|
||||||
Collecting lilcom>=1.1.0
|
|
||||||
Using cached lilcom-1.1.1-cp38-cp38-linux_x86_64.whl
|
|
||||||
Collecting numpy>=1.18.1
|
|
||||||
Using cached numpy-1.21.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.8 MB)
|
|
||||||
Collecting packaging
|
|
||||||
Using cached packaging-21.0-py3-none-any.whl (40 kB)
|
|
||||||
Collecting pyyaml>=5.3.1
|
Collecting pyyaml>=5.3.1
|
||||||
Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c8/6b/6600ac24725c7388255b2f5add93f91e58a5d7efaf4af244fdbcc11a541b/PyYAML-6.0.1-cp38-cp38-ma
|
||||||
|
nylinux_2_17_x86_64.manylinux2014_x86_64.whl (736 kB)
|
||||||
|
________________________________________ 736.6/736.6 kB 38.6 MB/s eta 0:00:00
|
||||||
|
Collecting dataclasses
|
||||||
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/26/2f/1095cdc2868052dd1e64520f7c0d5c8c550ad297e944e641dbf1ffbb9a5d/dataclasses-0.6-py3-none-
|
||||||
|
any.whl (14 kB)
|
||||||
|
Requirement already satisfied: torchaudio in ./test-icefall/lib/python3.8/site-packages (from lhotse==1.16.0.dev0+git.7640d66.clean) (0.13.0+cu116)
|
||||||
|
Collecting lilcom>=1.1.0
|
||||||
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a8/65/df0a69c52bd085ca1ad4e5c4c1a5c680e25f9477d8e49316c4ff1e5084a4/lilcom-1.7-cp38-cp38-many
|
||||||
|
linux_2_17_x86_64.manylinux2014_x86_64.whl (87 kB)
|
||||||
|
________________________________________ 87.1/87.1 kB 8.7 MB/s eta 0:00:00
|
||||||
Collecting tqdm
|
Collecting tqdm
|
||||||
Downloading tqdm-4.62.1-py2.py3-none-any.whl (76 kB)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/e6/02/a2cff6306177ae6bc73bc0665065de51dfb3b9db7373e122e2735faf0d97/tqdm-4.65.0-py3-none-any
|
||||||
|________________________________| 76 kB 2.7 MB/s
|
.whl (77 kB)
|
||||||
Collecting torchaudio==0.9.0
|
Requirement already satisfied: numpy>=1.18.1 in ./test-icefall/lib/python3.8/site-packages (from lhotse==1.16.0.dev0+git.7640d66.clean) (1.24.4)
|
||||||
Downloading torchaudio-0.9.0-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
|
Collecting audioread>=2.1.9
|
||||||
|________________________________| 1.9 MB 73.1 MB/s
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/5d/cb/82a002441902dccbe427406785db07af10182245ee639ea9f4d92907c923/audioread-3.0.0.tar.gz (
|
||||||
Requirement already satisfied: torch==1.9.0 in ./test-icefall/lib/python3.8/site-packages (from torchaudio==0.9.0->lhotse===0.8.0.dev
|
377 kB)
|
||||||
-2a1410b-clean) (1.9.0)
|
Preparing metadata (setup.py) ... done
|
||||||
Requirement already satisfied: typing-extensions in ./test-icefall/lib/python3.8/site-packages (from torch==1.9.0->torchaudio==0.9.0-
|
Collecting tabulate>=0.8.1
|
||||||
>lhotse===0.8.0.dev-2a1410b-clean) (3.10.0.0)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/40/44/4a5f08c96eb108af5cb50b41f76142f0afa346dfa99d5296fe7202a11854/tabulate-0.9.0-py3-none-
|
||||||
|
any.whl (35 kB)
|
||||||
|
Collecting click>=7.1.1
|
||||||
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.
|
||||||
|
whl (97 kB)
|
||||||
|
________________________________________ 97.9/97.9 kB 8.4 MB/s eta 0:00:00
|
||||||
|
Collecting packaging
|
||||||
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ab/c3/57f0601a2d4fe15de7a553c00adbc901425661bf048f2a22dfc500caf121/packaging-23.1-py3-none-
|
||||||
|
any.whl (48 kB)
|
||||||
|
Collecting intervaltree>=3.1.0
|
||||||
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/50/fb/396d568039d21344639db96d940d40eb62befe704ef849b27949ded5c3bb/intervaltree-3.1.0.tar.gz
|
||||||
|
(32 kB)
|
||||||
|
Preparing metadata (setup.py) ... done
|
||||||
|
Requirement already satisfied: torch in ./test-icefall/lib/python3.8/site-packages (from lhotse==1.16.0.dev0+git.7640d66.clean) (1.13.0+cu116)
|
||||||
|
Collecting SoundFile>=0.10
|
||||||
|
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ad/bd/0602167a213d9184fc688b1086dc6d374b7ae8c33eccf169f9b50ce6568c/soundfile-0.12.1-py2.py3-
|
||||||
|
none-manylinux_2_17_x86_64.whl (1.3 MB)
|
||||||
|
________________________________________ 1.3/1.3 MB 46.5 MB/s eta 0:00:00
|
||||||
Collecting toolz>=0.8.0
|
Collecting toolz>=0.8.0
|
||||||
Using cached toolz-0.11.1-py3-none-any.whl (55 kB)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/7f/5c/922a3508f5bda2892be3df86c74f9cf1e01217c2b1f8a0ac4841d903e3e9/toolz-0.12.0-py3-none-any.whl (55 kB)
|
||||||
Collecting sortedcontainers<3.0,>=2.0
|
Collecting sortedcontainers<3.0,>=2.0
|
||||||
Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
|
||||||
Collecting cffi>=1.0
|
Collecting cffi>=1.0
|
||||||
Using cached cffi-1.14.6-cp38-cp38-manylinux1_x86_64.whl (411 kB)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b7/8b/06f30caa03b5b3ac006de4f93478dbd0239e2a16566d81a106c322dc4f79/cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)
|
||||||
|
Requirement already satisfied: typing-extensions in ./test-icefall/lib/python3.8/site-packages (from torch->lhotse==1.16.0.dev0+git.7640d66.clean) (4.7.1)
|
||||||
Collecting pycparser
|
Collecting pycparser
|
||||||
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
|
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl (118 kB)
|
||||||
Collecting pyparsing>=2.0.2
|
Building wheels for collected packages: lhotse, audioread, intervaltree
|
||||||
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
|
Building wheel for lhotse (pyproject.toml) ... done
|
||||||
Building wheels for collected packages: lhotse
|
Created wheel for lhotse: filename=lhotse-1.16.0.dev0+git.7640d66.clean-py3-none-any.whl size=687627 sha256=cbf0a4d2d0b639b33b91637a4175bc251d6a021a069644ecb1a9f2b3a83d072a
|
||||||
Building wheel for lhotse (setup.py) ... done
|
Stored in directory: /tmp/pip-ephem-wheel-cache-wwtk90_m/wheels/7f/7a/8e/a0bf241336e2e3cb573e1e21e5600952d49f5162454f2e612f
|
||||||
Created wheel for lhotse: filename=lhotse-0.8.0.dev_2a1410b_clean-py3-none-any.whl size=342242 sha256=f683444afa4dc0881133206b4646a
|
Building wheel for audioread (setup.py) ... done
|
||||||
9d0f774224cc84000f55d0a67f6e4a37997
|
Created wheel for audioread: filename=audioread-3.0.0-py3-none-any.whl size=23704 sha256=5e2d3537c96ce9cf0f645a654c671163707bf8cb8d9e358d0e2b0939a85ff4c2
|
||||||
Stored in directory: /tmp/pip-ephem-wheel-cache-ftu0qysz/wheels/7f/7a/8e/a0bf241336e2e3cb573e1e21e5600952d49f5162454f2e612f
|
Stored in directory: /star-fj/fangjun/.cache/pip/wheels/e2/c3/9c/f19ae5a03f8862d9f0776b0c0570f1fdd60a119d90954e3f39
|
||||||
WARNING: Built wheel for lhotse is invalid: Metadata 1.2 mandates PEP 440 version, but '0.8.0.dev-2a1410b-clean' is not
|
Building wheel for intervaltree (setup.py) ... done
|
||||||
Failed to build lhotse
|
Created wheel for intervaltree: filename=intervaltree-3.1.0-py2.py3-none-any.whl size=26098 sha256=2604170976cfffe0d2f678cb1a6e5b525f561cd50babe53d631a186734fec9f9
|
||||||
Installing collected packages: pycparser, toolz, sortedcontainers, pyparsing, numpy, cffi, tqdm, torchaudio, SoundFile, pyyaml, packa
|
Stored in directory: /star-fj/fangjun/.cache/pip/wheels/f3/ed/2b/c179ebfad4e15452d6baef59737f27beb9bfb442e0620f7271
|
||||||
ging, lilcom, intervaltree, h5py, dataclasses, cytoolz, click, audioread, lhotse
|
Successfully built lhotse audioread intervaltree
|
||||||
Running setup.py install for lhotse ... done
|
Installing collected packages: sortedcontainers, dataclasses, tqdm, toolz, tabulate, pyyaml, pycparser, packaging, lilcom, intervaltree, click, audioread, cytoolz, cffi, SoundFile, lhotse
|
||||||
DEPRECATION: lhotse was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible
|
Successfully installed SoundFile-0.12.1 audioread-3.0.0 cffi-1.15.1 click-8.1.6 cytoolz-0.12.2 dataclasses-0.6 intervaltree-3.1.0 lhotse-1.16.0.dev0+git.7640d66.clean lilcom-1.7 packaging-23.1 pycparser-2.21 pyyaml-6.0.1 sortedcontainers-2.4.0 tabulate-0.9.0 toolz-0.12.0 tqdm-4.65.0
|
||||||
replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/is
|
|
||||||
sues/8368.
|
|
||||||
Successfully installed SoundFile-0.10.3.post1 audioread-2.1.9 cffi-1.14.6 click-8.0.1 cytoolz-0.11.0 dataclasses-0.6 h5py-3.4.0 inter
|
|
||||||
valtree-3.1.0 lhotse-0.8.0.dev-2a1410b-clean lilcom-1.1.1 numpy-1.21.2 packaging-21.0 pycparser-2.20 pyparsing-2.4.7 pyyaml-5.4.1 sor
|
|
||||||
tedcontainers-2.4.0 toolz-0.11.1 torchaudio-0.9.0 tqdm-4.62.1
|
|
||||||
|
|
||||||
(5) Download icefall
|
|
||||||
|
Verify that `lhotse`_ has been installed successfully:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:~$ python3 -c "import lhotse; print(lhotse.__version__)"
|
||||||
|
|
||||||
|
1.16.0.dev+git.7640d66.clean
|
||||||
|
|
||||||
|
(6) Download icefall
|
||||||
~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
.. code-block::
|
.. code-block:: bash
|
||||||
|
|
||||||
$ cd /tmp
|
(test-icefall) kuangfangjun:~$ cd /tmp/
|
||||||
$ git clone https://github.com/k2-fsa/icefall
|
|
||||||
|
(test-icefall) kuangfangjun:tmp$ git clone https://github.com/k2-fsa/icefall
|
||||||
|
|
||||||
Cloning into 'icefall'...
|
Cloning into 'icefall'...
|
||||||
remote: Enumerating objects: 500, done.
|
remote: Enumerating objects: 12942, done.
|
||||||
remote: Counting objects: 100% (500/500), done.
|
remote: Counting objects: 100% (67/67), done.
|
||||||
remote: Compressing objects: 100% (308/308), done.
|
remote: Compressing objects: 100% (56/56), done.
|
||||||
remote: Total 500 (delta 263), reused 307 (delta 102), pack-reused 0
|
remote: Total 12942 (delta 17), reused 35 (delta 6), pack-reused 12875
|
||||||
Receiving objects: 100% (500/500), 172.49 KiB | 385.00 KiB/s, done.
|
Receiving objects: 100% (12942/12942), 14.77 MiB | 9.29 MiB/s, done.
|
||||||
Resolving deltas: 100% (263/263), done.
|
Resolving deltas: 100% (8835/8835), done.
|
||||||
|
|
||||||
$ cd icefall
|
(test-icefall) kuangfangjun:tmp$ cd icefall/
|
||||||
$ pip install -r requirements.txt
|
|
||||||
|
|
||||||
Collecting kaldilm
|
|
||||||
Downloading kaldilm-1.8.tar.gz (48 kB)
|
|
||||||
|________________________________| 48 kB 574 kB/s
|
|
||||||
Collecting kaldialign
|
|
||||||
Using cached kaldialign-0.2-cp38-cp38-linux_x86_64.whl
|
|
||||||
Collecting sentencepiece>=0.1.96
|
|
||||||
Using cached sentencepiece-0.1.96-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
|
|
||||||
Collecting tensorboard
|
|
||||||
Using cached tensorboard-2.6.0-py3-none-any.whl (5.6 MB)
|
|
||||||
Requirement already satisfied: setuptools>=41.0.0 in /ceph-fj/fangjun/test-icefall/lib/python3.8/site-packages (from tensorboard->-r
|
|
||||||
requirements.txt (line 4)) (57.4.0)
|
|
||||||
Collecting absl-py>=0.4
|
|
||||||
Using cached absl_py-0.13.0-py3-none-any.whl (132 kB)
|
|
||||||
Collecting google-auth-oauthlib<0.5,>=0.4.1
|
|
||||||
Using cached google_auth_oauthlib-0.4.5-py2.py3-none-any.whl (18 kB)
|
|
||||||
Collecting grpcio>=1.24.3
|
|
||||||
Using cached grpcio-1.39.0-cp38-cp38-manylinux2014_x86_64.whl (4.3 MB)
|
|
||||||
Requirement already satisfied: wheel>=0.26 in /ceph-fj/fangjun/test-icefall/lib/python3.8/site-packages (from tensorboard->-r require
|
|
||||||
ments.txt (line 4)) (0.36.2)
|
|
||||||
Requirement already satisfied: numpy>=1.12.0 in /ceph-fj/fangjun/test-icefall/lib/python3.8/site-packages (from tensorboard->-r requi
|
|
||||||
rements.txt (line 4)) (1.21.2)
|
|
||||||
Collecting protobuf>=3.6.0
|
|
||||||
Using cached protobuf-3.17.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
|
|
||||||
Collecting werkzeug>=0.11.15
|
|
||||||
Using cached Werkzeug-2.0.1-py3-none-any.whl (288 kB)
|
|
||||||
Collecting tensorboard-data-server<0.7.0,>=0.6.0
|
|
||||||
Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
|
|
||||||
Collecting google-auth<2,>=1.6.3
|
|
||||||
Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
|
|
||||||
|________________________________| 152 kB 1.4 MB/s
|
|
||||||
Collecting requests<3,>=2.21.0
|
|
||||||
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
|
|
||||||
Collecting tensorboard-plugin-wit>=1.6.0
|
|
||||||
Using cached tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
|
|
||||||
Collecting markdown>=2.6.8
|
|
||||||
Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
|
|
||||||
Collecting six
|
|
||||||
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
|
|
||||||
Collecting cachetools<5.0,>=2.0.0
|
|
||||||
Using cached cachetools-4.2.2-py3-none-any.whl (11 kB)
|
|
||||||
Collecting rsa<5,>=3.1.4
|
|
||||||
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
|
|
||||||
Collecting pyasn1-modules>=0.2.1
|
|
||||||
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
|
|
||||||
Collecting requests-oauthlib>=0.7.0
|
|
||||||
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
|
|
||||||
Collecting pyasn1<0.5.0,>=0.4.6
|
|
||||||
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
|
|
||||||
Collecting urllib3<1.27,>=1.21.1
|
|
||||||
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
|
|
||||||
Collecting certifi>=2017.4.17
|
|
||||||
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
|
|
||||||
Collecting charset-normalizer~=2.0.0
|
|
||||||
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
|
|
||||||
Collecting idna<4,>=2.5
|
|
||||||
Using cached idna-3.2-py3-none-any.whl (59 kB)
|
|
||||||
Collecting oauthlib>=3.0.0
|
|
||||||
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
|
|
||||||
Building wheels for collected packages: kaldilm
|
|
||||||
Building wheel for kaldilm (setup.py) ... done
|
|
||||||
Created wheel for kaldilm: filename=kaldilm-1.8-cp38-cp38-linux_x86_64.whl size=897233 sha256=eccb906cafcd45bf9a7e1a1718e4534254bfb
|
|
||||||
f4c0d0cbc66eee6c88d68a63862
|
|
||||||
Stored in directory: /root/fangjun/.cache/pip/wheels/85/7d/63/f2dd586369b8797cb36d213bf3a84a789eeb92db93d2e723c9
|
|
||||||
Successfully built kaldilm
|
|
||||||
Installing collected packages: urllib3, pyasn1, idna, charset-normalizer, certifi, six, rsa, requests, pyasn1-modules, oauthlib, cach
|
|
||||||
etools, requests-oauthlib, google-auth, werkzeug, tensorboard-plugin-wit, tensorboard-data-server, protobuf, markdown, grpcio, google
|
|
||||||
-auth-oauthlib, absl-py, tensorboard, sentencepiece, kaldilm, kaldialign
|
|
||||||
Successfully installed absl-py-0.13.0 cachetools-4.2.2 certifi-2021.5.30 charset-normalizer-2.0.4 google-auth-1.35.0 google-auth-oaut
|
|
||||||
hlib-0.4.5 grpcio-1.39.0 idna-3.2 kaldialign-0.2 kaldilm-1.8 markdown-3.3.4 oauthlib-3.1.1 protobuf-3.17.3 pyasn1-0.4.8 pyasn1-module
|
|
||||||
s-0.2.8 requests-2.26.0 requests-oauthlib-1.3.0 rsa-4.7.2 sentencepiece-0.1.96 six-1.16.0 tensorboard-2.6.0 tensorboard-data-server-0
|
|
||||||
.6.1 tensorboard-plugin-wit-1.8.0 urllib3-1.26.6 werkzeug-2.0.1
|
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:icefall$ pip install -r ./requirements.txt
|
||||||
|
|
||||||
Test Your Installation
|
Test Your Installation
|
||||||
----------------------
|
----------------------
|
||||||
|
|
||||||
To test that your installation is successful, let us run
|
To test that your installation is successful, let us run
|
||||||
the `yesno recipe <https://github.com/k2-fsa/icefall/tree/master/egs/yesno/ASR>`_
|
the `yesno recipe <https://github.com/k2-fsa/icefall/tree/master/egs/yesno/ASR>`_
|
||||||
on CPU.
|
on ``CPU``.
|
||||||
|
|
||||||
Data preparation
|
Data preparation
|
||||||
~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
$ export PYTHONPATH=/tmp/icefall:$PYTHONPATH
|
(test-icefall) kuangfangjun:icefall$ export PYTHONPATH=/tmp/icefall:$PYTHONPATH
|
||||||
$ cd /tmp/icefall
|
|
||||||
$ cd egs/yesno/ASR
|
(test-icefall) kuangfangjun:icefall$ cd /tmp/icefall
|
||||||
$ ./prepare.sh
|
|
||||||
|
(test-icefall) kuangfangjun:icefall$ cd egs/yesno/ASR
|
||||||
|
|
||||||
|
(test-icefall) kuangfangjun:ASR$ ./prepare.sh
|
||||||
|
|
||||||
|
|
||||||
The log of running ``./prepare.sh`` is:
|
The log of running ``./prepare.sh`` is:
|
||||||
|
|
||||||
.. code-block::
|
.. code-block::
|
||||||
|
|
||||||
2023-05-12 17:55:21 (prepare.sh:27:main) dl_dir: /tmp/icefall/egs/yesno/ASR/download
|
2023-07-27 12:41:39 (prepare.sh:27:main) dl_dir: /tmp/icefall/egs/yesno/ASR/download
|
||||||
2023-05-12 17:55:21 (prepare.sh:30:main) Stage 0: Download data
|
2023-07-27 12:41:39 (prepare.sh:30:main) Stage 0: Download data
|
||||||
/tmp/icefall/egs/yesno/ASR/download/waves_yesno.tar.gz: 100%|_______________________________________________________________| 4.70M/4.70M [06:54<00:00, 11.4kB/s]
|
/tmp/icefall/egs/yesno/ASR/download/waves_yesno.tar.gz: 100%|___________________________________________________| 4.70M/4.70M [00:00<00:00, 11.1MB/s]
|
||||||
2023-05-12 18:02:19 (prepare.sh:39:main) Stage 1: Prepare yesno manifest
|
2023-07-27 12:41:46 (prepare.sh:39:main) Stage 1: Prepare yesno manifest
|
||||||
2023-05-12 18:02:21 (prepare.sh:45:main) Stage 2: Compute fbank for yesno
|
2023-07-27 12:41:50 (prepare.sh:45:main) Stage 2: Compute fbank for yesno
|
||||||
2023-05-12 18:02:23,199 INFO [compute_fbank_yesno.py:65] Processing train
|
2023-07-27 12:41:55,718 INFO [compute_fbank_yesno.py:65] Processing train
|
||||||
Extracting and storing features: 100%|_______________________________________________________________| 90/90 [00:00<00:00, 212.60it/s]
|
Extracting and storing features: 100%|_______________________________________________________________________________| 90/90 [00:01<00:00, 87.82it/s]
|
||||||
2023-05-12 18:02:23,640 INFO [compute_fbank_yesno.py:65] Processing test
|
2023-07-27 12:41:56,778 INFO [compute_fbank_yesno.py:65] Processing test
|
||||||
Extracting and storing features: 100%|_______________________________________________________________| 30/30 [00:00<00:00, 304.53it/s]
|
Extracting and storing features: 100%|______________________________________________________________________________| 30/30 [00:00<00:00, 256.92it/s]
|
||||||
2023-05-12 18:02:24 (prepare.sh:51:main) Stage 3: Prepare lang
|
2023-07-27 12:41:57 (prepare.sh:51:main) Stage 3: Prepare lang
|
||||||
2023-05-12 18:02:26 (prepare.sh:66:main) Stage 4: Prepare G
|
2023-07-27 12:42:02 (prepare.sh:66:main) Stage 4: Prepare G
|
||||||
/project/kaldilm/csrc/arpa_file_parser.cc:void kaldilm::ArpaFileParser::Read(std::istream&):79
|
/project/kaldilm/csrc/arpa_file_parser.cc:void kaldilm::ArpaFileParser::Read(std::istream&):79
|
||||||
[I] Reading \data\ section.
|
[I] Reading \data\ section.
|
||||||
/project/kaldilm/csrc/arpa_file_parser.cc:void kaldilm::ArpaFileParser::Read(std::istream&):140
|
/project/kaldilm/csrc/arpa_file_parser.cc:void kaldilm::ArpaFileParser::Read(std::istream&):140
|
||||||
[I] Reading \1-grams: section.
|
[I] Reading \1-grams: section.
|
||||||
2023-05-12 18:02:26 (prepare.sh:92:main) Stage 5: Compile HLG
|
2023-07-27 12:42:02 (prepare.sh:92:main) Stage 5: Compile HLG
|
||||||
2023-05-12 18:02:28,581 INFO [compile_hlg.py:124] Processing data/lang_phone
|
2023-07-27 12:42:07,275 INFO [compile_hlg.py:124] Processing data/lang_phone
|
||||||
2023-05-12 18:02:28,582 INFO [lexicon.py:171] Converting L.pt to Linv.pt
|
2023-07-27 12:42:07,276 INFO [lexicon.py:171] Converting L.pt to Linv.pt
|
||||||
2023-05-12 18:02:28,609 INFO [compile_hlg.py:48] Building ctc_topo. max_token_id: 3
|
2023-07-27 12:42:07,309 INFO [compile_hlg.py:48] Building ctc_topo. max_token_id: 3
|
||||||
2023-05-12 18:02:28,610 INFO [compile_hlg.py:52] Loading G.fst.txt
|
2023-07-27 12:42:07,310 INFO [compile_hlg.py:52] Loading G.fst.txt
|
||||||
2023-05-12 18:02:28,611 INFO [compile_hlg.py:62] Intersecting L and G
|
2023-07-27 12:42:07,314 INFO [compile_hlg.py:62] Intersecting L and G
|
||||||
2023-05-12 18:02:28,613 INFO [compile_hlg.py:64] LG shape: (4, None)
|
2023-07-27 12:42:07,323 INFO [compile_hlg.py:64] LG shape: (4, None)
|
||||||
2023-05-12 18:02:28,613 INFO [compile_hlg.py:66] Connecting LG
|
2023-07-27 12:42:07,323 INFO [compile_hlg.py:66] Connecting LG
|
||||||
2023-05-12 18:02:28,614 INFO [compile_hlg.py:68] LG shape after k2.connect: (4, None)
|
2023-07-27 12:42:07,323 INFO [compile_hlg.py:68] LG shape after k2.connect: (4, None)
|
||||||
2023-05-12 18:02:28,614 INFO [compile_hlg.py:70] <class 'torch.Tensor'>
|
2023-07-27 12:42:07,323 INFO [compile_hlg.py:70] <class 'torch.Tensor'>
|
||||||
2023-05-12 18:02:28,614 INFO [compile_hlg.py:71] Determinizing LG
|
2023-07-27 12:42:07,323 INFO [compile_hlg.py:71] Determinizing LG
|
||||||
2023-05-12 18:02:28,615 INFO [compile_hlg.py:74] <class '_k2.ragged.RaggedTensor'>
|
2023-07-27 12:42:07,341 INFO [compile_hlg.py:74] <class '_k2.ragged.RaggedTensor'>
|
||||||
2023-05-12 18:02:28,615 INFO [compile_hlg.py:76] Connecting LG after k2.determinize
|
2023-07-27 12:42:07,341 INFO [compile_hlg.py:76] Connecting LG after k2.determinize
|
||||||
2023-05-12 18:02:28,615 INFO [compile_hlg.py:79] Removing disambiguation symbols on LG
|
2023-07-27 12:42:07,341 INFO [compile_hlg.py:79] Removing disambiguation symbols on LG
|
||||||
2023-05-12 18:02:28,616 INFO [compile_hlg.py:91] LG shape after k2.remove_epsilon: (6, None)
|
2023-07-27 12:42:07,354 INFO [compile_hlg.py:91] LG shape after k2.remove_epsilon: (6, None)
|
||||||
2023-05-12 18:02:28,617 INFO [compile_hlg.py:96] Arc sorting LG
|
2023-07-27 12:42:07,445 INFO [compile_hlg.py:96] Arc sorting LG
|
||||||
2023-05-12 18:02:28,617 INFO [compile_hlg.py:99] Composing H and LG
|
2023-07-27 12:42:07,445 INFO [compile_hlg.py:99] Composing H and LG
|
||||||
2023-05-12 18:02:28,619 INFO [compile_hlg.py:106] Connecting LG
|
2023-07-27 12:42:07,446 INFO [compile_hlg.py:106] Connecting LG
|
||||||
2023-05-12 18:02:28,619 INFO [compile_hlg.py:109] Arc sorting LG
|
2023-07-27 12:42:07,446 INFO [compile_hlg.py:109] Arc sorting LG
|
||||||
2023-05-12 18:02:28,619 INFO [compile_hlg.py:111] HLG.shape: (8, None)
|
2023-07-27 12:42:07,447 INFO [compile_hlg.py:111] HLG.shape: (8, None)
|
||||||
2023-05-12 18:02:28,619 INFO [compile_hlg.py:127] Saving HLG.pt to data/lang_phone
|
2023-07-27 12:42:07,447 INFO [compile_hlg.py:127] Saving HLG.pt to data/lang_phone
|
||||||
|
|
||||||
|
|
||||||
Training
|
Training
|
||||||
~~~~~~~~
|
~~~~~~~~
|
||||||
@ -409,12 +430,13 @@ Now let us run the training part:
|
|||||||
|
|
||||||
.. code-block::
|
.. code-block::
|
||||||
|
|
||||||
$ export CUDA_VISIBLE_DEVICES=""
|
(test-icefall) kuangfangjun:ASR$ export CUDA_VISIBLE_DEVICES=""
|
||||||
$ ./tdnn/train.py
|
|
||||||
|
(test-icefall) kuangfangjun:ASR$ ./tdnn/train.py
|
||||||
|
|
||||||
.. CAUTION::
|
.. CAUTION::
|
||||||
|
|
||||||
We use ``export CUDA_VISIBLE_DEVICES=""`` so that ``icefall`` uses CPU
|
We use ``export CUDA_VISIBLE_DEVICES=""`` so that `icefall`_ uses CPU
|
||||||
even if there are GPUs available.
|
even if there are GPUs available.
|
||||||
|
|
||||||
.. hint::
|
.. hint::
|
||||||
@ -432,53 +454,52 @@ The training log is given below:
|
|||||||
|
|
||||||
.. code-block::
|
.. code-block::
|
||||||
|
|
||||||
2023-05-12 18:04:59,759 INFO [train.py:481] Training started
|
2023-07-27 12:50:51,936 INFO [train.py:481] Training started
|
||||||
2023-05-12 18:04:59,759 INFO [train.py:482] {'exp_dir': PosixPath('tdnn/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lr': 0.01, 'feature_dim': 23, 'weight_decay': 1e-06, 'start_epoch': 0,
|
2023-07-27 12:50:51,936 INFO [train.py:482] {'exp_dir': PosixPath('tdnn/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lr': 0.01, 'feature_dim': 23, 'weight_decay': 1e-06, 'start_epoch': 0, 'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 10, 'reset_interval': 20, 'valid_interval': 10, 'beam_size': 10, 'reduction': 'sum', 'use_double_scores': True, 'world_size': 1, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 15, 'seed': 42, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 30.0, 'bucketing_sampler': False, 'num_buckets': 10, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': False, 'return_cuts': True, 'num_workers': 2, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '4c05309499a08454997adf500b56dcc629e35ae5', 'k2-git-date': 'Tue Jul 25 16:23:36 2023', 'lhotse-version': '1.16.0.dev+git.7640d66.clean', 'torch-version': '1.13.0+cu116', 'torch-cuda-available': False, 'torch-cuda-version': '11.6', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '3fb0a43-clean', 'icefall-git-date': 'Thu Jul 27 12:36:05 2023', 'icefall-path': '/tmp/icefall', 'k2-path': '/star-fj/fangjun/test-icefall/lib/python3.8/site-packages/k2/__init__.py', 'lhotse-path': '/star-fj/fangjun/test-icefall/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'de-74279-k2-train-1-1220091118-57c4d55446-sph26', 'IP address': '10.177.77.20'}}
|
||||||
'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 10, 'reset_interval': 20, 'valid_interval': 10, 'beam_size': 10,
|
2023-07-27 12:50:51,941 INFO [lexicon.py:168] Loading pre-compiled data/lang_phone/Linv.pt
|
||||||
'reduction': 'sum', 'use_double_scores': True, 'world_size': 1, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 15, 'seed': 42, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 30.0,
|
2023-07-27 12:50:51,949 INFO [train.py:495] device: cpu
|
||||||
'bucketing_sampler': False, 'num_buckets': 10, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': False, 'return_cuts': True, 'num_workers': 2,
|
2023-07-27 12:50:51,965 INFO [asr_datamodule.py:146] About to get train cuts
|
||||||
'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '3b7f09fa35e72589914f67089c0da9f196a92ca4', 'k2-git-date': 'Mon May 8 22:58:45 2023',
|
2023-07-27 12:50:51,965 INFO [asr_datamodule.py:244] About to get train cuts
|
||||||
'lhotse-version': '1.15.0.dev+git.6fcfced.clean', 'torch-version': '2.0.0+cu118', 'torch-cuda-available': False, 'torch-cuda-version': '11.8', 'python-version': '3.1', 'icefall-git-branch': 'master',
|
2023-07-27 12:50:51,967 INFO [asr_datamodule.py:149] About to create train dataset
|
||||||
'icefall-git-sha1': '30bde4b-clean', 'icefall-git-date': 'Thu May 11 17:37:47 2023', 'icefall-path': '/tmp/icefall',
|
2023-07-27 12:50:51,967 INFO [asr_datamodule.py:199] Using SingleCutSampler.
|
||||||
'k2-path': 'tmp/lib/python3.10/site-packages/k2-1.24.3.dev20230512+cuda11.8.torch2.0.0-py3.10-linux-x86_64.egg/k2/__init__.py',
|
2023-07-27 12:50:51,967 INFO [asr_datamodule.py:205] About to create train dataloader
|
||||||
'lhotse-path': 'tmp/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'host', 'IP address': '0.0.0.0'}}
|
2023-07-27 12:50:51,968 INFO [asr_datamodule.py:218] About to get test cuts
|
||||||
2023-05-12 18:04:59,761 INFO [lexicon.py:168] Loading pre-compiled data/lang_phone/Linv.pt
|
2023-07-27 12:50:51,968 INFO [asr_datamodule.py:252] About to get test cuts
|
||||||
2023-05-12 18:04:59,764 INFO [train.py:495] device: cpu
|
2023-07-27 12:50:52,565 INFO [train.py:422] Epoch 0, batch 0, loss[loss=1.065, over 2436.00 frames. ], tot_loss[loss=1.065, over 2436.00 frames. ], batch size: 4
|
||||||
2023-05-12 18:04:59,791 INFO [asr_datamodule.py:146] About to get train cuts
|
2023-07-27 12:50:53,681 INFO [train.py:422] Epoch 0, batch 10, loss[loss=0.4561, over 2828.00 frames. ], tot_loss[loss=0.7076, over 22192.90 frames.], batch size: 4
|
||||||
2023-05-12 18:04:59,791 INFO [asr_datamodule.py:244] About to get train cuts
|
2023-07-27 12:50:54,167 INFO [train.py:444] Epoch 0, validation loss=0.9002, over 18067.00 frames.
|
||||||
2023-05-12 18:04:59,852 INFO [asr_datamodule.py:149] About to create train dataset
|
2023-07-27 12:50:55,011 INFO [train.py:422] Epoch 0, batch 20, loss[loss=0.2555, over 2695.00 frames. ], tot_loss[loss=0.484, over 34971.47 frames. ], batch size: 5
|
||||||
2023-05-12 18:04:59,852 INFO [asr_datamodule.py:199] Using SingleCutSampler.
|
2023-07-27 12:50:55,331 INFO [train.py:444] Epoch 0, validation loss=0.4688, over 18067.00 frames.
|
||||||
2023-05-12 18:04:59,852 INFO [asr_datamodule.py:205] About to create train dataloader
|
2023-07-27 12:50:55,368 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-0.pt
|
||||||
2023-05-12 18:04:59,853 INFO [asr_datamodule.py:218] About to get test cuts
|
2023-07-27 12:50:55,633 INFO [train.py:422] Epoch 1, batch 0, loss[loss=0.2532, over 2436.00 frames. ], tot_loss[loss=0.2532, over 2436.00 frames. ],
|
||||||
2023-05-12 18:04:59,853 INFO [asr_datamodule.py:252] About to get test cuts
|
batch size: 4
|
||||||
2023-05-12 18:04:59,986 INFO [train.py:422] Epoch 0, batch 0, loss[loss=1.065, over 2436.00 frames. ], tot_loss[loss=1.065, over 2436.00 frames. ], batch size: 4
|
2023-07-27 12:50:56,242 INFO [train.py:422] Epoch 1, batch 10, loss[loss=0.1139, over 2828.00 frames. ], tot_loss[loss=0.1592, over 22192.90 frames.], batch size: 4
|
||||||
2023-05-12 18:05:00,352 INFO [train.py:422] Epoch 0, batch 10, loss[loss=0.4561, over 2828.00 frames. ], tot_loss[loss=0.7076, over 22192.90 frames. ], batch size: 4
|
2023-07-27 12:50:56,522 INFO [train.py:444] Epoch 1, validation loss=0.1627, over 18067.00 frames.
|
||||||
2023-05-12 18:05:00,691 INFO [train.py:444] Epoch 0, validation loss=0.9002, over 18067.00 frames.
|
2023-07-27 12:50:57,209 INFO [train.py:422] Epoch 1, batch 20, loss[loss=0.07055, over 2695.00 frames. ], tot_loss[loss=0.1175, over 34971.47 frames.], batch size: 5
|
||||||
2023-05-12 18:05:00,996 INFO [train.py:422] Epoch 0, batch 20, loss[loss=0.2555, over 2695.00 frames. ], tot_loss[loss=0.484, over 34971.47 frames. ], batch size: 5
|
2023-07-27 12:50:57,600 INFO [train.py:444] Epoch 1, validation loss=0.07091, over 18067.00 frames.
|
||||||
2023-05-12 18:05:01,217 INFO [train.py:444] Epoch 0, validation loss=0.4688, over 18067.00 frames.
|
2023-07-27 12:50:57,640 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-1.pt
|
||||||
2023-05-12 18:05:01,251 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-0.pt
|
2023-07-27 12:50:57,847 INFO [train.py:422] Epoch 2, batch 0, loss[loss=0.07731, over 2436.00 frames. ], tot_loss[loss=0.07731, over 2436.00 frames.], batch size: 4
|
||||||
2023-05-12 18:05:01,389 INFO [train.py:422] Epoch 1, batch 0, loss[loss=0.2532, over 2436.00 frames. ], tot_loss[loss=0.2532, over 2436.00 frames. ], batch size: 4
|
2023-07-27 12:50:58,427 INFO [train.py:422] Epoch 2, batch 10, loss[loss=0.04391, over 2828.00 frames. ], tot_loss[loss=0.05341, over 22192.90 frames. ], batch size: 4
|
||||||
2023-05-12 18:05:01,637 INFO [train.py:422] Epoch 1, batch 10, loss[loss=0.1139, over 2828.00 frames. ], tot_loss[loss=0.1592, over 22192.90 frames. ], batch size: 4
|
2023-07-27 12:50:58,884 INFO [train.py:444] Epoch 2, validation loss=0.04384, over 18067.00 frames.
|
||||||
2023-05-12 18:05:01,859 INFO [train.py:444] Epoch 1, validation loss=0.1629, over 18067.00 frames.
|
2023-07-27 12:50:59,387 INFO [train.py:422] Epoch 2, batch 20, loss[loss=0.03458, over 2695.00 frames. ], tot_loss[loss=0.04616, over 34971.47 frames. ], batch size: 5
|
||||||
2023-05-12 18:05:02,094 INFO [train.py:422] Epoch 1, batch 20, loss[loss=0.0767, over 2695.00 frames. ], tot_loss[loss=0.118, over 34971.47 frames. ], batch size: 5
|
2023-07-27 12:50:59,707 INFO [train.py:444] Epoch 2, validation loss=0.03379, over 18067.00 frames.
|
||||||
2023-05-12 18:05:02,350 INFO [train.py:444] Epoch 1, validation loss=0.06778, over 18067.00 frames.
|
2023-07-27 12:50:59,758 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-2.pt
|
||||||
2023-05-12 18:05:02,395 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-1.pt
|
|
||||||
|
|
||||||
... ...
|
... ...
|
||||||
|
|
||||||
2023-05-12 18:05:14,789 INFO [train.py:422] Epoch 13, batch 0, loss[loss=0.01056, over 2436.00 frames. ], tot_loss[loss=0.01056, over 2436.00 frames. ], batch size: 4
|
2023-07-27 12:51:23,433 INFO [train.py:422] Epoch 13, batch 0, loss[loss=0.01054, over 2436.00 frames. ], tot_loss[loss=0.01054, over 2436.00 frames. ], batch size: 4
|
||||||
2023-05-12 18:05:15,016 INFO [train.py:422] Epoch 13, batch 10, loss[loss=0.009022, over 2828.00 frames. ], tot_loss[loss=0.009985, over 22192.90 frames. ], batch size: 4
|
2023-07-27 12:51:23,980 INFO [train.py:422] Epoch 13, batch 10, loss[loss=0.009014, over 2828.00 frames. ], tot_loss[loss=0.009974, over 22192.90 frames. ], batch size: 4
|
||||||
2023-05-12 18:05:15,271 INFO [train.py:444] Epoch 13, validation loss=0.01088, over 18067.00 frames.
|
2023-07-27 12:51:24,489 INFO [train.py:444] Epoch 13, validation loss=0.01085, over 18067.00 frames.
|
||||||
2023-05-12 18:05:15,497 INFO [train.py:422] Epoch 13, batch 20, loss[loss=0.01174, over 2695.00 frames. ], tot_loss[loss=0.01077, over 34971.47 frames. ], batch size: 5
|
2023-07-27 12:51:25,258 INFO [train.py:422] Epoch 13, batch 20, loss[loss=0.01172, over 2695.00 frames. ], tot_loss[loss=0.01055, over 34971.47 frames. ], batch size: 5
|
||||||
2023-05-12 18:05:15,747 INFO [train.py:444] Epoch 13, validation loss=0.01087, over 18067.00 frames.
|
2023-07-27 12:51:25,621 INFO [train.py:444] Epoch 13, validation loss=0.01074, over 18067.00 frames.
|
||||||
2023-05-12 18:05:15,783 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-13.pt
|
2023-07-27 12:51:25,699 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-13.pt
|
||||||
2023-05-12 18:05:15,921 INFO [train.py:422] Epoch 14, batch 0, loss[loss=0.01045, over 2436.00 frames. ], tot_loss[loss=0.01045, over 2436.00 frames. ], batch size: 4
|
2023-07-27 12:51:25,866 INFO [train.py:422] Epoch 14, batch 0, loss[loss=0.01044, over 2436.00 frames. ], tot_loss[loss=0.01044, over 2436.00 frames. ], batch size: 4
|
||||||
2023-05-12 18:05:16,146 INFO [train.py:422] Epoch 14, batch 10, loss[loss=0.008957, over 2828.00 frames. ], tot_loss[loss=0.009903, over 22192.90 frames. ], batch size: 4
|
2023-07-27 12:51:26,844 INFO [train.py:422] Epoch 14, batch 10, loss[loss=0.008942, over 2828.00 frames. ], tot_loss[loss=0.01, over 22192.90 frames. ], batch size: 4
|
||||||
2023-05-12 18:05:16,374 INFO [train.py:444] Epoch 14, validation loss=0.01092, over 18067.00 frames.
|
2023-07-27 12:51:27,221 INFO [train.py:444] Epoch 14, validation loss=0.01082, over 18067.00 frames.
|
||||||
2023-05-12 18:05:16,598 INFO [train.py:422] Epoch 14, batch 20, loss[loss=0.01169, over 2695.00 frames. ], tot_loss[loss=0.01065, over 34971.47 frames. ], batch size: 5
|
2023-07-27 12:51:27,970 INFO [train.py:422] Epoch 14, batch 20, loss[loss=0.01169, over 2695.00 frames. ], tot_loss[loss=0.01054, over 34971.47 frames. ], batch size: 5
|
||||||
2023-05-12 18:05:16,824 INFO [train.py:444] Epoch 14, validation loss=0.01077, over 18067.00 frames.
|
2023-07-27 12:51:28,247 INFO [train.py:444] Epoch 14, validation loss=0.01073, over 18067.00 frames.
|
||||||
2023-05-12 18:05:16,862 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-14.pt
|
2023-07-27 12:51:28,323 INFO [checkpoint.py:75] Saving checkpoint to tdnn/exp/epoch-14.pt
|
||||||
2023-05-12 18:05:16,865 INFO [train.py:555] Done!
|
2023-07-27 12:51:28,326 INFO [train.py:555] Done!
|
||||||
|
|
||||||
Decoding
|
Decoding
|
||||||
~~~~~~~~
|
~~~~~~~~
|
||||||
@ -487,42 +508,32 @@ Let us use the trained model to decode the test set:
|
|||||||
|
|
||||||
.. code-block::
|
.. code-block::
|
||||||
|
|
||||||
$ ./tdnn/decode.py
|
(test-icefall) kuangfangjun:ASR$ ./tdnn/decode.py
|
||||||
|
|
||||||
The decoding log is:
|
2023-07-27 12:55:12,840 INFO [decode.py:263] Decoding started
|
||||||
|
2023-07-27 12:55:12,840 INFO [decode.py:264] {'exp_dir': PosixPath('tdnn/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 23, 'search_beam': 20, 'output_beam': 8, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'epoch': 14, 'avg': 2, 'export': False, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 30.0, 'bucketing_sampler': False, 'num_buckets': 10, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': False, 'return_cuts': True, 'num_workers': 2, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '4c05309499a08454997adf500b56dcc629e35ae5', 'k2-git-date': 'Tue Jul 25 16:23:36 2023', 'lhotse-version': '1.16.0.dev+git.7640d66.clean', 'torch-version': '1.13.0+cu116', 'torch-cuda-available': False, 'torch-cuda-version': '11.6', 'python-version': '3.8', 'icefall-git-branch': 'master', 'icefall-git-sha1': '3fb0a43-clean', 'icefall-git-date': 'Thu Jul 27 12:36:05 2023', 'icefall-path': '/tmp/icefall', 'k2-path': '/star-fj/fangjun/test-icefall/lib/python3.8/site-packages/k2/__init__.py', 'lhotse-path': '/star-fj/fangjun/test-icefall/lib/python3.8/site-packages/lhotse/__init__.py', 'hostname': 'de-74279-k2-train-1-1220091118-57c4d55446-sph26', 'IP address': '10.177.77.20'}}
|
||||||
|
2023-07-27 12:55:12,841 INFO [lexicon.py:168] Loading pre-compiled data/lang_phone/Linv.pt
|
||||||
|
2023-07-27 12:55:12,855 INFO [decode.py:273] device: cpu
|
||||||
|
2023-07-27 12:55:12,868 INFO [decode.py:291] averaging ['tdnn/exp/epoch-13.pt', 'tdnn/exp/epoch-14.pt']
|
||||||
|
2023-07-27 12:55:12,882 INFO [asr_datamodule.py:218] About to get test cuts
|
||||||
|
2023-07-27 12:55:12,883 INFO [asr_datamodule.py:252] About to get test cuts
|
||||||
|
2023-07-27 12:55:13,157 INFO [decode.py:204] batch 0/?, cuts processed until now is 4
|
||||||
|
2023-07-27 12:55:13,701 INFO [decode.py:241] The transcripts are stored in tdnn/exp/recogs-test_set.txt
|
||||||
|
2023-07-27 12:55:13,702 INFO [utils.py:564] [test_set] %WER 0.42% [1 / 240, 0 ins, 1 del, 0 sub ]
|
||||||
|
2023-07-27 12:55:13,704 INFO [decode.py:249] Wrote detailed error stats to tdnn/exp/errs-test_set.txt
|
||||||
|
2023-07-27 12:55:13,704 INFO [decode.py:316] Done!
|
||||||
|
|
||||||
.. code-block::
|
|
||||||
|
|
||||||
2023-05-12 18:08:30,482 INFO [decode.py:263] Decoding started
|
**Congratulations!** You have successfully setup the environment and have run the first recipe in `icefall`_.
|
||||||
2023-05-12 18:08:30,483 INFO [decode.py:264] {'exp_dir': PosixPath('tdnn/exp'), 'lang_dir': PosixPath('data/lang_phone'), 'lm_dir': PosixPath('data/lm'), 'feature_dim': 23,
|
|
||||||
'search_beam': 20, 'output_beam': 8, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'epoch': 14, 'avg': 2, 'export': False, 'feature_dir': PosixPath('data/fbank'),
|
|
||||||
'max_duration': 30.0, 'bucketing_sampler': False, 'num_buckets': 10, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': False, 'return_cuts': True,
|
|
||||||
'num_workers': 2, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '3b7f09fa35e72589914f67089c0da9f196a92ca4', 'k2-git-date': 'Mon May 8 22:58:45 2023',
|
|
||||||
'lhotse-version': '1.15.0.dev+git.6fcfced.clean', 'torch-version': '2.0.0+cu118', 'torch-cuda-available': False, 'torch-cuda-version': '11.8', 'python-version': '3.1', 'icefall-git-branch': 'master',
|
|
||||||
'icefall-git-sha1': '30bde4b-clean', 'icefall-git-date': 'Thu May 11 17:37:47 2023', 'icefall-path': '/tmp/icefall',
|
|
||||||
'k2-path': '/tmp/lib/python3.10/site-packages/k2-1.24.3.dev20230512+cuda11.8.torch2.0.0-py3.10-linux-x86_64.egg/k2/__init__.py',
|
|
||||||
'lhotse-path': '/tmp/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'host', 'IP address': '0.0.0.0'}}
|
|
||||||
2023-05-12 18:08:30,483 INFO [lexicon.py:168] Loading pre-compiled data/lang_phone/Linv.pt
|
|
||||||
2023-05-12 18:08:30,487 INFO [decode.py:273] device: cpu
|
|
||||||
2023-05-12 18:08:30,513 INFO [decode.py:291] averaging ['tdnn/exp/epoch-13.pt', 'tdnn/exp/epoch-14.pt']
|
|
||||||
2023-05-12 18:08:30,521 INFO [asr_datamodule.py:218] About to get test cuts
|
|
||||||
2023-05-12 18:08:30,521 INFO [asr_datamodule.py:252] About to get test cuts
|
|
||||||
2023-05-12 18:08:30,675 INFO [decode.py:204] batch 0/?, cuts processed until now is 4
|
|
||||||
2023-05-12 18:08:30,923 INFO [decode.py:241] The transcripts are stored in tdnn/exp/recogs-test_set.txt
|
|
||||||
2023-05-12 18:08:30,924 INFO [utils.py:558] [test_set] %WER 0.42% [1 / 240, 0 ins, 1 del, 0 sub ]
|
|
||||||
2023-05-12 18:08:30,925 INFO [decode.py:249] Wrote detailed error stats to tdnn/exp/errs-test_set.txt
|
|
||||||
2023-05-12 18:08:30,925 INFO [decode.py:316] Done!
|
|
||||||
|
|
||||||
**Congratulations!** You have successfully setup the environment and have run the first recipe in ``icefall``.
|
|
||||||
|
|
||||||
Have fun with ``icefall``!
|
Have fun with ``icefall``!
|
||||||
|
|
||||||
YouTube Video
|
YouTube Video
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
We provide the following YouTube video showing how to install ``icefall``.
|
We provide the following YouTube video showing how to install `icefall`_.
|
||||||
It also shows how to debug various problems that you may encounter while
|
It also shows how to debug various problems that you may encounter while
|
||||||
using ``icefall``.
|
using `icefall`_.
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user