From 80dfdd1cfa4865697b7518c3d31103cd51ca3736 Mon Sep 17 00:00:00 2001 From: Guo Liyong Date: Fri, 27 May 2022 12:38:45 +0800 Subject: [PATCH] a short intro to distillation framework --- egs/librispeech/ASR/distillation_with_hubert.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/egs/librispeech/ASR/distillation_with_hubert.sh b/egs/librispeech/ASR/distillation_with_hubert.sh index e9b73ae2c..e18ba8f55 100644 --- a/egs/librispeech/ASR/distillation_with_hubert.sh +++ b/egs/librispeech/ASR/distillation_with_hubert.sh @@ -1,3 +1,20 @@ +# A short introduction about distillation framework. +# +# A typical traditional distillation method is +# Loss(teacher embedding, student embedding). +# +# Comparing to these, the proposed distillation framework contains two mainly steps: +# codebook indexes = quantizer.encode(teacher embedding) +# Loss(codebook indexes, student embedding) +# +# Things worth to meantion: +# 1. The float type teacher embedding is quantized into a sequence of +# 8-bit integer codebook indexes. +# 2. a middle layer 36(1-based) out of total 48 layers is used to extract +# teacher embeddings. +# 3. a middle layer 6(1-based) out of total 6 layers is used to extract +# student embeddings. + # This is an example to do distillation with librispeech clean-100 subset. # run with command: # bash distillation_with_hubert.sh [0|1|2|3|4]