Language modelling document

01/31/1 5 Automatic Speech Recognition 1 Quan. V, Ha. N Language Modelling 01/31/1 5 Automatic Speech Recognition 2 Quan. V, Ha. N Language Modelling • NPath Lists and Lattices • NGram Language Models • Word NetWord Expansion 01/31/1 5 Automatic Speech Recognition 3 Quan. V, Ha. N NPath Lists <S> W2 W1 W5 W4 W3 W7 W6 </S> 1 Best Path NPath n 01/31/1 5 Automatic Speech Recognition 4 Quan. V, Ha. N NPath HVite -n 4 -z lat -l “/lattice" -C config_hvite -H hmm30/macros -H hmm30/hmmdefs -S dtnvn1106a.scp -i rec_out_lattice.mlf -w wdnet_bigram -p 0.0 -s 5.0 dict.txt tiedlist 01/31/1 5 Automatic Speech Recognition 5 Quan. V, Ha. N Lattices VERSION=1.0 UTTERANCE=Dtnvn2307/DTNVN1106A_10_S1388583.mfc lmname=wdnet_bigram lmscale=5.00 wdpenalty=0.00 acscale=1.00 vocab=dict.txt N=8307 L=32095 I=0 t=0.00 W=!NULL … I=8306 t=3.19 W=!EXIT v=1 J=0 S=0 E=1 a=-159.04 l=0.000 J=1 S=0 E=2 a=-239.20 l=0.000 … J=32094 S=8304 E=8306 a=-199.93 l=-1.860 01/31/1 5 Automatic Speech Recognition 6 Quan. V, Ha. N Ngram Language Models • Database preparation • Mapping OOV words • Language Model Generation • Testing the LM perplexity • Generating and using count-besed models (dynamically adjusted) • Model interpolation (LMerge) • Class-bases models 01/31/1 5 Automatic Speech Recognition 7 Quan. V, Ha. N Ngram Language Models Gramfiles … HE SEEMED TO BE : 5 HE SEEMED TO TAKE : 1 HE SEEMS A VERY : 1 HE SEEMS TO BE : 2 … 01/31/1 5 Automatic Speech Recognition 8 Quan. V, Ha. N Database preparation Step 1: (new) LNewMap -f WFC Holmes empty.wmap Step 2: (count) LGPrep -T 1 -a 100000 -b 200000 -d holmes.0 -n 4 -s "Sherlock Holmes" empty.wmap -S listText.txt Text1.txt <s> QUOTE HOLMES QUOTE SAID I …</s> <s> IT SEEMS RATHER SAD THAT …</s> A text corpus of 10M word units is free for all researchers 01/31/1 5 Automatic Speech Recognition 9 Quan. V, Ha. N Database preparation holmes.0 gram.0 gram.1 gram.2 wmap Step 3: (sort+sequence) LGCopy -T 1 -b 200000 -d holmes.1 holmes.0/wmap holmes.0/gram.* holmes.1 data.0 data.1 data.2 01/31/1 5 Automatic Speech Recognition 10 Quan. V, Ha. N Mapping OOV words Step 4: LGCopy -T 1 -o -m lm_5k/5k.wmap -b 200000 -d lm_5k -w 5k.wlist holmes.0/wmap -S listdata.txt 5k.wlist:5000 most common words lm_5k/data.0 … <s> IT IS !!UNK : 17 <s> IT LOOKS !!UNK : 2 <s> IT MUST !!UNK : 1 <s> IT SEEMED !!UNK : 1 … [...].. .Language Model Generation Calculate Frequency of Frequency table Step 5: LFoF -T 1 -n 4 -f 32 lm_5k/5k.wmap lm_5k/5k.fof -S listdata.txt lm_5k\data.0 Step 6: LBuild -T 1 -c 2 1 -c 3 1 -n 3 lm_5k/5k.wmap . Recognition 1 Quan. V, Ha. N Language Modelling 01/31/1 5 Automatic Speech Recognition 2 Quan. V, Ha. N Language Modelling • NPath Lists and Lattices • NGram Language Models • Word NetWord. 01/31/1 5 Automatic Speech Recognition 6 Quan. V, Ha. N Ngram Language Models • Database preparation • Mapping OOV words • Language Model Generation • Testing the LM perplexity • Generating. interpolation (LMerge) • Class-bases models 01/31/1 5 Automatic Speech Recognition 7 Quan. V, Ha. N Ngram Language Models Gramfiles … HE SEEMED TO BE : 5 HE SEEMED TO TAKE : 1 HE SEEMS A VERY : 1 HE SEEMS