Speech and Language Processing

Transcription

1 Speech and Language rocessing Lecure 4 Variaional inference and sampling Informaion and Communicaions Engineering Course Takahiro Shinozaki 08//5

2 Lecure lan (Shinozaki s par) I gives he firs 6 lecures abou speech recogniion. Through hese lecures he backbone of he laes speech recogniion echniues is eplained.. 0/9 (remoe) Speech recogniion based on G and N gram. 0/6 (remoe) aimum likelihood esimaion and E algorihm 3. /5 (@TAIST) Bayesian nework and Bayesian inference 4. /5 (@TAIST) Variaional inference and sampling 5. /6 (@TAIST) Neural nework based acousic and language models 6. /6 (@TAIST) Weighed finie sae ransducer (WFST) and speech decoding

3 Today s Topic Answers for he previous eercises Variaional inference Sampling 3

4 Answers for he revious Eercises 4

5 Eercise 3. Is he direced graph a AG? Yes No Graph A Graph B 5

6 Eercise 3. Represen he following join probabiliy by a BN 6 ) ( ) ( E B B C A B A E C B A A B C E

7 Eercise 3.3 Fill he blanks so ha he following and he BN become euivalen Iniial sae Iniial sae probabiliy (a) (b) (S S ) S =a S =b S =a ( 0.3 ) ( 0.7 ) S =b ( 0.4 ) 0.6 a 0.4 b a a b b S S S 3 S T N X a a N X b b N X X X 3 X T X s s BN 7

8 Eercise 3.4 Assumes a probabilisic model (μ) a raining sample and a prior disribuion of a parameer (μ) are given as follows. Gaussian disribuion wih mean μ and variance ep μ ep Gaussian disribuion wih mean 0 and variance ) Esimae poserior disribuion ) Esimae predicive disribuion Noe: ep c d c d 8

9 Eercise 3.4 (Answer) 9 ep 4 ep ep ep ep ep ep N d d 3 3 ep 3 N d

10 Variaional inference 0

11 Variaional Bayes Evaluaing poserior disribuion is ofen no feasible Le s approimae i wih a simpler disribuion p p p p p d Simpler disribuion Too comple As he approimaion measure KL divergence can be used arg min KL p * (For simpliciy of noaion) The minimum of he KL divergence is found using variaional mehod

12 KL and Lower bound inimizing KL is eual o maimizing lower bound d p p d p p d p p KL d p p KL p odel evidence (Consan for ) Lower bound d p L

13 Approimaion by Facorizaion Assumes (groups of) hidden variables are condiionally independen I is called a mean filed approimaion by anay o physics No resricion on he funcional forms p i i i 3

14 oserior Inference wih ean Field Approimaion Suppose a probabiliy model consis of wo (groups of) hidden variables and a (group of) observaion variable We approimae p() as ()=()() arg ma L arg ma L d d The mehod of Lagrange muliplier arg ma F ] L d F[ d 4

15 aimizaion of F[] 5 ] [ d d dd p F 0 0 d p d p z d p C d p C ep ep C C : normalizaion consan Variaional mehod (See he appendi for he derivaion) The maimum is obained by alernaively updaing () and () saring from an iniial disribuion

16 Variaional G 6 X p X X X T Θ T π d d p p ep ep ean field approimaion:

17 Con. 7 d X d d d X X ep ep ep ep ep Cf. (compare wih he above resuls) X X X T Θ T π ep X ep X X X ep ep

18 Sampling ehods 8

19 seudo Random Generaor On digial compuer everyhing is deerminisically calculaed and here is no randomness owever someimes we wan random numbers os programing languages have a pseudo random generaor funcion yhon.6 > impor random > random.random() > random.random() > random.random()

20 Sampling From a Uniform isribuion Random numbers disribued uniformly over some region Eample: isogram of samples obained from a uniform disribuion over (0 ) (To make he graph scilab was used) 0 samples 000 samples Number of occurrence Number of occurrence hisplo(0rand(:0) normalizaion=%f) hisplo(0rand(:000) normalizaion=%f) 0

21 Sampling From a Gaussian isribuion Sandard normal (Gaussian) disribuion has a mean 0.0 and a variance.0 Eample: 00 samples 0000 samples Number of occurrence Number of occurrence

22 Transform of Random Variable Le be a random variable and f be a funcion y = f(). When follows p() y follows he following disribuion (y) y d p dy

23 Eample When p() and y = f() are given as follows obain disribuion (y) p( ) 0 Answer y 3 y d ( y) y 5 3 dy 3 3 p() () 0 /3 5 y 3

24 Eercise 4. When p() and y = f() are given as follows obain disribuion (y) p( ) 0 y 0 y 0 y ep d dy y ep y d ( y) p( ) ep( y) dy # of occurrence isogram of # of occurrence isogram of y y 4

25 Eercise 4. When p() and y = f() are given as follows obain disribuion (y) p( ) ep N 0 y 3 4 y 4 3 d dy 3 y 4 N 43 d ( y) p( ) ep y dy 3 3 isogram of isogram of y # of occurrence # of occurrence y 5

26 Sampling form Comple isribuions isribuions ha can be obained by a ransformaion from a simple disribuion (such as uniform disribuion) is limied We need sampling mehods ha do no reuire inegral and inverse of a funcion and can be applied o more comple disribuions Ancesral sampling Rejecion sampling arkov Chain one Carlo (CC) 6

27 Ancesral Sampling Assumpions: We wan samples from a join disribuion p( ) The join disribuion is given as a Bayesian nework Sampling from he condiional disribuions are easy Algorihm: Sample from he paren nodes o he child nodes in order For he child nodes use he already sampled paren value in he condiional par A B (A) (BA) (CB) (AB) (E) C E 7

28 Rejecion Sampling Assumpions: We wan samples from a disribuion p(). The normalizaion consan may be unknown. p ~ p p~ We have a disribuion () from which we can easily derive samples. We refer o as a proposal disracion 8

29 rocedure of Rejecion Sampling Algorihm:. Choose a consan k so ha he following holds. erive a sample s from () 3. erive a sample u from a uniform disribuion ranging [0 k(s)] 4. If u > p~ hen rejec he sample. Oherwise adop i p~ k ~ p u s k(s) k() 9

30 arkov Chain one Carlo General and powerful framework for sampling Scales well wih he dimensionaliy of he sample space ainains a sae ha forms a arkov chain. The se of he saes follows he desired disribuion eropolis algorihm eropolis asings algorihm Gibbs Sampling 30

31 eropolis Algorihm Assumpions: We wan samples from a disribuion p(x) p ~ X px The normalizaion consan may be unknown Iniializaion:. repare a symmeric proposal disribuion ( A B ) ha saisfy ( A B )=( B A ). repare an iniial sae 0 3

32 Eercise 4.3 Show ha N( A B ) = N( B A ) where N(mv) is he Gaussian disribuion wih mean m and variance v 3 ep ep A B A B B A B A N N

33 rocedure of eropolis Algorihm Algorihm:. Ge a candidae sample * from he proposal disribuion. ( ) based on he curren sae ~ p Accep he candidae wih probabiliy * A * min ~ or discard i p 3. If he candidae is acceped save i as he ne sae +. If i is discarded hen se + euals o 4. Goo sep * 33

34 eropolis asings Algorihm An eension of he eropolis algorihm No symmeric reuiremen for he proposal disribuion The accepance probabiliy of he candidae sae is defined as follows. Oher par is he same as he eropolis algorihm A k * min ~ p ~ p * k * k * 34

35 Gibbs Sampling roblem: We wan samples from a join disribuion p( ) Algorihm:. repare an iniial sae X 0 = < > 0. Selec one of he variables i in order or a random 3. Ge a sample from p i X \ i and updae i wih ha value 4. Goo sep. Afer enough ieraions he disribuion of follows p(x) Compared o he eropolis algorihm: Sampling from condiional disribuion of i given all oher variables need o be feasible There is no rejecion sep 35

36 Quick ome Work Q3. Q4. ue: 0:00 :00 Today Submission: Aach o an Tile: TAISTQ3Q4 Forma: Te file file name: your suden I (eg ) Your name Your I Q3. (A) (B) Q4. 36

37 Appendi 37

38 Variaional ehod (Ouline) Review of derivaives Funcion: a mapping from a value o a value f f Se of values Se of values f f

39 Funcional and Funcional erivaive Funcional Se of funcions Ff Se of values E. Enropy [p] akes a funcion p (probabiliy disribuion) and reurns a value F f If a funcional F akes a maimum/minimum a f 0 and f is close o f 0 Ff Ff f f f 0 f 39

40 F f Euler Lagrange Euaion f f F is a funcional of f Suppose F akes minimum/maimum a 0 f 0. Le η be an arbiral funcion of and ε is a scalar consan g F f g(ε) is a funcion of ε (akes and reurns a scalar) When ε is closed o 0 f is close o f 0. F f Ff Therefore. 0 0 This mus hold for arbiral η. 0 F f g erely a derivaive of a funcion 40

41 Con. C When F f h f d F f h h f f h f f d d d h d 0 f mus hold for arbiral η. h f 0 C.f. ow abou when Ff hf f d a b 0 b a 4