CS626 Speech, Web and natural Language Processing End Sem

Transcription

1 CS626 Speech, Web and naural Language Proceng End Sem Dae: 14/11/14 Tme: 9.30AM-12.30PM (no book, lecure noe allowed, bu ONLY wo page of any nformaon you deem f; clary and precon are very mporan; read he queon compleely) 1. a. Suppoe a naural language generaon yem generang a enence wh N word. I neceary o place he word n correc ynax order. The machne ued a Hopfeld Ne. Aume a noon of dance beween every par of word ha a funcon of he par dmlary (whch gven). Wha hould he archecure of he Hopfeld Ne be? In how many ep wll he Ne produce an orderng of he word (gve he wor cae complexy)? Prove. (3+2+4=9) The nework hould be a Marx of ze N 2, where here are N word and N poon. Each word can occur a only 1 poon and each poon can have only 1 word. Hence, he number of neuron N 2. Each of he N 2 neuron have an nerconnecng wegh. There are no elf-connecon, and all connecon are ymmerc (.e., W = 0 for all, and W = W ). IMP: no cred for gvng expreon mlar o he ravelng aleman problem. Irrelevan. The Energy of a Hopfeld Nework wh M node : M 1 M E w x x Suppoe he maxmum abolue value of he wegh n he nework W max and ha of mnmum value be W mn. 1 Then he upper bound on he energy E max= W max M ( M 1). 2 1 The lower bound E mn= W max M ( M 1). 2 The gap n energy O(M 2 ). Now can be hown ha a each frng or nhbng of neuron he decreae n energy M 1 w 1 x. If no 0, he mnmum ep ze, W mn. Th happen when w cancel one anoher, leavng only one wegh.

2 Hence, he nework hould ablze n O(M 2 ) ep. For he curren problem, he complexy O(N 4 ). b. In a Rerced Bolzmann Machne, prove ha he probably of ae ( macro probably ) and probably of frng of conuen neuron ( mcro probably ) are conen wh one anoher. Your anwer hould have boh forward and backward proof n he form of Macro Mcro and Mcro Macro. (4+4=8) 2. a. If a funcon F(X) lower bounded by anoher funcon G(X), wha condon() mu be afed o ha maxmzng G(X) uffce o fnd he maxma of F(X)? Th mean, one lookng for ha value X max of X ha maxmze G(X), becaue ha alo maxmze F(X). Of coure, X max may no necearly gve he global maxmum. (2) F(X)-G(X) hould be monoonc n he neghbourhood of he maxma. b. Eablh he relaon beween E(F(X)) and F(E(X)), where E denoe expecaon. (2) For a funcon F o be convex n R, E[F(X)] >= F(E[X]). If F concave, E[F(X)] <= F(E[X]) Proof: For dcree cae: E[F(X)] = F(x ). P (x ) = F(x ). λ where λ = 1 For connuou cae: E[F(X)] = F(x). P (x) = F(x). λ(x) where λ(x) = 1 Applyng Jenen nequaly F(x ). λ >= F( λ x ) = F(E(X)) QED (Gudelne: Whle he queon ak o eablh he relaonhp beween E(F(X)) and F(E(X)), paral mark have been gven o uden aumng convexy and ragh away ung Jenen nequaly). c. Argue wh rgour why uffcen o maxmze he lower bound on LL(X; θ) n order o fnd he be θ, where X he oberved daa and θ he parameer of he log lkelhood expreon LL. The lower bound E Z(LL(X, Z; θ)) whch he expecaon of LL(X, Z; θ) (Z he hdden varable).

3 Th queon wa aked n he quz, for whch nobody go he anwer compleely afacorly, and I wa omewha lenen n evaluang he anwer. (5) Commen: There a uoral by Sean Borman hp:// You have been gven 2 mark for gvng ha proof. However, ha NOT wan looked for n he anwer. You have o ufy WHY akng argmax over P(X;θ)-P(X;θ n) work! 3. Th queon on HMM and PCFG, and wan you o buld he heory of he laer from he former: a. Enumerae all he mlare beween HMM and PCFG (ae, ranon, obervaon, nal ae). (3) Oberved equence n HMM gven enence for parng 2. Sae equence of HMM pare ree 3. Model of HMM grammar of he PCFG 4. Sar ae of HMM S of grammar wk 5. Tranon n HMM Producon rule w k wk 6. Probable P ) = P ). P( w ) Probably of producon rule P( A B) ( ( k b. A wa done n he cla, for an HMM ranng ung Baum Welch Algorhm: M-ep E-ep Wk P( C( 0, n1 Wk P( S 0, n1 W ) ) 0, n T A l0 m0 ) n( Wk ) Wm l ) Wk, S where he quany of ulmae nere probably of ranng from any ae S o anoher ae S on a ymbol w k. 0, n1, w 0, n )

4 Gve wo mlar expreon for PCFG. You have o deduce he expreon for PCFG, rcly analogcally, mananng correpondence wh HMM a every ep. (5) M-ep (2.5 mark) P( ) T l0 ) l ) where = coun of me producon rule occurrng l ) = coun of me producon rule arng wh ymbol occur {2 mark f E-ep wrong} E-ep (2.5 mark) l ) = P( T S) n( T, T, S).e., Probably of pare ree number of me he producon rule appled. where T = pare ree for enence S 4. a. Defne chwa, ouchng on why chwa deleon a problem for peech proceng and ranleraon. Example are a mu. (2+2=4) A chwa hor a vowel ound aached wh every cononan by defaul. In Indo-Aryan language, a chwa may be deleed a ome place. Example of chwa deleon: (a) harka : The chwa afer r and are dropped n order o produce he correc pronuncaon. (b) agarba : In h cae, he chwa only afer r dropped. (c) Naman : he chwa afer he la n dropped. Schwa deleon a problem for ranleraon becaue may lead o ambguou ranleraon. For example, कमल may be ranleraed a Kamala or Kamal. On he oher hand, कमल may be ranleraed a Kamalaa or Kamlaa. A ranleraor whch underand chwa deleon wll underand ha he chwa a he end ha been dropped n cae of Kamal and chooe ha pellng, whle he chwa a he end of m ha been dropped n cae of Kamlaa and chooe ha pellng. For a peech proceng yem ha conver peech o ex, pronuncaon need o be appropraely rancrbed o ex and hence, mu handle chwa deleon. The problem ge

5 compounded epecally f your yem am o handle mullngual peech proceng becaue of lnguc dvergence uch a, n Dravdan language, he chwa a he end of a word no dropped (a n Aruna ). b. Gve a rule baed formulaon of chwa deleon, ha, a decon yem o decde f he chwa hould be deleed or no. Gve he Algorhm. (4) Aumpon: Th algorhm gve a rule baed formulaon for chwa deleon n cae of Hnd. Correpondng algorhm for oher language may be wren. Inpu: A ere of un n a word, w0, w1 wm Oupu: A vecor of lengh equal o number of un n he word. 0, 1, m. The vecor value can be one among 0, 1, 2. 0 ndcae deleed chwa, 1 ndcae chwa preen and 2 ndcae a ound apar from chwa (ay, a maaraa). The vecor ndcae he deleon decon. Algorhm: 1) Inalzaon: 0 = 1. m = 0. 2) From w=wm o w1: (rgh o lef cannng) 2a) If w end wh a maaraa (dependen vowel), e =2. 2b) Ele If (+1) = 0/2 or w(+1) a compound word, /* Te cae: m n Kamlaa. w n Dpaawl v/ ba n Agarba. */ Se =1. 2c) Ele If (+1)=1, Se = 0. /* Te cae: ga n Agarba */ 3) If w0 conan dependen vowel, 0=2. 4) Reurn. c. Gve a probablc formulaon for he ame problem. Gve he Algorhm for ranng and eng (called decodng). (4) We propoe a rgh-o-lef algorhm ha decde whch chwa canddae n a word can be deleed. Formulaon: The daa requred for h yem con of maaraa equence Inpu: Vecor m of lengh M. m a Boolean vecor whch ndcae wheher a word poon conan a maaraa or no. Example, gala 000, kamaal 010 and dpaawlee 1101.

6 Oupu: Vecor of lengh M. Value n are 0 or 1. 0: Schwa deleed, 1: Schwa no deleed. T* = argmax P(T m) = argmax P(m T).P(T) = argmax P(m ) P( ( 1)) Tranng: Learn probable ued n he above equaon ung an annoaed corpu. Decodng: Deermne a be value for T* ung an approprae algorhm. The fr erm ndcae an emon probably for a maaraa gven a chwa. (In cae of Hnd, P(m=rue =1)=1). The econd erm ndcae a ranon probably. Th model he alernang chwa requremen a depced n he rule-baed algorhm above). d. Suppoe you are gven yllabfed word a daa, lke he followng Lou <S>Lo</S> <S>u</S> Gve a probablc formulaon of he ak of yllabfyng a new word. Gve he algorhm for ranng and eng (called decodng) (4) For he npu characer equence C and canddae yllabfed repreenaon S he be poble repreenaon gven a S = argmax P(S C) = argmax P(C S)P(S) Bu gven S, C compleely deermned; hence P(C S) = 1. S = argmax P(S) Technque o compue P(S): Dcrmnave approach: P(S) = exp (c + Where, w f (S) ) f -> A feaure funcon ha ake a canddae yllabfed repreenaon and compue he feaure aocaed wh yllabfcaon (for e.g. Lengh of he word, number of vowel/cononan, feaure relaed o Sonory prncple ec). The feaure compuaon echnque have o be provded by he uer. w, c -> Model parameer

7 Tranng: For each example n he ranng corpu, he feaure are compued. The probable of he yllable are known ung corpu ac. Ung hee nformaon, he model parameer are compued. Decodng: For a new word, for each canddae yllabfcaon, he probably of he canddae obaned by compung he feaure. The canddae havng he hghe probably wn. A Verb lke algorhm work. (Gudelne: Only paral mark have been gven for no arng from he ab-no.e. he lkelhood expreon). =Paper end=