Speech, NLP and the Web

Transcription

1 peech L ad he Web uhpa Bhaacharyya CE Dep. IIT Bombay Lecure 79 0: Theorecal Uderpg- Mamum Lelhood ad Mamum Eropy rcple lecure 8 wa o LTK by Abhj ME

2 Fudameal prcple of mache learg Learg vacuum mpoblemporace of pror owledge Iducve Ba: Wha oo lear wha form o lear are pre-decded ME

3 rucure learg ad parameer learg rucure- par ad her relaohp arameer- probable ME 3

4 Eample /: rao able ^ V. aral equece graph ^ V O. ^ V O Th rao able wll chage from laguage o laguage due o laguage dvergece.

5 Eample /: Lecal robably Table Є people laugh... ^ V O ze of h able = # po ag age X vocabulary ze vocabulary ze = # uque word corpu

6 rucure ad parameer people laugh people V laugh V ME 6

7 CFG rule rucure + parameer V.0 DT V V 0.6 V VBD 0.4 DT he.0 guma 0.5 buldg 0.5 VBD prayed.0 bulle.0 wh.0 9 July 04 uhpa Bhaacharyya: arg 7

8 Epecao Mamzao Oe of he ey dea of acal AI ML L CV Ierave procedure Fd arameer Fd hdde varable Mamze daa lelhood ME 8

9 The co og problem Cae of co: uppoe here are oe of a co. H = The umber of Head Wha he probably of a head.e. H =? ME 9

10 Oberved varable #Obervao = X : where Therefore 3 0 H whe heo oherwe produce a head ME 0

11 Each obervao a Beroull Tral where H he probably of ucce.e. geg a head H he probably of falure.e. geg a al ME

12 Lelhood of X Lelhood of X.e. probably of Obervao equece X : LX Each ral decal ad depede. Mamum Lelhood of daa requre u o mae dl ad hu ge 0 d H he epreo for H - H H - H ME

13 Mahemacal Coveece Tae log of he lelhood. LL X ; Dffereag w.r.. H To ge he epreo for dll dh log H H H log mae H 0 H dh dll ME 3

14 Equag o 0 epreo for H H H H H ME 4

15 Mamum Eropy uppoe we do o ow how o ge he MLE or he lelhood epreo mpoble o ge he we ue: Mamum Eropy. Eample: I problem le co-referece reoluo. Eropy= log To be elaboraed laer. H H H log H ME 5

16 Cae for Epecao Mamzao Iead of oe co we o wo co. arameer < > = robably of choog fr co = robably of choog head from fr co = robably of choog head from ecod co : 3 X... We do o ow whch co he obervao came from ME 6

17 EM coued.. r X ; Z z z r X z 3 Z... z Z Z Z Z 3 Z he hdde equece rug alogde X X X 3 X Where Z = f he h obervao came from co =0 oherwe Y : z z z z ME 7

18 r Y Cd. ;. X Z ; z z. *.. We wa o wor wh LL X ; log Z X Z; Ivoe covey/cocavy ad epecao of Z ad wor wh logry;θ ME 8

19 Log Lelhood of he Daa LL X ; [ E zlog log log E zlog log log ] ME 9

20 IMORTAT OIT TO OTE Log move de he produc erm. Σ dappear gvg re o EZ place of Z Dffereae wr p p p equae o 0 ad ge he reul ME 0

21 z E z E p z E z E M p M= oberved o. of head z E p ME /. z z z z E

22 Aoher applcao of EM WD Meh Khapra all Joh ad uhpa Bhaacharyya I ae wo o Tago: A Blgual Uuperved Approach for emag ee Drbuo ug Epecao Mamzao 5h Ieraoal Jo Coferece o aural Laguage roceg IJCL 0 Chag Ma Thalad ovember 0. ME

23 Defo: WD Gve a coe: Ge meag of a e of word argeed wd or all word all word wd The Meag uually gve by he d of ee a ee repoory uually he worde ME 3

24 Eample: operao from rceo Worde Operao urgery urgcal operao urgcal procedure urgcal proce -- a medcal procedure volvg a co wh rume; performed o repar damage or arre deae a lvg body; "hey wll chedule he operao a oo a a operag room avalable"; "he ded whle udergog urgery" TOIC->ou urgery# Operao mlary operao -- acvy by a mlary or aval force a a maeuver or campag; " wa a jo operao of he avy ad ar force" TOIC->ou mlary# armed force# armed ervce# mlary mache# war mache# mahemacal proce mahemacal operao operao -- mahemac calculao by mahemacal mehod; "he problem a he ed of he chaper demoraed he mahemacal procee volved he dervao"; "hey were learg he bac operao of arhmec" TOIC->ou mahemac# mah# mah# ME 4

25 ar Worde Urdu Worde WD for ALL Ida laguage: Crcal reource: IDOWORDET Begal Worde Dravda Laguage Worde Kahmr Worde Orya Worde Hd Worde ujab Worde orh Ea Laguage Worde Marah Worde Koa Worde Gujara Worde Eglh uhpa Worde Bhaacharyya: ML ad ME 5

26 3 ye Baed Mullgual Dcoary Hd Marah A ample ery from he MulDc Epao approach for creag worde [Mohay e. al. 008] Iead of creag from crach l o he ye of eg worde Relao ge borrowed from eg worde ME 6

27 Hypohe ee drbuo acro laguage vara!! roporo of me a ee appear a laguage uform acro laguage! E.g. proporo of me he ee of u appear ay laguage hrough u ad yoym rema he ame! ME 7

28 ETIMATIG EE DITRIBUTIO If ee agged Marah corpu were avalable we could have emaed Bu uch a corpu o avalable ME 8

29 EM for emag ee drbuo roblem: galaa elf ambguou I raw cou cao be ued a oluo: I cou hould be weghed by ME 9

30 Word correpodece ee Eglh mar Marah ee umber word mar paral l h = π mar projeced Hd ee umber word mar paral l of word projeced Hd ee ec maa greeva garda galaa Repec maa aaar amaa 3 zza aadar Voce 3 awaaz war galaa ME 30

31 EM for emag ee drbuo M-ep E-ep.#.#.#.#.#.# war war awaaj awaaj greeva greeva maa maa greeva greeva maa maa gala mar mar mar mar mar mar h.#.#.#.#.#.# zza zza aadar aadar gala gala garda garda gala gala garda garda maa h h h h h h mar ME 3

32 Geeral Algo ep E v v u L j L L L v L L j L L j L L.#.# 3.#.# L L L L m L y L L v L where ep M y y v v v L m L L m L L ME 3

33 Reul Algorhm Marah % R % F % IWD rag o elf corpora; o parameer projeco IWD rag o Hd ad projecg parameer for Marah EM o ee corpora eher Hd or Marah Worde Baele ME 33

34 Reul & Dcuo Our value Maual Cro Lage robablc Cro Lage yle - elf rag daa avalable Worde fr ee baele -O-T-A Kowledge Baed Approach -O-T-A Uuperved Approach erformace of projeco ug maual cro lage wh 7% of elf- Trag erformace of projeco ug probablc cro lage wh 0- % of elf-trag remarable ce o addoal co curred arge laguage Boh MCL ad CL gve 0-4% mproveme over Worde Fr ee Baele o prude o c o owledge baed ad uuperved approache hey come owhere cloe o MCL or CL ME 34

35 Covey ME 35

36 Movao: argma compuao acal pell Checg Auomac peech Recogo ar of peech Taggg robablc arg acal Mache Tralao ME 36

37 ome geeral obervao A*= argma [A B] A = argma [A.B A] A Compug ad ug A ad B A boh eed loog a he eral rucure of A ad B mag depedece aumpo pug ogeher a compuao from maller par

38 roblem : pell checer: apply Baye Rule W*= argma [W T] = argma [W.T W] W=correc word T=mpel word Why apply Baye rule? Fdg pw v. p w? Aumpo : obaed from w by a gle error. The word co of oly alphabe Jurafy ad Mar peech ad L 000

39 roblem-: Iolaed word recogo roblem Defo : Gve a equece of peech gal defy he word. ep : egmeao Word Boudary Deeco Idefy he word Iolaed Word Recogo : Idefy W gve peech gal ^ W arg ma W W

40 roblem-3: acal MT Fd he Eglh ralao e correpodg o a gve Foreg eece f Thu we ee e be uch ha e be = argma e e f = argma e [e * f e] Laguage Model e Tralao Model f e Tralao are produced o he ba of acal model arameer are emaed ug blgual parallel corpora

41 Covey: uly Jee equaly Kullbac Lebler dace/dvergece EM formulao ME 4

42 f f f f f z ME 4

43 Crera for covey A fuco f ad o be cove he erval [ab] ff f f f [ a b] ME 43

44 Jee equaly For ay cove fuco f f f Where ad 0 ME 44

45 roof of Jee equaly Mehod:- By duco o Bae cae:- fλ. λf where λ λ f frvally rue ME 45

46 Aoher bae cae = ce f cove ce f f f f ME 46

47 Hypohe uppoe rue for.e f f ME 47

48 Iduco ep gve ha how 3 f f ME 48

49 roof where By covey 3 3 f f f f f f ME 49

50 Coued... Eame µ becaue ME 50

51 Coued... Therefore proved equaly Jee Thu ep duco a he Fally f f f f f f f f ME 5

52 KL -dvergece We wll do he dcree form of probably drbuo. Gve wo probably drbuo Q o he radom varable X : 3... :p =p p =p... p =p Q:q =q q =q... q =q ME 5

53 KLD defo KLQ D D aymmerc ad p D log 0 p q p q alo wre a KLQ D E p log E p logq ME 53

54 roof: KLD>=0 p p ] [ p q p q p p q p p KLQ log log o 0 cove log log log : - roof 0 log ME 54

55 roof cd. Apply Jee equaly 0 log log log log log o q q p p q p p q p q p p q p ME 55

56 Covey of log log log log.. log log log y y y e ME 56

57 Iereg problem Try o prove:- w w w w w w w w ME 57

58 d defo of covey Theorem: If f wce dffereable [ab] ad f '' 0 [ab] he f cove o -log cove. [ab]. ME 58

59 Lemma If f '' 0 [ a b] he f ' f ' ad [ a b] a z b ME 59

60 Mea Value Theorem fz fa z af ' za For ay fuco f f fm mf ' p where m p ME 60

61 Alerave form of z z λ λ Add λz o boh de λz λ λ z z λz λ ME 6

62 Alerave form of covey fλ λ λf λf Add λfz o boh de fz λfz λfz λfz λf λf λf λf fz fz λfz λf λf ME 6

63 roof: ecod dervave >=0 mple covey / We have ha z f z f [ f - [ f f - z] [ z z] [ f z ] f ] ME 63

64 ecod dervave >=0 mple covey / I equvale o f. f z For ome ad where z ow ce f >=0 f ' f ' Combg h wh he reul proved ME 64

65 Why all h I EM we mamze he epecao of log lelhood of he daa Log a cocave fuco We have o ae erave ep o ge o he mamum There are wo uow value: Z uoberved daa ad θ parameer From θ ge ew value of Z E-ep From Z ge ew uhpa Bhaacharyya: value of ML ad θ M-ep ME 65

66 Recap: a mple EM uao To of wo co: arameer < > = robably of choog fr co = robably of choog head from fr co = robably of choog head from ecod co : 3 X... We do o ow whch co he obervao came from ME 66

67 EM coued.. r X ; Z z z r X z 3 Z... z Z Z Z Z 3 Z he hdde equece rug alogde X X X 3 X Where Z = f he h obervao came from co =0 oherwe Y : z z z z ME 67

68 r Y Cd. ;. X Z ; z z. *.. We wa o wor wh LL X ; log Z X Z; Ivoe covey/cocavy ad epecao of Z ad wor wh logry;θ ME 68

69 Log Lelhood of he Daa LL X ; [ E zlog log log E zlog log log ] ME 69

70 IMORTAT OIT TO OTE Log move de he produc erm. Σ dappear gvg re o EZ place of Z Dffereae wr p p p equae o 0 ad ge he reul ME 70

71 z E z E p z E z E M p M= oberved o. of head z E p ME 7 /. z z z z E

72 How o fd θ How o chooe he e θ? Tae Where arg ma LL X Z X: oberved daa Z: uoberved daa Θ: parameer : LL X Z : LLXZ:θ : log lelhood of complee daa wh parameer value a θ Th leu of for eample grade ace θ Θ A every ep LL. wll Icreae ulmaely reachg local/global mamum ME 7

73 Why epecao of log lelhood? /3 X:θ he obervao lelhood Deal wh XZ:θ margalzed over Z LogΣ Z XZ:θ mahemacally proceed wh mulplyg by Z X: θ whch for each Z bewee 0 ad ad um o ME 73

74 Why epecao of log lelhood? /3 The Jee equaly wll gve ; ; log ; he probably a ; where ; mulply ad devde by ; ; ; log ; log z z z X Z Z X X Z X Z X Z X Z Z X X Z Z X ME 74

75 Why epecao of log lelhood? 3/3 Z. w.r.. complee daa of log llhood he epecao of. where ; log ; log ; ; ; arg ma o ; ; log ; ; ce ;. ; ; log ; ; log ; ; ; log ; log ; log ; ; z z Z Z Z Z Z Z E Z X E Z X X Z X LL X LL Z X Z X X Z X Z X X Z Z X X Z X X Z Z X X Z X Z X X LL X LL ME 75

76 Why epecao of Z? If he log lelhood a lear fuco of Z he he epecao ca be carred de of he log lelhood ad EZ compued The above rue whe he hdde varable form a mure of drbuo e..g oe of wo co ad Each drbuo a epoeal drbuo le mulomal/ormal/poo ME 76

77 Applcao of EM: HMM Trag Baum Welch or Forward Bacward Algorhm ME 77

78 A problem cearo Uuperved O aggg Cover he Brow corpu o a corpu wh OLY he followg ag: ou V verb J adjecve R adverb F fuco word le prepoo ad cojuco A arcle a a he ad O oher Aume raw corpu ad he creae a O agger ME 78

79 Key Iuo a a b a q r b a b b Gve: Ialzao: Compue: Approach: Trag equece robably value r ae eq rag eq ge epeced cou of rao compue rule probable Ialze he probable ad recompue hem EM le approach ME 79

80 Baum-Welch algorhm: cou ab q a b r ab ab rg = abb aaa bbb aaa equece of ae wh repec o pu ymbol o/p eq ae eq a q r b b a q q r a a b b b a a a q r q q q r q r ME 80

81 Calculag probable from able a q r b q b T=#ae w A=#alphabe ymbol j 5/ 8 3/ 8 T c A l m c w w m Table of cou rc De O/ Cou q r a 5 q q b 3 r q a 3 r q b ow f we have a o-deermc rao he mulple ae eq poble for he gve o/p eq ref. o prevou lde feaure. Our am o fd epeced cou hrough h. j l ME 8

82 Ierplay Bewee Two Equao W j T c A l0 m0 W c Wm j l C 0 W 0 j W 0 W j 0 w 0 o. of me he rao j occur he rg w ME 8

83 Illurao b:0.7 a:0.6 q a:0.67 b:.0 r Acual Dered HMM b:0.48 q a:0.04 r a:0.48 b:.0 Ial gue ME 83

84 a Oe ru of Baum-Welch algorhm: rg ababb a b b a a b b b b pah a q r b r q a q q q r q r q q q r q q q q q q q r q q q q q q q q b q q Rouded Toal ew robable 0.06 =0.0/0. ae equece * ε codered a arg ad edg ymbol of he pu equece rg. Through mulple erao he probably value wll coverge. ME 84

85 Compuaoal par / j j j W j W j W W w W W W w W W W W W W W C ] [ ] [ ] [ ] [ w 0 w w w w - w 0 j - + ME 85

86 Compuaoal par / j B w F j B w W F j B w W F W W w W W W w W W W w W j j j j j j j w 0 w w w w - w 0 j - + ME 86

87 Dcuo. ymmery breag: Eample: ymmery breag lead o o chage al value b:.0 b:0.5 a:0.5 a:.0 Dered a:0.5 b:0.5 a:0.5 b:0.5 a:0.5 a:0.5 b:0.5 b:0.5 Ialzed ruc Local mama 3. Label ba problem robable have o um o. Value ca re a he co of fall of value for oher. ME 87