PCFG estimation with EM. The Inside-Outside Algorithm

Transcription

1 PCFG estiation with EM The Inside-Outside Alorith

2 Presentation order l otation l Calculatin inside robabilities l Calculatin outside robabilities l General schea or EM aloriths l The inside-outside alorith

3 Soe notation l l l { 1... n } on-terinal sybols hidden variables w 1... w 1 w Sentence observed data sans w... w in strin VP 13 w 1 w 2 w 3

4 Inside robability l Deinition: l Couted recursively base case: l Induction: G w P k k kk k β G w P k kk G w P k + s r d s r s r d d P 1 1 β β β... G w P G w w P β

5 Inside robability exale l Consider the ollowin PCFG raent P DET 0.8 P 0.2 DET a 0.6 DET the 0.4 ale 0.8 orane 0.2 β DET β 11 P the DET11 G P DET the G 22 P orane G 0.2 β P 12 P P DET β DET 11 β β P

6 Inside robability exale l Consider the ollowin PCFG raent P DET 0.8 P 0.2 DET a 0.6 DET the 0.4 ale 0.8 orane 0.2 β DET β 11 P the DET11 G P DET the G 22 P orane G 0.2 β P 12 P P DET β DET 11 β β P l What is the robability o a sentence under a PCFG? β 1 P S w1... w G S 0.4

7 Outside robability l Deinition α P w1 1 w + 1 G l That is the oint robability o startin with 1 coonly called S and eneratin words w 1 w -1 the non-terinal and words w +1 w. 1 S w 1 w -1 w w w +1 w

8 Calculatin outside robability l Couted recursively base case: α1 1 1 α 1 1 l How do we calculate this base case? l Recall the deinition: 0 α S 1 1 in ters o l Intuition: ust be either the L or R child o a arent node. We irst consider the case when it is the L child. α α P w1 1 w + 1 G

9 Outside robabilities: decoosin the roble The shaded area reresents the outside robability α which we need to calculate. How can this be decoosed? e +1 e w 1... w -1 w... w w w e w e+1... w

10 Outside robabilities: decoosin the roble e Ste 1: We assue that is the arent o. Its outside robability α e reresented by the yellow shadin is available recursively. How do we calculate the cross-hatched robability? e +1 e w 1... w -1 w... w w w e w e+1... w

11 Outside robabilities: decoosin the roble Ste 2: The red shaded area is the inside robability o which is available as β + 1 e. + 1 e e +1 e w 1... w -1 w... w w w e w e+1... w

12 Outside robabilities: decoosin the roble Ste 3: The blue shaded art corresonds to the roduction which because o the contextreeness o the raar is not deendent on the ositions o the words. It s robability is sily P G and is available ro the PCFG without calculation. e +1 e w 1... w -1 w... w w w e w e+1... w

13 Outside robabilities: decoosin the roble Multilyin the ters toether we have the oint robability corresondin to the yellow red and blue areas assuin was the let child o and iven ixed non-terinals and as well as a ixed artition e in [+1]. α β + 1 e P e +1 e w 1... w -1 w... w w w e w e+1... w

14 Outside robabilities: decoosin the roble The total oint robability or a let sided can be calculated by suin over all non-terinals and and artition e. e + 1 α β + 1 e P e +1 e w 1... w -1 w... w w w e w e+1... w

15 Outside robabilities: decoosin the roble The total oint robability or a riht-sided is shown scheatically in this diara. The relevant calculation is: 1 e 1 α β e 1 P e e 1 w 1... w e-1 w e... w -1 w... w w w

16 Calculatin the outside robability: inal or e e +1 e e 1 w 1... w -1 w... w w ww e e+1... w w 1... w e-1 w e... w -1 w... ww w Since ay be either the let or riht child we have to add both ters. And since / will et counted twice when it ust be discounted on one side. e + 1 α β + 1 e P α 1 + e 1 α β e 1 P

17 General schea or certain EM aloriths l Given two events x and y the axiu likelihood estiation MLE or their conditional robability is: P x y count x y count x l I they are observable it s easy to see what to do: ust count the events in a reresentative corus and use the MLE or a soothed distribution.

18 General schea or certain EM aloriths l What these are hidden variables that cannot be observed directly? Use a odel µ and iteratively irove the odel based on a corus o observable data O enerated by the hidden variables: Eµ [ count x y O] Pˆ µ x y E [ count x O] µ l It is worth notin that i you know how to calculate the nuerator the denoinator is trivially derivable.

19 General schea or certain EM aloriths l By udatin µ and iteratin the odel converes to at least a local axiu. l This can be roven but I will not do it here.

20 The inside-outside alorith l Goal: estiate a odel µ that is a PCFG in Chosky noral or that characterizes a corus o text. l Reuired inut: l l Size o non-terinal vocabulary n At least one sentence to be odeled O

21 The inside-outside alorith l Stated with the eneral schea described earlier we seek to the MLE robabilities or roductions in the raar. r r s count P count s l Observe that this would be trivially easy to calculate this with a treebank since the non-terinals are observable in a treebank

22 The inside-outside alorith l Since the non-terinals are not visible we can use EM to estiate the robabilities iteratively: ] [ ] [ ˆ O count E O count E P s r s r µ µ µ

23 The inside-outside alorith l We bein by takin the nuerator alone: E µ [ count O] r s

24 The inside-outside alorith What we want is or iven non-terinals r and s a robability that is both used at soe oint in the derivation and accounts or san w. Since there are two rules on the RHS we need to ick a artition between the and call it d: r s α P β d β d 1 r s + α r d P r s s d +1 β d β r s d + 1 w 1... w -1 w... w d w d+1... w w w

25 The inside-outside alorith l Suin ives the total robability or any artition d: r s α P d l Exectation ust involves suin the robabilities o all ossible oortunities or usin this rule in the derivation o w 1. Each such oortunity is a san o 2 words or ore in w 1 since we are dealin with binary rules. E µ [count r s O] 1 β d β d r 1 +1 s P + 1 r s Oµ

26 The inside-outside alorith l We can use the deinition o conditional robability to turn P r s Oµ into P l Thereore the exected value o the nuerator in the EM euation is r s Oµ/PO µ α P r s β r d β s d +1 d PO µ l PO µ is ust the inside robability β 1 1

27 The inside-outside alorith l otice the analoy with the orwardbackward alorith. Probability o ettin ro the start to the oint where the latent event haens accordin to µ. Outside Forward 1 +1 uber o oortunities or the unobservable event to haen. Sans Tie stes Probability o the latent event accordin to µ. Rule Transition Probability o ettin the rest o the way accordin to µ. Inside Backward 1 α P r s β r d β s d +1 d PO µ Probability o the entire observed strin bein enerated accordin to µ uses solution to irst undaental roble

28 The inside-outside alorith l What is the denoinator E [ count O]? l One ossibility is to calculate the value o the nuerator and su the result over all non-terinals r s. µ

29 The inside-outside alorith l Also intuitively it can be thouht o as a su o the robabilities over ALL sans in the w 1 that enerated. The robability or a roduction in a iven san is: l Thus the exectation count o usin the roduction in a iven sentence is: P P P β β α µ µ µ β β α

30 The inside-outside alorith l Puttin the ieces toether yields: l otice that the indices on the suations are slihtly dierent. This is because the nuerator deals exclusively with binary rules which ust san at least two terinals! + + s d r s r s r d d P P ˆ 1 β α β β α µ