Probability and Information Theory for Language Modeling. Statistical Linguistics. Statistical Linguistics: Adult Monolingual Speaker

Transcription

1 Probability ad Iformatio Theory for Laguage Modelig Statistical vs. Symbolic NLP Elemetary Probability Theory Laguage Modelig Iformatio Theory Statistical Liguistics Statistical approaches are clearly useful for egieerig tasks. Are statistical approaches appropriate for scietific study? Laguage Acquisitio Laguage Chage Laguage Variatio 1 2 Statistical Liguistics: Adult Mooligual Speaker Elemetary Probability Theory Error tolerace Laguage comprehesio: A average setece has a huge umber of "possible" sytactic structures. Termiology: Experimet Sample a repeatable process a possible outcome Sample space all samples for a experimet Defiig which aalysis is correct is ot a computatioal problem. Possible Couter: These are "performace issues" But ote: "performace issues" is ot the same as "computatioal issues" Evet a set of samples Probability assigs a probability distributio to each sample. P(A) P x x A Uiform All samples are Distributio equi-probable. 3 4

2 Elemetary Probability Theory (2) A B Prior probability of evet A is P(A) We are told that B is true Now the probability of A is P(A ad B)/P(B) This is the coditioal probability of A give B: P(A B) = P(A B)/P(B) Elemetary Probability Theory (3) Multiplicatio Rule: P(A B) = P(A B)P(B) = P(B A)P(A) Bayes rule: P(A B) = P(B A)P(A)/P(B) Idepedace: P(A B) = P(A) P(B A) = P(B) P(A B) = P(A)P(B) 5 6 Bayes Theorem: Example Laguage Modelig Evet A = a setece cotais a certai liguistic costructio P(A) = Evet B = our program reports that it foud the liguistic costructio P(B A) = 0.9 P(B A) = 0.1 (true positive) (false positive) Suppose the program fids the costructio. What is the chace that it is correct? Goal: Fid the probability of a "text" "text" ca be a word, a utterace, a documet, etc. Texts are geerated by a ukow probability distributio Use laguage model to capture a priori iformatio about the likelihood of a text. We are more likely to predict a text with a higher a priori probability. P(A B) = P(B A)P(A)/P(B) 7 8

3 Statistical Iferece Why Do Laguage Modelig? Use traiig data to make ifereces about the ukow distributio. Defie a class of cadidate probability distributios. Divide texts ito equivalace classes. Examples: P w = P w.type P w i = P w i.type w i 1.type Select the "best" cadidate Use traiig data to evaluate each cadidate probability distributio. Speech recogitio Predict word sequeces that are more likely. Spellig correctio Suggest words that are more likely. Machie traslatio Suggest traslatios that are more likely. Geeratio Laguage models help us geerate "likely" seteces Output Forms Fidig the Source Text Each text geerates a output form, which we ca directly observe. Speech recogitio: a sequece of souds Spellig correctio: a sequece of characters Bayes Rule: Laguage Model P text P output text P text output = P output Machie traslatio: a source laguage text Texts ad their output forms are geerated by a ukow distributio: P text,output Fid the most likely text for a output form: P text output 11 Recoverig the uderlyig form: P text output = P text P output text P output P text P output text 12 =

4 Divide & Coquor Equivalace Classes P(output) has a very large sample space. P(w w 1,..., w -1 ) has a large sample space. We ca ot directly model P(output). Reduce the problem of modelig P(output) to simpler estimatio problems, whose solutios ca be combied. Estimate the probability of a text oe word at a time: P w 1, w 2,...,w = P w i w 1,...,w i1 i=1 Divide P(w w 1,..., w -1 ) ito equivalace classes Example: P(w w 1,..., w -1 ) P(w w -1 ) Estimate the probability of each equivalace class. Cout the umber of traiig istaces i each equivalace class. Use these couts to estimate the probability for each equivalace class Maximum Likelihood Estimatio Problems with MLE Predict the probability of a equivalace class usig its relative frequecy i the traiig data: P x = C x N C(x) = frequecy of x i the traiig data N = umber of traiig istaces Uderestimates the probability for usee data C(x)=0 Maybe we just did't have eough traiig data. Overestimates the probability for rare data C(x)=1 Estimates based o oe traiig sample are ureliable

5 Statistical Estimators Statistical Estimators: ltk Use the traiig data to form a more advaced estimate of P(x) ltk.probability.probdisti defies a iterface for probability distributios. Laplace, Lidstoe, ad ELE: Reserve a small portio of the probability distributio for usee data. See Maig pp Held out estimatio: Use a small amout of held-out data to decide the probability of usee data. Probability distributios are typically costructed from frequecy distributios: >>> pdist = ELEProbDist(fdist) >>> prit pdist.prob('the') 0.02 >>> prit pdist.cod_prob('the', SetEvet('the', 'a')) 0.6 See the ltk referece documetatio for more iformatio Noisy Chael Model Noisy Chael Model (2) ecoder chael I O decoder P(o i) Î ecoder chael I O decoder P(o i) Î Chael itroduces "oise." Task: optimize throughput ad accuracy More redudacy -> More accuracy Less redudacy -> More throughput May statistical NLP problems ca be thought of as decodig problems. No cotrol over ecodig Example: optical character recogitio i o P(i) = actual text = text with mistakes = laguage model P(o i) = model of OCR errors See Maig Table 2.2 (p. 71) 19 20

6 Etropy Example: Simplified Polyesia The iformatio cotet of a probability distributio. Letter p t k a i u P(Letter) 1/8 1/4 1/8 1/4 1/8 1/8 The average legth of the message eeded to trasmit the outcome. H X = P x log P x x Per-letter etropy: H X = P x logp x = log1 8 plus2 1 4 log1 4 Example: fair 8-sided die H X = x= 1 8 P x logp x 8 = x=1 1 8 log1 8 = 2.5bits = log1 8 = log 1 8 = Optimal Biary Ecodigs Example: Simplified Polyesia (2) O average, how may yes/o questios do you eed to ask to fid the outcome? Example questios: Is the letter t or a? Is the letter a cosoat? Biary ecodig: each bit is a yes/o questio. Etropy is the average message legth, usig this ecodig. t p, k, u, or i? y a? u or i? y y a p k? Optimal ecodig for Simplified Polyesia: Letter p t k a i u Ecodig y k i u? y u 23 24

7 Relative Etropy Readigs Relative etropy measures the differece betwee two probability distributios. D p q = p x log p x x X q x If we ecode the actual data usig the laguage model, how may extra bits do we use, o average? Use relative etropy to measure performace: p = actual distributio (from test data) q = laguage model performace = D(p q) Maig 1, 2 Abey #