Efficient Bayesian Parameter Estimation in Large Discrete Domains

Transcription

1 fficient Bayeian Parameter timation in Large Dicrete Domain Nir Friedman Computer Science Diviion Univerity of California 387 Soda Hall, Berkeley, CA Yoram Singer AT&T Lab, Room A Park Avenue Florham Park, NJ Abtract In thi paper we examine the problem of etimating the parameter of a multinomial ditribution over a large number of dicrete outcome, mot of which do not appear in the training data. We analye thi problem from a Bayeian perpective and develop a hierarchical prior that incorporate the aumption that the oberved outcome contitute only a mall ubet of the poible outcome. We how how to efficiently perform exact inference with thi form of hierarchical prior and compare our method to tandard approache and demontrate it merit. Category: Algorithm and Architecture Preentation preference: none Thi paper wa not ubmitted elewhere nor will be ubmitted during NIPS review period. Introduction One of the mot important problem in tatitical inference i multinomial etimation: Given a pat hitory of obervation independent trial with a dicrete et of outcome, predict the probability of the next trial. Such etimator are the baic building block in more complex tatitical model, uch a prediction tree [, 5, 4], hidden Markov model [2] and Bayeian network [3, 7]. The root of multinomial etimation go back to Laplace work in the 8 th century [0]. In Bayeian theory, the claic approach to multinomial etimation i via the ue of the Dirichlet ditribution (ee for intance [4]). Laplace law of ucceion and other common method can be derived uing Bayeian inference with the Dirichlet ditribution a a prior ditribution. The Dirichlet ditribution entertain everal propertie which become very ueful in tatitical inference. In particular, etimate derived uing Dirichlet prior are conitent (the etimate converge with probability one to the true ditribution), conjugate (the poterior ditribution i alo a Dirichlet ditribution), and can be computed efficiently (all querie of interet have a cloed-form olution). Furthermore, theoretical tudie of online prediction of individual equence how that prediction uing Dirichlet prior i competitive with any other prior ditribution (ee for intance [9, 2, 5] and the reference therein). Unfortunately, in ome key application, Dirichlet prior are unwieldy. Thee application are characteried by everal ditinct feature: The et of poible outcome i very large, and often not known in advance. The number of training example i mall compared to the number of poible outcome. The outcome that have poitive probability contitute a relatively mall ubet of the poible outcome. However, thi ubet i not known in advance.

2 F % % In thi ituation, a prediction baed on a Dirichlet prior, in particular, the uniform ditribution, tend to aign mot of the probability ma to outcome that were not een in the training et. For example, conider a natural language application, where outcome are word drawn from an nglih dictionary, and the problem i predicting the probability of word that follow a particular word, ay Bonia. If we do not have any prior knowledge, we can conider any word in the dictionary a a poible candidate. Yet, our knowledge of language would lead u to believe that in fact, only few word, uch a Heregovina, hould naturally follow the word Bonia. Furthermore, even in a large corpora, we do not expect to ee many training example that involve thi phrae. A another example conider the problem of etimating the parameter of a dicrete dynamical ytem. Here the tak i to find a ditribution over the tate that can be reached from a particular tate (poibly after the ytem receive a particular external control ignal). Again, although the number of poible tate can be large, we often believe that the et of reachable tate, from any tate, i much maller. In thi paper, we preent a Bayeian treatment of thi problem uing an hierarchical prior that average over an exponential number of hypothee each of which repreent a ubet of the feaible outcome. Such a prior wa previouly ued in a pecific context of online prediction uing uffix tree tranducer [4]. A we how, although thi prior involve exponentially many hypothee, we can efficiently perform prediction. Moreover, our approach allow u to deal with countably infinite number of outcome. The paper i organied a follow. We tart in Section 2 with a hort review of Dirichlet prior. In Section 3 we decribe the hierarchical prior for multinomial ditribution. In Section 4, we examine how to apply our approach to countably infinite et of outcome. Section 5 we decribe experimental reult that how the effectivene of our approach in language modeling. We conclude in Section 6. 2 Dirichlet prior Let be a random variable that can take poible value from a et Σ. Without lo of generality, let Σ. We are given a training et that contain the outcome of independent draw of from an unknown multinomial ditribution. We denote by be the number of occurrence of the ymbol in the training data. The multinomial etimation problem i to find a good approximation for (which i alo a multinomial ditribution). Thi problem can be tated a the problem of predicting the outcome given. Given a prior ditribution over the poible multinomial ditribution, the Bayeian etimate i:!#"$ &' ( *) +!#",& )- +!#",. ) () where ) 0/ i a vector that decribe poible value of the (unknown) probabilitie "7 ' 8", and! i the context variable that denote all other aumption about the domain. (We conider particular context in the next ection.) The poterior probability of ) can rewritten uing Baye law a: )- +!#"89: ;)!#"<& )=!#"$ > )-!#"@ 8A (2) The Dirichlet ditribution i a parametric family that i conjugate to the multinomial ditribution. That i, if the prior ditribution i from thi family, o i the poterior. A Dirichlet prior for i pecified by hyperparameter B BC3, and ha the form: & )-!#"D BC" B " ;G AIH 'J and 8K 0 for all '" where L" NM8O 0 P,Q H R HTS. P i the gamma function. Given a Dirichlet prior, the initial prediction for each value of i U!#"8 )-!#",. ) B WV B V

3 J It i eay to ee that, if the prior i a Dirichlet prior with hyperparameter B B 3, then the poterior i a Dirichlet with hyperparameter B X BC3 X 3. Thu, we get that the prediction for ( i & ( > +!#"Y B X V 'B V X V " We can think of the hyperparameter B a the number of imaginary example in which we aw outcome. Thu, the ratio between hyperparameter correpond to our initial aement of the relative probability of the correponding outcome. The total weight of the hyperparameter repreent our confidence (or entrenchment) in the prior knowledge. A we can ee, if thi weight i large, our etimate for the parameter tend to be further off from the empirical frequencie oberved in the training data. 3 A hierarchical prior We now decribe a more tructured prior that capture our uncertainty about the et of feaible value of. We define a random variable Z that take value from the et 2 Σ of poible ubet of Σ. The intended emantic for thi variable, i that if we know the value of Z, then 78[ 0 iff $\]Z. Clearly, the hypothei ZN Σ^ (for Σ^D_ Σ) i conitent with training data only if Σ^ contain all the indice for which [ 0. We denote by Σ` the et of oberved ymbol. That i, Σ` a ; : [ 0, and we let b@`( Σ`. Suppoe we know the value of Z. Given thi aumption, we can define a Dirichlet prior over poible multinomial ditribution ) if we ue the ame hyper-parameter B for each ymbol in Z. Formally, we define the prior: & )8 Zc"$ Z B$" 'BY"ed fdd Uing q. (2), we have that: & 'g f ;G H > jt 7Zc"8 lk J and 0 for all ih \-Z" (3) d f$d G L A G L if $\]Z 0 otherwie Now conider the cae where we are uncertain about the actual et of feaible outcome. We contruct a two tiered prior over the value of Z. We tart with a prior over the ie of Z, and then aume that all et of the ame cardinality have the ame prior probability. We let the random variable m denote the cardinality of Z. We aume that we are given a ditribution m] >b@" for bn. We define the prior over et to be: &Z m- Ub@"$ po We now examine how to compute the poterior prediction given thi hierarchical prior. Let denote the training data. Then it i eay to verify that U &"r J; B X t b@b X b6q H (4) (5) Z &" (6) fuvd f$d w u g f Let u now examine which et Z actually contribute to thi um. Firt, we note that et that do not contain Σ` have ero poterior probability, ince they are inconitent with the oberved data. Thu, we can examine only et Z that contain Σ`. Second, a we noted above, & Zc" i the ame for all et of cardinality b that contain Σ`. Moreover, by definition the prior for all thee et i the ame. Uing Baye rule, we conclude that &Z &" i the ame for all et of ie b that contain Σ`. Thu, we can implify the inner ummation in q. (6), by multiplying the number of et in the core of the ummation by the poterior probability of uch et. There are two cae. If x\ Σ`, then any et Z that ha non-ero poterior appear in the um. Thu, in thi cae we can write: & ( U &"D J B X b@b X &'m- Ub &" if x\ Σ`

4 ˆ Ž q k 3 J w 3 w Ž If yh \ Σ`, then we need to etimate the fraction of ubet of Z with non-ero poterior that contain. Thi lead to an equation imilar to the one above, but with a correction for thi fraction. Note, however, that all unoberved outcome have the ame poterior probability. Thu, we can imply divide the ma that wa not aigned to the oberved outcome among the uneen ymbol. Notice that the ingle term in q. (3) that depend on can be moved outide the ummation. Thu, to make prediction, we only need to etimate the quantity: { 8"$ ; b ` B X bb X &b &" and then & > &"8 G LDA G '{ 8" if C\ Σ` L j H } '{ D"," if h \ Σ` We can therefore think of { 8" a caling factor that we apply to the Dirichlet prediction that aume that we have een all of the feaible ymbol. The quantity } '{ D" i the probability ma aigned to novel (i.e., uneen) outcome. Uing propertie of Dirichlet prior we get the following characteriation of '{ D". Propoition 3.: m] >b &"D ; L where U&m] >b" b! b }Wb ` "! ƒ bby" 'b@b X =" Proof: To compute '{ 8", we need to compute &'mw b &". Uing Baye rule, we have that &'b &"D By introduction of variable, we have that: m >b@"<&m] :b@" &' m- Ub ^ ",&'m- Ub ^ " m >b@"$ J Zt"<&Z m Ub@"4 fd Σ uvd f$d w Uing tandard propertie of Dirichlet prior, we have that if Σ`(_UZ, then &' Zc"8 Z B$" B X " (8) Z B X =" 'g 'BY" f Now, uing q. (8) and (5), we get that if Σ` _UZ, and b& Z, then & Zt"<&Z m Ub@"$ o H b@b$" b q 'b@b X =" B$" H 'B X "4 (9) 'g Σ Thu, yˆ Š Œ Ž Œ Ž ÎŽ š Ž Ž ÎŽ œ=ž& Î š Îñœ=ž o@ ot q Œ Ž Î š Î tœ]ž ÎŽ š ÎŽ Ž ÎŽ œ=ž&!! & Σ Note that the term in the quare bracket doe not depend on the choice of b. Thu, it cancel out when plug q. (0) in q.(7). The deired equality follow directly. From the above propoition we immediately get that Corollary 3.2: '{ D"8 (7) T; G L G L ()

5 «O O H ƒ G ª ª P(S = k D ) k N = 20 N = 50 N = 00 N = 200 N = 400 C(D,L) N = 25 N = 50 N = 00 N = 200 N = Figure : Left: Illutration of the poterior ditribution &m &" for different value of, with b@` 20, 00, B= l 25, and &m] ab@" Right: Illutration howing the change in '{ 8" for different value of, with b@`( 25, B=, and &'m- Ub@"$ Note that m] >b &" and { 8" depend only on b@` and and doe not depend on the ditribution of count among the b` oberved ymbol. Note that when,ª i ufficiently larger than b@` (and thi depend on the choice of B$"!, then the term! ƒ i G L much maller than. Thi implie that the poterior for larger et decay rapidly. We can ee thi behavior on the left hand ide of Figure that how the poterior ditribution of &'m &" for different dataet ie. 4 Unbounded alphabet By examining the analytic form of '{ 8", we ee that the dependency on i expreed only in the number of term in the ummation. If the term vanih for large b, then '{ 8" become inenitive to the exact ie of the alphabet. We can ee thi behavior on the right hand ide of Figure, which how '{ 8" a a function of. A we can ee, when i cloe to b@`, then { 8" i cloe to. A grow, '{ 8" aymptote to a value that depend on and b ` (a well a B and the prior m] :b@" ). Thi dicuion ugget that we can apply our prior in cae where we do not know in advance. In fact, we can aume that i unbounded. That i, Σ i iomorphic to 2. Aume that we aign the prior &m0 pb@" for each choice of, and that lim3t«&'m: lb@" exit for all b. We define '{ "( lim3t«{ 8". We then ue for prediction the term U &"8 G L8A G { ". L For thi method to work, we have to enure that '{ " i well defined; that i, that the limit exit. Two uch cae are identified by the following propoition. Propoition 4.: If &'m ab@" i exponentially decreaing in b or if B K and &m ab" i polynomially decreaing in b, then { " i well-defined. To prove the # above propoition we ue Stirling approximation to " and how that lim O 0 for ome contant [ (detail omitted due to lack of pace). In practice we evaluate { " by computing ucceive value of (the logarithm of), until we reach value that are ignificantly maller than the larget value beforehand. Since i exponentially decaying, we can ignore the ma in the tail of the equence. A we can ee from the right hand ide of Figure, there i not much difference between the prediction uing a large, and unbounded one. 5 mpirical evaluation We have ued the propoed etimation method to contruct a tatitical model for predicting the probability of character in the context of the previouly oberved character. Such model, often referred to a bigram model, are of great interet in application uch optical character recognition and text compreion. We teted two of prior ditribution for the alphabet ie 0 ma b@" : an exponential prior, 0 mw ab" 9U±, and a polynomial prior, 0 m ab@"9ab HL². The training and tet L

6 ª A Perplexity Method Oberved Novel Overall A ( ) B (Approximated (L³ Good-Turing) Spare-Multinomial (Poly) Spare-Multinomial (xp) Table : Perplexity reult on heterogeneou character data. material were derived from variou archive and included different type of file uch C program, core dump, and acii text file. The alphabet for the algorithm conit of all the (acii and non-acii) 256 poible character. The training data conited of around 70 mega byte and for teting we ued 35 mega byte. ach model we compared had to aign a probability to any character. If a character wa not oberved in the context of the previou character, the new character i aigned the probability of the total ma of novel event. Note that thi tak i different than common language modeling benchmark where the probability of each individual word i etimated either uing the full context or by backing off to a horter context [8], which i the null context for bigram model. We compared our approach with two etimation technique that have been hown to perform well on natural data et [6]. The firt etimate the probability of a ymbol in the context of a given word a DA where i the number of different character oberved at given context (the previou character). L³ The econd method, baed on an approximation of the Good-Turing etimation cheme [6], etimate the probability of a ymbol a HLµ 8A, where i the number of different character that have been oberved only once at for the given context. Thi cheme require a fall-back etimate when l, a decribed in more detail in [6]. For evaluation we ued the perplexity which i imply the exponentiation of the average log-lo on the tet data. Table ummarie the average tet-et perplexity for oberved character, novel event, and the overall perplexity. In the experiment we fixed B] ¹ 2 for the parameter of the Dirichlet prior and ±- 2 for the exponentially and polynomially decaying prior of the alphabet ie. One can ee from the table that prediction uing pare-multinomial achieve the lowet overall perplexity. (The difference are tatitically ignificant due to the ie of the data.) The performance baed on the two different prior for the alphabet ie i comparable. The reult indicate the all the leverage in uing pare-multinomial for prediction i due to more accurate prediction for oberved event. Indeed, the perplexity of novel event uing pare-multinomial i much higher than when uing either method º or». Put another way, our approach prefer to acrifice event with low probability (novel event) and uffer high lo in favor of more accurate prediction for frequently occurring event. The net effect i a lower overall perplexity. 6 Dicuion In thi paper we preented a Bayeian approach for the problem of etimating the parameter of a multinomial ource over a large alphabet. Our method i baed on an efficient inference algorithm that i baed on hierarchical prior. Among the numerou technique that have been ued for multinomial etimation the one propoed by Ritad [3] i the cloet to our. Though the methodology ued by Ritad i ubtantially different than our, hi method can been een a a pecial cae of pare-multinomial with B et to and pecific form for the prior over the alphabet ie. The main advantage of fixing Ba i an even impler inference procedure. The impler inference procedure demand, however, a price which i lo of flexibility. In addition, our method explicitly repreent the poterior ditribution. Hence, it i more uitable for tak, uch a tochatic ampling, where an explicit repreentation of the approximated ditribution i required. Our method can be

7 combined with other Bayeian approache for language modeling uch a the one propoed by Mackay and Peto []. A briefly dicued in the introduction, there are application other than language modeling that can make ue the propoed modeling cheme. Reinforcement Learning (RL) i an example of uch a domain. One of the approache in RL i to firt etimate the parameter of an underlying Markov deciion proce. However, the number of tate of a typical Markov proce might be very large. Hence, naive etimation cheme often yield poor reult. Intead, one can ue pare multinomial to build a robut etimate for the tranition probabilitie of each tate, depite the fact that the number of time each tate wa viited might very mall. Another poible domain i parameter etimation of large Bayeian network where one might want to keep the explicit form of poterior ditribution for the parameter. Reference [] W. Buntine. Learning claification tree. In D. J. Hand, editor, Artificial Intelligence Frontier in Statitic, number III in AI and Statitic. Chapman & Hall, London, 993. [2] B.S. Clarke and A.R. Barron. Jeffrey prior i aymptotically leae favorable under entropic rik. J. Stat. Planning and Inference, 4:37 60, 994. [3] G. F. Cooper and. Herkovit. A Bayeian method for the induction of probabilitic network from data. Machine Learning, 9: , 992. [4] M. H. DeGroot. Optimal Statitical Deciion. McGraw-Hill, New York, 970. [5] Y. Freund. Predicting a binary equence almot a well a the optimal biaed coin. In Proceeding of the Ninth Annual Conference on Computational Learning Theory, 996. [6] I.J. Good. The population frequencie of pecie and the etimation of population parameter. Biometrika, 40(3): , 953. [7] D. Heckerman, D. Geiger, and D. M. Chickering. Learning Bayeian network: The combination of knowledge and tatitical data. Machine Learning, 20:97 243, 995. [8] S.M. Kat. timation of probabilitie from pare data for the language model component of a peech recognier. I Tran. on ASSP, 35(3):400 40, 987. [9] R.. Krichevky and V.K. Trofimov. The performance of univeral coding. I Tranaction on Information Theory, 27:99 207, 98. [0] P.S. Laplace. Philopuical ay on Probabilitie. Springer-Verlag, 995. [] D.J.C. MacKay and L. Peto. A hierarchical Dirichlet language model. Natural Language ngineering, (3): 9, 995. [2] L. R. Rabiner and B. H. Juang. An introduction to hidden markov model. I ASSP Magaine, 3():4 6, January 986. [3]. Ritad. A natural law of ucceion. Technical Report CS-TR , Department of Computer Science, Princeton Univerity, 995. [4] Y. Singer. Adaptive mixture of probabilitic tranducer. Neural Computation, 9(8):7 734, 997. [5] F.M.J. Willem, Y.M. Shtarkov, and T.J. Tjalken. The context tree weighting method: baic propertie. I Tranaction on Information Theory, 4(3): , 995. [6] I.H. Witten and T.C. Bell. The ero-frequency problem: etimating the probabilitie of novel event in adaptive text compreion. I Tran. on Info. Theory, 37(4): , 99.