On Modeling On Minimum Description Length Modeling. M-closed

Transcription

1 O Modelig O Miiu Descriptio Legth Modelig M M-closed M-ope Do you believe that the data geeratig echais really is i your odel class M? 7 73 Miiu Descriptio Legth Priciple o-m-closed predictive iferece Explicitly iclude predictio ad itervetio) i odelig MDL is a ethod related to odelig, iductive iferece, achie learig Rissae 978-; Barro, Rissae ad Yu 998 Models are a eas a laguage) to describe iterestig properties of the pheoeo to be studied, but they are ot itrisic to the pheoeo itself. tasks Model selectio Paraeter estiatio All odels are false, but soe are useful. Predictio Fro arithetic codig to odelig 74 Model selectio 75 Descriptive coplexity 000 bit strigs Soloooff-Kologorov-Chaiti coplexity shortest possible ecodig with the help of L code based o a uiversal coputer laguage L too strog a descriptio laguage - ucoputability 76 77

2 The idea a good odel M captures regular features costraits) of the observed data ay set of regularities we fid reduces our ucertaity of the data D, ad we ca use it to ecode the data i a shorter ad less redudat way The ore we are able to copress a sequece of data, the ore regularity you have detected i the data ad the ore you have leared fro the data to ake predictios of future data) For exaple regressio There is a trade-off betwee the odel coplexity ad fit to the data MDL coplex siple Types of MDL algorithic, ideal MDL Li ad Vitáyi 97) MML Wallace 68, 87) two-part code MDL Rissae 78, 83) uiversal odel based MDL Rissae 96, Barro, Rissae, Yu 98, Grüwald 0) + P saple space Probability set of all sequeces of outcoes set of arbitrary legth sequeces set of ifiite sequeces probability distributio over 80 8 Code Aalogy coutable) data alphabet A uiquely decodable) code C is a oe-to-oe ap fro to { 0,} + L C x) deotes the legth i bits eeded to describe x let Pbe a probability distributio. Sice P x) x oly very few x ca have large probability let C be a code for { 0,}. Sice the fractio of sequeces that ca be copressed by ore tha k bits is less tha k k = oly very few sybols ca have sall code legth 8 83

3 Correspodece Uiversal codes L is a set of codelegth fuctio)s available to ecode data x there is a - correspodece betwee probability distributios ad code legth fuctios such that large code legths correspod to sall probabilities ad vice versa assue that oe of the codelegth fuctio)s i L allows for substatial copressio of x TASK: ecode x usig iiu uber of bits for all x # + : L x ) = log P x ) 84 Uiversal code ore) 85 Uiversal Models Let M be aprobabilistic odel, i.e., a faily set) of probability distributios Assue M fiite: M = {P ),K, P M )} For exaple: L fiite There exists a code LL such that for soe costat K, for all, x, all L L : There exists a code LM s.t. for all, x, : L L x ) L x ) + K Specifically L M x ) log P x # ) + K hece, exists distributio PM s.t. L L x ) if LL L x ) + K - log PM x ) log P x # ) + K K does ot deped o, while typically L x ) grows liearly i i.e. PM x ) K'#x $ ) PM is a uiversal odel distributio) for M 86 Bayesia ixture as a uiveral odel 87 Two-part MDL code as a uiversal odel let W be a prior over M. The Bayesia argial likelihood is defied as: The ML axiu likelihood) distributio is ˆ x ) ifp # )$M {%log Px # )} PBayes x M ) = P x j )W j ) code x by first codig ˆ x ), the codig x with the help of ˆ x ) : L x M ) = log W ˆ x )) log P x ˆ x )) p j = This is a uiversal odel, sice For all, x, : log PBayes x M ) = Bayes ixture assigs larger probability shorter code legth) to outcoes... log # P x $ j )W $ j ) j = what prior leads to short code legths? log P x # ) log W # ) j =

4 Optial Uiversal Model Optial Uiversal Model Look for P* such that regret ifp * sup x # $log P * x ) $ $log Px %ˆ x )) ) is sall o atter what x are; i.e. look for ifp * sup x # { )} )} is achieved by Noralized Maxiu Likelihood NML) distributio $log P * x ) $ $log Px %ˆ x )) { log P * x ) log Px #ˆ x )) PNML x M ) = P x $ˆ x )) y # P y $ˆ y )) 90 MDL Model Selectio Geoetric Iterpretatio of MDL Uder regularity coditios log PNML x M ) = Select M i iiizig log PNML x M i ), i.e. k log P x #ˆi x )) + log + log det I # ) d# + o) $ # log P x %ˆi x )) + log y $ P y %ˆi y )) error=ius fit) ter coplexity ter log M ) error ter 9 Space of probability distributios #paraeters volue of odel viewed as aifold i space of all distributios The Rieaia volue easure is related to the uber of all possible distiguishable probability distributios that are idexed by the odel faily Balasubraaia, 997): d 93 Cout oly distiguishable distributios The faily of probability distributios fors a Rieaia aifold iforatio geoetry; Rao, 945; Efro, 975; Aari, 980) det I ) 95 4

5 Bayes vs. MDL Uder regularity coditios log PNML x M ) = ˆ k log P x # i x )) + log + log det I ) d + o) # # $ Uder regularity coditios log PBayes x M ) ˆ k log P x i x )) + log log w ˆ) + log det I ) d + o) # If we take Jeffrey s prior det I ) w ) = det I ) d Predictive Iterpretatio iterpret log P x) as loss icurred whe predictig usig P while actual outcoe was x Bayesia argial likelihood ca be writte as cuulative log-loss predictio error = i PBayes x ) log PBayes x ) log = i PBayes x ) i log PBayes xi x, K, xi ) = Loss xi, PBayes x )) i= 97 Philosophy what do we do whe the data geeratig echais is ot i the faily of odels M we cosider? what is prior?) MDL priors are techical i ature Jeffreys prior is uifor prior o the space of distributios with the atural etric that easures distaces betwee distributios by how distiguishable they are 98 5