Why study large deviations? The problem of estimating buer overow frequency The performance of many systems is limited by events which have a small pr

Transcription

1 Why study lrge devitios? The problem of estimtig buer overow frequecy The performce of my systems is ited by evets which hve smll probbility of occurrig, but which hve severe cosequeces whe they occur. The theory dels with rre evets, d is symptotic i ture. It c be viewed s reemet of the lw of lrge umbers. The gure below shows 2 2 switch, where output liks re served t rte c. c B It is useful whe simultio or umericl techiques become icresigly dicult s prmeter teds to its it. It hs my pplictios: queueig d commuictios models, iformtio theory, simultio techiques, prmeter estimtio, hypothesis testig, : : : I order to kow how my virtul circuits my be llowed to use this output lik, for give Qulity of Service costrit, we eed to estimte the probbility tht the cotet of the queue, Q t, eceeds the buer of size B. (Q t B) should be smll. These slides were writte to support iforml discussio of Chpters d 2 of \Lrge Devitios for erformce Alysis", by Shwrtz d Weiss. Richrd Weber, 2 October The overow probbility i MMB queue Elemets of lrge devitio theory Here is other result of lrge devitio theory. Simply to illustrte ides cosider sigle server MMB queue, with ite buer, here beig shred by two trc sources, with combied oisso rrivls t rte We kow Hece B (Q t B) where mes This is typicl. (c) (c) B : (c) B+ (Q t B) e B log(c) for lrge B; B B log (Q t B) log(c): 3 c Suppose ; 2 ; : : : re i.i.d. r.v.s the i i 2 [; b] We hd for the queue: e [if 2[;b] `()+o()] (Q t B) e B log(c) for lrge B: These re typicl. The geerl coclusios re: The symptotic frequecy of occurece of rre evets depeds i epoetil mer o some prmeters of the problem. E.g.,, B. If rre evets occurs the it occurs i the most likely wy. E.g., if 2[;b] `(). Rre evets occur s oisso process. 4

2 Chero's theorem (upper boud) Suppose ; 2 ; : : : is sequece of i.i.d. rdom vribles d E. Let S + +. The for ll >, Hece (S ) E [ + + ] E e [ ++ ] e [ log Ee ] (S ) e sup [ log Ee ] Note tht by Jese's iequlity tht for ll, Ee e E d hece log Ee ( E ). Thus d we coclude Note `(E ). `() def sup sup log Ee log Ee (S ) e `() 5 Observtios Note the key role of momet geertig fuctio, M() Ee d logrithmic momet geertig fuctio, log M() (lso clled the cumult geertig fuctio.) log M() is cove fuctio of. `() : sup [ log M()] is clled the Legedre trsform of log M(). `() is cove fuctio of. `() d log M() re Legedre trsform duls, i.e., sup[ `()] sup[ sup( log M())] sup if[log M() ( )] if log M() : log M() The optimizig, sy, stises M ( )M( ): 6 A typicl rte fuctio Suppose i ; with probbilities q, p. The d `() < : log log M() log(q + pe ); p + ( ) log p ; ; otherwise: : :6 :4 :2 Here E p :6. `() is cove. `() :2 :4 :6 : j`()j s boudry of the set where `() is ite. `(E ). 7 Chero's theorem (lower boud) Suppose F is the distributio of d dee y G(y) M( ) e df () where is s bove. The G is distributio. It is clled tilted distributio. Note tht if ~ G, y E(~) M( ) e df () M ( ) M( ) : Now dg(y) M( ) e y df (y), so (S ) M( ) M( ) y ++y y ++y +y ++y e [ (+) log M ( )] df (y ) : : : df (y ) e (y ++y ) dg(y ) : : : dg(y ) e (y ++y ) dg(y ) : : : dg(y ) +y ++y dg(y ) : : : dg(y ) e `() ( + ~ + + ~ ) e `() ~ + + ~ p

3 Chero's theorem (lower boud), cotiued (S ) e `() Now ~ + + ~ p ~ + + ~ p 2 So sice is rbitrrily smll, if log (S ) `() The upper d lower bouds together imply (S ) e [`()+o()] We eed coditios to esure tht log M() is dieretible t d tht its derivtive is. It is eough to ssume tht M() is ite i some eighborhood of d tht there is i the iterior of this eighborhood such tht `() log M( ). 9 Illustrtio with the orml distributio I the simple cse tht i N(; 2 ), d log M() `() ( ) A more reed estimte c be obtied from y + y e 2 y2 (S ) y e 2 t2 dt y e 2 y2 ) p 2( ) e ( )2 2 2 p 2( ) e `() The pperce of p ( e 2 log ) is typicl. A pplictio of the theory would be to pproimte (S ) by e ( ) Sometimes oe c get reed pproimtios, e.g., s bove, or the Bhdur-Ro pproimtio for the biomil distributio. Geerliztio to i.i.d. vectors Theorem.22. Suppose ; 2 ; : : : 2 R d is sequece of rdom vectors d M() Ee <; > : Dee the rte fuctio `() sup[< ; > log M()]: The for y set C R d log log t t t 2 C t 2 C if 2C o `(); if 2 C `(); where C o d C re respectively the iterior d closure of Note: If you go bck to the proof of Chero's theorem, you will see tht you c esily eted the proof to sttemets bout (S 2 C). You c tke C closed set whe doig the upper boud, but will eed to tke C to be ope set for the lower boud. (You'll wt to let be the miimizer of `() d boud the probbility of beig i C by the probbility of beig i eighbouhood of ; so you'll eed tht if 2 C the eighborhood of is lso i ) Geerl sttemet of lrge devitio priciple Suppose z ; z 2 ; : : : is sequece of rdom vectors i probbility spce (X ; ; F). Here X might be R d, or perhps C[; T ], the spce of cotiuous fuctios. E.g., thik of z ( + + ). Deitio 2.. A rel vlued fuctio I o X is clled \rte fuctio" if (i) I(), (ii) I is lower semi-cotiuous; i.e., if y ; y 2 ; : : : is sequece such tht y y i X the if I(y ) I(y). Deitio 2.2. We sy z ; z 2 ; : : : stisfy lrge devitio priciple with rte fuctio I if for every set C X log (z 2 C) if I(); 2C o log (z 2 C) if I(); 2C where C o d C re respectively the iterior d closure of If if 2C o I() if 2 C the the two bouds coicide d C is sid to be I-cotiuity set for I. 2

4 Vrdh's lemm The cotrctio priciple Theorem 2.2. Suppose tht z ; z 2 ; : : : stisfy lrge devitio priciple with rte fuctio I. The for y bouded cotiuous fuctio g o X, log E e g(z ) The ituitive ide is tht E e g(z ) sup[g() I()]: e g() (z )d e g() e I() d e sup [g() I()] where the lst lie follows from Lplce's rgumet, tht the rte of growth of itegrl (or sum) is obtied by pproimtig it by its lrgest term. E.g., for lrge. 4e 2 + 6e 3 + e 4e 2 Deitio 2.. A rte fuctio is sid to be good rte fuctio if (iii) The set f : I() g is compct for every. Suppose tht z ; z 2 ; : : : stisfy lrge devitio priciple with rte fuctio I. Let f be cotiuous fuctio d let y i f(z i ). Dee I (y) Theorem 2.3. < : iffi() : 2 X ; f() yg ; if y f() for o 2 X (i) If I is good rte fuctio the I is good rte fuctio. (ii) If z ; z 2 ; : : : stisfy lrge devitio priciple with good rte fuctio I the y ; y 2 ; : : : stisfy lrge devitio priciple with good rte fuctio I. Agi, Lplce's rgumet gives the right ituitio why this is true. 3 4 Sov's theorem Deitio A.2. Let ; 2 ; : : : be sequece rdom vribles with distributio F d vlues i some metric spce X. The empiricl distributio of mesurble set A is (A) : i [ i 2 A]: Suppose ; 2 ; : : : re i.i.d. with distributio, i.e., ( y) (( ; y]). Dee I() log d d (y) d(y): I the cse tht of discrete distributio (p ; : : : ; p d ), over discrete set of d poits this would be I(q) q j log j H(q j p): p j Theorem.22. Cosider the sequece ; 2 ; : : :. For every set C cotied i the spce of probbility distributios, log ( 2 C) if I(); 2C o log ( 2 C) if I(); 2C where C o d C re respectively the iterior d closure of 5 Sov's theorem for discrete distributio Suppose X f; : : : ; dg. Give i j, let y i 2 f; g d be vector whose jth compoet is equl to d ll others re equl to. For y z 2 R d, let jzj m jd jz j j. The i y i q if q:jq j q j log A : p j Emple. Suppose we roll die times d the totl is 4. The epected vlue is 3:5. So we hve see rre evet. How did this hppe? For q (:3; :23; :46; :74; :27; :247), we i y i q if q:jq ; j jq j 4 if q: j jq j >4 j j j j j 4A. q j log 6 q j log 6 A A where q is chose s imizer of the deomitor. 6

5 Iformtio theory d lrge devitios Suppose source geertes letters from lphbet of d symbols. Letters re i.i.d. choices mogst the d symbols, with probbilities q ; : : : ; q d. The empiricl distributio of the symbols i strig of symbols will be close to q, so without losig much iformtio, we could igore strigs for which the empiricl distributio is fr from q. There re d possible strigs of legth, but we would be usig oly frctio of these. The umber we would be usig, sy M, is give by m d j where h(q) j q j log 2 q j. Hece 2 q j log A h(q) d d ; m 2 h(q) 2 log 2 d : This shows tht the source hs iformtio rte h(q) log 2 d. 7