Minimax Rényi Redundancy

Transcription

1 Miimax Réyi Rdudacy Smih Yagli Pricto Uivrsity Yücl Altuğ Natra, Ic Srgio Vrdú Pricto Uivrsity Abstract Th rdudacy for uivrsal losslss comprssio i Campbll s sttig is charactrizd as a miimax Réyi divrgc, which is show to b qual to th maximal α-mutual iformatio via a gralizd rdudacy-capacity thorm Spcial atttio is placd o th aalysis of th asymptotics of miimax Réyi divrgc, which is dtrmid up to a trm vaishig i bloclgth I INTRODUCTION I variabl lgth sourc codig, xpctd cod lgth is th usual cost fuctio that o aims to miimiz For discrt mmorylss sourcs, asymptotically, th miimal achivabl pr-lttr xpctd cod lgth is qual to th tropy Howvr, if P Y V is a sourc distributio with a uow paramtr ad th codig systm assums a distributio, th o ds to pay a xtra palty for th mismatch giv by D(P Y V ) + o() () I light of (), th covtioal worst-cas masur of rdudacy i uivrsal losslss comprssio is R if D(P Y V ), () th ifimizatio is ovr all distributios o Y, ad th rmum is ovr all possibl valus of th uow paramtr I this zro-sum gam, is chos by th cod dsigr, ad is chos by atur A rlatio btw R ad th maximal mutual iformatio is giv by th Rdudacy-Capacity Thorm (g, [3], ad [4]) that stats that R I(, P Y V ), (3) th rmizatio is ovr all possibl probability distributios o th paramtr spac Through (), () ad (3), w s a plasig rlatioship btw tropy, rlativ tropy ad mutual iformatio i th cotxt of losslss data comprssio Lt Y P Y V, ad ot that D(P Y V ) E ı PY V (Y ), (4) For prfix cods, () is wll ow [, Thorm 543] Morovr, th loss i rat icurrd du to th prfix coditio is ow to b asymptotically gligibl [] th rlativ iformatio btw th probability mass fuctios o th valu st A is dotd by ı P Q (a) log P (a) Q(a) with a A A much mor strigt prformac guarat tha th avrag of rlativ iformatio is its poitwis maximum I particular, if o rplacs (4) with max y ı PY V (y ), th coutrpart of () bcoms th miimax rgrt, r if max ı y Y P Y V (y ), (5) which has foud applicatios i various sttigs, g, [5]-[9] A aalogy to th Rdudacy-Capacity Thorm is giv by [6] r log P Y V (y ) (6) y Y I (, P Y V ), (7) I (P X, P Y X ) dots th α-mutual iformatio of ifiit ordr, whos dfiitio is giv i (30) Th avrag ad poitwis formulatios ar two xtrms of prformac guarats, which ar ot quit suitabl for crtai applicatios For this raso, o ss a compromis btw thos two For xampl, i th coomics litratur, avrag ad poitwis guarats ar rfrrd as ris-utral ad ris-avoidig, rspctivly, ad th otio of ris-avrsio has b itroducd to provid a compromis [0] I this papr, w itroduc th otio of ris-avrsio withi th uivrsal sourc codig cotxt ad w quatify its ffct o th fudamtal limit I th o-uivrsal sttig, i, wh th sourc distributio is ow, a classical rsult of Campbll [] itroducs such a ris-avrs cost fuctio i a discrt mmorylss sttig Spcifically, [] proposs to graliz th covtioal otio of miimizig th xpctd cod lgth with th cost fuctio L (Y ) log E [xp(l(f(y )))], (8) 3 (0, ), f dots th cod, ad l( ) dots th lgth fuctio I this cas, for a discrt mmorylss sourc Y, Campbll [] shows that th miimum pr-lttr cost asymptotically achivabl by prfix cods is giv by th Réyi Ulss othrwis statd, logarithms ad xpotials ar of arbitrary basis throughout this papr 3 Campbll s rsult is still valid wh (, 0) Howvr, such a formulatio corrspods to a ris-sig schm, which falls outsid th philosophy spousd i this papr

2 tropy H + (Y ) Notic that L (Y ) capturs th otio of ris-avrsio through th paramtr sic L (Y ) 0 E [l(f(y ))], (9) L (Y ) max y Y l(f(y )) (0) A atural way to itroduc ris-avrsio i uivrsal sourc codig is to us Campbll s formulatio ad charactriz th palty for th mismatch ai to () Idd, about forty yars aftr Campbll s wor, Sudarsa [, Thorm 8] showd that if o uss L (Y ) as th cost fuctio, th palty paid for uivrsality ca b writt as 4 D +( P + Y V Q + Y ) + o(), () D + (P Q) dots th Réyi divrgc of ordr + (dfid i (9)), ad P Y α dots th scald distributio of P Y : Th distac masur P α Y (y) PY α(y) b Y P Y α () (b) S α (P Q) D α ( P α Q α ) (3) is ow as th Sudarsa divrgc of ordr α btw P ad Q Followig [], th rlvat masur of rdudacy for uivrsal losslss comprssio udr Campbll s prformac critrio is R () if S + (P Y V ) (4) Th covtioal miimax rdudacy i () corrspods to R 0 () whil th miimax rgrt i (5) corrspods to R () Although, i gral, S α (P Q) D α (P Q), w ar abl to stablish a plasig aalog to th classical rdudacy rsults such as (), (3) ad (5), (7): R () if D + (P Y V ) (5) I + (, P Y V ) (6) Not that (5) is aalogous to () with Réyi divrgc rplacig th rlativ tropy Thus, w rfr R () as th miimax Réyi rdudacy Morovr, (6) gralizs th Rdudacy-Capacity Thorm to α-mutual iformatio thrby fidig aothr opratioal maig for th maximal α-mutual iformatio byod thos that hav b show i th litratur o rror probability bouds for data trasmissio (g [3], [4]) Morovr, th α-mutual iformatio smoothly itrpolats btw two xtrms, amly I(, P Y V ) i (3) ad I (, P Y V ) i (7) Fially, (5) ad (6), coupld with Campbll s rsult [], provid a plasig rlatioship btw Réyi tropy, Réyi divrgc ad α-mutual iformatio i th cotxt of uivrsal losslss data comprssio 4 Although it is ot cosidrd hr, Sudarsa s rsult is still valid wh (, 0) Th asymptotic bhaviors of th miimax rdudacy ad miimax rgrt hav also rcivd cosidrabl atttio i th litratur (g, [5], [6], [8], [5]-[0]) sic, i additio to comprssio, thy ar rlvat i applicatios such as machi larig, fiac, prdictio, gamblig, ad so o I particular, Xi ad Barro i thir y cotributios [5], [5] show that R R 0 () r R () π + log Γ (/) Γ(/) log + o(), (7) π + log Γ (/) + o(), (8) Γ(/) ad ar th umbr of obsrvatios ad th alphabt siz, rspctivly, ad o() vaishs as Whil Mrhav [, Thorm ] givs R () + o(), w quatify asymptotically th ffct of th risavrsio paramtr o th fudamtal limit i uivrsal sourc codig by providig a plasig itrpolatio btw (7) ad (8): R () π + log Γ (/) Γ(/) log( + ) + o() (9) II NOTATION, DEFINITIONS AND STATEMENT OF THE A Notatio ad Dfiitios RESULTS Lt Y {,,, } ad dot th ( )-dimsioal simplx of probability mass fuctios dfid o Y by For ach paramtr (,, ), w dfi our obsrvatio modl P Y V : Y such that P Y V (i) i, (0) ad th idpdt idtically distributd (iid) xtsio of this modl P Y V : Y such that P Y V (y ) P Y V (y i ) () t t, () t i j {y j i} dots th umbr of tims i Y appars i th vctor y It ca b vrifid that th dtrmiat of th Fishr iformatio matrix (i ats) of P Y V for th paramtr vctor i is 5 J(, PY V ) i (3) i A importat probability masur o is Jffrys prior [] dfid as PV () J(, P Y V ) / J(ξ, P Y V ) / dξ (4) 5 Not that th Fishr iformatio matrix is ( ) ( ) sic thr ar ( ) fr paramtrs i th modl Nvrthlss, it is otatioally covit to dot th paramtr vctor as if it wr -dimsioal

3 / / D (/,, /) (5) D (α,, α ) dots a spcial form of th Dirichlt itgrals of typ which ca b writt i trms of th Gamma fuctio: D (α,, α ) Γ(α ) Γ(α ) Γ(α + + α ) (6) Th sourc distributio w gt by assumig Jffrys prior o th paramtr spac is rfrrd as Jffrys mixtur which w dot by Q Y (y ) P Y V (y )dpv () (7) D (t + /,, t + /) (8) D (/,, /) For discrt probability masurs P ad Q o th st Y such that Q domiats P, i, P Q, Réyi divrgc of ordr 6 α btw P ad Q is dfid as D α (P Q) D(P Q), α α log E xp((α )ı P Q (Y )), α (, ) max ı P Q(b), α, b Y (9) Y P Giv (P X, P Y X ), a aalogous gralizatio ca b mad for mutual iformatio rsultig i th α- mutual iformatio [4]: I α (P X, P Y X ) I(P X, P Y X ), α if D α (P Y X P X Q Y P X ), α (, ) Q Y log E ss xp ı X;Y (X; Ȳ ), α, X (30) Ȳ P Y, idpdt of X P X, ad w hav usd th covtioal otatio for iformatio dsity ı X;Y (x; y) ı PY Xx P Y (y) B Statmt of th Rsults Thorm Gralizd Rdudacy-Capacity Thorm For ay (0, ), ad positiv itgr R () if D + (P Y V ) (3) I + (, P Y V ) (3) As w show i th proof i Sctio III, (3) is du to th fact that scalig a distributio is a o-to-o opratio that prsrvs mmorylssss whil th miimax thorm for Réyi divrgc [3, Thorm 34] is th gatway to showig th gralizd rdudacy-capacity thorm i (3) 6 W ar ot cocrd with Réyi divrgcs of ordr α (0, ) A mor gral dfiitio ca b foud i [3] Thorm Asymptotic Bhavior of Miimax Réyi Rdudacy For ay (0, ) lim R () π log Γ (/) Γ(/) log( + ) (33) W prov th covrs dirctio for Thorm i Sctio IV Omittd bcaus of th spac costraits, th achivability dirctio uss mthods ai to thos i [5] ad [5], s [4] for a complt proof of Thorm III PROOF OF THEOREM To stablish (3), for ay (0, ), dfi th bijctio f : as f (,, ) +,, +, (34) κ κ b Y P + Y V (b) (35) Th, for ay (,, ) ad y Y, th scald vrsio of th coditioal distributio (s ()) satisfis P + Y V (y ) P + Y V (y i) (36) i P Y V f ()(y i ) (37) i P Y V f ()(y ) (38) Thrfor, for ay giv distributio R Y o Y As a rsult of (40), D + ( P + Y V R Y ) D + (P Y V f ()R Y ) (39) D + (P Y V R Y ) (40) if S + (P Y Q V ) Y if if if D + ( P + Y V Q + Y ) (4) D + (P Y V Q + Y ) (4) D + PY V, (43) (43) follows bcaus vry probability masur i is a scald vrsio of aothr probability masur i To stablish (3), ot that if D + PY V

4 if if E D + PY V ( V ) E D + PY V ( V ) (44) (45) I + (, P Y V ), (46) th xpctatio i (44) is with rspct to V, ad (45) follows from [3, Thorm 34], which holds wh Y is fiit To vrify (46) (s also [3, Propositio ] ad [4, Thorm 5] i th fiit alphabt cas), ot that th α-mutual iformatio ca b writt as I + (, P Y V ) if log E E xp ı PY V (Y ) V, (47) ad ot that if E D + PY V ( V ) if if log if if log E E xp ı PY V (Y ) V E E xp ı PY V (Y ) V (48) (49) D + (P Y V ) (50) E D + (P Y V ( V )) (5) if E D + PY V ( V ), (5) (48) follows from Js s iquality, (49) follows from th fact that th maximi valu is always lss tha or qual to th miimax valu, ad (5) is agai du to [3, Thorm 34] IV PROOF FOR THE CONVERSE OF THEOREM This sctio is dvotd to th proof of i (33) for ay (0, ) Dfi M a (a,, a ) Z + : a i, (53) i M δ M a (a,, a ) Z + : for ay δ (0, ) Cosidr th followig + R () log log t M y Y δ a i i (54) + I +(, P Y V ) (55) P + + Y V (y )d () (56) t t ( t t ) + dp V () + (57) log t M δ t t ( t t ) + dp V () + (58) (55) is du to Thorm, (56) follows from [4, Thorm ] (also s [4, Appdix A]), (57) is du to th suboptimal choic of Jffrys prior, ad (58) follows bcaus M δ M Usig Robbis sharpig [5] of Stirlig s approximatio, ad th fact that t M δ, o ca show that t t H( P y ) (π) i t i (+) δ, (59) th tropy is i ats ad P y dots th mpirical distributio of th vctor y With th aid of (5) ad (6) w ca xprss th itgral i th right sid of (58) as t t + dp V () i Γ(( + )t (60) i + /) D (/,, /) Γ(( + ) + /) Th gamma fuctio gralizatio of Stirlig s approximatio (show to b valid for positiv ral umbrs by Whittar ad Watso [6]) yilds Γ(x) πx x / x ( + r), x > 0, (6) r /(x) I particular, for i,,, Γ(( + )t i + /) π(( + )t i + /) (+)ti (+)ti / ( + r i ), (6) Γ(( + ) + /) π(( + ) + /) (+)+( )/ (+) / ( + r 0 ), (63) r i xp, (64) ( + )t i + 6 r 0 xp (65) ( + ) + 6 It follows from (6) ad (63) that i Γ(( + )t i + /) (+)H( P y ) (π) Γ(( + ) + /) ( + ) (+)ti i + (+)t i ( + ri ) (+)+ + (+) ( + r 0 ) Combiig (60) ad (66), w ca writ t t + dp V () (π) D (/,, /) (+)H( P y ) ( + ) (66)

5 (+)ti i + (+)t i ( + ri ) (+)+ + (+) ( + r 0 ) (π) D (/,, /) + (+)δ + (+) (+)H( P y ) ( + ) (+)δ (+)+ (67) (+)δ+6 (+)+6, (68) (68) is du to th dfiitio of M δ, (54), th fact that for ay positiv costat c, ( + c/x) x is a mooto icrasig fuctio of x, ad th fact that th rror trms (s (64) ad (65)) satisfy i ( + r i) (+)δ+6 (69) + r 0 (+)+6 Uitig th lowr bouds i (58), (59) ad (68), R () log π + log(β(, δ, )) log(d (/,, /)) (70) log( + ) + + log((, δ,, )), β(, δ, ) /, (7) tj t M δ j (, δ,, ) (+) Notic that δ ( + (+)δ + (+) δ + (+) (+)δ+6 ) (+)+6 + (7) lim (, δ,, ), for ay δ (0, ), (73) lim lim β(, δ, ) τ / δ 0 τ / dτ (74) D (/,, /), (75) (73) follows aftr oticig that ach factor of (, δ, ) gos to, ad (74) follows from th dfiitio of th Rima itgral Assmblig (70), (73) ad (75), w obtai i (33) ACKNOWLEDGEMENT This wor has b portd by ARO-MURI cotract umbr W9NF ad i part by th Ctr for Scic of Iformatio, a NSF Scic ad Tchology Ctr udr Grat CCF REFERENCES [] T M Covr ad J A Thomas, Elmts of Iformatio Thory, d d Hobo, NJ: Wily, 006 [] I Kotoyiais ad S Vrdú, Optimal losslss data comprssio: o-asymptotics ad asymptotics, IEEE Trasactios o Iformatio Thory, vol 60, o, pp , Fb 04 [3] R G Gallagr, Sourc codig with sid iformatio ad uivrsal codig, Sptmbr 976, upublishd mauscript Availabl from: [4] B Y Ryabo, Codig of a sourc with uow but ordrd probabilitis, Problms of Iformatio Trasmissio, vol 5, o, pp 34 38, Oct 979 [5] Q Xi ad A R Barro, Asymptotic miimax rgrt for data comprssio, gamblig, ad prdictio, IEEE Trasactios o Iformatio Thory, vol 46, o, pp , Mar 000 [6] Y Shtarov, Uivrsal squtial codig of sigl mssags, Problmy Prdachi Iformatsii, vol 3, o 3, pp 3 7, 987 [7] J Forstr ad M K Warmuth, Rlativ xpctd istataous loss bouds, Joural of Computr ad Systm Scics, vol 64, o, pp 76 0, Fb 00 [8] M Drmota, ad W Szpaowsi, Prcis miimax rdudacy ad rgrt, IEEE Trasactios o Iformatio Thory, vol 50, o, pp , Nov 004 [9] F Liag ad A R Barro, Exact miimax stratgis for prdictiv dsity stimatio, data comprssio, ad modl slctio, IEEE Trasactios o Iformatio Thory, vol 50, o, pp , Nov 004 [0] J W Pratt, Ris avrsio i th small ad i th larg, Ecoomtrica, vol 3, o -, pp 36, Ja-Apr 964 [] L L Campbll, A codig thorm ad Réyi s tropy, Iformatio ad Cotrol, vol 8, o 4, pp 43 49, Aug 965 [] R Sudarsa, Gussig udr sourc ucrtaity, IEEE Trasactios o Iformatio Thory, vol 53, o, pp 69 87, Ja 007 [3] I Csiszár, Gralizd cutoff rats ad Réyi s iformatio masurs, IEEE Trasactios o Iformatio Thory, vol 4, o, pp 6 34, Ja 995 [4] S Vrdú, α-mutual iformatio, i 05 Iformatio Thory ad Applicatios Worshop, Sa Digo, pp 6, 05 [5] Q Xi ad A R Barro, Miimax rdudacy for th class of mmorylss sourcs, IEEE Trasactios o Iformatio Thory, vol 43, o, pp , Mar 997 [6] L D Davisso, R J McElic, M B Pursly, ad M S Wallac, Efficit uivrsal oislss sourc cods, IEEE Trasactios o Iformatio Thory, vol 7, o 3, pp 69 79, May 98 [7] L Györfi, I Páli, ad E C va dr Mul, Thr is o uivrsal sourc cod for a ifiit sourc alphabt, IEEE Trasactios o Iformatio Thory, vol 40, o, pp 67-7, Ja 994 [8] R E Krichvsy, ad V K Trofimov, Th prformac of uivrsal codig, IEEE Trasactios o Iformatio Thory, vol 7, o, pp 99 07, Mar 98 [9] J Rissa, Uivrsal codig, iformatio, prdictio ad stimatio, IEEE Trasactios o Iformatio Thory, vol 30, o 4, pp , Jul 984 [0] J Rissa, Stochastic complxity ad modlig, Aals of Statistics, vol 4, o 3, pp , Sp 986 [] N Mrhav, O optimum stratgis for miimizig th xpotial momts of a loss fuctio, Commuicatios i Iformatio ad Systms, vol, o 4, pp , 0 [] H Jffrys, A ivariat form for th prior probability i stimatio problms, Procdigs of th Royal Socity of Lodo Sris A: Mathmatical ad Physical Scics, vol 86, o 007, pp , Sp 946 [3] T va Erv ad P Harrmoës, Réyi divrgc ad Kullbac-Liblr divrgc, IEEE Trasactios o Iformatio Thory, vol 60, o 7, pp , Jul 04 [4] S Yagli, Y Altuğ, ad S Vrdú, Miimax Réyi rdudacy, [Oli] Availabl: [5] H Robbis, A rmar o Stirlig s formula, Amrica Mathmatical Mothly, vol 6, o, pp 6 9, Ja 955 [6] E T Whittar ad G N Watso, A Cours of Modr Aalysis, 4th d Cambridg, UK: Cambridg Uiv Prss, 963