Analysis of Remaining Uncertainties and Exponents under Various Conditional Rényi Entropies

Transcription

1 nlyi of Rmining Uncrini nd Exponn undr Vriou Condiionl Rényi Enropi Vincn Y F Tn, Snior mbr, IEEE, nd hio Hyhi, Snior mbr, IEEE rxiv: v cit] 3 y 206 brc In hi ppr, w nlyz h ympoic of h normlizd rmining uncriny of ourc whn comprd or hhd vrion of i nd corrld id-informion i obrvd For hi ym, commonly known Slpin-Wolf ourc coding, w blih h opiml minimum) r of comprion of h ourc o nur h h rmining uncrini vnih W lo udy h xponnil r of dcy of h rmining uncriny o zro whn h r i bov h opiml r of comprion In our udy, w conidr vriou cl of rndom univrl hh funcion Ind of muring rmining uncrini uing rdiionl Shnnon informion mur, w do o uing wo form of h condiionl Rényi nropy mong ohr chniqu, w mploy nw on-ho bound nd h momn of yp cl numror mhod for h vluion W how h h ympoic rul r gnrlizion of h rong convr xponn nd h rror xponn of h Slpin-Wolf problm undr mximum poriori P) dcoding Indx Trm Rmining uncriny, Condiionl Rényi nropi, Rényi divrgnc, Error xponn, Srong convr xponn, Slpin- Wolf coding, Univrl hh funcion, Informion-horic curiy, omn of yp cl numror mhod I INTRODUCTION In informion-horic curiy ], 2], i i of fundmnl impornc o udy h rmining uncriny of rndom vribl n givn comprd vrion of ilf f n ) nd nohr corrld ignl E n Thi modl, rminicn of h h Slpin-Wolf ourc coding problm 3], i illurd in Fig modl omwh imilr o h on w udy hr w udid by Tndon, Uluku nd Rmchndrn 4] who nlyzd h problm of cur ourc coding wih hlpr In priculr, pry would lik o rconruc ourc n givn hlpr ignl or comprd vrion of i) bu n vdroppr, who cn p on f n ) i lo prn in h ym Th uhor in 4] nlyzd h rdoff bwn h comprion r nd h quivocion of n givn f n ) Villrd nd Pinnid 5] nd Bro 6] conidrd h ing in which h vdroppr lo h cc o mmoryl id-informion E n h i corrld wih n Howvr, hr r mny wy h on could mur h quivocion or rmining uncriny Th rdiionl wy, ring from Wynr minl ppr on h wirp chnnl 7] nd lo in ], 2], 4] 6]), i o do o uing h condiionl Shnnon nropy H n f n ), E n ), lding o ndrd quivocion mur In hi ppr, w udy h ympoic of rmining uncrini bd on h fmily of Rényi informion mur 8] Th mur w conidr includ h condiionl Rényi nropy H n f n ), E n ) nd i o-clld Gllgr form, which w dno H n f n ), E n ) W no h unlik h condiionl Shnnon nropy, hr i no univrlly ccpd dfiniion for h condiionl Rényi nropy, o w dfin h qunii h w udy crfully in Scion II- Exniv dicuion of vriou pluibl noion of h condiionl Rényi nropy r providd in h rcn work by Tixir, o nd nun 9] nd Fhr nd Brn 0] W moiv our udy by fir howing h h limi of h normlizd) rmining uncriny n H n f n ), E n ) nd h xponn of h rmining uncriny n log H n f n ), E n ) for pproprily chon Rényi prmr ) r, rpcivly, gnrlizion of h rong convr xponn nd h rror xponn for dcoding n givn f n ), E n ) Rcll h h rong convr xponn ], 2] i h xponnil r which h probbiliy of corrc dcoding nd o zro whn on opr r blow h fir-ordr coding r, i, h condiionl Shnnon nropy H E) In conr, h rror xponn 3] 6] i h xponnil r which h probbiliy of incorrc dcoding nd o zro whn on opr r bov H E) Thu, udying h ympoic of h condiionl Rényi nropy llow u no only o undrnding h rmining uncriny for vriou cl of hh funcion 7], 8] bu lo llow u o provid ddiionl informion nd inuiion concrning h rong convr xponn nd h rror xponn for Slpin-Wolf coding 3] W lo moiv our udy by conidring cnrio in informion-horic curiy whr h hh funcion w udy ppr nurlly, nd coding cn b don in compuionlly fficin mnnr Th prn work cn b rgrdd follow-on from h uhor prviou work in 9] on h ympoic of h quivocion whr w udid h bhvior of C := nr H f n ) E n ) Vincn Y F Tn i wih h Dprmn of Elcricl nd Compur Enginring nd h Dprmn of hmic, Nionl Univriy of Singpor Emil: vn@nudug) hio Hyhi i wih h Grdu School of hmic, Ngoy Univriy, nd h Cnr for Qunum Tchnologi CQT), Nionl Univriy of Singpor Emil: mhio@mhngoy-ucjp) Thi ppr w prnd in pr h 206 Inrnionl Sympoium on Informion Thory ISIT) in Brclon, Spin In hi ppr, w bu rminology nd u h rm Slpin-Wolf coding 3] nd lol ourc coding wih dcodr id-informion inrchngbly

2 2 n f n f n n ) g n  H ± n f n n ), E n ) H ± n f n n ), E n ) E n Fig Th Slpin-Wolf 3] ourc coding problm W r inring in qunifying h ympoic bhvior of h rmining uncriny of n givn f n n ), E n ) murd ccording o h condiionl Rényi nropi H ± nd H ± dfind in 0) nd 2) nd C := nr H fn ) E n ) whr R = n log f i h r of h crdinliy of h rng of f) In 9], w lo udid h xponn nd cond-ordr ympoic of h quivocion Howvr, w no h bcu w conidr h rmining uncriny ind of h quivocion, vrl novl chniqu, including nw on-ho bound nd lrg-dviion chniqu, hv o b dvlopd o ingl-lriz vriou xprion Ppr Orgnizion: Thi ppr i orgnizd follow In Scion II, w rcp h dfiniion of ndrd Shnnon informion mur nd om l common Rényi informion mur 9], 0] W lo inroduc om nw qunii nd rlvn propri of h informion mur W om noion concrning h mhod of yp 6] In Scion III, w furhr moiv our udy by rling h qunii w wih o chrcriz o h rror xponn nd rong convr xponn of Slpin-Wolf coding Propoiion ) In Scion IV, w dfin vriou imporn cl of hh funcion 7], 8] uch univrl 2 nd rong univrl hh funcion) nd furhr moiv h udy of h qunii of inr by dicuing fficin implmnion of univrl 2 hh funcion vi circuln mric 20] Th finl pr of Scion IV conin our min rul concrning h ympoic of h normlizd rmining uncrini Thorm 2), h opiml r of comprion of h min ourc o nur h h rmining uncrini vnih Thorm 3), nd h xponn of h rmining uncrini Thorm 4) W how h h opiml r r igh in crin rng of h Rényi prmr For h vluion, w mk u of vrl novl on-ho bound, lrg-dviion chniqu wll h momn of yp cl numror mhod 2] 25] Thorm 2, 3 nd 4 r provd in Scion V, VI nd VII rpcivly W conclud our dicuion nd ugg furhr vnu for rrch in Scion VIII Som chnicl rul g, on-ho bound, concnrion inqulii) r rlgd o h ppndic Bic Shnnon nd Rényi Informion Qunii II INFORTION ESURES ND OTHER PRELIINRIES W now inroduc om informion mur h gnrliz Shnnon informion mur Fix normlizd diribuion P P) nd ub-diribuion non-ngiv vcor bu no ncrily umming o on) Q P) uppord on fini Thn h rliv nropy nd h Rényi divrgnc of ordr + r rpcivly dfind DP Q ) := P ) log P ) Q ) D P Q ) := log P ) Q ), 2) ) whr hroughou, log i o h nurl b I i known h lim 0 D P Q ) = DP Q ) o pcil limiing) c of h Rényi divrgnc i h uul rliv nropy I i lo known h h mp D P Q ) i concv in R nd hnc D P Q ) i monooniclly incring for R Furhrmor, h following d procing or informion procing inqulii for Rényi divrgnc hold for, ], DP W Q W ) DP Q ) 3) D P W Q W ) D P Q ) 4) Hr W : B i ny ochic mrix chnnl) nd P W b) := W b )P ) i h oupu diribuion inducd by W nd P W lo inroduc condiionl nropi on h produc lphb E bd on h divrgnc bov L I ) = for ch If P E i diribuion on E, h condiionl nropy, h condiionl Rényi nropy of ordr + nd h min-nropy rliv o nohr normlizd diribuion Q E on E H E P E Q E ) := DP E I Q E ), 5) H E P E Q E ) := D P E I Q E ), 6) P E, ) H min E P E Q E ) = log mx,):q E )>0 Q E ) 7)

3 3 I i known h lim 0 H E P E Q E ) = H E P E Q E ) nd lim H E P E Q E ) = H E P E Q E ) = H min E P E Q E ) 8) If Q E = P E, w implify h bov noion nd dno h condiionl nropy, h condiionl Rényi nropy of ordr + nd h min-nropy H E P E ) := H E P E P E ) = P E ) P E ) log P E ), 9) H E P E ) := H E P E P E ) = log P E ) P E ), 0) H min E P E ) := H min E P E P E ) = log mx P E ) ),):P E )>0 Th mp H E P E ) i concv, nd H E P E Q E ) i monooniclly dcring for R \ 0} Th dfiniion of h condiionl Rényi nropy in 0) i du o Hyhi 26, Scion II] nd Škorić l 27, Dfiniion 7] W r lo inrd in h o-clld Gllgr form of h condiionl Rényi nropy nd h min-nropy for join diribuion P E P E): H E P E) := + log P E ) P E ) ) H min E P E) := H E P E ) = log P E ) mx P E ) 3) By dfining h fmilir Gllgr funcion 3], 4] prmrizd lighly diffrnly) φ ) ) E P E := log P E ) P E ) 4) w cn xpr 2) H E P E) = + φ E P E ), 5) + hu looly) juifying h nomnclur Gllgr form of h condiionl Rényi nropy in 2) No h H nd H r rpcivly dnod H 4 nd H in h ppr by Fhr nd Brn 0] Th Gllgr form of h condiionl Rényi nropy, lo commonly known rimoo condiionl Rényi nropy 28], w hown in 0] o ify wo nurl propri for, nmly, monooniciy undr condiioning or imply monooniciy) nd h chin rul H B, E P BE) H E P E), 6) H B, E P BE) H E P E) log B 7) Th monooniciy propry of H w lo hown oprionlly by Bun nd Lpidoh in h conx of lol ourc coding wih li nd id-informion 29] nd ncoding k wih id-informion 30] W xploi h propri in h qul Th qunii H nd H cn b hown o b rld follow 0, Thorm 4] mx H E P E Q E ) = H E P E) 8) Q E PE) for, ) \ 0} Th mximum on h lf-hnd-id i ind for h ild diribuion Q E ) = P E, ) ) P 9) E, ) ) Th mp H E P E) i concv nd h mp H E P E) i monooniclly dcring for, ) I cn b hown by L Hôpil rul h lim 0 H E P E) = H E P E ) Thu, w rgrd H E P E) H E P E ), i, whn = 0, h condiionl Rényi nropy nd i Gllgr form coincid nd r qul o h condiionl Shnnon nropy W lo find i uful o conidr wo-prmr fmily of h condiionl Rényi nropy 2 H + E P E ) := + log ) P E ) P E ) ã 2) P E ã ) + ) + 20) 2 Thi nw informion-horic quniy i omwh rld o H +θ,+θ in h work by Hyhi nd Wnb 3, Eq 4)-5)] bu i diffrn nd no o b confud wih H +θ,+θ

4 4 Clrly H E P E ) = H E P E), o h wo-prmr condiionl Rényi nropy i gnrlizion of h Gllgr form of h condiionl Rényi nropy in 2) For fuur rfrnc, givn join ourc P E, dfin h criicl r ˆR := d d H + E P E ), nd, 2) = ˆR := d d H + E P E ) 22) = B Noion for Typ Th proof of our rul lvrg on h mhod of yp 6, Ch 2], o w ummriz om rlvn noion hr Th of ll diribuion probbiliy m funcion) on fini i dnod P) Th yp or mpiricl diribuion of qunc n i h diribuion Q) = n n i= i = }, Th of ll qunc n wih yp Q P) i h yp cl nd i dnod T Q n Th of ll n-yp yp formd from lngh-n qunc) on lphb i n dnod P n ) Whn w wri n b n, w mn h inquliy on n xponnil cl, i, lim n n log b n 0 Th noion nd = r dfind nlogouly Throughou, w will u h fc h h numbr of yp P n ) n+) = III OTIVTION FOR STUDYING REINING UNCERTINTIES mniond in h inroducion, in hi ppr, w udy h rmining uncriny nd i r of xponnil dcy murd uing vriou Rényi informion mur In hi cion, w furhr moiv h rlvnc of hi udy by rling h rmining uncriny o h rong convr xponn for dcoding =, 2,, n ) n givn id informion =, 2,, n ) E n nd h comprd vrion of, nmly m = f) Slpin-Wolf problm) W lo rl h xponnil r of dcy of h rmining uncriny for ourc coding r bov h fir-ordr fundmnl limi o h rror xponn of h Slpin-Wolf problm Rlion o h Srong Convr Exponn for Slpin-Wolf Coding Conidr h Slpin-Wolf ourc coding problm hown in Fig For givn funcion ncodr) f n : n n nd id informion vcor E n, w my dfin h mximum -poriori P) dcodr g fn : n E n n follow: g fn m, ) := rg mx PE, n ) = rg mx P E n ) 23) n :f n)=m n :f n)=m Dfin h probbiliy of corrcly dcoding givn h ncodr f n nd h P dcodr g fn follow: P n) c f n ) := PE, n ) 24) Thn, by h dfiniion of H in 3), w immdily h :=g fn f n),) log Pn) c f n ) = n n H n f n n ), E n PE) n 25) Whn opimizd ovr f n } n=, h quniy on h lf of 25) or i limi) i clld h rong convr xponn i chrcriz h opiml xponnil r which h probbiliy of corrc dcoding h ru ourc givn f n ), ) dcy o zro Thu, by udying h ympoic of n H for ll 0, ) nd, in priculr, h limiing c of which w do in 53) in Pr 2) of Thorm 2), w obin gnrlizion of h rong convr xponn for h Slpin-Wolf problm In fc, i i known h lim n n log Pn) c f n ) > 0 for ny qunc of ncodr f n } n= if nd only if h r lim n n log f n < H E P E ) 2, Thorm 2] Thi fc will b uilizd in h proof of Thorm 3 B Rlion o h Error Exponn for Slpin-Wolf Coding Similrly, w my dfin h probbiliy of incorrcly dcoding givn h ncodr f n nd P dcodr g fn follow: P n) f n ) := PE, n ) 26) : g fn f n),) Thn w hv h following propoiion concrning h xponn of P n) f n ) Propoiion um h P n) f n ) nd o zro xponnilly f for givn qunc of hh funcion f n } n=, i, lim n n log Pn) f n ) > 0 h xinc of h limi i pr of h umpion) Thn for ny, w hv lim n n log Pn) f n ) = lim n n log H n f n n ), E n PE) n 27) = lim n n log H n f n n ), E n P n E) 28)

5 5 W rcll, by h Slpin-Wolf horm 3], h hr xi qunc of ncodr f n } n= uch h P n) f n ) nd o zro if nd only if lim n n log f n H E P E ) Whn opimizd ovr f n } n=, h quniy on h lf of 27) i clld h opiml rror xponn nd i chrcriz h opiml xponnil r which h rror probbiliy of dcoding givn f n ), ) dcy o zro Thu, Propoiion y h h xponn of H nd H for r gnrlizion of h rror xponn of dcoding n givn f n n ), E n ) W blih bound on h limi for crin cl of hh funcion in Pr 2) of Thorm 4 Proof: W fir conidr h Gllgr form of h condiionl Rényi nropy H For brviy, w l f = f n uppring h dpndnc on n) nd w lo dfin h probbiliy diribuion P := P f n ),E n nd Q := P n f n ),En Rcll h dfiniion of h P dcodr g f m, ) in 23) W hv H n f n ),E n P n E ) =,m P m, ),m Q m, ) ) 29) P m, ) Qg f m, ) m, ) ) 30) =,m P m, )Qg f m, ) m, ) 3) = P n) f) 32) In h following chin of inqulii, w will mploy Tylor horm wih h Lgrng form of h rmindr for h funcion + ) = 0, i, + ) = + + ) + + ) + ξ) 2 33) 2 for om ξ, 0] W choo o b Qg f m, ) m, ) in our pplicion in 36) o follow L ξm, ) b gnric lmn of Qg f m, ) m, ), 0], 0] king h rol of ξ in h Tylor ri xpnion in 33) W bound h condiionl Rényi nropy follow: Hn f n ),E n P n E ) = P m, ),m =,m P m, ) =,m P m, ) Q m, ) 34) Qg f m, ) m, ) + : g f m,) + + ) Qg f m, ) m, ) ] Q m, ) } + ) ) + + ξm, ) Qgf m, ) m, ) ] P m, ) + + ) Qg f m, ) m, ) ],m + ) + Qgf m, ) m, ) ] : g f m,) = + P m, ) + ) + ) Q m, ) +,m =,m P m, ) : g f m,) = P n) + ) f) + 2 : g f m,) Q m, ) + P m, ),m + ) 2 : g f m,) Q m, ),m : g f m,) } + ) 2 P m, ) Q m, ) } : g f m,) : g f m,) ] } 2 Q m, ) 35) 36) 37) 38) Q m, )] 2 39) Q m, )] 2 40) In 37), noing h 0, w uniformly uppr boundd + ξm, ) ) by W lo uppr boundd Q m, ) by Q m, ) In 40), w ud h dfiniion of P n) f) d in 26) Bcu P n) f) i umd o dcy xponnilly f,

6 6 w hv P n) f) = log P n) f) ) 4) H n f n ), E n PE) n 42) H n f n ), E n PE) n 43) log P n) + ) ] } 2 f) + P m, ) Q m, ) 44) 2,m : g f m,) = P n) + ) 2 f) P m, ) Q m, )], 45) 2,m : g f m,) whr 4) nd 45) follow from log ) = + O 2 ), 42) u 32), 43) u h fc h H H cf 8)) nd 44) u 40) Th cond rm in 45) i xponnilly mllr hn P n) f) bcu of h qur oprion nd h fc h : g f m,) Q m, ) < Now, inc i conn, h xponn of h qunii on h lf nd righ id of h bov chin r qul Thu hy r qul o h xponn of H n f n ), E n PE n ) nd H n f n ), E n PE n ) for vry Thi compl h proof of Propoiion IV IN RESULTS: SYPTOTICS OF THE REINING UNCERTINTIES In hi cion w prn our rul concrning h ympoic bhvior of h rmining uncrini nd i xponnil bhvior mniond in Scion III, h formr i gnrlizion of h rong convr xponn for h Slpin-Wolf problm 3], whil h lr i gnrlizion of h rror xponn for h m problm Bfor doing o, w dfin vriou cl of rndom hh funcion nd furhr moiv our nlyi uing n xmpl from informion-horic curiy Dfiniion of Vriou Cl of Hh Funcion W now dfin vriou cl of hh funcion W r by ing ligh gnrlizion of h cnonicl dfiniion of univrl 2 hh funcion by Crr nd Wgmn 7] Dfiniion rndom 3 hh funcion f X i ochic mp from o :=,, }, whr X dno rndom vribl dcribing i ochic bhvior Th of ll rndom hh funcion mpping from o i dnod R = R, ) hh funcion f X i clld n ɛ-lmo univrl 2 hh funcion if i ifi h following condiion: For ny diinc, 2, Pr f X ) = f X 2 ) ) ɛ 46) Whn ɛ = in 46), w imply y h f X i univrl 2 hh funcion 7] W dno h of univrl 2 hh funcion mpping from o by U 2 = U 2, ) Th following dfiniion i du o Wgmn nd Crr 8] Dfiniion 2 rndom hh funcion f X :,, } i clld rongly univrl whn h rndom vribl f X ) : } r indpndn nd ubjc o uniform diribuion, i, Pr f X ) = m ) = 47) for ll m,, } If f X i rongly univrl hh funcion, w mphiz hi fc by wriing f X n xmpl, if f X indpndnly nd uniformly ign ch lmn of ino on of bin indxd by m i, h fmilir rndom binning proc inroducd by Covr in h conx of Slpin-Wolf coding 32]), hn 47) hold, yilding rongly univrl hh funcion Th hirrchy of hh funcion i hown in Fig 2 univrl 2 hh funcion f cn b implmnd fficinly vi circuln pcil c of Topliz) mric Th complxiy i low pplying f o n m-bi ring rquir Om log m) oprion gnrlly For dil, h dicuion in Hyhi nd Turumru 20] nd h ubcion o follow So, i i nurl o um h h ncoding funcion f w nlyz in hi ppr r univrl 2 hh funcion B nohr oivion for nlyzing Rmining Uncrini To nur ronbl lvl of curiy in prcic, w ofn nd our mg vi mulipl ph in nwork um h lic wn o nd n m-bi mg o Bob vi l N ph, nd h Ev h cc o id-informion E 3 For brviy, w will omim omi h qulifir rndom I i undrood, hncforh, h ll o-mniond hh funcion r rndom hh funcion

7 7 Srongly univrl Drminiic Univrl 2 ε-univrl 2 Rndom Fig 2 Hirrchy of hh funcion S Dfiniion nd 2 E Ev f) f) 2 f f - Â lic Bob f) k Fig 3 cur communicion cnrio h moiv our udy of rmining uncrini S Scion IV-B for dicuion corrld o nd inrcp on of h l ph W lo uppo m = kl for om k N lic ppli n invribl funcion f o nd divid f) ino k qul-izd pr f), f) 2,, f) k ) F m 2 = F l 2 F l 2 k im) S Fig 3 Bob rciv ll of hm, nd ppli f o dcod Hnc, Bob cn rcovr h originl mg lolly Howvr, if Ev omhow mng o p on h j-h pr f) j whr j, 2,, k}), Ev cn poibly im h mg from E nd f) j in Fig 3, w um Ev p on h fir pic of informion j = ) Ev uncriny wih rpc o i H f) j, E P E ) H hr i gnric nropy funcion; i will b kn o b vriou condiionl Rényi nropi in h ubqun ubcion) In hi cnrio, i i no y o im h uncriny H f) j, E P E ) i dpnd on h choic of j To void uch difficuly, w propo o pply rndom invribl funcion f X o To furhr rolv h formniond iu from compuionl prpciv, w rgrd F m 2 h fini xnion fild F 2 m Whn lic nd Bob choo invribl lmn X in h fini fild F 2 m ubjc o h uniform diribuion, nd f X ) i dfind f) := X, h mp f) j i univrl 2 hh funcion Thn, Ev uncriny wih rpc o cn b dcribd H f) j, E, X P E P X ) Whn, E) i kn o b n, E n ) = i, E i )} n i= whr h i, E i ) r indpndn nd idniclly diribud, our rul in h following ubcion r dircly pplicbl in vluing Ev uncriny murd ccording o vriou condiionl Rényi nropi W rmrk h if m i no mulipl of l, w cn mk h finl block mllr hn l bi wihou ny lo of gnrliy ympoiclly Indd, hi proocol cn b fficinly implmnd wih low) complxiy of Om log m) 20] bcu muliplicion in h fini fild F 2 m cn b rlizd by n pproprily-dignd circuln mrix, lding o f Fourir rnform-lik lgorihm Thrfor, hi communicion up, which conin n vdroppr, i prcicl in h n h ncoding nd dcoding cn b rlizd fficinly C ympoic of Rmining Uncrini Our rul in Thorm 2 o follow prin o h wor-c rmining uncrini ovr ll univrl 2 hh funcion W r inrd in up fxn U 2 n H ± nd up fxn U 2 n H ±, whr H ± i horhnd for H ± n f Xn n ), E n, X n PE n P Xn ) imilrly for H ± ) nd P E n i h n-fold produc mur W mphiz h h vluion of up f Xn U 2 n H ± nd up fxn U 2 n H ± r rongr hn ho in ndrd chivbiliy rgumn in Shnnon hory whr on ofn u

8 8 rndom lcion rgumn o r h n objc g, cod) wih good propri xi In our clculion of h ympoic of up fxn U 2 n H ± nd up fxn U 2 n H ±, w r h ll hh funcion in U 2 hv crin dirbl propry; nmly, h h rmining uncrini cn b pproprily uppr boundd In ddiion, in Thorm 3 o follow, w lo qunify h minimum r R uch h h b-c rmining uncrini ovr ll rndom hh funcion inf fxn R n H ± nd inf fxn R n H ± vnih For mny vlu of, w how h minimum r for h wo diffrn vluion wor-c ovr ll f Xn U 2 nd b-c ovr ll f Xn R) coincid, blihing ighn for h opiml comprion r L + := mx0, } Th following i our fir min rul Thorm 2 Rmining Uncrini) For ch n N, l h iz 4 of h rng of f Xn b n = nr Fix join diribuion P E P E) Dfin h wor-c limiing normlizd rmining uncrini ovr ll univrl 2 hh funcion GR, ) := lim n n up H n f Xn n ), E n, X n PE n P Xn ), f Xn U 2 nd 48) G R, ) := lim n n up H n f Xn n ), E n, X n PE n P Xn ) f Xn U 2 49) Rcll h dfiniion of h criicl r ˆR nd ˆR in 2) nd 22) rpcivly Th following chivbiliy mn hold: ) For ny 0, ], w hv GR, ) H E P E ) R +, 50) nd for ny 0, /2], w hv 2) For 0, ), w hv nd GR, ) G R, ) G R, ) H E P E) R + 5) H E P E ) R R ˆR mx 0,] H + E P E ) R) R > ˆR, 52) H E P E) R R ˆR mx 0,] H + E P E ) R) R > ˆR 53) Thorm 2 i provd in Scion V nd u vrl novl on-ho bound on h rmining uncriny ummrizd in ppndix ) coupld wih ppropri u of lrg-dviion rul uch Crmér horm nd Snov horm 33] In Fig 4 nd 5, w plo h uppr bound in 50) 53) for corrld ourc P E P0, } 2 ) wih P E 0, 0) = 07 nd P E 0, ) = P E, 0) = P E, ) = 0 For h uppr bound in 50) nd 5), w from Fig 4 h h r which h curv rniion from poiiv quniy o zro r clrly h condiionl Rényi nropi H E P E ) nd H E P E) In conr, from Fig 5, w obrv h h r which h normlizd rmining uncrini rniion from poiiv qunii o zro r h m nd r qul o h condiionl Shnnon nropy H E P E ) 044 n D Opiml R for Vnihing Rmining Uncrini Th ighn of h bound in Thorm 2 i prilly ddrd in h following horm whr w r concrnd wih h minimum comprion r R uch h h vriou normlizd rmining uncrini nd o zro To h nx rul uccincly, w rquir fw ddiionl dfiniion L P P) b givn diribuion L γ) := H + P ) = log P ) + nd l P ) ) := P ) + γ) b ild diribuion 5 rliv o P Dfin 0 P ) := mx 0, ] : H P ) H P ) ) } 54) W clim h 0 P ) i lwy poiiv; hi i bcu H P ) nd H P ) ) r coninuou nd H P ) = 0 H P ) =, nd 55) log = H P ) log = 0 ) = H P ) = 56) If P i no uniform on, 0 P ) 0, ) In fc inc H P ) nd H P ) ) r monooniclly incring nd dcring 6 rpcivly, 0 P ) 0, ) cn lo b xprd h uniqu oluion o h quion 4 Whn w wri n = nr, w mn h n i h ingr nr 5 P ) ) i indd vlid diribuion P ) ) = 6 Inuiivly, H P ) ) i monooniclly dcring bcu incr, P ) convrg o drminiic diribuion, which h h low Shnnon nropy 0

9 9 Rmining Uncriny Uppr bound on GR, ) = 00 = 0 = 02 = 03 Rmining Uncriny Uppr bound on G R, ) = 00 = 0 = 02 = R R R R Fig 4 Illurion of h uppr bound on h rmining uncrini GR, ) nd G R, ) in 50) nd 5) rpcivly ll curv rniion from poiiv quniy o zro h Rényi condiionl nropi H E P E ) lf) nd H E P E) righ) Rmining Uncriny Uppr bound on GR, ) = 00 = 0 = 02 = 03 Rmining Uncriny Uppr bound on G R, ) = 00 = 0 = 02 = R R R R Fig 5 Illurion of h uppr bound on h rmining uncrini GR, ) nd G R, ) in 52) nd 53) rpcivly ll curv rniion from poiiv quniy o zro h condiionl Shnnon nropy H E P E ) 044 n 0 0 P = 0 8, 0 8 ] 07 P = 025, 075] 0693 P = 049, 05] Enropi 0 4 Enropi Enropi Fig 6 Plo of H P ) olid) nd H P ) ) dod) for P P0, }) whr P 0) = 0 8 lf; lmo drminiic), P 0) = 025 middl) nd P 0) = 049 righ; lmo uniform) For h hr c, 0 P ) 0549, 0 P ) 065 nd 0 P ) 068 rpcivly H P ) = H P ) ) If P i uniform on, H P ) = H P ) ) for ny 0, ] nd uch, 0 P ) = S Fig 6 for illurion of h rgumn Now, givn P E P E), dfin 0 = 0 E P E ) := min 0 P E= ) : E} 57) Clrly, by h prcding rgumn nd h fc h E i fini, 0 i poiiv Thorm 3 Opiml R for Vnihing Normlizd Rmining Uncrini) For ch n N, l h iz of h rng of

10 0 f Xn b n = nr Dfin h b-c limiing normlizd rmining uncrini ovr ll rndom hh funcion GR, ) := lim n n G R, ) := lim n n inf H n f Xn n ), E n, X n PE n P Xn ), nd 58) f Xn R inf f Xn R H n f Xn n ), E n, X n PE n P Xn ) 59) lo dfin h limiing normlizd rmining uncriny for rongly univrl hh funcion f Xn : n,, n } GR, ) := lim n n H n f Xn n ), E n, X n PE n P Xn ) 60) Now dfin h opiml comprion r ) For 0, ], w hv nd for 0, 0 ], w hv 2) For 0, ), w hv 3) For 0, /2], w hv nd for 0, ), w hv T := infr R : GR, ) = 0}, 6) T := infr R : GR, ) = 0}, 62) T := infr R : G R, ) = 0}, 63) T := infr R : G R, ) = 0}, nd 64) T := infr R : GR, ) = 0} 65) T H E P E ) 66) T H E P E ) 67) T = T = H E P E ) 68) T = T = H E P E), 69) T = T = H E P E ) 70) Th proof of hi rul i providd in Scion VI For Pr ) of h bov rul, unforunly, w do no hv mching lowr bound o T Howvr, for 0, 0 ], h bound in 67) y h rricd o h imporn cl of rongly univrl hh funcion g, h ubiquiou rndom binning procdur 32]), h rul in 66) i igh hr i mching lowr bound Hnc, 67) rv pril convr o 66) In ohr word, 66) i igh wih rpc o h nmbl vrg 34] whn h nmbl i chon o b rongly univrl hh funcion Th qulii in 68) 70) imply h in h pcifid rng of, h opiml r for h b-c rmining uncriny ovr ll hh funcion nd wor-c rmining uncriny ovr ll univrl 2 hh funcion r h m I i inring o obrv h h opiml r for h c in 69) dpnd on 0, 0 ] bu h opiml r for h + c in 68) nd 70) do no Thi i lo clrly obrvd in Fig 4 nd 5 Th proof of h chivbiliy pr uppr bound) of h rul follow dircly from Thorm 2 For h convr pr lowr bound), w ppl o h mhod of yp 6, Ch 2], h momn of yp cl numror mhod 2] 25], nd h xponnil rong convr for Slpin-Wolf coding 2, Thorm 2] W lo xploi rul by Fhr nd Brn 0, Thorm 3] concrning h monooniciy 6) nd chin rul 7) for Gllgr form of h condiionl Rényi nropy H E P E) E Exponnil R of Dcr of Rmining Uncrini Lly, w conidr h r of xponnil dcr of h vriou wor-c rmining uncrini Thorm 4 Exponn of Rmining Uncrini) For ch n N, l h iz of h rng of f Xn b n = nr Fix join diribuion P E P E) Dfin h xponn of 48) nd 49) ] ER, ) := lim n n log up H n f Xn n ), E n, X n PE n P Xn ), nd 7) f Xn U 2 ] E R, ) := lim n n log up H n f Xn n ), E n, X n PE n P Xn ) f Xn U 2 72)

11 = 00 = 0 = 02 = 03 Lowr bound o xponn ER, ) = 00 = 0 = 02 = 03 Lowr bound o xponn E R, ) Exponn R R Exponn R R Fig 7 Illurion of h lowr bound on h xponn ER, ) nd E R, ) in 73) nd 74) rpcivly Th curv rniion from 0 o poiiv quniy H E P E ) lf) nd H E P E) righ) Th r h xponn of h wor-c rmining uncrini ovr ll univrl 2 hh funcion Th following chivbiliy mn hold: ) For 0, ], w hv nd for ny 0, /2], w hv ER, ) up E R, ),) up,/2) R H E P E ) ) +, 73) ) + R H E P E) 74) 2) For 0, ), w hv nd ER, ) E R, ) up 0,/2) up 0,/2) ) R H E P E), 75) ) R H E P E) 76) Thorm 4 i provd in Scion VII W obrv from Propoiion h h righ-hnd-id of h bound in Pr 2), which cn b hown o b non-ngiv for R H E P E ), r lowr bound on h opiml rror xponn 2], 4], 5] for h Slpin-Wolf 3] problm, dnod ESW R) In fc, i cn b infrrd from Gllgr work 4] or 6, Problm 25)] for h E = c) h if w rplc h domin of h opimizion ovr from 0, /2) o 0, ), h lowr bound in 75) 76) r qul o ESW R) for crin rng of coding r bov H E P E ) Th ron why w obin ponilly mllr xponn i bcu w conidr h wor-c ovr ll univrl 2 hh funcion f Xn U 2 in h dfiniion of ER, ) nd E R, ) in 7) nd 72) rpcivly For h Slpin-Wolf problm, w cn choo h b qunc of hh funcion In Fig 7, w plo h lowr bound in 73) nd 74) for h m ourc P E in Fig 4 nd 5 in Scion IV-C W no h h r which h lowr bound on h xponn rniion from bing zro o poiiv i givn by H E P E ) for 73) nd H E P E) for 74) Th lr obrvion corrobor 69) of Thorm 3 Furhrmor, dicud in h prviou prgrph, h = 0 c in h righ plo of Fig 7 i lowr bound on ESW R) For hi ourc, if w chng h domin of opimizion of from 0, /2) o 0, ), h plo do no chng i, h opiml < /2) o for r in mll nighborhood bov H E P E ) 044 n, h curv indd rc ou h opiml rror xponn ESW R) V PROOF OF THEORE 2 W prov mn 50), 5), 52), nd 53) in Subcion V-, V-B, V-C, nd V-D rpcivly

12 2 Proof of 50) in Thorm 2 To prov h uppr bound in 50), w u h on-ho bound in 76) in Lmm 3 ppndix ) W fir um h H E P E ) R > 0 In hi c, ɛ H n E n P n E ), 77) for n ufficinly lrg Thn h on-ho bound in 76) impli h for ny ɛ-lmo univrl 2 hh funcion f Xn, H n f Xn n ), E n, X n P n E P Xn ) = log E X n H n f Xn n ),E n P n )) E 78) log + ɛ H n E n P n E ) ) 79) log 2 ɛ H n E n P n E ) ) 80) = log 2ɛ ) + n H E P E ) R ), 8) whr in 80) w ud 77) nd in 8) w ud h fc h h condiionl Rényi nropy i ddiiv for indpndn rndom vribl, i, H n E n PE n ) = nh E P E ) Sinc hi bound hold for ll ɛ-univrl 2 hh funcion f Xn U 2 including ɛ = ), normlizing by n, king h lim, nd ppling o h dfiniion GR, ) in 48) blih h GR, ) H E P E ) R if H E P E ) R 0 Now, whn H E P E ) R 0, w follow h p lding o 79) bu u log + ) o blih h H n f Xn n ), E n, X n P n E P Xn ) ɛ nh E P E) R) 82) From 82), w conclud h if H E P E ) R 0, w hv GR, ) = 0 bcu GR, ) cnno b ngiv) Sinc h wo bound in 8) nd 82) hold for ll qunc of univrl 2 hh funcion f Xn U 2 king ɛ = bov), oghr hy blih 50) B Proof of 5) in Thorm 2 To prov h uppr bound in 5), w u h on-ho bound in 77) in Lmm 3 ppndix ) Similrly, o h nlyi in Scion V-, w my conidr wo c H E P E) R > 0 or H E P E) R 0 W will only conidr h formr inc h nlyi of h lr prlll h in Scion V- Undr h formr condiion, w my um h ɛ H n E n P n E ) 83) for n ufficinly lrg Th on-ho bound in 77) impli h for ny ɛ-lmo univrl 2 hh funcion f Xn, H n f Xn n ), E n, X n PE n P Xn ) = log E Xn H n f Xn n ),E n P n )) E 84) ) log + ɛ H n E n P n E ) 85) = log 2 ɛ H n E n P n E ) log ) 2ɛ + n H E P E) R ) 86) ), 87) whr in 86) w ud 83) Sinc hi bound hold for ll qunc of univrl 2 hh funcion f Xn U 2 king ɛ = bov), normlizing by n nd ppling o h dfiniion G R, ) in 49), w blih h uppr bound in 5) C Proof of 52) in Thorm 2 To prov h uppr bound in 52), w will ror o h on-ho bound in 56) in Lmm ppndix ) W fir obrv by Crmér horm 33, Scion 22] h lim n n log P E n, ) : P E n ) ɛ nr} = up H+ E P E ) R) } 88) 0

13 3 Thi i bcu h cumuln gnring funcion of h rndom vribl log P E E) whr, E) i diribud P E i ] log E log P E E) = log P E ) P E ) + = H + E P E ) 89) Nx, w pply gnrlizion of Crmér horm concrning rbirry fini non-ngiv mur 7 no ncrily probbiliy mur) o h qunc of rndom vribl log P E n n E n ) = n i= log P E i E i ) undr h qunc of non-ngiv fini join mur B,) B P E n, )P E n ) o blih h lim n n log =,):P n E )<ɛ nr P n E, )P n E ) nr Th mn in 90) hold bcu h rlvn cumuln gnring funcion i H E P E ) R) R ˆR mx 0,] H + E P E ) R) R ˆR 90) τ ) = log, P E, )P E ) log P E ) 9) from h dfiniion of h condiionl Rényi nropy in 0) Hnc, Γ := lim n n log = )H + ) E P E ), 92),):P n E )<ɛ nr P n E, )P n E ) 93) = up R τ )} 94) 0 = up R + )H+ ) E P E ) } 95) 0 By diffrniing h objciv funcion in 95), w h if R ˆR = d d H + = cf h dfiniion of h criicl r in 2)), h opiml oluion i ind = 0 rcll h H + i concv o ˆR i dcring) nd o Γ = H E P E ), lding o h fir clu in 90) Convrly, whn R > ˆR, h opiml oluion i ind > 0 Thi ld o h cond clu on h righ-hnd-id of 90) bcu h lf-hnd-id of 90) i now Γ R = up )R + )H+ ) E P E ) } 96) 0 = mx H + E P E ) R} 97) 0,] Sinc 88) i no mllr hn 90), h lr domin Now uing h on-ho bound in 56) in Lmm w h for ny qunc of ɛ-lmo univrl 2 hh funcion f Xn, lim n n H n f Xn n ), E n, X n PE n P Xn ) = lim n n log E X n lim n n log = Hn f Xn n ),E n P n E )] 98) 2 PE, n ),):P n E ) ɛ nr ] + 2 PE, n )P E n ) nr,):p n E )<ɛ nr H E P E ) R R ˆR mx 0,] H + E P E ) R) R ˆR 00) Sinc hi bound hold for ll qunc of univrl 2 hh funcion f Xn U 2 king ɛ = bov), w hv blihd h uppr bound in 52) 7 Th ndrd Crmér horm 33, Scion 22] or Snov horm 33, Scion 2]) i lrg-dviion rul concrning h xponn of P n B) whr P i probbiliy mur nd B i n vn in h mpl pc Ω If P i no ncrily probbiliy mur bu fini non-ngiv mur i i in our pplicion), y µ, Crmér horm clrly lo ppli by dfining h nw probbiliy mur B P B) := µb)/µω) 99)

14 4 D Proof of 53) in Thorm 2 W now prov h uppr bound in 53) For hi purpo, w u h on-ho bound 67) in Lmm 2 ppndix ) W mploy Crmér horm 33, Scion 22] wih h qunc of rndom vribl log P E n ) ã P E n ã ) ) whr ã = ã, ã 2,, ã n ) n ) undr h qunc of join diribuion PE n, ) W clim h lim n n log PE) n P :P n E ) ɛ E n ) nr ã P n E ã ) = lim n n log PE, n ) 0),):P n E ) ã P n E ã ) ) ɛ n R = mx 0 + H + E P E ) R) 02) L u juify h clim in 02) crfully Th drivion hr i imilr o h in 92) 95) nd involv clculing h rlvn cumuln gnring funcion τ ) := log ) ]) P E, ) xp log P E ) P E ã ) 03), = log ) ) P E ) P E ) + P E ã ) 04) ã = + H + E P E ), 05) whr h l p rul from h dfiniion of h wo-prmr condiionl Rényi nropy in 20) By n pplicion of Crmér horm, h corrponding xponn i 02) In ddiion, w pply h gnrlizd vrion of Crmér horm foono 7) o compu nohr lrg dviion quniy Conidr h qunc of rndom vribl log P E n ) ã P E n ã ) ) diribud ccording o h qunc of non-ngiv fini join mur B,) B P E n)p E n )) ã P E n ã ) ) W clim h h xponn cn b clculd o b lim n n log = n R P n E) ã ) ] P n :P n E ) <ɛ E ) P E n ã ) 06) nr ã P n E ã ) ã mx 0,] H E P E) R) R ˆR H + E P E ) R) R > ˆR 07) L u juify h clim in 07) crfully Thi p follow bcu h rlvn cumuln gnring funcion i τ ) := log xp = log P E ) log P E ) P E ) ã P E ) ã P E ) ã P E ã ) ) P E ã ) ) ] ) 08) P E ã ) ) 09) Thu 06) rduc o mx τ ) } 0 + R = mx log P E ) ) P E ) + P E ã ) + R 0) ã = mx + H + E P E ) } + R, )

15 5 whr h l p follow from h dfiniion of h wo-prmr condiionl Rényi nropy in 20) Now from h dfiniion of h criicl r ˆR in 22) nd h fc h H = H, w know h if R ˆR, h mximizion in ) i ind =, ruling in h fir c in 07) Convrly, h cond c rul from R > ˆR whr h domin of i 0, ] inc h vnul xponn cnno b ngiv Thi prov 07) Sinc 07) i no grr hn 02), h formr domin h xponnil bhvior of G R, ), nd o plugging h vluion ino h on-ho bound in 67) which hold for ny ɛ-lmo univrl 2 hh funcion f Xn, lim n n H n f Xn n ), E n, X n PE n P Xn ) = lim + n n log E X n H n f Xn n ),E n P n )] E 2) lim + n n log 2 PE) n :P n E ) ɛ nr ã P n E ã ) P n E ) + 2 ɛ n R PE) n ) ] P n :P n E ) <ɛ E ) P E n ã ) 3) nr ã P n E ã ) ã H = E P E) R R ˆR mx 0,] H + E P E ) R) R > ˆR 4) Sinc hi bound hold for ll qunc of univrl 2 hh funcion f Xn U 2 king ɛ = bov), w hv blihd h uppr bound in 53) VI PROOF OF THEORE 3 Th bound on h opiml comprion r corrponding o h condiionl Rényi nropy nd Gllgr form of h condiionl Rényi nropy r provd in Subcion VI- nd VI-B rpcivly Proof of 66), 67), nd 68) Proof: Rcll h dfiniion of h opiml r T, T, nd T in Thorm 3 Sinc GR, ) GR, ), nd boh funcion r monooniclly non-incring in R, i hold h T T for ll Fir, w prov h uppr bound o T nd T in Scion VI-; nx w prov h lowr bound o T in Scion VI-2; nd finlly w prov h lowr bound o T in Scion VI-3 For h + c, h uppr nd lowr bound mch for ll 0, ) nd o hi prov 68) ) Uppr Bound: W rfr o h mn in 50) W obrv h if R H E P E ), GR, ) = 0 inc GR, ) i uppr boundd by H E P E ) R + nd GR, ) i non-ngiv Hnc, T H E P E ) for ll 0, ] Thi prov 66) Nx w rfr o h mn in 52) If R H E P E ), w know from h monooniclly dcring nur of H E P E ) h H + E P E ) R i non-poiiv for 0, ] Thu, h opiml in h opimizion in mx 0,] H + E P E ) R) i = 0 nd conqunly, h opiml objciv vlu i lo 0 On h ohr hnd, for R ˆR, H E P E )), h opiml 0, ] nd o h h opiml objciv vlu i poiiv W conclud for 0, ) h h opiml ky gnrion r i uppr boundd by h condiionl Shnnon nropy H E P E ) In ummry, w conclud h T H E P E ) for ll 0, ), proving h uppr bound for 68) 2) Lowr Bound o T : W now conidr rongly univrl hh funcion 8] cf Dfiniion 2) nd 0, 0 ], whr 0 = 0 E P E ) i dfind in 57) or prcily, w hll how h for h qunc of rongly univrl hh funcion f Xn : n,, nr }} n N nd ny 0, 0 ], w hv GR, ) = lim n n H n f Xn n ), E n, X n PE n P Xn ) H E P E ) R +, 5) immdily implying h T H E P E ) In fc, for hi rng of, no only i i ru h h minimum vlu of R uch h GR, ) = 0 coincid wih h in 66), h bound in 5) rv igh lowr bound o h chivbiliy uppr) bound for GR, ) in 50) l for rongly univrl hh funcion) In h following, w mk u hvy u of h mhod of yp; rlvn noion i ummrizd in Scion II-B

16 6 For of xpoiion, w fir conidr h c in which E = or quivlnly, E = Subqunly, w gnrliz our rul o h gnrl c in which E > Sring wih h on-ho bound in 80) in Lmm 3), w hv E Xn H n f Xn n ),X n P n )] = E Xn P) n n P ã f Xn f Xn )) 2 E Xn P) n P) n + = E Xn P) n P) n + = E Xn P) n P) n + = = P) n P) n + E Xn P) n P) n + Q P n) ã) 6) n P ã f Xn f Xn ))\} Q P n) ã T Q \}:f Xn )=f Xn ã) mx Q P n) ã T Q \}:f Xn )=f Xn ã) mx Q P n) E Xn ã T Q \}:f Xn )=f Xn ã) ã T Q \}:f Xn )=f Xn ã) ] ã) 7) ] Pã) n 8) ] Pã) n 9) ] Pã) n 20) ] Pã) n, 2) whr in 7) w ud h bound b + c) 2 b + c ) for b, c 0 nd 0, ] conqunc of Jnn inquliy pplid o h concv funcion for 0, ]), in 8), w pli h innr um ino n-yp on, nd in 9) nd 2) w ud h fc h hr r polynomilly mny yp o w cn inrchng um ovr yp wih mximum ovr yp nd vic vr Thi drivion i imilr o 2, Eqn 20)] Now, w w um h R < H P ) nd lo h 0 P ) Th lr umpion mn h H P ) H P ) ) Scion IV-D) In hi c, w my u h bound in 205) in Lmm 4 in ppndix B) o lowr bound h innr) um ovr xpcion in 2) W hv E Xn ] Pã) n Q P n) ã T Q \}:f Xn )=f Xn ã) Q P n) E ] X n Pã) n 22) ã T Q \}:f Xn )=f Xn ã) ] E Xn Pã) n 23) Q P n) ã T Q \}:f Xn )=f Xn ã) = E ] X n Pã) n 24) ã :f Xn )=f Xn ã) = P n ã) Pr f Xn ) = f Xn ã) } 25) ã = nr Pã) n, 26) ã whr 22) u 205) in Lmm 4, nd 23) u h inquliy i b i i b i) which hold for non-ngiv b i nd 0, ] 3, Problm 45f)] nd 26) u h fc h f Xn i rongly univrl hh funcion cf Dfiniion 2)

17 7 Subiuing 26) ino 2), w obin E Xn H n f Xn n ),X n P n )] P) n P) n + nr Pã) n 27) ã = + nr P) n P) n } 28) = + xp nh P ) R] ) 29) = xp nh P ) R] ), 30) whr in 29) w ud h fc h mx P n) 2 y) for n lrg nough o P n)} P n) 2 ) > 0, nd in 30), w ud h condiion H P ) > R o h xprion in 29) i xponnilly lrg In h ohr c, whn R H P ), w imply lowr bound h um ovr yp rm in 2) by 0 nd hnc, h nir xprion in 2) cn b lowr boundd by Thu, w conclud h E Xn H n f Xn n ),X n P n )] xp n H P ) R +) 3) For h c E =, hi blih 5) for 0, 0 P )] Now w xnd our nlyi o E > Nurlly, w opr on yp-by-yp bi ovr E n nlogouly o h drivion of 3) vi Lmm 4, w h if 0 0 = min 0 P E= ), w hv E Xn H n f Xn n ),E n,x n P n )] E +) Q E P ne) P n ET QE ) xp n Q E )H P E= ) R E 32) S Rmrk 2 in ppndix B for dild dcripion of hi p In fc, hi drivion i imilr o h corrponding clculion in rhv work in 2, Scion IV-C] nd 22, Scion IV-D] Bcu PE nt Q E ) = ndq E P E ), E Xn H n f Xn n ),E n,x n P n )] E xp n mx Q E PE) +}) DQ E P E ) + Q E )H P E= ) R 33) Dno h opimizr in h mximizion in 33) Q E If h + i inciv for Q E, by righforwrd clculu, Q E) = E P E ) H P E=) E P, E, 34) E ) H P E= ) whil if h + i civ for Q E, obviouly Q E ) = P E) for ll E By uing h form of h opimizr Q E nd h fc h h condiionl Rényi nropy H E P E ) dfind in 0)) i rld o h diribuion P E nd h uncondiionl Rényi nropi H P E= ) : E} follow H E P E ) = log E P E ) H P E=), 35) w h h mximizion in 33) rduc o H E P E ) R + Upon king h log, dividing by n, nd king h lim, w compl h proof of 5) for h c whr E >, uming 0, 0 ] uch, w hv compld h proof of h lowr bound on h opiml comprion r for rongly univrl hh funcion in 67) 3) Lowr Bound o T : For h lowr bound o T, w no from h work by Hyhi in 35, Lmm 5] h for ny, ) \ 0} h h condiionl Rényi nropy nd i Gllgr form ify Furhrmor, bcu H i monooniclly non-incring, n H n f n n ), E n PE) n H E P E ) H E P E ) 36) n H n f n n ), E n PE) n 37) = n H n f n n ), E n P n E) 38)

18 8 W rquir h h rm on h lfmo rm of hi inquliy o vnih o inc h conrin GR, ) = 0 i prn Thi impli h h normlizd Gllgr min-nropy n H ncrily vnih From h rlion bwn h rong convr xponn nd H in 25), w h n log Pn) c f n ) 0, whr P n) c f n ) i h probbiliy of corrc opiml P) dcoding undr ncodr f n By h xponnil rong convr for Slpin-Wolf coding by Oohm nd Hn 2, Thorm 2], w know h if R < H E P E ), i i lo ncrily ru h lim n n log Pn) c f n ) > 0 Hnc, by conrpoiion, R H E P E ) Thu, T H E P E ) Thi, oghr wih h corrponding uppr bound provd in Scion VI-, immdily blih h lowr bound o 68) for ll 0, ) B Proof of 69) nd 70) Proof: To prov 69) nd 70), fir rcll h dfiniion of h opiml r T nd T in Thorm 3 Sinc G R, ) G R, ), nd boh funcion r monooniclly non-incring in R, i hold h T T for ll ) Uppr Bound: Th uppr bound for T nd T cn b hown in h m wy h rgumn o uppr bound T nd T in Scion VI- nd uing h rul in 5) nd 53) Dil r omid for brviy 2) Lowr Bound: For h lowr bound o T, w u n imporn rul by Fhr nd Brn 0, Thorm 3], which in our conx h for ny hh funcion f n :,, }, w hv H f n), E P E ) H E P E) log 39) for ll, ) No h, ) includ h rng of inr for T which i 0, /2] Th inquliy in 39) i conqunc of monooniciy undr condiioning nd h chin rul for h Gllgr form of h condiionl Rényi nropy S 6) nd 7) Sinc = nr nd H n E n PE n ) = nh E P E), hi impli h for 0, /2], w hv G R, ) H E P E) R + which immdily ld o h bound T H E P E), blihing 69) Now, for h lowr bound of T in 70), w no from h monooniclly dcring nur of H h n H n f n n ), E n PE) n n H n f n n ), E n PE) n 40) for ll 0, ) W rquir h h rm on h lf o vnih inc h conrin G R, ) = 0 i prn Hnc, n H lo vnih Similrly o h rgumn in Scion VI-3 fr 38), by invoking h xponnil rong convr o h Slpin-Wolf horm 2, Thorm 2], w know h T H E P E ) Thi blih 70) Rmrk W rmrk h h proof of h lowr bound o T bov i much implr hn h proof of h lowr bound of T for rongly univrl hh funcion in Scion VI-2 bcu w cn lvrg wo uful propri of H monooniciy nd chin rul) lding o 40) In conr, H do no po h propri Obrv h h proof in h fir hlf of Scion VI-B2 lo llow u o conclud h h uppr bound in 5) for h dirc pr i coincidn wih h lowr bound on G R, ) for ll 0, /2], i, G R, ) = G R, ) = H E P E) R + 4) VII PROOF OF THEORE 4 W prov mn 73) nd 74) in Subcion VII- nd VII-B rpcivly Smn 75) nd 76) r joinly provd in Subcion VII-C Proof of 73) in Thorm 4 Fir w no h ll h xponn r non-ngiv inc H n f Xn n ), E n, X n P n E P X n ) = On) nd imilrly for ll h ohr Rényi informion qunii Thi giv h + ign in ll lowr bound in 73) 76) Fix, ] Th on-ho bound in 76) in Lmm 3 impli h for ny ɛ-lmo univrl 2 hh funcion f Xn, log E Xn H n f Xn n ), E n, X n P n E)] log E Xn H n f Xn n ), E n, X n PE)] n 42) = log E X n log H n f Xn n ),E n,x n P n )]]} E 43) log log E X n H n f Xn n ),E n,x n P n )]} E 44) log log + ɛ H n E n P n E ) )} 45) log ɛ nh E PE) } 46) } = log ɛ + nr H E P E )), 47)

19 9 whr in 42) w ud h fc h H i monooniclly non-dcring, in 44) w pplid Jnn inquliy o h concv funcion log, nd in 46) w mployd h bound log + ) Th bound in 47) hold for ll f Xn U 2 nd ll, ] Now, w normliz by n nd k h lim n Finlly, w mximiz ovr ll, ] Thi yild 73), concluding h proof B Proof of 74) in Thorm 4 Fix, /2] Th on-ho bound in 77) in Lmm 3 impli h for ny ɛ-lmo univrl 2 hh funcion f Xn, ] log E Xn H n f Xn n ), E n, X n PE) n ] log E Xn H n f Xn n ), E n, X n PE) n 48) = log E Xn log H n f Xn n ),E n,x n P n )]]} E 49) log log E Xn H n f Xn n ),E n,x n P n )]} E 50) )} log log + ɛ H n E n P n E ) 5) log ɛ = log ɛ n H E P E) } + n } R H E P E) In 48), w ud h fc h for H i monooniclly non-dcring Thi bound hold for ll f X n U 2 nd ll, /2] Now, w normliz by n nd k h lim n Finlly, w mximiz ovr ll, /2] Thi yild 74), concluding h proof C Proof of 75) nd 76) in Thorm 4 W prov 76) bfor proving 75) W no h lim n n log H n f Xn n ), E n, X n PE n P Xn ) ) 52) 53) lim n n log H n f Xn n ), E n, X n PE n P Xn ) 54) for ll 0, ) nd 0, ] Thi i bcu α Hα i monooniclly non-incring Thi bound impli h E R, ) E R, ) Combining hi wih h lowr bound on h xponn of H in 74) nd noing h h lowr bound i mximizd = 0 immdily blih 76) Finlly, w no from 8) h H H o lim n n log H n f Xn n ), E n, X n PE n P Xn ) lim n n log H n f Xn n ), E n, X n PE n P Xn ) 55) Combining hi wih h lowr bound on E R, ) in 76) compl h proof of 75) VIII CONCLUSION ND FUTURE WORK In hi ppr, w hv dvlopd novl chniqu o bound h ympoic bhvior of rmining uncrini murd ccording o vriou condiionl Rényi nropi Thi i in conr o ohr work ], 2], 4] 7] h qunify uncriny uing Shnnon informion mur W moivd our udy by howing h h qunii w chrcriz r gnrlizion of h rror xponn nd h rong convr xponn for h Slpin-Wolf problm W udid vriou imporn cl of hh funcion, including univrl 2 nd rongly univrl hh funcion Finlly, w lo howd h in mny c, h opiml comprion r o nur h h normlizd rmining uncrini vnih cn b chrcrizd xcly, nd h hy xhibi bhvior h r omwh diffrn o whn Shnnon informion mur r ud In h fuur, w hop o driv lowr bound o h normlizd rmining uncrini nd uppr bound on hir xponn h mch or pproximly mch hir chivbiliy counrpr in Thorm 2 nd 4 In ddiion, ju in h uhor rlir work in 9], w my lo udy h cond-ordr or n bhvior 36] of h rmining uncrini Th chllnging ndvor rquir h dvlopmn of nw on-ho bound wll h pplicion of nw lrg-dviion nd cnrl-limiyp bound on vriou probbilii

20 20 PPENDIX ONE-SHOT DIRECT PRT BOUNDS In hi ppndix, w nd prov vrl on-ho bound on h vriou condiionl Rényi nropi Lmm nd 2 r ud in h proof for h rmining uncrini Thorm 2) Lmm 3 i ud in h proof for h xponn Thorm 4) Lmm For ɛ-lmo univrl 2 hh funcion f X : =,, }, w hv E X H f X ),E,X P E ) 2 P E ) P E ) :P E ) ɛ ɛ ) + 2 P E ) for ny 0, ] f Xn :P E )< ɛ P E ) 56) Proof: W fir blih om bic inqulii: For ny, ) E, nd ny ɛ-lmo univrl 2 hh funcion : =,, }, E X P E ) P E ) + ɛ P E ) 57) Uing 59), w hv f X f X )) P E ) + ɛ 58) ɛ } 2 mx P E ), 59) E X H f X ),E,X P E ) = E X P E ) P E ) ) P E ) i f X i) f X i) f X i) P 60) E ) = E X P E ) P E ) P E ) 6) i f X i) f X i) = E X P E ) P E ) P E ) 62) f X f X )) P E ) P E ) E X P E ) 63) = 2 P E ) P E ) P E ) 2 mx f X f X )) P E ), P E ) mx P E ), ɛ }) 64) ɛ ) } 65) = 2 P E ) ɛ ) P E ) + 2 P E ) P E ) 66) :P E ) ɛ :P E )< ɛ Thu, w obin 56) Lmm 2 For ɛ-lmo univrl 2 hh funcion f X : =,, }, w hv E X H f X ),E,X P E ) 2 P E ) P E ) :P E ) ɛ P E ) + 2 ɛ ) P E ) P E ) P E ) P E ) for ny 0, ) :P E ) < ɛ ) 67)

21 2 Proof: Uing 59), w hv E X H f X ),E,X P E ) = E X P E ) P E ) ) P E ) i f X i) f X i) f X i) P E ) = E X P E ) P E ) P E ) i f X i) f X i) = E X P E ) P E ) P E ) f X f X )) P E ) P E ) E X P E ) P E ) P E ) + P E ) = 2 P E ) P E ) 2 mx :P E ) ɛ f X f X )) P E ), }) ɛ P E ) P E ) P E ) 2P E ) ) :P E ) < ɛ :P E ) ɛ + 2 ɛ ) Thu, w obin 67) P E ) P E ) P E ) P E ) P E ) :P E ) < ɛ 2 ɛ P E ) P E ) P E ) Lmm 3 For ɛ-lmo univrl 2 hh funcion f X : =,, }, w hv for ll 0, ] In ddiion, w lo hv for ll 0, /2] Proof: W hv E X H f X ),E,X P E ) + ɛ H E P E) E X H f X ),E,X P E ) + ɛ H E P E) ) P E ) ) 68) 69) 70) 7) 72) 73) 74) 75), 76), 77) E X H f X ),E,X P E ) = E X P E ) P E ) ) P E ) i f X i) f X i) f X i) P 78) E ) = E X P E ) P E ) P E ) 79) i f X i) f X i) = E X P E ) P E ) P E ) 80) f X f X )) P E ) P E ) E X P E ) 8) f X f X ))

22 22 P E ) P E ) P E ) P E ) + ɛ ) 82) ɛ ) ) P E ) P E ) + = P E ) P E ) + P E ) ɛ ) ) 84) = + P E ) P E ) ɛ ) 85) = + ɛ H E P E) 86) In 8), w pplid Jnn inquliy wih h concv funcion Hr i whr h condiion 0, ] i ud In 82), w ud 58) nd in 83), w ud h fc h b + c) b + c for b, c 0 nd 0, ] 3, Problm 45f)] Thu, w obin 76) For 77), conidr, E X H f X ),E,X P E ) = E X P E ) P E ) ) P E ) i f X i) f X i) f X i) P E ) = E X P E ) P E ) i f X i) = E X P E ) P E ) P E ) i f X i) f X i) = E X P E ) P E ) P E ) f X f X )) P E ) P E ) E X P E ) f X f X )) P E ) P E ) P E ) + ɛ P E ) P E ) P E ) P E ) + ɛ P E ) = + P E ) P E ) ɛ ) = + ɛ ) = + P E ) P E ) + ɛ P E ) ɛ P E ) ) + P E ) P E ) P E ) = + ɛ H E P E) P E ) ) P E ) ) 83) 87) 88) 89) 90) 9) 92) 93) ) 94) 95) 96) 97) 98)

23 23 In 9), w pplid Jnn inquliy wih h concv funcion Hr i whr h condiion 0, /2] i ud Th xplnion for h ohr bound prlll ho for h proof of 76) nd r omid for h k of brviy Thi compl h proof of 77) PPENDIX B BOUND ON THE SU OF EXPECTTIONS IN 2) Rcll h dfiniion of γ) = H + P ) nd h ild diribuion P ) ) = P ) + γ) inroducd in Scion IV-D I i ily n cf Scion II-) h γ) i ricly concv for > I lo hold h DP ) P ) = γ) γ ), nd 99) HP ) ) = + )γ ) γ) 200) fc w u in h qul i h if >, h funcion HP ) ) i monooniclly dcring; hi cn b n by conidring h driviv d ) dhp ) = + )γ ) < 0 Lmm 4 L h um of xpcion in 2) b dnod Ψ n := E Xn Q P n) ã T Q \}:f Xn )=f Xn ã) ] Pã) n 20) L R, ) b h uniqu numbr ifying HP R) ) = R log Thn, Ψ n xp nλ, R)) 202) whr Λ, R) := R + DP R ) P )) R R + γ ) > R 203) Furhrmor, if R < H P ), hn for ll 0, 0 P )] cf dfiniion in 54)), Ψ n xp nr + DP R) P ))]) 204) = E ] X n Pã) n 205) Q P n) ã T Q \}:f Xn )=f Xn ã) Th inquliy in 204) impli h undr h d condiion on R nd, h fir clu in 203) i civ Th clculion hr r omwh imilr o ho in rhv work in 2, Scion IV-C] nd 22, Scion IV-D] bu h whol proof for h c in which E = i includd for compln S Rmrk 2 for kch of how o xnd h nlyi o h mmoryl bu non-ionry c in which E = W rmrk h whn = R, Λ, R) =+R = + R )γ R ) 206) By uing HP R) ) = R, 99), nd 200), w immdily h hi boundry c coincid wih h wo c of 203) o Λ, R) i coninuou + R Proof: L G R := Q P) : HQ) > R}, G R,n := G R P n ) nd clg R ) b h clour of G R W pli Ψ n ino h following wo um Ψ n = ] E Xn Pã) n Q G R,n ã T Q \}:f Xn )=f Xn ã) }} α n + ] E Xn Pã) n 207) Q G R,n c ã T Q \}:f Xn )=f Xn ã) }} β n

24 24 Dfin N Q := ã T Q \} f X n ) = f Xn ã)} Thi i um of L Q := T Q \ } indpndn nd idniclly diribud 0, }-rndom vribl Y i } L i= wih PrY i = ) = nr =: p propry of rong univrl 2 hh funcion) L Q T Q b ny gnric vcor of yp Q L u now lowr bound α n nd β n For h xpcion wihin α n, w hv E Xn ã T Q \}:f Xn )=f Xn ã) = P n Q ) E Xn = P n Q ) E NQ ] ] Pã) n ã T Q \}:f Xn )=f Xn ã) ] 208) 209) P n Q ) E N Q ]} 20) = E ] X n Pã) n, 2) ã T Q \}:f Xn )=f Xn ã) whr 208) follow bcu ll ã in h um hv h m yp Q, 209) from h dfiniion of N Q, 20) follow from Lmm 5 in ppndix C) undr h condiion h Q G R,n o L Q p = EN Q ] n+) xpnhq) R]) nd 228) ppli) Thu, w conclud h α n EN Q ]) P n Q ) 22) Q G R,n = mx EN Q]) P n Q ) 23) Q clg R ) By furhr uing h fc h P n Q) = xp ndq P ) + HQ)]) 6, Lmm 26], w hv ) α n xp nr) xp n min DQ P ) =: α n 24) Q clg R ) Nx, w lowr bound h xpcion in β n Th p from 208) o 209) rmin h m bu w bound EN Q ] diffrnly W hv E NQ] PrN Q = ) 25) ) = LQ p p) L Q 26) = L Q p xp L Q ) log p) ) 27) ) p L Q p xp L Q ) 28) p L Q p = EN Q ], 29) whr 28) follow from h bic inquliy log ) nd 29) follow from h fc h L Q )p nr T Q \ } xpnhq) R]) whn Q GR,n c = Q P n) : HQ) R} Thu, w hv β n Q G c R,n = mx Q G c R = xp nr) xp EN Q ]P n Q ) 220) EN Q ]P n Q ) 22) n min DQ P ) )HQ) ]) =: β n 222) Q GR c W rmrk h h vluion in 20) nd 29) r, in fc, xponnilly igh 8 2, Eqn 34)] Thi impli h α n nd β n = βn Howvr, w only rquir h lowr bound = αn 8 Th inuiion hr i h if HQ) > R, h rndom vribl N Q concnr doubly-xponnilly f o i xpcion, which ilf i xponnilly lrg On h ohr hnd, if HQ) < R, N Q i ypiclly xponnilly mll, o EN Q ] i domind by h rm PrN Q = )

25 25 I i y o h h opiml diribuion Q in h opimizion in h xponn of α n ifi HQ ) = R i, Q li on h boundry of G R ) In fc, h xponn which i h lol ourc coding rror xponn 6, Thorm 25]) cn b xprd DP R) P ) whr R i chon uch h HP R) ) = + R)γ R ) γ R ) = R Thu, α n = xp nr) xp nd P R) P )) 223) = xp nr + γ R ) R γ R )]) 224) Now, i i y o vrify Shyviz 37, Scion IV-8] for xmpl) by diffrniing h convx funcion gq) := DQ P ) )HQ) h h unconrind minimum in h xponn in β n k h form of iling of P, i, Q ) := P ) P ) = P ) ), 225) Now dpnding on h vlu of, w hv wo diffrn cnrio Fir, if > R or quivlnly, Q GR c β, hn n = xp nr + gq )]) = xp nr + γ )]) uing 99) nd 200)) On h ohr hnd, if R, h opiml oluion in h opimizion in β n i gin ind h boundry of G R nd GR c i, h conrin Q Gc R i civ) Thu, β n = αn whr α n i in 224) In ummry, β n = xp nλ, R)) whr Λ, R) i dfind in 203) Now, clrly βn lwy domin α n i, β n i xponnilly l lrg α n ) Thi i bcu whn R, hy r h m, nd whn > R, w r king n unconrind minimum of gq) in h xponn, mking h ovrll xprion lrgr W hu obin h concluion in 202) For h mn in 204) 205), w um h R < H P ) nd 0, 0 P )] W clim h h imply h R, i, h fir clu in 203) i civ No from h dfiniion of 0 P ) in 54) h 0 P ) mn h H P ) H P ) ) 226) Sinc R < H P ), i hold h R < H P ) ), bu hi in urn impli h R bcu H P ) ) i monooniclly non-incring Thu 204) hold Th 204) i xponnilly qul o 205) follow from h fc h whn R < H P ) nd 0, 0 P )], β n i of h m xponnil ordr α n nd h lr i lowr boundd on h xponnil cl) by 2) Rmrk 2 To driv condiionl vrion of Lmm 4 o obin 32), w um h h yp of E n i Q E P n E) Th bov drivion go hrough nilly unchngd by vrging wih rpc o Q E vrywhr Spcificlly, w conidr nd ã o blong o vriou V E -hll T V E ) := n :, ) T QE V E } 6, Ch 2] Th nropi HQ) r rplcd by condiionl nropi HV E Q E ), h rliv nropi DQ P ) by condiionl rliv nropi DV E P E Q E ) nd h ild condiionl diribuion i dfind P ) E ) P E ) + nd o on For h nlogu of 204) 205) o hold, w firly rquir R < Q E)H P E= ) If w furhr um h 0 0 = min 0 P E= ), H P E= ) H P ) E )) for ll E, nd o R < Q E)H P ) E )) giving h fir clu in h condiionl nlogu of 203), i, h Λ, R) = R + DP R) E P E Q E )) whr R ifi HP R) E Q E) = R Th obrvion yild 32) upon vrging ovr ll yp on E PPENDIX C USEFUL CONCENTRTION BOUND Th following lmm i nilly rmn of Lmm 0 in 38] Lmm 5 L Y,, Y L b indpndn rndom vribl, ch king vlu in 0, } uch h PrY i = ) = p L N := L i= Y i For vry 0, ] nd ny 0 < ɛ <, EN ] Lp ɛ) xp L p2 )] ɛ2 227) In priculr, if Lp i qunc in n h nd o infiniy n nd o infiniy) xponnilly f, hn by king ɛ = /2 y) in 227), EN ] Lp) = EN]} 228) In fc, w hv EN ] = EN]} bcu EN ] EN]} by Jnn inquliy cknowldgmn VYFT i prilly uppord n NUS Young Invigor wrd R B37-33) nd Singpor iniry of Educion Tir 2 grn Nwork Communicion wih Synchronizion Error: Fundmnl Limi nd Cod R B6-2) H i prilly uppord by EXT Grn-in-id for Scinific Rrch ) No H i lo prilly uppord by h Okw Rrch Grn nd Kymori Foundion of Informionl Scinc dvncmn Th Cnr for Qunum Tchnologi i fundd by h Singpor iniry of Educion nd h Nionl Rrch Foundion pr of h Rrch Cnr of Excllnc progrmm