On Exponentially Concave Functions and Their Impact in Information Theory

Transcription

1 O Epoetally Cocave Fuctos ad Ther Impact Iformato Theory Abstract Cocave fuctos play a cetral role optmzato So-called epoetally cocave fuctos are of smlar mportace formato theory I ths paper, we comprehesvely dscuss mathematcal propertes of the class of epoetally cocave fuctos, lke closedess uder lear ad cove combato ad relatos to quas-, Jese- ad Schur-cocavty Iformato theoretc quattes such as self-formato ad scaled etropy are show to be epoetally cocave Furthermore, ew equaltes for the Kullback-Lebler dvergece, for the etropy of mture dstrbutos, ad for mutual formato are derved Ide Terms Iequaltes, cove optmzato, Schurcocave, quascocave, Shao theory, self-formato, etropy, mutual formato, dvergece, mture dstrbuto I INTRODUCTION More tha 00 years ago, Joha Jese publshed hs semal paper [] whch the fudametals of covety are vestgated Nowadays, cove ad cocave fuctos are cetral compoets both optmzato theory ad set theory Furthermore, the cocept of covety ad cocavty of fuctos s eteded varous drectos to tackle the eeds of physcsts, egeers ad mathematcas For eample, the class of logarthmcally cove fuctos provdes more accurate bouds ad equaltes tha the oes derved from cove fuctos, see [2], [3] Moreover, covety ad cocavty allow for very elegat proofs equalty theory, cf [4] I cotrast to logarthmcally cove log-cove fuctos, ther couterpart, the so-called epoetally cocave epcocave fuctos, are rarely dscussed the lterature, probably due to ther trcate structure Epoetally cocave fuctos play a mportat role formato theory, as we wll see later For stace, the scaled dscrete etropy, as a cove combato of logarthms, s epoetally cocave It s eactly ths type of combato that makes the mathematcal vestgato of epoetally cocave fuctos cumbersome Besdes the applcato formato theory, a thorough vestgato of epoetally cocave fuctos s of geeral mathematcal terest The growth of research o bg data aalytcs ad deep learg has recetly creased the terest epoetally cocave fuctos I [5], the smoothess of epoetally cocave fuctos s eploted for statstcal learg, sequetal predcto ad stochastc optmzato, whch are mportat topcs mache learg Lower ad upper bouds o solutos of stochastc epoetally cocave optmzato problems are dscussed [6] Emprcal rsk mmzato, Gholamreza Alrezae ad Rudolf Mathar Isttute for Theoretcal Iformato Techology RWTH Aache Uversty, D Aache, Germay {alrezae, mathar}@trwth-aachede whch s a geeral optmzato framework that captures several mportat learg problems cludg lear ad logstc regresso, learg support vector maches SVMs wth the squared hge-loss, ad portfolo selecto, see [7], s vestgated for epoetally cocave loss fuctos [8] To the best of our kowledge, a comprehesve vestgato of epoetally cocave fuctos as the preset paper s ew I the preset paper, we preset basc propertes of epoetally cocave fuctos ad equaltes Afterwards, formato theoretc quattes are cosdered We especally show that both self-formato ad the scaled dscrete etropy are epoetally cocave fuctos Fally, these results are appled to deduce a ew lower boud o the Kullback- Lebler dvergece, ew etropy equaltes for mtures of dstrbutos, ad equaltes for the mutual formato of dscrete chaels II PRELIMINARIES Let D be a cove 2 subset 3 of R A fucto g : D R s called cocave, f βg + βgy gβ + βy holds for all, y D ad for all β [0, ] wth β = β Fucto g s called strctly cocave, f strct equalty holds for all y ad for all β 0, A fucto s called strctly cove, f ts egatve s strctly cocave, cf [2] Note that cove ad cocave fuctos are ecessarly cotuous fte spaces If addto g s dfferetable, the the equalty gy g g T y 2 holds for all, y D For a twce dfferetable fucto g, a equvalet defto of cocavty s gve by v T 2 g v 0 3 for ay vector v R ad for all D The set of atural, real, ad oegatve real umbers are deoted by N, R ad R +, respectvely The sets a, b, a, b], ad [a, b] are the ope, the half-ope, ad the closed terval, respectvely Vectors are wrtte boldface ad ther traspose s dcated by the superscrpt T Gradet ad Hessa of a fucto f are show by f ad 2 f, respectvely 2 A set D s cove f the le segmet jog ay two pots D s part of D 3 Uless otherwse stated, we cosder fte-dmesoal sub-spaces ad fte sub-sets wth cardaltes deoted by m N ad N throughout the preset paper I few cases we also allow coutably fte spaces

2 The cocept of covety ad cocavty ca be eteded varous drectos A crucal ad well-kow represetatve s the class of logarthmcally cove fuctos, from whch tghter bouds optmzato theory evolve As we wll see later, ts couterpart, the class of epoetally cocave fuctos plays a mportat role formato theory The defto s aalogous to the oe of logarthmcally cove fuctos A fucto f : D R s epoetally strctly cocave D, f the fucto ep f s strctly cocave Hece, equvalet deftos for epoetally cocave fuctos are obtaed by applyg, 2 ad 3 to ep f Straghtforward algebra yelds the followg: a equvalet defto 4, y D, β [0, ] : log β ep[f] + β ep[fy] fβ + βy, 4 b assumg dfferetablty, y D : fy f log + f T y, 5 c assumg twce-dfferetablty D, v R : v T 2 f + f f T v 0 6 Comparg -3 wth 4-6 shows that tghter equaltes for epoetally cocave fuctos est I partcular, we have ad βf + βfy log β ep[f] + β ep[fy] fβ + βy, fy f log + f T y f T y, v T 2 f v v T 2 f + f f T v 0 9 Iequaltes 7-9 also show that every epoetally cocave fucto s a cocave fucto, too By usg mathematcal ducto, we ca geeralze equalty 4 as follows Corollary Let f : D R be a epoetally cocave fucto o the cove set D R The for all w R + wth w = the equaltes w f log w ep f f w 0 hold O the rght sde, equalty s attaed for f = log c 0 + j c j j wth properly chose costats c0, c, Note that the left equalty 0 s a smple cosequece of Jese s equalty However, we have observed that especally for epoetally cocave fuctos the left equalty usually acheves smaller gaps compared to the rght oe whch s terestg ts ow 4 The base of the logarthm s the Euler s umber e whle ep refers to the atural epoetal fucto Wheever the base becomes mportat, we wrte log a ad a stead of log ad ep, respectvely, to hghlght the correspodg base a of both the logarthm ad the epoetal fucto 7 8 A geeralzato of 5 yelds a squeeze-equalty for the dfferece f j w j j w f, whch s the cotet of the et two corollares By choosg y = ad = j w j j, ad multplyg both sdes of 5 wth w ad summg up over all, the followg lower boud s derved Corollary 2 Let f : D R be a epoetally cocave fucto o the cove set D R The for all w R + wth w = the double-equalty 0 T w log + f j w j j w j j j f j w j j w f holds The logarthmc terms o the lefthad sde of are always egatve, as ca be see by elargg ther argumet by the ad of AM-GM equalty, cf [4] By replacg = ad y = j w j j 5 ad proceedg aalogously to the above, we obta the followg upper boud Corollary 3 Let f : D R be a epoetally cocave fucto o the cove set D R The for all w R + wth w = the equalty f j w j j w f w log f T 2 w j j j holds Itegral versos of 0-2 ca be derved uder addtoal costrats o f, eg, boudedess ad Rema tegrablty We wll ow dscuss mathematcal propertes of epoetally cocave fuctos III MATHEMATICAL ATTRIBUTES I the followg we wll show that uder certa assumptos the shft, scale, combato, ad composto of epoetally cocave fuctos preserve epoetal cocavty Proposto 4 Let f be a epoetally cocave fucto Addg a costat c R to f or multplyg f by a factor c 2 [0, ] preserves ts epoetal cocavty Proof The proof s easy, sce ep c + f = epc ep f ad ep c 2 f = [ ep f ] c 2 are cocave fuctos for ay c R ad ay c 2 [0, ] wheever ep f s cocave For certa fuctos the multplcatve costat may be greater tha oe, sce for a twce-dfferetable ad epoetally cocave fucto f : D R R we obta for f cf the largest possble costat by c ma = f D [f ] 2 from 6 For epoetally cocave fuctos c ma s ecessarly greater tha or equal to oe Hece, fuctos wth c ma < caot be epoetally cocave

3 Proposto 5 The cove combato w f of epoetally cocave fuctos f : D R defed o cove sets D R wth weghts w R +, w =, s epoetally cocave o D Proof We show that 4 holds for w f By usg the geeralzed Hölder s equalty, cf [4], we obta the followg cha of equaltes [ ] log β ep w f + β [ ] ep w f y = log β ep[w f ]+ β ep[w jf j y ] j log β ep[f ] + β ep[f y ] w = w log β ep[f ] + β ep[f y ] w f β + βy, whch completes the proof Proposto 6 The sum f + f 2 of epoetally cocave fuctos f, f 2 : D R, o the cove set D R, s epoetally cocave, f [f f y][f 2 f 2 y] 0 3 for all, y D holds For = relato 3 holds f f ad f 2 are cotra-mootoc Proof Sce the epoetal fucto s mootoc, relato 3 s equvalet to ep f + f 2 + ep f y + f 2 y ep f + f 2 y + ep f y + f 2 4 By usg the epoetal cocavty of f ad f 2, ad applyg 4 afterwards, we fer ep f β + βy + f 2 β + βy [ β ep f + β ep f y ] [β ep f 2 + β ep f 2 y ] = β 2 ep f + f 2 + β 2 ep f y + f 2 y + β β [ ep f + f 2 y + ep f y + f 2 ] β ep f + f 2 + β ep f y + f 2 y, whch s equvalet to 4 ad thus completes the proof Proposto 7 Let f : D f R be a epoetally cocave fucto o the cove set D f R m Let g : D g D f have compoets g, m, where D g R s a cove set If f s o-decreasg each argumet ad g cocave for all, or f s o-creasg each argumet ad g cove for all, the the composto f g s epoetally cocave Proof By applyg equalty 4 to f, the usg the mootocty of f coecto wth cocavty or covety of each g, we obta log β ep [ f g ] + β ep [ f gy ] f βg + βgy f g[β + βy], whch proves the asserto A obvous cosequece of Proposto 7 s that log w g s epoetally cocave for ay cove combato w g of cocave fuctos g A degeerate case of Proposto 7 s whe each g becomes the detty fucto I ths case we obta the followg statemet Proposto 8 Let f : D f R be a epoetally cocave fucto o the cove set D f R m The the composto f w, wth D f, w 0 ad w =, s a epoetally cocave fucto of, 2, Proof By applyg equalty 4 to f ad usg the learty of w, we obta [ ] log β ep f w + β [ ] ep f w y f β w + β w y f w [β + βy ], whch shows the asserto We wll ow vestgate the relato betwee epoetally cocave ad Schur-cocave fuctos It s well-kow that symmetrc cocave fuctos are Schur-cocave, see [4] wth may etesos [9] Thus, for symmetrc ad epoetally cocave fuctos oe ca easly show the Schur-cocavty The coverse, that a Schur-cocave fucto s epoetally cocave, s ot true geeral ad eeds addtoal assumptos, oe of whch s troduced the et proposto Proposto 9 Let D be a terval ad f a cotuous real fucto o D If the fucto φ, 2 = ep f + ep f 2 s Schur-cocave o D 2, e, φy, y 2 φ, 2 for all matrces 5, 2 that are row majorzed 6 by y, y 2, the the fucto f s epoetally cocave o D Proof Sce φ s Schur-cocave, t holds that φy, y 2 φ, 2 for the specfc choce = 2 = y +y 2 /2 Ths yelds ep fy + ep fy 2 2 ep fy /2 + y 2 /2, whch shows that ep f s a mdcocave fucto of Sce f s cotuous, the cocavty of ep f follows from Jese s theorem, see [3, p 25] Hece, f s epoetally cocave A smple coecto to quascocave fuctos s cosdered et, whch shows that epoetally cocave fuctos are as well quascocave Proposto 0 The epoetal fucto of ay cocave fucto s quascocave Proof Let f : D R be a cocave fucto o the cove set D R The the equalty βf + βfy 5 Note that, 2, y ad y 2 are colum vectors such that X =, 2 ad Y = y, y 2 are 2 matrces 6 The statemet Proposto 9 remas vald for dfferet multvarate majorzato techques, eg, the cha majorzato or the ordary majorzato, wheever the row majorzato s mpled by them, cf [9, p 620]

4 fβ + βy holds for ay, y D, β [0, ] For g = ep f to be a quascocave fucto of D we have to show the equalty m{g, gy} gβ + βy for ay, y D, β [0, ] Assumg wlog that g gy, whch s equvalet to f fy, we coclude g = ep βf + βf ep βf + βfy ep fβ + βy = gβ + βy, whch completes the proof The coverse of the above statemet does ot hold as ca be see from smple eamples The perspectve of epoetally cocave fuctos ca be defed aalogously to [0, p 89] Ufortuately, the correspodg perspectve s ot homogeeous Proposto Let f : D R be a epoetally cocave fucto o the cove set D R The ts perspectve gy, = logy + f y s also epoetally cocave o R + D Proof The asserto s prove by the followg cha of equatos ep [ gβy + βy 2, β + β 2 ] [ = ep logβy + βy β f β ] 2 βy + βy 2 =βy + βy [ βy βy 2 ] 2 2 ep f βy + βy + 2 y βy + βy 2 y [ 2 ] βy ep f + y βy [ 2 ] 2 ep f y 2 = β ep [ gy, ] + ep [ gy 2, 2 ], whch satsfes 4 Sce epoetally cocave fuctos are cocave, mamzato of epoetally cocave fuctos has the advatage that every local optmum s also a global mamum Moreover, epoetally cocave fuctos ca be upper-bouded by cocave ad suffcetly smooth hypersurfaces Proposto 2 Let f : D R be a epoetally cocave fucto o the cove set D R The the equalty fy f + log + v T y holds for all, y D ad proper vector v Proof It s well-kow that for ay cocave fucto g at each pot D there ests some ṽ such that gy g + ṽ T y for all y D If g s dfferetable at the ṽ = g may be chose Replacg g by epf ad applyg the logarthm o both sdes yelds fy f + log + ep fṽ T y wth v = ep fṽ, whch proves the statemet, cf 5 Cosderg the argumet of the logarthm equato 5, a ew ecessary codto for epoetally cocave ad dfferetable fuctos s obtaed Corollary 3 For a epoetally cocave ad dfferetable fucto f : D R the equalty fy T y 5 holds for all, y D Partcularly for = 0, the relato fy T y lmts the er product betwee the slope ad the posto of ay pot y Corollary 3 provdes a quck test for ecludg fuctos from the class of epoetally cocave fuctos For eample, the fucto log + s epoetally cocave, but the sum log + + s ot Sce multplyg the dervatve [log + + ] = 2+ + by y ad choosg y = 2 + yelds 2 + > for all > Hece, equalty 5 s volated, whch meas that log+ + s ot epoetally cocave The coverse of Corollary 3 s ot vald geeral as smple eamples show IV EXPONENTIAL CONCAVITY OF INFORMATION THEORETIC QUANTITIES I ths secto, we show epoetal cocavty of formato theoretc quattes lke self-formato ad etropy We start wth the self-formato ad wll cosequetly show that the scaled etropy s epoetally cocave whch results a equalty for the etropy of mtures of dscrete dstrbutos Theorem 4 Self-formato ρ = log s epoetally cocave for [0, ] Proof Sce ρ s dfferetable, we cosder the secod dervatve ep ρ = ep ρ v wth v = log 2 e as well as v = + 2 loge ad v = 2 2 loge Due to the equalty log y y, we ca elarge v to obta v 2 3 e < 0 Sce v s egatve, v s cocave ad v s decreasg Wth v = 3 > 0 t follows that v s postve ad hece v s creasg Wth v = 0 we deduce that v s o-postve Thus ep ρ 0 yelds epoetal cocavty of ρ for [0, ] Sce self-formato s epoetally cocave, we ca smply use Proposto 5 to show the followg equalty Corollary 5 The weghted etropy [], defed by H, u = u log, wth u, [0, ] ad u = =, s epoetally cocave for all, e, β ep u ρ + β ep u ρy holds for ay β [0, ] ep u ρ β + βy 6 The compostos ep H 2, ad ep ρ of the epoetal fucto wth both the bary etropy ad the selfformato are depcted Fgure

5 Fg : The fuctos ep H 2, ad ep ρ are show by a blue dashed ad a red sold curve, respectvely I addto to the above relatoshp, we ca deduce smlar equaltes for the ordary etropy for a lmted umber of dmesos I partcular, we show that oly the bary ad the terary etropes are epoetally cocave Theorem 6 The etropy H = log, wth [0, ] ad =, s epoetally cocave oly for {2, 3} Proof Sce ρ, ad cosequetly H, s dfferetable, we cosder the secod dervatve of gβ = ep ρ β + βy, order to show that g s cocave betwee ay two feasble pots ad y as log as the umber of elemets s less tha four The secod dervatve of g reads as g β = gβvβ wth vβ gve by ρ β + βy 2 y + ρ β + βy y 2 Sce g s oegatve, we oly eed to check the sg of v for the proof By the substtutos z = β + βy ad d = y we obta 2 v d, z = d logz d 2 z Note that z has to fulfll z = ad z 0 for all whle d = 0 ad d for all have to be take to accout Frst we show that for all 4 the fucto v d, z ca be postve for partcularly chose d ad z whch dsproves the epoetal cocavty of etropy H for all 4 Cosder the choce 0 = ε 2, ε 2, ε 2,, 2 ε ad y0 = ε 2, ε 2, ε 2,, 2 ε wth a suffcetly small ε > 0 wth whch we determe d 0 = ε, ε, ε,, ε ad z0 = z, z, z,, z wth z = β + 2β ε 2 For 0 < β < 2, correspodg to 0 < z <, t leads to the quatty v d 0, z 0 νz, = 2 = log 2 z ε z z z, whch s a fucto of z ad, ad s creasg Selectg z = yelds ν 85 00, = log log > log2 e 8 8 = 0, where the frst equalty arses from the mootocty ad the secod from modfcato of the costats Hece, we obta that νz,, ad cosequetly v d, z, ca be postve for all 4 whch dsproves the epoetal cocavty of etropy for 4 Now, we show that v 3 d, z s always egatve order to prove the epoetal cocavty of the terary etropy Therefore we apply the Cauchy-Buyakovsky-Schwarz equalty, cf [4], o the frst term v 3 d, z to obta the equalty z d logz 2 = d log λ 3 d z log z 2 z λ 3 d 2 3 z log 2 z, z λ for ay postve λ Rearragg z such that z z 2 z 3 holds ad choosg λ = z 3, t leads to 3 z log 2 z [ z = z 3 log 2 z + z 2 log 2 z ] 2 λ z 3 z 3 z 3 z 3 Note that the fucto y log 2 y + y 2 log 2 y 2 s Schurcocave y, y 2 ad hece we obta the upper boud [ z z 3 log 2 z + z 2 log 2 z ] 2 z 3 z 3 z 3 z 3 z 3 [ 2 z 3 2z 3 log 2 z 3 2z 3 ] z 3 log 2 z 3, 2z 3 for all 3 z 3 The last quatty ca smply be mamzed by umercal methods due to ts quascocavty Ths leads to a upper boud of < at z3 = Hece, the equalty d d logz 2 3 z log 2 z z a 3 holds, whch proves v 3 d, z 0 Thus, the terary etropy s epoetally cocave It remas to show the epoetal cocavty of the bary etropy, whch ca easly be doe by forcg d = 0 ad the followg the same steps as for the proof of terary etropy As ca be see from Corollary 5 ad Theorem 6 there must est a value <, depedg o the umber of dmesos, such that H becomes epoetally cocave for all > The et theorem uses ths prcple ad hece mproves ad cosoldates Corollary 5 ad Theorem 6 d 2 z

6 Theorem 7 Let H be the etropy as defed Theorem 6 The the fucto H s epoetally cocave for all > wth ma z z log2 z /<z< z Proof Aalogously to the proof of the terary etropy from Theorem 6, we have to show that the fucto gβ = ep ρ β + βy s cocave β, or equvaletly we ca show that v d, z = 2 d d logz 2 z s o-postve Aga we elarge v d, z by applyg the Cauchy-Buyakovsky-Schwarz equalty to obta v d, z = z 2 d 2 d log λ z d 2 z log 2 z d 2 z λ z d 2 = z log 2 z z λ By optmzg λ the gap of the above equalty ca be reduced We frst cosder the case λ = whch yelds a weak lower boud for Afterwards we optmze λ to obta a sharper lower boud as stated the asserto Wlog we assume z z 2 z what follows For λ = the sum z log 2 z s Schur-cocave z whch yelds z log 2 z log 2 at z = From v d, z 0 we hece obta log 2 0 ad tur log 2, whch ca be mproved as follows For λ z the fucto y log 2 y λ s quascocave y for y λ ad cove for y λ Thus, mamzg z log 2 z λ subject to z =, z 0, wll atta ts mamum at z = z2 = = z = z λ z, e, z log 2 z λ z log 2 z λ +z log 2 z λ Sce the last term s quascove λ we ca reduce the gap by mmzg over λ Wth λ = z z z z we ed up z log 2 z λ + z log 2 z λ = z z log 2 z z From v d, z 0 we hece obta z z log 2 z z 0 ad tur ma z z log2 z /<z< z Note that by usg the relato betwee geometrc ad logarthmc meas oe ca smply show z z log 2 z z log 2 log 2 Table I lsts some eamples for comparso Aalogously to Theorem 7, the epoetal cocavty of Réy etropy [2], [3], scaled by a umber, ca also be show wth some more effort The followg fuctoal structures are key elemets for dealg wth the mutual formato of certa fudametal chael models, see [4] [7] TABLE I: Multplcatve factors of the etropy for becomg epoetally cocave log 2 z ma z z log2 /<z< z = = = = = = = = = = Theorem 8 The dfferece ρ βγ + βγ 2 βργ βργ 2 s epoetally cocave β [0, ] for all γ, γ 2 [0, ] Proof Sce self-formato s dfferetable, we show that equalty 6 holds Let fβ = ρ βγ + βγ 2 βργ βργ 2 ad vβ = f β + [f β] 2 If fβ s epoetally cocave β, the vβ 0 must hold By smple calculato we obta f β = γ γ 2 ρ βγ + βγ 2 ργ ργ 2 ad f β = γ γ 2 2 ρ βγ + βγ 2 wth ρ β = log β ad ρ β = β It s easy to check that vβ = 0 for γ = γ 2, so o further vestgato s eeded for ths case Sce f β s o-postve, we ca represet vβ the form f β f β f β + f β Now we oly cosder the case γ > γ 2 the followg, sce the opposte case ca be treated aalogously The we have ad f β f β = γ + ργ γ 2 ργ 2 γ γ 2 γ0 + log γ 0 7 f β + f β = γ + ργ γ 2 ργ 2 + γ γ 2 γ0 log γ 0 8 wth γ 0 = βγ + βγ 2 Cosder ow the fucto γ+ργ wth [γ+ργ] = log γ 0 for all γ 0, ] Sce ts dervatve s o-egatve, the fucto γ + ργ s creasg γ Ths meas that the quatty γ + ργ γ 2 ργ 2 s oegatve for γ > γ 2 It s easy 7 to show that γ0 log γ 0 + log γ 0 2 log 2 0 for γ 0 [0, ] Hece, the fucto 7 s egatve Sce γ0 log γ 0 s decreasg γ0 7 The fucto γ + log γ has the dervatve 2 γ 2γ 3/2, whch s postve for all γ > /4 ad o-postve otherwse Hece, γ = /4 s a global mmum, whch leads to γ + log γ 2 log 2 0 I addto, sce γ [0, ], t holds γ log γ γ + log γ

7 γ 0, we mamze γ 0 by replacg t wth γ to obta f β + f β γ + ργ γ 2 ργ 2 + γ γ 2 γ log γ = γ γ2 γ γ + γ 2 log γ γ 2 0 Hece, the product of f β f β wth f β + f β s egatve, whch shows the egatvty of vβ Ths completes the proof For the sake of compactess, we hereafter deote the bary etropy H 2, by H Theorem 9 The dfferece Hβγ + βγ 2 βhγ βhγ 2 s epoetally cocave β [0, ] for all γ, γ 2 [0, ] Proof We cosder the fucto fβ = Hβγ + βγ 2 βhγ βhγ 2 ad ts frst ad secod dervatves wrt β as gve by ad f β = H βγ + βγ 2 γ γ 2 Hγ + Hγ 2 9 f β = H βγ + βγ 2 γ γ 2 2, 20 where H = log ad H = Wthout loss of geeralty we assume γ 2 < γ, sce for the case γ = γ 2 the fucto f becomes zero Smlar to the above proof, we defe vβ = f β+[f β] 2 ad show that v s o-postve to prove the epoetal cocavty of f The decomposto of v yelds f β f β = γ γ γ0 γ 0 Hγ Hγ 2 ad γ0 + γ γ 2 log γ 0 2 f β + f β = + γ γ 2 γ0 γ 0 Hγ Hγ 2 γ0 + γ γ 2 log γ 0 22 wth γ 0 = βγ + βγ 2 Note that both fuctos log γ 0 γ 0 + ad log γ 0 γ0 γ 0 γ 0 are decreasg γ 0 γ0 γ 0 for γ 0 2 ad γ 0 2, respectvely, sce the dervatves are o-postve, e, γ0 log ± γ 0 γ 0 γ0 γ 0 = ± γ 0 γ 0 2γ γ 0 γ 0 I the followg we oly cosder the partcular case γ 2 < γ < γ 2 alog wth the subcases γ 0 2 ad γ 0 2 The case γ 2 < γ 2 < γ ca be prove smlarly, after multplcato of both 2 ad 22 wth mus oe I the subcase γ 2 < γ < γ 2 wth γ 0 2, we have log γ 0 γ 0 0, γ 2, ad Hγ > Hγ 2 0 whch lead to f β f β 0 Usg addto the above mootocty we obta f β+ f β + γ γ γ γ Hγ Hγ 2 γ + γ γ 2 log γ γ = Hγ 2 γ 2 log γ }{{} + γ γ γ γ 0 Hece, the product vβ = f β f β f β + f β s o-postve I the subcase γ 2 < γ < γ 2 wth γ 0 2, we have log γ 0 γ 0 0 ad Hγ > Hγ 2 whch lead to f β f β γ γ γ0 γ 0 Hγ Hγ 2 2γ 0 + γ γ 2 γ0 γ 0 γ γ γ0 γ 0 Hγ Hγ 2 + γ γ γ0 γ 0 = Hγ Hγ 2 0, where we have used the equalty log 2 2 from Proposto 26 Wth more effort we also deduce f β+ f β + γ γ γ2 γ 2 Hγ Hγ 2 γ2 + γ γ 2 log γ 2 = + γ γ γ2 γ 2 log γ2 γ }{{ } γ2 γ + γ log γ γ 2 + γ γ γ2 γ 2 γ γ γ γ2 γ2 γ + γ log γ γ 2 + γ γ γ2 γ 2 γ γ γ2 γ 2 γ2 γ γ2 + γ log = γ log γ γ 2 γ }{{} γ γ 2 }{{} 0, where we aga have used the equalty log 2 2 ad γ 2 < γ, that follows from γ < γ 2 Thus, the product vβ = f β f β f β+ f β s aga opostve I summary, vβ o-postve, whch proves the epoetal cocavty of fβ

8 As ca smply be show by coutereamples, the dffereces H β γ β Hγ uder the costrats β = ad β 0 ca ever be epoetally cocave β for ay 3 As a eample we cosder the fucto ep [ H 3 β 3 γ β Hγ ], whch leads to the obvously cove fucto ep [ β 2 H 4 5] for the partcular choce γ = 0, γ 2 = 4 5, γ 3 =, β = β2 5, ad β 3 = 4 β2 5 Epoetal cocavty ca be eteded may more drectos, eg, to Jese-Steffese or Hermte-Hadamard-lke equaltes ad eve to dfferetal etropy, whch are devoted to future works V APPLICATION OF EXPONENTIAL CONCAVITY Epoetal cocavty s a useful tool for provg formato theoretc equaltes ad bouds Ths s demostrated the preset secto, where we derve e ew bouds Proposto 20 Let H be the etropy ad be as gve Theorem 7 Let D y = log y be the Kullback-Lebler dvergece,, y [0, ], = y = It holds that D y log + H y T y H +H y 0 23 Proof By smple calculato, we observe the detty D y = H y T y H + H y Comparg the above detty wth equalty 8 ad recallg that H s epoetally cocave, yelds the lower boud The etropy power equalty s a well-kow equalty to descrbe the relatoshp betwee the etropes of two depedet radom varables ad ther sum, cf [8] [20] I the et two propostos, we carry over ths prcple to derve ew equaltes betwee the weghted etropes of radom varables ad ther mture dstrbutos Proposto 2 Cosder the weghted etropes H, u of m probablty vectors R + ad correspodg weghts u R + The the equaltes m m ep w H, u w ep H, u hold for all w [0, ] wth m w = m 24 ep H w, u Proof Sce the weghted etropy fucto s epoetally cocave as stated Corollary 5, we ca use equalty 0 to deduce the asserto Other useful equaltes for comparg the etropes of two dstrbutos are the followg oes Proposto 22 Cosder the weghted etropes H, u ad H y, u of probablty vectors, y, u R + The the equalty ep H y,u u y logy holds ep H, u u y log 25 Proof Sce the weghted etropy fucto s epoetally cocave as stated Corollary 5, the frst dervatve of gβ = ep H β + βy, u wrt β must be decreasg, e, the equalty g g 0 holds Ths completes the proof Smlar to the last statemet, we ca deduce the followg asserto Proposto 23 Cosder the weghted etropes H, u ad H y, u of probablty vectors, y, u R + The the equalty ep H y, u H, u + u y log holds 26 Proof Sce the weghted etropy fucto s dfferetable ad epoetally cocave as stated Corollary 5, the equalty 5 holds whch s equvalet to 26 We ca further sharpe 24, 25 ad 26 by eplotg the epoetal cocavty of H, as descrbed Theorem 7 Hece, we derve m m ep w H ep H y y logy ad w ep H m ep H w, 27 ep H 28 y log ep H y H + y log 29 It s well-kow that mutual formato I 2 β of dscrete bary chaels s gve by I 2 z = H zγ + zγ 2 zhγ zhγ 2 30 for the trasto matr γ γ γ 2 γ 2 wth γ, γ 2 [0, ] The etry the th row ad the y th colum of the matr deotes the codtoal probablty that y s receved whe s set The probablty of the two put symbols are deoted by z ad

9 z, cf [2], [22] Note that the mutual formato of bary symmetrc chaels BSC, bary asymmetrc chaels BAC, ad the oe-bt quatzer are specal cases of 30 The correspodg capacty s derved [23] ad eteded [4] by a decoder specfc quatty The et theorem helps to fd proper equaltes for such classes of chaels Proposto 24 For the mutual formato I 2 of dscrete bary chaels, the equalty ep w I 2 z w ep I 2 z 3 ep I 2 w z holds for all z, w [0, ] wth w = Proof Due to Theorem 9, the mutual formato I 2 z s epoetally cocave By the ad of 0 we fer the statemet Cosderg the geeralzed mutual formato I z, γ, γ 2, = H z j γ j z j H γ j 32 j j depedg o the put dstrbuto z for gve probablty vectors γ j Sce Hγ s epoetally cocave as stated Theorem 7, we ca fd lower ad upper bouds for I z, γ, γ 2, by the ad of Corollares 2 ad 3 as follows Corollary 25 For the mutual formato I of dscrete chaels, the equalty cha T z log + H z jγ j γ z jγ j j j I z, γ, γ 2, z log H γ T γ z jγ j j holds for all probablty vectors z, γ, γ 2, 33 VI SIDE INEQUALITIES FROM EXPONENTIAL CONCAVITY I the prevous sectos, we have vestgated several equaltes ad provded dfferet proofs for them, from whch we easly ca fer the followg equaltes Proposto 26 The equalty log 2 2 holds for all > 0 wth equalty at = 34 Proof Sce the bary etropy s epoetally cocave, the quatty v = H + H 2 = log 2 s o-postve Moreover, dscussg u = 4+v 0 alog wth ts frst ad secod dervatves reveals that actually u s o-postve Substtutg by u 0 yelds the equalty uder cosderato Suppose that the weghted arthmetc, geometrc, ad harmoc meas are defed by A j= j, w j = w j j, 35 j= ad G j= j, w j = j= j= wj j 36 H w j j= j, w j =, 37 j for ay R ad w R + wth j= w j = The the famous equalty cha H j= j, w j G j= j, w j A j= j, w j s well-kow, cf [24] Oe ca fd may mprovemets for ther relatoshp the lterature Due to the epoetal cocavty we have acheved a ew med mea equalty, whch s the ve of Herc s, Najudah s ad Serpsk s equalty [25], ad s precsely stated the et proposto Proposto 27 Let X be a real m matr wth oegatve elemets,j wth j=,j = for all Let w R m + wth m w = The wth from Theorem 7 we have [ G j= A m,j, w,a m,j, w ] / H m [ G j=,j,,j ] /, w 38 Proof From Theorem 7, we kow the epoetal cocavty of the scaled etropy Remdg the relatoshp ep k ρy k = k y y k k, we fer m c w,j j= m w,j m w j=,j c,j, from 27 whch s equal to 38 There are may epoetally cocave fuctos, that are mportat commucato theory Oe of the famous oes s the error-fucto erf = 2 π 0 e t2 dt, 39 that becomes epoetally cocave the form erf for all 0 Its complemetary usually descrbes the bt/symbol error probablty of commucato systems depedg o the square root of the uderlyg sgal-to-ose rato Ths eample shows that epoetally cocave fuctos ca have a crucal role also commucato theory VII CONCLUSION Epoetally cocave fuctos seem to play a mportat role formato theory Sce they are rarely dscussed the lterature, we have vestgated ther mathematcal propertes alog wth ther geeral applcatos Especally the self-formato ad the scaled dscrete etropy have bee dscussed ad t has bee show that they are epoetally cocave fuctos I addto, we have derved ew equaltes for the Kullback-Lebler dvergece, the etropy of mtures of dstrbutos, ad the mutual formato of dscrete chaels

10 REFERENCES [] J L W V Jese, Sur les foctos covees et les égaltés etre les valeurs moyees, Acta Mathematca, vol 30, o, pp 75 93, 906 [2] R T Rockafellar, Cove Aalyss, ser Prceto ladmarks mathematcs ad physcs Prceto Uversty Press, 997 [3] A W Roberts ad D E Varberg, Cove Fuctos, ser Pure ad Appled Mathematcs; a Seres of Moographs ad Tetbooks, 57 Academc Press, 973 [4] G H Hardy, J E Lttlewood, ad G Pólya, Iequaltes, ser Cambrdge Mathematcal Lbrary Cambrdge Uversty Press, 952 [5] M Mahdav, Eplotg Smoothess Statstcal Learg, Sequetal Predcto, ad Stochastc Optmzato East Lasg, MI, USA: Mchga State Uversty, 204 [6] M Mahdav, L Zhag, ad R J, Lower ad upper bouds o the geeralzato of stochastc epoetally cocave optmzato, Proceedgs of The 28th Coferece o Learg Theory Joural of Mache Learg Research JMLR, 205, pp [7] S Pal, Epoetally cocave fuctos ad hgh dmesoal stochastc portfolo theory, ArXv e-prts, Mar 206 [8] T Kore ad K Y Levy, Fast rates for ep-cocave emprcal rsk mmzato, Proceedgs of the 28th Iteratoal Coferece o Neural Iformato Processg Systems, ser NIPS 5 Cambrdge, MA, USA: MIT Press, 205, pp [9] A W Marshall, I Olk, ad B C Arold, Iequaltes: Theory of Majorzato ad Its Applcatos, 2d ed New York: Sprger, 20 [0] S Boyd ad L Vadeberghe, Cove Optmzato New York, NY, USA: Cambrdge Uversty Press, 2004 [] S Guaşu, Weghted etropy, Reports o Mathematcal Physcs, vol 2, o 3, pp 65 79, 97 [2] Z Daróczy, Über Mttelwerte ud Etrope vollstädger Wahrschelchketsverteluge, Acta Mathematca Hugarca, vol 5, pp , 964 [3] I Csszár ad J Körer, Iformato Theory, Codg Theorems for Dscrete Memoryless Systems, 2d ed Cambrdge Uversty Press, 20 [4] G Alrezae ad R Mathar, Optmum oe-bt quatzato, The IEEE Iformato Theory Workshop ITW 5, Jeju Islad, Korea, Oct 205 [5], A upper boud o the capacty of cesored chaels, The 9th Iteratoal Coferece o Sgal Processg ad Commucato Systems ICSPCS 5, Cars Barrer Reef, Australa, Dec 205 [6], O the formato capacty of hge fuctos, The IEEE Iteratoal Symposum o Iformato Theory ad Its Applcatos ISITA 6, Moterey, Calfora, USA, 206 [7] A Behbood, G Alrezae, ad R Mathar, O the capacty of cesored chaels, th Iteratoal ITG Coferece o Systems, Commucatos ad Codg SCC 7, Hamburg, Germay, Feb 207 [8] T M Cover ad J A Thomas, Elemets of Iformato Theory New York, NY, USA: Wley-Iterscece, 99 [9] V Jog ad V Aatharam, O the geometry of cove typcal sets, 205 IEEE Iteratoal Symposum o Iformato Theory ISIT, Jue 205, pp [20] T A Courtade, Stregtheg the etropy power equalty, 206 IEEE Iteratoal Symposum o Iformato Theory ISIT, July 206, pp [2] C E Shao, A mathematcal theory of commucato, The Bell System Techcal Joural, vol 27, o 3, pp , July 948 [22] S Muroga, O the capacty of a dscrete chael I, Joural of the Physcal Socety of Japa, vol 8, o 4, pp , 953 [23] R Slverma, O bary chaels ad ther cascades, IRE Trasactos o Iformato Theory, vol, o 3, pp 9 27, December 955 [24] P S Bulle, Hadbook of Meas ad Ther Iequaltes, 2d ed, ser Mathematcs ad Its Applcatos Sprger Netherlads, 2003 [25], Dctoary of Iequaltes, 2d ed, ser Chapma & Hall/CRC Moographs ad Research Notes Mathematcs CRC Press, 205