Estimation of entropy rate and Rényi entropy rate for Markov chains

Size: px

Start display at page:

Download "Estimation of entropy rate and Rényi entropy rate for Markov chains"

Lilian Lester
5 years ago
Views:

1 06 IEEE Intenational Symposium on Infomation Theoy Estimation of entopy ate and Rényi entopy ate fo Makov chains Sudeep Kamath, Segio Vedú Depatment of Electical Engineeing Pinceton Univesity sukamath, Abstact Estimation of the entopy ate of a stochastic pocess with unknown statistics, fom a single sample path is a classical poblem in infomation theoy. While univesal estimatos fo geneal families of pocesses exist, the estimates have not been accompanied by guaantees fo fixed-length sample paths. We povide finite sample bounds on the convegence of a plug-in type estimato fo the entopy ate of a Makov chain in tems of its alphabet size and its mixing popeties. We also discuss Rényi entopy ate estimation fo evesible Makov chains. I. INTRODUCTION Given a stochastic pocess {X t } t=, whee each X t takes values ove the finite alphabet X of size K, the entopy ate of the pocess defined wheneve the limit exists by H := lim n n E[ı X nxn ] is a fundamental notion of uncetainty pe unit time contained within the pocess hee and elsewhee, ı Y y := log p Y y denotes the infomation in y. Estimating the entopy ate of a pocess with unknown statistics fom the obsevation of a sample path is an impotant poblem with applications in divese aeas such as data compession, bioinfomatics [] and image pocessing []. This poblem has enjoyed a ich histoy in infomation theoy with its oigin in the study of the entopy ate of the English language [3]. Any univesal data compession algoithm that achieves the entopy ate can be used as a univesal entopy ate estimato. Inspied by the analysis of the Lempel-Ziv algoithm [4], Wyne and Ziv [5] poposed an estimato based on ecuence times and poved that it conveges to H in pobability fo all stationay egodic pocesses. Analogously, the ode-α Rényi entopy ate fo 0 < α <, α of the stochastic pocess {X t } t= is defined by H α := lim n n α log E[e αı Xn X n ], All logaithms in this pape ae natual logaithms. and the min-entopy ate ode- Rényi entopy ate is defined by H := lim n n min ı x n X nxn. 3 The Rényi entopy ate plays a fundamental ole in uncetainty in seach poblems [6], biological sequence analysis [7], and data compession unde a isk-avese length citeion [8]. Univesal estimatos fo the Rényi entopy ate have been obtained fo stationay egodic pocesses unde a stong mixing condition [9]. These univesal estimatos fo the entopy ate and Rényi entopy ate convege asymptotically fo vey geneal pocesses. Howeve, so fa thee is no analysis of thei finite-sample pefomance. At the othe exteme, if the pocess is known to be i.i.d., such finite-sample bounds can be povided fo lage K fom some ecent esults: it has been shown [0] that fo any fixed additive K log K eo toleance and confidence inteval, Θ i.i.d. samples is both necessay and sufficient fo estimation of the entopy of the unknown distibution. The numbe of samples necessay and sufficient fo estimation of the ode-α Rényi entopy of an unknown distibution fom its i.i.d. samples has been studied in []. To bidge the gap between asymptotic esults fo the geneal stationay egodic pocess and finite-sample bounds fo the i.i.d. pocess, we conside the simplest and most impotant family of dependent pocesses, namely Makov chains. While plug-in estimates fo the entopy ate of Makov chains have been investigated befoe, the focus has been only on poving asymptotic convegence, e.g. [], [3]. In this pape, we study finite-sample bounds fo estimation of the entopy ate of Makov chains. We ae specifically inteested in the case whee the alphabet size of the chain is lage, as this is the case in many applications of contempoay inteest. Even if the alphabet size is small such as say, the English alphabet, a highe ode Makov souce is sometimes employed as a moe accuate model fo eal data and such a souce may be viewed as a fist ode Makov chain ove a lage alphabet, so ou esults ae elevant to such highe ode Makov souces as well. To povide /6/$ IEEE 685

2 06 IEEE Intenational Symposium on Infomation Theoy any guaantees, a bound on the alphabet size alone is not sufficient and some assumption on the mixing popeties of the chain must be made. We assume an uppe bound on the elaxation time ecipocal of the absolute spectal gap fo evesible Makov chains, and an uppe bound on the ecipocal of thei pseudo spectal gap, a quantity ecently intoduced in [4] fo geneal Makov chains i.e. without assuming evesibility. While spectal bounds ae available fo many impotant Makov chain models, in pactice, we may obseve a sample path of a chain fo which no such spectal bounds ae known. In such cases, we could esot to estimation of its spectal popeties; fo instance, [5] studies estimation of the absolute spectal gap and the minimum stationay pobability fo evesible Makov chains. Ou esults emphasize the need fo suitable mixing bounds to guaantee convegence of any estimato and also help claify the dependence on such mixing popeties fo the plug-in type estimato. Fo Rényi entopy ate estimation of Makov chains, we show that bounds on the alphabet size and mixing time cannot suffice to poduce finite-sample bounds fo any estimate of the ode-α Rényi entopy ate. By assuming in addition, a lowe bound on the minimum stationay pobability of any state of the chain, we give possibly suboptimal finite-sample bounds on the convegence of a plug-in type estimato of the odeα Rényi entopy ate fo evesible Makov chains. We also povide a fomula fo the min-entopy ate of a Makov chain and finite-sample bounds on its estimation fo evesible Makov chains. The est of the pape is oganized as follows. In Section II, we state basic Makov chain teminology. In Section III, we discuss the necessity of mixing assumptions fo poviding finite-sample bounds on entopy ate estimates. In Section IV, we show finite-sample bounds on convegence of a simple plug-in type estimato fo the entopy ate of a Makov chain. In Section V, we study finite-sample bounds fo Rényi entopy ate estimation. We conclude with a discussion and open questions in Section VI. II. MARKOV CHAIN PRELIMINARIES In this section, we set up basic teminology about Makov chains. A. Geneal Makov chains Let P be the tansition matix of a discete-time ieducible apeiodic Makov chain ove a finite alphabet X which we assume fo simplicity to be X := {,,..., K}. Let {X t } n t= be a sample path of the Makov chain, with X q fo some initial distibution q, and P[X t+ = j X t = i] = P ij, t n. 4 Let π denote its unique stationay distibution. The minimum stationay pobability is defined as π min := min i X π i > 0, 5 whee the inequality assumes that π chages all points of X. Fo such a Makov chain, the entopy ate and Rényi entopy ates defined by the limits in, always exist, do not depend on the initial distibution, and fo 0 < α <, α ae given by the explicit fomulae see [6], [7] HP = K K π i i= j= P ij log P ij, 6 H α P = α log ρ P α, 7 whee P α is the α th Hadamad powe of P, namely a matix with i, j th enty given by Pij α, and ρa is the spectal adius of a matix A. In Theoem of Section V, we povide a fomula fo the min-entopy ate H P. If the eigenvalues of the tansition matix P ae = λ, λ,..., λ K, then the absolute spectal gap of the Makov chain is defined to be γ P := max i K λ i > 0, 8 whee the inequality in 8 follows fom the egodicity of the Makov chain. The elaxation time of the Makov chain is defined to be t el := γ P. 9 If d TV P, Q = sup A P A QA denotes the total vaiation distance between distibutions P and Q, then fo 0 < ɛ <, the ɛ-mixing time of the chain is defined by t mix ɛ := min{t : d TV P t i,, π ɛ, i X }. 0 It is easy to ague that fo 0 < τ < ɛ <, see e.g. [8, Sec 4.5] logτ t mix ɛ t mix τ log ɛ t mix ɛ. We choose the standad teminology, t mix := t mix /4 fo conceteness, although the bounds we pesent ae easy to adapt to othe aguments. The elationship between the mixing time of a Makov chain and the spectal popeties of its tansition matix can be found in tems of the pseudo spectal gap intoduced in [4], which we define biefly. 686

3 06 IEEE Intenational Symposium on Infomation Theoy Fist, let P denote the tansition matix of the evese chain, namely it satisfies π i P ij = π j P j,i i, j X. 3 The chain is defined to be evesible if P = P. If the chain is evesible, then the eigenvalues of the tansition matix P ae eal. In this case, we define its spectal gap as γp := max i K λ i. 4 The pseudo spectal gap of a geneal Makov chain is then defined as γ ps P := max γp P, 5 whee we note that fo each, P P is the tansition matix of a evesible Makov chain and hence, that its spectal gap is well-defined. If we define the pseudo elaxation time of a geneal Makov chain evesible o not as t ps := γ ps P, 6 then, the pseudo elaxation time and the mixing time ae elated to each othe as [4, Pop 3.4] t ps t 4 mix t ps + log. 7 π min Futhemoe, fom the definitions 4, 5 and the fact that P has an eigenvalue with absolute value max i K λ i, we get max i K λ i γ ps P max 8 = γ P 9 = γ P γ P γ P, 0 whee 9 follows fom 8 and Benoulli s inequality a / fo 0 a,, Hence, a t el t ps. Fo a geneal Makov chain, [9, Pop.] gives an uppe bound on the mixing time in tems of the elaxation time and the alphabet size without invoking π min. Using that bound in conjunction with yields t mix 4t ps Klog t ps + + log 8 + log 4. a 3 4 a Fig.. Random walk on the weighted gaph fo a, a {, } B. Revesible Makov chains Fo evesible chains, the elaxation time and the mixing time ae intimately elated by [8, Thm..3,.4] as t el log t mix t el log π min. 3 A ecent esult fo evesible chains [9, Pop.] povides an uppe bound on the mixing time in tems of only the elaxation time and alphabet size, without invoking π min : t mix t el K + log 4 + K log 4. III. NEED FOR MIXING ASSUMPTIONS 4 Conside the evesible andom walk on the six-node weighted gaph in Fig., whee tansition pobabilities out of any state ae popotional to the weights on its outgoing edges. Suppose that the weights a, a {, } and the weight θ on the bottleneck edge is vey small. The entopy ate of the chain is H = log + i= + a ih + Ofθ, 4 + a + a +a i 5 whee h is the binay entopy function in nats, and fθ 0 as θ 0. Fo any initial distibution, the pobability that the chain neve cosses the bottleneck edge in a sample path n nθ. If nθ, with of length n is at least +θ high pobability, the sample path does not coss ove the bottleneck edge. Thus, we cannot infe both the weights a, a and it is not possible to estimate the entopy ate within a given level of accuacy with a sufficiently small pobability of eo. This example illustates the necessity of mixing assumptions. Among the most well-known of the mixing popeties of a Makov chain ae the elaxation time t el and the mixing time t mix. 3 shows that the elaxation time is always smalle than the mixing time up to a constant facto. Futhe, [9] shows that the inequality 4 is essentially shap: fo a evesible Makov chain on an alphabet of size K, t mix can be as lage as 3 687

4 06 IEEE Intenational Symposium on Infomation Theoy ΘKt el. In this pape, we assume uppe bounds on the elaxation time fo evesible chains and on the pseudo elaxation time fo geneal chains. This leads to a moe geneal setting than imposing the same uppe bounds on the mixing time. Let M ev K, T el, MK, T ps denote the set of all tansition matices of ieducible apeiodic evesible and ieducible apeiodic geneal Makov chains espectively on alphabets of size at most K and t el T el and t ps T ps espectively. Let M ev K, T el, π denote the set of all tansition matices of evesible Makov chains on alphabets of size at most K, t el T el, and minimum stationay pobability π min π. IV. ENTROPY RATE ESTIMATION In this section, we obtain a finite-sample bound on the pefomance of a plug-in-type estimato fo the entopy ate of a Makov chain. If {X t } n t= is a sample path of a Makov chain ove alphabet X = {,,..., K}, then we can define a plug-in estimato fo its entopy ate. Fo i, j X, N ij := { t n : X t, X t+ = i, j}, 6 N i := { t n : X t = i}. 7 K N i K Ĥ plug in = N ij log N i. 8 n N i N ij i= j= A simple vaiant of this estimato is used to obtain Theoem whose poof is placed in Appendix A of the full pape [0]. Theoem. Let {X t } n t= be a sample path of a Makov chain with any tansition matix P M ev K, T el initiated at any distibution. Fo any 0 < ɛ <, thee exists an estimate Ĥn such that with pobability at least ɛ, we have Ĥn HP CK T el n ɛ + C KT el log n n ɛ, 9 whee n = max{n C KT el log ɛ, 0} fo some absolute constants C, C > 0. If P MK, T ps instead, fo any 0 < ɛ <, thee exists an estimate Ĥ n such that with pobability at least ɛ, Ĥn HP C3K Tps C3KTpslog Tpslog n ɛ + n n ɛ, 30 whee n = max{n C 4 KT ps log T ps log ɛ, 0} fo some absolute constants C 3, C 4 > 0. Remak. In paticula, if P M ev K, T el, and if the chain is not too slow mixing, i.e. T el << e K /K, then fo any fixed desied accuacy and specified uppe bound on the eo pobability, n = OK T el length sample path is sufficient fo estimation of the entopy ate. Remak. On the ight hand sides of 9 and 30, the fist tem deives fom bounds on the bias of the estimato and the second fom those on the vaiance. Remak 3. One of the impotant featues of Thm. is that its bounds do not depend on the minimum stationay pobability π min. In contast, we shall see in Section V that such dependence on π min is unavoidable fo Rényi entopy ate estimation. V. RÉNYI ENTROPY RATE ESTIMATION We stat by poviding a simple fomula fo the minentopy ate ode- Rényi entopy ate of a Makov chain. The poof of Theoem is placed in Appendix B of the full pape [0]. Given a state space of any Makov chain with tansition matix P, a loop is a sequence of distinct states of the chain i, i,..., i l with l such that P is,i s+ > 0 fo s =,,..., l whee i l+ i. If P i,i > 0, then i is a loop. The set of all loops of length l is denoted by C l P. Theoem. Let P be the tansition matix of an ieducible apeiodic Makov chain on a finite alphabet X. The min-entopy ate of the Makov chain is given by H P = min l K min l l ı X X i s+ i s, 3 s= whee the inne minimum is taken ove all loops i,..., i l C l P, and ı X X j i := log P ij. Fo evesible Makov chains, 3 simplifies to [ H P = min ıx X i,j X j i + ı X X i j ]. 3 In paallel with Section III, it can be shown that in addition to a bound on the alphabet size, a bound on the elaxation time is necessay in ode to povide guaantees fo the estimation of Rényi entopy ates. Howeve, we show that uppe bounds on the alphabet size and elaxation time alone do not suffice to povide finite-sample bounds on the accuacy with which the ode-α Rényi entopy ate may be estimated fo any α, even fo stationay evesible chains. The poof is placed in Appendix C of the full pape [0]. Theoem 3. Fix any α 0,, ]. Thee does not exist any estimate Ĥn α based on a sample path {X t } n t= of length n of a stationay evesible Makov chain such that fo any tansition matix P M ev K, T el, we have P[ Ĥn α H α P δ] ɛ, fo sufficiently small constants ɛ, δ, if the length n of the sample path is only allowed to depend on K, T el, ɛ, δ

5 06 IEEE Intenational Symposium on Infomation Theoy Howeve as the next esult shows, additional knowledge of a lowe bound on the stationay pobability π min opens the possibility of such bounds. The poof of Theoem 4 is in Appendix D of the full pape [0]. Theoem 4. Fix any α 0,, ]. Let {X t } n t= be a sample path of a evesible Makov chain with tansition matix P M ev K, T el, π initiated at any distibution. If α 0,,, then thee exists an estimate Ĥn α such that fo any 0 < ɛ <, with pobability at least ɛ, Ĥn α H α P C α K α Tel log K ɛ log n π ɛ π n + T el log T el n α, 33 whee C α > 0 is an absolute constant, a b = max{a, b}, a b = min{a, b}. If α =, and 0 < ɛ <, then thee exists an estimate of the min-entopy ate Ĥn such that Ĥn H P Tel log C K K ɛ log n π ɛ π n + T el log T el n, 34 fo some absolute constant C > 0, with pobability at least ɛ. VI. DISCUSSION The estimato studied in Section IV is of the plugin type. The analysis of this estimato in the poof of Thm. shows a lage bias. Effots to educe this bias should geneally impove pefomance []. A few open questions ae as follows. Chaacteizing up to constant factos the minimax isk fo estimating the entopy ate fo a family of Makov chains such as all evesible Makov chains with a bound on thei alphabet size and elaxation time. Ou bounds fo entopy ate estimation of non-evesible chains involve bounds on the pseudo elaxation time. We do not know if such bounds could be obtained using bounds on the elaxation time instead. 3 Shape bounds fo Rényi entopy ate estimation could be obtained via a multiplicative eo analysis, athe than the additive eo analysis we have pefomed in this pape capitalizing on [5]. Fo nonevesible Makov chains, an eigenvalue petubation analysis could be caied out fo the estimation of the spectal adius of the Hadamad powe of the tansition matix. ACKNOWLEDGEMENTS We would like to thank Ramon van Handel, Ayeh Kontoovich, and Daniel Paulin fo helpful discussions. SK would like to acknowledge suppot fom the Cente fo Science of Infomation CSoI, an NSF Science and Technology Cente, unde Gant CCF SV would like to acknowledge suppot fom MURI Gant 67470CSMUR. REFERENCES [] J. Lanctot, M. Li, and E.-H. Yang, Estimating DNA sequence entopy, in Poc. Symp. Discete Algoithms, San Fancisco, CA, 000. [] A. O. Heo, B. Ma, O. Michel, and J. Goman, Applications of entopic spanning gaphs, IEEE Signal Pocessing Mag., vol. 9, pp , Septembe 00. [3] C. E. Shannon, Pediction and Entopy of pinted English, Bell Syst. Tech. J., pp , 95. [4] J. Ziv and A. Lempel, A univesal algoithm fo sequential data compession, IEEE Tans. Infom. Theoy, vol. 3, pp , May 977. [5] A. D. Wyne and J. Ziv, Some asymptotic popeties of the entopy of a stationay egodic data souce with applications to data compession, IEEE Tans. Infom. Theoy, vol. 35, pp , 989. [6] L. Ponzato, H. P. Wynn, and A. A. Zhigljavsky, Using Rényi entopies to measue uncetainty in seach poblems, Lectues in Applied Mathematics, vol. 33, pp , 997. [7] S. Vinga, Infomation theoy applications fo biological sequence analysis, Biefings in bioinfomatics, vol. 5, no. 3, pp , 03. [8] L. L. Campbell, A coding theoem and Rényi s entopy, Infomation and Contol, vol. 8, pp , 965. [9] W. Szpankowski, A genealized suffix tee and its unexpected asymptotic behavios, SIAM Jounal on Computing, vol., no. 6, pp , 993. [0] G. Valiant and P. Valiant, Estimating the unseen: an n/ logn- sample estimato fo entopy and suppot size, shown optimal via new CLTs, in Poc. of the 43d annual ACM symposium on Theoy of Computing, 0. [] J. Achaya, A. Olitsky, H. Tyagi, and A. T. Suesh, The complexity of estimating Rényi entopy, in Poc. of the ACM- SIAM Symposium on Discete Algoithms, 05. [] H. Cai, S. R. Kulkani, and S. Vedú, Univesal Entopy Estimation Via Block Soting, IEEE Tans. Infom. Theoy, vol. 50, no. 7, pp , July 004. [3] G. Ciupeca and V. Giadin, On the estimation of the entopy ate of finite Makov chains, in Poc. of the Intenational Symposium on Applied Stochastic Models and Data Analysis, 005. [4] D. Paulin, Concentation inequalities fo Makov chains by Maton couplings and spectal methods, axiv:.05v4 [math.pr], Jan. 05. [5] D. J. Hsu, A. Kontoovich, and C. Szepesvái, Mixing time estimation in evesible Makov chains fom a single sample path, in Advances in Neual Infomation Pocessing Systems, 05, pp [6] C. E. Shannon, A mathematical theoy of communication, Bell Syst. Tech. J., vol. 7, pp , , Jul.-Oct [7] Z. Rached, F. Alajaji, and L.L. Campbell, Rényi s entopy ate fo Discete Makov souces, in Poc. of CISS, Baltimoe, MD, Mach 999. [8] D. A. Levin, Y. Pees, and E. L. Wilme, Makov chains and mixing times, Ameican Mathematical Society, Povidence, RI, 009. [9] D. Jeison, Geneal mixing time bounds fo finite Makov chains via the absolute spectal gap, axiv:30.80v [math.pr], Oct. 03. [0] S. Kamath and S. Vedú, Estimation of entopy ate and Rényi entopy ate fo Makov chains, ~vedu/epints/isit_entopy_ate_makov_chains.pdf, 06, [Online]. [] L. Paninski, Estimation of entopy and mutual infomation, Neual computation, vol. 5, no. 6, pp. 9 53,

Temporal-Difference Learning

Temporal-Difference Learning .997 Decision-Making in Lage-Scale Systems Mach 17 MIT, Sping 004 Handout #17 Lectue Note 13 1 Tempoal-Diffeence Leaning We now conside the poblem of computing an appopiate paamete, so that, given an appoximation