arxiv: v4 [cs.it] 17 Oct 2015
|
|
- Bertha Golden
- 5 years ago
- Views:
Transcription
1 Upper Bounds on the Relative Entropy and Rényi Divergence as a Function of Total Variation Distance for Finite Alphabets Igal Sason Department of Electrical Engineering Technion Israel Institute of Technology Haifa 3, Israel sason@ee.technion.ac.il Sergio Verdú Department of Electrical Engineering Princeton University Princeton, New Jersey 8544, USA verdu@princeton.edu arxiv:53.347v4 [cs.it] 7 Oct 5 Abstract A new upper bound on the relative entropy is derived as a function of the total variation distance for probability measures defined on a common finite alphabet. The bound improves a previously reported bound by Csiszár and Talata. It is further extended to an upper bound on the Rényi divergence of an arbitrary non-negative order including as a function of the total variation distance. Keywords: Pinsker s inequality, relative entropy, relative information, Rényi divergence, total variation distance.. INTRODUCTION Consider two probability distributions P and Q defined on a common measurable space A, F. The Csiszár- Kemperman-Kullback-Pinsker inequality a.k.a. Pinsker s inequality states that where [ DP Q = E P log dp ] dq P Q log e DP Q = A d log d dq designates the relative entropy a.k.a. the Kullback-Leibler divergence from P to Q, and P Q = sup P F QF 3 F F is the total variation distance between P and Q. A reverse Pinsker inequality providing an upper bound on the relative entropy in terms of the total variation distance does not exist in general since we can find distributions that are arbitrarily close in total variation but with arbitrarily high relative entropy. Nevertheless, it is possible to introduce constraints under which such reverse Pinsker inequalities can be obtained. In the case where the probability measures P and Q are defined on a common discrete i.e., finite or countable set A, DP Q = log, 4 P Q =. 5 One of the implications of is that convergence in relative entropy implies convergence in total variation distance. The total variation distance is bounded P Q, whereas the relative entropy is an unbounded information measure. Improved versions of Pinsker s inequality were studied, e.g., in [9], [], [4], [7], []. A reverse Pinsker inequality providing an upper bound on the relative entropy in terms of the total variation distance does not exist in general since we can find distributions that are arbitrarily close in total variation but with arbitrarily high relative entropy. Nevertheless, it is possible to introduce constraints under which such reverse Pinsker inequalities can be obtained. In the case of a finite alphabet A, Csiszár and Talata [6, p. ] show that log e DP Q P Q, 6 where. 7 Recent applications of 6 can be found in [, Appendix D] and [, Lemma 7] for the analysis of the thirdorder asymptotics of the discrete memoryless channel with or without cost constraints. In addition to in 7, the bounds in this paper involve β =, 8 β = 9 so, β, β [, ]. In this paper, Section derives a reverse Pinsker inequality for probability measures defined on a common finite set, improving the bound in 6. The utility of this inequality is studied in Section 3, and it is extended in Section 4 to Rényi divergences of an arbitrary non-negative order.. A NEW REVERSE PINSKER INEQUALITY FOR DISTRIBUTIONS ON A FINITE SET The present section introduces a strengthened version of 6, followed by some remarks and an example.
2 A. Main Result and Proof Theorem. Let P and Q be probability measures defined on a common finite set A, and assume that Q is strictly positive on A. Then, the following inequality holds: DP Q log + log + P Q P Q β log e P Q where and β are given in 7 and 9, respectively. Proof: Theorem is proved by obtaining upper and lower bounds on the χ -divergence from P to Q χ P Q. A lower bound follows by invoking Jensen s inequality χ P Q = 3 = exp log 4 exp log 5 = exp DP Q. 6 Alternatively, 6 can be obtained by combining the equality χ P Q = exp D P Q 7 with the monotonicity of the Rényi divergence D α P Q in α, which implies that D P Q DP Q. A refined version of 6 is derived in the following. The starting point is a refined version of Jensen s inequality in [, Lemma ], generalizing a result from [7, Theorem ], which leads to see [, Theorem 7] DQ P log + χ P Q DP Q 8 max DQ P. 9 From 9 and the definition of β in 9, we have χ P Q exp DP Q + β DQ P exp DP Q + β log e P Q where follows from 8 and the definition of β in 9, and follows from Pinsker s inequality. Note that the lower bound in refines the lower bound in 6 since β [, ]. An upper bound on χ P Q is derived as follows: and, from 3, χ P Q = = P Q max 3 P Q max. 4 Combining 3 and 4 yields χ P Q P Q. 5 Finally, follows by combining the upper and lower bounds on the χ -divergence in and 5. Remark. It is easy to check that Theorem strengthens the bound by Csiszár and Talata in 6 by at least a factor of since upper bounding the logarithm in gives DP Q β log e P Q. 6 In the finite-alphabet case, we can obtain another upper bound on DP Q as a function of the l norm P Q : DP Q log + P Q β log e P Q 7 which follows by combining,, and P Q P Q. Using the inequality log + x x log e for x in the right side of 7, and also loosening this bound by β log e ignoring the term P Q, we recover the bound DP Q P Q log e 8 which appears in the proof of Property 4 of [, Lemma 7], and also used in [, 74]. Remark. The lower bounds on the χ -divergence in 6 and improve the one in [6, Lemma 6.3] which states that DP Q χ P Q log e. Remark 3. Reverse Pinsker inequalities have been also derived in quantum information theory [], [], providing upper bounds on the relative entropy of two quantum states as a function of the trace norm distance when the imal eigenvalues of the states are positive c.f. [, Theorem 6] and [, Theorem ]. These type of bounds are akin to the weakend form in. When the variational distance is much smaller than the imal eigenvalue see [, Eq. 57], the latter bounds have a quadratic scaling in this distance, similarly to ; they are also inversely proportional to the imal eigenvalue, similarly to the dependence of in.
3 3. APPLICATIONS OF THEOREM A. The Exponential Decay of the Probability for a Non- Typical Sequence To exemplify the utility of Theorem, we bound the function L δ Q = DP Q 9 P T δ Q where we have denoted the subset of probability measures on A, F which are δ-close to Q as } T δ Q = P : a A, δ 3 Note that a,..., a n is strongly δ-typical according to Q if its empirical distribution belongs to T δ Q. According to Sanov s theorem e.g. [5, Theorem.4.], if the random variables are independent distributed according to Q, then the probability that Y,..., Y n, is not δ-typical vanishes exponentially with exponent L δ Q. To state the next result, we invoke the following notions from [4]. Given a probability measure Q, its balance coefficient is given by β Q = inf QA. 3 A F : QA The function φ:, ] [ log e, is given by 4 p φp = log p p, p,, 3 log e, p =. Theorem. If >, then φ β Q Q δ L δ Q 33 where 34 holds if δ Q. log + δ 34 Proof: Ordentlich and Weinberger [4, Section 4] show the refinement of Pinsker s inequality: φ β Q P Q DP Q. 35 Note that if > then β Q <, and therefore φ β Q is well defined and finite. If P T δ Q the simple bound P Q > δ 36 together with 35 yields 33. The upper bound 34 follows from and the fact that if δ Q, then P Q = δ. 37 P T δ Q If δ Q, the ratio between the upper and lower bounds in 34, satisfies log e φ β Q log + Q δ log e Q 4 38 δ where 38 follows from the fact that its second and third factors are less than or equal to and 4, respectively. Note that the bounds in 33 and 34 scale like δ for δ. B. Distance from Equiprobable If P is a distribution on a finite set A, HP gauges the distance from U, the equiprobable distribution, since HP = log A DP U. 39 Thus, it is of interest to explore the relationship between HP and P U. Particularizing, [4,.] see also [4, pp. 3 3], and we obtain P U log e log A HP, 4 P U A exp HP, 4 respectively P U b c exp HP, 4 A A =4 a b c HP bits A = 6 a HP bits Fig.. Bounds on P U as a function of HP for A = 4, and A = 6. The point HP, P U =, A is depicted on the y-axis. In the curves of the two plots, the bounds a, b and c refer, respectively, to 4, 4 and 4. The bounds in 4 4 are illustrated for A = 4, 6 in Figure. For HP =, P U = A is shown for reference in Figure ; as the cardinality of the alphabet
4 increases, the gap between P U and its upper bound is reduced and this gap decays asymptotically to zero. Results on the more general problem of finding bounds on HP HQ based on P Q can be found in [5, Theorem 7.3.3], [], [6], [8], [6, Section.7] and [7]. 4. EXTENSION OF THEOREM TO RÉNYI DIVERGENCES Definition. The Rényi divergence of order α [, ] from P to Q is defined for α,, as D α P Q α log P α a Q α a. 43 Recall that D P Q DP Q is defined to be the analytic extension of D α P Q at α = if DP Q <, L Hôpital s rule gives that DP Q = lim α D α P Q. The extreme cases of α =, are defined as follows: If α = then D P Q = log QSupportP, If α = + then D P Q = log sup. Pinsker s inequality was extended by Gilardoni [] for a Rényi divergence of order α, ] see also [8, Theorem 3], and it gets the form α P Q log e D α P Q. A tight lower bound on the Rényi divergence of order α > as a function of the total variation distance is given in [9], which is consistent with Vajda s tight lower bound for f- divergences in [3, Theorem 3]. Motivated by these findings, we extend the upper bound on the relative entropy in Theorem to Rényi divergences of an arbitrary order. Theorem 3. Assume that P, Q are strictly positive with imum masses denoted by P and, respectively. Let β and β be given in 8 and 9, respectively, and abbreviate δ P Q [, ]. Then, the Rényi divergence of order α [, ] satisfies D α P Q f, α, ] f, α [, ] f, f 3, f 4 }, α, } log δ, f, f 3, f 4, α [ ], 44 where, for α [, ], f α, β, δ α log + δβ α β α [,, δ β log β, α =, log β, for α [, ] α = f α, β,, δ } f α, β, δ, log + δ and, for α [,, f 3 and f 4 are given by f 3 α, P, β, δ [ α log + δ α f 4 β,, δ log + δ P ] β δ log e, 47 β δ log e, log + δ, } δ }. 48 Proof: See [, Section 7.C]. Remark 4. A simple bound, albeit looser than the one in Theorem 3 is P Q D α P Q log + 49 which is asymptotically tight as α in the case of a binary alphabet with equiprobable Q. Example. Figure illustrates the bound in 45, which is valid for all α [, ] see [, Theorem 3], and the upper bounds of Theorem 3 in the case of binary alphabets. 5. SUMMARY We derive in this paper some reverse Pinsker inequalities for probability measures P Q defined on a common finite set, which provide lower bounds on the total variation distance P Q as a function of the relative entropy DP Q under the assumption of a bounded relative information or >. More general results for an arbitrary alphabet are available in [, Section 5]. In [], we study bounds among various f-divergences, dealing with arbitrary alphabets and deriving bounds on the ratios of various distance measures. New expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to upper and lower bounds on the Rényi divergence in terms of the variational distance.
5 .7 nats b a D P kq Fig.. The Rényi divergence D αp Q for P and Q which are defined on a binary alphabet with P = Q = 5, compared to a its upper bound in 44, and b its upper bound in 45 see [, Theorem 3]. The two bounds coincide here when α,.9,. ACKNOWLEDGMENT The work of I. Sason has been supported by the Israeli Science Foundation ISF under Grant /, and the work of S. Verdú has been supported by the US National Science Foundation under Grant CCF-665, and in part by the Center for Science of Information, an NSF Science and Technology Center under Grant CCF REFERENCES [] K. M. R. Audenaert and J. Eisert, Continuity bounds on the quantum relative entropy, Journal of Mathematical Physics, vol. 46, paper 4, October 5. [] K. M. R. Audenaert and J. Eisert, Continuity bounds on the quantum relative entropy - II, Journal of Mathematical Physics, vol. 5, paper, November. [3] G. Böcherer and B. C. Geiger, Optimal quantization for distribution synthesis, March 5. Available at [4] J. Bretagnolle and C. Huber, Estimation des densités: risque imax, Probability Theory and Related Fields, vol. 47, no., pp. 9 37, 979. [5] T. M. Cover and J. A. Thomas, Elements of Information Theory, second edition, John Wiley & Sons, 6. [6] I. Csiszár and Z. Talata, Context tree estimation for not necessarily finite memory processes, via BIC and MDL, IEEE Trans. on Information Theory, vol. 5, no. 3, pp. 7 6, March 6. [7] S. S. Dragomir, Bounds for the normalized Jensen functional, Bulletin of the Australian Mathematical Society, vol. 74, no. 3, pp , 6. [8] T. van Erven and P. Harremoës, Rényi divergence and Kullback- Leibler divergence, IEEE Trans. on Information Theory, vol. 6, no. 7, pp , July 4. [9] A. A. Fedotov, P. Harremoës and F. Topsøe, Refinements of Pinsker s inequality, IEEE Trans. on Information Theory, vol. 49, no. 6, pp , June 3. [] G. L. Gilardoni, On Pinsker s and Vajda s type inequalities for Csiszár s f-divergences, IEEE Trans. on Information Theory, vol. 56, no., pp , November. [] S. W. Ho and R. W. Yeung, The interplay between entropy and variational distance, IEEE Trans. on Information Theory, vol. 56, no., pp , December. [] V. Kostina and S. Verdú, Channels with cost constraints: strong converse and dispersion, to appear in the IEEE Trans. on Information Theory, vol. 6, no. 5, May 5. [3] M. Kraj ci, C. F. Liu, L. Mike s and S. M. Moser, Performance analysis of Fano coding, Proceedings of the IEEE 5 International Symposium on Information Theory, Hong Kong, June 4 9, 5. [4] E. Ordentlich and M. J. Weinberger, A distribution dependent refinement of Pinsker s inequality, IEEE Trans. on Information Theory, vol. 5, no. 5, pp , May 5. [5] M. S. Pinsker, Information and Information Stability of Random Variables and Random Processes, San-Fransisco: Holden-Day, 964, originally published in Russian in 96. [6] V. V. Prelov and E. C. van der Meulen, Mutual information, variation, and Fano s inequality, Problems of Information Transmission, vol. 44, no. 3, pp , September 8. [7] M. D. Reid and R. C. Williamson, Information, divergence and risk for binary experiments, Journal of Machine Learning Research, vol., no. 3, pp , March. [8] I. Sason, Entropy bounds for discrete random variables via maximal coupling, IEEE Trans. on Information Theory, vol. 59, no., pp , November 3. [9] I. Sason, On the Rényi divergence and the joint range of relative entropies, Proceedings of the 5 IEEE International Symposium on Information Theory, pp. 6 64, Hong Kong, June 4 9, 5. [] I. Sason and S. Verdú, Bounds among f-divergences, submitted to the IEEE Trans. on Information Theory, July 5. [Online]. Available at [] M. Tomamichel and V. Y. F. Tan, A tight upper bound for the third-order asymptotics for most discrete memoryless channels, IEEE Trans. on Information Theory, vol. 59, no., pp , November 3. [] I. Vajda, Note on discriation information and variation, IEEE Trans. on Information Theory, vol. 6, no. 6, pp , November 97. [3] I. Vajda, On f-divergence and singularity of probability measures, Periodica Mathematica Hungarica, vol., no. 4, pp. 3 34, 97. [4] V. N. Vapnik, Statistical Learning Theory, John Wiley & Sons, 998. [5] S. Verdú, Total variation distance and the distribution of the relative information, Proceedings of the Information Theory and Applications Workshop, pp , San-Diego, California, USA, February 4. [6] S. Verdú, Information Theory, in preparation. [7] Z. Zhang, Estimating mutual information via Kolmogorov distance, IEEE Trans. on Information Theory, vol. 53, no. 9, pp , September 7.
Tight Bounds for Symmetric Divergence Measures and a New Inequality Relating f-divergences
Tight Bounds for Symmetric Divergence Measures and a New Inequality Relating f-divergences Igal Sason Department of Electrical Engineering Technion, Haifa 3000, Israel E-mail: sason@ee.technion.ac.il Abstract
More informationOn Improved Bounds for Probability Metrics and f- Divergences
IRWIN AND JOAN JACOBS CENTER FOR COMMUNICATION AND INFORMATION TECHNOLOGIES On Improved Bounds for Probability Metrics and f- Divergences Igal Sason CCIT Report #855 March 014 Electronics Computers Communications
More informationarxiv: v4 [cs.it] 8 Apr 2014
1 On Improved Bounds for Probability Metrics and f-divergences Igal Sason Abstract Derivation of tight bounds for probability metrics and f-divergences is of interest in information theory and statistics.
More informationTight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding
APPEARS IN THE IEEE TRANSACTIONS ON INFORMATION THEORY, FEBRUARY 015 1 Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding Igal Sason Abstract Tight bounds for
More informationf-divergence Inequalities
SASON AND VERDÚ: f-divergence INEQUALITIES f-divergence Inequalities Igal Sason, and Sergio Verdú Abstract arxiv:508.00335v7 [cs.it] 4 Dec 06 This paper develops systematic approaches to obtain f-divergence
More informationf-divergence Inequalities
f-divergence Inequalities Igal Sason Sergio Verdú Abstract This paper develops systematic approaches to obtain f-divergence inequalities, dealing with pairs of probability measures defined on arbitrary
More informationConvexity/Concavity of Renyi Entropy and α-mutual Information
Convexity/Concavity of Renyi Entropy and -Mutual Information Siu-Wai Ho Institute for Telecommunications Research University of South Australia Adelaide, SA 5095, Australia Email: siuwai.ho@unisa.edu.au
More informationArimoto-Rényi Conditional Entropy. and Bayesian M-ary Hypothesis Testing. Abstract
Arimoto-Rényi Conditional Entropy and Bayesian M-ary Hypothesis Testing Igal Sason Sergio Verdú Abstract This paper gives upper and lower bounds on the minimum error probability of Bayesian M-ary hypothesis
More informationOn the Entropy of Sums of Bernoulli Random Variables via the Chen-Stein Method
On the Entropy of Sums of Bernoulli Random Variables via the Chen-Stein Method Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, Israel ETH, Zurich,
More informationMMSE Dimension. snr. 1 We use the following asymptotic notation: f(x) = O (g(x)) if and only
MMSE Dimension Yihong Wu Department of Electrical Engineering Princeton University Princeton, NJ 08544, USA Email: yihongwu@princeton.edu Sergio Verdú Department of Electrical Engineering Princeton University
More informationLower Bounds on the Graphical Complexity of Finite-Length LDPC Codes
Lower Bounds on the Graphical Complexity of Finite-Length LDPC Codes Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, Israel 2009 IEEE International
More informationArimoto Channel Coding Converse and Rényi Divergence
Arimoto Channel Coding Converse and Rényi Divergence Yury Polyanskiy and Sergio Verdú Abstract Arimoto proved a non-asymptotic upper bound on the probability of successful decoding achievable by any code
More informationLiterature on Bregman divergences
Literature on Bregman divergences Lecture series at Univ. Hawai i at Mānoa Peter Harremoës February 26, 2016 Information divergence was introduced by Kullback and Leibler [25] and later Kullback started
More informationChannels with cost constraints: strong converse and dispersion
Channels with cost constraints: strong converse and dispersion Victoria Kostina, Sergio Verdú Dept. of Electrical Engineering, Princeton University, NJ 08544, USA Abstract This paper shows the strong converse
More informationThe Information Bottleneck Revisited or How to Choose a Good Distortion Measure
The Information Bottleneck Revisited or How to Choose a Good Distortion Measure Peter Harremoës Centrum voor Wiskunde en Informatica PO 94079, 1090 GB Amsterdam The Nederlands PHarremoes@cwinl Naftali
More informationA GENERAL CLASS OF LOWER BOUNDS ON THE PROBABILITY OF ERROR IN MULTIPLE HYPOTHESIS TESTING. Tirza Routtenberg and Joseph Tabrikian
A GENERAL CLASS OF LOWER BOUNDS ON THE PROBABILITY OF ERROR IN MULTIPLE HYPOTHESIS TESTING Tirza Routtenberg and Joseph Tabrikian Department of Electrical and Computer Engineering Ben-Gurion University
More informationOn the Concentration of the Crest Factor for OFDM Signals
On the Concentration of the Crest Factor for OFDM Signals Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology, Haifa 3, Israel E-mail: sason@eetechnionacil Abstract
More informationA new converse in rate-distortion theory
A new converse in rate-distortion theory Victoria Kostina, Sergio Verdú Dept. of Electrical Engineering, Princeton University, NJ, 08544, USA Abstract This paper shows new finite-blocklength converse bounds
More informationInequalities for the L 1 Deviation of the Empirical Distribution
Inequalities for the L 1 Deviation of the Emirical Distribution Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, Marcelo J. Weinberger June 13, 2003 Abstract We derive bounds on the robability
More informationUniversal Estimation of Divergence for Continuous Distributions via Data-Dependent Partitions
Universal Estimation of for Continuous Distributions via Data-Dependent Partitions Qing Wang, Sanjeev R. Kulkarni, Sergio Verdú Department of Electrical Engineering Princeton University Princeton, NJ 8544
More informationSoft Covering with High Probability
Soft Covering with High Probability Paul Cuff Princeton University arxiv:605.06396v [cs.it] 20 May 206 Abstract Wyner s soft-covering lemma is the central analysis step for achievability proofs of information
More informationRefined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities
Refined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities Maxim Raginsky and Igal Sason ISIT 2013, Istanbul, Turkey Capacity-Achieving Channel Codes The set-up DMC
More informationInformation Theory and Hypothesis Testing
Summer School on Game Theory and Telecommunications Campione, 7-12 September, 2014 Information Theory and Hypothesis Testing Mauro Barni University of Siena September 8 Review of some basic results linking
More informationCorrelation Detection and an Operational Interpretation of the Rényi Mutual Information
Correlation Detection and an Operational Interpretation of the Rényi Mutual Information Masahito Hayashi 1, Marco Tomamichel 2 1 Graduate School of Mathematics, Nagoya University, and Centre for Quantum
More informationDispersion of the Gilbert-Elliott Channel
Dispersion of the Gilbert-Elliott Channel Yury Polyanskiy Email: ypolyans@princeton.edu H. Vincent Poor Email: poor@princeton.edu Sergio Verdú Email: verdu@princeton.edu Abstract Channel dispersion plays
More informationJensen-Shannon Divergence and Hilbert space embedding
Jensen-Shannon Divergence and Hilbert space embedding Bent Fuglede and Flemming Topsøe University of Copenhagen, Department of Mathematics Consider the set M+ 1 (A) of probability distributions where A
More informationEntropy measures of physics via complexity
Entropy measures of physics via complexity Giorgio Kaniadakis and Flemming Topsøe Politecnico of Torino, Department of Physics and University of Copenhagen, Department of Mathematics 1 Introduction, Background
More informationAn Extended Fano s Inequality for the Finite Blocklength Coding
An Extended Fano s Inequality for the Finite Bloclength Coding Yunquan Dong, Pingyi Fan {dongyq8@mails,fpy@mail}.tsinghua.edu.cn Department of Electronic Engineering, Tsinghua University, Beijing, P.R.
More informationA Single-letter Upper Bound for the Sum Rate of Multiple Access Channels with Correlated Sources
A Single-letter Upper Bound for the Sum Rate of Multiple Access Channels with Correlated Sources Wei Kang Sennur Ulukus Department of Electrical and Computer Engineering University of Maryland, College
More informationJournal of Inequalities in Pure and Applied Mathematics
Journal of Inequalities in Pure and Applied Mathematics http://jipam.vu.edu.au/ Volume, Issue, Article 5, 00 BOUNDS FOR ENTROPY AND DIVERGENCE FOR DISTRIBUTIONS OVER A TWO-ELEMENT SET FLEMMING TOPSØE DEPARTMENT
More informationBounds for entropy and divergence for distributions over a two-element set
Bounds for entropy and divergence for distributions over a two-element set Flemming Topsøe Department of Mathematics University of Copenhagen, Denmark 18.1.00 Abstract Three results dealing with probability
More informationOn the Capacity of Free-Space Optical Intensity Channels
On the Capacity of Free-Space Optical Intensity Channels Amos Lapidoth TH Zurich Zurich, Switzerl mail: lapidoth@isi.ee.ethz.ch Stefan M. Moser National Chiao Tung University NCTU Hsinchu, Taiwan mail:
More informationStrong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach Silas Fong (Joint work with Vincent Tan) Department of Electrical & Computer Engineering National University
More informationSequential prediction with coded side information under logarithmic loss
under logarithmic loss Yanina Shkel Department of Electrical Engineering Princeton University Princeton, NJ 08544, USA Maxim Raginsky Department of Electrical and Computer Engineering Coordinated Science
More informationAnalytical Bounds on Maximum-Likelihood Decoded Linear Codes: An Overview
Analytical Bounds on Maximum-Likelihood Decoded Linear Codes: An Overview Igal Sason Department of Electrical Engineering, Technion Haifa 32000, Israel Sason@ee.technion.ac.il December 21, 2004 Background
More informationAmobile satellite communication system, like Motorola s
I TRANSACTIONS ON INFORMATION THORY, VOL. 45, NO. 4, MAY 1999 1111 Distributed Source Coding for Satellite Communications Raymond W. Yeung, Senior Member, I, Zhen Zhang, Senior Member, I Abstract Inspired
More informationThe Method of Types and Its Application to Information Hiding
The Method of Types and Its Application to Information Hiding Pierre Moulin University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ moulin/talks/eusipco05-slides.pdf EUSIPCO Antalya, September 7,
More informationUpper Bounds on the Capacity of Binary Intermittent Communication
Upper Bounds on the Capacity of Binary Intermittent Communication Mostafa Khoshnevisan and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame, Indiana 46556 Email:{mhoshne,
More informationLarge Deviations Performance of Knuth-Yao algorithm for Random Number Generation
Large Deviations Performance of Knuth-Yao algorithm for Random Number Generation Akisato KIMURA akisato@ss.titech.ac.jp Tomohiko UYEMATSU uematsu@ss.titech.ac.jp April 2, 999 No. AK-TR-999-02 Abstract
More informationAn Improved Sphere-Packing Bound for Finite-Length Codes over Symmetric Memoryless Channels
An Improved Sphere-Packing Bound for Finite-Length Codes over Symmetric Memoryless Channels Gil Wiechman Igal Sason Department of Electrical Engineering Technion, Haifa 3000, Israel {igillw@tx,sason@ee}.technion.ac.il
More informationThe Poisson Channel with Side Information
The Poisson Channel with Side Information Shraga Bross School of Enginerring Bar-Ilan University, Israel brosss@macs.biu.ac.il Amos Lapidoth Ligong Wang Signal and Information Processing Laboratory ETH
More informationFunctional Properties of MMSE
Functional Properties of MMSE Yihong Wu epartment of Electrical Engineering Princeton University Princeton, NJ 08544, USA Email: yihongwu@princeton.edu Sergio Verdú epartment of Electrical Engineering
More informationSome Expectations of a Non-Central Chi-Square Distribution With an Even Number of Degrees of Freedom
Some Expectations of a Non-Central Chi-Square Distribution With an Even Number of Degrees of Freedom Stefan M. Moser April 7, 007 Abstract The non-central chi-square distribution plays an important role
More informationMUTUAL INFORMATION (MI) specifies the level of
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 58, NO. 7, JULY 2010 3497 Nonproduct Data-Dependent Partitions for Mutual Information Estimation: Strong Consistency and Applications Jorge Silva, Member, IEEE,
More informationAn Achievable Error Exponent for the Mismatched Multiple-Access Channel
An Achievable Error Exponent for the Mismatched Multiple-Access Channel Jonathan Scarlett University of Cambridge jms265@camacuk Albert Guillén i Fàbregas ICREA & Universitat Pompeu Fabra University of
More information(each row defines a probability distribution). Given n-strings x X n, y Y n we can use the absence of memory in the channel to compute
ENEE 739C: Advanced Topics in Signal Processing: Coding Theory Instructor: Alexander Barg Lecture 6 (draft; 9/6/03. Error exponents for Discrete Memoryless Channels http://www.enee.umd.edu/ abarg/enee739c/course.html
More informationA Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback
A Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback Vincent Y. F. Tan (NUS) Joint work with Silas L. Fong (Toronto) 2017 Information Theory Workshop, Kaohsiung,
More informationInformation Theory in Intelligent Decision Making
Information Theory in Intelligent Decision Making Adaptive Systems and Algorithms Research Groups School of Computer Science University of Hertfordshire, United Kingdom June 7, 2015 Information Theory
More informationIN THIS PAPER, we consider a class of continuous-time recurrent
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 51, NO. 4, APRIL 2004 161 Global Output Convergence of a Class of Continuous-Time Recurrent Neural Networks With Time-Varying Thresholds
More informationarxiv: v8 [cs.it] 20 Feb 2014
arxiv:1206.6544v8 [cs.it] 20 Feb 2014 Minimum KL-divergence on complements of L 1 balls Daniel Berend, Peter Harremoës, and Aryeh Kontorovich May 5, 2014 Abstract Pinsker s widely used inequality upper-bounds
More informationLecture 2: August 31
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 2: August 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy
More informationEE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15
EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15 1. Cascade of Binary Symmetric Channels The conditional probability distribution py x for each of the BSCs may be expressed by the transition probability
More informationTwo Applications of the Gaussian Poincaré Inequality in the Shannon Theory
Two Applications of the Gaussian Poincaré Inequality in the Shannon Theory Vincent Y. F. Tan (Joint work with Silas L. Fong) National University of Singapore (NUS) 2016 International Zurich Seminar on
More informationBounded Infinite Sequences/Functions : Orders of Infinity
Bounded Infinite Sequences/Functions : Orders of Infinity by Garimella Ramamurthy Report No: IIIT/TR/2009/247 Centre for Security, Theory and Algorithms International Institute of Information Technology
More informationConvergence of generalized entropy minimizers in sequences of convex problems
Proceedings IEEE ISIT 206, Barcelona, Spain, 2609 263 Convergence of generalized entropy minimizers in sequences of convex problems Imre Csiszár A Rényi Institute of Mathematics Hungarian Academy of Sciences
More informationSuperposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels
Superposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels Nan Liu and Andrea Goldsmith Department of Electrical Engineering Stanford University, Stanford CA 94305 Email:
More information5 Mutual Information and Channel Capacity
5 Mutual Information and Channel Capacity In Section 2, we have seen the use of a quantity called entropy to measure the amount of randomness in a random variable. In this section, we introduce several
More informationOn bounded redundancy of universal codes
On bounded redundancy of universal codes Łukasz Dębowski Institute of omputer Science, Polish Academy of Sciences ul. Jana Kazimierza 5, 01-248 Warszawa, Poland Abstract onsider stationary ergodic measures
More informationOn Bayes Risk Lower Bounds
Journal of Machine Learning Research 17 (2016) 1-58 Submitted 4/16; Revised 10/16; Published 12/16 On Bayes Risk Lower Bounds Xi Chen Stern School of Business New York University New York, NY 10012, USA
More informationNecessary and Sufficient Conditions for High-Dimensional Salient Feature Subset Recovery
Necessary and Sufficient Conditions for High-Dimensional Salient Feature Subset Recovery Vincent Tan, Matt Johnson, Alan S. Willsky Stochastic Systems Group, Laboratory for Information and Decision Systems,
More informationChannel Polarization and Blackwell Measures
Channel Polarization Blackwell Measures Maxim Raginsky Abstract The Blackwell measure of a binary-input channel (BIC is the distribution of the posterior probability of 0 under the uniform input distribution
More informationLecture 21: Minimax Theory
Lecture : Minimax Theory Akshay Krishnamurthy akshay@cs.umass.edu November 8, 07 Recap In the first part of the course, we spent the majority of our time studying risk minimization. We found many ways
More informationChapter 2: Entropy and Mutual Information. University of Illinois at Chicago ECE 534, Natasha Devroye
Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Log-sum inequality
More informationCapacity-Achieving Ensembles for the Binary Erasure Channel With Bounded Complexity
Capacity-Achieving Ensembles for the Binary Erasure Channel With Bounded Complexity Henry D. Pfister, Member, Igal Sason, Member, and Rüdiger Urbanke Abstract We present two sequences of ensembles of non-systematic
More information5.1 Inequalities via joint range
ECE598: Information-theoretic methods in high-dimensional statistics Spring 2016 Lecture 5: Inequalities between f-divergences via their joint range Lecturer: Yihong Wu Scribe: Pengkun Yang, Feb 9, 2016
More informationThe Information Lost in Erasures Sergio Verdú, Fellow, IEEE, and Tsachy Weissman, Senior Member, IEEE
5030 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 54, NO. 11, NOVEMBER 2008 The Information Lost in Erasures Sergio Verdú, Fellow, IEEE, Tsachy Weissman, Senior Member, IEEE Abstract We consider sources
More informationGuesswork Subject to a Total Entropy Budget
Guesswork Subject to a Total Entropy Budget Arman Rezaee, Ahmad Beirami, Ali Makhdoumi, Muriel Médard, and Ken Duffy arxiv:72.09082v [cs.it] 25 Dec 207 Abstract We consider an abstraction of computational
More informationA View on Extension of Utility-Based on Links with Information Measures
Communications of the Korean Statistical Society 2009, Vol. 16, No. 5, 813 820 A View on Extension of Utility-Based on Links with Information Measures A.R. Hoseinzadeh a, G.R. Mohtashami Borzadaran 1,b,
More informationFeedback Capacity of the Compound Channel
Feedback Capacity of the Compound Channel The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation As Published Publisher Shrader,
More informationChannel Dispersion and Moderate Deviations Limits for Memoryless Channels
Channel Dispersion and Moderate Deviations Limits for Memoryless Channels Yury Polyanskiy and Sergio Verdú Abstract Recently, Altug and Wagner ] posed a question regarding the optimal behavior of the probability
More informationGaussian Estimation under Attack Uncertainty
Gaussian Estimation under Attack Uncertainty Tara Javidi Yonatan Kaspi Himanshu Tyagi Abstract We consider the estimation of a standard Gaussian random variable under an observation attack where an adversary
More informationIntermittent Communication
Intermittent Communication Mostafa Khoshnevisan, Student Member, IEEE, and J. Nicholas Laneman, Senior Member, IEEE arxiv:32.42v2 [cs.it] 7 Mar 207 Abstract We formulate a model for intermittent communication
More informationBlock 2: Introduction to Information Theory
Block 2: Introduction to Information Theory Francisco J. Escribano April 26, 2015 Francisco J. Escribano Block 2: Introduction to Information Theory April 26, 2015 1 / 51 Table of contents 1 Motivation
More informationEECS 750. Hypothesis Testing with Communication Constraints
EECS 750 Hypothesis Testing with Communication Constraints Name: Dinesh Krithivasan Abstract In this report, we study a modification of the classical statistical problem of bivariate hypothesis testing.
More informationSeries 7, May 22, 2018 (EM Convergence)
Exercises Introduction to Machine Learning SS 2018 Series 7, May 22, 2018 (EM Convergence) Institute for Machine Learning Dept. of Computer Science, ETH Zürich Prof. Dr. Andreas Krause Web: https://las.inf.ethz.ch/teaching/introml-s18
More informationIEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 12, DECEMBER
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 53, NO 12, DECEMBER 2007 4457 Joint Source Channel Coding Error Exponent for Discrete Communication Systems With Markovian Memory Yangfan Zhong, Student Member,
More information4488 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 54, NO. 10, OCTOBER /$ IEEE
4488 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 54, NO. 10, OCTOBER 2008 List Decoding of Biorthogonal Codes the Hadamard Transform With Linear Complexity Ilya Dumer, Fellow, IEEE, Grigory Kabatiansky,
More informationIntroduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information.
L65 Dept. of Linguistics, Indiana University Fall 205 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission rate
More informationDept. of Linguistics, Indiana University Fall 2015
L645 Dept. of Linguistics, Indiana University Fall 2015 1 / 28 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission
More informationCut-Set Bound and Dependence Balance Bound
Cut-Set Bound and Dependence Balance Bound Lei Xiao lxiao@nd.edu 1 Date: 4 October, 2006 Reading: Elements of information theory by Cover and Thomas [1, Section 14.10], and the paper by Hekstra and Willems
More informationQuantum Sphere-Packing Bounds and Moderate Deviation Analysis for Classical-Quantum Channels
Quantum Sphere-Packing Bounds and Moderate Deviation Analysis for Classical-Quantum Channels (, ) Joint work with Min-Hsiu Hsieh and Marco Tomamichel Hao-Chung Cheng University of Technology Sydney National
More informationSubset Universal Lossy Compression
Subset Universal Lossy Compression Or Ordentlich Tel Aviv University ordent@eng.tau.ac.il Ofer Shayevitz Tel Aviv University ofersha@eng.tau.ac.il Abstract A lossy source code C with rate R for a discrete
More informationMultiaccess Channels with State Known to One Encoder: A Case of Degraded Message Sets
Multiaccess Channels with State Known to One Encoder: A Case of Degraded Message Sets Shivaprasad Kotagiri and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame,
More informationA Formula for the Capacity of the General Gel fand-pinsker Channel
A Formula for the Capacity of the General Gel fand-pinsker Channel Vincent Y. F. Tan Institute for Infocomm Research (I 2 R, A*STAR, Email: tanyfv@i2r.a-star.edu.sg ECE Dept., National University of Singapore
More informationAn Alternative Proof for the Capacity Region of the Degraded Gaussian MIMO Broadcast Channel
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 58, NO. 4, APRIL 2012 2427 An Alternative Proof for the Capacity Region of the Degraded Gaussian MIMO Broadcast Channel Ersen Ekrem, Student Member, IEEE,
More informationThe Compound Capacity of Polar Codes
The Compound Capacity of Polar Codes S. Hamed Hassani, Satish Babu Korada and Rüdiger Urbanke arxiv:97.329v [cs.it] 9 Jul 29 Abstract We consider the compound capacity of polar codes under successive cancellation
More informationThe Fading Number of a Multiple-Access Rician Fading Channel
The Fading Number of a Multiple-Access Rician Fading Channel Intermediate Report of NSC Project Capacity Analysis of Various Multiple-Antenna Multiple-Users Communication Channels with Joint Estimation
More informationTightened Upper Bounds on the ML Decoding Error Probability of Binary Linear Block Codes and Applications
on the ML Decoding Error Probability of Binary Linear Block Codes and Moshe Twitto Department of Electrical Engineering Technion-Israel Institute of Technology Haifa 32000, Israel Joint work with Igal
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationSimple Channel Coding Bounds
ISIT 2009, Seoul, Korea, June 28 - July 3,2009 Simple Channel Coding Bounds Ligong Wang Signal and Information Processing Laboratory wang@isi.ee.ethz.ch Roger Colbeck Institute for Theoretical Physics,
More informationRényi Information Dimension: Fundamental Limits of Almost Lossless Analog Compression
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 8, AUGUST 2010 3721 Rényi Information Dimension: Fundamental Limits of Almost Lossless Analog Compression Yihong Wu, Student Member, IEEE, Sergio Verdú,
More informationOptimal Distributed Detection Strategies for Wireless Sensor Networks
Optimal Distributed Detection Strategies for Wireless Sensor Networks Ke Liu and Akbar M. Sayeed University of Wisconsin-Madison kliu@cae.wisc.edu, akbar@engr.wisc.edu Abstract We study optimal distributed
More informationApproaching Blokh-Zyablov Error Exponent with Linear-Time Encodable/Decodable Codes
Approaching Blokh-Zyablov Error Exponent with Linear-Time Encodable/Decodable Codes 1 Zheng Wang, Student Member, IEEE, Jie Luo, Member, IEEE arxiv:0808.3756v1 [cs.it] 27 Aug 2008 Abstract We show that
More informationSHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe
SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe 2/41 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel
More information4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information
4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information Ramji Venkataramanan Signal Processing and Communications Lab Department of Engineering ramji.v@eng.cam.ac.uk
More informationMismatched Multi-letter Successive Decoding for the Multiple-Access Channel
Mismatched Multi-letter Successive Decoding for the Multiple-Access Channel Jonathan Scarlett University of Cambridge jms265@cam.ac.uk Alfonso Martinez Universitat Pompeu Fabra alfonso.martinez@ieee.org
More informationOn Competitive Prediction and Its Relation to Rate-Distortion Theory
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 49, NO. 12, DECEMBER 2003 3185 On Competitive Prediction and Its Relation to Rate-Distortion Theory Tsachy Weissman, Member, IEEE, and Neri Merhav, Fellow,
More informationVariable Length Codes for Degraded Broadcast Channels
Variable Length Codes for Degraded Broadcast Channels Stéphane Musy School of Computer and Communication Sciences, EPFL CH-1015 Lausanne, Switzerland Email: stephane.musy@ep.ch Abstract This paper investigates
More informationOn Third-Order Asymptotics for DMCs
On Third-Order Asymptotics for DMCs Vincent Y. F. Tan Institute for Infocomm Research (I R) National University of Singapore (NUS) January 0, 013 Vincent Tan (I R and NUS) Third-Order Asymptotics for DMCs
More informationGoodness of Fit Test and Test of Independence by Entropy
Journal of Mathematical Extension Vol. 3, No. 2 (2009), 43-59 Goodness of Fit Test and Test of Independence by Entropy M. Sharifdoost Islamic Azad University Science & Research Branch, Tehran N. Nematollahi
More information