Literature on Bregman divergences
|
|
- Edward Greer
- 5 years ago
- Views:
Transcription
1 Literature on Bregman divergences Lecture series at Univ. Hawai i at Mānoa Peter Harremoës February 26, 2016 Information divergence was introduced by Kullback and Leibler [25] and later Kullback started using information theory in statistics [24] and here information divergence play a crusial role. Information divergence was used already by Wald [35] although he did not give this quantity a name. The basic properties of information divergence are now described in many textbooks. Optimization with information divergence was described in a systematic way by Topsøe [33]. Relation to the conditional limit theorem can be found in [9]. Alternating minimization was studied in [11]. Information projections and reversed projections are described in [10]. Information divergence can also be used to define a topology with some strange properties [15]. There have been many attempts to generalize the notion of information divergence to a wider class of divergence. There have been two different types of motivation for generalizing information divergence. One motivation has been that quantities that share some properties with information divergence are used in physics, statistics, probability theory or other parts of information theory. If this is the motivation one often has to compare related quantities by inequalities or similar results. This motivation has lead to a great number of good results. Another motivation has been generalization in the hope that some generalized version of information divergence will turn out to be useful. There are many papers that take this approach but most of the divergences that have emerged in this way have never been used again. One important exception is Rényi divergence introduced [31]. All the basic properties of information divergence and Rényi divergence were recently described in [34]. The class of f-divergences were introduced independently by Csiszár and Morimoto [8, 29] and a little later again by Ali and Silvey [2]. The f-divergences generalize information divergence in such a way that convexity and the data processing inequality are still satisfied. It includes various quantities used in statistics including the χ 2 - divergence. In statistics a major question therefore is which f-divergence to use for a specific problem. The standard reference is [26]. An important result is that if the probability measures are close together then it does not make much difference which divergence is used [27]. If the notion of Bahadur efficiency is used information divergence should normally be preferred [21]. In some cases the distribution information 1
2 divergence is closer to a χ 2 -distribution that other f-divergences [20, 16]. There are many papers on inequalities between f-divergences but it has been shown that an inequality that holds for a binary alphabet holds for any alphabet [22]. Bregman divergences were introduced in [6], but for a long time they did not received the attention they deserve. The Bregman divergence may be characterized as divergences that satisfy the Bregman identity. In the context of Information theory Bregman divergences were partly reinvented by Rao and Nayak[30] where the name cross entropy were proposed and this term is still in use in some groups of scientists. Until 2005 there were only few papers on Bregman divergences but in the paper Clustering with Bregman Divergences [3] all the basic properties of Bregman divergences were described. The paper also clarify the relation between Bregman divergences and exponential families. The sufficiency condition was first used to characterize divergences in [19] and in [23] it was proved that the sufficiency condition can be used to characterize information divergence. This idea was further developed in [17, 18]. The relation between Bregman divergences and metrics was described in [7] and [1]. Inspired by results from 2-person 0-sum games in the 1940 ties Wald developed the idea that in situations with uncertainty one should make decisions that maximize the minimal payoff. This decision criterion is very robust but is often too pessimistic for real world decisions. In 1951 Sevage introduced the minimax regret criterion in decision theory as an alternate criterion for decision making. Regret was introduced as an inference criteria in statistics in 1978 by Rissanen[32] but this idea took slowly momentum, but has now developed into a competitor to Bayesian statistics and the frequential interpretation of statistics allthough it is still not so widely known[4, 14, 13]. The use of regret in economy did not take momentum before 1982 where the idea was revived in a number of papers [12, 28, 5]. Now there is also active research in psykological aspect of using regret as a decision criterion. The relation between Bregman divergences, decision theory and regret is described in [17, 18]. References [1] S. Acharyya, A. Banerjee, and D. Boley. Bregman Divergences and Triangle Inequality, chapter 52, pages [2] S. M. Ali and S. D. Silvey. A general class of coefficients of divergence of one distribution from another. J. Roy. Statist. Soc. Ser B, 28: , [3] Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with Bregman divergences. Journal of Machine Learning Research, 6: , [4] A. R. Barron, J. Rissanen, and B. Yu. The minimum description length principle in coding and modeling. IEEE Trans. Inform. Theory, 44(6): , Oct Commemorative issue. 2
3 [5] D. E. Bell. Regret in decision making under uncertainty. Operations research, 30(5): , [6] L. M. Bregman. The relaxation method of finding the common point of convex sets and itsapplication to the solution of problems in convex programming. USSR Comput. Math. and Math. Phys., 7: , Translated from Russian. [7] P. Chen, Y. Chen, and M. Rao. Metrics defined by bregman divergences. Commun. Math. Sci., 6(4): , [8] I. Csiszár. Eine informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der ergodizität von Markoffschen Ketten. Publ. Math. Inst. Hungar. Acad., 8:95 108, [9] I. Csiszár. Sanov property, generalized I-projection and a conditional limit theorem. Ann. Probab., 12: , [10] I. Csiszár and F. Matús. Information projections revisited. IEEE Trans. Inform. Theory, 49(6): , June [11] I. Csiszár and G. Tusnady. Information geometry and alternating minimization procedures. Statistics and Decisions, Supplementary Issue 1: , [12] P. C. Fishburn. The foundations of expected utility. Theory & Decision Library, [13] P. Grünwald. the Minimum Description Length principle. MIT Press, [14] P. D. Grünwald and A. P. Dawid. Game theory, maximum entropy, minimum discrepancy, and robust Bayesian decision theory. Annals of Mathematical Statistics, 32(4): , [15] P. Harremoës. The information topology. In Proceedings IEEE International Symposium on Information Theory, page 431, Lausanne, June IEEE. [16] P. Harremoës. Mutual information of contingency tables and related inequalities. In Proceedings ISIT 2014, pages IEEE, June [17] P. Harremoës. Proper scoring and sufficiency. In J. Rissanen, P. Harremoës, S. Forchhammer, T. Roos, and P. Myllymäke, editors, Proceeding of the The Eighth Workshop on Information Theoretic Methods in Science and Engineering, number Report B in Series of Publications B, pages 19 22, University of Helsinki, Department of Computer Science, An appendix with proofs only exists in the arxiv version of the paper. [18] P. Harremoës. Sufficiency on the stock market. Submitted, Jan
4 [19] P. Harremoës and N. Tishby. The information bottleneck revisited or how to choose a good distortion measure. In Proceedings ISIT 2007, Nice, pages IEEE Information Theory Society, June [20] P. Harremoës and G. Tusnády. Information divergence is more χ 2 -distributed than the χ 2 -statistic. In International Symposium on Information Theory (ISIT 2012), pages , Cambridge, Massachusetts, USA, July IEEE. [21] P. Harremoës and I. Vajda. On the Bahadur-efficient testing of uniformity by means of the entropy. IEEE Trans. Inform Theory, 54(1): , Jan [22] P. Harremoës and I. Vajda. On pairs of f-divergences and their joint range. IEEE Tranns. Inform. Theory, 57(6): , June [23] Jiantao Jiao, Thomas Courtade amd Albert No, Kartik Venkat, and Tsachy Weissman. Information measures: the curious case of the binary alphabet. Trans. Inform. Theory, 60(12): , Dec [24] S. Kullback. Information Theory and Statistics. Wiley, New York, [25] S. Kullback and R. Leibler. On information and sufficiency. Ann. Math. Statist., 22:79 86, [26] F. Liese and I. Vajda. Convex Statistical Distances. Teubner, Leipzig, [27] F. Liese and I. Vajda. On divergence and informations in statistics and information theory. IEEE Tranns. Inform. Theory, 52(10): , Oct [28] G. Loomes and R. Sugden. Regret theory: An alternative theory of rational choice under uncertainty. Economic Journal, 92(4): , [29] T. Morimoto. Markov processes and the h-theorem. J. Phys. Soc. Jap., 12: , [30] C. R. Rao and T. K. Nayak. Cross entropy, dissimilarity measures, and characterizations of quadratic entropy. IEEE Trans. Inform. Theory, 31(5): , September [31] Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages , [32] J. Rissanen. Modelling by shortest data description. Automatica, 14: , [33] F. Topsøe. Information theoretical optimization techniques. Kybernetika, 15(1):8 27,
5 [34] T. van Erven and P. Harremoës. Rényi divergence and Kullback-Leibler divergence. IEEE Trans Inform. Theory, 60(7): , July [35] A. Wald. Sequensial Analysis. Wiley,
The Information Bottleneck Revisited or How to Choose a Good Distortion Measure
The Information Bottleneck Revisited or How to Choose a Good Distortion Measure Peter Harremoës Centrum voor Wiskunde en Informatica PO 94079, 1090 GB Amsterdam The Nederlands PHarremoes@cwinl Naftali
More informationPROPERTIES. Ferdinand Österreicher Institute of Mathematics, University of Salzburg, Austria
CSISZÁR S f-divergences - BASIC PROPERTIES Ferdinand Österreicher Institute of Mathematics, University of Salzburg, Austria Abstract In this talk basic general properties of f-divergences, including their
More informationON A CLASS OF PERIMETER-TYPE DISTANCES OF PROBABILITY DISTRIBUTIONS
KYBERNETIKA VOLUME 32 (1996), NUMBER 4, PAGES 389-393 ON A CLASS OF PERIMETER-TYPE DISTANCES OF PROBABILITY DISTRIBUTIONS FERDINAND OSTERREICHER The class If, p G (l,oo], of /-divergences investigated
More informationInformation Divergences and the Curious Case of the Binary Alphabet
Information Divergences and the Curious Case of the Binary Alphabet Jiantao Jiao, Thomas Courtade, Albert No, Kartik Venkat, and Tsachy Weissman Department of Electrical Engineering, Stanford University;
More informationarxiv: v4 [cs.it] 17 Oct 2015
Upper Bounds on the Relative Entropy and Rényi Divergence as a Function of Total Variation Distance for Finite Alphabets Igal Sason Department of Electrical Engineering Technion Israel Institute of Technology
More informationConvexity/Concavity of Renyi Entropy and α-mutual Information
Convexity/Concavity of Renyi Entropy and -Mutual Information Siu-Wai Ho Institute for Telecommunications Research University of South Australia Adelaide, SA 5095, Australia Email: siuwai.ho@unisa.edu.au
More informationSome New Information Inequalities Involving f-divergences
BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 12, No 2 Sofia 2012 Some New Information Inequalities Involving f-divergences Amit Srivastava Department of Mathematics, Jaypee
More informationCONVEX FUNCTIONS AND MATRICES. Silvestru Sever Dragomir
Korean J. Math. 6 (018), No. 3, pp. 349 371 https://doi.org/10.11568/kjm.018.6.3.349 INEQUALITIES FOR QUANTUM f-divergence OF CONVEX FUNCTIONS AND MATRICES Silvestru Sever Dragomir Abstract. Some inequalities
More informationA View on Extension of Utility-Based on Links with Information Measures
Communications of the Korean Statistical Society 2009, Vol. 16, No. 5, 813 820 A View on Extension of Utility-Based on Links with Information Measures A.R. Hoseinzadeh a, G.R. Mohtashami Borzadaran 1,b,
More informationFull text available at: Information Theory and Statistics: A Tutorial
Information Theory and Statistics: A Tutorial Information Theory and Statistics: A Tutorial Imre Csiszár Rényi Institute of Mathematics, Hungarian Academy of Sciences POB 127, H-1364 Budapest, Hungary
More informationExperimental Design to Maximize Information
Experimental Design to Maximize Information P. Sebastiani and H.P. Wynn Department of Mathematics and Statistics University of Massachusetts at Amherst, 01003 MA Department of Statistics, University of
More informationBregman Divergences for Data Mining Meta-Algorithms
p.1/?? Bregman Divergences for Data Mining Meta-Algorithms Joydeep Ghosh University of Texas at Austin ghosh@ece.utexas.edu Reflects joint work with Arindam Banerjee, Srujana Merugu, Inderjit Dhillon,
More informationTight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding
APPEARS IN THE IEEE TRANSACTIONS ON INFORMATION THEORY, FEBRUARY 015 1 Tight Bounds for Symmetric Divergence Measures and a Refined Bound for Lossless Source Coding Igal Sason Abstract Tight bounds for
More informationarxiv: v2 [cs.it] 10 Apr 2017
Divergence and Sufficiency for Convex Optimization Peter Harremoës April 11, 2017 arxiv:1701.01010v2 [cs.it] 10 Apr 2017 Abstract Logarithmic score and information divergence appear in information theory,
More informationEntropy measures of physics via complexity
Entropy measures of physics via complexity Giorgio Kaniadakis and Flemming Topsøe Politecnico of Torino, Department of Physics and University of Copenhagen, Department of Mathematics 1 Introduction, Background
More informationNash equilibrium in a game of calibration
Nash equilibrium in a game of calibration Omar Glonti, Peter Harremoës, Zaza Khechinashvili and Flemming Topsøe OG and ZK: I. Javakhishvili Tbilisi State University Laboratory of probabilistic and statistical
More informationarxiv: v1 [cs.lg] 22 Oct 2018
The Bregman chord divergence arxiv:1810.09113v1 [cs.lg] Oct 018 rank Nielsen Sony Computer Science Laboratories Inc, Japan rank.nielsen@acm.org Richard Nock Data 61, Australia The Australian National &
More informationWE start with a general discussion. Suppose we have
646 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 2, MARCH 1997 Minimax Redundancy for the Class of Memoryless Sources Qun Xie and Andrew R. Barron, Member, IEEE Abstract Let X n = (X 1 ; 111;Xn)be
More informationMinimum Phi-Divergence Estimators and Phi-Divergence Test Statistics in Contingency Tables with Symmetry Structure: An Overview
Symmetry 010,, 1108-110; doi:10.3390/sym01108 OPEN ACCESS symmetry ISSN 073-8994 www.mdpi.com/journal/symmetry Review Minimum Phi-Divergence Estimators and Phi-Divergence Test Statistics in Contingency
More informationRobustness and duality of maximum entropy and exponential family distributions
Chapter 7 Robustness and duality of maximum entropy and exponential family distributions In this lecture, we continue our study of exponential families, but now we investigate their properties in somewhat
More informationOn The Asymptotics of Minimum Disparity Estimation
Noname manuscript No. (will be inserted by the editor) On The Asymptotics of Minimum Disparity Estimation Arun Kumar Kuchibhotla Ayanendranath Basu Received: date / Accepted: date Abstract Inference procedures
More informationTHEOREM AND METRIZABILITY
f-divergences - REPRESENTATION THEOREM AND METRIZABILITY Ferdinand Österreicher Institute of Mathematics, University of Salzburg, Austria Abstract In this talk we are first going to state the so-called
More informationA New Quantum f-divergence for Trace Class Operators in Hilbert Spaces
Entropy 04, 6, 5853-5875; doi:0.3390/e65853 OPEN ACCESS entropy ISSN 099-4300 www.mdpi.com/journal/entropy Article A New Quantum f-divergence for Trace Class Operators in Hilbert Spaces Silvestru Sever
More informationarxiv: v2 [math.pr] 8 Feb 2016
Noname manuscript No will be inserted by the editor Bounds on Tail Probabilities in Exponential families Peter Harremoës arxiv:600579v [mathpr] 8 Feb 06 Received: date / Accepted: date Abstract In this
More informationJensen-Shannon Divergence and Hilbert space embedding
Jensen-Shannon Divergence and Hilbert space embedding Bent Fuglede and Flemming Topsøe University of Copenhagen, Department of Mathematics Consider the set M+ 1 (A) of probability distributions where A
More informationGoodness of Fit Test and Test of Independence by Entropy
Journal of Mathematical Extension Vol. 3, No. 2 (2009), 43-59 Goodness of Fit Test and Test of Independence by Entropy M. Sharifdoost Islamic Azad University Science & Research Branch, Tehran N. Nematollahi
More informationMinimum message length estimation of mixtures of multivariate Gaussian and von Mises-Fisher distributions
Minimum message length estimation of mixtures of multivariate Gaussian and von Mises-Fisher distributions Parthan Kasarapu & Lloyd Allison Monash University, Australia September 8, 25 Parthan Kasarapu
More informationMathematical Foundations of the Generalization of t-sne and SNE for Arbitrary Divergences
MACHINE LEARNING REPORTS Mathematical Foundations of the Generalization of t-sne and SNE for Arbitrary Report 02/2010 Submitted: 01.04.2010 Published:26.04.2010 T. Villmann and S. Haase University of Applied
More informationTight Bounds for Symmetric Divergence Measures and a New Inequality Relating f-divergences
Tight Bounds for Symmetric Divergence Measures and a New Inequality Relating f-divergences Igal Sason Department of Electrical Engineering Technion, Haifa 3000, Israel E-mail: sason@ee.technion.ac.il Abstract
More informationConjugate Predictive Distributions and Generalized Entropies
Conjugate Predictive Distributions and Generalized Entropies Eduardo Gutiérrez-Peña Department of Probability and Statistics IIMAS-UNAM, Mexico Padova, Italy. 21-23 March, 2013 Menu 1 Antipasto/Appetizer
More informationMutual information of Contingency Tables and Related Inequalities
Mutual information of Contingency Tables and Related Inequalities Peter Harremoës Copenhagen Business College Copenhagen, Denmark Email: harremoes@ieeeorg arxiv:4020092v [mathst] Feb 204 Abstract For testing
More informationInformation Measures: The Curious Case of the Binary Alphabet
7616 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 60, NO. 12, DECEMBER 2014 Information Measures: The Curious Case of the Binary Alphabet Jiantao Jiao, Student Member, IEEE, ThomasA.Courtade,Member, IEEE,
More informationBayesian Properties of Normalized Maximum Likelihood and its Fast Computation
Bayesian Properties of Normalized Maximum Likelihood and its Fast Computation Andre Barron Department of Statistics Yale University Email: andre.barron@yale.edu Teemu Roos HIIT & Dept of Computer Science
More informationAlpha/Beta Divergences and Tweedie Models
Alpha/Beta Divergences and Tweedie Models 1 Y. Kenan Yılmaz A. Taylan Cemgil Department of Computer Engineering Boğaziçi University, Istanbul, Turkey kenan@sibnet.com.tr, taylan.cemgil@boun.edu.tr arxiv:1209.4280v1
More informationRisk Measurement Robust under Model Uncertainty
Risk Measurement Robust under Model Uncertainty Thomas Breuer Imre Csiszár Abstract Systematic model stress tests identify worst case risk factor distributions (models) satisfying some plausibility constraint.
More informationInformation, Utility & Bounded Rationality
Information, Utility & Bounded Rationality Pedro A. Ortega and Daniel A. Braun Department of Engineering, University of Cambridge Trumpington Street, Cambridge, CB2 PZ, UK {dab54,pao32}@cam.ac.uk Abstract.
More informationReceived: 20 December 2011; in revised form: 4 February 2012 / Accepted: 7 February 2012 / Published: 2 March 2012
Entropy 2012, 14, 480-490; doi:10.3390/e14030480 Article OPEN ACCESS entropy ISSN 1099-4300 www.mdpi.com/journal/entropy Interval Entropy and Informative Distance Fakhroddin Misagh 1, * and Gholamhossein
More informationInformation Measures: the Curious Case of the Binary Alphabet
Information Measures: the Curious Case of the Binary Alphabet Jiantao Jiao, Student Member, IEEE, Thomas A. Courtade, Member, IEEE, Albert No, Student Member, IEEE, Kartik Venkat, Student Member, IEEE,
More informationOn Improved Bounds for Probability Metrics and f- Divergences
IRWIN AND JOAN JACOBS CENTER FOR COMMUNICATION AND INFORMATION TECHNOLOGIES On Improved Bounds for Probability Metrics and f- Divergences Igal Sason CCIT Report #855 March 014 Electronics Computers Communications
More informationInstitut für Mathematik
U n i v e r s i t ä t A u g s b u r g Institut für Mathematik Christoph Gietl, Fabian P. Reffel Continuity of f-projections and Applications to the Iterative Proportional Fitting Procedure Preprint Nr.
More informationSTATISTICAL CURVATURE AND STOCHASTIC COMPLEXITY
2nd International Symposium on Information Geometry and its Applications December 2-6, 2005, Tokyo Pages 000 000 STATISTICAL CURVATURE AND STOCHASTIC COMPLEXITY JUN-ICHI TAKEUCHI, ANDREW R. BARRON, AND
More informationBelief Propagation, Information Projections, and Dykstra s Algorithm
Belief Propagation, Information Projections, and Dykstra s Algorithm John MacLaren Walsh, PhD Department of Electrical and Computer Engineering Drexel University Philadelphia, PA jwalsh@ece.drexel.edu
More informationOn the Entropy of Sums of Bernoulli Random Variables via the Chen-Stein Method
On the Entropy of Sums of Bernoulli Random Variables via the Chen-Stein Method Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, Israel ETH, Zurich,
More informationCOMPSCI 650 Applied Information Theory Jan 21, Lecture 2
COMPSCI 650 Applied Information Theory Jan 21, 2016 Lecture 2 Instructor: Arya Mazumdar Scribe: Gayane Vardoyan, Jong-Chyi Su 1 Entropy Definition: Entropy is a measure of uncertainty of a random variable.
More informationInformation Measure Estimation and Applications: Boosting the Effective Sample Size from n to n ln n
Information Measure Estimation and Applications: Boosting the Effective Sample Size from n to n ln n Jiantao Jiao (Stanford EE) Joint work with: Kartik Venkat Yanjun Han Tsachy Weissman Stanford EE Tsinghua
More informationAn Introduction to Functional Derivatives
An Introduction to Functional Derivatives Béla A. Frigyik, Santosh Srivastava, Maya R. Gupta Dept of EE, University of Washington Seattle WA, 98195-2500 UWEE Technical Report Number UWEETR-2008-0001 January
More informationGeneralized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels
Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels Liming Wang, Miguel Rodrigues, Lawrence Carin Dept. of Electrical & Computer Engineering, Duke University,
More informationGeneralized Neyman Pearson optimality of empirical likelihood for testing parameter hypotheses
Ann Inst Stat Math (2009) 61:773 787 DOI 10.1007/s10463-008-0172-6 Generalized Neyman Pearson optimality of empirical likelihood for testing parameter hypotheses Taisuke Otsu Received: 1 June 2007 / Revised:
More informationMaximum Likelihood Approach for Symmetric Distribution Property Estimation
Maximum Likelihood Approach for Symmetric Distribution Property Estimation Jayadev Acharya, Hirakendu Das, Alon Orlitsky, Ananda Suresh Cornell, Yahoo, UCSD, Google Property estimation p: unknown discrete
More informationArimoto Channel Coding Converse and Rényi Divergence
Arimoto Channel Coding Converse and Rényi Divergence Yury Polyanskiy and Sergio Verdú Abstract Arimoto proved a non-asymptotic upper bound on the probability of successful decoding achievable by any code
More information3. If a choice is broken down into two successive choices, the original H should be the weighted sum of the individual values of H.
Appendix A Information Theory A.1 Entropy Shannon (Shanon, 1948) developed the concept of entropy to measure the uncertainty of a discrete random variable. Suppose X is a discrete random variable that
More informationUncertainty. Jayakrishnan Unnikrishnan. CSL June PhD Defense ECE Department
Decision-Making under Statistical Uncertainty Jayakrishnan Unnikrishnan PhD Defense ECE Department University of Illinois at Urbana-Champaign CSL 141 12 June 2010 Statistical Decision-Making Relevant in
More informationInformation Theory in Intelligent Decision Making
Information Theory in Intelligent Decision Making Adaptive Systems and Algorithms Research Groups School of Computer Science University of Hertfordshire, United Kingdom June 7, 2015 Information Theory
More informationInformation Geometry on Hierarchy of Probability Distributions
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 47, NO. 5, JULY 2001 1701 Information Geometry on Hierarchy of Probability Distributions Shun-ichi Amari, Fellow, IEEE Abstract An exponential family or mixture
More informationA Modification of Linfoot s Informational Correlation Coefficient
Austrian Journal of Statistics April 07, Volume 46, 99 05. AJS http://www.ajs.or.at/ doi:0.773/ajs.v46i3-4.675 A Modification of Linfoot s Informational Correlation Coefficient Georgy Shevlyakov Peter
More informationExponentiated Gradient Descent
CSE599s, Spring 01, Online Learning Lecture 10-04/6/01 Lecturer: Ofer Dekel Exponentiated Gradient Descent Scribe: Albert Yu 1 Introduction In this lecture we review norms, dual norms, strong convexity,
More informationarxiv: v4 [cs.it] 8 Apr 2014
1 On Improved Bounds for Probability Metrics and f-divergences Igal Sason Abstract Derivation of tight bounds for probability metrics and f-divergences is of interest in information theory and statistics.
More informationA SYMMETRIC INFORMATION DIVERGENCE MEASURE OF CSISZAR'S F DIVERGENCE CLASS
Journal of the Applied Mathematics, Statistics Informatics (JAMSI), 7 (2011), No. 1 A SYMMETRIC INFORMATION DIVERGENCE MEASURE OF CSISZAR'S F DIVERGENCE CLASS K.C. JAIN AND R. MATHUR Abstract Information
More informationDIVERGENCES (or pseudodistances) based on likelihood
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 56, NO 11, NOVEMBER 2010 5847 Estimating Divergence Functionals the Likelihood Ratio by Convex Risk Minimization XuanLong Nguyen, Martin J Wainwright, Michael
More informationA GENERAL CLASS OF LOWER BOUNDS ON THE PROBABILITY OF ERROR IN MULTIPLE HYPOTHESIS TESTING. Tirza Routtenberg and Joseph Tabrikian
A GENERAL CLASS OF LOWER BOUNDS ON THE PROBABILITY OF ERROR IN MULTIPLE HYPOTHESIS TESTING Tirza Routtenberg and Joseph Tabrikian Department of Electrical and Computer Engineering Ben-Gurion University
More informationDivergence measures for statistical data processing
Divergence measures for statistical data processing Michèle Basseville To cite this version: Michèle Basseville. Divergence measures for statistical data processing. [Research Report] PI-1961, 2010, pp.23.
More informationSome History of Optimality
IMS Lecture Notes- Monograph Series Optimality: The Third Erich L. Lehmann Symposium Vol. 57 (2009) 11-17 @ Institute of Mathematical Statistics, 2009 DOl: 10.1214/09-LNMS5703 Erich L. Lehmann University
More informationLearning Methods for Online Prediction Problems. Peter Bartlett Statistics and EECS UC Berkeley
Learning Methods for Online Prediction Problems Peter Bartlett Statistics and EECS UC Berkeley Course Synopsis A finite comparison class: A = {1,..., m}. Converting online to batch. Online convex optimization.
More informationGaussian Estimation under Attack Uncertainty
Gaussian Estimation under Attack Uncertainty Tara Javidi Yonatan Kaspi Himanshu Tyagi Abstract We consider the estimation of a standard Gaussian random variable under an observation attack where an adversary
More informationEstimation of signal information content for classification
Estimation of signal information content for classification The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation As Published
More informationOn the Chi square and higher-order Chi distances for approximating f-divergences
c 2013 Frank Nielsen, Sony Computer Science Laboratories, Inc. 1/17 On the Chi square and higher-order Chi distances for approximating f-divergences Frank Nielsen 1 Richard Nock 2 www.informationgeometry.org
More informationSpeech Recognition Lecture 7: Maximum Entropy Models. Mehryar Mohri Courant Institute and Google Research
Speech Recognition Lecture 7: Maximum Entropy Models Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.com This Lecture Information theory basics Maximum entropy models Duality theorem
More informationBregman Divergences. Barnabás Póczos. RLAI Tea Talk UofA, Edmonton. Aug 5, 2008
Bregman Divergences Barnabás Póczos RLAI Tea Talk UofA, Edmonton Aug 5, 2008 Contents Bregman Divergences Bregman Matrix Divergences Relation to Exponential Family Applications Definition Properties Generalization
More informationDivergences, surrogate loss functions and experimental design
Divergences, surrogate loss functions and experimental design XuanLong Nguyen University of California Berkeley, CA 94720 xuanlong@cs.berkeley.edu Martin J. Wainwright University of California Berkeley,
More informationExtremal properties of the variance and the quantum Fisher information; Phys. Rev. A 87, (2013).
1 / 24 Extremal properties of the variance and the quantum Fisher information; Phys. Rev. A 87, 032324 (2013). G. Tóth 1,2,3 and D. Petz 4,5 1 Theoretical Physics, University of the Basque Country UPV/EHU,
More informationCurvilinear Components Analysis and Bregman Divergences
and Machine Learning. Bruges (Belgium), 8-3 April, d-side publi., ISBN -9337--. Curvilinear Components Analysis and Bregman Divergences Jigang Sun, Malcolm Crowe and Colin Fyfe Applied Computational Intelligence
More informationA Generalized Fuzzy Inaccuracy Measure of Order ɑ and Type β and Coding Theorems
International Journal of Fuzzy Mathematics and Systems. ISSN 2248-9940 Volume 4, Number (204), pp. 27-37 Research India Publications http://www.ripublication.com A Generalized Fuzzy Inaccuracy Measure
More informationResearch Article On Some Improvements of the Jensen Inequality with Some Applications
Hindawi Publishing Corporation Journal of Inequalities and Applications Volume 2009, Article ID 323615, 15 pages doi:10.1155/2009/323615 Research Article On Some Improvements of the Jensen Inequality with
More informationComputational Systems Biology: Biology X
Bud Mishra Room 1002, 715 Broadway, Courant Institute, NYU, New York, USA L#8:(November-08-2010) Cancer and Signals Outline 1 Bayesian Interpretation of Probabilities Information Theory Outline Bayesian
More informationInformation Theory and Hypothesis Testing
Summer School on Game Theory and Telecommunications Campione, 7-12 September, 2014 Information Theory and Hypothesis Testing Mauro Barni University of Siena September 8 Review of some basic results linking
More informationHOPFIELD neural networks (HNNs) are a class of nonlinear
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 52, NO. 4, APRIL 2005 213 Stochastic Noise Process Enhancement of Hopfield Neural Networks Vladimir Pavlović, Member, IEEE, Dan Schonfeld,
More informationONE SILVESTRU SEVER DRAGOMIR 1;2
SOME -DIVERGENCE MEASURES RELATED TO JENSEN S ONE SILVESTRU SEVER DRAGOMIR ; Abstract. In this paper we introuce some -ivergence measures that are relate to the Jensen s ivergence introuce by Burbea an
More informationTesting Goodness-of-Fit for Exponential Distribution Based on Cumulative Residual Entropy
This article was downloaded by: [Ferdowsi University] On: 16 April 212, At: 4:53 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 172954 Registered office: Mortimer
More informationarxiv: v1 [cs.dm] 27 Jan 2014
Randomized Minmax Regret for Combinatorial Optimization Under Uncertainty Andrew Mastin, Patrick Jaillet, Sang Chin arxiv:1401.7043v1 [cs.dm] 27 Jan 2014 January 29, 2014 Abstract The minmax regret problem
More informationConvergence of generalized entropy minimizers in sequences of convex problems
Proceedings IEEE ISIT 206, Barcelona, Spain, 2609 263 Convergence of generalized entropy minimizers in sequences of convex problems Imre Csiszár A Rényi Institute of Mathematics Hungarian Academy of Sciences
More information6.891 Games, Decision, and Computation February 5, Lecture 2
6.891 Games, Decision, and Computation February 5, 2015 Lecture 2 Lecturer: Constantinos Daskalakis Scribe: Constantinos Daskalakis We formally define games and the solution concepts overviewed in Lecture
More informationKeep it Simple Stupid On the Effect of Lower-Order Terms in BIC-Like Criteria
Keep it Simple Stupid On the Effect of Lower-Order Terms in BIC-Like Criteria Teemu Roos and Yuan Zou Helsinki Institute for Information Technology HIIT Department of Computer Science University of Helsinki,
More informationFinding the best mismatched detector for channel coding and hypothesis testing
Finding the best mismatched detector for channel coding and hypothesis testing Sean Meyn Department of Electrical and Computer Engineering University of Illinois and the Coordinated Science Laboratory
More informationAn Analysis of the Difference of Code Lengths Between Two-Step Codes Based on MDL Principle and Bayes Codes
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 47, NO. 3, MARCH 2001 927 An Analysis of the Difference of Code Lengths Between Two-Step Codes Based on MDL Principle Bayes Codes Masayuki Goto, Member, IEEE,
More informationLecture 3: Lower Bounds for Bandit Algorithms
CMSC 858G: Bandits, Experts and Games 09/19/16 Lecture 3: Lower Bounds for Bandit Algorithms Instructor: Alex Slivkins Scribed by: Soham De & Karthik A Sankararaman 1 Lower Bounds In this lecture (and
More informationA NEW INFORMATION THEORETIC APPROACH TO ORDER ESTIMATION PROBLEM. Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A.
A EW IFORMATIO THEORETIC APPROACH TO ORDER ESTIMATIO PROBLEM Soosan Beheshti Munther A. Dahleh Massachusetts Institute of Technology, Cambridge, MA 0239, U.S.A. Abstract: We introduce a new method of model
More informationStar-Structured High-Order Heterogeneous Data Co-clustering based on Consistent Information Theory
Star-Structured High-Order Heterogeneous Data Co-clustering based on Consistent Information Theory Bin Gao Tie-an Liu Wei-ing Ma Microsoft Research Asia 4F Sigma Center No. 49 hichun Road Beijing 00080
More informationNoncommutative Uncertainty Principle
Noncommutative Uncertainty Principle Zhengwei Liu (joint with Chunlan Jiang and Jinsong Wu) Vanderbilt University The 12th East Coast Operator Algebras Symposium, Oct 12, 2014 Z. Liu (Vanderbilt) Noncommutative
More informationINFORMATION PROCESSING ABILITY OF BINARY DETECTORS AND BLOCK DECODERS. Michael A. Lexa and Don H. Johnson
INFORMATION PROCESSING ABILITY OF BINARY DETECTORS AND BLOCK DECODERS Michael A. Lexa and Don H. Johnson Rice University Department of Electrical and Computer Engineering Houston, TX 775-892 amlexa@rice.edu,
More informationLecture 2: Basic Concepts of Statistical Decision Theory
EE378A Statistical Signal Processing Lecture 2-03/31/2016 Lecture 2: Basic Concepts of Statistical Decision Theory Lecturer: Jiantao Jiao, Tsachy Weissman Scribe: John Miller and Aran Nayebi In this lecture
More informationH(X) = plog 1 p +(1 p)log 1 1 p. With a slight abuse of notation, we denote this quantity by H(p) and refer to it as the binary entropy function.
LECTURE 2 Information Measures 2. ENTROPY LetXbeadiscreterandomvariableonanalphabetX drawnaccordingtotheprobability mass function (pmf) p() = P(X = ), X, denoted in short as X p(). The uncertainty about
More informationBioinformatics: Biology X
Bud Mishra Room 1002, 715 Broadway, Courant Institute, NYU, New York, USA Model Building/Checking, Reverse Engineering, Causality Outline 1 Bayesian Interpretation of Probabilities 2 Where (or of what)
More informationInformation-theoretic foundations of differential privacy
Information-theoretic foundations of differential privacy Darakhshan J. Mir Rutgers University, Piscataway NJ 08854, USA, mir@cs.rutgers.edu Abstract. We examine the information-theoretic foundations of
More informationAsymptotic Minimax Regret for Data Compression, Gambling, and Prediction
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 46, NO. 2, MARCH 2000 431 Asymptotic Minimax Regret for Data Compression, Gambling, Prediction Qun Xie Andrew R. Barron, Member, IEEE Abstract For problems
More informationBregman divergence and density integration Noboru Murata and Yu Fujimoto
Journal of Math-for-industry, Vol.1(2009B-3), pp.97 104 Bregman divergence and density integration Noboru Murata and Yu Fujimoto Received on August 29, 2009 / Revised on October 4, 2009 Abstract. In this
More informationSOLVING an optimization problem over two variables in a
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 3, MARCH 2009 1423 Adaptive Alternating Minimization Algorithms Urs Niesen, Student Member, IEEE, Devavrat Shah, and Gregory W. Wornell Abstract The
More informationON MINIMAL PAIRWISE SUFFICIENT STATISTICS
Journal of Applied Analysis Vol. 7, No. 2 (2001), pp. 285 292 ON MINIMAL PAIRWISE SUFFICIENT STATISTICS A. KUŚMIEREK Received September 15, 2000 and, in revised form, January 23, 2001 Abstract. Each statistic,
More informationSeries 7, May 22, 2018 (EM Convergence)
Exercises Introduction to Machine Learning SS 2018 Series 7, May 22, 2018 (EM Convergence) Institute for Machine Learning Dept. of Computer Science, ETH Zürich Prof. Dr. Andreas Krause Web: https://las.inf.ethz.ch/teaching/introml-s18
More informationOn Some New Measures of Intutionstic Fuzzy Entropy and Directed Divergence
Global Journal of Mathematical Sciences: Theory and Practical. ISSN 0974-3200 Volume 3, Number 5 (20), pp. 473-480 International Research Publication House http://www.irphouse.com On Some New Measures
More informationOn the Borel-Cantelli Lemma
On the Borel-Cantelli Lemma Alexei Stepanov, Izmir University of Economics, Turkey In the present note, we propose a new form of the Borel-Cantelli lemma. Keywords and Phrases: the Borel-Cantelli lemma,
More information