Information in Biology
|
|
- Linette Dixon
- 1 years ago
- Views:
Transcription
1 Information in Biology CRI - Centre de Recherches Interdisciplinaires, Paris May 2012 Information processing is an essential part of Life. Thinking about it in quantitative terms may is useful. 1
2 Living information is carried by molecular channels Living systems I. Self-replicating information processors Environment II. III. Evolve collectively. Made of molecules. Generic properties of molecular channels subject to evolution? Information theory approach? Other biological information channels. 2
3 Outline Information in Biology Information in Biology Concept of information is found in many living systems: DNA, signaling, neuron, ribosomes, evolution. Goals: (1) Formalize and quantify biological information. (2) Application to various biological systems. (3) Looking for common principles. I. Information and Statistical Mechanics: Shannon s information theory and its relation to statistical mechanics. II. III. Overview: Living Information: molecules, neurons, population and evolution. Living systems as information sources, channels and processors. Molecular information and noise. IV. Neural networks and coding theory. V. Population dynamics, social interaction and sensing. 3
4 I. Basics of Information Theory (Shannon) 4
5 Shannon s Information theory Information theory: a branch of applied math and electrical engineering. Developed by Claude E. Shannon. Main results: fundamental limits on signal processing such as, How well data can be compressed? What is the reliability of communicating signals? Numerous applications (besides communication eng.): Physics (stat mech), Math (statistical inference), linguistics, Computer science (cryptography, complexity), Economics (portfolio theory). The key quantity which measures information is entropy: Quantifies the uncertainty involved in predicting the value of a random variable (e.g., a coin flip or a die). What are the biological implications? 5
6 Claude Elwood Shannon ( ) 1937 master's thesis: A Symbolic Analysis of Relay and Switching Circuits Ph.D. thesis: An Algebra for Mandelian Genetics. WWII (Bell labs) works on cryptography and fire-control systems: Data Smoothing and Prediction in Fire-Control Systems. Communication theory of secrecy systems. 1948: Mathematical Theory of Communication. 1949: Sampling theory: Analog to digital. 1951: Prediction and Entropy of Printed English. 1950: Shannon s mouse: 1 st artificial learning machine. 1950: Programming a Computer for Playing Chess. 1960: 1 st wearable computer, Las Vegas. 6
7 A Mathematical Theory of Communication Shannon s seminal paper: "A Mathematical Theory of Communication". Bell System Technical Journal 27 (3): (1948). Basic scheme of communication: Information source produces messages. Transmitter converts message to signal. Channel conveys the signal with Noise. Receiver transforms the signal back into the message Destination: machine, person, organism receiving the message. Introduces information entropy measured in bits. 7
8 What is information? "The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point." Engineering perspective How to make good transmission channels Problem with telegraph lines, Define information as measure for the surprise If a binary channel transmits only 1 s there is no information (no surprise). If the channel transmits 0 s and 1 s with equal probability max. information. 8
9 Intuition: 20 questions game Try to guess the object from: {barrel, cat, dog, ball, fish, box, building}. First strategy: wild guess. 9
10 Intuition: 20 questions game Optimal strategy: equalized tree Information = # of yes/no questions in an optimal tree. 10
11 Introducing the bit If I have a (equal) choice between two alternatives the information is: I=1 bit = log 2 (#Alternatives) Harry Nyquist (1924): 1 bit = Certain Factors Affecting Telegraph Speed Example: How many bits are in a genome of length N? 11
12 Shannon s axioms of information For random variable that distributes as p 1 p n,search for a function H(p 1,..,p n ) that measures information that obeys: 1) Continuous in p i 2) For equal probabilities it is Nyquist expression: H(1/n,,1/n)=log 2 (n). 3) Tree property : invariant to redistribution into stages. 12
13 Information from Shannon s axioms Shannon showed that the only function that obeys these axioms is H p log p log p i i 2 i 2 i (up to proportion constant). Example : if X is uniformly distributed over 128 outcomes H p log p log log 2 7 bits i i 2 i i 2 2 Example: very uneven coin shows head only 1 in 1024 times ilog2 i 2 log22 (1 2 )log 2(1 2 ) (10 log 2 )2 bits 13 i H p p e
14 Entropy of a Binary Channel H plog p (1 p)log (1 p)
15 Why call it entropy? Shannon discussed this problem with John von Neumann: My greatest concern was what to call it. I thought of calling it information, but the word was overly used, so I decided to call it uncertainty. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage. M. Tribus, E.C. McIrvine, Energy and information, Scientific American, 224 (1971). 15
16 Conditional probability Consider two random variables X,Y with a joint probability distribution P(X,Y) The joint entropy is H(X,Y) The mutual entropy is H(X,Y)-H(X)-H(Y) The conditional entropies are H(X Y) and H(Y X) 16
17 Joint entropy H(X,Y) measures total entropy of the joint distribution P(X,Y) Mutual entropy I(X;Y) measures correlations ( P(x,y)=P(x)P(y) => I=0 ) Conditional entropy H(X Y) measures remaining uncertainty of X given Y 17
18 More information measures 18
19 Whose entropy? E. T. Jaynes 19
20 Kullback Leibler entropy Problem: Suppose a random variable X is distributed according to P(X) but we expect it to be distributed according to Q(X). What is the level of our surprise? Answer: The Kullback Leibler divergence pi DKL( p q) pi log2 q i i Mutual information I( X, Y ) D ( p( x, y) p( x) p( y)) KL Appears in many circumstances Example Markov chains 20
21 Shannon Entropy and Statistical Mechanics 21
22 I. Maxwell s Demon Entropy in statistical mechanics: measure of uncertainty of a system after specifying its macroscopic observables such as temperature and pressure. Given macroscopic variables, entropy measures the degree of spreading probability over different possible states. Boltzmann famous formula: S ln 22
23 The second law of thermodynamics The second law of thermodynamics: In general, the total entropy of a system isolated from its environment, the entropy of that system will tend not to decrease. Consequences: (i) (ii) heat will not flow from a colder body to a hotter body without work. No perpetuum mobile: One cannot produce net work from a single temperature reservoir (production of net work requires flow of heat from a hotter reservoir to a colder reservoir). 23
24 Maxwell s thought experiment How to violate the Second Law? - Container divided by an insulated wall. - Door can be opened and closed by a demon. - Demon opens the door to allow only "hot" molecules of gas to flow to the favored side. - One side heats up while other side cools down: decreasing entropy. Breaking 2 nd law! 24
25 Solution: entropy = -information Demon reduces the thermodynamic entropy of a system using information about individual molecules (their direction) Landauer (1961) showed that he demon must increase thermodynamic entropy by at least the amount of Shannon information he acquires and stores; total thermodynamic entropy does not decrease! Landauer s principle : Any logically irreversible manipulation of information, such as the erasure of a bit, must be accompanied by a corresponding entropy increase in the information processing apparatus or its environment. min E k T ln 2 B 25
26 II. Second law in Markov chains Random walk on a graph:. W is transition matrix : p(t+1)=wp(t) p* be the steady state solution: Wp*=p* Theorem: distribution approaches steady-state t D(p p*) <=0 Also t D(p p*)=0 <=> p=p* In other words: Markov dynamics dissipates any initial information. 26
27 Maximum entropy inference (E. T. Jaynes) Problem: Given partial knowledge e.g. the average value <X> of X how should we assign probabilities to outcomes P(X)? Answer: choose the probability distribution that maximizes the entropy (surprise) and is consistent with what we already know. Example: given energies E i and measurement <E> what is p i? Exercise: which dice X={1,2,3,4,5,6} gives <X>=3? 27
28 Maximum Entropy principle relates thermodynamics and information At equilibrium, TD entropy = Shannon information needed to define the microscopic state of the system, given its macroscopic description. Gain in entropy always means loss of information about this state. Equivalently, TD entropy = minimum number of yes/no questions needed to be answered in order to fully specify the microstate. 28
Introduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information.
L65 Dept. of Linguistics, Indiana University Fall 205 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission rate
Dept. of Linguistics, Indiana University Fall 2015
L645 Dept. of Linguistics, Indiana University Fall 2015 1 / 28 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission
Introduction to Information Theory
Introduction to Information Theory Gurinder Singh Mickey Atwal atwal@cshl.edu Center for Quantitative Biology Kullback-Leibler Divergence Summary Shannon s coding theorems Entropy Mutual Information Multi-information
Classification & Information Theory Lecture #8
Classification & Information Theory Lecture #8 Introduction to Natural Language Processing CMPSCI 585, Fall 2007 University of Massachusetts Amherst Andrew McCallum Today s Main Points Automatically categorizing
Entropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
Introduction to Information Theory. B. Škorić, Physical Aspects of Digital Security, Chapter 2
Introduction to Information Theory B. Škorić, Physical Aspects of Digital Security, Chapter 2 1 Information theory What is it? - formal way of counting information bits Why do we need it? - often used
Machine Learning Srihari. Information Theory. Sargur N. Srihari
Information Theory Sargur N. Srihari 1 Topics 1. Entropy as an Information Measure 1. Discrete variable definition Relationship to Code Length 2. Continuous Variable Differential Entropy 2. Maximum Entropy
CS 630 Basic Probability and Information Theory. Tim Campbell
CS 630 Basic Probability and Information Theory Tim Campbell 21 January 2003 Probability Theory Probability Theory is the study of how best to predict outcomes of events. An experiment (or trial or event)
Chapter I: Fundamental Information Theory
ECE-S622/T62 Notes Chapter I: Fundamental Information Theory Ruifeng Zhang Dept. of Electrical & Computer Eng. Drexel University. Information Source Information is the outcome of some physical processes.
CHAPTER 4 ENTROPY AND INFORMATION
4-1 CHAPTER 4 ENTROPY AND INFORMATION In the literature one finds the terms information theory and communication theory used interchangeably. As there seems to be no wellestablished convention for their
Thermodynamics: More Entropy
Thermodynamics: More Entropy From Warmup On a kind of spiritual note, this could possibly explain how God works some miracles. Supposing He could precisely determine which microstate occurs, He could heat,
Information Theory, Statistics, and Decision Trees
Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010
MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK UNIT V PART-A. 1. What is binary symmetric channel (AUC DEC 2006)
MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK SATELLITE COMMUNICATION DEPT./SEM.:ECE/VIII UNIT V PART-A 1. What is binary symmetric channel (AUC DEC 2006) 2. Define information rate? (AUC DEC 2007)
Shannon's Theory of Communication
Shannon's Theory of Communication An operational introduction 5 September 2014, Introduction to Information Systems Giovanni Sileno g.sileno@uva.nl Leibniz Center for Law University of Amsterdam Fundamental
An Elementary Notion and Measurement of Entropy
(Source: Wikipedia: entropy; Information Entropy) An Elementary Notion and Measurement of Entropy Entropy is a macroscopic property of a system that is a measure of the microscopic disorder within the
6.02 Fall 2012 Lecture #1
6.02 Fall 2012 Lecture #1 Digital vs. analog communication The birth of modern digital communication Information and entropy Codes, Huffman coding 6.02 Fall 2012 Lecture 1, Slide #1 6.02 Fall 2012 Lecture
Information Theory - Entropy. Figure 3
Concept of Information Information Theory - Entropy Figure 3 A typical binary coded digital communication system is shown in Figure 3. What is involved in the transmission of information? - The system
(Classical) Information Theory II: Source coding
(Classical) Information Theory II: Source coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract The information content of a random variable
1 Introduction to information theory
1 Introduction to information theory 1.1 Introduction In this chapter we present some of the basic concepts of information theory. The situations we have in mind involve the exchange of information through
3F1: Signals and Systems INFORMATION THEORY Examples Paper Solutions
Engineering Tripos Part IIA THIRD YEAR 3F: Signals and Systems INFORMATION THEORY Examples Paper Solutions. Let the joint probability mass function of two binary random variables X and Y be given in the
Chapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
Information Theory. David Rosenberg. June 15, New York University. David Rosenberg (New York University) DS-GA 1003 June 15, / 18
Information Theory David Rosenberg New York University June 15, 2015 David Rosenberg (New York University) DS-GA 1003 June 15, 2015 1 / 18 A Measure of Information? Consider a discrete random variable
Lecture 18: Quantum Information Theory and Holevo s Bound
Quantum Computation (CMU 1-59BB, Fall 2015) Lecture 1: Quantum Information Theory and Holevo s Bound November 10, 2015 Lecturer: John Wright Scribe: Nicolas Resch 1 Question In today s lecture, we will
Chapter 1. Introduction
Chapter 1 Introduction Symbolical artificial intelligence is a field of computer science that is highly related to quantum computation. At first glance, this statement appears to be a contradiction. However,
Entropy. Physics 1425 Lecture 36. Michael Fowler, UVa
Entropy Physics 1425 Lecture 36 Michael Fowler, UVa First and Second Laws of A quick review. Thermodynamics First Law: total energy conserved in any process: joules in = joules out Second Law: heat only
Example: Letter Frequencies
Example: Letter Frequencies i a i p i 1 a 0.0575 2 b 0.0128 3 c 0.0263 4 d 0.0285 5 e 0.0913 6 f 0.0173 7 g 0.0133 8 h 0.0313 9 i 0.0599 10 j 0.0006 11 k 0.0084 12 l 0.0335 13 m 0.0235 14 n 0.0596 15 o
An introduction to basic information theory. Hampus Wessman
An introduction to basic information theory Hampus Wessman Abstract We give a short and simple introduction to basic information theory, by stripping away all the non-essentials. Theoretical bounds on
A Holevo-type bound for a Hilbert Schmidt distance measure
Journal of Quantum Information Science, 205, *,** Published Online **** 204 in SciRes. http://www.scirp.org/journal/**** http://dx.doi.org/0.4236/****.204.***** A Holevo-type bound for a Hilbert Schmidt
The binary entropy function
ECE 7680 Lecture 2 Definitions and Basic Facts Objective: To learn a bunch of definitions about entropy and information measures that will be useful through the quarter, and to present some simple but
1. Basics of Information
1. Basics of Information 6.004x Computation Structures Part 1 Digital Circuits Copyright 2015 MIT EECS 6.004 Computation Structures L1: Basics of Information, Slide #1 What is Information? Information,
Information Theory. M1 Informatique (parcours recherche et innovation) Aline Roumy. January INRIA Rennes 1/ 73
1/ 73 Information Theory M1 Informatique (parcours recherche et innovation) Aline Roumy INRIA Rennes January 2018 Outline 2/ 73 1 Non mathematical introduction 2 Mathematical introduction: definitions
Lecture 25 Goals: Chapter 18 Understand the molecular basis for pressure and the idealgas
Lecture 5 Goals: Chapter 18 Understand the molecular basis for pressure and the idealgas law. redict the molar specific heats of gases and solids. Understand how heat is transferred via molecular collisions
Basic information theory
Basic information theory Communication system performance is limited by Available signal power Background noise Bandwidth limits. Can we postulate an ideal system based on physical principles, against
Random Bit Generation
.. Random Bit Generation Theory and Practice Joshua E. Hill Department of Mathematics, University of California, Irvine Math 235B January 11, 2013 http://bit.ly/xwdbtv v. 1 / 47 Talk Outline 1 Introduction
Entropy, Relative Entropy and Mutual Information Exercises
Entrop, Relative Entrop and Mutual Information Exercises Exercise 2.: Coin Flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips required. (a) Find the entrop H(X)
04. Information and Maxwell's Demon. I. Dilemma for Information-Theoretic Exorcisms.
04. Information and Maxwell's Demon. I. Dilemma for Information-Theoretic Exorcisms. Two Options: (S) (Sound). The combination of object system and demon forms a canonical thermal system. (P) (Profound).
Chapter 20 The Second Law of Thermodynamics
Chapter 20 The Second Law of Thermodynamics When we previously studied the first law of thermodynamics, we observed how conservation of energy provided us with a relationship between U, Q, and W, namely
Chapter 20. Heat Engines, Entropy and the Second Law of Thermodynamics. Dr. Armen Kocharian
Chapter 20 Heat Engines, Entropy and the Second Law of Thermodynamics Dr. Armen Kocharian First Law of Thermodynamics Review Review: The first law states that a change in internal energy in a system can
Conceptual Physics Phase Changes Thermodynamics
Conceptual Physics Phase Changes Thermodynamics Lana Sheridan De Anza College July 25, 2016 Last time heat capacity thermal expansion heat transfer mechanisms Newton s law of cooling the greenhouse effect
An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
Block 2: Introduction to Information Theory
Block 2: Introduction to Information Theory Francisco J. Escribano April 26, 2015 Francisco J. Escribano Block 2: Introduction to Information Theory April 26, 2015 1 / 51 Table of contents 1 Motivation
Chapter 12. The Laws of Thermodynamics
Chapter 12 The Laws of Thermodynamics First Law of Thermodynamics The First Law of Thermodynamics tells us that the internal energy of a system can be increased by Adding energy to the system Doing work
Active Learners Reflective Learners Use both ways equally frequently More Inclined to be Active More inclined to be Reflective
Active Learners Reflective Learners Use both ways equally frequently More Inclined to be Active More inclined to be Reflective Sensory Leaners Intuitive Learners Use both ways equally frequently More inclined
3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output
CS546:Learning and NLP Lec 4: Mathematical and Computational Paradigms
CS546:Learning and NLP Lec 4: Mathematical and Computational Paradigms Spring 2009 February 3, 2009 Lecture Some note on statistics Bayesian decision theory Concentration bounds Information theory Introduction
Chapter 11 Heat Engines and The Second Law of Thermodynamics
Chapter 11 Heat Engines and The Second Law of Thermodynamics Heat Engines Heat engines use a temperature difference involving a high temperature (T H ) and a low temperature (T C ) to do mechanical work.
Information Theory CHAPTER. 5.1 Introduction. 5.2 Entropy
Haykin_ch05_pp3.fm Page 207 Monday, November 26, 202 2:44 PM CHAPTER 5 Information Theory 5. Introduction As mentioned in Chapter and reiterated along the way, the purpose of a communication system is
SIDDHARTH GROUP OF INSTITUTIONS :: PUTTUR Siddharth Nagar, Narayanavanam Road UNIT I
SIDDHARTH GROUP OF INSTITUTIONS :: PUTTUR Siddharth Nagar, Narayanavanam Road 517583 QUESTION BANK (DESCRIPTIVE) Subject with Code : CODING THEORY & TECHNIQUES(16EC3810) Course & Branch: M.Tech - DECS
Example: Letter Frequencies
Example: Letter Frequencies i a i p i 1 a 0.0575 2 b 0.0128 3 c 0.0263 4 d 0.0285 5 e 0.0913 6 f 0.0173 7 g 0.0133 8 h 0.0313 9 i 0.0599 10 j 0.0006 11 k 0.0084 12 l 0.0335 13 m 0.0235 14 n 0.0596 15 o
Information Theory and Communication
Information Theory and Communication Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/8 General Chain Rules Definition Conditional mutual information
Chapter 12. The Laws of Thermodynamics. First Law of Thermodynamics
Chapter 12 The Laws of Thermodynamics First Law of Thermodynamics The First Law of Thermodynamics tells us that the internal energy of a system can be increased by Adding energy to the system Doing work
T s change via collisions at boundary (not mechanical interaction)
Lecture 14 Interaction of 2 systems at different temperatures Irreversible processes: 2nd Law of Thermodynamics Chapter 19: Heat Engines and Refrigerators Thermal interactions T s change via collisions
Discovering Correlation in Data. Vinh Nguyen Research Fellow in Data Science Computing and Information Systems DMD 7.
Discovering Correlation in Data Vinh Nguyen (vinh.nguyen@unimelb.edu.au) Research Fellow in Data Science Computing and Information Systems DMD 7.14 Discovering Correlation Why is correlation important?
Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006
Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006 Fabio Grazioso... July 3, 2006 1 2 Contents 1 Lecture 1, Entropy 4 1.1 Random variable...............................
Chapter 16 Thermodynamics
Nicholas J. Giordano www.cengage.com/physics/giordano Chapter 16 Thermodynamics Thermodynamics Introduction Another area of physics is thermodynamics Continues with the principle of conservation of energy
Distance between physical theories based on information theory
Distance between physical theories based on information theory Jacques Calmet 1 and Xavier Calmet 2 Institute for Cryptography and Security (IKS) Karlsruhe Institute of Technology (KIT), 76131 Karlsruhe,
First Law Limitations
First Law Limitations First Law: During any process, the energy of the universe is constant. du du du ZERO!!! universe sys surroundings Any energy transfer between system and surroundings is accomplished
Entropy & Information
Entropy & Information Pre- Lab: Maxwell and His Demon A Bit of History We begin this lab with the story of a demon. Imagine a box filled with two types of gas molecules. We ll call them blue and green.
PERFECT SECRECY AND ADVERSARIAL INDISTINGUISHABILITY
PERFECT SECRECY AND ADVERSARIAL INDISTINGUISHABILITY BURTON ROSENBERG UNIVERSITY OF MIAMI Contents 1. Perfect Secrecy 1 1.1. A Perfectly Secret Cipher 2 1.2. Odds Ratio and Bias 3 1.3. Conditions for Perfect
2.2 Classical circuit model of computation
Chapter 2 Classical Circuit Models for Computation 2. Introduction A computation is ultimately performed by a physical device. Thus it is a natural question to ask what are the fundamental limitations
Lecture 2. Capacity of the Gaussian channel
Spring, 207 5237S, Wireless Communications II 2. Lecture 2 Capacity of the Gaussian channel Review on basic concepts in inf. theory ( Cover&Thomas: Elements of Inf. Theory, Tse&Viswanath: Appendix B) AWGN
Introduction to Information Theory and Its Applications
Introduction to Information Theory and Its Applications Radim Bělohlávek Information Theory: What and Why information: one of key terms in our society: popular keywords such as information/knowledge society
Lecture 3: Channel Capacity
Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.
Machine Learning Recitation 8 Oct 21, Oznur Tastan
Machine Learning 10601 Recitation 8 Oct 21, 2009 Oznur Tastan Outline Tree representation Brief information theory Learning decision trees Bagging Random forests Decision trees Non linear classifier Easy
The Communication Complexity of Correlation. Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan
The Communication Complexity of Correlation Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan Transmitting Correlated Variables (X, Y) pair of correlated random variables Transmitting
X 1 : X Table 1: Y = X X 2
ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access
Decision Tree Learning - ID3
Decision Tree Learning - ID3 n Decision tree examples n ID3 algorithm n Occam Razor n Top-Down Induction in Decision Trees n Information Theory n gain from property 1 Training Examples Day Outlook Temp.
Exercise 1. = P(y a 1)P(a 1 )
Chapter 7 Channel Capacity Exercise 1 A source produces independent, equally probable symbols from an alphabet {a 1, a 2 } at a rate of one symbol every 3 seconds. These symbols are transmitted over a
Introduction to Information Theory. Part 4
Introduction to Information Theory Part 4 A General Communication System CHANNEL Information Source Transmitter Channel Receiver Destination 10/2/2012 2 Information Channel Input X Channel Output Y 10/2/2012
Information Theory. Coding and Information Theory. Information Theory Textbooks. Entropy
Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is
Basic Thermodynamics. Prof. S. K. Som. Department of Mechanical Engineering. Indian Institute of Technology, Kharagpur.
Basic Thermodynamics Prof. S. K. Som Department of Mechanical Engineering Indian Institute of Technology, Kharagpur Lecture - 06 Second Law and its Corollaries I Good afternoon, I welcome you all to this
Application of Information Theory, Lecture 7. Relative Entropy. Handout Mode. Iftach Haitner. Tel Aviv University.
Application of Information Theory, Lecture 7 Relative Entropy Handout Mode Iftach Haitner Tel Aviv University. December 1, 2015 Iftach Haitner (TAU) Application of Information Theory, Lecture 7 December
Simulation of the Evolution of Information Content in Transcription Factor Binding Sites Using a Parallelized Genetic Algorithm
Simulation of the Evolution of Information Content in Transcription Factor Binding Sites Using a Parallelized Genetic Algorithm Joseph Cornish*, Robert Forder**, Ivan Erill*, Matthias K. Gobbert** *Department
12 The Laws of Thermodynamics
June 14, 1998 12 The Laws of Thermodynamics Using Thermal Energy to do Work Understanding the laws of thermodynamics allows us to use thermal energy in a practical way. The first law of thermodynamics
ECE598: Information-theoretic methods in high-dimensional statistics Spring 2016
ECE598: Information-theoretic methods in high-dimensional statistics Spring 06 Lecture : Mutual Information Method Lecturer: Yihong Wu Scribe: Jaeho Lee, Mar, 06 Ed. Mar 9 Quick review: Assouad s lemma
Entropy in Classical and Quantum Information Theory
Entropy in Classical and Quantum Information Theory William Fedus Physics Department, University of California, San Diego. Entropy is a central concept in both classical and quantum information theory,
Quantum Information Types
qitd181 Quantum Information Types Robert B. Griffiths Version of 6 February 2012 References: R. B. Griffiths, Types of Quantum Information, Phys. Rev. A 76 (2007) 062320; arxiv:0707.3752 Contents 1 Introduction
Multimedia Communications. Mathematical Preliminaries for Lossless Compression
Multimedia Communications Mathematical Preliminaries for Lossless Compression What we will see in this chapter Definition of information and entropy Modeling a data source Definition of coding and when
Part I. C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS
Part I C. M. Bishop PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 8: GRAPHICAL MODELS Probabilistic Graphical Models Graphical representation of a probabilistic model Each variable corresponds to a
8 Lecture 8: Thermodynamics: Principles
8. LECTURE 8: THERMODYNMICS: PRINCIPLES 69 8 Lecture 8: Thermodynamics: Principles Summary Phenomenological approach is a respectable way of understanding the world, especially when we cannot expect microscopic
Feature Selection based on the Local Lift Dependence Scale
Feature Selection based on the Local Lift Dependence Scale Diego Marcondes Adilson Simonis Junior Barrera arxiv:1711.04181v2 [stat.co] 15 Nov 2017 Abstract This paper uses a classical approach to feature
Information Theory. Week 4 Compressing streams. Iain Murray,
Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 4 Compressing streams Iain Murray, 2014 School of Informatics, University of Edinburgh Jensen s inequality For convex functions: E[f(x)]
18.13 Review & Summary
5/2/10 10:04 PM Print this page 18.13 Review & Summary Temperature; Thermometers Temperature is an SI base quantity related to our sense of hot and cold. It is measured with a thermometer, which contains
An Extended Fano s Inequality for the Finite Blocklength Coding
An Extended Fano s Inequality for the Finite Bloclength Coding Yunquan Dong, Pingyi Fan {dongyq8@mails,fpy@mail}.tsinghua.edu.cn Department of Electronic Engineering, Tsinghua University, Beijing, P.R.
Homework Set #2 Data Compression, Huffman code and AEP
Homework Set #2 Data Compression, Huffman code and AEP 1. Huffman coding. Consider the random variable ( x1 x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0.11 0.04 0.04 0.03 0.02 (a Find a binary Huffman code
One Lesson of Information Theory
Institut für One Lesson of Information Theory Prof. Dr.-Ing. Volker Kühn Institute of Communications Engineering University of Rostock, Germany Email: volker.kuehn@uni-rostock.de http://www.int.uni-rostock.de/
CS225/ECE205A, Spring 2005: Information Theory Wim van Dam Engineering 1, Room 5109
CS225/ECE205A, Spring 2005: Information Theory Wim van Dam Engineering 1, Room 5109 vandam@cs http://www.cs.ucsb.edu/~vandam/teaching/s06_cs225/ Formalities There was a mistake in earlier announcement.
Lecture 6. Statistical Processes. Irreversibility. Counting and Probability. Microstates and Macrostates. The Meaning of Equilibrium Ω(m) 9 spins
Lecture 6 Statistical Processes Irreversibility Counting and Probability Microstates and Macrostates The Meaning of Equilibrium Ω(m) 9 spins -9-7 -5-3 -1 1 3 5 7 m 9 Lecture 6, p. 1 Irreversibility Have
Approximating Discrete Probability Distributions with Causal Dependence Trees
Approximating Discrete Probability Distributions with Causal Dependence Trees Christopher J. Quinn Department of Electrical and Computer Engineering University of Illinois Urbana, Illinois 680 Email: quinn7@illinois.edu
Axiomatic Information Thermodynamics. Received: 1 February 2018; Accepted: 26 March 2018; Published: 29 March 2018
entropy Article Axiomatic Information Thermodynamics Austin Hulse 1, Benjamin Schumacher 1, * and Michael D. Westmoreland 2 1 Department of Physics, Kenyon College, Gambier, OH 43022, USA; hulsea@kenyon.edu
B L U E V A L L E Y D I S T R I C T C U R R I C U L U M Science Physics
B L U E V A L L E Y D I S T R I C T C U R R I C U L U M Science Physics ORGANIZING THEME/TOPIC UNIT 1: KINEMATICS Mathematical Concepts Graphs and the interpretation thereof Algebraic manipulation of equations
Conditional Probability
Conditional Probability When we obtain additional information about a probability experiment, we want to use the additional information to reassess the probabilities of events given the new information.
Universality for mathematical and physical systems
Universality for mathematical and physical systems Percy Deift, Courant Institute 1 All physical systems in equilibrium obey the laws of thermodynamics. The first law asserts the conservation of energy.
Demon Dynamics: Deterministic Chaos, the Szilard Map, & the Intelligence of Thermodynamic Systems.
Demon Dynamics: Deterministic Chaos, the Szilard Map, & the Intelligence of Thermodynamic Systems http://csc.ucdavis.edu/~cmg/ Jim Crutchfield Complexity Sciences Center Physics Department University of
Classification: Decision Trees
Classification: Decision Trees Outline Top-Down Decision Tree Construction Choosing the Splitting Attribute Information Gain and Gain Ratio 2 DECISION TREE An internal node is a test on an attribute. A
MCMC notes by Mark Holder
MCMC notes by Mark Holder Bayesian inference Ultimately, we want to make probability statements about true values of parameters, given our data. For example P(α 0 < α 1 X). According to Bayes theorem:
An Introduction to Independent Components Analysis (ICA)
An Introduction to Independent Components Analysis (ICA) Anish R. Shah, CFA Northfield Information Services Anish@northinfo.com Newport Jun 6, 2008 1 Overview of Talk Review principal components Introduce
Password Cracking: The Effect of Bias on the Average Guesswork of Hash Functions
Password Cracking: The Effect of Bias on the Average Guesswork of Hash Functions Yair Yona, and Suhas Diggavi, Fellow, IEEE Abstract arxiv:608.0232v4 [cs.cr] Jan 207 In this work we analyze the average
Lecture 6: Entropy Rate
Lecture 6: Entropy Rate Entropy rate H(X) Random walk on graph Dr. Yao Xie, ECE587, Information Theory, Duke University Coin tossing versus poker Toss a fair coin and see and sequence Head, Tail, Tail,
(each row defines a probability distribution). Given n-strings x X n, y Y n we can use the absence of memory in the channel to compute
ENEE 739C: Advanced Topics in Signal Processing: Coding Theory Instructor: Alexander Barg Lecture 6 (draft; 9/6/03. Error exponents for Discrete Memoryless Channels http://www.enee.umd.edu/ abarg/enee739c/course.html