Anatomy of a Bit. Reading for this lecture: CMR articles Yeung and Anatomy.
|
|
- Cori Wilson
- 6 years ago
- Views:
Transcription
1 Anatomy of a Bit Reading for this lecture: CMR articles Yeung and Anatomy. 1
2 Information in a single measurement: Alternative rationale for entropy and entropy rate. k measurement outcomes Stored in log 10 k decimal digits or log 2 k bits. Series of n measurements stored in n log 2 k bits. Number M of possible sequences of n measurements with n 1,n 2,...,n k counts: n M = n 1 n 2 n k n! = n 1!n 2! n k! 2
3 To specify/code which particular sequence occurred store: k log 2 n + log 2 M + log 2 n + Maximum number of bits to store the count ni of each of the k output values Number of bits to specify particular observed sequence within the class of sequences that have counts n1, n2,..., nk Bits to specify the number of bits in n itself bits Apply Stirlings approximation: n! = p 2 n(n/e) n kx To M: log 2 M n n i n log 2 n i n i=1 = nh[ n 1 n, n 2 n,..., n k n ] = nh[x] 3
4 Can compress if H[X] < log 2 k R 1 1-Symbol Redundancy R 1 = log 2 k R 1 H[X] log 2 k H[X] h µ (a) (b) (c) 4
5 Typical sources are not IID, therefore use: h µ = lim L!1 H(L) L R 1 R 1 Intrinsic Redundancy log 2 k R 1 = log 2 k h µ (Total Predictability) H[X] h µ Intrinsic Randomness (a) (b) (c) 5
6 In addition, measurements occur in the context of past and future. A semantically rich decomposition of a single measurement: µ q µ w µ H[X] h µ r µ r µ (a) (b) (c) (d) 6
7 A semantically rich decomposition of a single measurement... µ Anticipated from prior observations q µ w µ H[X] h µ Random component r µ r µ (a) (b) (c) (d) 7
8 A semantically rich decomposition of a single measurement... µ q µ Shared between the past and the current observation, but relevance stops there (stationary process only) Anticipated by the past, present currently, and plays a w µ role in future behavior. Can be negative! H[X] Relevant for predicting the future h µ r µ r µ Ephemeral: Exists only for the present (a) (b) (c) (d) 8
9 A semantically rich decomposition of a single measurement... µ q µ w µ Structural information H[X] h µ r µ r µ Ephemeral: exists only for the present (a) (b) (c) (d) 9
10 Settings: Before: Predict the present from the past, h µ = H[X 0 X ] X :0 X 0 X 1: X 3 X 2 X 1 X 0 X 1 X 2 X 3 10
11 Settings: Before: Predict the present from the past, h µ = H[X 0 X ] X :0 X 0 X 1: X 3 X 2 X 1 X 0 X 1 X 2 X 3 Today, prediction from the omniscient view: The present in the context of the past and the future. X :0 X 0 X 1: X 3 X 2 X 1 X 0 X 1 X 2 X 3 10
12 Notations: l-blocks: Past: Present: Future: X t:t+` = X t X t+1...x t+` 1 X :0 =...X 2 X 1 X 0 X 1: = X 1 X 2... Any information measure F : F(`) =F[X 0:`] 11
13 Recall block entropy: H(`) E + h µ` where: E = I[X :0 ; X 0: ] Recall total correlation: X T [X 0:N ]= H[X A ] H[X 0:N ] A2P (N) A =1 For stationary time series, get block total correlation: T (`) =`H[X 0 ] H(`) with T (0) = 0 T (1) = 0 Compares process s block entropy to the case of IID RVs. 12
14 Block entropy versus total correlation... Monotone increasing, with linear asymptote: T (`) E + µ` Convex up. H(`) T (`) Information [bits] E 0 E Block length ` [symbols] 13
15 Block entropy versus total correlation... Note: H(`)+T (`) =`H(1) Plug in asymptotes gives first decomposition: H(1) = h µ + µ µ Anticipated from prior observations q µ w µ H[X] h µ Random component r µ r µ (a) (b) (c) (d) 14
16 Scaling of other block informations: Bound information Enigmatic B(`) =H(`) Q(`) =T (`) R(`) B(`) Local Exogenous W (`) =B(`)+T (`) Asymptotes: R(`) E R + `r µ B(`) E B + ` R(`) B(`) Q(`) W (`) Q(`) E Q + `q µ W (`) E W + `w µ E R E B E Q, E W Block length ` [symbols] 15
17 Scaling of other block informations: Block entropy: H(`) =B(`)+R(`) =(E B + E R )+`( + r µ ) Thus, h µ = + r µ and E = E B + E R. Block total correlation: T (`) =B(`)+Q(`) =(E B + E Q )+`( + q µ ) Thus, µ = + q µ and E = E B E Q. µ Shared between the past and the current observation, but relevance stops there Gives second decomposition of H[X]. H[X] q µ Anticipated w µ by the past, present currently, and plays a role in future behavior. Can be negative! Relevant for predicting the future h µ r µ Ephemeral: r µ Exists only for the present Lecture 20: Natural Computation & Self-Organization, Physics 256A (a) (Winter 2014); (b) Jim Crutchfield (c) (d) 16
18 Scaling of other block informations: Block local exogenous: W (`) =B(`)+T (`) Thus, w µ = + µ. From definitions: =(E B E)+`( + µ ) R(`)+W (`) =`H(1) Plug in asymptotes gives third decomposition of H[X] : H(1) = w µ + r µ µ q µ w µ Structural information H[X] h µ r µ r µ Ephemeral: exists only for the present (a) (b) (c) (d) 17
19 Block multivariate mutual information: X I(`) =H(`) I[X A X Ā ] A2P (`) 0< A <` Asymptote: I(`) I + `i µ? Observations of finite-state process tend to independence: h µ > 0 : lim I(`) =0 i µ =0 I =0 `!1 I(`) Information [bits] I Block length ` [symbols] 18
20 Measurement Decomposition in the Process I-diagram: H[X 0 ] r µ H[X :0 ] H[X 1: ] q µ µ 19
21 Predictive decomposition: H[X 0 ] h µ µ 20
22 Atomic decomposition: H[X 0 ] r µ q µ 21
23 Structural decomposition: H[X 0 ] r µ w µ 22
24 Measurement Decomposition in the Process I-diagram... H[X 0 ] r µ H[X :0 ] H[X 1: ] q µ µ What is? µ Not a component of the information in a single observation. Information shared between past and future, not in present! Hidden state information! 23
25 Measurement Decomposition in the Process I-diagram: H[X 0 ] r µ H[X :0 ] H[X 1: ] q µ µ Decomposition of Excess Entropy: E = + q µ + µ 24
26 Example Processes: A B Even A B Golden Mean 1 0 B C A 1 1 D 1 0 E Noisy Random Phase Slip 25
27 Example: Parametrized Golden Mean Process: p Entropy rate [bits/symbol] h µ r µ 1 1 A B p Self loop probability p ~ Period 2: Randomness affects phase and so is remembered. r µ 0 ~ Period1: Randomness disrupts all 1s, but is not remembered. 0 h µ h µ r µ 26
28 Entropy-rate Decomposition: Logistic Map 27
29 Entropy-rate decomposition: Tent Map h µ = log 2 a 28
30 What is information? Depends on the question! Uncertainty, surprise, randomness,... Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound
31 What is information? Depends on the question! Uncertainty, surprise, randomness,... H(X) Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... h µ 29
32 What is information? Depends on the question! Uncertainty, surprise, randomness,... Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... H(X) h µ R = log 2 A h µ 29
33 What is information? Depends on the question! Uncertainty, surprise, randomness,... H(X) h µ Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... R = log 2 A I[X; Y ] h µ 29
34 What is information? Depends on the question! Uncertainty, surprise, randomness,... Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... H(X) h µ R = log 2 A h µ I[X; Y ] E 29
35 What is information? Depends on the question! Uncertainty, surprise, randomness,... Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... H(X) h µ R = log 2 A h µ I[X; Y ] E T 29
36 What is information? Depends on the question! Uncertainty, surprise, randomness,... Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... H(X) h µ R = log 2 A h µ I[X; Y ] E T rµ 29
37 What is information? Depends on the question! Uncertainty, surprise, randomness,... Compressibility. Transmission rate. Memory, apparent stored information,... Synchronization. Ephemeral. Bound.... H(X) h µ R = log 2 A h µ I[X; Y ] E T rµ 29
38 Analysis Pipeline: 1 A 1 B 0 0 C α γ 0 β δ 0 System Instrument Process Modeller 30
39 Analysis Pipeline: 1 A 1 B 0 0 C α γ 0 β δ 0 System Instrument Process Modeller 1. An information source: a. Dynamical System: deterministic or stochastic low-dimensional or high-dimensional (spatial?) b. Design instrument (partition) 2. Information-theoretic analysis: a. How much information produced? b. How much stored information? Calculate or estimate c. How does observer synchronize? H(L) h µ E T Pr(s L ) 30
40 Course summary: Dynamical systems as sources of complexity: 1. Chaotic attractors (State) 2. Basins of attraction (Initial conditions) 3. Bifurcation sequences (Parameters) Dynamical systems as information processors: 1. Randomness: Entropy hierarchy: block entropy, entropy convergence, rate, Information storage: 1. Total predictability 2. Excess entropy 3. Transient information 31
41 Preview of 256B: Physics of Computation Answers a number of questions 256A posed: What is a model? What are the hidden states and equations of motion? Are these always subjective, depending on the observer? Or is there a principled way to model processes? To discover states? What are cause and effect? To what can we apply these ideas? How much can we calculate analytically? How much numerically? How much from data? 32
42 Preview of 256B: Physics of Computation Punch line: How nature is organized is How nature computes 33
43 Reading for next lecture: CMR articles RURO (Introduction), CAO, and ROIC.... next quarter! 34
Example Chaotic Maps (that you can analyze)
Example Chaotic Maps (that you can analyze) Reading for this lecture: NDAC, Sections.5-.7. Lecture 7: Natural Computation & Self-Organization, Physics 256A (Winter 24); Jim Crutchfield Monday, January
More informationAnatomy of a Bit: Information in a Time Series Measurement
Anatomy of a Bit: Information in a Time Series Measurement Ryan G. James Christopher J. Ellison James P. Crutchfield SFI WORKING PAPER: 2011-05-019 SFI Working Papers contain accounts of scientific work
More informationCommunications Theory and Engineering
Communications Theory and Engineering Master's Degree in Electronic Engineering Sapienza University of Rome A.A. 2018-2019 AEP Asymptotic Equipartition Property AEP In information theory, the analog of
More informationReconstruction Deconstruction:
Reconstruction Deconstruction: A Brief History of Building Models of Nonlinear Dynamical Systems Jim Crutchfield Center for Computational Science & Engineering Physics Department University of California,
More informationITCT Lecture IV.3: Markov Processes and Sources with Memory
ITCT Lecture IV.3: Markov Processes and Sources with Memory 4. Markov Processes Thus far, we have been occupied with memoryless sources and channels. We must now turn our attention to sources with memory.
More informationInformation Theory. Lecture 5 Entropy rate and Markov sources STEFAN HÖST
Information Theory Lecture 5 Entropy rate and Markov sources STEFAN HÖST Universal Source Coding Huffman coding is optimal, what is the problem? In the previous coding schemes (Huffman and Shannon-Fano)it
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More informationInformation Theoretic Properties of Quantum Automata
Information Theoretic Properties of Quantum Automata D. Gier Department of Physics and Astronomy, The University of Kansas, Lawrence, KS 66045 and Complexity Sciences Center and Physics Department, University
More informationLecture 6: Entropy Rate
Lecture 6: Entropy Rate Entropy rate H(X) Random walk on graph Dr. Yao Xie, ECE587, Information Theory, Duke University Coin tossing versus poker Toss a fair coin and see and sequence Head, Tail, Tail,
More informationHierarchical Thermodynamics
Hierarchical Thermodynamics James P Crutchfield Complexity Sciences Center Physics Department University of California, Davis http://csc.ucdavis.edu/~chaos/ Workshop on Information Engines at the Frontier
More informationLecture 22: Final Review
Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information
More informationComplex Systems Methods 3. Statistical complexity of temporal sequences
Complex Systems Methods 3. Statistical complexity of temporal sequences Eckehard Olbrich MPI MiS Leipzig Potsdam WS 2007/08 Olbrich (Leipzig) 02.11.2007 1 / 24 Overview 1 Summary Kolmogorov sufficient
More informationEE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018
Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code
More informationStatistical Complexity of Simple 1D Spin Systems
Statistical Complexity of Simple 1D Spin Systems James P. Crutchfield Physics Department, University of California, Berkeley, CA 94720-7300 and Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501
More informationComputing and Communications 2. Information Theory -Entropy
1896 1920 1987 2006 Computing and Communications 2. Information Theory -Entropy Ying Cui Department of Electronic Engineering Shanghai Jiao Tong University, China 2017, Autumn 1 Outline Entropy Joint entropy
More informationInformational and Causal Architecture of Discrete-Time Renewal Processes
Entropy 2015, 17, 4891-4917; doi:10.3390/e17074891 OPEN ACCESS entropy ISSN 1099-4300 www.mdpi.com/journal/entropy Article Informational and Causal Architecture of Discrete-Time Renewal Processes Sarah
More informationInformation Theory. Coding and Information Theory. Information Theory Textbooks. Entropy
Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is
More informationEECS 229A Spring 2007 * * (a) By stationarity and the chain rule for entropy, we have
EECS 229A Spring 2007 * * Solutions to Homework 3 1. Problem 4.11 on pg. 93 of the text. Stationary processes (a) By stationarity and the chain rule for entropy, we have H(X 0 ) + H(X n X 0 ) = H(X 0,
More informationLecture 5: Asymptotic Equipartition Property
Lecture 5: Asymptotic Equipartition Property Law of large number for product of random variables AEP and consequences Dr. Yao Xie, ECE587, Information Theory, Duke University Stock market Initial investment
More information10-704: Information Processing and Learning Spring Lecture 8: Feb 5
10-704: Information Processing and Learning Spring 2015 Lecture 8: Feb 5 Lecturer: Aarti Singh Scribe: Siheng Chen Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal
More informationPredictive information in Gaussian processes with application to music analysis
Predictive information in Gaussian processes with application to music analysis Samer Abdallah 1 and Mark Plumbley 2 1 University College London 2 Queen Mary University of London Abstract. We describe
More informationMARKOV CHAINS A finite state Markov chain is a sequence of discrete cv s from a finite alphabet where is a pmf on and for
MARKOV CHAINS A finite state Markov chain is a sequence S 0,S 1,... of discrete cv s from a finite alphabet S where q 0 (s) is a pmf on S 0 and for n 1, Q(s s ) = Pr(S n =s S n 1 =s ) = Pr(S n =s S n 1
More informationNoisy channel communication
Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 6 Communication channels and Information Some notes on the noisy channel setup: Iain Murray, 2012 School of Informatics, University
More informationLecture 11: Continuous-valued signals and differential entropy
Lecture 11: Continuous-valued signals and differential entropy Biology 429 Carl Bergstrom September 20, 2008 Sources: Parts of today s lecture follow Chapter 8 from Cover and Thomas (2007). Some components
More informationEE5585 Data Compression April 18, Lecture 23
EE5585 Data Compression April 18, 013 Lecture 3 Instructor: Arya Mazumdar Scribe: Trevor Webster Differential Encoding Suppose we have a signal that is slowly varying For instance, if we were looking at
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More information4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information
4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information Ramji Venkataramanan Signal Processing and Communications Lab Department of Engineering ramji.v@eng.cam.ac.uk
More informationLecture 11: Quantum Information III - Source Coding
CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that
More informationThe Organization of Intrinsic Computation: Complexity-Entropy Diagrams and the Diversity of Natural Information Processing
Santa Fe Institute Working Paper 8-6-XXX arxiv.org:86.4789 [nlin.cd] The Organization of Intrinsic Computation: Complexity-Entropy Diagrams and the Diversity of Natural Information Processing David P.
More informationBayesian Inference and the Symbolic Dynamics of Deterministic Chaos. Christopher C. Strelioff 1,2 Dr. James P. Crutchfield 2
How Random Bayesian Inference and the Symbolic Dynamics of Deterministic Chaos Christopher C. Strelioff 1,2 Dr. James P. Crutchfield 2 1 Center for Complex Systems Research and Department of Physics University
More informationLecture 8: Shannon s Noise Models
Error Correcting Codes: Combinatorics, Algorithms and Applications (Fall 2007) Lecture 8: Shannon s Noise Models September 14, 2007 Lecturer: Atri Rudra Scribe: Sandipan Kundu& Atri Rudra Till now we have
More information(Classical) Information Theory II: Source coding
(Classical) Information Theory II: Source coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract The information content of a random variable
More informationIntrinsic Quantum Computation
Intrinsic Quantum Computation James P. Crutchfield Karoline Wiesner SFI WORKING PAPER: 2006-11-045 SFI Working Papers contain accounts of scientific work of the author(s and do not necessarily represent
More informationSolutions to Set #2 Data Compression, Huffman code and AEP
Solutions to Set #2 Data Compression, Huffman code and AEP. Huffman coding. Consider the random variable ( ) x x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0. 0.04 0.04 0.03 0.02 (a) Find a binary Huffman code
More informationCS 229r Information Theory in Computer Science Feb 12, Lecture 5
CS 229r Information Theory in Computer Science Feb 12, 2019 Lecture 5 Instructor: Madhu Sudan Scribe: Pranay Tankala 1 Overview A universal compression algorithm is a single compression algorithm applicable
More informationRegularities Unseen, Randomness Observed: Levels of Entropy Convergence
Regularities Unseen, Randomness Observed: evels of Entropy Convergence James P. Crutchfield David P. Feldman SFI WORKING PAPER: -- SFI Working Papers contain accounts of scientific work of the author(s)
More informationLecture 7: DecisionTrees
Lecture 7: DecisionTrees What are decision trees? Brief interlude on information theory Decision tree construction Overfitting avoidance Regression trees COMP-652, Lecture 7 - September 28, 2009 1 Recall:
More informationRegularities unseen, randomness observed: Levels of entropy convergence
CHAOS VOUME 13, NUMBER 1 MARCH 2003 Regularities unseen, randomness observed: evels of entropy convergence James P. Crutchfield a) Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, New Mexico 87501 David
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationCapacity of a channel Shannon s second theorem. Information Theory 1/33
Capacity of a channel Shannon s second theorem Information Theory 1/33 Outline 1. Memoryless channels, examples ; 2. Capacity ; 3. Symmetric channels ; 4. Channel Coding ; 5. Shannon s second theorem,
More informationLECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem
LECTURE 15 Last time: Feedback channel: setting up the problem Perfect feedback Feedback capacity Data compression Lecture outline Joint source and channel coding theorem Converse Robustness Brain teaser
More informationEE5139R: Problem Set 4 Assigned: 31/08/16, Due: 07/09/16
EE539R: Problem Set 4 Assigned: 3/08/6, Due: 07/09/6. Cover and Thomas: Problem 3.5 Sets defined by probabilities: Define the set C n (t = {x n : P X n(x n 2 nt } (a We have = P X n(x n P X n(x n 2 nt
More informationInformation. = more information was provided by the outcome in #2
Outline First part based very loosely on [Abramson 63]. Information theory usually formulated in terms of information channels and coding will not discuss those here.. Information 2. Entropy 3. Mutual
More informationA Gentle Tutorial on Information Theory and Learning. Roni Rosenfeld. Carnegie Mellon University
A Gentle Tutorial on Information Theory and Learning Roni Rosenfeld Mellon University Mellon Outline First part based very loosely on [Abramson 63]. Information theory usually formulated in terms of information
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More information1 Stat 605. Homework I. Due Feb. 1, 2011
The first part is homework which you need to turn in. The second part is exercises that will not be graded, but you need to turn it in together with the take-home final exam. 1 Stat 605. Homework I. Due
More informationHow Random is a Coin Toss? Bayesian Inference and the Symbolic Dynamics of Deterministic Chaos
How Random is a Coin Toss? Bayesian Inference and the Symbolic Dynamics of Deterministic Chaos Christopher C. Strelioff Center for Complex Systems Research and Department of Physics University of Illinois
More informationLECTURE 3. Last time:
LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate
More informationIntroduction to information theory and coding
Introduction to information theory and coding Louis WEHENKEL Set of slides No 4 Source modeling and source coding Stochastic processes and models for information sources First Shannon theorem : data compression
More informationChapter 4. Data Transmission and Channel Capacity. Po-Ning Chen, Professor. Department of Communications Engineering. National Chiao Tung University
Chapter 4 Data Transmission and Channel Capacity Po-Ning Chen, Professor Department of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30050, R.O.C. Principle of Data Transmission
More informationBasic Principles of Video Coding
Basic Principles of Video Coding Introduction Categories of Video Coding Schemes Information Theory Overview of Video Coding Techniques Predictive coding Transform coding Quantization Entropy coding Motion
More informationLecture 7. Order Out of Chaos
Lecture 7 Order Out of Chaos Lyapunov Exponents: Recall from Last Time The Lyapunov exponent for maps is: < 0if f 0 (x ) < 1 =ln f 0 (x ) = > 0if f 0 (x ) > 1 Lyapunov Exponents: Numerical Estimation Lyapunov
More informationIntroduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information.
L65 Dept. of Linguistics, Indiana University Fall 205 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission rate
More informationDept. of Linguistics, Indiana University Fall 2015
L645 Dept. of Linguistics, Indiana University Fall 2015 1 / 28 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission
More informationMassachusetts Institute of Technology
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.0J Information and Entropy Spring 2005 Issued: March 7, 2005
More informationMAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK UNIT V PART-A. 1. What is binary symmetric channel (AUC DEC 2006)
MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK SATELLITE COMMUNICATION DEPT./SEM.:ECE/VIII UNIT V PART-A 1. What is binary symmetric channel (AUC DEC 2006) 2. Define information rate? (AUC DEC 2007)
More informationReconstruction. Reading for this lecture: Lecture Notes.
ɛm Reconstruction Reading for this lecture: Lecture Notes. The Learning Channel... ɛ -Machine of a Process: Intrinsic representation! Predictive (or causal) equivalence relation: s s Pr( S S= s ) = Pr(
More informationLecture 1. Introduction
Lecture 1. Introduction What is the course about? Logistics Questionnaire Dr. Yao Xie, ECE587, Information Theory, Duke University What is information? Dr. Yao Xie, ECE587, Information Theory, Duke University
More informationRevision of Lecture 4
Revision of Lecture 4 We have completed studying digital sources from information theory viewpoint We have learnt all fundamental principles for source coding, provided by information theory Practical
More informationLecture 11: Information theory THURSDAY, FEBRUARY 21, 2019
Lecture 11: Information theory DANIEL WELLER THURSDAY, FEBRUARY 21, 2019 Agenda Information and probability Entropy and coding Mutual information and capacity Both images contain the same fraction of black
More informationInformation Theory CHAPTER. 5.1 Introduction. 5.2 Entropy
Haykin_ch05_pp3.fm Page 207 Monday, November 26, 202 2:44 PM CHAPTER 5 Information Theory 5. Introduction As mentioned in Chapter and reiterated along the way, the purpose of a communication system is
More information3F1 Information Theory, Lecture 1
3F1 Information Theory, Lecture 1 Jossy Sayir Department of Engineering Michaelmas 2013, 22 November 2013 Organisation History Entropy Mutual Information 2 / 18 Course Organisation 4 lectures Course material:
More informationSource Coding. Master Universitario en Ingeniería de Telecomunicación. I. Santamaría Universidad de Cantabria
Source Coding Master Universitario en Ingeniería de Telecomunicación I. Santamaría Universidad de Cantabria Contents Introduction Asymptotic Equipartition Property Optimal Codes (Huffman Coding) Universal
More informationHomework Set #2 Data Compression, Huffman code and AEP
Homework Set #2 Data Compression, Huffman code and AEP 1. Huffman coding. Consider the random variable ( x1 x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0.11 0.04 0.04 0.03 0.02 (a Find a binary Huffman code
More informationMAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK. SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A
MAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK DEPARTMENT: ECE SEMESTER: IV SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A 1. What is binary symmetric channel (AUC DEC
More informationInformation Theory, Statistics, and Decision Trees
Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010
More informationMechanisms of Chaos: Stable Instability
Mechanisms of Chaos: Stable Instability Reading for this lecture: NDAC, Sec. 2.-2.3, 9.3, and.5. Unpredictability: Orbit complicated: difficult to follow Repeatedly convergent and divergent Net amplification
More informationEntropy as a measure of surprise
Entropy as a measure of surprise Lecture 5: Sam Roweis September 26, 25 What does information do? It removes uncertainty. Information Conveyed = Uncertainty Removed = Surprise Yielded. How should we quantify
More informationComputational Systems Biology: Biology X
Bud Mishra Room 1002, 715 Broadway, Courant Institute, NYU, New York, USA L#8:(November-08-2010) Cancer and Signals Outline 1 Bayesian Interpretation of Probabilities Information Theory Outline Bayesian
More informationLecture 5 Linear Quadratic Stochastic Control
EE363 Winter 2008-09 Lecture 5 Linear Quadratic Stochastic Control linear-quadratic stochastic control problem solution via dynamic programming 5 1 Linear stochastic system linear dynamical system, over
More informationBeyond the typical set: Fluctuations in Intrinsic Computation
Beyond the typical set: Fluctuations in Intrinsic Computation Cina Aghamohammadi Complexity Sciences Center and Department of Physics, niversity of California at Davis, One Shields Avenue, Davis, CA 9566
More informationCommunication Theory II
Communication Theory II Lecture 15: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 29 th, 2015 1 Example: Channel Capacity of BSC o Let then: o For
More informationCan Feedback Increase the Capacity of the Energy Harvesting Channel?
Can Feedback Increase the Capacity of the Energy Harvesting Channel? Dor Shaviv EE Dept., Stanford University shaviv@stanford.edu Ayfer Özgür EE Dept., Stanford University aozgur@stanford.edu Haim Permuter
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More informationLecture 1: Introduction, Entropy and ML estimation
0-704: Information Processing and Learning Spring 202 Lecture : Introduction, Entropy and ML estimation Lecturer: Aarti Singh Scribes: Min Xu Disclaimer: These notes have not been subjected to the usual
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science Transmission of Information Spring 2006
MASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science 6.44 Transmission of Information Spring 2006 Homework 2 Solution name username April 4, 2006 Reading: Chapter
More informationThe Communication Complexity of Correlation. Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan
The Communication Complexity of Correlation Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan Transmitting Correlated Variables (X, Y) pair of correlated random variables Transmitting
More informationLECTURE 13. Last time: Lecture outline
LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to
More informationEE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018
Please submit the solutions on Gradescope. Some definitions that may be useful: EE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018 Definition 1: A sequence of random variables X
More informationInformation Theory and Coding Techniques
Information Theory and Coding Techniques Lecture 1.2: Introduction and Course Outlines Information Theory 1 Information Theory and Coding Techniques Prof. Ja-Ling Wu Department of Computer Science and
More informationFrom Determinism to Stochasticity
From Determinism to Stochasticity Reading for this lecture: (These) Lecture Notes. Lecture 8: Nonlinear Physics, Physics 5/25 (Spring 2); Jim Crutchfield Monday, May 24, 2 Cave: Sensory Immersive Visualization
More informationInformation Dynamics Foundations and Applications
Gustavo Deco Bernd Schürmann Information Dynamics Foundations and Applications With 89 Illustrations Springer PREFACE vii CHAPTER 1 Introduction 1 CHAPTER 2 Dynamical Systems: An Overview 7 2.1 Deterministic
More informationPrinciples of Communications
Principles of Communications Weiyao Lin Shanghai Jiao Tong University Chapter 10: Information Theory Textbook: Chapter 12 Communication Systems Engineering: Ch 6.1, Ch 9.1~ 9. 92 2009/2010 Meixia Tao @
More informationModelling Non-linear and Non-stationary Time Series
Modelling Non-linear and Non-stationary Time Series Chapter 2: Non-parametric methods Henrik Madsen Advanced Time Series Analysis September 206 Henrik Madsen (02427 Adv. TS Analysis) Lecture Notes September
More information10-704: Information Processing and Learning Fall Lecture 9: Sept 28
10-704: Information Processing and Learning Fall 2016 Lecturer: Siheng Chen Lecture 9: Sept 28 Note: These notes are based on scribed notes from Spring15 offering of this course. LaTeX template courtesy
More informationLecture 5: GPs and Streaming regression
Lecture 5: GPs and Streaming regression Gaussian Processes Information gain Confidence intervals COMP-652 and ECSE-608, Lecture 5 - September 19, 2017 1 Recall: Non-parametric regression Input space X
More informationUpper Bounds on the Capacity of Binary Intermittent Communication
Upper Bounds on the Capacity of Binary Intermittent Communication Mostafa Khoshnevisan and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame, Indiana 46556 Email:{mhoshne,
More informationCh. 8 Math Preliminaries for Lossy Coding. 8.4 Info Theory Revisited
Ch. 8 Math Preliminaries for Lossy Coding 8.4 Info Theory Revisited 1 Info Theory Goals for Lossy Coding Again just as for the lossless case Info Theory provides: Basis for Algorithms & Bounds on Performance
More informationChannel capacity. Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5.
Channel capacity Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5. Exercices Exercise session 11 : Channel capacity 1 1. Source entropy Given X a memoryless
More informationRevision of Lecture 5
Revision of Lecture 5 Information transferring across channels Channel characteristics and binary symmetric channel Average mutual information Average mutual information tells us what happens to information
More information1 Probability and Random Variables
1 Probability and Random Variables The models that you have seen thus far are deterministic models. For any time t, there is a unique solution X(t). On the other hand, stochastic models will result in
More informationChapter 8: Differential entropy. University of Illinois at Chicago ECE 534, Natasha Devroye
Chapter 8: Differential entropy Chapter 8 outline Motivation Definitions Relation to discrete entropy Joint and conditional differential entropy Relative entropy and mutual information Properties AEP for
More informationELEC546 Review of Information Theory
ELEC546 Review of Information Theory Vincent Lau 1/1/004 1 Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random
More informationCS 630 Basic Probability and Information Theory. Tim Campbell
CS 630 Basic Probability and Information Theory Tim Campbell 21 January 2003 Probability Theory Probability Theory is the study of how best to predict outcomes of events. An experiment (or trial or event)
More informationChaos, Complexity, and Inference (36-462)
Chaos, Complexity, and Inference (36-462) Lecture 5: Symbolic Dynamics; Making Discrete Stochastic Processes from Continuous Deterministic Dynamics Cosma Shalizi 27 January 2009 Symbolic dynamics Reducing
More informationShannon s A Mathematical Theory of Communication
Shannon s A Mathematical Theory of Communication Emre Telatar EPFL Kanpur October 19, 2016 First published in two parts in the July and October 1948 issues of BSTJ. First published in two parts in the
More informationMassachusetts Institute of Technology. Solution to Problem 1: The Registrar s Worst Nightmare
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.0J Information and Entropy Spring 2006 Issued: March 4, 2006
More informationAn instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
More informationLecture 2: August 31
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 2: August 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy
More information