Fundamentals of Information Theory Lecture 1. Introduction. Prof. CHEN Jie Lab. 201, School of EIE BeiHang University

Fundamentals of Information Theory Lecture 1. Introduction Prof. CHEN Jie Lab. 201, School of EIE BeiHang University

Teaching Staff Dr. YU Ze Office: F617, NMB Email: yz613@buaa.edu.cn Dr. SUN Bing Office: F617, NMB Email: bingsun@buaa.edu.cn Prof. CHEN Jie Dean of Depart. of Info.& Com. Eng., SEIE Office: F615, New Main Building (NMB) Email: chenjie@buaa.edu.cn Dr. HUANG Qin Office: F509, NMB Email: qinhuang@buaa.edu.cn

Text Book Thomas M. Cover (1938 2012) Fellow Members of the IEEE IEEE R. W. Hamming Medal Recipients

Lecture Notes

Laboratory Manual

Course Website http://infortheory.buaa.edu.cn/

Outline 1.Introduction and Preview 2.Entropy and Mutual Information 3.Asymptotic Equipartition Property 4.Markov chains 5.Data Compression 6.Channel Capacity 7.Differential Entropy 8.Gaussian Channel 9.Maximum Entropy and Spectral Estimation 10.Rate Distortion Theory 11.Network Information Theory

1. Introduction to Information Theoretical model of a typical communication system? How to distinguish information, signal And message What is Information?

1. Introduction to Information Information, in its general sense, is Knowledge communicated or received concerning a parti -cular fact or circumstance. " Information can t be predicted and resolves uncertainty. The uncertainty of an event is measured by its probability of occurrence and is inversely proportional to that. The more uncertain an event is, the more information is required to resolve uncertainty of that event. The amount of information is measured in bits. Example: information in fair one coin flip: log2(2/1) = 1 bit whereas in fair two coin flip is log2(4/1) = 2 bits.. http://en.wikipedia.org/

http://en.wikipedia.org/ 1. Introduction to Information Information, in its most restricted technical sense, is a sequence of symbols that can be interpreted as a message. Information can be recorded as signs, or transmitted as signals. Information is any kind of event that affects the stat e of a dynamic system. Information is the message being conveyed. Information is closely related to notions of constraint, communication, control, data, instruction, knowled ge, meaning, understanding, mental stimuli, pattern, perception, representation, and entropy.

1.1 Concept of information What is information? fair one coin flip S={T,F} operator will receive a call in next one hour

1.1 Concept of information Variations in value of 10 resistor Electromagnetic Interference Noisy Channel transmission

http://en.wikipedia.org/ 1.1 Concept of information To be or not to be, that is the question. William Shakespeare's play Hamlet

1.1 Concept of information Can you give some more examples?

1.1 Concept of information Can we measure information?

http://en.wikipedia.org/ 1.1 Concept of information The logarithmic connection between entropy and probability was first stated by L. Boltzmann in his kinetic theory of gases The famous formula for entropy S S k log e W k = 1.3806505(24) 10 23 J/K, Boltzmann's constant W is the Wahrscheinlichkeit, the frequency of occurrence of a macrostate, more precisely, the number of possible microstates corresponding to the macroscopic state of a system L. Boltzmann(1844-1906)

http://en.wikipedia.org/ 1.1 Concept of information Nyquist s logarithm law (1924) Harry Nyquist (1889 1976)

1.1 Concept of information Hartley s law (1928) Ralph Hartley (1888 1970) http://en.wikipedia.org/

1.1 Concept of information The uncertainty measure Uncertainty log p( x) The average uncertainty, Entropy H ( X ) p( x)log p( x) x Claude Shannon (1916-2001) Atheists /Electrical engineers Mathematicians & Statisticians Computer pioneers IEEE Medal of Honor recipients http://en.wikipedia.org/

http://en.wikipedia.org/ 1.1 Concept of information Claude Shannon (1916-2001) Atheists /Electrical engineers Mathematicians & Statisticians Computer pioneers IEEE Medal of Honor recipients

1.1 Concept of information What is Information: Information causes change; If it doesn t, it isn t information Claude Shannon (1916-2001) http://en.wikipedia.org/

1.2 Timeline of information theory

1.3 Information, Message and Signals Information: The uncertainty of source transmitted by communication system, which is contained by message and is still an abstract conception Message: More specific concept with all kinds of forms such as language, symbol, image which can be understood by both sides of communication system, or can be acquired/processed/stored by an information systems, e.g. remote sensing, GNSS. Signal: The most physical concept, which is carrier of message, being measurable, visible and physical

http://s4.sinaimg.cn/mw690/ 1.3 Information, Message and Signals Earth Observation System Configuration

http://www.jpl.nasa.gov/ Example1.3.1 VHF Band-Apollo-17/ALSE The Apollo 17 moon craft launched by U.S.(Dec.1972) made the SAR firstly perform in the space VHF radar antenna This SAR was named as Apollo Lunar Sounder Experiment (ALSE) ALSE was the first application in the human history to study the Moon's surface and interior using SAR based on the space probe

Example1.3.2 VHF Band- MARS Express http://www.jpl.nasa.gov/

Example1.3.3 S Band Cassini Huygens Radar image: Titan North Pole Lakes Saturn http://en.wikipedia.org/

Example1.3.5 Shuttle Radar Topography Mission Shuttle Radar Topography Mission (STRM,U.S.) use two radar antenna on board the space shuttle to implement the single-pass SAR interferometry Demonstration of STRM http://www.jpl.nasa.gov/ Demonstration of interferogram acquired by INSAR processing

Example1.3.5 Shuttle Radar Topography Mission http://www.jpl.nasa.gov/

Example1.3.6 SAR image: DEM of volcano Etna http://www.jpl.nasa.gov/

Example1.3.9 TerraSAR-X http://www.dlr.de/

Microwave EO Satellites TerraSAR-X Interferometric tandem X band Resolution 1m~18m RADARSAT-2 Polarimetric radar C band Resolution 1m-100m

Example1.3.10 IKONOS optical satellite IKONOS is one of the most advanced commercial optical satellites. IKONOS played an important role in the modern warfare and military application

Example1.3.10 IKONOS image of Beijing

Optical EO Satellites IKONOS Resolution: Pan=1 m M S(B,G,R,NIR)=4 m Scale: 1: 5,000 Mono and stereo GeoEye-1 Resolution: Pan=0.41/0.5m MS(B,G,R,NIR)=1.6/2 m Scale: 1: 2,000 Mono and stereo

Optical EO Satellites Space Imaging's IKONOS satellite captured these one-meter resolution colour images of the World Trade Center before and after the terrorist attack

Optical EO Satellites QuickBird-2 Resolution: Pan=0.65 m MS(B,G,R,NIR)= 2.62 m Scale: 1: 5,000 Mono only WorldView-1 Resolution: Pan=0.5 m Scale: 1: 2,000 Mono and stereo WorldView-2 Resolution: Pan=0.5 m MS1(B,G,R,NIR) & MS2(CB,Y,RE,NIR2)=2 m Scale: 1: 2,000 Mono and stereo

Optical EO Satellites Pléiades-1/2 Commercial June, 2012 Resolution: Pan=0.7/0.5m MS(B,G,R,NIR)=2.8/2m Scale: 1: 2 000 Mono and stéréo WorldView-3 (2014) Resolution: Pan= 0.30/0.5m Scale: 1: 2,000 Mono and stereo 16bands 4 additional SWIR bands

Example1.3.11 Terahertz image

1.3 Communication system model Channel

1.3 Communication system model

1.3 Communication system model Sound of ring bell Channel p=1/3 Received sound (simulated) Channel p=1/3 Satellite Remote sensing image of NMB, Beihang University http://map.google.com Received image (simulated)

1.3 Communication system model source Binary Symmetric Channel p=0.01 source Binary Symmetric Channel p=0.1 source Binary Symmetric Channel p=0.5

1.3 Communication system model Entropy Shannon argued that random processes such as music and speech have an irreducible complexity below which the signal cannot be compressed. This he named the entropy. Claude Shannon (1916-2001)

1.3 Communication system model Channel capacity In the early 1940s, it was thought that increasing the transmission rate of information over a communication channel increased the probability of error. Shannon surprised the communication theory community by proving that this was not true as long as the communication rate was below channel capacity.

http://en.wikipedia.org/ 1.3 Communication system model Information Theory answers: What is the bound on data compression The entropy rate H What is the limit on transmission rate The channel capacity C

Comparison of Transmission Property Before and After Huffman Coding BSC p=0.01 Huffman Coder BSC p=0.01 Decoder BSC p=0.1 Huffman Coder BSC p=0.1 Decoder

Demonstration of (M,n) Channel Coding BSC p=0.01 (2,3) Channel Coder BSC p=0.01 Decoder BSC p=0.1 (2,3) Channel Coder BSC p=0.1 Decoder

1.4 Information theory applications Information Theory intersects: Physics (statistical mechanics) Mathematics (probability theory) Electrical engineering (communication theory) Computer science (algorithmic complexity) Neurobiology Understanding of black holes Invention of the compact disc Voyage missions Development of to deep space the Internet

Physics AEP Thermodynamics Application Information theory Fisher information Hypothesis testing Statistics Inequalities Mathematics Figure1.1 The relationship of information theory with other fields

Thanks