Fundamentals of Information Theory Lecture 1. Introduction. Prof. CHEN Jie Lab. 201, School of EIE BeiHang University

Size: px
Start display at page:

Download "Fundamentals of Information Theory Lecture 1. Introduction. Prof. CHEN Jie Lab. 201, School of EIE BeiHang University"

Transcription

1 Fundamentals of Information Theory Lecture 1. Introduction Prof. CHEN Jie Lab. 201, School of EIE BeiHang University

2 Teaching Staff Dr. YU Ze Office: F617, NMB Dr. SUN Bing Office: F617, NMB Prof. CHEN Jie Dean of Depart. of Info.& Com. Eng., SEIE Office: F615, New Main Building (NMB) Dr. HUANG Qin Office: F509, NMB

3 Text Book Thomas M. Cover ( ) Fellow Members of the IEEE IEEE R. W. Hamming Medal Recipients

4 Lecture Notes

5 Laboratory Manual

6 Course Website

7 Outline 1.Introduction and Preview 2.Entropy and Mutual Information 3.Asymptotic Equipartition Property 4.Markov chains 5.Data Compression 6.Channel Capacity 7.Differential Entropy 8.Gaussian Channel 9.Maximum Entropy and Spectral Estimation 10.Rate Distortion Theory 11.Network Information Theory

8 1. Introduction to Information Theoretical model of a typical communication system? How to distinguish information, signal And message What is Information?

9 1. Introduction to Information Information, in its general sense, is Knowledge communicated or received concerning a parti -cular fact or circumstance. " Information can t be predicted and resolves uncertainty. The uncertainty of an event is measured by its probability of occurrence and is inversely proportional to that. The more uncertain an event is, the more information is required to resolve uncertainty of that event. The amount of information is measured in bits. Example: information in fair one coin flip: log2(2/1) = 1 bit whereas in fair two coin flip is log2(4/1) = 2 bits..

10 1. Introduction to Information Information, in its most restricted technical sense, is a sequence of symbols that can be interpreted as a message. Information can be recorded as signs, or transmitted as signals. Information is any kind of event that affects the stat e of a dynamic system. Information is the message being conveyed. Information is closely related to notions of constraint, communication, control, data, instruction, knowled ge, meaning, understanding, mental stimuli, pattern, perception, representation, and entropy.

11 1.1 Concept of information What is information? fair one coin flip S={T,F} operator will receive a call in next one hour

12 1.1 Concept of information Variations in value of 10 resistor Electromagnetic Interference Noisy Channel transmission

13 1.1 Concept of information To be or not to be, that is the question. William Shakespeare's play Hamlet

14 1.1 Concept of information Can you give some more examples?

15 1.1 Concept of information Can we measure information?

16 1.1 Concept of information The logarithmic connection between entropy and probability was first stated by L. Boltzmann in his kinetic theory of gases The famous formula for entropy S S k log e W k = (24) J/K, Boltzmann's constant W is the Wahrscheinlichkeit, the frequency of occurrence of a macrostate, more precisely, the number of possible microstates corresponding to the macroscopic state of a system L. Boltzmann( )

17 1.1 Concept of information Nyquist s logarithm law (1924) Harry Nyquist ( )

18 1.1 Concept of information Hartley s law (1928) Ralph Hartley ( )

19 1.1 Concept of information The uncertainty measure Uncertainty log p( x) The average uncertainty, Entropy H ( X ) p( x)log p( x) x Claude Shannon ( ) Atheists /Electrical engineers Mathematicians & Statisticians Computer pioneers IEEE Medal of Honor recipients

20 1.1 Concept of information Claude Shannon ( ) Atheists /Electrical engineers Mathematicians & Statisticians Computer pioneers IEEE Medal of Honor recipients

21 1.1 Concept of information What is Information: Information causes change; If it doesn t, it isn t information Claude Shannon ( )

22 1.2 Timeline of information theory

23 1.2 Timeline of information theory

24 1.2 Timeline of information theory

25 1.2 Timeline of information theory

26 1.2 Timeline of information theory

27 1.3 Information, Message and Signals Information: The uncertainty of source transmitted by communication system, which is contained by message and is still an abstract conception Message: More specific concept with all kinds of forms such as language, symbol, image which can be understood by both sides of communication system, or can be acquired/processed/stored by an information systems, e.g. remote sensing, GNSS. Signal: The most physical concept, which is carrier of message, being measurable, visible and physical

28 1.3 Information, Message and Signals Earth Observation System Configuration

29 Example1.3.1 VHF Band-Apollo-17/ALSE The Apollo 17 moon craft launched by U.S.(Dec.1972) made the SAR firstly perform in the space VHF radar antenna This SAR was named as Apollo Lunar Sounder Experiment (ALSE) ALSE was the first application in the human history to study the Moon's surface and interior using SAR based on the space probe

30 Example1.3.2 VHF Band- MARS Express

31 Example1.3.3 S Band Cassini Huygens Radar image: Titan North Pole Lakes Saturn

32 Example1.3.5 Shuttle Radar Topography Mission Shuttle Radar Topography Mission (STRM,U.S.) use two radar antenna on board the space shuttle to implement the single-pass SAR interferometry Demonstration of STRM Demonstration of interferogram acquired by INSAR processing

33 Example1.3.5 Shuttle Radar Topography Mission

34 Example1.3.6 SAR image: DEM of volcano Etna

35 Example1.3.9 TerraSAR-X

36 Example1.3.9 TerraSAR-X

37 Example1.3.9 TerraSAR-X

38 Example1.3.9 TerraSAR-X

39 Example1.3.9 TerraSAR-X

40 Example1.3.9 TerraSAR-X

41 Example1.3.9 TerraSAR-X

42 Microwave EO Satellites TerraSAR-X Interferometric tandem X band Resolution 1m~18m RADARSAT-2 Polarimetric radar C band Resolution 1m-100m

43 Example IKONOS optical satellite IKONOS is one of the most advanced commercial optical satellites. IKONOS played an important role in the modern warfare and military application

44 Example IKONOS image of Beijing

45 Optical EO Satellites IKONOS Resolution: Pan=1 m M S(B,G,R,NIR)=4 m Scale: 1: 5,000 Mono and stereo GeoEye-1 Resolution: Pan=0.41/0.5m MS(B,G,R,NIR)=1.6/2 m Scale: 1: 2,000 Mono and stereo

46 Optical EO Satellites Space Imaging's IKONOS satellite captured these one-meter resolution colour images of the World Trade Center before and after the terrorist attack

47 Optical EO Satellites QuickBird-2 Resolution: Pan=0.65 m MS(B,G,R,NIR)= 2.62 m Scale: 1: 5,000 Mono only WorldView-1 Resolution: Pan=0.5 m Scale: 1: 2,000 Mono and stereo WorldView-2 Resolution: Pan=0.5 m MS1(B,G,R,NIR) & MS2(CB,Y,RE,NIR2)=2 m Scale: 1: 2,000 Mono and stereo

48 Optical EO Satellites Pléiades-1/2 Commercial June, 2012 Resolution: Pan=0.7/0.5m MS(B,G,R,NIR)=2.8/2m Scale: 1: Mono and stéréo WorldView-3 (2014) Resolution: Pan= 0.30/0.5m Scale: 1: 2,000 Mono and stereo 16bands 4 additional SWIR bands

49 Example Terahertz image

50 Example Terahertz image

51 1.3 Communication system model Channel

52 1.3 Communication system model

53 1.3 Communication system model Sound of ring bell Channel p=1/3 Received sound (simulated) Channel p=1/3 Satellite Remote sensing image of NMB, Beihang University Received image (simulated)

54 1.3 Communication system model source Binary Symmetric Channel p=0.01 source Binary Symmetric Channel p=0.1 source Binary Symmetric Channel p=0.5

55 1.3 Communication system model Entropy Shannon argued that random processes such as music and speech have an irreducible complexity below which the signal cannot be compressed. This he named the entropy. Claude Shannon ( )

56 1.3 Communication system model Channel capacity In the early 1940s, it was thought that increasing the transmission rate of information over a communication channel increased the probability of error. Shannon surprised the communication theory community by proving that this was not true as long as the communication rate was below channel capacity.

57 1.3 Communication system model Information Theory answers: What is the bound on data compression The entropy rate H What is the limit on transmission rate The channel capacity C

58 Comparison of Transmission Property Before and After Huffman Coding BSC p=0.01 Huffman Coder BSC p=0.01 Decoder BSC p=0.1 Huffman Coder BSC p=0.1 Decoder

59 Demonstration of (M,n) Channel Coding BSC p=0.01 (2,3) Channel Coder BSC p=0.01 Decoder BSC p=0.1 (2,3) Channel Coder BSC p=0.1 Decoder

60 1.4 Information theory applications Information Theory intersects: Physics (statistical mechanics) Mathematics (probability theory) Electrical engineering (communication theory) Computer science (algorithmic complexity) Neurobiology Understanding of black holes Invention of the compact disc Voyage missions Development of to deep space the Internet

61 Physics AEP Thermodynamics Application Information theory Fisher information Hypothesis testing Statistics Inequalities Mathematics Figure1.1 The relationship of information theory with other fields

62 Thanks

Outline of the Lecture. Background and Motivation. Basics of Information Theory: 1. Introduction. Markku Juntti. Course Overview

Outline of the Lecture. Background and Motivation. Basics of Information Theory: 1. Introduction. Markku Juntti. Course Overview : Markku Juntti Overview The basic ideas and concepts of information theory are introduced. Some historical notes are made and the overview of the course is given. Source The material is mainly based on

More information

Information in Biology

Information in Biology Lecture 3: Information in Biology Tsvi Tlusty, tsvi@unist.ac.kr Living information is carried by molecular channels Living systems I. Self-replicating information processors Environment II. III. Evolve

More information

ELEMENTS O F INFORMATION THEORY

ELEMENTS O F INFORMATION THEORY ELEMENTS O F INFORMATION THEORY THOMAS M. COVER JOY A. THOMAS Preface to the Second Edition Preface to the First Edition Acknowledgments for the Second Edition Acknowledgments for the First Edition x

More information

Digital Communications III (ECE 154C) Introduction to Coding and Information Theory

Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 2014 1 / 8 I

More information

Information in Biology

Information in Biology Information in Biology CRI - Centre de Recherches Interdisciplinaires, Paris May 2012 Information processing is an essential part of Life. Thinking about it in quantitative terms may is useful. 1 Living

More information

Entropies & Information Theory

Entropies & Information Theory Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information

More information

Murray Gell-Mann, The Quark and the Jaguar, 1995

Murray Gell-Mann, The Quark and the Jaguar, 1995 Although [complex systems] differ widely in their physical attributes, they resemble one another in the way they handle information. That common feature is perhaps the best starting point for exploring

More information

An introduction to basic information theory. Hampus Wessman

An introduction to basic information theory. Hampus Wessman An introduction to basic information theory Hampus Wessman Abstract We give a short and simple introduction to basic information theory, by stripping away all the non-essentials. Theoretical bounds on

More information

(Classical) Information Theory III: Noisy channel coding

(Classical) Information Theory III: Noisy channel coding (Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way

More information

The Liar Game. Mark Wildon

The Liar Game. Mark Wildon The Liar Game Mark Wildon Guessing Games Ask a friend to thinks of a number between 0 and 15. How many NO/YES questions do you need to ask to find out the secret number? Guessing Games Ask a friend to

More information

CSCI 2570 Introduction to Nanocomputing

CSCI 2570 Introduction to Nanocomputing CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication

More information

Multimedia Communications. Mathematical Preliminaries for Lossless Compression

Multimedia Communications. Mathematical Preliminaries for Lossless Compression Multimedia Communications Mathematical Preliminaries for Lossless Compression What we will see in this chapter Definition of information and entropy Modeling a data source Definition of coding and when

More information

A Mathematical Theory of Communication

A Mathematical Theory of Communication A Mathematical Theory of Communication Ben Eggers Abstract This paper defines information-theoretic entropy and proves some elementary results about it. Notably, we prove that given a few basic assumptions

More information

Part I. Entropy. Information Theory and Networks. Section 1. Entropy: definitions. Lecture 5: Entropy

Part I. Entropy. Information Theory and Networks. Section 1. Entropy: definitions. Lecture 5: Entropy and Networks Lecture 5: Matthew Roughan http://www.maths.adelaide.edu.au/matthew.roughan/ Lecture_notes/InformationTheory/ Part I School of Mathematical Sciences, University

More information

Lecture 8: Shannon s Noise Models

Lecture 8: Shannon s Noise Models Error Correcting Codes: Combinatorics, Algorithms and Applications (Fall 2007) Lecture 8: Shannon s Noise Models September 14, 2007 Lecturer: Atri Rudra Scribe: Sandipan Kundu& Atri Rudra Till now we have

More information

Revision of Lecture 4

Revision of Lecture 4 Revision of Lecture 4 We have completed studying digital sources from information theory viewpoint We have learnt all fundamental principles for source coding, provided by information theory Practical

More information

Noisy channel communication

Noisy channel communication Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 6 Communication channels and Information Some notes on the noisy channel setup: Iain Murray, 2012 School of Informatics, University

More information

CSE468 Information Conflict

CSE468 Information Conflict CSE468 Information Conflict Lecturer: Dr Carlo Kopp, MIEEE, MAIAA, PEng Lecture 02 Introduction to Information Theory Concepts Reference Sources and Bibliography There is an abundance of websites and publications

More information

6.02 Fall 2012 Lecture #1

6.02 Fall 2012 Lecture #1 6.02 Fall 2012 Lecture #1 Digital vs. analog communication The birth of modern digital communication Information and entropy Codes, Huffman coding 6.02 Fall 2012 Lecture 1, Slide #1 6.02 Fall 2012 Lecture

More information

6.02 Fall 2011 Lecture #9

6.02 Fall 2011 Lecture #9 6.02 Fall 2011 Lecture #9 Claude E. Shannon Mutual information Channel capacity Transmission at rates up to channel capacity, and with asymptotically zero error 6.02 Fall 2011 Lecture 9, Slide #1 First

More information

1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H.

1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H. Problem sheet Ex. Verify that the function H(p,..., p n ) = k p k log p k satisfies all 8 axioms on H. Ex. (Not to be handed in). looking at the notes). List as many of the 8 axioms as you can, (without

More information

Channel capacity. Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5.

Channel capacity. Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5. Channel capacity Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5. Exercices Exercise session 11 : Channel capacity 1 1. Source entropy Given X a memoryless

More information

Information Theory and Coding Techniques

Information Theory and Coding Techniques Information Theory and Coding Techniques Lecture 1.2: Introduction and Course Outlines Information Theory 1 Information Theory and Coding Techniques Prof. Ja-Ling Wu Department of Computer Science and

More information

Chapter 9 Fundamental Limits in Information Theory

Chapter 9 Fundamental Limits in Information Theory Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For

More information

ECE INFORMATION THEORY. Fall 2011 Prof. Thinh Nguyen

ECE INFORMATION THEORY. Fall 2011 Prof. Thinh Nguyen 1 ECE 566 - INFORMATION THEORY Fall 2011 Prof. Thinh Nguyen LOGISTICS Prerequisites: ECE 353, strong mathematical background Office hours: M, W 11-Noon, KEC3115 Course Content: Entropy, Relative Entropy,

More information

Principles of Communications

Principles of Communications Principles of Communications Weiyao Lin Shanghai Jiao Tong University Chapter 10: Information Theory Textbook: Chapter 12 Communication Systems Engineering: Ch 6.1, Ch 9.1~ 9. 92 2009/2010 Meixia Tao @

More information

Computing and Communications 2. Information Theory -Entropy

Computing and Communications 2. Information Theory -Entropy 1896 1920 1987 2006 Computing and Communications 2. Information Theory -Entropy Ying Cui Department of Electronic Engineering Shanghai Jiao Tong University, China 2017, Autumn 1 Outline Entropy Joint entropy

More information

UNIT I INFORMATION THEORY. I k log 2

UNIT I INFORMATION THEORY. I k log 2 UNIT I INFORMATION THEORY Claude Shannon 1916-2001 Creator of Information Theory, lays the foundation for implementing logic in digital circuits as part of his Masters Thesis! (1939) and published a paper

More information

9. Distance measures. 9.1 Classical information measures. Head Tail. How similar/close are two probability distributions? Trace distance.

9. Distance measures. 9.1 Classical information measures. Head Tail. How similar/close are two probability distributions? Trace distance. 9. Distance measures 9.1 Classical information measures How similar/close are two probability distributions? Trace distance Fidelity Example: Flipping two coins, one fair one biased Head Tail Trace distance

More information

Channel Coding I. Exercises SS 2017

Channel Coding I. Exercises SS 2017 Channel Coding I Exercises SS 2017 Lecturer: Dirk Wübben Tutor: Shayan Hassanpour NW1, Room N 2420, Tel.: 0421/218-62387 E-mail: {wuebben, hassanpour}@ant.uni-bremen.de Universität Bremen, FB1 Institut

More information

Welcome to Comp 411! 2) Course Objectives. 1) Course Mechanics. 3) Information. I thought this course was called Computer Organization

Welcome to Comp 411! 2) Course Objectives. 1) Course Mechanics. 3) Information. I thought this course was called Computer Organization Welcome to Comp 4! I thought this course was called Computer Organization David Macaulay ) Course Mechanics 2) Course Objectives 3) Information L - Introduction Meet the Crew Lectures: Leonard McMillan

More information

Information Theory and Coding Techniques: Chapter 1.1. What is Information Theory? Why you should take this course?

Information Theory and Coding Techniques: Chapter 1.1. What is Information Theory? Why you should take this course? Information Theory and Coding Techniques: Chapter 1.1 What is Information Theory? Why you should take this course? 1 What is Information Theory? Information Theory answers two fundamental questions in

More information

Lecture 1: Shannon s Theorem

Lecture 1: Shannon s Theorem Lecture 1: Shannon s Theorem Lecturer: Travis Gagie January 13th, 2015 Welcome to Data Compression! I m Travis and I ll be your instructor this week. If you haven t registered yet, don t worry, we ll work

More information

Communication Theory II

Communication Theory II Communication Theory II Lecture 15: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 29 th, 2015 1 Example: Channel Capacity of BSC o Let then: o For

More information

Shannon's Theory of Communication

Shannon's Theory of Communication Shannon's Theory of Communication An operational introduction 5 September 2014, Introduction to Information Systems Giovanni Sileno g.sileno@uva.nl Leibniz Center for Law University of Amsterdam Fundamental

More information

Classification & Information Theory Lecture #8

Classification & Information Theory Lecture #8 Classification & Information Theory Lecture #8 Introduction to Natural Language Processing CMPSCI 585, Fall 2007 University of Massachusetts Amherst Andrew McCallum Today s Main Points Automatically categorizing

More information

Remote Sensing I: Basics

Remote Sensing I: Basics Remote Sensing I: Basics Kelly M. Brunt Earth System Science Interdisciplinary Center, University of Maryland Cryospheric Science Laboratory, Goddard Space Flight Center kelly.m.brunt@nasa.gov (Based on

More information

Chapter 2 Review of Classical Information Theory

Chapter 2 Review of Classical Information Theory Chapter 2 Review of Classical Information Theory Abstract This chapter presents a review of the classical information theory which plays a crucial role in this thesis. We introduce the various types of

More information

3F1 Information Theory, Lecture 1

3F1 Information Theory, Lecture 1 3F1 Information Theory, Lecture 1 Jossy Sayir Department of Engineering Michaelmas 2013, 22 November 2013 Organisation History Entropy Mutual Information 2 / 18 Course Organisation 4 lectures Course material:

More information

The Continuing Miracle of Information Storage Technology Paul H. Siegel Director, CMRR University of California, San Diego

The Continuing Miracle of Information Storage Technology Paul H. Siegel Director, CMRR University of California, San Diego The Continuing Miracle of Information Storage Technology Paul H. Siegel Director, CMRR University of California, San Diego 10/15/01 1 Outline The Shannon Statue A Miraculous Technology Information Theory

More information

Revision of Lecture 5

Revision of Lecture 5 Revision of Lecture 5 Information transferring across channels Channel characteristics and binary symmetric channel Average mutual information Average mutual information tells us what happens to information

More information

Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006

Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006 Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006 Fabio Grazioso... July 3, 2006 1 2 Contents 1 Lecture 1, Entropy 4 1.1 Random variable...............................

More information

Lecture 22: Final Review

Lecture 22: Final Review Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information

More information

Multiple-Input Multiple-Output Systems

Multiple-Input Multiple-Output Systems Multiple-Input Multiple-Output Systems What is the best way to use antenna arrays? MIMO! This is a totally new approach ( paradigm ) to wireless communications, which has been discovered in 95-96. Performance

More information

Introduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information.

Introduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information. L65 Dept. of Linguistics, Indiana University Fall 205 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission rate

More information

Dept. of Linguistics, Indiana University Fall 2015

Dept. of Linguistics, Indiana University Fall 2015 L645 Dept. of Linguistics, Indiana University Fall 2015 1 / 28 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission

More information

MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK UNIT V PART-A. 1. What is binary symmetric channel (AUC DEC 2006)

MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK UNIT V PART-A. 1. What is binary symmetric channel (AUC DEC 2006) MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK SATELLITE COMMUNICATION DEPT./SEM.:ECE/VIII UNIT V PART-A 1. What is binary symmetric channel (AUC DEC 2006) 2. Define information rate? (AUC DEC 2007)

More information

4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information

4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information 4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information Ramji Venkataramanan Signal Processing and Communications Lab Department of Engineering ramji.v@eng.cam.ac.uk

More information

Roll No. :... Invigilator's Signature :.. CS/B.TECH(ECE)/SEM-7/EC-703/ CODING & INFORMATION THEORY. Time Allotted : 3 Hours Full Marks : 70

Roll No. :... Invigilator's Signature :.. CS/B.TECH(ECE)/SEM-7/EC-703/ CODING & INFORMATION THEORY. Time Allotted : 3 Hours Full Marks : 70 Name : Roll No. :.... Invigilator's Signature :.. CS/B.TECH(ECE)/SEM-7/EC-703/2011-12 2011 CODING & INFORMATION THEORY Time Allotted : 3 Hours Full Marks : 70 The figures in the margin indicate full marks

More information

CS 630 Basic Probability and Information Theory. Tim Campbell

CS 630 Basic Probability and Information Theory. Tim Campbell CS 630 Basic Probability and Information Theory Tim Campbell 21 January 2003 Probability Theory Probability Theory is the study of how best to predict outcomes of events. An experiment (or trial or event)

More information

One Lesson of Information Theory

One Lesson of Information Theory Institut für One Lesson of Information Theory Prof. Dr.-Ing. Volker Kühn Institute of Communications Engineering University of Rostock, Germany Email: volker.kuehn@uni-rostock.de http://www.int.uni-rostock.de/

More information

EC2252 COMMUNICATION THEORY UNIT 5 INFORMATION THEORY

EC2252 COMMUNICATION THEORY UNIT 5 INFORMATION THEORY EC2252 COMMUNICATION THEORY UNIT 5 INFORMATION THEORY Discrete Messages and Information Content, Concept of Amount of Information, Average information, Entropy, Information rate, Source coding to increase

More information

Lecture 1. Introduction

Lecture 1. Introduction Lecture 1. Introduction What is the course about? Logistics Questionnaire Dr. Yao Xie, ECE587, Information Theory, Duke University What is information? Dr. Yao Xie, ECE587, Information Theory, Duke University

More information

ELEMENT OF INFORMATION THEORY

ELEMENT OF INFORMATION THEORY History Table of Content ELEMENT OF INFORMATION THEORY O. Le Meur olemeur@irisa.fr Univ. of Rennes 1 http://www.irisa.fr/temics/staff/lemeur/ October 2010 1 History Table of Content VERSION: 2009-2010:

More information

Notes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel

Notes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic

More information

Neural coding Ecological approach to sensory coding: efficient adaptation to the natural environment

Neural coding Ecological approach to sensory coding: efficient adaptation to the natural environment Neural coding Ecological approach to sensory coding: efficient adaptation to the natural environment Jean-Pierre Nadal CNRS & EHESS Laboratoire de Physique Statistique (LPS, UMR 8550 CNRS - ENS UPMC Univ.

More information

to mere bit flips) may affect the transmission.

to mere bit flips) may affect the transmission. 5 VII. QUANTUM INFORMATION THEORY to mere bit flips) may affect the transmission. A. Introduction B. A few bits of classical information theory Information theory has developed over the past five or six

More information

Noisy-Channel Coding

Noisy-Channel Coding Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/05264298 Part II Noisy-Channel Coding Copyright Cambridge University Press 2003.

More information

Chaos, Complexity, and Inference (36-462)

Chaos, Complexity, and Inference (36-462) Chaos, Complexity, and Inference (36-462) Lecture 7: Information Theory Cosma Shalizi 3 February 2009 Entropy and Information Measuring randomness and dependence in bits The connection to statistics Long-run

More information

(Classical) Information Theory II: Source coding

(Classical) Information Theory II: Source coding (Classical) Information Theory II: Source coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract The information content of a random variable

More information

ITCT Lecture IV.3: Markov Processes and Sources with Memory

ITCT Lecture IV.3: Markov Processes and Sources with Memory ITCT Lecture IV.3: Markov Processes and Sources with Memory 4. Markov Processes Thus far, we have been occupied with memoryless sources and channels. We must now turn our attention to sources with memory.

More information

1. Basics of Information

1. Basics of Information 1. Basics of Information 6.004x Computation Structures Part 1 Digital Circuits Copyright 2015 MIT EECS 6.004 Computation Structures L1: Basics of Information, Slide #1 What is Information? Information,

More information

Reliable Computation over Multiple-Access Channels

Reliable Computation over Multiple-Access Channels Reliable Computation over Multiple-Access Channels Bobak Nazer and Michael Gastpar Dept. of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA, 94720-1770 {bobak,

More information

EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018

EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code

More information

MAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK. SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A

MAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK. SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A MAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK DEPARTMENT: ECE SEMESTER: IV SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A 1. What is binary symmetric channel (AUC DEC

More information

Lecture 16. Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code

Lecture 16. Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code Lecture 16 Agenda for the lecture Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code Variable-length source codes with error 16.1 Error-free coding schemes 16.1.1 The Shannon-Fano-Elias

More information

Information Theory. Coding and Information Theory. Information Theory Textbooks. Entropy

Information Theory. Coding and Information Theory. Information Theory Textbooks. Entropy Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is

More information

Information and Entropy

Information and Entropy Information and Entropy Shannon s Separation Principle Source Coding Principles Entropy Variable Length Codes Huffman Codes Joint Sources Arithmetic Codes Adaptive Codes Thomas Wiegand: Digital Image Communication

More information

Lecture Notes for Communication Theory

Lecture Notes for Communication Theory Lecture Notes for Communication Theory February 15, 2011 Please let me know of any errors, typos, or poorly expressed arguments. And please do not reproduce or distribute this document outside Oxford University.

More information

Shannon s noisy-channel theorem

Shannon s noisy-channel theorem Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for

More information

1 Background on Information Theory

1 Background on Information Theory Review of the book Information Theory: Coding Theorems for Discrete Memoryless Systems by Imre Csiszár and János Körner Second Edition Cambridge University Press, 2011 ISBN:978-0-521-19681-9 Review by

More information

Information Theory - Entropy. Figure 3

Information Theory - Entropy. Figure 3 Concept of Information Information Theory - Entropy Figure 3 A typical binary coded digital communication system is shown in Figure 3. What is involved in the transmission of information? - The system

More information

ECE Information theory Final (Fall 2008)

ECE Information theory Final (Fall 2008) ECE 776 - Information theory Final (Fall 2008) Q.1. (1 point) Consider the following bursty transmission scheme for a Gaussian channel with noise power N and average power constraint P (i.e., 1/n X n i=1

More information

ECE Information theory Final

ECE Information theory Final ECE 776 - Information theory Final Q1 (1 point) We would like to compress a Gaussian source with zero mean and variance 1 We consider two strategies In the first, we quantize with a step size so that the

More information

Lecture 1: Introduction, Entropy and ML estimation

Lecture 1: Introduction, Entropy and ML estimation 0-704: Information Processing and Learning Spring 202 Lecture : Introduction, Entropy and ML estimation Lecturer: Aarti Singh Scribes: Min Xu Disclaimer: These notes have not been subjected to the usual

More information

Some of the best pictures of the planets in our solar system 19 January 2015, by Matt Williams

Some of the best pictures of the planets in our solar system 19 January 2015, by Matt Williams Some of the best pictures of the planets in our solar system 19 January 2015, by Matt Williams Our Solar System is a pretty picturesque place. Between the Sun, the Moon, and the Inner and Outer Solar System,

More information

Information Theory, Statistics, and Decision Trees

Information Theory, Statistics, and Decision Trees Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010

More information

Compression and Coding

Compression and Coding Compression and Coding Theory and Applications Part 1: Fundamentals Gloria Menegaz 1 Transmitter (Encoder) What is the problem? Receiver (Decoder) Transformation information unit Channel Ordering (significance)

More information

ELECTRONICS & COMMUNICATIONS DIGITAL COMMUNICATIONS

ELECTRONICS & COMMUNICATIONS DIGITAL COMMUNICATIONS EC 32 (CR) Total No. of Questions :09] [Total No. of Pages : 02 III/IV B.Tech. DEGREE EXAMINATIONS, APRIL/MAY- 207 Second Semester ELECTRONICS & COMMUNICATIONS DIGITAL COMMUNICATIONS Time: Three Hours

More information

VID3: Sampling and Quantization

VID3: Sampling and Quantization Video Transmission VID3: Sampling and Quantization By Prof. Gregory D. Durgin copyright 2009 all rights reserved Claude E. Shannon (1916-2001) Mathematician and Electrical Engineer Worked for Bell Labs

More information

Coding into a source: an inverse rate-distortion theorem

Coding into a source: an inverse rate-distortion theorem Coding into a source: an inverse rate-distortion theorem Anant Sahai joint work with: Mukul Agarwal Sanjoy K. Mitter Wireless Foundations Department of Electrical Engineering and Computer Sciences University

More information

Intro to Information Theory

Intro to Information Theory Intro to Information Theory Math Circle February 11, 2018 1. Random variables Let us review discrete random variables and some notation. A random variable X takes value a A with probability P (a) 0. Here

More information

Nanotechnology-inspired Information Processing Systems of the Future

Nanotechnology-inspired Information Processing Systems of the Future Nanotechnology-inspired Information Processing Systems of the Future Lav R. Varshney University of Illinois at Urbana-Champaign August 31, 2016 Cross-cutting Panel 3 Putting intelligence in (trillions

More information

Introduction to Information Theory

Introduction to Information Theory Introduction to Information Theory Gurinder Singh Mickey Atwal atwal@cshl.edu Center for Quantitative Biology Kullback-Leibler Divergence Summary Shannon s coding theorems Entropy Mutual Information Multi-information

More information

National University of Singapore Department of Electrical & Computer Engineering. Examination for

National University of Singapore Department of Electrical & Computer Engineering. Examination for National University of Singapore Department of Electrical & Computer Engineering Examination for EE5139R Information Theory for Communication Systems (Semester I, 2014/15) November/December 2014 Time Allowed:

More information

Channel Coding I. Exercises SS 2017

Channel Coding I. Exercises SS 2017 Channel Coding I Exercises SS 2017 Lecturer: Dirk Wübben Tutor: Shayan Hassanpour NW1, Room N 2420, Tel.: 0421/218-62387 E-mail: {wuebben, hassanpour}@ant.uni-bremen.de Universität Bremen, FB1 Institut

More information

UNIT E: SPACE EXPLORATION

UNIT E: SPACE EXPLORATION UNIT E: SPACE EXPLORATION S C I E N C E 9 1 Science 9 Unit E Section 3.0 OPTICAL TELESCOPES, RADIO TELESCOPES, AND OTHER TECHNOLOGIES ADVANCE OUR UNDERSTANDING OF SPACE SECTI ON 3.0 Science 9 Unit E Section

More information

log 2 N I m m log 2 N + 1 m.

log 2 N I m m log 2 N + 1 m. SOPHOMORE COLLEGE MATHEMATICS OF THE INFORMATION AGE SHANNON S THEOREMS Let s recall the fundamental notions of information and entropy. To repeat, Shannon s emphasis is on selecting a given message from

More information

Massachusetts Institute of Technology

Massachusetts Institute of Technology Name (%): Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mechanical Engineering 6.050J/2.0J Information and Entropy Spring 2005 Issued: May

More information

INTRODUCTION TO MICROWAVE REMOTE SENSING. Dr. A. Bhattacharya

INTRODUCTION TO MICROWAVE REMOTE SENSING. Dr. A. Bhattacharya 1 INTRODUCTION TO MICROWAVE REMOTE SENSING Dr. A. Bhattacharya Why Microwaves? More difficult than with optical imaging because the technology is more complicated and the image data recorded is more varied.

More information

LECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem

LECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem LECTURE 15 Last time: Feedback channel: setting up the problem Perfect feedback Feedback capacity Data compression Lecture outline Joint source and channel coding theorem Converse Robustness Brain teaser

More information

Introduction to Information Theory

Introduction to Information Theory Introduction to Information Theory Impressive slide presentations Radu Trîmbiţaş UBB October 2012 Radu Trîmbiţaş (UBB) Introduction to Information Theory October 2012 1 / 19 Transmission of information

More information

16.36 Communication Systems Engineering

16.36 Communication Systems Engineering MIT OpenCourseWare http://ocw.mit.edu 16.36 Communication Systems Engineering Spring 2009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. 16.36: Communication

More information

Lecture #27: Saturn. The Main Point. The Jovian Planets. Basic Properties of Saturn. Saturn:

Lecture #27: Saturn. The Main Point. The Jovian Planets. Basic Properties of Saturn. Saturn: Lecture #27: Saturn Saturn: General properties. Atmosphere. Interior. Origin and evolution. Reading: Chapters 7.1 (Saturn) and 11.1. The Main Point Saturn is a large Jovian-class planet with a composition

More information

Introduction to Information Theory. Part 2

Introduction to Information Theory. Part 2 Introduction to Information Theory Part 2 1 A General Communication System CHANNEL Information Source Transmitter Channel Receiver Destination 2 Information: Definition Information is quantified using

More information

Geological Mapping Using EO Data for Onshore O&G Exploration

Geological Mapping Using EO Data for Onshore O&G Exploration Geological Mapping Using EO Data for Onshore O&G Exploration Michael Hall ESA Oil and Gas Workshop, Frascati, Italy michael.hall@infoterra-global.com Why use EO data for Geological Mapping? Availability

More information

Audio Coding. Fundamentals Quantization Waveform Coding Subband Coding P NCTU/CSIE DSPLAB C.M..LIU

Audio Coding. Fundamentals Quantization Waveform Coding Subband Coding P NCTU/CSIE DSPLAB C.M..LIU Audio Coding P.1 Fundamentals Quantization Waveform Coding Subband Coding 1. Fundamentals P.2 Introduction Data Redundancy Coding Redundancy Spatial/Temporal Redundancy Perceptual Redundancy Compression

More information

Midterm Exam Information Theory Fall Midterm Exam. Time: 09:10 12:10 11/23, 2016

Midterm Exam Information Theory Fall Midterm Exam. Time: 09:10 12:10 11/23, 2016 Midterm Exam Time: 09:10 12:10 11/23, 2016 Name: Student ID: Policy: (Read before You Start to Work) The exam is closed book. However, you are allowed to bring TWO A4-size cheat sheet (single-sheet, two-sided).

More information

CS6304 / Analog and Digital Communication UNIT IV - SOURCE AND ERROR CONTROL CODING PART A 1. What is the use of error control coding? The main use of error control coding is to reduce the overall probability

More information

Lecture 11: Information theory THURSDAY, FEBRUARY 21, 2019

Lecture 11: Information theory THURSDAY, FEBRUARY 21, 2019 Lecture 11: Information theory DANIEL WELLER THURSDAY, FEBRUARY 21, 2019 Agenda Information and probability Entropy and coding Mutual information and capacity Both images contain the same fraction of black

More information