Digital Communications III (ECE 154C) Introduction to Coding and Information Theory
|
|
- Claire Newton
- 6 years ago
- Views:
Transcription
1 Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring / 26
2 Lossy Coding Distortion Coding With Distortion 2 / 26
3 Coding with Distortion Lossy Coding Distortion Placeholder Figure A ǫ 2 = lim T 1 T T/2 T/2 E(x(t) ˆx(t)) 2 dt = M.S.E. 3 / 26
4 Discrete-Time Signals Lossy Coding Distortion If signals are bandlimited, one can sample at nyquist rate and convert continuous-time problem to discrete-time problem. This sampling is part of the A/D converter. Placeholder Figure A ǫ 2 = lim m 1 m m E(x i ˆx i ) 2 i=1 4 / 26
5 A/D A/D Conversion and D/A Conversion 5 / 26
6 A/D Conversion A/D Assume a random variablex which falls into the range (X min,x max ). The goal is forx to be converted intok binary digits. Let M = 2 k. The usual A/D converter first subdivides the interval (X min,x max ) intom equal sub-intervals. Subintervals are of width = (X max X min )/M. Shown below for the case ofk = 3 andm = 8. Placeholder Figure A We call thei th sub-interval,r i. 6 / 26
7 D/A Conversion A/D Assume that ifx falls in the regionr i, i.e.x R i converter uses as an estimate ofx, the value ˆX = Y which is the center of thei th region. The mean-squared error betweenx and ˆX is ǫ 2 = E[(X ˆX) 2 ] = Xmax X min (X ˆX) 2 f x (x)dx wheref x (x) is the probability density function of the random variablex. Letf X Ri (x) be the conditional density function ofx given that X falls in the regionr i. Then ǫ 2 = M P[x R i ] (x y i ) 2 f x Ri (x)dx x R i i=1 7 / 26
8 D/A Conversion A/D Note that for i = 1,2,...,M M P[x R i ] = i=1 and x R i f X Ri (x)dx = 1. Make the further assumption thatk is large enough so that f X Ri (x) is a constant over the regionr i. Thenf X Ri (x) = 1 for alli, and x R i (x y i ) 2 f X Ri (x)dx = 1 = 1 b a 2 2 = (x ( b a 2 ))2 dx (x 0) 2 dx ( ) 3 = / 26
9 D/A Conversion A/D In other words,ǫ 2 = M i=1 P[x R i ] 2 12 = 2 12 IfX has varianceσx 2, the signal-to-noise ratio of the A to the D (& D to A) converter is often defined as ( σ 2 x/ 2 12 IfX min is equal to and/orx max = +, then the last and first intervals can be infinite in extent. Howeverf x (x) is usually small enough in those intervals so that the result is still approximately the same. ) Placeholder Figure A 9 / 26
10 Quantization Quantizer Optimal Quantizer Iterative Solution SCALAR QUANTIZATION of GAUSSIAN SAMPLES 10 / 26
11 Quantization Quantizer Optimal Quantizer Iterative Solution ENCODER: Placeholder Figure A x 3b < x < b 100 3b < x 2b 001 b < x 2b 101 2b x b 010 2b < x 3b 110 b x b < x 111 DECODER: b b b b b b 011.5b b 11 / 26
12 Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution Let us construct boundariesb i,(b 0 =,b M = + and quantization symbolsa i such that b i 1 x < b i ˆx = a i i = 1,2,...,M Placeholder Figure A The question is how to optimize{b i } and{a i } to minimize distortionǫ 2 ǫ 2 = M i=1 bi b i 1 (x a i ) 2 f x (x)dx 12 / 26
13 Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution To optimizeǫ 2 = M i=1 bi b i 1 (x a i ) 2 f x (x)dx we take derivatives and putting them equal to zero: δǫ 2 δa j = 0 And use Leibnitz s Rule: δǫ 2 δb j = 0 δ δt b(t) a(t) f(x,t)dx =f(b(t),t) δb(t) δt f(a(t),t) δa(t) δt b(t) δ + δt f(x,t)dt a(t) 13 / 26
14 Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution δ δb j bj δ δb j ( M i=1 bi b j 1 (x a j ) 2 f x (x)dx+ δ δb j b i 1 (x a i ) 2 f x (x)dx bj+1 b j ) = (x a j+1 ) 2 f x (x)dx = (b j a j ) 2 f x (x) x=bj (b j a j+1 ) 2 f x (x) x=bj = 0 b 2 j 2a j b j +a 2 j = b 2 j 2b j a j+1 +a 2 j+1 2b j (a j+1 a j ) = a 2 j+1 a 2 j b j = a j+1+a j 2 (I) 14 / 26
15 Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution δ δa j ( M i=1 b i 1 b i (x a i ) 2 f x (x)dx a j bj a j = b j 1 f x (x)dx = ) bj b j b j 1 xf x (x)dx b j b j 1 f x (x)dx = 2 bj b j 1 xf x (x)dx b j 1 (x a j )f x (x)dx = (II) 15 / 26
16 Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution Note that the{b i } can be found from (I) once the{a i } is known. In fact, the{b i } are the midpoints of the{a i }. The{a i } can also be solved from (II) once the{b i } are known. The{a i } are centroids of the corresponding regions. Thus one can use a computer to iteratively solve for the{a i } and the{b i } 1. One starts with an initial guess for the{b i }. 2. One uses (II) to solve for the{a i }. 3. One uses (I) to solve for the{b i }. 4. One repeats steps 2 and 3 until the{a i } and the{b i } stop changing. 16 / 26
17 Comments on Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution 1. This works for anyf x (x) 2. Iff x (x) only has a finite support one adjustsb 0 &b M to be the limits of the support. 3. One needs to know Placeholder Figure β f x (x)dx and α A β xf x (x)dx (true for anyf x (x)) 4. For a Gaussian, we can integrate by parts or lety = x 2 α β α 1 β e 1 2 x2 dx = Q(β) Q(α) 2π α x 1 2π e 1 2 x2 dx = / 26
18 Comments on Optimum Scalar Quantizer Quantization Quantizer Optimal Quantizer Iterative Solution 5. IfM = 2 a one could useabinary digits to represent the quantized value. However since the quantized values are not necessarily equally likely, one could use a HUFFMAN CODE to use fewer binary digits(on the average). 6. After the{a i } and{b i } are known, one computesǫ 2 from M ǫ 2 = i=1 bi b i 1 (x a i ) 2 f x (x)dx 7. ForM = 2 andf x (x) = 1 2πσ 2 e 1 2 x 2 σ 2 we have b 0 =,b 1 = 0,b 2 = +, anda 2 = a 1 = 2σ 2 π 8. Also easy to show thatǫ 2 = (1 2 π )σ2 =.3634σ / 26
19 Quantization Gaussain DMS Gaussain DMS 19 / 26
20 Quantization Gaussain DMS Gaussain DMS One can achieve a smallerǫ 2 by quantizing several samples at a time. We would then use regions in anm-dimensional space Placeholder Figure A Shannon characterized this in terms of rate-distortion formula which tells us how smallǫ 2 can be (m ). For a Gaussian source with one binary digit per sample, ǫ 2 σ2 4 = 0.25σ2 This follows from the result on the next page. Contrast this with scalar case: ǫ 2 s = (1 2 π )σ2 =.3634σ / 26
21 VQ: Discrete Memoryless Gaussian Source Quantization Gaussain DMS Gaussain DMS Let source produce i.i.d. Gaussian samplesx 1,X 2,... where f X (x) = 1 2πσ 2 e 1 2 Let the source encoder produce a sequence of binary digits at a rate ofrbinary digits/source symbol. x 2 σ 2 In our previous terminologyr = logm Let the source decoder produce the sequence ˆX 1, ˆX 2,..., ˆX i,... such that the mean-squared error between{x i } and{ ˆX i } isǫ 2. ǫ 2 = 1 n n E{(X i ˆX i ) 2 } i=1 21 / 26
22 VQ: Discrete Memoryless Gaussian Source Quantization Gaussain DMS Gaussain DMS Then one can prove that for any such system R 1 2 log 2( σ2 ǫ 2 ) for ǫ2 σ 2 Note thatr = 0 forǫ 2 σ 2. What does this mean? Note that forr = logm = 1, log 2( σ2 ǫ 2 ) 2 log 2( σ2 ǫ 2 ) 4 σ2 ǫ 2 ǫ2 (1/4)σ 2 This is an example of Rate-Distortion Theory. 22 / 26
23 MP3 CD MPEG-1 Layer 3 Reduced Fidelity Audio Compression 23 / 26
24 Reduced Fidelity MP3 players use a form of audio compression called MPEG-1 Audio Layer 3. MP3 CD MPEG-1 Layer 3 It takes advantage of a psycho-acoustic phenomena whereby a loud tone at one frequency masks the presence of softer tones at neighboring frequencies; hence, these softer neighbouring tones need not be stored(or transmitted). Compression efficiency of an audio compression scheme is usually described by the encoded bit rate (prior to the introduction of coding bits.) 24 / 26
25 Reduced Fidelity MP3 CD MPEG-1 Layer 3 The CD has a bit rate of( ) = bits/second. The term is the sampling rate which is approximately the Nyquist frequency of the audio to be compressed. The term2comes from the fact that there are two channels in a stereo audio system. The term16 comes from the 16-bit (or2 16 = level) A to D converter. Note that a slightly higher sampling rate samples/second is used for a DAT recorder. 25 / 26
26 Reduced Fidelity MP3 CD MPEG-1 Layer 3 Different standards are used in MP3 players. Several bit rates are specified in the MPEG-1, Layer 3 standard. These are32,40,48,56,64,80,96,112,128,144,160, 192, 224, 256 and 320 kilobits/sec. The sampling rates allowed are32,44.1 and48 kilohz but the sampling rate of Hz is almost always used. The basic idea behind the scheme is as follows. A block of 576 time domain samples are converted into 576 frequency domain samples using a DFT. The coefficients then modified using psycho-acoustic principles. The processed coefficients are then converted into a bit stream using various schemes including Huffman Encoding. The process is reversed at the receiver: bits frequency domain coefficients time domain samples. 26 / 26
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 2014 1 / 8 I
More informationPrinciples of Communications
Principles of Communications Weiyao Lin, PhD Shanghai Jiao Tong University Chapter 4: Analog-to-Digital Conversion Textbook: 7.1 7.4 2010/2011 Meixia Tao @ SJTU 1 Outline Analog signal Sampling Quantization
More informationDigital Communications III (ECE 154C) Introduction to Coding and Information Theory
Digital Communications III (ECE 54C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 204 / 2 Noiseless
More informationGaussian source Assumptions d = (x-y) 2, given D, find lower bound of I(X;Y)
Gaussian source Assumptions d = (x-y) 2, given D, find lower bound of I(X;Y) E{(X-Y) 2 } D
More informationC.M. Liu Perceptual Signal Processing Lab College of Computer Science National Chiao-Tung University
Quantization C.M. Liu Perceptual Signal Processing Lab College of Computer Science National Chiao-Tung University http://www.csie.nctu.edu.tw/~cmliu/courses/compression/ Office: EC538 (03)5731877 cmliu@cs.nctu.edu.tw
More information18.2 Continuous Alphabet (discrete-time, memoryless) Channel
0-704: Information Processing and Learning Spring 0 Lecture 8: Gaussian channel, Parallel channels and Rate-distortion theory Lecturer: Aarti Singh Scribe: Danai Koutra Disclaimer: These notes have not
More informationLecture 20: Quantization and Rate-Distortion
Lecture 20: Quantization and Rate-Distortion Quantization Introduction to rate-distortion theorem Dr. Yao Xie, ECE587, Information Theory, Duke University Approimating continuous signals... Dr. Yao Xie,
More informationMultimedia Networking ECE 599
Multimedia Networking ECE 599 Prof. Thinh Nguyen School of Electrical Engineering and Computer Science Based on lectures from B. Lee, B. Girod, and A. Mukherjee 1 Outline Digital Signal Representation
More information4. Quantization and Data Compression. ECE 302 Spring 2012 Purdue University, School of ECE Prof. Ilya Pollak
4. Quantization and Data Compression ECE 32 Spring 22 Purdue University, School of ECE Prof. What is data compression? Reducing the file size without compromising the quality of the data stored in the
More informationCoding for Discrete Source
EGR 544 Communication Theory 3. Coding for Discrete Sources Z. Aliyazicioglu Electrical and Computer Engineering Department Cal Poly Pomona Coding for Discrete Source Coding Represent source data effectively
More informationEE368B Image and Video Compression
EE368B Image and Video Compression Homework Set #2 due Friday, October 20, 2000, 9 a.m. Introduction The Lloyd-Max quantizer is a scalar quantizer which can be seen as a special case of a vector quantizer
More informationECE Information theory Final
ECE 776 - Information theory Final Q1 (1 point) We would like to compress a Gaussian source with zero mean and variance 1 We consider two strategies In the first, we quantize with a step size so that the
More informationQuantization. Introduction. Roadmap. Optimal Quantizer Uniform Quantizer Non Uniform Quantizer Rate Distorsion Theory. Source coding.
Roadmap Quantization Optimal Quantizer Uniform Quantizer Non Uniform Quantizer Rate Distorsion Theory Source coding 2 Introduction 4 1 Lossy coding Original source is discrete Lossless coding: bit rate
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More informationBasic Principles of Video Coding
Basic Principles of Video Coding Introduction Categories of Video Coding Schemes Information Theory Overview of Video Coding Techniques Predictive coding Transform coding Quantization Entropy coding Motion
More information2018/5/3. YU Xiangyu
2018/5/3 YU Xiangyu yuxy@scut.edu.cn Entropy Huffman Code Entropy of Discrete Source Definition of entropy: If an information source X can generate n different messages x 1, x 2,, x i,, x n, then the
More informationRevision of Lecture 5
Revision of Lecture 5 Information transferring across channels Channel characteristics and binary symmetric channel Average mutual information Average mutual information tells us what happens to information
More informationImage Compression. Fundamentals: Coding redundancy. The gray level histogram of an image can reveal a great deal of information about the image
Fundamentals: Coding redundancy The gray level histogram of an image can reveal a great deal of information about the image That probability (frequency) of occurrence of gray level r k is p(r k ), p n
More informationScalar and Vector Quantization. National Chiao Tung University Chun-Jen Tsai 11/06/2014
Scalar and Vector Quantization National Chiao Tung University Chun-Jen Tsai 11/06/014 Basic Concept of Quantization Quantization is the process of representing a large, possibly infinite, set of values
More informationPrinciples of Communications
Principles of Communications Weiyao Lin Shanghai Jiao Tong University Chapter 10: Information Theory Textbook: Chapter 12 Communication Systems Engineering: Ch 6.1, Ch 9.1~ 9. 92 2009/2010 Meixia Tao @
More informationE303: Communication Systems
E303: Communication Systems Professor A. Manikas Chair of Communications and Array Processing Imperial College London Principles of PCM Prof. A. Manikas (Imperial College) E303: Principles of PCM v.17
More informationMAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK. SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A
MAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK DEPARTMENT: ECE SEMESTER: IV SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A 1. What is binary symmetric channel (AUC DEC
More informationLecture 22: Final Review
Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information
More informationMultimedia Systems Giorgio Leonardi A.A Lecture 4 -> 6 : Quantization
Multimedia Systems Giorgio Leonardi A.A.2014-2015 Lecture 4 -> 6 : Quantization Overview Course page (D.I.R.): https://disit.dir.unipmn.it/course/view.php?id=639 Consulting: Office hours by appointment:
More informationDigital communication system. Shannon s separation principle
Digital communication system Representation of the source signal by a stream of (binary) symbols Adaptation to the properties of the transmission channel information source source coder channel coder modulation
More informationMAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK UNIT V PART-A. 1. What is binary symmetric channel (AUC DEC 2006)
MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK SATELLITE COMMUNICATION DEPT./SEM.:ECE/VIII UNIT V PART-A 1. What is binary symmetric channel (AUC DEC 2006) 2. Define information rate? (AUC DEC 2007)
More informationMultimedia Communications. Scalar Quantization
Multimedia Communications Scalar Quantization Scalar Quantization In many lossy compression applications we want to represent source outputs using a small number of code words. Process of representing
More informationEC2252 COMMUNICATION THEORY UNIT 5 INFORMATION THEORY
EC2252 COMMUNICATION THEORY UNIT 5 INFORMATION THEORY Discrete Messages and Information Content, Concept of Amount of Information, Average information, Entropy, Information rate, Source coding to increase
More informationRun-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE
General e Image Coder Structure Motion Video x(s 1,s 2,t) or x(s 1,s 2 ) Natural Image Sampling A form of data compression; usually lossless, but can be lossy Redundancy Removal Lossless compression: predictive
More informationVector Quantization Encoder Decoder Original Form image Minimize distortion Table Channel Image Vectors Look-up (X, X i ) X may be a block of l
Vector Quantization Encoder Decoder Original Image Form image Vectors X Minimize distortion k k Table X^ k Channel d(x, X^ Look-up i ) X may be a block of l m image or X=( r, g, b ), or a block of DCT
More informationAudio Coding. Fundamentals Quantization Waveform Coding Subband Coding P NCTU/CSIE DSPLAB C.M..LIU
Audio Coding P.1 Fundamentals Quantization Waveform Coding Subband Coding 1. Fundamentals P.2 Introduction Data Redundancy Coding Redundancy Spatial/Temporal Redundancy Perceptual Redundancy Compression
More information7.1 Sampling and Reconstruction
Haberlesme Sistemlerine Giris (ELE 361) 6 Agustos 2017 TOBB Ekonomi ve Teknoloji Universitesi, Guz 2017-18 Dr. A. Melda Yuksel Turgut & Tolga Girici Lecture Notes Chapter 7 Analog to Digital Conversion
More informationVector Quantization. Institut Mines-Telecom. Marco Cagnazzo, MN910 Advanced Compression
Institut Mines-Telecom Vector Quantization Marco Cagnazzo, cagnazzo@telecom-paristech.fr MN910 Advanced Compression 2/66 19.01.18 Institut Mines-Telecom Vector Quantization Outline Gain-shape VQ 3/66 19.01.18
More informationEE-597 Notes Quantization
EE-597 Notes Quantization Phil Schniter June, 4 Quantization Given a continuous-time and continuous-amplitude signal (t, processing and storage by modern digital hardware requires discretization in both
More informationECE Advanced Communication Theory, Spring 2009 Homework #1 (INCOMPLETE)
ECE 74 - Advanced Communication Theory, Spring 2009 Homework #1 (INCOMPLETE) 1. A Huffman code finds the optimal codeword to assign to a given block of source symbols. (a) Show that cannot be a Huffman
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationSolutions to Homework Set #1 Sanov s Theorem, Rate distortion
st Semester 00/ Solutions to Homework Set # Sanov s Theorem, Rate distortion. Sanov s theorem: Prove the simple version of Sanov s theorem for the binary random variables, i.e., let X,X,...,X n be a sequence
More informationECE472/572 - Lecture 11. Roadmap. Roadmap. Image Compression Fundamentals and Lossless Compression Techniques 11/03/11.
ECE47/57 - Lecture Image Compression Fundamentals and Lossless Compression Techniques /03/ Roadmap Preprocessing low level Image Enhancement Image Restoration Image Segmentation Image Acquisition Image
More informationMARKOV CHAINS A finite state Markov chain is a sequence of discrete cv s from a finite alphabet where is a pmf on and for
MARKOV CHAINS A finite state Markov chain is a sequence S 0,S 1,... of discrete cv s from a finite alphabet S where q 0 (s) is a pmf on S 0 and for n 1, Q(s s ) = Pr(S n =s S n 1 =s ) = Pr(S n =s S n 1
More informationLecture 2: Introduction to Audio, Video & Image Coding Techniques (I) -- Fundaments
Lecture 2: Introduction to Audio, Video & Image Coding Techniques (I) -- Fundaments Dr. Jian Zhang Conjoint Associate Professor NICTA & CSE UNSW COMP9519 Multimedia Systems S2 2006 jzhang@cse.unsw.edu.au
More informationat Some sort of quantization is necessary to represent continuous signals in digital form
Quantization at Some sort of quantization is necessary to represent continuous signals in digital form x(n 1,n ) x(t 1,tt ) D Sampler Quantizer x q (n 1,nn ) Digitizer (A/D) Quantization is also used for
More informationCS578- Speech Signal Processing
CS578- Speech Signal Processing Lecture 7: Speech Coding Yannis Stylianou University of Crete, Computer Science Dept., Multimedia Informatics Lab yannis@csd.uoc.gr Univ. of Crete Outline 1 Introduction
More informationLecture 2: Introduction to Audio, Video & Image Coding Techniques (I) -- Fundaments. Tutorial 1. Acknowledgement and References for lectures 1 to 5
Lecture : Introduction to Audio, Video & Image Coding Techniques (I) -- Fundaments Dr. Jian Zhang Conjoint Associate Professor NICTA & CSE UNSW COMP959 Multimedia Systems S 006 jzhang@cse.unsw.edu.au Acknowledgement
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output
More informationCommunication Theory II
Communication Theory II Lecture 15: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 29 th, 2015 1 Example: Channel Capacity of BSC o Let then: o For
More information1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H.
Problem sheet Ex. Verify that the function H(p,..., p n ) = k p k log p k satisfies all 8 axioms on H. Ex. (Not to be handed in). looking at the notes). List as many of the 8 axioms as you can, (without
More informationDigital Communications III (ECE 154C) Introduction to Coding and Information Theory
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 2014 1 / 14 Statement
More informationECE Information theory Final (Fall 2008)
ECE 776 - Information theory Final (Fall 2008) Q.1. (1 point) Consider the following bursty transmission scheme for a Gaussian channel with noise power N and average power constraint P (i.e., 1/n X n i=1
More informationGaussian channel. Information theory 2013, lecture 6. Jens Sjölund. 8 May Jens Sjölund (IMT, LiU) Gaussian channel 1 / 26
Gaussian channel Information theory 2013, lecture 6 Jens Sjölund 8 May 2013 Jens Sjölund (IMT, LiU) Gaussian channel 1 / 26 Outline 1 Definitions 2 The coding theorem for Gaussian channel 3 Bandlimited
More informationLecture 1: Shannon s Theorem
Lecture 1: Shannon s Theorem Lecturer: Travis Gagie January 13th, 2015 Welcome to Data Compression! I m Travis and I ll be your instructor this week. If you haven t registered yet, don t worry, we ll work
More informationExercise 1. = P(y a 1)P(a 1 )
Chapter 7 Channel Capacity Exercise 1 A source produces independent, equally probable symbols from an alphabet {a 1, a 2 } at a rate of one symbol every 3 seconds. These symbols are transmitted over a
More information! Where are we on course map? ! What we did in lab last week. " How it relates to this week. ! Compression. " What is it, examples, classifications
Lecture #3 Compression! Where are we on course map?! What we did in lab last week " How it relates to this week! Compression " What is it, examples, classifications " Probability based compression # Huffman
More informationThe Secrets of Quantization. Nimrod Peleg Update: Sept. 2009
The Secrets of Quantization Nimrod Peleg Update: Sept. 2009 What is Quantization Representation of a large set of elements with a much smaller set is called quantization. The number of elements in the
More informationThe information loss in quantization
The information loss in quantization The rough meaning of quantization in the frame of coding is representing numerical quantities with a finite set of symbols. The mapping between numbers, which are normally
More informationELECTRONICS & COMMUNICATIONS DIGITAL COMMUNICATIONS
EC 32 (CR) Total No. of Questions :09] [Total No. of Pages : 02 III/IV B.Tech. DEGREE EXAMINATIONS, APRIL/MAY- 207 Second Semester ELECTRONICS & COMMUNICATIONS DIGITAL COMMUNICATIONS Time: Three Hours
More informationRemote Estimation Games over Shared Networks
October st, 04 Remote Estimation Games over Shared Networks Marcos Vasconcelos & Nuno Martins marcos@umd.edu Dept. of Electrical and Computer Engineering Institute of Systems Research University of Maryland,
More information8/19/16. Fourier Analysis. Fourier analysis: the dial tone phone. Fourier analysis: the dial tone phone
Patrice Koehl Department of Biological Sciences National University of Singapore http://www.cs.ucdavis.edu/~koehl/teaching/bl5229 koehl@cs.ucdavis.edu Fourier analysis: the dial tone phone We use Fourier
More informationELEC546 Review of Information Theory
ELEC546 Review of Information Theory Vincent Lau 1/1/004 1 Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random
More informationDigital Modulation 1
Digital Modulation 1 Lecture Notes Ingmar Land and Bernard H. Fleury Navigation and Communications () Department of Electronic Systems Aalborg University, DK Version: February 5, 27 i Contents I Basic
More informationLecture 5 Channel Coding over Continuous Channels
Lecture 5 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 14, 2014 1 / 34 I-Hsiang Wang NIT Lecture 5 From
More informationEE/Stat 376B Handout #5 Network Information Theory October, 14, Homework Set #2 Solutions
EE/Stat 376B Handout #5 Network Information Theory October, 14, 014 1. Problem.4 parts (b) and (c). Homework Set # Solutions (b) Consider h(x + Y ) h(x + Y Y ) = h(x Y ) = h(x). (c) Let ay = Y 1 + Y, where
More informationImage Data Compression
Image Data Compression Image data compression is important for - image archiving e.g. satellite data - image transmission e.g. web data - multimedia applications e.g. desk-top editing Image data compression
More informationEE 5345 Biomedical Instrumentation Lecture 12: slides
EE 5345 Biomedical Instrumentation Lecture 1: slides 4-6 Carlos E. Davila, Electrical Engineering Dept. Southern Methodist University slides can be viewed at: http:// www.seas.smu.edu/~cd/ee5345.html EE
More informationIterative Quantization. Using Codes On Graphs
Iterative Quantization Using Codes On Graphs Emin Martinian and Jonathan S. Yedidia 2 Massachusetts Institute of Technology 2 Mitsubishi Electric Research Labs Lossy Data Compression: Encoding: Map source
More informationLORD: LOw-complexity, Rate-controlled, Distributed video coding system
LORD: LOw-complexity, Rate-controlled, Distributed video coding system Rami Cohen and David Malah Signal and Image Processing Lab Department of Electrical Engineering Technion - Israel Institute of Technology
More informationEXAMPLE OF SCALAR AND VECTOR QUANTIZATION
EXAMPLE OF SCALAR AD VECTOR QUATIZATIO Source sequence : This could be the output of a highly correlated source. A scalar quantizer: =1, M=4 C 1 = {w 1,w 2,w 3,w 4 } = {-4, -1, 1, 4} = codeboo of quantization
More informationBlock 2: Introduction to Information Theory
Block 2: Introduction to Information Theory Francisco J. Escribano April 26, 2015 Francisco J. Escribano Block 2: Introduction to Information Theory April 26, 2015 1 / 51 Table of contents 1 Motivation
More informationInformation Dimension
Information Dimension Mina Karzand Massachusetts Institute of Technology November 16, 2011 1 / 26 2 / 26 Let X would be a real-valued random variable. For m N, the m point uniform quantized version of
More informationInformation and Entropy
Information and Entropy Shannon s Separation Principle Source Coding Principles Entropy Variable Length Codes Huffman Codes Joint Sources Arithmetic Codes Adaptive Codes Thomas Wiegand: Digital Image Communication
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationlossless, optimal compressor
6. Variable-length Lossless Compression The principal engineering goal of compression is to represent a given sequence a, a 2,..., a n produced by a source as a sequence of bits of minimal possible length.
More informationTransform Coding. Transform Coding Principle
Transform Coding Principle of block-wise transform coding Properties of orthonormal transforms Discrete cosine transform (DCT) Bit allocation for transform coefficients Entropy coding of transform coefficients
More informationSOURCE CODING WITH SIDE INFORMATION AT THE DECODER (WYNER-ZIV CODING) FEB 13, 2003
SOURCE CODING WITH SIDE INFORMATION AT THE DECODER (WYNER-ZIV CODING) FEB 13, 2003 SLEPIAN-WOLF RESULT { X i} RATE R x ENCODER 1 DECODER X i V i {, } { V i} ENCODER 0 RATE R v Problem: Determine R, the
More informationReview of Quantization. Quantization. Bring in Probability Distribution. L-level Quantization. Uniform partition
Review of Quantization UMCP ENEE631 Slides (created by M.Wu 004) Quantization UMCP ENEE631 Slides (created by M.Wu 001/004) L-level Quantization Minimize errors for this lossy process What L values to
More informationEE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018
Please submit the solutions on Gradescope. Some definitions that may be useful: EE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018 Definition 1: A sequence of random variables X
More informationL. Yaroslavsky. Fundamentals of Digital Image Processing. Course
L. Yaroslavsky. Fundamentals of Digital Image Processing. Course 0555.330 Lec. 6. Principles of image coding The term image coding or image compression refers to processing image digital data aimed at
More informationDesign of Optimal Quantizers for Distributed Source Coding
Design of Optimal Quantizers for Distributed Source Coding David Rebollo-Monedero, Rui Zhang and Bernd Girod Information Systems Laboratory, Electrical Eng. Dept. Stanford University, Stanford, CA 94305
More informationCompression and Coding
Compression and Coding Theory and Applications Part 1: Fundamentals Gloria Menegaz 1 Transmitter (Encoder) What is the problem? Receiver (Decoder) Transformation information unit Channel Ordering (significance)
More informationETSF15 Analog/Digital. Stefan Höst
ETSF15 Analog/Digital Stefan Höst Physical layer Analog vs digital Sampling, quantisation, reconstruction Modulation Represent digital data in a continuous world Disturbances Noise and distortion Synchronization
More informationChannel capacity. Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5.
Channel capacity Outline : 1. Source entropy 2. Discrete memoryless channel 3. Mutual information 4. Channel capacity 5. Exercices Exercise session 11 : Channel capacity 1 1. Source entropy Given X a memoryless
More informationEE376A - Information Theory Final, Monday March 14th 2016 Solutions. Please start answering each question on a new page of the answer booklet.
EE376A - Information Theory Final, Monday March 14th 216 Solutions Instructions: You have three hours, 3.3PM - 6.3PM The exam has 4 questions, totaling 12 points. Please start answering each question on
More informationECS 332: Principles of Communications 2012/1. HW 4 Due: Sep 7
ECS 332: Principles of Communications 2012/1 HW 4 Due: Sep 7 Lecturer: Prapun Suksompong, Ph.D. Instructions (a) ONE part of a question will be graded (5 pt). Of course, you do not know which part will
More informationCompression methods: the 1 st generation
Compression methods: the 1 st generation 1998-2017 Josef Pelikán CGG MFF UK Praha pepca@cgg.mff.cuni.cz http://cgg.mff.cuni.cz/~pepca/ Still1g 2017 Josef Pelikán, http://cgg.mff.cuni.cz/~pepca 1 / 32 Basic
More informationUCSD ECE250 Handout #27 Prof. Young-Han Kim Friday, June 8, Practice Final Examination (Winter 2017)
UCSD ECE250 Handout #27 Prof. Young-Han Kim Friday, June 8, 208 Practice Final Examination (Winter 207) There are 6 problems, each problem with multiple parts. Your answer should be as clear and readable
More informationInformation Theory. Coding and Information Theory. Information Theory Textbooks. Entropy
Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is
More informationQuantization for Distributed Estimation
0 IEEE International Conference on Internet of Things ithings 0), Green Computing and Communications GreenCom 0), and Cyber-Physical-Social Computing CPSCom 0) Quantization for Distributed Estimation uan-yu
More informationExample: for source
Nonuniform scalar quantizer References: Sayood Chap. 9, Gersho and Gray, Chap.'s 5 and 6. The basic idea: For a nonuniform source density, put smaller cells and levels where the density is larger, thereby
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Lesson 7 Delta Modulation and DPCM Instructional Objectives At the end of this lesson, the students should be able to: 1. Describe a lossy predictive coding scheme.
More informationLecture 1. Introduction
Lecture 1. Introduction What is the course about? Logistics Questionnaire Dr. Yao Xie, ECE587, Information Theory, Duke University What is information? Dr. Yao Xie, ECE587, Information Theory, Duke University
More informationEE5356 Digital Image Processing
EE5356 Digital Image Processing INSTRUCTOR: Dr KR Rao Spring 007, Final Thursday, 10 April 007 11:00 AM 1:00 PM ( hours) (Room 111 NH) INSTRUCTIONS: 1 Closed books and closed notes All problems carry weights
More informationTools for Signal Compression
Tools for Signal Compression Tools for Signal Compression Nicolas Moreau First published 2011 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc. Adapted and updated from Outils
More informationExercises with solutions (Set D)
Exercises with solutions Set D. A fair die is rolled at the same time as a fair coin is tossed. Let A be the number on the upper surface of the die and let B describe the outcome of the coin toss, where
More informationLeast-Squares Performance of Analog Product Codes
Copyright 004 IEEE Published in the Proceedings of the Asilomar Conference on Signals, Systems and Computers, 7-0 ovember 004, Pacific Grove, California, USA Least-Squares Performance of Analog Product
More informationUncertainity, Information, and Entropy
Uncertainity, Information, and Entropy Probabilistic experiment involves the observation of the output emitted by a discrete source during every unit of time. The source output is modeled as a discrete
More informationCHAPTER 3. P (B j A i ) P (B j ) =log 2. j=1
CHAPTER 3 Problem 3. : Also : Hence : I(B j ; A i ) = log P (B j A i ) P (B j ) 4 P (B j )= P (B j,a i )= i= 3 P (A i )= P (B j,a i )= j= =log P (B j,a i ) P (B j )P (A i ).3, j=.7, j=.4, j=3.3, i=.7,
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2011, 28 November 2011 Memoryless Sources Arithmetic Coding Sources with Memory 2 / 19 Summary of last lecture Prefix-free
More informationTTIC 31230, Fundamentals of Deep Learning David McAllester, April Information Theory and Distribution Modeling
TTIC 31230, Fundamentals of Deep Learning David McAllester, April 2017 Information Theory and Distribution Modeling Why do we model distributions and conditional distributions using the following objective
More informationSCALABLE AUDIO CODING USING WATERMARKING
SCALABLE AUDIO CODING USING WATERMARKING Mahmood Movassagh Peter Kabal Department of Electrical and Computer Engineering McGill University, Montreal, Canada Email: {mahmood.movassagh@mail.mcgill.ca, peter.kabal@mcgill.ca}
More informationCommunications Theory and Engineering
Communications Theory and Engineering Master's Degree in Electronic Engineering Sapienza University of Rome A.A. 2018-2019 AEP Asymptotic Equipartition Property AEP In information theory, the analog of
More informationImage Compression. Qiaoyong Zhong. November 19, CAS-MPG Partner Institute for Computational Biology (PICB)
Image Compression Qiaoyong Zhong CAS-MPG Partner Institute for Computational Biology (PICB) November 19, 2012 1 / 53 Image Compression The art and science of reducing the amount of data required to represent
More information