U Logo Use Guidelines
|
|
|
- Zoe Murphy
- 5 years ago
- Views:
Transcription
1 COMP2610/ Information Theory Lecture 15: Arithmetic Coding U Logo Use Guidelines Mark Reid and Aditya Menon logo is a contemporary n of our heritage. presents our name, d and our motto: arn the nature of things. authenticity of our brand identity, there are n how our logo is used. go - horizontal logo go should be used on a white background. ludes black text with the crest in Deep Gold in MYK. Research School of Computer Science The Australian National University Preferred logo September 23rd, 2014 Black version inting is not available, the black logo can hite background. Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
2 1 The Trouble with Huffman Coding 2 Guessing Game 3 Interval Coding Shannon-Fano-Elias Coding Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
3 Huffman Coding: Advantages and Disadvantages Advantages: Huffman Codes are provably optimal Algorithm is simple and efficient Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
4 Huffman Coding: Advantages and Disadvantages Advantages: Huffman Codes are provably optimal Algorithm is simple and efficient Disadvantages: Assumes a fixed distribution of symbols The extra bit in the SCT If H(X ) is large not a problem If H(X ) is small (e.g., 1 bit for English) codes are 2 optimal Huffman codes are the best possible symbol code but symbol coding is not always the best type of code Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
5 1 The Trouble with Huffman Coding 2 Guessing Game 3 Interval Coding Shannon-Fano-Elias Coding Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
6 A Guessing Game Let s play! Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
7 An Idealised Guessing Game Encoding: Given message x and guesser G: For i from 1 to x : 1 Set count n i = 1 2 While G guesses x i incorrectly: 1 n i n i Output n i Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
8 An Idealised Guessing Game Encoding: Given message x and guesser G: For i from 1 to x : 1 Set count n i = 1 2 While G guesses x i incorrectly: 1 n i n i Output n i Decoding: Given counts n and guesser G: For i from 1 to n and guesser G: 1 While G has made fewer than n i guesses: 1 x i next guess from G 2 Output x i Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
9 An Idealised Guesser If the guesser G is deterministic (i.e., its next output depends only on the history of values it has seen), the same guesser can be used to encode and decode. Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
10 An Idealised Guesser If the guesser G is deterministic (i.e., its next output depends only on the history of values it has seen), the same guesser can be used to encode and decode. Advantages Scheme works regardless of how the sequence x was generated Only need to compute codes for observed sequence (not all possible blocks of some size) Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
11 An Idealised Guesser If the guesser G is deterministic (i.e., its next output depends only on the history of values it has seen), the same guesser can be used to encode and decode. Advantages Scheme works regardless of how the sequence x was generated Only need to compute codes for observed sequence (not all possible blocks of some size) How can we use this? Have the guesser guess distributions rather than symbols (repeatedly) Use bits instead of counts (with fewer bits for lower counts) Define a clever guesser (one that learns ) Prove that this scheme is close to optimal (at least, not much worse than Huffman on probabilistic sources) Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
12 1 The Trouble with Huffman Coding 2 Guessing Game 3 Interval Coding Shannon-Fano-Elias Coding Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
13 Real Numbers in Binary Real numbers are commonly expressed in decimal: Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
14 Real Numbers in Binary Real numbers are commonly expressed in decimal: Some real numbers have infinite, repeating decimal expansions: 1 3 = = and 7 = = Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
15 Real Numbers in Binary Real numbers are commonly expressed in decimal: Some real numbers have infinite, repeating decimal expansions: 1 3 = = and 7 = = Real numbers can also be similarly expressed in binary: = = and 7 = = Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
16 Real Numbers in Binary Real numbers are commonly expressed in decimal: Some real numbers have infinite, repeating decimal expansions: 1 3 = = and 7 = = Real numbers can also be similarly expressed in binary: = = and 7 = = Question: 1) What is in decimal? 2) What is 7 5 in binary? Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
17 Intervals in Binary An interval [a, b) is the set of all the numbers at least as big as a but smaller than b. That is, [a, b) = {x : a x < b}. Examples: [0, 1), [0.3, 0.6), [0.2, 0.4). Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
18 Intervals in Binary An interval [a, b) is the set of all the numbers at least as big as a but smaller than b. That is, [a, b) = {x : a x < b}. Examples: [0, 1), [0.3, 0.6), [0.2, 0.4). The set of numbers in [0, 1) that start with a given sequence of bits b = b 1... b n form the interval [0.b 1... b n, 0.(b 1... b n + 1)). 1 [0.1, 1.0) [0.5, 1] [0.01, 0.10) [0.25, 0.5) [0.1101, ) [0.8125, 0.875) 10 Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
19 Intervals in Binary An interval [a, b) is the set of all the numbers at least as big as a but smaller than b. That is, [a, b) = {x : a x < b}. Examples: [0, 1), [0.3, 0.6), [0.2, 0.4). The set of numbers in [0, 1) that start with a given sequence of bits b = b 1... b n form the interval [0.b 1... b n, 0.(b 1... b n + 1)). 1 [0.1, 1.0) [0.5, 1] [0.01, 0.10) [0.25, 0.5) [0.1101, ) [0.8125, 0.875) 10 If b is a prefix of b the interval for b is contained in the interval for b. b = 01 is prefix of b = 0101 so [0.0101, ) [0.01, 0.10) }{{}}{{} [0.3125,0.375) 10 [0.25,0.5) 10 Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
20 Cumulative Distribution Suppose p = (p 1,..., p K ) is a distribution over A = {a 1,..., a K }. Assume the alphabet A is ordered and define a i a j to mean i j. Consider F the cumulative distribution for p: F (a) = P(x a) = a i a p i Each symbol a i A is associated with the interval [F (a i 1 ), F (a i )). (The value of F (a 0 ) = 0). Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
21 Cumulative Distribution Suppose p = (p 1,..., p K ) is a distribution over A = {a 1,..., a K }. Assume the alphabet A is ordered and define a i a j to mean i j. Consider F the cumulative distribution for p: F (a) = P(x a) = a i a p i Each symbol a i A is associated with the interval [F (a i 1 ), F (a i )). (The value of F (a 0 ) = 0). Example: A = {r, g, b} = a 1 = r, a 2 = g, a 3 = b = r b since 1 3. If p = ( 1 2, 1 4, 1 4 ) then F (r) = 1 2, F (g) = 3 4, F (b) = 1. Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
22 Cumulative Distribution Example 1 a F(a i ) F(a i )-½p i a F(a i-1 ) a 2 a a 1 a 2 a 3 a 4 0 Cumulative distribution for p = ( 2 9, 1 9, 1 3, 1 3 ) Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
23 Shannon-Fano-Elias Coding 1 [0, 2 9 ) 10 [0, ) 2 a [ 2 9, 1 3 ) 10 [ , 0.01) 2 F(a i ) F(a i )-½p i a [ 1 3, 2 3 ) 10 [0.01, 0.10) 2 F(a i-1 ) a 2 a [ 2 3, 1) 10 [0.10, 1) 2 a 1 a 2 a 3 a 4 0 Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
24 Shannon-Fano-Elias Coding 1 [0, 2 9 ) 10 [0, ) 2 a [ 2 9, 1 3 ) 10 [ , 0.01) 2 F(a i ) F(a i )-½p i a [ 1 3, 2 3 ) 10 [0.01, 0.10) 2 F(a i-1 ) a a [ 2 3, 1) 10 [0.10, 1) 2 a 1 a 2 a 3 a 0 4 Define the midpoint F (a i ) = F (a i ) 1 2 p 1 i and length l(a i ) = log 2 p i + 1. Code x A using first l(x) bits of F (x). x p(x) F (x) F (x) F (x) 2 l(x) Code a 1 2/9 2/9 1/ a 2 1/9 1/3 5/ a 3 1/3 2/3 1/ a 4 1/3 1 5/ Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
25 Shannon-Fano-Elias Coding 1 [0, 2 9 ) 10 [0, ) 2 a [ 2 9, 1 3 ) 10 [ , 0.01) 2 F(a i ) F(a i )-½p i a [ 1 3, 2 3 ) 10 [0.01, 0.10) 2 F(a i-1 ) a a [ 2 3, 1) 10 [0.10, 1) 2 a 1 a 2 a 3 a 0 4 Define the midpoint F (a i ) = F (a i ) 1 2 p 1 i and length l(a i ) = log 2 p i + 1. Code x A using first l(x) bits of F (x). x p(x) F (x) F (x) F (x) 2 l(x) Code a 1 2/9 2/9 1/ a 2 1/9 1/3 5/ a 3 1/3 2/3 1/ a 4 1/3 1 5/ Example: Sequence x = a 3 a 3 a 1 coded as Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
26 Shannon-Fano-Elias Decoding Let p = { 2 9, 1 9, 1 3, 1 3 }. Suppose we want to decode 01000: Find symbol whose interval contains interval for a 4 a a a 1 0 [0,0.5) [0.25,0.5) [0.25,0.375) [0.25,0.3125) [0.25,0.2815) Note: We did not need to know all the codewords in C. Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
27 Expected Code Length of SFE Code The extra bit for the code lengths is because we code p i 2 and log 2 2 p i = log 2 1 p i + log 2 2 = log 2 1 p i + 1 What is the expected length of a SFE code C for ensemble X with probabilities p? L(C, X ) = K p i l(a i ) = i=1 K ( ) 1 p i log p i K ( ) 1 p i log p i i=1 i=1 = H(X ) + 2 Similarly, H(X ) + 1 L(C, X ) for the SFE codes. Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
28 What does the extra bit buy us? Let X be an ensemble, C SFE be a Shannon-Fano-Elias code for X and C H be a Huffman code for X. H(X ) L(C H, X ) H(X ) + 1 L(C }{{} SFE, X ) H(X ) + 2 Source Coding Theorem Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
29 What does the extra bit buy us? Let X be an ensemble, C SFE be a Shannon-Fano-Elias code for X and C H be a Huffman code for X. H(X ) L(C H, X ) H(X ) + 1 L(C }{{} SFE, X ) H(X ) + 2 Source Coding Theorem The extra bit guarantees that the interval for each code word lies entirely within the interval for each symbol. [Why?] Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
30 What does the extra bit buy us? Let X be an ensemble, C SFE be a Shannon-Fano-Elias code for X and C H be a Huffman code for X. H(X ) L(C H, X ) H(X ) + 1 L(C }{{} SFE, X ) H(X ) + 2 Source Coding Theorem The extra bit guarantees that the interval for each code word lies entirely within the interval for each symbol. [Why?] Example: SFE Code for p = { 2 9, 1 9, 1 3, 1 3 } is C = {0001, 01000, 100, 110}. Intervals for p are [0, 0.22), [0.22, 0.33), [0.33, 0.66), [0.66, 1) Intervals for {000, 0100, 10, 11} are [0, 0.125), [0.25, ), [0.5, 0.75), [0.75, 1) Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
31 Summary and Reading Main points: Problems with Huffman coding Guessing game for coding without assuming fixed symbol distribution Binary strings to/from intervals in [0, 1] Shannon-Fano-Elias Coding: Code C via cumulative distribution function for p H(X ) + 1 L(C, X ) H(X ) + 2 Extra bit guarantees interval containment Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
32 Summary and Reading Main points: Problems with Huffman coding Guessing game for coding without assuming fixed symbol distribution Binary strings to/from intervals in [0, 1] Shannon-Fano-Elias Coding: Code C via cumulative distribution function for p H(X ) + 1 L(C, X ) H(X ) + 2 Extra bit guarantees interval containment Reading: Guessing game and interval coding: MacKay 6.1 and 6.2 Shannon-Fano-Elias Coding: Cover & Thomas 5.9 Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
33 Summary and Reading Main points: Problems with Huffman coding Guessing game for coding without assuming fixed symbol distribution Binary strings to/from intervals in [0, 1] Shannon-Fano-Elias Coding: Code C via cumulative distribution function for p H(X ) + 1 L(C, X ) H(X ) + 2 Extra bit guarantees interval containment Reading: Guessing game and interval coding: MacKay 6.1 and 6.2 Shannon-Fano-Elias Coding: Cover & Thomas 5.9 Next time: Extending SFE Coding to sequences of symbols Mark Reid and Aditya Menon (ANU) COMP2610/ Information Theory September 23rd, / 17
U Logo Use Guidelines
COMP2610/6261 - Information Theory Lecture 22: Hamming Codes U Logo Use Guidelines Mark Reid and Aditya Menon logo is a contemporary n of our heritage. presents our name, d and our motto: arn the nature
U Logo Use Guidelines
Information Theory Lecture 3: Applications to Machine Learning U Logo Use Guidelines Mark Reid logo is a contemporary n of our heritage. presents our name, d and our motto: arn the nature of things. authenticity
Chapter 3 Source Coding. 3.1 An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code
Chapter 3 Source Coding 3. An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code 3. An Introduction to Source Coding Entropy (in bits per symbol) implies in average
Entropy as a measure of surprise
Entropy as a measure of surprise Lecture 5: Sam Roweis September 26, 25 What does information do? It removes uncertainty. Information Conveyed = Uncertainty Removed = Surprise Yielded. How should we quantify
SIGNAL COMPRESSION Lecture Shannon-Fano-Elias Codes and Arithmetic Coding
SIGNAL COMPRESSION Lecture 3 4.9.2007 Shannon-Fano-Elias Codes and Arithmetic Coding 1 Shannon-Fano-Elias Coding We discuss how to encode the symbols {a 1, a 2,..., a m }, knowing their probabilities,
3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2011, 28 November 2011 Memoryless Sources Arithmetic Coding Sources with Memory 2 / 19 Summary of last lecture Prefix-free
3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output
Lecture 16. Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code
Lecture 16 Agenda for the lecture Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code Variable-length source codes with error 16.1 Error-free coding schemes 16.1.1 The Shannon-Fano-Elias
Chapter 5: Data Compression
Chapter 5: Data Compression Definition. A source code C for a random variable X is a mapping from the range of X to the set of finite length strings of symbols from a D-ary alphabet. ˆX: source alphabet,
EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018
Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code
Motivation for Arithmetic Coding
Motivation for Arithmetic Coding Motivations for arithmetic coding: 1) Huffman coding algorithm can generate prefix codes with a minimum average codeword length. But this length is usually strictly greater
Shannon-Fano-Elias coding
Shannon-Fano-Elias coding Suppose that we have a memoryless source X t taking values in the alphabet {1, 2,..., L}. Suppose that the probabilities for all symbols are strictly positive: p(i) > 0, i. The
Chapter 2 Date Compression: Source Coding. 2.1 An Introduction to Source Coding 2.2 Optimal Source Codes 2.3 Huffman Code
Chapter 2 Date Compression: Source Coding 2.1 An Introduction to Source Coding 2.2 Optimal Source Codes 2.3 Huffman Code 2.1 An Introduction to Source Coding Source coding can be seen as an efficient way
Communications Theory and Engineering
Communications Theory and Engineering Master's Degree in Electronic Engineering Sapienza University of Rome A.A. 2018-2019 AEP Asymptotic Equipartition Property AEP In information theory, the analog of
Lecture 3 : Algorithms for source coding. September 30, 2016
Lecture 3 : Algorithms for source coding September 30, 2016 Outline 1. Huffman code ; proof of optimality ; 2. Coding with intervals : Shannon-Fano-Elias code and Shannon code ; 3. Arithmetic coding. 1/39
Source Coding Techniques
Source Coding Techniques. Huffman Code. 2. Two-pass Huffman Code. 3. Lemple-Ziv Code. 4. Fano code. 5. Shannon Code. 6. Arithmetic Code. Source Coding Techniques. Huffman Code. 2. Two-path Huffman Code.
Data Compression Techniques (Spring 2012) Model Solutions for Exercise 2
582487 Data Compression Techniques (Spring 22) Model Solutions for Exercise 2 If you have any feedback or corrections, please contact nvalimak at cs.helsinki.fi.. Problem: Construct a canonical prefix
DCSP-3: Minimal Length Coding. Jianfeng Feng
DCSP-3: Minimal Length Coding Jianfeng Feng Department of Computer Science Warwick Univ., UK Jianfeng.feng@warwick.ac.uk http://www.dcs.warwick.ac.uk/~feng/dcsp.html Automatic Image Caption (better than
Information Theory. Week 4 Compressing streams. Iain Murray,
Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 4 Compressing streams Iain Murray, 2014 School of Informatics, University of Edinburgh Jensen s inequality For convex functions: E[f(x)]
Data Compression. Limit of Information Compression. October, Examples of codes 1
Data Compression Limit of Information Compression Radu Trîmbiţaş October, 202 Outline Contents Eamples of codes 2 Kraft Inequality 4 2. Kraft Inequality............................ 4 2.2 Kraft inequality
Stream Codes. 6.1 The guessing game
About Chapter 6 Before reading Chapter 6, you should have read the previous chapter and worked on most of the exercises in it. We ll also make use of some Bayesian modelling ideas that arrived in the vicinity
Source Coding. Master Universitario en Ingeniería de Telecomunicación. I. Santamaría Universidad de Cantabria
Source Coding Master Universitario en Ingeniería de Telecomunicación I. Santamaría Universidad de Cantabria Contents Introduction Asymptotic Equipartition Property Optimal Codes (Huffman Coding) Universal
Bandwidth: Communicate large complex & highly detailed 3D models through lowbandwidth connection (e.g. VRML over the Internet)
Compression Motivation Bandwidth: Communicate large complex & highly detailed 3D models through lowbandwidth connection (e.g. VRML over the Internet) Storage: Store large & complex 3D models (e.g. 3D scanner
Basic Principles of Lossless Coding. Universal Lossless coding. Lempel-Ziv Coding. 2. Exploit dependences between successive symbols.
Universal Lossless coding Lempel-Ziv Coding Basic principles of lossless compression Historical review Variable-length-to-block coding Lempel-Ziv coding 1 Basic Principles of Lossless Coding 1. Exploit
Chapter 2: Source coding
Chapter 2: meghdadi@ensil.unilim.fr University of Limoges Chapter 2: Entropy of Markov Source Chapter 2: Entropy of Markov Source Markov model for information sources Given the present, the future is independent
Data Compression Techniques
Data Compression Techniques Part 1: Entropy Coding Lecture 4: Asymmetric Numeral Systems Juha Kärkkäinen 08.11.2017 1 / 19 Asymmetric Numeral Systems Asymmetric numeral systems (ANS) is a recent entropy
Run-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE
General e Image Coder Structure Motion Video x(s 1,s 2,t) or x(s 1,s 2 ) Natural Image Sampling A form of data compression; usually lossless, but can be lossy Redundancy Removal Lossless compression: predictive
Information Theory. David Rosenberg. June 15, New York University. David Rosenberg (New York University) DS-GA 1003 June 15, / 18
Information Theory David Rosenberg New York University June 15, 2015 David Rosenberg (New York University) DS-GA 1003 June 15, 2015 1 / 18 A Measure of Information? Consider a discrete random variable
Lecture 1: Shannon s Theorem
Lecture 1: Shannon s Theorem Lecturer: Travis Gagie January 13th, 2015 Welcome to Data Compression! I m Travis and I ll be your instructor this week. If you haven t registered yet, don t worry, we ll work
Information Theory. Coding and Information Theory. Information Theory Textbooks. Entropy
Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is
Information Theory and Statistics Lecture 2: Source coding
Information Theory and Statistics Lecture 2: Source coding Łukasz Dębowski ldebowsk@ipipan.waw.pl Ph. D. Programme 2013/2014 Injections and codes Definition (injection) Function f is called an injection
4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information
4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information Ramji Venkataramanan Signal Processing and Communications Lab Department of Engineering ramji.v@eng.cam.ac.uk
Summary of Last Lectures
Lossless Coding IV a k p k b k a 0.16 111 b 0.04 0001 c 0.04 0000 d 0.16 110 e 0.23 01 f 0.07 1001 g 0.06 1000 h 0.09 001 i 0.15 101 100 root 1 60 1 0 0 1 40 0 32 28 23 e 17 1 0 1 0 1 0 16 a 16 d 15 i
Using an innovative coding algorithm for data encryption
Using an innovative coding algorithm for data encryption Xiaoyu Ruan and Rajendra S. Katti Abstract This paper discusses the problem of using data compression for encryption. We first propose an algorithm
Kolmogorov complexity ; induction, prediction and compression
Kolmogorov complexity ; induction, prediction and compression Contents 1 Motivation for Kolmogorov complexity 1 2 Formal Definition 2 3 Trying to compute Kolmogorov complexity 3 4 Standard upper bounds
10-704: Information Processing and Learning Fall Lecture 9: Sept 28
10-704: Information Processing and Learning Fall 2016 Lecturer: Siheng Chen Lecture 9: Sept 28 Note: These notes are based on scribed notes from Spring15 offering of this course. LaTeX template courtesy
PART III. Outline. Codes and Cryptography. Sources. Optimal Codes (I) Jorge L. Villar. MAMME, Fall 2015
Outline Codes and Cryptography 1 Information Sources and Optimal Codes 2 Building Optimal Codes: Huffman Codes MAMME, Fall 2015 3 Shannon Entropy and Mutual Information PART III Sources Information source:
Fibonacci Coding for Lossless Data Compression A Review
RESEARCH ARTICLE OPEN ACCESS Fibonacci Coding for Lossless Data Compression A Review Ezhilarasu P Associate Professor Department of Computer Science and Engineering Hindusthan College of Engineering and
Lecture 4 : Adaptive source coding algorithms
Lecture 4 : Adaptive source coding algorithms February 2, 28 Information Theory Outline 1. Motivation ; 2. adaptive Huffman encoding ; 3. Gallager and Knuth s method ; 4. Dictionary methods : Lempel-Ziv
ELEC 515 Information Theory. Distortionless Source Coding
ELEC 515 Information Theory Distortionless Source Coding 1 Source Coding Output Alphabet Y={y 1,,y J } Source Encoder Lengths 2 Source Coding Two coding requirements The source sequence can be recovered
ECE Advanced Communication Theory, Spring 2009 Homework #1 (INCOMPLETE)
ECE 74 - Advanced Communication Theory, Spring 2009 Homework #1 (INCOMPLETE) 1. A Huffman code finds the optimal codeword to assign to a given block of source symbols. (a) Show that cannot be a Huffman
Lecture 3. Mathematical methods in communication I. REMINDER. A. Convex Set. A set R is a convex set iff, x 1,x 2 R, θ, 0 θ 1, θx 1 + θx 2 R, (1)
3- Mathematical methods in communication Lecture 3 Lecturer: Haim Permuter Scribe: Yuval Carmel, Dima Khaykin, Ziv Goldfeld I. REMINDER A. Convex Set A set R is a convex set iff, x,x 2 R, θ, θ, θx + θx
10-704: Information Processing and Learning Fall Lecture 10: Oct 3
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 0: Oct 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy of
COMP2610/COMP Information Theory
COMP2610/COMP6261 - Information Theory Lecture 9: Probabilistic Inequalities Mark Reid and Aditya Menon Research School of Computer Science The Australian National University August 19th, 2014 Mark Reid
Information and Entropy
Information and Entropy Shannon s Separation Principle Source Coding Principles Entropy Variable Length Codes Huffman Codes Joint Sources Arithmetic Codes Adaptive Codes Thomas Wiegand: Digital Image Communication
Information & Correlation
Information & Correlation Jilles Vreeken 11 June 2014 (TADA) Questions of the day What is information? How can we measure correlation? and what do talking drums have to do with this? Bits and Pieces What
Lecture 1: September 25, A quick reminder about random variables and convexity
Information and Coding Theory Autumn 207 Lecturer: Madhur Tulsiani Lecture : September 25, 207 Administrivia This course will cover some basic concepts in information and coding theory, and their applications
CSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H.
Problem sheet Ex. Verify that the function H(p,..., p n ) = k p k log p k satisfies all 8 axioms on H. Ex. (Not to be handed in). looking at the notes). List as many of the 8 axioms as you can, (without
EE5585 Data Compression January 29, Lecture 3. x X x X. 2 l(x) 1 (1)
EE5585 Data Compression January 29, 2013 Lecture 3 Instructor: Arya Mazumdar Scribe: Katie Moenkhaus Uniquely Decodable Codes Recall that for a uniquely decodable code with source set X, if l(x) is the
Lecture 22: Final Review
Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information
Lec 05 Arithmetic Coding
ECE 5578 Multimedia Communication Lec 05 Arithmetic Coding Zhu Li Dept of CSEE, UMKC web: http://l.web.umkc.edu/lizhu phone: x2346 Z. Li, Multimedia Communciation, 208 p. Outline Lecture 04 ReCap Arithmetic
EE5139R: Problem Set 4 Assigned: 31/08/16, Due: 07/09/16
EE539R: Problem Set 4 Assigned: 3/08/6, Due: 07/09/6. Cover and Thomas: Problem 3.5 Sets defined by probabilities: Define the set C n (t = {x n : P X n(x n 2 nt } (a We have = P X n(x n P X n(x n 2 nt
Chapter 2 Source Models and Entropy. Any information-generating process can be viewed as. computer program in executed form: binary 0
Part II Information Theory Concepts Chapter 2 Source Models and Entropy Any information-generating process can be viewed as a source: { emitting a sequence of symbols { symbols from a nite alphabet text:
UNIT I INFORMATION THEORY. I k log 2
UNIT I INFORMATION THEORY Claude Shannon 1916-2001 Creator of Information Theory, lays the foundation for implementing logic in digital circuits as part of his Masters Thesis! (1939) and published a paper
Multimedia. Multimedia Data Compression (Lossless Compression Algorithms)
Course Code 005636 (Fall 2017) Multimedia Multimedia Data Compression (Lossless Compression Algorithms) Prof. S. M. Riazul Islam, Dept. of Computer Engineering, Sejong University, Korea E-mail: riaz@sejong.ac.kr
Data Compression Techniques
Data Compression Techniques Part 2: Text Compression Lecture 5: Context-Based Compression Juha Kärkkäinen 14.11.2017 1 / 19 Text Compression We will now look at techniques for text compression. These techniques
lossless, optimal compressor
6. Variable-length Lossless Compression The principal engineering goal of compression is to represent a given sequence a, a 2,..., a n produced by a source as a sequence of bits of minimal possible length.
ECE 587 / STA 563: Lecture 5 Lossless Compression
ECE 587 / STA 563: Lecture 5 Lossless Compression Information Theory Duke University, Fall 2017 Author: Galen Reeves Last Modified: October 18, 2017 Outline of lecture: 5.1 Introduction to Lossless Source
EE376A - Information Theory Midterm, Tuesday February 10th. Please start answering each question on a new page of the answer booklet.
EE376A - Information Theory Midterm, Tuesday February 10th Instructions: You have two hours, 7PM - 9PM The exam has 3 questions, totaling 100 points. Please start answering each question on a new page
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 13 Competitive Optimality of the Shannon Code So, far we have studied
( c ) E p s t e i n, C a r t e r a n d B o l l i n g e r C h a p t e r 1 7 : I n f o r m a t i o n S c i e n c e P a g e 1
( c ) E p s t e i n, C a r t e r a n d B o l l i n g e r 2 0 1 6 C h a p t e r 1 7 : I n f o r m a t i o n S c i e n c e P a g e 1 CHAPTER 17: Information Science In this chapter, we learn how data can
ECE 587 / STA 563: Lecture 5 Lossless Compression
ECE 587 / STA 563: Lecture 5 Lossless Compression Information Theory Duke University, Fall 28 Author: Galen Reeves Last Modified: September 27, 28 Outline of lecture: 5. Introduction to Lossless Source
1 Introduction to information theory
1 Introduction to information theory 1.1 Introduction In this chapter we present some of the basic concepts of information theory. The situations we have in mind involve the exchange of information through
EE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018
Please submit the solutions on Gradescope. Some definitions that may be useful: EE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018 Definition 1: A sequence of random variables X
Lecture 1 : Data Compression and Entropy
CPS290: Algorithmic Foundations of Data Science January 8, 207 Lecture : Data Compression and Entropy Lecturer: Kamesh Munagala Scribe: Kamesh Munagala In this lecture, we will study a simple model for
Exercise 1. = P(y a 1)P(a 1 )
Chapter 7 Channel Capacity Exercise 1 A source produces independent, equally probable symbols from an alphabet {a 1, a 2 } at a rate of one symbol every 3 seconds. These symbols are transmitted over a
Chapter 5. Data Compression
Chapter 5 Data Compression Peng-Hua Wang Graduate Inst. of Comm. Engineering National Taipei University Chapter Outline Chap. 5 Data Compression 5.1 Example of Codes 5.2 Kraft Inequality 5.3 Optimal Codes
CS4800: Algorithms & Data Jonathan Ullman
CS4800: Algorithms & Data Jonathan Ullman Lecture 22: Greedy Algorithms: Huffman Codes Data Compression and Entropy Apr 5, 2018 Data Compression How do we store strings of text compactly? A (binary) code
(Classical) Information Theory II: Source coding
(Classical) Information Theory II: Source coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract The information content of a random variable
Intro to Information Theory
Intro to Information Theory Math Circle February 11, 2018 1. Random variables Let us review discrete random variables and some notation. A random variable X takes value a A with probability P (a) 0. Here
Entropy Coding. Connectivity coding. Entropy coding. Definitions. Lossles coder. Input: a set of symbols Output: bitstream. Idea
Connectivity coding Entropy Coding dd 7, dd 6, dd 7, dd 5,... TG output... CRRRLSLECRRE Entropy coder output Connectivity data Edgebreaker output Digital Geometry Processing - Spring 8, Technion Digital
Homework Set #2 Data Compression, Huffman code and AEP
Homework Set #2 Data Compression, Huffman code and AEP 1. Huffman coding. Consider the random variable ( x1 x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0.11 0.04 0.04 0.03 0.02 (a Find a binary Huffman code
EECS 229A Spring 2007 * * (a) By stationarity and the chain rule for entropy, we have
EECS 229A Spring 2007 * * Solutions to Homework 3 1. Problem 4.11 on pg. 93 of the text. Stationary processes (a) By stationarity and the chain rule for entropy, we have H(X 0 ) + H(X n X 0 ) = H(X 0,
1 Computational Problems
Stanford University CS254: Computational Complexity Handout 2 Luca Trevisan March 31, 2010 Last revised 4/29/2010 In this lecture we define NP, we state the P versus NP problem, we prove that its formulation
TTIC 31230, Fundamentals of Deep Learning David McAllester, April Information Theory and Distribution Modeling
TTIC 31230, Fundamentals of Deep Learning David McAllester, April 2017 Information Theory and Distribution Modeling Why do we model distributions and conditional distributions using the following objective
Shannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 13, 2015 Lucas Slot, Sebastian Zur Shannon s Noisy-Channel Coding Theorem February 13, 2015 1 / 29 Outline 1 Definitions and Terminology
CSEP 590 Data Compression Autumn Arithmetic Coding
CSEP 590 Data Compression Autumn 2007 Arithmetic Coding Reals in Binary Any real number x in the interval [0,1) can be represented in binary as.b 1 b 2... where b i is a bit. x 0 0 1 0 1... binary representation
Introduction to information theory and coding
Introduction to information theory and coding Louis WEHENKEL Set of slides No 5 State of the art in data compression Stochastic processes and models for information sources First Shannon theorem : data
Shannon s noisy-channel theorem
Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 2014 1 / 8 I
Notes for Lecture Notes 2
Stanford University CS254: Computational Complexity Notes 2 Luca Trevisan January 11, 2012 Notes for Lecture Notes 2 In this lecture we define NP, we state the P versus NP problem, we prove that its formulation
Lecture 5: Asymptotic Equipartition Property
Lecture 5: Asymptotic Equipartition Property Law of large number for product of random variables AEP and consequences Dr. Yao Xie, ECE587, Information Theory, Duke University Stock market Initial investment
SIGNAL COMPRESSION Lecture 7. Variable to Fix Encoding
SIGNAL COMPRESSION Lecture 7 Variable to Fix Encoding 1. Tunstall codes 2. Petry codes 3. Generalized Tunstall codes for Markov sources (a presentation of the paper by I. Tabus, G. Korodi, J. Rissanen.
Lecture 11: Continuous-valued signals and differential entropy
Lecture 11: Continuous-valued signals and differential entropy Biology 429 Carl Bergstrom September 20, 2008 Sources: Parts of today s lecture follow Chapter 8 from Cover and Thomas (2007). Some components
Sources: The Data Compression Book, 2 nd Ed., Mark Nelson and Jean-Loup Gailly.
Lossless ompression Multimedia Systems (Module 2 Lesson 2) Summary: daptive oding daptive Huffman oding Sibling Property Update lgorithm rithmetic oding oding and ecoding Issues: OF problem, Zero frequency
Chapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
17.1 Binary Codes Normal numbers we use are in base 10, which are called decimal numbers. Each digit can be 10 possible numbers: 0, 1, 2, 9.
( c ) E p s t e i n, C a r t e r, B o l l i n g e r, A u r i s p a C h a p t e r 17: I n f o r m a t i o n S c i e n c e P a g e 1 CHAPTER 17: Information Science 17.1 Binary Codes Normal numbers we use
Coding of memoryless sources 1/35
Coding of memoryless sources 1/35 Outline 1. Morse coding ; 2. Definitions : encoding, encoding efficiency ; 3. fixed length codes, encoding integers ; 4. prefix condition ; 5. Kraft and Mac Millan theorems
ECS 452: Digital Communication Systems 2015/2. HW 1 Due: Feb 5
ECS 452: Digital Communication Systems 2015/2 HW 1 Due: Feb 5 Lecturer: Asst. Prof. Dr. Prapun Suksompong Instructions (a) Must solve all non-optional problems. (5 pt) (i) Write your first name and the
Lecture 4 Channel Coding
Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity
LECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem
LECTURE 15 Last time: Feedback channel: setting up the problem Perfect feedback Feedback capacity Data compression Lecture outline Joint source and channel coding theorem Converse Robustness Brain teaser
COMM901 Source Coding and Compression. Quiz 1
German University in Cairo - GUC Faculty of Information Engineering & Technology - IET Department of Communication Engineering Winter Semester 2013/2014 Students Name: Students ID: COMM901 Source Coding
Compressing Tabular Data via Pairwise Dependencies
Compressing Tabular Data via Pairwise Dependencies Amir Ingber, Yahoo! Research TCE Conference, June 22, 2017 Joint work with Dmitri Pavlichin, Tsachy Weissman (Stanford) Huge datasets: everywhere - Internet
2018/5/3. YU Xiangyu
2018/5/3 YU Xiangyu yuxy@scut.edu.cn Entropy Huffman Code Entropy of Discrete Source Definition of entropy: If an information source X can generate n different messages x 1, x 2,, x i,, x n, then the
Entropy and Ergodic Theory Lecture 3: The meaning of entropy in information theory
Entropy and Ergodic Theory Lecture 3: The meaning of entropy in information theory 1 The intuitive meaning of entropy Modern information theory was born in Shannon s 1948 paper A Mathematical Theory of
Exercises with solutions (Set B)
Exercises with solutions (Set B) 3. A fair coin is tossed an infinite number of times. Let Y n be a random variable, with n Z, that describes the outcome of the n-th coin toss. If the outcome of the n-th
CISC 876: Kolmogorov Complexity
March 27, 2007 Outline 1 Introduction 2 Definition Incompressibility and Randomness 3 Prefix Complexity Resource-Bounded K-Complexity 4 Incompressibility Method Gödel s Incompleteness Theorem 5 Outline
6.02 Fall 2012 Lecture #1
6.02 Fall 2012 Lecture #1 Digital vs. analog communication The birth of modern digital communication Information and entropy Codes, Huffman coding 6.02 Fall 2012 Lecture 1, Slide #1 6.02 Fall 2012 Lecture
Information and Entropy. Professor Kevin Gold
Information and Entropy Professor Kevin Gold What s Information? Informally, when I communicate a message to you, that s information. Your grade is 100/100 Information can be encoded as a signal. Words