Ch. 2 Math Preliminaries for Lossless Compression. Section 2.4 Coding
|
|
- Dora Hubbard
- 5 years ago
- Views:
Transcription
1 Ch. 2 Math Preliminaries for Lossless Compression Section 2.4 Coding
2 Some General Considerations Definition: An Instantaneous Code maps each symbol into a codeord Notation: a i φ (a i ) Ex. : a 0 For Ex. : φ (a a 2 3 ) = 00 a 3 00 a 4 Ex. 2: a 0 a 2 0 a 3 0 a 4 This code has a tree structure: a a 2 a 3 a 4 2
3 What characteristics must a code φ have? Unambiguous (UA): For a i a j, φ(a i ) φ(a j ) The codes in Ex. and Ex.2 each are UA Is UA enough?? No! Consider Ex. coding to different source sequences: a a 2 a a a 2 a 2 a a 2 a 3 a 4 a a 2 a a a 2 a 2 They each get coded to the bit stream: a a 2 a 3 a 4 Can t uniquely decode this bit sequence!! So UA guarantees that can decode each symbol by itself but not necessarily a stream of coded symbols!! 3
4 Define mapping of sequences under code φ Φ ( a ) ( ) ( ) ( ) ( ) ( ) i a i a 2 i a 3 i a 4 i =φ a N i φ a i φ a 2 i φ a 3 i φ a 4 in S i Concatenation of code ords Don t ant to sequences of symbols to map to the same bit stream: S i Φ S j Source Sequence Space Φ Code Sequence Space Leads to need for Uniquely Decodable (UD): Let S i & S j be to sequences from the same source (not necessarily of the same length). Then code φ is UD if the only ay that Φ(S i ) = Φ(S j ) is for S i S j 4
5 Does UD UA??? YES! UA Codes UD Codes Then UD is enough??? YES! But in practice it is helpful to restrict to a subset of UD codes called Prefix Codes. 5
6 Prefix Code: A UD code in hich no codeord may be the prefix of another codeord. Ex. 2 above is a prefix code: Ex. 2: 0 a 0 0 a 2 0 a 3 0 a 4 0 a a 2 a 3 a 4 This code is not prefix Ex. 3: a 0 a 2 0 a 3 0 a 4 0 Does Prefix UD??? YES! UA Codes UD Codes Prefix Codes Do e lose anything by restricting to prefix codes? No as e ll see later! 6
7 Ho do e compare various UD codes??? (i.e., What is our measure of performance?) Average Code Length: Info theory says to use average code length per symbol For a source ith symbols a, a 2, a N and a code φ the average code length is define by N l ( φ) = P( ai) n{ φ( ai)} Prob. of a i # of bits in codeord for a i Optimum Code: The UD code ith the smallest average code length 7
8 Example: For P(a ) = ½ P(a 2 ) = ¼ P(a 3 ) = P(a 3 ) = /8 This source has a entropy of.75 bits Here are three possible codes and their average lengths: Symbol UA Non-UD Code (Ex. ) Prefix Code (Ex. 2) UD Non-Prefix (Ex. 3) Info of symbol -log 2 [P(a i )] a a a a Avg. Length:.25 bits.75 bits.875 bits H(S) =.75 bits Length better than H(S)!! Length equals H(S) Length larger than H(S) BUT not usable because it is Non-UD Prefix Code gives smallest usable code!! 8
9 Info Theory Says: Optimum Code is alays a prefix code!! UA Codes UD Codes Prefix Codes The proof of this uses the Kraft-McMillan Inequality hich e ll discuss next. Ho do e find the optimum prefix code? (Note: not just any prefix code ill be optimum!) We ll discuss this later. 9
10 2.4.3 Kraft-McMillan Inequality This result tells us that an optimal code can alays be chosen to be a prefix code!!! The Theorem has 2 parts. Theorem Part #: Let C be a code having N codeords ith codeord lengths of l, l 2, l 3,, l N If C is uniquely decodable, then For notation: N = l KC ( ) Δ 2 i N li 2 Proof: Here is the main idea used in the proof If K(C) >, then [K(C)] n gros exponentially.r.t. n So if e can sho that [K(C)] n gros, say, no more than linearly e have our proof. Thus e need to sho that [ KC ( )] n αn+ β Some constants 0
11 For arbitrary integer n: [ KC] n N n N N N l l li i i2 in ( ) = 2 = i2= in = = N N N i = i = i = 2 n 2 ( l l l ) i i2 i n l Note use of n different dummy variables! ( ) Note that this exponent is nothing more than the length of a sequence of selected codeords of code C Let this be L(i, i 2, i 3,, i n ) and e can re-rite ( ) as [ KC] N N N n L i i i L L L N N N (,,, ) (,,,) (,,,2 ) (,,, ) 2 n ( ) = 2 = i = i = i = 2 n The smallest L(i, i 2, i 3,, i n ) can be is n (hen each codeord in the sequence is bit long) The longest L(i, i 2, i 3,, i n ) can be is nl here l is the longest codeord in C. So then: [ KC] (,,,) (,,,2 ) (,,, ) n L L L N N N ( ) = = A 2 + A A 2 n ( n+ ) ( nl) n n+ nl ( ) [ ] nl n KC ( ) = Ak 2 k= n k A k = # times L(i, i 2, i 3,, i n ) = n
12 Remember that e are trying to establish this bound: [ KC ( )] n αn+ β e don t need the A k values exactly just need a good upper bound on them! There may be some in the 2 k that are not valid First: The # of k-bit binary sequences = 2 k The If part of the theorem! Second: If our code is uniquely decodable, then each of these can represent one and only one sequence of codeords hose total length = k bits A We can no use this bound in ( ) to get a bound on [K(C)] n : nl k 2 k n k k k [ ] k KC ( ) = A2 2 2 = nl n+ nl k= n k= n Thus [K(C)] n gros sloer than exponentially = Hence K(C) <End of Proof> 2
13 Part # says: If code ith lengths {l, l 2, l N } is uniquely decodable, then the lengths satisfy the inequality Part #2 says: Given lengths {l, l 2, l N } that satisfy the inequality, then e can alays find a prefix code / these lengths li Theorem Part #2: Given integers {l, l 2, l N } such that 2 We can alays find a prefix code ith lengths {l, l 2, l N } Proof: This is a Proof by Construction : e ill sho ho to construct the desired prefix code. WLOG. Assume that l l 2 l N Define the numbers, 2,, N using j = 0 j j i = 2, j > l l N Think of this in terms of a binary representation (see next slide for an example) 3
14 Example of Creating the li l = l = 3 l = 3 l = 5 l = 5 2 = < l2 li l3 li l4 li l5 li = 2 = = = 0 = = = = = = + = = = = + + = = j = For j >, the binary representation of j uses log 2 j bits Easy to sho (see textbook) that: # bits in j l j log 2 j l j, for j This is here e use that N li 2 4
15 No use the binary reps of the j to construct the prefix codeords having lengths {l, l 2,, l N } If log 2 j = l j log 2 j < l j then set j th codeord = binary j then set j th codeord = [binary j 0 0] So no e ve constructed a code ith the desired lengths Is it a prefix code??? Append enough 0 s to get l j total bits Sho it is by using contradiction Assume that it is NOT a prefix code and sho that it leads to something that contradicts a knon condition 5
16 Suppose that the constructed code is not prefix thus, for some j < k the codeord C j is a prefix of codeord C k (l j MSBs of k ) = j Right-Shift & Chop k = j k l 2 k l = j j ( ) But by design : k k = l 2 k l i There is alays a But in a proof by contradiction! So see if ( ) contradicts this required condition: Put this k into ( ) and sho that something goes rong 6
17 ( ) 2 l k k l j k = 2 l j l i Thus, k l 2 k j j j + l j + > j k l l l l = j j i j i k 0 l 2 2 j l = i j + + j+ Integer Must fall here j Integer j by Defn j + k l 2 k l + j j hich contradicts ( ) So code is prefix! <End of Proof> 7
18 Meaning of Kraft-McMillan Theorem Question: So hat do these to parts of the theorem tell us??? Anser: Shortest Avg. Length We are looking for the optimal UD code. Once e find it e kno its codeord lengths satisfy the K-M inequality Part # of the theorem tells us that!!! Once e have such lengths (that satisfy the K-M ineq.) e can construct a prefix code having those optimal lengths This is guaranteed by Part #2 of the theorem This gives us a prefix code that is optimal!!! So everytime e find the optimal code, if it isn t already prefix e can replace it ith a prefix code that is just as optimal! Can focus on finding optimal prefix codes /o orrying that e could find a better code that is not prefix! 8
An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
More informationChapter 2 Date Compression: Source Coding. 2.1 An Introduction to Source Coding 2.2 Optimal Source Codes 2.3 Huffman Code
Chapter 2 Date Compression: Source Coding 2.1 An Introduction to Source Coding 2.2 Optimal Source Codes 2.3 Huffman Code 2.1 An Introduction to Source Coding Source coding can be seen as an efficient way
More informationChapter 3 Source Coding. 3.1 An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code
Chapter 3 Source Coding 3. An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code 3. An Introduction to Source Coding Entropy (in bits per symbol) implies in average
More information10-704: Information Processing and Learning Fall Lecture 10: Oct 3
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 0: Oct 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy of
More informationMultimedia Communications. Mathematical Preliminaries for Lossless Compression
Multimedia Communications Mathematical Preliminaries for Lossless Compression What we will see in this chapter Definition of information and entropy Modeling a data source Definition of coding and when
More informationChapter 3. Systems of Linear Equations: Geometry
Chapter 3 Systems of Linear Equations: Geometry Motiation We ant to think about the algebra in linear algebra (systems of equations and their solution sets) in terms of geometry (points, lines, planes,
More informationConsider this problem. A person s utility function depends on consumption and leisure. Of his market time (the complement to leisure), h t
VI. INEQUALITY CONSTRAINED OPTIMIZATION Application of the Kuhn-Tucker conditions to inequality constrained optimization problems is another very, very important skill to your career as an economist. If
More informationEE5139R: Problem Set 4 Assigned: 31/08/16, Due: 07/09/16
EE539R: Problem Set 4 Assigned: 3/08/6, Due: 07/09/6. Cover and Thomas: Problem 3.5 Sets defined by probabilities: Define the set C n (t = {x n : P X n(x n 2 nt } (a We have = P X n(x n P X n(x n 2 nt
More informationChapter 5: Data Compression
Chapter 5: Data Compression Definition. A source code C for a random variable X is a mapping from the range of X to the set of finite length strings of symbols from a D-ary alphabet. ˆX: source alphabet,
More informationLecture 6: Kraft-McMillan Inequality and Huffman Coding
EE376A/STATS376A Information Theory Lecture 6-0/25/208 Lecture 6: Kraft-McMillan Inequality and Huffman Coding Lecturer: Tsachy Weissman Scribe: Akhil Prakash, Kai Yee Wan In this lecture, we begin with
More informationLecture 1 : Data Compression and Entropy
CPS290: Algorithmic Foundations of Data Science January 8, 207 Lecture : Data Compression and Entropy Lecturer: Kamesh Munagala Scribe: Kamesh Munagala In this lecture, we will study a simple model for
More informationUC Berkeley Department of Electrical Engineering and Computer Sciences. EECS 126: Probability and Random Processes
UC Berkeley Department of Electrical Engineering and Computer Sciences EECS 26: Probability and Random Processes Problem Set Spring 209 Self-Graded Scores Due:.59 PM, Monday, February 4, 209 Submit your
More informationCh. 8 Math Preliminaries for Lossy Coding. 8.5 Rate-Distortion Theory
Ch. 8 Math Preliminaries for Lossy Coding 8.5 Rate-Distortion Theory 1 Introduction Theory provide insight into the trade between Rate & Distortion This theory is needed to answer: What do typical R-D
More informationBandwidth: Communicate large complex & highly detailed 3D models through lowbandwidth connection (e.g. VRML over the Internet)
Compression Motivation Bandwidth: Communicate large complex & highly detailed 3D models through lowbandwidth connection (e.g. VRML over the Internet) Storage: Store large & complex 3D models (e.g. 3D scanner
More informationCOMM901 Source Coding and Compression. Quiz 1
German University in Cairo - GUC Faculty of Information Engineering & Technology - IET Department of Communication Engineering Winter Semester 2013/2014 Students Name: Students ID: COMM901 Source Coding
More informationLecture 1: Shannon s Theorem
Lecture 1: Shannon s Theorem Lecturer: Travis Gagie January 13th, 2015 Welcome to Data Compression! I m Travis and I ll be your instructor this week. If you haven t registered yet, don t worry, we ll work
More informationAnnouncements Wednesday, September 06
Announcements Wednesday, September 06 WeBWorK due on Wednesday at 11:59pm. The quiz on Friday coers through 1.2 (last eek s material). My office is Skiles 244 and my office hours are Monday, 1 3pm and
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More informationMinentropy and its Variations for Cryptography
1 Minentropy and its Variations for Cryptography Leonid Reyzin May 23, 2011 5 th International Conference on Information Theoretic Security 2 guessability and entropy Many ays to measure entropy If I ant
More informationInformation Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 13 Competitive Optimality of the Shannon Code So, far we have studied
More informationPart I consists of 14 multiple choice questions (worth 5 points each) and 5 true/false question (worth 1 point each), for a total of 75 points.
Math 131 Exam 1 Solutions Part I consists of 14 multiple choice questions (orth 5 points each) and 5 true/false question (orth 1 point each), for a total of 75 points. 1. The folloing table gives the number
More informationlossless, optimal compressor
6. Variable-length Lossless Compression The principal engineering goal of compression is to represent a given sequence a, a 2,..., a n produced by a source as a sequence of bits of minimal possible length.
More informationChapter 2: Source coding
Chapter 2: meghdadi@ensil.unilim.fr University of Limoges Chapter 2: Entropy of Markov Source Chapter 2: Entropy of Markov Source Markov model for information sources Given the present, the future is independent
More informationBasic Principles of Lossless Coding. Universal Lossless coding. Lempel-Ziv Coding. 2. Exploit dependences between successive symbols.
Universal Lossless coding Lempel-Ziv Coding Basic principles of lossless compression Historical review Variable-length-to-block coding Lempel-Ziv coding 1 Basic Principles of Lossless Coding 1. Exploit
More informationCh. 8 Math Preliminaries for Lossy Coding. 8.4 Info Theory Revisited
Ch. 8 Math Preliminaries for Lossy Coding 8.4 Info Theory Revisited 1 Info Theory Goals for Lossy Coding Again just as for the lossless case Info Theory provides: Basis for Algorithms & Bounds on Performance
More informationOptimal codes - I. A code is optimal if it has the shortest codeword length L. i i. This can be seen as an optimization problem. min.
Huffman coding Optimal codes - I A code is optimal if it has the shortest codeword length L L m = i= pl i i This can be seen as an optimization problem min i= li subject to D m m i= lp Gabriele Monfardini
More informationEntropy as a measure of surprise
Entropy as a measure of surprise Lecture 5: Sam Roweis September 26, 25 What does information do? It removes uncertainty. Information Conveyed = Uncertainty Removed = Surprise Yielded. How should we quantify
More informationA Generalization of a result of Catlin: 2-factors in line graphs
AUSTRALASIAN JOURNAL OF COMBINATORICS Volume 72(2) (2018), Pages 164 184 A Generalization of a result of Catlin: 2-factors in line graphs Ronald J. Gould Emory University Atlanta, Georgia U.S.A. rg@mathcs.emory.edu
More informationTight Upper Bounds on the Redundancy of Optimal Binary AIFV Codes
Tight Upper Bounds on the Redundancy of Optimal Binary AIFV Codes Weihua Hu Dept. of Mathematical Eng. Email: weihua96@gmail.com Hirosuke Yamamoto Dept. of Complexity Sci. and Eng. Email: Hirosuke@ieee.org
More informationData Compression. Limit of Information Compression. October, Examples of codes 1
Data Compression Limit of Information Compression Radu Trîmbiţaş October, 202 Outline Contents Eamples of codes 2 Kraft Inequality 4 2. Kraft Inequality............................ 4 2.2 Kraft inequality
More informationEE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018
Please submit the solutions on Gradescope. Some definitions that may be useful: EE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018 Definition 1: A sequence of random variables X
More informationIntroduction To Resonant. Circuits. Resonance in series & parallel RLC circuits
Introduction To esonant Circuits esonance in series & parallel C circuits Basic Electrical Engineering (EE-0) esonance In Electric Circuits Any passive electric circuit ill resonate if it has an inductor
More informationEcon 201: Problem Set 3 Answers
Econ 20: Problem Set 3 Ansers Instructor: Alexandre Sollaci T.A.: Ryan Hughes Winter 208 Question a) The firm s fixed cost is F C = a and variable costs are T V Cq) = 2 bq2. b) As seen in class, the optimal
More informationRun-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE
General e Image Coder Structure Motion Video x(s 1,s 2,t) or x(s 1,s 2 ) Natural Image Sampling A form of data compression; usually lossless, but can be lossy Redundancy Removal Lossless compression: predictive
More informationThe Dot Product
The Dot Product 1-9-017 If = ( 1,, 3 ) and = ( 1,, 3 ) are ectors, the dot product of and is defined algebraically as = 1 1 + + 3 3. Example. (a) Compute the dot product (,3, 7) ( 3,,0). (b) Compute the
More informationLogic Effort Revisited
Logic Effort Revisited Mark This note ill take another look at logical effort, first revieing the asic idea ehind logical effort, and then looking at some of the more sutle issues in sizing transistors..0
More informationBloom Filters and Locality-Sensitive Hashing
Randomized Algorithms, Summer 2016 Bloom Filters and Locality-Sensitive Hashing Instructor: Thomas Kesselheim and Kurt Mehlhorn 1 Notation Lecture 4 (6 pages) When e talk about the probability of an event,
More informationEE5585 Data Compression January 29, Lecture 3. x X x X. 2 l(x) 1 (1)
EE5585 Data Compression January 29, 2013 Lecture 3 Instructor: Arya Mazumdar Scribe: Katie Moenkhaus Uniquely Decodable Codes Recall that for a uniquely decodable code with source set X, if l(x) is the
More informationBusiness Cycles: The Classical Approach
San Francisco State University ECON 302 Business Cycles: The Classical Approach Introduction Michael Bar Recall from the introduction that the output per capita in the U.S. is groing steady, but there
More informationOn the approximation of real powers of sparse, infinite, bounded and Hermitian matrices
On the approximation of real poers of sparse, infinite, bounded and Hermitian matrices Roman Werpachoski Center for Theoretical Physics, Al. Lotnikó 32/46 02-668 Warszaa, Poland Abstract We describe a
More informationInformation Theory and Statistics Lecture 2: Source coding
Information Theory and Statistics Lecture 2: Source coding Łukasz Dębowski ldebowsk@ipipan.waw.pl Ph. D. Programme 2013/2014 Injections and codes Definition (injection) Function f is called an injection
More informationGRADIENT DESCENT. CSE 559A: Computer Vision GRADIENT DESCENT GRADIENT DESCENT [0, 1] Pr(y = 1) w T x. 1 f (x; θ) = 1 f (x; θ) = exp( w T x)
0 x x x CSE 559A: Computer Vision For Binary Classification: [0, ] f (x; ) = σ( x) = exp( x) + exp( x) Output is interpreted as probability Pr(y = ) x are the log-odds. Fall 207: -R: :30-pm @ Lopata 0
More informationRandomized Smoothing Networks
Randomized Smoothing Netorks Maurice Herlihy Computer Science Dept., Bron University, Providence, RI, USA Srikanta Tirthapura Dept. of Electrical and Computer Engg., Ioa State University, Ames, IA, USA
More informationComplex Numbers and the Complex Exponential
Complex Numbers and the Complex Exponential φ (2+i) i 2 θ φ 2+i θ 1 2 1. Complex numbers The equation x 2 + 1 0 has no solutions, because for any real number x the square x 2 is nonnegative, and so x 2
More information/633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Matroids and Greedy Algorithms Date: 10/31/16
60.433/633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Matroids and Greedy Algorithms Date: 0/3/6 6. Introduction We talked a lot the last lecture about greedy algorithms. While both Prim
More informationSource Coding. Master Universitario en Ingeniería de Telecomunicación. I. Santamaría Universidad de Cantabria
Source Coding Master Universitario en Ingeniería de Telecomunicación I. Santamaría Universidad de Cantabria Contents Introduction Asymptotic Equipartition Property Optimal Codes (Huffman Coding) Universal
More informationLecture 3 : Algorithms for source coding. September 30, 2016
Lecture 3 : Algorithms for source coding September 30, 2016 Outline 1. Huffman code ; proof of optimality ; 2. Coding with intervals : Shannon-Fano-Elias code and Shannon code ; 3. Arithmetic coding. 1/39
More informationComputer Science & Engineering 423/823 Design and Analysis of Algorithms
Computer Science & Engineering 423/823 Design and Analysis of Algorithms Lecture 03 Dynamic Programming (Chapter 15) Stephen Scott and Vinodchandran N. Variyam sscott@cse.unl.edu 1/44 Introduction Dynamic
More informationLecture 3. Mathematical methods in communication I. REMINDER. A. Convex Set. A set R is a convex set iff, x 1,x 2 R, θ, 0 θ 1, θx 1 + θx 2 R, (1)
3- Mathematical methods in communication Lecture 3 Lecturer: Haim Permuter Scribe: Yuval Carmel, Dima Khaykin, Ziv Goldfeld I. REMINDER A. Convex Set A set R is a convex set iff, x,x 2 R, θ, θ, θx + θx
More informationSIGNAL COMPRESSION Lecture Shannon-Fano-Elias Codes and Arithmetic Coding
SIGNAL COMPRESSION Lecture 3 4.9.2007 Shannon-Fano-Elias Codes and Arithmetic Coding 1 Shannon-Fano-Elias Coding We discuss how to encode the symbols {a 1, a 2,..., a m }, knowing their probabilities,
More informationCoding of memoryless sources 1/35
Coding of memoryless sources 1/35 Outline 1. Morse coding ; 2. Definitions : encoding, encoding efficiency ; 3. fixed length codes, encoding integers ; 4. prefix condition ; 5. Kraft and Mac Millan theorems
More informationExploiting Source Redundancy to Improve the Rate of Polar Codes
Exploiting Source Redundancy to Improe the Rate of Polar Codes Ying Wang ECE Department Texas A&M Uniersity yingang@tamu.edu Krishna R. Narayanan ECE Department Texas A&M Uniersity krn@tamu.edu Aiao (Andre)
More informationWorksheet Week 1 Review of Chapter 5, from Definition of integral to Substitution method
Worksheet Week Review of Chapter 5, from Definition of integral to Substitution method This worksheet is for improvement of your mathematical writing skill. Writing using correct mathematical expressions
More informationMinimize Cost of Materials
Question 1: Ho do you find the optimal dimensions of a product? The size and shape of a product influences its functionality as ell as the cost to construct the product. If the dimensions of a product
More information3 - Vector Spaces Definition vector space linear space u, v,
3 - Vector Spaces Vectors in R and R 3 are essentially matrices. They can be vieed either as column vectors (matrices of size and 3, respectively) or ro vectors ( and 3 matrices). The addition and scalar
More informationStat 8112 Lecture Notes Weak Convergence in Metric Spaces Charles J. Geyer January 23, Metric Spaces
Stat 8112 Lecture Notes Weak Convergence in Metric Spaces Charles J. Geyer January 23, 2013 1 Metric Spaces Let X be an arbitrary set. A function d : X X R is called a metric if it satisfies the folloing
More informationLec 03 Entropy and Coding II Hoffman and Golomb Coding
CS/EE 5590 / ENG 40 Special Topics Multimedia Communication, Spring 207 Lec 03 Entropy and Coding II Hoffman and Golomb Coding Zhu Li Z. Li Multimedia Communciation, 207 Spring p. Outline Lecture 02 ReCap
More informationAlgebra II Solutions
202 High School Math Contest 6 Algebra II Eam 2 Lenoir-Rhyne University Donald and Helen Schort School of Mathematics and Computing Sciences Algebra II Solutions This eam has been prepared by the folloing
More information1 Basic Definitions. 2 Proof By Contradiction. 3 Exchange Argument
1 Basic Definitions A Problem is a relation from input to acceptable output. For example, INPUT: A list of integers x 1,..., x n OUTPUT: One of the three smallest numbers in the list An algorithm A solves
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationAlgorithms and Data Structures 2016 Week 5 solutions (Tues 9th - Fri 12th February)
Algorithms and Data Structures 016 Week 5 solutions (Tues 9th - Fri 1th February) 1. Draw the decision tree (under the assumption of all-distinct inputs) Quicksort for n = 3. answer: (of course you should
More informationCS173 Lecture B, November 3, 2015
CS173 Lecture B, November 3, 2015 Tandy Warnow November 3, 2015 CS 173, Lecture B November 3, 2015 Tandy Warnow Announcements Examlet 7 is a take-home exam, and is due November 10, 11:05 AM, in class.
More informationLecture 17: Trees and Merge Sort 10:00 AM, Oct 15, 2018
CS17 Integrated Introduction to Computer Science Klein Contents Lecture 17: Trees and Merge Sort 10:00 AM, Oct 15, 2018 1 Tree definitions 1 2 Analysis of mergesort using a binary tree 1 3 Analysis of
More informationSolving with Absolute Value
Solving with Absolute Value Who knew two little lines could cause so much trouble? Ask someone to solve the equation 3x 2 = 7 and they ll say No problem! Add just two little lines, and ask them to solve
More informationLecture 1: September 25, A quick reminder about random variables and convexity
Information and Coding Theory Autumn 207 Lecturer: Madhur Tulsiani Lecture : September 25, 207 Administrivia This course will cover some basic concepts in information and coding theory, and their applications
More information10-704: Information Processing and Learning Fall Lecture 9: Sept 28
10-704: Information Processing and Learning Fall 2016 Lecturer: Siheng Chen Lecture 9: Sept 28 Note: These notes are based on scribed notes from Spring15 offering of this course. LaTeX template courtesy
More informationPGSS Discrete Math Solutions to Problem Set #4. Note: signifies the end of a problem, and signifies the end of a proof.
PGSS Discrete Math Solutions to Problem Set #4 Note: signifies the end of a problem, and signifies the end of a proof. 1. Prove that for any k N, there are k consecutive composite numbers. (Hint: (k +
More informationCMPT 365 Multimedia Systems. Lossless Compression
CMPT 365 Multimedia Systems Lossless Compression Spring 2017 Edited from slides by Dr. Jiangchuan Liu CMPT365 Multimedia Systems 1 Outline Why compression? Entropy Variable Length Coding Shannon-Fano Coding
More informationConstrained and Unconstrained Optimization Prof. Adrijit Goswami Department of Mathematics Indian Institute of Technology, Kharagpur
Constrained and Unconstrained Optimization Prof. Adrijit Goswami Department of Mathematics Indian Institute of Technology, Kharagpur Lecture - 01 Introduction to Optimization Today, we will start the constrained
More informationLecture 16. Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code
Lecture 16 Agenda for the lecture Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code Variable-length source codes with error 16.1 Error-free coding schemes 16.1.1 The Shannon-Fano-Elias
More informationLinear models: the perceptron and closest centroid algorithms. D = {(x i,y i )} n i=1. x i 2 R d 9/3/13. Preliminaries. Chapter 1, 7.
Preliminaries Linear models: the perceptron and closest centroid algorithms Chapter 1, 7 Definition: The Euclidean dot product beteen to vectors is the expression d T x = i x i The dot product is also
More informationMATH 115, SUMMER 2012 LECTURE 4 THURSDAY, JUNE 21ST
MATH 115, SUMMER 2012 LECTURE 4 THURSDAY, JUNE 21ST JAMES MCIVOR Today we enter Chapter 2, which is the heart of this subject. Before starting, recall that last time we saw the integers have unique factorization
More informationCSE 190, Great ideas in algorithms: Pairwise independent hash functions
CSE 190, Great ideas in algorithms: Pairwise independent hash functions 1 Hash functions The goal of hash functions is to map elements from a large domain to a small one. Typically, to obtain the required
More informationCS173 Strong Induction and Functions. Tandy Warnow
CS173 Strong Induction and Functions Tandy Warnow CS 173 Introduction to Strong Induction (also Functions) Tandy Warnow Preview of the class today What are functions? Weak induction Strong induction A
More informationSummary of Last Lectures
Lossless Coding IV a k p k b k a 0.16 111 b 0.04 0001 c 0.04 0000 d 0.16 110 e 0.23 01 f 0.07 1001 g 0.06 1000 h 0.09 001 i 0.15 101 100 root 1 60 1 0 0 1 40 0 32 28 23 e 17 1 0 1 0 1 0 16 a 16 d 15 i
More informationCOS 511: Theoretical Machine Learning. Lecturer: Rob Schapire Lecture # 12 Scribe: Indraneel Mukherjee March 12, 2008
COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture # 12 Scribe: Indraneel Mukherjee March 12, 2008 In the previous lecture, e ere introduced to the SVM algorithm and its basic motivation
More informationSemi-simple Splicing Systems
Semi-simple Splicing Systems Elizabeth Goode CIS, University of Delaare Neark, DE 19706 goode@mail.eecis.udel.edu Dennis Pixton Mathematics, Binghamton University Binghamton, NY 13902-6000 dennis@math.binghamton.edu
More informationComputer Science & Engineering 423/823 Design and Analysis of Algorithms
Computer Science & Engineering 423/823 Design and Analysis of s Lecture 09 Dynamic Programming (Chapter 15) Stephen Scott (Adapted from Vinodchandran N. Variyam) 1 / 41 Spring 2010 Dynamic programming
More informationENGINEERING MATH 1 Fall 2009 VECTOR SPACES
ENGINEERING MATH 1 Fall 2009 VECTOR SPACES A vector space, more specifically, a real vector space (as opposed to a complex one or some even stranger ones) is any set that is closed under an operation of
More informationLecture 8 January 30, 2014
MTH 995-3: Intro to CS and Big Data Spring 14 Inst. Mark Ien Lecture 8 January 3, 14 Scribe: Kishavan Bhola 1 Overvie In this lecture, e begin a probablistic method for approximating the Nearest Neighbor
More informationPolynomial and Rational Functions
Polnomial and Rational Functions Figure -mm film, once the standard for capturing photographic images, has been made largel obsolete b digital photograph. (credit film : modification of ork b Horia Varlan;
More informationAPC486/ELE486: Transmission and Compression of Information. Bounds on the Expected Length of Code Words
APC486/ELE486: Transmission and Compression of Information Bounds on the Expected Length of Code Words Scribe: Kiran Vodrahalli September 8, 204 Notations In these notes, denotes a finite set, called the
More informationMathematics-I Prof. S.K. Ray Department of Mathematics and Statistics Indian Institute of Technology, Kanpur. Lecture 1 Real Numbers
Mathematics-I Prof. S.K. Ray Department of Mathematics and Statistics Indian Institute of Technology, Kanpur Lecture 1 Real Numbers In these lectures, we are going to study a branch of mathematics called
More informationData Compression Techniques (Spring 2012) Model Solutions for Exercise 2
582487 Data Compression Techniques (Spring 22) Model Solutions for Exercise 2 If you have any feedback or corrections, please contact nvalimak at cs.helsinki.fi.. Problem: Construct a canonical prefix
More informationARE211, Fall2012. Contents. 2. Linear Algebra (cont) Vector Spaces Spanning, Dimension, Basis Matrices and Rank 8
ARE211, Fall2012 LINALGEBRA2: TUE, SEP 18, 2012 PRINTED: SEPTEMBER 27, 2012 (LEC# 8) Contents 2. Linear Algebra (cont) 1 2.6. Vector Spaces 1 2.7. Spanning, Dimension, Basis 3 2.8. Matrices and Rank 8
More informationShannon-Fano-Elias coding
Shannon-Fano-Elias coding Suppose that we have a memoryless source X t taking values in the alphabet {1, 2,..., L}. Suppose that the probabilities for all symbols are strictly positive: p(i) > 0, i. The
More informationAnnouncements Wednesday, September 06. WeBWorK due today at 11:59pm. The quiz on Friday covers through Section 1.2 (last weeks material)
Announcements Wednesday, September 06 WeBWorK due today at 11:59pm. The quiz on Friday coers through Section 1.2 (last eeks material) Announcements Wednesday, September 06 Good references about applications(introductions
More information1 Introduction to information theory
1 Introduction to information theory 1.1 Introduction In this chapter we present some of the basic concepts of information theory. The situations we have in mind involve the exchange of information through
More informationBefore this course is over we will see the need to split up a fraction in a couple of ways, one using multiplication and the other using addition.
CH MORE FRACTIONS Introduction I n this chapter we tie up some loose ends. First, we split a single fraction into two fractions, followed by performing our standard math operations on positive and negative
More informationIntroduction to information theory and coding
Introduction to information theory and coding Louis WEHENKEL Set of slides No 4 Source modeling and source coding Stochastic processes and models for information sources First Shannon theorem : data compression
More information1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H.
Problem sheet Ex. Verify that the function H(p,..., p n ) = k p k log p k satisfies all 8 axioms on H. Ex. (Not to be handed in). looking at the notes). List as many of the 8 axioms as you can, (without
More informationLogistic Regression. Machine Learning Fall 2018
Logistic Regression Machine Learning Fall 2018 1 Where are e? We have seen the folloing ideas Linear models Learning as loss minimization Bayesian learning criteria (MAP and MLE estimation) The Naïve Bayes
More informationMath 249B. Geometric Bruhat decomposition
Math 249B. Geometric Bruhat decomposition 1. Introduction Let (G, T ) be a split connected reductive group over a field k, and Φ = Φ(G, T ). Fix a positive system of roots Φ Φ, and let B be the unique
More informationDigital communication system. Shannon s separation principle
Digital communication system Representation of the source signal by a stream of (binary) symbols Adaptation to the properties of the transmission channel information source source coder channel coder modulation
More informationOn the redundancy of optimum fixed-to-variable length codes
On the redundancy of optimum fixed-to-variable length codes Peter R. Stubley' Bell-Northern Reserch Abstract There has been much interest in recent years in bounds on the redundancy of Huffman codes, given
More informationIntro to Information Theory
Intro to Information Theory Math Circle February 11, 2018 1. Random variables Let us review discrete random variables and some notation. A random variable X takes value a A with probability P (a) 0. Here
More informationHomework 6 Solutions
Homeork 6 Solutions Igor Yanovsky (Math 151B TA) Section 114, Problem 1: For the boundary-value problem y (y ) y + log x, 1 x, y(1) 0, y() log, (1) rite the nonlinear system and formulas for Neton s method
More informationOptimisation and Operations Research
Optimisation and Operations Research Lecture 15: The Greedy Heuristic Matthew Roughan http://www.maths.adelaide.edu.au/matthew.roughan/ Lecture_notes/OORII/ School of
More informationCoding for Discrete Source
EGR 544 Communication Theory 3. Coding for Discrete Sources Z. Aliyazicioglu Electrical and Computer Engineering Department Cal Poly Pomona Coding for Discrete Source Coding Represent source data effectively
More informationDynamic Programming. Shuang Zhao. Microsoft Research Asia September 5, Dynamic Programming. Shuang Zhao. Outline. Introduction.
Microsoft Research Asia September 5, 2005 1 2 3 4 Section I What is? Definition is a technique for efficiently recurrence computing by storing partial results. In this slides, I will NOT use too many formal
More information