Kotebe Metropolitan University Department of Computer Science and Technology Multimedia (CoSc 4151)

Transcription

1 Kotebe Metropolitan University Department of Computer Science and Technology Multimedia (CoSc 4151) Chapter Three Multimedia Data Compression Part I: Entropy in ordinary words Claude Shannon developed the core elements of information theory in the 1940s. There are many aspects to this theory and much has been done beyond Shannon (see Chaitin and Kolmogorov). However, a key question I want to pose is this: Suppose you are given a sequence of symbols (say bits (on/off or heads/tails), or letters of the alphabet). How much information is in the sequence? Shannon showed that information is, in a useful sense, a measure of entropy. More entropy, more information. This may seem very counterintuitive, but think about it like this: Say you've seen a long chain of symbols in the sequence so far. Based on all you've seen so far, you could now compute the probability of each possible next symbol. Suppose one symbol was much more probable than the others. Then you would be less surprised to see it next, than the others. Information is amount of data that must be communicated in order to bridge between what you would expect next and what is actually coming next. If the next symbol will unsurprising to you based on seeing the previous symbols, then it can be communicated with less data than some symbol that would more surprising. You can measure the average surprise over all the different symbols and use it to determine how much information per symbol actually exists. This information is measured in *bits* (ones/zeros) Think of a flipping a coin again and again and sending the outcomes to your friend over the telephone. If it's a fair coin, the information content of both "head" and "tail" is one bit. You need to tell your friend, on every flip, 1 bit of information: head or tail. Suppose it's a coin that comes up 80% of the time as heads. Then, when you flip it an comes up heads, you can send *less than* 1 bit of information (keep in mind, this is averaged over many flips) for a head and more than 1 bit of information for a tail. On average, though, over both heads and tails, you could send less than one 1 bit of information per flip. Suppose it's a double-headed coin. Then, at the beginning, you'd just tell your friend this, and your average bits per flip would be zero. Suppose the coin followed a pattern, say HHTTHHTTHHTT...: then you could tell your friend a fixed amount of information: "HHTT [repeat]", which, amortized over many flips would approach zero bits per flip. Very little information. Shannon's classic experiment on computing the information content (entropy) of letters of the alphabet within an English sentence is a fun thing to play with: With 27 characters (A-Z + space), English text can at most have about 4.8 bits per character of information (see if you'd like to compute for yourself). But the various measures (including the above game), show that actual English text has closer to 1 bit per character of information (and probably closer to 0.5 bits/ character). What about Amharic and other Ethiopian languages? So, what does this all have to do with computer science? Well, one easy connection is compression. When you compress a file, you are re- encoding its sequence of symbols such that the file size gets as close as possible to its information content/entropy. That re-encoded file will have much more "surprise" than the original. If you measure the information content of its symbols, it will be much closer to the Shannon limit. If you look at a compressed file on your computer (jpg, mpg, zip, etc), you'll 1

2 find that its bits are very close to the random sequence you might expect from flipping a fair coin. You should now understand, btw, why you should never believe someone who claims that you can shrink a file further by compressing it a second time. Part II: Information and Entropy Information Equation I(p) = log b p Where: p = probability of the event happening b = base (base 2 is mostly used in information theory) *unit of information is determined by base base 2 = bits; base 3 = trits; base 10 = Hartleys; base e = nats; Example of Calculating Information Coin Toss There are two probabilities in fair coin, which are head(.5) and tail(.5). So if you get either head or tail you will get 1 bit of information through following formula. I(head) = - log (.5) = 1 bit Another Example Balls in the bin The information you will get by choosing a ball from the bin are calculated as following. I (red ball) = - log(4/9) = bits I (yellow ball) = - log(2/9) = bits I (green ball) = - log(3/9) = bits Then, what is Entropy? - Entropy is simply the average (expected) amount of the information from the event. Entropy Equation n Entropy = i p i log b p i Where n = number of different outcomes How the entropy equation was derived n I = i (N p i ) log b p i Where: I = total information from N occurrences N = number of occurrences (N*Pi) = Approximated number that the certain result will come out in N occurrence 2

3 n Entropy = i p i log b p i So when you look at the difference between the total Information from N occurrences and the Entropy equation, only thing that changed in the place of N. The N is moved to the right, which means that I/N is Entropy. Therefore, Entropy is the average (expected) amount of information in a certain event. Let s look at the previous example again Calculating the entropy In this example there are three outcomes possible when you choose the ball, it can be either red, yellow, or green. (n = 3) So the equation will be following. n Entropy = i p i log b p i Entropy = - (4/9) log(4/9) + -(2/9) log(2/9) + - (3/9) log(3/9) = Therefore, you are expected to get information each time you choose a ball from the bin Does Entropy have range from 0 to 1? No. However, the range is set based on the number of outcomes. Equation for calculating the range of Entropy: 0 Entropy log(n), where n is number of outcomes Entropy 0(minimum entropy) occurs when one of the probabilities is 1 and rest are 0 s Entropy log(n) (maximum entropy) occurs when all the probabilities have equal values of 1/n. What is the difference between Information and knowledge? Information different from knowledge Concerned with abstract possibilities, not their meaning Information = reduction in uncertainty Imagine: #1 you re about to observe the outcome of a coin flip #2 you re about to observe the outcome of a die roll There is more uncertainty in #2 Next: 1. You observed outcome of #1 uncertainty reduced to zero. 2. You observed outcome of #2 uncertainty reduced to zero. = more information was provided by the outcome in #2 3

4 Information is Additive 1 I (k fair coin tosses) = 1 2 k = k bits So: Random word from a1 00,000 word vocabulary: o I(word) = log100,000 = 16.61bits o A 1000 word document from same source I(document) = 16,610bits o A 480x640 pixel, 16-grayscale video picture: I(picture) = 307,200 * log16 = 1,228,800 bits A (VGA) picture is worth (a lot more than) a 1000 words! (In reality, both are gross over estimates.) Part III: Alternative Explanations of Entropy (THIS IS VERY IMPORTANT) n Entropy = i p i log b p i 1. Average amount of information provided per symbol 2. Average amount surprise when observing a symbol 3. Uncertainty an observer has before seeing the symbol 4. Average number of bits needed to communicate each symbol Shannon: there are codes that will communicate these symbols with efficiency arbitrarily close to Entropy bits/symbol; there are no codes that will do it with efficiency less than Entropy bits/symbol NOTE: RELATE THE ABOVE INTERPRETATIONS WITH SOLUTIONS OF PREVIOS PROBLEMS Special Case: k = 2 Flipping a coin with P ( head ) = p, P ( tail ) = 1 - p Entropy = p log 2 1 p + (p 1) log 2 1 (p 1) Notice: Zero uncertainty/information/surprise at edges. Maximum info at 0.5 (1bit). Drops off quickly. So a sequence of (independent) 0 s-and-1 s can provide up to 1 bit of information per digit, provided the 0 s and 1 s are equally likely at any point. If they are not equally likely the sequence provides less information and can be compressed. Entropy doesn t tell you the amount of information you get on the happening of a particular event (one occurrence of head in coin toss), rather it tells you the average amount of information you get (per event head in coin toss) out of repeated occurrences of that event such as happening of head out of repeated occurrences of coin toss experiment. At this point, be sure you understand the concept and examples of entropy, it is very useful in information science. The concept of Entropy is very wide and crosses several disciplines, it is not limited to Information Science. For those from other discipline (or interested in being interdisciplinary), you can entertain by connecting the concept of entropy to other discipline/s you want. If you want to read further, please visit this link (especially read pages 15 to 34) 4

5 Modeling and Compression We will look at models whose purpose is primarily compression of multimedia data. We are interested in modeling multimedia data. To model means to replace something complex with a simpler (= shorter) analog. Some models help understand the original phenomenon/data better: Example: Laws of physics Huge arrays of astronomical observations (e.g. Tycho Brahe s logbooks) summarised in a few characters (e.g. Kepler, Newton): F = G M 1M 2 r 2. This model helps us understand gravity better. Is an example of tremendous compression of data.

6 Recap: The Need for Compression Raw video, image, and audio files can be very large. Example: One minute of uncompressed audio. Audio Type 44.1 KHz KHz KHz 16 Bit Stereo: 10.1 MB 5.05 MB 2.52 MB 16 Bit Mono: 5.05 MB 2.52 MB 1.26 MB 8 Bit Mono: 2.52 MB 1.26 MB 630 KB Example: Uncompressed images. Image Type File Size 512 x 512 Monochrome 0.25 MB 512 x bit colour image 0.25 MB 512 x bit colour image 0.75 MB

7 Recap: The Need for Compression Example: Videos (involves a stream of audio plus video imagery). Raw Video uncompressed image frames 512x512 True Colour at 25 FPS = 1125 MB/min. HDTV ( ) Gigabytes per minute uncompressed, True Colour at 25 FPS = 8.7 GB/min. Relying on higher bandwidths is not a good option M25 Syndrome: traffic will always increase to fill the current bandwidth limit whatever this is. Compression HAS TO BE part of the representation of audio, image, and video formats.

8 Basics of Information Theory Suppose we have an information source (random variable) S which emits symbols {s 1, s 2,..., s n } with probabilities p 1, p 2,..., p n. According to Shannon, the entropy of S is defined as: H(S) = 1 p i log 2, p i i where p i is the probability that symbol s i will occur. When a symbol with probability p i is transmitted, it reduces the amount of uncertainty in the receiver by a factor of 1 pi. log 2 1 pi = log 2 p i indicates the amount of information conveyed by s i, i.e., the number of bits needed to code s i (Shannon s coding theorem).

9 Entropy Example Example: Entropy of a fair coin. The coin emits symbols s 1 = heads and s 2 = tails with p 1 = p 2 = 1/2. Therefore, the entropy if this source is: H(coin) = (1/2 log 2 1/2 + 1/2 log 2 1/2) = (1/ /2 1) = ( 1/2 1/2) = 1 bit. Example: Grayscale image In an image with uniform distribution of gray-level intensity (and all pixels independent), i.e. p i = 1/256, then The # of bits needed to code each gray level is 8 bits. The entropy of this image is 8.

10 Entropy Example Example: Breakfast order #1. Alice: What do you want for breakfast: pancakes or eggs? I am unsure, because you like them equally (p 1 = p 2 = 1/2)... Bob: I want pancakes. Question: How much information has Bob communicated to Alice?

11 Entropy Example Example: Breakfast order #1. Alice: What do you want for breakfast: pancakes or eggs? I am unsure, because you like them equally (p 1 = p 2 = 1/2)... Bob: I want pancakes. Question: How much information has Bob communicated to Alice? Answer: He has reduced the uncertainty by a factor of 2, therefore 1 bit.

12 Entropy Example Example: Breakfast order #2. Alice: What do you want for breakfast: pancakes, eggs, or salad? I am unsure, because you like them equally (p 1 = p 2 = p 3 = 1/3)... Bob: Eggs. Question: What is Bob s entropy assuming he behaves like a random variable = how much information has Bob communicated to Alice?

13 Entropy Example Example: Breakfast order #2. Alice: What do you want for breakfast: pancakes, eggs, or salad? I am unsure, because you like them equally (p 1 = p 2 = p 3 = 1/3)... Bob: Eggs. Question: What is Bob s entropy assuming he behaves like a random variable = how much information has Bob communicated to Alice? Answer: H(Bob) = 3 i=1 1 3 log 2 3 = log bits.

14 Entropy Example Example: Breakfast order #3. Alice: What do you want for breakfast: pancakes, eggs, or salad? I am unsure, because you like them equally (p 1 = p 2 = p 3 = 1/3)... Bob: Dunno. I definitely do not want salad. Question: How much information has Bob communicated to Alice?

15 Entropy Example Example: Breakfast order #3. Alice: What do you want for breakfast: pancakes, eggs, or salad? I am unsure, because you like them equally (p 1 = p 2 = p 3 = 1/3)... Bob: Dunno. I definitely do not want salad. Question: How much information has Bob communicated to Alice? Answer: He has reduced her uncertainty by a factor of 3/2 (leaving 2 out of 3 equal options), therefore transmitted log 2 3/ bits.

16 Shannon s Experiment (1951) Estimated entropy for English text: H English bits/letter. (If all letters and space were equally probable, then it would be H 0 = log bits/letter.) External link: Shannon s original 1951 paper. External link: Java applet recreating Shannon s experiment.

17 Shannon s coding theorem Shannon 1948 Basically: The optimal code length for an event with probability p is L(p) = log 2 p ones and zeros (or generally, log b p if instead we use b possible values for codes). External link: Shannon s original 1948 paper.

18 Shannon vs Kolmogorov What if we have a finite string? Shannon s entropy is a statistical measure of information. We can cheat and regard a string as infinitely long sequence of i.i.d. random variables. Shannon s theorem then approximately applies. Kolmogorov Complexity: Basically, the length of the shortest program that ouputs a given string. Algorithmical measure of information. K(S) is not computable! Practical algorithmic compression is hard.

19 Compression in Multimedia Data Compression basically employs redundancy in the data: Temporal in 1D data, 1D signals, audio, between video frames etc. Spatial correlation between neighbouring pixels or data items. Spectral e.g. correlation between colour or luminescence components. This uses the frequency domain to exploit relationships between frequency of change in data. Psycho-visual exploit perceptual properties of the human visual system.

20 Lossless vs Lossy Compression Compression can be categorised in two broad ways: Lossless Compression: after decompression gives an exact copy of the original data. Example: Entropy encoding schemes (Shannon-Fano, Huffman coding), arithmetic coding, LZW algorithm (used in GIF image file format). Lossy Compression: after decompression gives ideally a close approximation of the original data, ideally perceptually lossless. Example: Transform coding FFT/DCT based quantisation used in JPEG/MPEG differential encoding, vector quantisation.

21 Why Lossy Compression? Lossy methods are typically applied to high resoultion audio, image compression. Have to be employed in video compression (apart from special cases). Basic reason: Compression ratio of lossless methods (e.g. Huffman coding, arithmetic coding, LZW) is not high enough for audio/video. By cleverly making a small sacrifice in terms of fidelity of data, we can often achieve very high compression ratios. Cleverly = sacrifice information that is psycho-physically unimportant.

22 Lossless Compression Algorithms Repetitive Sequence Suppression. Run-Length Encoding (RLE). Pattern Substitution. Entropy Encoding: Shannon-Fano Algorithm. Huffman Coding. Arithmetic Coding. Lempel-Ziv-Welch (LZW) Algorithm.