Source Coding Fundamentals

Transcription

1 Surce Cding Fundamentals Surce Cding Fundamentals Thmas Wiegand Digital Image Cmmunicatin 1 / 54

2 Surce Cding Fundamentals Outline Intrductin Lssless Cding Huffman Cding Elias and Arithmetic Cding Rate-Distrtin Thery Rate-Distrtin Functin Shannn Lwer Bund Quantizatin Scalar Quantizatin Vectr Quantizatin Predictive Cding Linear Predictin Differential Pulse Cde Mdulatin (DPCM) Transfrm Cding Orthgnal Transfrms and Bit Allcatin Karhunen Léve Transfrm (KLT) Discrete Csine Transfrm (DCT) Thmas Wiegand Digital Image Cmmunicatin 2 / 54

3 Surce Cding Fundamentals Practical Cmmunicatin Prblem Surce cdecs are primarily characterized in terms f: Thrughput f the channel, a characteristic influenced by transmissin channel bit rate and amunt f prtcl and errr-crrectin cding verhead incurred by transmissin system Distrtin f the decded signal, which is primarily induced by surce encder and by channel errrs intrduced in path t surce decder The fllwing additinal cnstraints must als be cnsidered: Delay (start-up latency and end-t-end delay) including prcessing delay, buffering, structural delays f surce and channel cdecs, and speed at which data are cnveyed thrugh transmissin channel Cmplexity (cmputatin, memry capacity, memry access) f surce cdec, prtcl stacks and netwrk Thmas Wiegand Digital Image Cmmunicatin 3 / 54

4 Surce Cding Fundamentals Frmulatin f the Practical Cmmunicatin Prblem The practical surce cding design prblem can be psed as fllws: Given a maximum allwed delay and a maximum allwed cmplexity, achieve an ptimal trade-ff between bit rate and distrtin fr the transmissin prblem in the targeted applicatins Scpe f the cnsideratin in this lecture: Surce cdec Delay is nly evaluated fr surce cdec Cmplexity is als nly assessed fr the algrithm used in surce cdec Thmas Wiegand Digital Image Cmmunicatin 4 / 54

5 Surce Cding Fundamentals Types f Cmpressin Lssless cding: Uses redundancy reductin as the nly principle and is therefre reversible Als referred t as niseless cding, invertible cding, data cmpactin, r entrpy cding Well knwn use fr this type f cmpressin fr data is Lempel-Ziv cding (e.g., gzip) and fr picture and vide signals JPEG-LS is well knwn Lssy cding: Uses redundancy reductin and irrelevancy reductin and is therefre nt reversible It is the primary cding type in cmpressin fr speech, audi, picture, and vide signals The practically relevant bit rate reductin that is achievable thrugh lssy cmpressin is typically mre than an rder f magnitude larger than with lssless cmpressin Well knwn examples are fr audi cding are the MPEG-1 Layer 3 (mp3), fr still picture cding JPEG, and fr vide cding H.264/AVC Thmas Wiegand Digital Image Cmmunicatin 5 / 54

6 Surce Cding Fundamentals Distrtin Measures Usage f distrtin measures The use f lssy cmpressin requires the ability t measure distrtin Often, the distrtin that a human perceives in cded cntent is a very difficult quantity t measure, as the characteristics f human perceptin are cmplex Perceptual mdels are far mre advanced fr speech and audi cdecs than fr picture r vide cdecs In picture and vide cding, Perceptual mdels have limited use t guide encding decisins (mainly fcusing n prperties f the human visual system) Viewing tests are used t determine subjective quality f cding results Fr investigating surce cding techniques, simple bjective distrtin measures such as MSE and PSNR are ften used Thmas Wiegand Digital Image Cmmunicatin 6 / 54

7 Surce Cding Fundamentals Objective Distrtin Measures Mean Squared Errr (MSE) Pictures (X: picture height, Y : picture width): MSE = 1 X Y X 1 Y 1 x=0 y=0 ( s [x, y] s[x, y] ) 2 Vides (N: number f pictures, MSE n: MSE f picture n): MSE = 1 N N 1 n=0 Peak Signal-t-Nise Rati (PSNR) Pictures (2 k 1: maximum amplitude f the picture samples) ( ) (2 k 1) 2 PSNR = 10 lg 10 Thmas Wiegand Digital Image Cmmunicatin 7 / 54 (1) MSE n (2) MSE Vides (N: number f pictures, PSNR n: PSNR f picture n) PSNR = 1 N N 1 n=0 (3) PSNR n (4)

8 Surce Cding Fundamentals Distrtin Measures fr Picture and Vide Cding Typical artifacts in cmpressed vides Blckiness, mtin errrs, blur, ringing Picture and vide encding Measurement f prperties f the human visual system have been used t derived spatial cntrast sensitivity functins that are used in the encding f pictures Tempral sensitivity functins that have been measured have s far nt been included in vide encding algrithms knwn in the public dmain Mean squared errr is a widely used measure Picture and vide quality assessment Objective quality measures ften have very limited crrelatin with the actual subjective quality f a cmpressed picture r vide signal Picture and vide quality is assessed in viewing tests Viewing test with human subjects are cstly and time cnsuming Thmas Wiegand Digital Image Cmmunicatin 8 / 54

9 Surce Cding Fundamentals Vide Quality Measurement Viewing cnditins The viewing cnditins are fixed and described in rder t be able t reprduce the subjective test ITU-R Rec. BT specifies viewing cnditins including Lw rm illuminatin; Viewing screen is the main light surce Screen size and preferred viewing distance as rati between height f the screen and distance frm screen Example fr a viewing test methd Duble stimulus cntinuus quality scale (DSCQS) is a test methd where the subject view the stimuli cmpressed by a cdec under test and a reference Cded sequence and reference are presented alternately Each sequence is presented twice Sequences are separated by mid-gray level sequence f 3 sec duratin Subjects have n knwledge f display chrnlgy Subjects rate vide n a cntinuus scale is partitined int intervals that are labeled Excellent, Gd, Fair, Pr r Bad Thmas Wiegand Digital Image Cmmunicatin 9 / 54

10 Surce Cding Fundamentals Lssless Cding Lssless Cding Thmas Wiegand Digital Image Cmmunicatin 10 / 54

11 Surce Cding Fundamentals Lssless Cding Discrete Randm Variables and Prbability Mass Functin Definitins A randm variable S is a functin f the sample space O that assigns a value S(ζ) t each utcme ζ O f a randm experiment A randm variable S called discrete randm variable if it takes values f cuntable alphabet A = {a 0, a 1,...} Prbability mass functin (pmf) fr discrete randm variables Examples fr pmfs Binary pmf: Unifrm pmf: Gemetric pmf: p S(a) = P (S = a) = P ( {ζ : S(ζ)= a} ) (5) A = {a 0, a 1} p S(a 0) = p, p S(a 1) = 1 p (6) A = {a 0, a 1,, a M 1} p S(a i) = 1/M a i A (7) A = {a 0, a 1, } p S(a i) = (1 p) p i a i A (8) Thmas Wiegand Digital Image Cmmunicatin 11 / 54

12 Surce Cding Fundamentals Lssless Cding Jint and Cnditinal Pmfs Jint prbability mass functin The N-dimensinal pmf r jint pmf fr a randm vectr S = (S 0,, S N 1 ) T is defined by p S (a) = P (S = a) = P (S 0 = a 0,, S N 1 = a N 1 ) (9) Jint pmf f tw randm vectrs X and Y p XY (a x, a y ) = P (X =a x, Y =a y ) (10) Cnditinal prbability mass functins The cnditinal pmf p S B (a B) f a randm variable S given an event B, with P (B) > 0, is defined by p S B (a B) = P (S = a B) (11) Cnditinal pmf f a randm vectr X given anther randm vectr Y p X Y (a x a y ) = p XY (a x, a y ) p Y (a y ) (12) Thmas Wiegand Digital Image Cmmunicatin 12 / 54

13 Surce Cding Fundamentals Lssless Cding Example fr Jint Pmf Example: Jint pmf fr neighbring picture samples Samples in picture and vide signals typically shw strng statistical dependencies Belw: Histgram f tw hrizntally adjacent samples fr the picture Lena Relative frequency f ccurence Amplitude f adjacent pixel Amplitude f current pixel Thmas Wiegand Digital Image Cmmunicatin 13 / 54

14 Surce Cding Fundamentals Lssless Cding Expectatin Expectatin value r expectatin Definitin fr discrete randm variables S E {g(s)} = g(a) p S (a) (13) a A Imprtant expectatin values are mean µ S and variance σ 2 S µ S = E {S} and σ 2 S = E { (S µ s ) 2} (14) Cnditinal expectatin The cnditinal expectatin value f functin g(s) given an event B, with P (B) > 0, is defined by E {g(s) B} = a A g(a) p S (a B) = a A g(a) P (S = a, B) P (B) (15) Thmas Wiegand Digital Image Cmmunicatin 14 / 54

15 Surce Cding Fundamentals Lssless Cding Discrete Randm Prcesses Discrete randm prcess Series f randm experiments at time instants t n, with n = 0, 1, 2,..., characterized by a series f randm variables S = {S n} Statistical prperties f discrete-time randm prcess S: N-th rder jint pmf p Sn (a) = P (S n = a 0,, S n+n 1 = a N 1) (16) Statinary randm prcess Statistical prperties are invariant t a shift in time: p Sn (a) = p S (a) Memryless randm prcess The randm variables S n are independent f each ther Independent and identical distributed (iid) randm prcess Statinary and memryless: p S (a) = p S(a 0) p S(a 1) p S(a N 1) Markv prcess Future utcmes d nt depend n past utcmes, but nly n the present utcme p Sn (a n a n 1, ) = p Sn (a n a n 1) (17) Thmas Wiegand Digital Image Cmmunicatin 15 / 54

16 Surce Cding Fundamentals Lssless Cding Lssless Surce Cding Overview Lssless surce cding Reversible mapping f sequence f discrete surce symbls int sequences f cdewrds Other names: Niseless cding, entrpy cding Original surce sequence can be exactly recnstructed nt the case in lssy cding Bit rate reductin is pssible, if surce data cntain statistical prperties that are explitable fr data cmpressin Thmas Wiegand Digital Image Cmmunicatin 16 / 54

17 Surce Cding Fundamentals Lssless Cding Lssless Surce Cding Terminlgy Terminlgy Message s (L) ={s 0,, s L 1 } drawn frm stchastic prcess S ={S n } Sequence b (K) ={b 0,, b K 1 } f K bits (b k B ={0, 1}) Prcess f lssless cding: Message s (L) is cnverted t b (K) Assume: Subsequence s (N) = {s n,, s n+n 1} with 1 N L and Bits b (l) (s (N) ) = {b 0,, b l 1 } assigned t it Lssless surce cde Encder mapping: b (l) = γ ( s (N) ) (18) Decder mapping: s (N) = γ 1( b (l) ) = γ 1( γ ( s (N) ) ) (19) Thmas Wiegand Digital Image Cmmunicatin 17 / 54

18 Surce Cding Fundamentals Lssless Cding Classificatin f Lssless Surce Cdes Lssless surce cde Encder mapping: Decder mapping: b (l) = γ ( s (N) ) (20) s (N) = γ 1( b (l) ) = γ 1( γ ( s (N) ) ) (21) Classificatin Fixed-t-fixed mapping: N and l are bth fixed (discussed as special case f fixed-t-variable) Fixed-t-variable mapping: N fixed and l variable Huffman algrithm fr scalars and vectrs (discussed in lecture) Variable-t-fixed mapping: N variable and l fixed Tunstall cdes (nt discussed in lecture) Variable-t-variable mapping: l and N are bth variable Arithmetic cdes (discussed in lecture) Thmas Wiegand Digital Image Cmmunicatin 18 / 54

19 Surce Cding Fundamentals Lssless Cding Variable-Length Cding fr Scalars Assigning cdewrds t scalar symbls Assign a separate cdewrd t each scalar symbl s n f a message s (L) Assume: Message s (L) generated by statinary randm prcess S = {S n } Randm variables S n = S with symbl alphabet A = {a 0,, a M 1 } and marginal pmf p(a) = P (S = a) Lssless surce cde: Assign a binary cdewrd b i = {b i 0,, b i l(a i) 1 } t each alphabet letter a i The length (number f bits) f the cdewrd b i that is assigned t a i is dented by l(a i ), with l(a i ) 1 Average cdewrd length l = E {l(s)} = M 1 i=0 p(a i ) l(a i ) (22) Thmas Wiegand Digital Image Cmmunicatin 19 / 54

20 Surce Cding Fundamentals Lssless Cding Optimizatin Prblem Average cde wrd length is given as l = K 1 k=0 p(a k ) l(a k ) (23) Gal f lssless cde design: Minimize average cdewrd length l while prviding unique decdability a i p(a i) cde A cde B cde C cde D cde E a a a a l Thmas Wiegand Digital Image Cmmunicatin 20 / 54

21 Surce Cding Fundamentals Lssless Cding Unique Decdability and Prefix Cdes Unique decdability The cde γ has t specify a mapping a i b i such that if a k a j then b k b j (24) Fr sequences f symbls, abve cnstraint needs t be extended t the cncatenatin f multiple symbls Fr a uniquely decdable cde, a sequence f cdewrds can nly be generated by ne pssible sequence f surce symbls. Prefix cdes One class f cdes that satisfies the cnstraint f unique decdability is called prefix cdes A cde is called a prefix cde if n cdewrd is a prefix f any ther cdewrd It is bvius that if cnditin (24) is satisfied and the cde is a prefix cde, then any cncatenatin f symbls is uniquely decdable Thmas Wiegand Digital Image Cmmunicatin 21 / 54

22 Surce Cding Fundamentals Lssless Cding Binary Cde Trees 0 terminal 0 nde 10 rt nde branch interir 1 nde Representatin f prefix cdes with binary trees Prefix cdes can be represented by trees A binary tree cntains ndes with tw branches (labelled as 0 and 1 ) leading t ther ndes starting frm a rt nde A nde frm which branches depart is called an interir nde while a nde frm which n branches depart is called a terminal nde A prefix cde can be cnstructed by assigning letters f the alphabet A t terminal ndes f a binary tree Thmas Wiegand Digital Image Cmmunicatin 22 / 54

23 Surce Cding Fundamentals Lssless Cding Parsing f Prefix Cdes Given the cde wrd assignment t terminal ndes f the binary tree, the parsing rule fr this prefix cde is given as fllws: 1 Set the current nde n i equal t the rt nde. 2 Read the next bit b frm the bitstream. 3 Fllw the branch labeled with the value f b frm the current nde n i t the descendant nde n j. 4 If n j is a terminal nde, return the assciated alphabet letter and prceed with step 1. Otherwise, set the current nde n i equal t n j and repeat the previus tw steps. Prefix cdes are nt nly uniquely decdable, but als instantaneusly decdable. Thmas Wiegand Digital Image Cmmunicatin 23 / 54

24 Surce Cding Fundamentals Lssless Cding Kraft Inequality fr Prefix Cdes Prperty f cdewrd length fr prefix cdes Assume fully balanced tree with depth l max (=lngest cde wrd) Cdewrds assigned t ndes with cdewrd length l(a k ) l max Each chice with l(a k ) l max eliminates 2 lmax l(a k) pssibilities f cde wrd assignment at level l max, fr example: l max l(a k ) = 0, ne ptin is cvered l max l(a k ) = 1, tw ptins are cvered Number f terminal ndes is less than r equal t number f terminal ndes in balanced tree with depth l max, which is 2 lmax K 1 k=0 2 lmax l(a k) 2 lmax (25) Thmas Wiegand Digital Image Cmmunicatin 24 / 54

25 Surce Cding Fundamentals Lssless Cding Kraft Inequality and Unique Decdability Kraft inequality fr lssless cdes Observatin fr prefix cdes can be generalized Can be shwn that the Kraft inequality ζ(γ) = K 1 k=0 2 l(a k) 1 (26) is a necessary cnditin fr the unique decdability f a cde γ Prf can be fund in [Cver and Thmas, 2006, p. 116] r [Wiegand and Schwarz, 2011, p. 25] Thmas Wiegand Digital Image Cmmunicatin 25 / 54

26 Surce Cding Fundamentals Lssless Cding Bund fr Scalar Variable-Length Cding: The Entrpy Lwer bund fr average cdewrd length Based n the Kraft inequality it can be shwn that the average cdewrd length l fr uniquely decdable scalar cdes is bunded by M 1 l H(S) = E { lg 2 p(s)} = p(a i ) lg 2 p(a i ) (27) An example prf can be fund in [Wiegand and Schwarz, 2011, p. 27] The lwer bund H(S) is called the entrpy f the surce S i=0 Redundancy f a cde The redundancy f a scalar cde is given by the difference ϱ = l H(S) 0 (28) The redundancy is zer nly if fr all alphabet letters a i the length f the crrespnding cdewrds are l(a i ) = lg 2 p(a i ) The redundancy can nly be zer if all prbability masses p(a i ) represent negative integer pwers f 2 Thmas Wiegand Digital Image Cmmunicatin 26 / 54

27 Surce Cding Fundamentals Lssless Cding Upper Bund fr Minimum Average Cdewrd Length Bunds fr achievable average cdewrd length The fundamental lwer bund fr l is given by the entrpy H(S), but it is nt always achievable (cdewrds must have integer number f bits) A cde with the cdewrd lengths l(a i ) = lg 2 p(a i ), a i A can always be cnstructed, yielding the upper bund l = < M 1 i=0 M 1 i=0 p(a i ) lg 2 p(a i ) p(a i ) (1 lg 2 p(a i )) = H(S) + 1 (29) The minimum achievable average cdewrd length l min is bunded by H(S) l min < H(S) + 1 (30) Thmas Wiegand Digital Image Cmmunicatin 27 / 54

28 Surce Cding Fundamentals Lssless Cding Entrpy f a Binary Surce Binary entrpy functin A binary surce has prbabilities p(0) = p and p(1) = 1 p The entrpy f the binary surce is given as H(S) = p lg 2 p (1 p) lg 2 (1 p) = H b (p) (31) with H b (x) being the s-called binary entrpy functin R [bit/symbl] P(a ) 0 Thmas Wiegand Digital Image Cmmunicatin 28 / 54

29 Surce Cding Fundamentals Lssless Cding The Huffman Algrithm Cnstructing lssless cdes with minimum redundancy Questin: Hw t generate a prefix cde with minimum redundancy? The answer was given by D. A. Huffman in 1952 [Huffman, 1952] The Huffman algrithm always yields a prefix cde with minimum redundancy Fr a prf that Huffman cdes are ptimal instantaneus cdes (with minimum expected length), see [Cver and Thmas, 2006, p. 124ff] The Huffman algrithm 1 Given an ensemble representing a memryless discrete surce 2 Pick the tw symbls with lwest prbabilities and merge them int a new auxiliary symbl and calculate its prbability 3 If mre than ne symbl remains, repeat the previus step 4 Cnvert the cde tree int a prefix cde Thmas Wiegand Digital Image Cmmunicatin 29 / 54

30 Surce Cding Fundamentals Lssless Cding Example fr the Design f a Huffman Cde P(7)=0.29 P(6)= P= P(5)=0.16 P(4)=0.14 P(3)=0.07 P(2)=0.03 P(1)=0.02 P(0)= P= P= P= P= P= Thmas Wiegand Digital Image Cmmunicatin 30 / 54

31 Surce Cding Fundamentals Lssless Cding Cnditinal Huffman Cdes Reduce average cdewrd length fr surces with memry Randm prcess {S n } with memry: Design VLC fr cnditinal pmf Example: Statinary discrete Markv prcess, A = {a 0, a 1, a 2} Cnditinal pmfs p(a a k ) = P (S n =a S n 1 =a k ) with k = 0, 1, 2 a a 0 a 1 a 2 entrpy p(a a 0 ) H(S n a 0 ) = p(a a 1 ) H(S n a 1 ) = H(S n S n 1 ) = p(a a 2 ) H(S n a 2 ) = p(a) H(S) = Design Huffman cde fr cnditinal pmfs Huffman cdes fr cnditinal pmfs Huffman cde a i S n 1 = a 0 S n 1 = a 1 S n 1 = a 2 fr marginal pmf a a a l 0 = 1.1 l1 = 1.2 l2 = 1.4 l = l c = Thmas Wiegand Digital Image Cmmunicatin 31 / 54

32 Surce Cding Fundamentals Lssless Cding Average Cdewrd Length f Cnditinal Huffman Cdes Bunds fr the average cdewrd length Average cdewrd length l k = l(s n 1 =a k ) is bunded by H(S n a k ) l k < H(S n a k ) + 1 (32) with cnditinal entrpy f S n given event {S n 1 =a k } M 1 H(S n a k ) = H(S n S n 1 =a k ) = p(a i a k ) lg 2 p(a i a k ) (33) Taking the expectatin yields M 1 k=0 p(a k ) H(S n a k ) M 1 k=0 i=0 p(a k ) l k < M 1 Thmas Wiegand Digital Image Cmmunicatin 32 / 54 k=0 p(a k ) H(S n a k ) + 1 (34) where the average cdewrd length f the cnditinal cde is given by l = M 1 k=0 p(a k ) l k (35)

33 Surce Cding Fundamentals Lssless Cding Cnditinal Entrpy Minimum average cdewrd length and cnditinal entrpy The lwer bund fr the average cdewrd length f cnditinal cdes is called the cnditinal entrpy f S n given the randm variable S n 1 H(S n S n 1 ) = E { lg 2 p(s n S n 1 )} = M 1 k=0 p(a k ) H(S n S n 1 =a k ) (36) The minimum achievable average cdewrd length l min fr cnditinal cdes is bunded by H(S n S n 1 ) l min < H(S n S n 1 ) + 1 (37) Cnditining may reduce the average cdewrd length The cnditinal entrpy is less than r equal t the marginal entrpy (with equality if and nly if the prcess {S n } is i.i.d.) H(S n S n 1 ) H(S n ) (38) Thmas Wiegand Digital Image Cmmunicatin 33 / 54

34 Surce Cding Fundamentals Lssless Cding Huffman Cding f Vectrs Jint cding f blcks f N symbls Statinary discrete randm surces S = {S n } with an M-ary alphabet A = {a 0,, a M 1 } N symbls are cded jintly: Design Huffman cde fr jint pmf p(a 0,, a N 1 ) = P (S n =a 0,, S n+n 1 =a N 1 ) Minimum average cdewrd length l min per symbl is bunded by H(S n,, S n+n 1 ) N l min < H(S n,, S n+n 1 ) N + 1 N (39) where the blck entrpy is defined by H(S n,, S n+n 1 ) = E { lg 2 p(s n,, S n+n 1 )} (40) The fllwing limit is called entrpy rate H(S 0,, S N 1 ) H(S) = lim N N (41) Thmas Wiegand Digital Image Cmmunicatin 34 / 54

35 Surce Cding Fundamentals Lssless Cding Entrpy Rate Entrpy rate as fundamental bund fr lssless cding Entrpy rate H(S 0,, S N 1 ) H(S) = lim N N The limit in (42) always exists fr statinary surces [Gallager, 1968] (42) The entrpy rate H(S) is the greatest lwer bund fr the average cdewrd length l per symbl fr all lssless cding techniques l H(S) (43) Entrpy rate fr iid prcesses H(S) = E { lg lim 2 p(s 0, S 1,, S N 1 )} N N = lim N N 1 n=0 E { lg 2 p(s n )} = H(S) (44) N Thmas Wiegand Digital Image Cmmunicatin 35 / 54

36 Surce Cding Fundamentals Lssless Cding Entrpy Rate fr Markv Prcesses Entrpy rate fr statinary Markv prcesses E { lg H(S) = lim 2 p(s 0, S 1,, S N 1 )} N N E { lg = lim 2 p(s 0 )} + N 1 n=1 E { lg 2 p(s n S n 1 )} N N = H(S n S n 1 ) (45) Example: Jint Huffman cding f 2 events and l vs. table size N C a i a k p(a i, a k ) cdewrds a 0 a a 0 a a 0 a a 1 a a 1 a a 1 a a 2 a a 2 a a 2 a N l NC Thmas Wiegand Digital Image Cmmunicatin 36 / 54

37 Surce Cding Fundamentals Lssless Cding Elias Cding and Arithmetic Cding Mtivatin Main drawbacks f blck Huffman cdes: Large table sizes Anther class f uniquely decdable cdes are Elias cdes and arithmetic cdes Encding: Mapping f a string f N symbls s = {s 0, s 1,..., s N 1 } nt a string f K bits b = {b 0, b 1,..., b K 1 } γ : s b (46) Decding r parsing: Mapping the bit string nt the string f symbls Cmplexity f cde cnstructin: Linear per symbl γ 1 : b s (47) Cnstructin methd: Recursive subdivisin f the unit interval [0, 1) Iterative encding and decding prcedures Thmas Wiegand Digital Image Cmmunicatin 37 / 54

38 Surce Cding Fundamentals Lssless Cding Mapping f Symbl Sequences t Numbers Representing a symbl sequence by a number in the interval [0, 1) Cnsider a sequence f N randm variables S (N) = {S 0, S 1,..., S N 1 }, each S i with alphabet f size M i Order alphabet symbls and let η i (s i ) be a functin that returns the crrespnding symbl index in the range frm 0 t M i 1 A realizatin s (N) = {s 0, s 1,..., s N 1 } f S (N) can be represented by a unqiue real number r [0, 1) r = ζ ( s (N)) = N 1 i=0 η i (s i ) B i with B i = i j=0 M 1 j (48) Nte that when all M j = M, the basis simplifies t B i = M i 1 Define cmparisn peratrs fr symbl sequences s (N) a > s (N) b ζ ( s (N) ) ( (N)) a > ζ s b (49) Thmas Wiegand Digital Image Cmmunicatin 38 / 54

39 Surce Cding Fundamentals Lssless Cding Mapping f Symbl Sequences t Intervals Representing a symbl sequence by a prbability interval The prbability f the symbl sequence s (N) can be written as p ( s (N)) = P ( S (N) =s (N)) = P ( S (N) s (N)) P ( S (N) <s (N)) (50) A symbl sequence s (N) can be represented by an interval I N between tw successive levels f the cumulative prbability mass functin I N = [ L N, L N + W N ) = [P ( S (N) <s (N)), P ( S (N) s (N))) (51) with L N = P ( S (N) <s (N)) and W N = P ( S (N) =s (N)) (52) The intervals I N fr different symbl sequences s (N) are disjint A symbl sequence s (N) can be uniquely represented by any value v inside the interval I N Thmas Wiegand Digital Image Cmmunicatin 39 / 54

40 Surce Cding Fundamentals Lssless Cding Hw Many Bits fr Identifying an Interval? Bit sequence b = {b 0, b 1,, b K 1 } f K bits fr representing an interval I N Represent the value v as binary fractin v = K 1 i=0 b i 2 i 1 = 0.b 0b 1 b K 1 with v I N ( s (N) ) (53) Size f interval p ( s (N)) gverns number K f required bits p ( s (N)) =1/2 B={.0,.1} p ( s (N)) =1/4 B={.00,.01,.10,.11} p ( s (N)) =1/8 B={.000,.001,.010,.011,.100,.101,.110,.111} Minimum number f bits is K = K ( s (N)) = lg 2 p ( s (N)) (54) Binary number v that identifies the interval I N and determines the bit string b v = L N 2 K 2 K (55) Thmas Wiegand Digital Image Cmmunicatin 40 / 54

41 Surce Cding Fundamentals Lssless Cding Redundancy f Elias Cding Average cdewrd length Cnsider cding f N symbls Average cdewrd length per symbl l = 1 } {K(S N E (N) ) = 1 { N E lg2 p(s (N) ) } (56) Applying inequalities x x and x < x + 1 yields 1 { } N E lg 2 p(s (N) ) l < 1 { } N E 1 lg 2 p(s (N) ) (57) Average cdewrd length is bunded 1 N H(S(N) ) l < 1 N H(S(N) ) + 1 N (58) The redundancy appraches zer as the number N f cded symbls appraches infinity (similar t blck Huffman cding) Thmas Wiegand Digital Image Cmmunicatin 41 / 54

42 Surce Cding Fundamentals Lssless Cding Example: IID Surce Example fr an iid surce fr which an ptimum Huffman cde exists symbl a k pmf p(a k ) Huffman cde a 0= A 0.25 = a 1= B 0.25 = a 2= C 0.50 = Suppse we intend t send the symbl string s = CABAC Using the Huffman cde, the bit string wuld be b = An alternative t Huffman cding is Elias cding The prbability p(s) is given by p(s) = p( C ) p( A ) p( B ) p( A ) p( C ) = = The size f the bit string is lg 2 p(s) = 8 bit Thmas Wiegand Digital Image Cmmunicatin 42 / 54

43 Surce Cding Fundamentals Lssless Cding Iterative Algrithm fr Elias Cding Iterative interval subdivisin Cnsider sub-sequences s (n) = {s 0, s 1,, s n 1 } with 1 n N Iteratin rule fr the interval width W n+1 = W n p(s n s 0, s 1,..., s n 1 ) (59) Iteratin rule f lwer interval bundary L n+1 = L n + W n c(s n s 0, s 1,..., s n 1 ) (60) with the cumulative prbability mass functin (cmf) c( ) being defined as c(s n s 0, s 1,..., s n 1 ) = p(a s 0, s 1,..., s n 1 ) (61) a A n: a<s n An interval I n+1 is always nested inside the interval I n Thmas Wiegand Digital Image Cmmunicatin 43 / 54

44 Surce Cding Fundamentals Lssless Cding Iterative Interval Subdivisin fr Different Surces Iterative interval subdivisin fr different surces Derivatin abve fr general case f dependent and differently distributed randm variables Fr i.i.d. surces, interval refinement can be simplified W n = W n 1 p(s n 1 ) (62) L n = L n 1 + W n 1 c(s n 1 ) (63) Fr Markv surces with cnditinal pmf p(s n s n 1 ) and cnditinal cmf c(s n s n 1 ) W n = W n 1 p(s n 1 s n 2 ) (64) L n = L n 1 + W n 1 c(s n 1 s n 2 ) (65) Fr nn-statinary surces, the prbabilities p( ) can be adapted during the cding prcess Thmas Wiegand Digital Image Cmmunicatin 44 / 54

45 Surce Cding Fundamentals Lssless Cding Encding Algrithm fr Elias Cdes Encding algrithm: 1 Given is a sequence {s 0,, s N 1 } f N symbls 2 Initializatin f the iterative prcess by W 0 = 1, L 0 = 0 3 Fr each n = 0, 1,, N 1, determine the interval I n+1 by W n+1 = W n p(s n s 0,, s n 1 ) L n+1 = L n + W n c(s n s 0,, s n 1 ) 4 Determine the cdewrd length by K = lg 2 W N 5 Transmit the cdewrd b f K bits that represents the fractinal part f v = L N 2 K 2 K Thmas Wiegand Digital Image Cmmunicatin 45 / 54

46 Surce Cding Fundamentals Lssless Cding Example fr Elias Encding s 0= C s 1= A s 2= B W 1 = W 0 p( C ) W 2 = W 1 p( A ) W 3 = W 2 p( B ) = = 2 1 = = 2 3 = = 2 5 = (0.1) b = (0.001) b = ( ) b L 1 = L 0 + W 0 c( C ) L 2 = L 1 + W 1 c( A ) L 3 = L 2 + W 2 c( B ) = L = L = L = 2 1 = 2 1 = = (0.1) b = (0.100) b = ( ) b s 3= A s 4= C terminatin W 4 = W 3 p( A ) W 5 = W 4 p( C ) K = lg 2 W 5 = 8 = = 2 7 = = 2 8 = ( ) b = ( ) b v = L 5 2 K 2 K L 4 = L 3 + W 3 c( A ) L 5 = L 4 + W 4 c( C ) = = L = L = = b = = ( ) b = ( ) b Thmas Wiegand Digital Image Cmmunicatin 46 / 54

47 Surce Cding Fundamentals Lssless Cding Illustratin f Iterative Cding Thmas Wiegand Digital Image Cmmunicatin 47 / 54

48 Surce Cding Fundamentals Lssless Cding Decding Algrithm fr Elias Cdes Decding algrithm: 1 Given is the number N f symbls t be decded and a cdewrd b = {b 0,, b K 1 } f K bits 2 Determine the interval representative v accrding t v = K 1 i=0 b i 2 i 3 Initializatin f the iterative prcess by W 0 = 1, L 0 = 0 4 Fr each n = 0, 1,, N 1, d the fllwing: 1 Fr each a i A n, determine the interval I n+1(a i) by W n+1(a i) = W n p(a i s 0,..., s n 1) L n+1(a i) = L n + W n c(a i s 0,..., s n 1) 2 Select the letter a i A n fr which v I n+1(a i) and set s n = a i, W n+1 = W n+1(a i), L n+1 = L n+1(a i) Thmas Wiegand Digital Image Cmmunicatin 48 / 54

49 Surce Cding Fundamentals Lssless Cding Arithmetic Cding Arithmetic cding as finite precisin implementatin f Elias cding Prblem with Elias cdes: Precisin requirement fr W N and L N Arithmetic cdes: Variant f Elias cdes with fixed-precisin arithmetic Represent pmfs p(a) and cmfs c(a) and width W n with finite number f bits Lss in cding efficiency due t runding is typically negligible Representatin f the lwer interval bundary L n has the structure L n = 0. aaaaa a }{{} settled bits }{{} utstanding bits xxxxx x }{{} active bits }{{} trailing bits where settled bits are nt mdified in fllwing interval updates (can be utput) utstanding bits may be mdified by a carry frm the active bits active bits directly mdified by fllwing interval update Mst practical variant f arithmetic cding is binary arithmetic cding Symbls are first binarized using a variable length cde Decding search is reduced t ne cmparisn Multiplicatin-free algrithms (e.g., M cder) fr binary arithmetic cding Thmas Wiegand Digital Image Cmmunicatin 49 / 54

50 Surce Cding Fundamentals Lssless Cding Cmparisn f Lssless Cding Techniques Example: Markv surce Instantaneus entrpy rate H inst (S, L) = 1 L H(S 0, S 1,..., S L 1 ) Thmas Wiegand Digital Image Cmmunicatin 50 / 54

51 Surce Cding Fundamentals Lssless Cding Cnditinal and Adaptive Cdes Cding f surce with memry and/r varying statistics One apprach wuld be a switch Huffman cde trained n the cnditinal prbabilities The resulting number f Huffman cde tables is ften t large in practice Hence, cnditinal entrpy cding is typically dne using arithmetic cdes In adaptive arithmetic cding, prbabilities p(a k ) are estimated/adapted simultaneusly at encder and decder Statistical dependencies can be explited using s-called cntext mdeling techniques: Cnditinal prbabilities p(a k z k ) with z k being a cntext/state that is simultaneusly cmputed at encder and decder Thmas Wiegand Digital Image Cmmunicatin 51 / 54

52 Surce Cding Fundamentals Lssless Cding Frward and Backward Adaptatin The tw basic appraches fr adaptatin are Frward adaptatin: Gather statistics fr a large enugh blck f surce symbls Transmit adaptatin signal t decder as side infrmatin Disadvantage: Increased bit rate due t side infrmatin Backward adaptatin: Gather statistics simultaneusly at cder and decder Drawback: Errr resilience With tday s packet-switched transmissin systems, an efficient cmbinatin f the tw adaptatin appraches can be achieved: 1 Gather statistics fr the entire packet and prvide initializatin f entrpy cde at the beginning f the packet 2 Cnduct backwards adaptatin fr each symbl inside the packet in rder t minimize the size f the packet Thmas Wiegand Digital Image Cmmunicatin 52 / 54

53 Surce Cding Fundamentals Lssless Cding Illustratin f Adaptive Cding Frward Adaptatin Cmputatin f adaptatin signal Surce symbls Delay Encding Channel Decding Recnstructed symbls Backward Adaptatin Surce symbls Delay Cmputatin f adaptatin signal Encding Channel Cmputatin f adaptatin signal Decding Delay Recnstructed symbls Thmas Wiegand Digital Image Cmmunicatin 53 / 54

54 Surce Cding Fundamentals Lssless Cding Summary Entrpy is the lwer bund fr the average number f bits/symbl fr uniquely decdable scalar cdes Entrpy rate is the lwer bund fr the average number f bits/symbl fr all uniquely decdable lssless cdes Huffman cding is an efficient and simple entrpy cding methd needs cde table can be inefficient fr certain prbabilities difficult t use fr expliting statistical dependencies and time-varying prbabilities Arithmetic cding is a universal methd fr encding strings f symbls des nt need a cde table, but a table fr string prbabilities typically requires serial cmputatin f interval and prbability estimatin update (in case prbabilities are adapted) appraches entrpy fr lng strings is well suited fr expliting statistical dependencies and cding with time-varying prbabilities Thmas Wiegand Digital Image Cmmunicatin 54 / 54