Lec 02 Entropy and Lossless Coding I

Transcription

1 Multmeda Communcaton, Fall 208 Lec 02 Entroy and Lossless Codng I Zhu L Z. L Multmeda Communcaton, Fall 208.

2 Outlne Lecture 0 ReCa Info Theory on Entroy Lossless Entroy Codng Z. L Multmeda Communcaton, Fall 208.2

3 Vdeo Comresson n Summary Comresson Pelne: Z. L Multmeda Communcaton, Fall 208.3

4 Vdeo Codng Standards: Rate-Dstorton Performance Pre-HEVC Z. L Multmeda Communcaton, Fall 208.4

5 PSS over managed IP networks Managed moble core IP networks Z. L Multmeda Communcaton, Fall 208.5

6 MPEG DASH OTT HTTP Adatve Streamng of Vdeo Z. L Multmeda Communcaton, Fall 208.6

7 Outlne Lecture 0 ReCa Info Theory on Entroy Self Info of an event Entroy of the source Relatve Entroy Mutual Info Entroy Codng Thanks for SFU s Prof. Je Lang s sld Z. L Multmeda Communcaton, Fall 208.7

8 Entroy and ts Alcaton Entroy codng: the last art of a comresson system Encoder Transform Quantzato n Entroy codng Losslessly reresent symbols Key dea: Assgn short codes for common symbols Assgn long codes for rare symbols Queston: How to evaluate a comresson method? o Need to know the lower bound we can acheve. o Entroy Z. L Multmeda Communcaton, Fall 208.8

9 Claude Shannon: A dstant relatve of Thomas Edson 932: Went to Unversty of Mchgan. 937: Master thess at MIT became the foundaton of dgtal crcut desgn: o The most mortant, and also the most famous, master's thess of the century 940: PhD, MIT : Bell Lab back to MIT after that 948: The brth of Informaton Theory o A mathematcal theory of communcaton, Bell System Techncal Journal. Z. L Multmeda Communcaton, Fall 208.9

10 Aom Defnton of Informaton Informaton s a measure of uncertanty or surrse Aom : Informaton of an event s a functon of ts robablty: A = f PA. What s the eresson of f? Aom 2: Rare events have hgh nformaton content Water found on Mars!!! Common events have low nformaton content It s ranng n Vancouver. Informaton should be a decreasng functon of the robablty: Stll numerous choces of f. Aom 3: Informaton of two ndeendent events = sum of ndvdual nformaton: If PAB=PAPB AB = A + B. Only the logarthmc functon satsfes these condtons. Z. L Multmeda Communcaton, Fall 208.0

11 Self-nformaton Shannon s Defnton [948]: X: dscrete random varable wth alhabet {A, A2,, AN} Probablty mass functon: = Pr{ X = } Self-nformaton of an event X = : = logb = -logb If b = 2, unt of nformaton s bt - log b P Self nformaton ndcates the number of bts needed to reresent an event. 0 P Z. L Multmeda Communcaton, Fall 208.

12 Entroy of a Random Varable H X = å log Also wrte as H : functon of the dstrbuton of X, not the value of X. Recall: the mean of a functon gx: E å g X = g Entroy s the eected self-nformaton of the r.v. X: æ ö H = E çlog = -E log X è X ø The entroy reresents the mnmal number of bts needed to losslessly reresent one outut of the source. Z. L Multmeda Communcaton, Fall 208.2

13 Eamle PX=0 = /2 PX= = /4 PX=2 = /8 PX=3 = /8 Fnd the entroy of X. Soluton: H X = å log = log 2 + log 4 + log 8 + log 8 = = bts/samle Z. L Multmeda Communcaton, Fall 208.3

14 Eamle A bnary source: only two ossble oututs: 0, Source outut eamle: X=0 =, X==. Entroy of X: H = -log log2- H = 0 when = 0 or = ofed outut, no nformaton H s largest when = /2 ohghest uncertanty oh = bt n ths case Proertes: H 0 H concave roved later Entroy Equal rob mamze entroy Z. L Multmeda Communcaton, Fall 208.4

15 Jont entroy We can get better understandng of the source S by lookng at a block of outut XX2 Xn: The jont robablty of a block of outut: Jont entroy X =, X =, X = 2 2 n n H X, X, X = 2 2 åå å X =, X 2 = 2, Xn = n log X =, X 2 = 2, Xn = n = -E n n X,... X log n Jont entroy s the number of bts requred to reresent the sequence XX2 Xn: Ths s the lower bound for entroy codng. Z. L Multmeda Communcaton, Fall 208.5

16 Condtonal Entroy Condtonal Self-Informaton of an event X =, gven that event Y = y has occurred: y y = log = log y, y Note: y,, y and y are three dfferent dstrbutons: y, 2, y and 3y. Condtonal Entroy HY X: Average cond. self-nfo. Remanng uncertanty about Y gven the knowledge of X. H Y X = H Y X = = - y log y = - y log y = -, ylog y = -E åå, y å å å y y y log y åå Z. L Multmeda Communcaton, Fall 208.6

17 Chan Rule HX, Y = HX + HY X = HY + HX Y Proof: H X, Y = - = - = - åå åå å y y = - åå log y, ylog, ylog, ylog y - + H Y åå, y y, ylog y X = H X + H Y X. Total area: HX, Y HX Y HY X HX HY Smler notaton: H X, Y = -Elog X, Y = -Elog X + log Y X = H X + H Y X Z. L Multmeda Communcaton, Fall 208.7

18 Condtonal Entroy Eamle: for the followng jont dstrbuton, y, fnd HY X. X Y PX: [ ½, ¼, /8, /8 ] >> HX = 7/4 b /8 /6 /32 /32 PY: [ ¼, ¼, ¼, ¼ ] >> HY = 2 bts 2 /6 /8 /32 /32 3 /6 /6 /6 /6 4 / Indeed, HX Y = HX, Y HY= 27/8 2 = /8 bts Z. L Multmeda Communcaton, Fall 208.8

19 General Chan Rule Addng HZ: HX, Y Z + Hz = HX, Y, Z = Hz + HX Z + HY X, Z General form of chan rule: H X, X 2,... X n = å H X X -,... X The jont encodng of a sequence can be broken nto the sequental encodng of each samle, e.g. HX, X 2, X 3 =HX + HX 2 X + HX 3 X 2, X Advantages: Jont encodng needs jont robablty: dffcult Sequental encodng only needs condtonal entroy, can use local neghbors to aromate the condtonal entroy contet-adatve arthmetc codng. n = Z. L Multmeda Communcaton, Fall 208.9

20 General Chan Rule,......, = n n n Proof:.,...,... log,...,... log,...,... log,...,... log,...,...,...,...,...,... å å å å å å Õ å = - = - - = = - = - = - = - = - = n n n n n n n n n n n n n n X X X H X X H Z. L Multmeda Communcaton, Fall

21 General Chan Rule The comlety of the condtonal robablty,... - grows as the ncrease of. In many cases we can aromate the cond. robablty wth some nearest neghbors contets:,...», L a b c b c a b c b a b c b a The low-dm cond rob s more manageable How to measure the qualty of the aromaton? Relatve entroy Z. L Multmeda Communcaton, Fall 208.2

22 Relatve Entroy Cost of Codng wth Wrong Dstr D æ X ö q = å log = E çlog q è q X ø Also known as Kullback Lebler K-L Dstance, Informaton Dvergence, Informaton Gan A measure of the dstance between two dstrbutons: In many alcatons, the true dstrbuton X s unknown, and we only know an estmaton dstrbuton qx What s the neffcency n reresentng X? o The true entroy: o The actual rate: o The dfference: R = -å log R = -å q 2 log R2 - R = D q Z. L Multmeda Communcaton, Fall

23 Relatve Entroy D Proertes: æ X ö q = å log = E çlog q è q X ø D q ³ 0. Proved later. D q = 0 f and only f q =. What f >0, but q=0 for some? D q= Cauton: D q s not a true dstance Not symmetrc n general: D q Dq Does not satsfy trangular nequalty. Z. L Multmeda Communcaton, Fall

24 Relatve Entroy How to make t symmetrc? Many ossbltes, for eamle: 2 D q D q + D q D q D q + D q can be useful for attern classfcaton. Z. L Multmeda Communcaton, Fall

25 Mutual Informaton y: condtonal self-nformaton y = -log y Mutual nformaton between two events: y ; y = - y = log = log, y y A measure of the amount of nformaton that one event contans about another one. or the reducton n the uncertanty of one event due to the knowledge of the other. Note: ; y can be negatve, f y <. Z. L Multmeda Communcaton, Fall

26 Mutual Informaton IX; Y: Mutual nformaton between two random varables:, y I X ; Y = åå, y ; y = åå, ylog y y y æ X, Y ö = D, y y = E, y ç log è X Y ø Mutual nformaton s a relatve entroy: But t s symmetrc: IX; Y = IY; X Dfferent from ; y, IX; Y >=0 due to averagng If X, Y are ndeendent:, y = y I X; Y = 0 Knowng X does not reduce the uncertanty of Y. Z. L Multmeda Communcaton, Fall

27 Entroy and Mutual Informaton. I X ; Y = H X - H X Y, y y I X ; Y = åå, ylog = åå, ylog y åå =, ylog y -, ylog = H X - H X Y y y åå y y 2. Smlarly: I X ; Y = H Y - H Y X 3. IX; Y = HX + HY HX, Y Proof: Eand the defnton: I X ; Y = H X + = åå y H Y -, y H X, Y log, y - log - log y Z. L Multmeda Communcaton, Fall

28 Entroy and Mutual Informaton Total area: HX, Y HX Y IX; Y HY X HX HY It can be seen from ths fgure that IX; X = HX: Proof: Let X = Y n IX; Y = HX + HY HX, Y, or n IX; Y = HX HX Y and use HX X=0. Z. L Multmeda Communcaton, Fall

29 Alcaton of Mutual Informaton a b c b c a b c b a b c b a Mutual nformaton can be used n the otmzaton of contet quantzaton. Eamle: If each neghbor has 26 ossble values a to z, then 5 neghbors have 26 5 combnatons: too many cond robs to estmate. To reduce the number, can grou smlar data attern together contet quantzaton,...» f, Z. L Multmeda Communcaton, Fall

30 Alcaton of Mutual Informaton H X, X 2,... X n = å H X X -,... X,...» f, We need to desgn the functon f to mnmze the condtonal entroy But n = H X f X -,... X H X Y = H X - I X ; Y The roblem s equvalent to mamzng the mutual nformaton between {X} and f, -. For further nfo: Lu and Karam, Mutual Informaton-Based Analyss of JPEG2000 Contets, IEEE Trans Image Processng, VOL. 4, NO. 4, APRIL 2005, Z. L Multmeda Communcaton, Fall

31 Outlne Lecture 0 ReCa Info Theory and Entroy Entroy Codng Pref Codng Kraft-McMllan Inequalty Shannon Codes Z. L Multmeda Communcaton, Fall 208.3

32 Varable Length Codng Desgn the mang from source symbols to codewords Lossless mang Dfferent codewords may have dfferent lengths Goal: mnmzng the average codeword length The entroy s the lower bound. Z. L Multmeda Communcaton, Fall

33 Classes of Codes Non-sngular code: Dfferent nuts are maed to dfferent codewords nvertble. Unquely decodable code: any encoded strng has only one ossble source strng, but may need delay to decode. Pref-free code or smly ref, or nstantaneous: No codeword s a ref of any other codeword. The focus of our studes. Questons: o Characterstc? o How to desgn? o Is t otmal? All codes Non-sngular codes Unquely decodable codes Pref-free codes Z. L Multmeda Communcaton, Fall

34 Pref Code Eamles X Sngular Non-sngular, But not unquely decodable Unquely decodable, but not ref-free Pref-free Need unctuaton 00 Need to look at net bt to decode revous code. Z. L Multmeda Communcaton, Fall

35 Carter-Gll s Conjecture [974] Carter-Gll s Conjecture [974] Every unquely decodable code can be relaced by a ref-free code wth the same set of codeword comostons. So we only need to study ref-free code. Z. L Multmeda Communcaton, Fall

36 Pref-free Code Can be unquely decoded. No codeword s a ref of another one. Also called ref code Goal: construct ref code wth mnmal eected length. Can ut all codewords n a bnary tree: Root node Internal node leaf node 0 Pref-free code contans leaves only. How to eress the requrement mathematcally? Z. L Multmeda Communcaton, Fall

37 Kraft-McMllan Inequalty The characterstc of ref-free codes: The codeword lengths l, =, N of a ref code over an alhabet of sze D=2 satsfes the nequalty N å = 2 - l Conversely, f a set of {l} satsfes the nequalty above, then there ests a ref code wth codeword lengths l, =, N. Z. L Multmeda Communcaton, Fall

38 Kraft-McMllan Inequalty Consder D=2: eand the bnary code tree to full deth L = mal : l s the code length L = 3 N å = 2 -l N L l Û å - 2 = 2 L Eamle: {0, 0, 0, } Number of nodes n the last level: Each code corresonds to a sub-tree: The number of off srngs n the last level: 2 L-l K-M nequalty: # of L-th level offsrngs of all codes s less than 2^L! Z. L Multmeda Communcaton, Fall L 2^3=8 {4, 2, 0, 0}

39 Kraft-McMllan Inequalty K-M nequalty: å = 2 - l Invald code: {0, 0,, 0, }: l =[ ] Leads to more than 2^L offsrng: 2> 2 3 Z. L Multmeda Communcaton, Fall

40 Etended Kraft Inequalty Countably nfnte ref code also satsfes the Kraft nequalty: Has nfnte number of code words. å = 2 - l Eamle: 0, 0, 0, 0, 0, 0, Golomb-Rce code, net lecture Each codeword can be maed to a subnterval n [0, ] that s dsjont wth others revsted n arthmetc codng L = Z. L Multmeda Communcaton, Fall

41 Otmal Codes Advanced Toc How to desgn the ref code wth the mnmal eected length? Otmzaton Problem: fnd {l} to mn l s. t. å å D l -l Lagrangan soluton: Ignore the nteger codeword length constrant for now Assume equalty holds n the Kraft nequalty Mnmze J å l å -l = l + D Z. L Multmeda Communcaton, Fall 208.4

42 Otmal Codes J å Let J = - - l = l å -l = l + l D l ln D D 0 å - l Substtut ng nto D = D - l l = = l ln D ln D -l D = or l * = - log D The otmal codeword length s the self-nformaton of an event. Eected codeword length: L * = å l = å - log H X Entroy of X! D = D Z. L Multmeda Communcaton, Fall

43 Otmal Code Theorem: The eected length L of any ref code s greater or equal to the entroy l * = -log D s not nteger n general. wth equalty holds ff Proof: L ³ H D X -l D = P s Dadc, ½, ¼, /8, /6 å å L - H X = l + log D D å å å l = log D + log = log D D D l D - Ths remnds us the defnton of relatve entroy D q, but we need to normalze D -l. Z. L Multmeda Communcaton, Fall

44 Otmal Code because D q >= 0, and N å = l D - The equalty holds ff both terms are 0: -l D = for ref code. or -log D s an nteger. Z. L Multmeda Communcaton, Fall

45 Otmal Code D-adc: a robablty dstrbuton s called D-adc wth resect to D f each robablty s equal to D -n for some nteger n: Eamle: D=2, P= {/2, /4, /8, /8} Therefore the otmalty can be acheved by ref code ff the dstrbuton s D-adc. Prevous eamle: - log = {, 2,3,3} D Possble codewords: o {0, 0, 0, } Z. L Multmeda Communcaton, Fall

46 Shannon Code: Bounds on Otmal Code Code length s not nteger n general. D - l log * = Practcal codewords have to be nteger. ú ú ù ê ê é = D l log Shannon Code: Is ths a vald ref code? Check Kraft nequalty. log log = = = å å å å - ú ú ù ê ê é - - l D D D D D log log + < D D l ú ú ù ê ê é = D l log + < X H L X H D D Yes! Ths s just one choce. May not be otmal see eamle later Z. L Multmeda Communcaton, Fall

47 Otmal Code The otmal code wth nteger lengths should be better than Shannon code H D X * L H X D + To reduce the overhead er symbol: Encode a block of symbols {, 2,, n} together Ln = å, 2,..., n l, 2,..., n = E n n H X, X,..., X E l,,..., H X { l,,..., } 2 n { }, X,..., X 2 n 2 n 2 n + Assume..d. samles: H X, X 2,..., X n = nh X H X L H X + n n L n H X entroy rate f statonary. Z. L Multmeda Communcaton, Fall

48 Otmal Code Imact of wrong df: what s the enalty f the df we use s dfferent from the true df? True df: Codeword length: l Estmated df: q Eected length: ElX + + < + q D H X l E q D H Proof: assume Shannon code ú ú ù ê ê é = log q l å å ø ö ç è æ + < ú ú ù ê ê é = log log q q X E l log + + = ø ö ç ç è æ + ø ö ç è æ = å q D X H q The lower bound s derved smlarly. Z. L Multmeda Communcaton, Fall

49 Shannon Code s not otmal Eamle: Bnary r.v. X: 0=0.9999, = Entroy: bts/samle Assgn bnary codewords by Shannon code: l = é log ê ù ú é ù log 2 =. ê ú é ù log 2 = 4. ê ú Eected length: =.003. Wthn the range of [HX, HX + ]. But we can easly beat ths by the code {0, } Eected length:. Z. L Multmeda Communcaton, Fall

50 Summary Entroy: H Condtonal Entroy: HX Y Mutual Info: IX, Y = Relatve Entroy: DX Y K-M Inequalty: Bounds on Pref Codes: Z. L Multmeda Communcaton, Fall