Surce Cding and Cmpressin Heik Schwarz Cntact: Dr.-Ing. Heik Schwarz heik.schwarz@hhi.fraunhfer.de Heik Schwarz Surce Cding and Cmpressin September 22, 2013 1 / 60
PartI: Surce Cding Fundamentals Heik Schwarz Surce Cding and Cmpressin September 22, 2013 31 / 60
Prbability, Randm Variables and Randm Prcesses Prbability, Randm Variables and Randm Prcesses Heik Schwarz Surce Cding and Cmpressin September 22, 2013 32 / 60
Prbability, Randm Variables and Randm Prcesses Outline Part I: Surce Cding Fundamentals Review: Prbability, Randm Variables and Randm Prcesses Prbability Randm Variables Randm Prcesses Lssless Surce Cding Rate-Distrtin Thery Quantizatin Predictive Cding Transfrm Cding Part II: Applicatin in Image and Vide Cding Still Image Cding / Intra-Picture Cding Hybrid Vide Cding (Frm MPEG-2 Vide t H.265/HEVC) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 33 / 60
Prbability, Randm Variables and Randm Prcesses Prbability Prbability Prbability thery: Branch f mathematics fr descriptin and mdelling f randm events Mdern prbability thery the aximatic definitin f prbability intrduced by Klmgrv Heik Schwarz Surce Cding and Cmpressin September 22, 2013 34 / 60
Prbability, Randm Variables and Randm Prcesses Prbability Definitin f Prbability Experiment with an uncertain utcme: Randm experiment Unin f all pssible utcmes ζ f the randm experiment: Certain event r sample space O f the randm experiment Event: Subset A O Prbability: Measure P (A) assigned t A satisfying the fllwing three axims 1 Prbabilities are nn-negative real numbers: P (A) 0, A O 2 Prbability f the certain event: P (O) = 1 3 If {A i : i = 0, 1, } is a cuntable set f events such that A i A j = fr i j, then ( ) P A i = P (A i ) (14) i i Heik Schwarz Surce Cding and Cmpressin September 22, 2013 35 / 60
Prbability, Randm Variables and Randm Prcesses Prbability Independence and Cnditinal Prbability Tw events A i and A j are independent if P (A i A j ) = P (A i ) P (A j ) (15) The cnditinal prbability f an event A i given anther event A j, with P (A j ) > 0 is P (A i A j ) = P (A i A j ) P (A j ) (16) Direct cnsequence: Bayes therem P (A i A j ) = P (A j A i ) P (A i) P (A j ) with P (A i ), P (A j ) > 0 (17) Definitins (15) and (16) als imply that, if A i and A j are independent and P (A j ) > 0, then P (A i A j ) = P (A i ) (18) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 36 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Randm Variables Randm variable S: Functin f the sample space O that assigns a real value S(ζ) t each utcme ζ O f a randm experiment Define: Cumulative distributin functin (cdf) f a randm variable S: F S (s) = P (S s) = P ( {ζ : S(ζ) s} ) (19) Prperties f cdfs: F S (s) is nn-decreasing F S ( ) = 0 F S ( ) = 1 Heik Schwarz Surce Cding and Cmpressin September 22, 2013 37 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Jint Cumulative Distributin Functin Jint cdf r jint distributin f tw randm variables X and Y F XY (x, y) = P (X x, Y y) (20) N-dimensinal randm vectr S = (S 0,, S N 1 ) T : Vectr f randm variables S 0, S 1,, S N 1 N-dimensinal cdf, jint cdf, r jint distributin: F S (s) = P (S s) = P (S 0 s 0,, S N 1 s N 1 ) (21) with S = (S 0,, S N 1 ) T being a randm vectr Jint cdf f tw randm vectrs X and Y F XY (x, y) = P (X x, Y y) (22) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 38 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Cnditinal Cumulative Distributin Functin Cnditinal cdf f randm variable S given event B with P (B) > 0 F S B (s B) = P (S s B) = P ({S s} B) P (B) (23) Cnditinal cdf f a randm variable X given anther randm variable Y F X Y (x y) = F XY (x, y) F Y (y) = P (X x, Y y) P (Y y) (24) Cnditinal cdf f a randm vectr X given anther randm vectr Y F X Y (x y) = F XY (x, y) F Y (y) (25) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 39 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Cntinuus Randm Variables A randm variables S is called a cntinuus randm variable, if and nly if its cdf F S (s) is a cntinuus functin Define: Prbability density functin (pdf) fr cntinuus randm variables f S (s) = df S(s) ds F S (s) = s f S (t) dt (26) Prperties f pdfs: f S (s) 0, s f S(t) dt = 1 Heik Schwarz Surce Cding and Cmpressin September 22, 2013 40 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Examples fr Pdfs Unifrm pdf: f S (s) = { 1/A : A/2 s A/2 0 : therwise, A > 0 (27) Laplacian pdf: f S (s) = 1 e s µ S 2/σS, σ S > 0 (28) σ S 2 Gaussian pdf: f S (s) = 1 e (s µ S) 2 /(2σ 2 S ), σ S > 0 (29) σ S 2π Heik Schwarz Surce Cding and Cmpressin September 22, 2013 41 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Generalized Gaussian Distributin f S (s) = β 2 α Γ(1/β) β e ( x µ /α) Γ(z) = 0 e t t z 1 dt (30) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 42 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Jint and Cnditinal Pdfs N-dimensinal pdf, jint pdf, r jint density f S (s) = N F S (s) s 0 s N 1 (31) Cnditinal pdf r cnditinal density f S B (s B) f a randm variable S given an event B f S B (s B) = d F S B(s B) d s (32) Cnditinal density f a randm variable X given anther randm variable Y f X Y (x y) = f XY (x, y) f Y (y) (33) Cnditinal density f a randm vectr X given anther randm vectr Y f X Y (x y) = f XY (x, y) f Y (y) (34) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 43 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Discrete Randm Variables A randm variable S is called a discrete randm variable, if and nly if its cdf F S (s) represents a staircase functin Discrete randm variable S takes values f cuntable set A = {a 0, a 1,...} Define: Prbability mass functin (pmf) fr discrete randm variables: p S (a) = P (S = a) = P ( {ζ : S(ζ)= a} ) (35) Cdf f discrete randm variable F S (s) = a s p(a) (36) Pdf can be cnstructed using the Dirac delta functin δ f S (s) = a A δ(s a) p S (a) (37) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 44 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Examples fr Pmfs Binary pmf: A = {a 0, a 1 } p S (a 0 ) = p, p S (a 1 ) = 1 p (38) Unifrm pmf: A = {a 0, a 1,, a M 1 } p S (a i ) = 1/M a i A (39) Gemetric pmf: A = {a 0, a 1, } p S (a i ) = (1 p) p i a i A (40) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 45 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Jint and Cnditinal Pmfs N-dimensinal pmf r jint pmf fr a randm vectr S = (S 0,, S N 1 ) T p S (a) = P (S = a) = P (S 0 = a 0,, S N 1 = a N 1 ) (41) Jint pmf f tw randm vectrs X and Y : p XY (a x, a y ) Cnditinal pmf p S B (a B) f a randm variable S given an event B, with P (B) > 0 p S B (a B) = P (S = a B) (42) Cnditinal pmf f a randm variable X given anther randm variable Y p X Y (a x a y ) = p XY (a x, a y ) p Y (a y ) (43) Cnditinal pmf f a randm vectr X given anther randm vectr Y p X Y (a x a y ) = p XY (a x, a y ) p Y (a y ) (44) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 46 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Example fr a Jint Pmf Fr example, samples in picture and vide signals typically shw strng statistical dependencies Belw: Histgram f tw hrizntally adjacent sampels fr the picture Lena Relative frequency f ccurence Amplitude f adjacent pixel Amplitude f current pixel Heik Schwarz Surce Cding and Cmpressin September 22, 2013 47 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Expectatin Expectatin value r expected value f a cntinuus randm variable S E{g(S)} = f a discrete randm variable S g(s) f S (s) ds (45) E{g(S)} = a A g(a) p S (a) (46) Imprtant expectatin values are mean µ S and variance σ 2 S µ S = E{S} and σ 2 S = E { (S µ s ) 2} (47) Expectatin value f a functin g(s) f a set f N randm variables S = {S 0,, S N 1 } E{g(S)} = g(s) f S (s) ds (48) R N Heik Schwarz Surce Cding and Cmpressin September 22, 2013 48 / 60
Prbability, Randm Variables and Randm Prcesses Randm Variables Cnditinal Expectatin Cnditinal expectatin value f functin g(s) given an event B, with P (B) > 0 E{g(S) B} = g(s) f S B (s B) ds (49) Cnditinal expectatin value f functin g(x) given a particular value y fr anther randm variable Y E{g(X) y} = E{g(X) Y =y} = Nte: E{g(X) y} is a deterministic functin f y g(x) f X Y (x, y) dx (50) Cnditinal expectatin value f functin g(x) given a randm variable Y, E{g(X) Y } = g(x) f X Y (x, Y ) dx, (51) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 49 / 60 is anther randm variable
Prbability, Randm Variables and Randm Prcesses Randm Variables Iterative Expectatin Rule Expectatin value E{Z} f a randm variable Z = E{g(X) Y } E{E{g(X) Y }} = = = = E{g(X) y} f Y (y) dy ( g(x) f X Y (x, y) dx ( g(x) g(x) f X (x) dx ) f Y (y) dy ) dx f X Y (x, y) f Y (y) dy = E{g(X)} (52) = E{E{g(X) Y }} des nt depend n the statistical prperties f the randm variable Y, but nly n thse f X Heik Schwarz Surce Cding and Cmpressin September 22, 2013 50 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Randm Prcesses Series f randm experiments at time instants t n, with n = 0, 1, 2,... Outcme f experiment: Randm variable S n = S(t n ) Discrete-time randm prcess: Series f randm variables S = {S n } Statistical prperties f discrete-time randm prcess S: N-th rder jint cdf F Sk (s) = P (S (N) k s) = P (S k s 0,, S k+n 1 s N 1 ) (53) Cntinuus randm prcess f Sk (s) = Discrete randm prcess N s 0 s N 1 F Sk (s) (54) F Sk (s) = A N prduct space f the alphabets A n and a A N p Sk (a) (55) p Sk (a) = P (S k = a 0,, S k+n 1 = a N 1 ) (56) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 51 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Autcvariance and Autcrrelatin Matrix N-th rder autcvariance matrix { ( ) ( C N (t k ) = E S (N) k µ N (t k ) N-th rder autcrrelatin matrix { ( ) ( R N (t k ) = E S (N) k ) } T k µ N (t k ) S (N) S (N) k ) T } (57) (58) Nte the fllwing relatinship { ( ) ( C N (t k ) = E S (N) k µ N (t k ) { ( ) ( = E S (N) k S (N) k { µ N (t k ) E S (N) k ) } T k µ N (t k ) S (N) ) } T { E S (N) k } µ N (t k ) T } T + µn (t k ) µ N (t k ) T = R N (t k ) µ N (t k ) µ N (t k ) T (59) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 52 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Statinary Randm Prcess Statinary randm prcess: Statistical prperties are invariant t a shift in time = F Sk (s), f Sk (s) and p Sk (a) are independent f t k and are dented by F S (s), f S (s) and p S (a), respectively = µ N (t k ), C N (t k ) and R N (t k ) are independent f t k and are dented by µ N, C N and R N, respectively N-th rder autcvariance matrix { C N = E (S (N) µ N )(S (N) µ N ) T} (60) is a symmetric Teplitz matrix 1 ρ 1 ρ 2 ρ N 1 ρ 1 1 ρ 1 ρ N 2 C N = σs 2 ρ 2 ρ 1 1 ρ N 3......... ρ N 1 ρ N 2 ρ N 3 1 with ρ k = 1 { } σs 2 E (S l µ S ) (S l+k µ S ) (61) (62) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 53 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Memryless and IID Randm Prcesses Memryless randm prcess: Randm prcess S = {S n } fr which the randm variables S n are independent Independent and identical distributed (iid) randm prcess: Statinary and memryless randm prcess N-th rder cdf F S (s), pdf f S (s), and pmf p S (a) fr iid prcesses, with s = (s 0,, s N 1 ) T and a = (a 0,, a N 1 ) T F S (s) = f S (s) = p S (a) = N 1 k=0 N 1 k=0 N 1 k=0 F S (s k ) (63) f S (s k ) (64) p S (a k ) (65) F S (s), f S (s), and p S (a) are the marginal cdf, pdf, and pmf, respectively Heik Schwarz Surce Cding and Cmpressin September 22, 2013 54 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Markv Prcesses Markv prcess: Future utcmes d nt depend n past utcmes, but nly n the present utcme, P (S n s n S n 1 =s n 1, ) = P (S n s n S n 1 =s n 1 ) (66) Discrete Markv prcesses p Sn (a n a n 1, ) = p Sn (a n a n 1 ) (67) Example fr a discrete Markv prcess a a 0 a 1 a 2 p(a a 0 ) 0.90 0.05 0.05 p(a a 1 ) 0.15 0.80 0.05 p(a a 2 ) 0.25 0.15 0.60 p(a) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 55 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Cntinuus Markv Prcesses Cntinuus Markv prcesses f Sn (s n s n 1, ) = f Sn (s n s n 1 ) (68) Cnstructin f cntinuus statinary Markv prcess S = {S n } with mean µ S, given a zer-mean iid prcess Z = {Z n } S n = Z n + ρ (S n 1 µ S ) + µ S, with ρ < 1 (69) = Variance σ 2 S f statinary Markv prcess S σ 2 S = E { (S n µ S ) 2} = E { (Z n + ρ (S n 1 µ S ) ) 2} = = Autcvariance functin f statinary Markv prcess S σ2 Z 1 ρ 2 (70) φ k,l = φ k l = E{(S k µ S )(S l µ S )} = σ 2 S ρ k l (71) Heik Schwarz Surce Cding and Cmpressin September 22, 2013 56 / 60
Prbability, Randm Variables and Randm Prcesses Randm Prcesses Gaussian Prcesses Gaussian prcess: Cntinuus prcess S = {S n } with the prperty that all finite cllectins f randm variables S n represent Gaussian randm vectrs N-th rder pdf f statinary Gaussian prcess with N-th rder autcrrelatin matrix C N and mean µ S f S (s) = 1 (2π)N C N e 1 2 (s µ S )T C 1 N (s µ S ) with µ S = Statinary Gauss-Markv prcess: Statinary prcess that is a Gaussian prcess and a Markv prcess IID prcess Z = {Z n} in (69) has a Gaussian pdf Statistical prperties are cmpletely determined by mean µ S variance σ 2 S crrelatin factr ρ s(t) µ s. µ S (72) 1 0 10 20 30 40 50 t Heik Schwarz Surce Cding and Cmpressin September 22, 2013 57 / 60 1 0.5 0 0.5
Prbability, Randm Variables and Randm Prcesses Chapter Summary Chapter Summary Randm variables Discrete and cntinuus randm variables Cumulative distributin functin (cdf) Prbability density functin (pdf) Prbability mass functin (pmf) Jint and cnditinal cdfs, pdfs, pmfs Expectatin values and cnditinal expectatin values Randm prcesses Statinary prcesses Memryless prcesses IID prcesses Markv prcesses Gaussian prcesses Gauss-Markv prcesses Heik Schwarz Surce Cding and Cmpressin September 22, 2013 58 / 60
Prbability, Randm Variables and Randm Prcesses Exercises (Set A) Exercise 1 Given is a statinary discrete Markv prcess with the alphabet A = {a 0, a 1, a 2 } and the cnditinal pmfs listed in the table belw a a 0 a 1 a 2 p(a a 0 ) 0.90 0.05 0.05 p(a a 1 ) 0.15 0.80 0.05 p(a a 2 ) 0.25 0.15 0.60 p(a) Determine the marginal pmf p(a). Heik Schwarz Surce Cding and Cmpressin September 22, 2013 59 / 60
Prbability, Randm Variables and Randm Prcesses Exercises (Set A) Exercise 2 Investigate the relatinship between independence and crrelatin. (a) Tw randm variables X and Y are said t be crrelated if and nly if their cvariance C XY is nt equal t 0. Can tw independent randm variables X and Y be crrelated? (b) Let X be a cntinuus randm variable with a variance σx 2 > 0 and a pdf f X (x). The pdf shall be nn-zer fr all real numbers, f X (x) > 0, x R. Furthermre, the pdf f X (x) shall be symmetric arund zer, f X (x) = f X ( x), x R. Let Y be a randm variable given by Y = a X 2 + b X + c with a, b, c R. Fr which values f a, b, and c are X and Y uncrrelated? Fr which values f a, b, and c are X and Y independent? (c) Which f the fllwing statements fr tw randm variables X and Y are true? If X and Y are uncrrelated, they are als independent. If X and Y are independent, E{XY } = 0. If X and Y are crrelated, they are als dependent. Heik Schwarz Surce Cding and Cmpressin September 22, 2013 60 / 60