Lec 07 Transforms and Quantization II

Transcription

1 Outlne CS/EE 559 / ENG 4 Specal Topcs (784, 785, 78) Lec 7 Transforms and Quantzaton II Lecture 6 Re-Cap Scalar Quantzaton Vector Quantzaton Zhu L Course We: Z. L ultmeda Communcaton, 6 Sprng p. Z. L ultmeda Communcaton, 6 Sprng p. Untary Transforms Untary Transform Propertes y=a,,y n R d, A: dd Preserve Energy: y = A= = [a T, a T,, a T d ] Inner product<, a k > = = = = = = = = Untary Transforms: A s untary f: A - =A T, AA T = I d The ass of A s orthogonal to each other <, > = <, > = Preserve Angles: the angles etween vectors are preserved Eamples: cos sn sn cos untary transform: rotate a vector n R n,.e., rotate the ass coordnates DoF of Untary Transforms k-dmenson projectons n d-dmensonal space: kd k. Aove eample: - = ; normal ponts to the unt sphere n Z. L ultmeda Communcaton, 6 Sprng p. Z. L ultmeda Communcaton, 6 Sprng p.4

2 Energy Compacton and De-correlaton Energy Compacton any common untary transforms tend to pack a large fracton of sgnal energy nto just a few transform coeffcents De-correlaton Hghly correlated nput elements qute uncorrelated output coeffcents Covarance matr = = { {} } DCT Eample: y=dct(), Queston: Is there an optmal transform that do est n ths?,,, 6 R y,y,, y 6 R yy Karhunen-Loève Transform (KLT) a untary transform wth the ass vectors n A eng the orthonormalzed egenvectors of R =, = A T = [a, a,, a d ] =, =,,, assume real nput, wrte A T nstead of A H denote the nverse transform matr as A, AA T =I R s symmetrc for real nput, Hermtan for comple nput.e. R T =R, R H = R R nonnegatve defnte,.e. has real non-negatve egen values lnear dsplay scale: g Attrutons Kar Karhunen 947, chel Loève 948 a.k.a Hotellng transform (Harold Hotellng, dscrete formulaton 9) a.k.a. Prncple Component Analyss (PCA, estmate R from samples) : columns of mage pels dsplay scale: log(+as(g)) Decorrelaton y constructon: Propertes of K-L Transform Energy Compacton Comparson = { } = = = nmzng Error under lmted coeffcents reconstructon,! = =, = Bass restrcton: Keep only a suset of m transform coeffcents and then perform nverse transform ( m N) Keep the coeffcents w.r.t. the egenvectors of the frst m largest egenvalues (ndcaton of energy)

3 Transform on D sgnals Gven a m n mage lock, how to compute ts D transform? By applyng D DCT to the rows and then y the columns. (Separale) DCT transform matr s a kronecker product of D DCT ass ufuncton u N(=8)-pt D DCT ass 8-pt D DCT ass atla Eercse: SVD, PCA, and DCT appromaton In compresson: DCT: not data dependent, motvated y DFT, no need to sgnal ass PCA: data drven, otaned from a class of sgnals, need to sgnal per class SVD: drectly appro. from the sgnal, need to sgnal per mage lock o Queston: can we encode ass etter? u 7 Z. L ultmeda Communcaton, 6 Sprng p.9 Z. L ultmeda Communcaton, 6 Sprng p. DNA Sequence Compresson Seq Data n real world: Qualty score any Reads that are algned, wth mutatons/errors:. llon t each = 8 B reads + qualty (confdence) score + laelng =.5 TB Queston: how to compress sequence (lossless) and confdence (lossy) reads Z. L ultmeda Communcaton, 6 Sprng p. Z. L ultmeda Communcaton, 6 Sprng p.

4 FastQ and SA Current solutons: Remnds of zgzag and run-level codng Lecture 6 Re-Cap Scalar Quantzaton Unform Quantzaton Non-Unform Quantzaton Vector Quantzaton Outlne Z. L ultmeda Communcaton, 6 Sprng p. Z. L ultmeda Communcaton, 6 Sprng p.4 Rate Dstorton Encoder and Decoder n nr f ( ),,..., X n n X Encoder Decoder n Encoder: Represent a sequence X n = {X, X,, Xn} y an nde f n (X n ) n {,,, nr }. Decoder: ap f n (X n ) to a reconstructon sequence. Scalar quantzer: n =, quantze each sample ndvdually. Vector quantzer: n >, quantze a group of samples jontly. Z. L, ultmeda Communcaton, 6 Sprng p.5 Xˆ Scalar Quantzaton y y y y4 y5 y6 y7 y8 = = Fed len Code: Encoder: Partton the real lne nto dsjont ntervals: I [, ),... I: Quantzaton ns. : Inde of quantzaton n.,,..., -. : Decson oundares. y : Reconstructon levels. Encoder: sends the code word of each nterval/n nde to the decoder. Decoder: represents all values n an nterval y a reconstructon level. Z. L, ultmeda Communcaton, 6 Sprng p.6.

5 Rate-Dstorton Tradeoff odel of Quantzaton Thngs to e determned: Numer of ns Decson oundares Reconstructon levels Codewords for n ndees. Dstorton The desgn of quantzaton s a tradeoff etween rate and dstorton: To reduce the sze of the encoded ts, we need to reduce the numer of ns ore dstortons The performance s governed y the ratedstorton theory A B Rate Z. L, ultmeda Communcaton, 6 Sprng p.7 A Quantzaton: q = A(): map to an nde Inverse Quantzaton: ˆ B( q) B( A( )) Q( ) ˆ B() s not eactly the nverse functon of A(), ecause Quantzaton error: Q ˆ q B e( ) ˆ Comnng quantzer and de-quantzer: e() or Z. L, ultmeda Communcaton, 6 Sprng p.8 ˆ ˆ Quantzaton error: easure of Dstorton ean Squared Error (SE) for Quantzaton Average quantzaton error of all nput values Need to know the proalty dstruton of the nput Numer of ns: Decson oundares:, =,, Reconstructon Levels: y, =,, Reconstructon: ˆ y ff d SE e( ) ˆ y y y y4 y5 y6 y7 y ˆ f ( ) d y f ( ) d Z. L, ultmeda Communcaton, 6 Sprng p.9 Unform drse Quantzer All ns have the same sze ecept possly for the two outer ntervals: and y are spaced evenly The spacng of and y are oth (step sze) y for nner ntervals. Unform drse Quantzer For fnte Xma and Xmn: Reconstructon Input Even numer of recon levels s not a recon level ma Xmn=- 6 ns For nfnte Xma and Xmn: ns ma ma= The two outer-most recon levels are stll one step sze away from the nner ones. Z. L ultmeda Communcaton, 6 Sprng p.

6 Unform dtread Quantzer Unform dtread Quantzer Unform dtread Quantzer Reconstructon Input Odd numer of recon levels - s a recon level - Desred n mage/vdeo codng For fnte Xma and Xmn: ma For nfnte Xma and Xmn: Xmn=- 5 ns ns ma ma= Z. L ultmeda Communcaton, 6 Sprng p Reconstructon Input Quantzaton mappng: Output s an nde q A( ) sgn( ). 5 Eample: =.8, q =. De-quantzaton mappng: Eample: q = ˆ B( q) q ˆ Z. L ultmeda Communcaton, 6 Sprng p. Quantzaton of a Unformly Dstruted Source Input X: unformly dstruted n [-Xma, Xma]: f()= / ( Xma) Numer of ns: (even for mdrse quantzer) Step sze: = Xma /. y Xma y -.5 e( ) ˆ - - y y y y6.5 6 y7.5 7 y Xma s unformly dstruted n [- /, /] Z. L ultmeda Communcaton, 6 Sprng p. Quantzaton of a Unformly Dstruted Source SE Prove that d Proof: The pdf s f ( ) d / d How to choose, the numer of ns, such that the dstorton s less than a desred level D? d D ˆ f ( ) d y X ma D X ma f ( ) d D Z. L ultmeda Communcaton, 6 Sprng p.4

7 Sgnal to Nose Rato (SNR) Varance of a random varale unformly dstruted n [- L/, L/]: Let = R, each n nde can e represented y R ts. SNR( db) log log 6.R db X X ma ma / Sgnal Energy log Nose Energy L / L / d L log L log / X / R ma ( log ) R Lecture 6 Re-Cap Scalar Quantzaton Unform Quantzaton Non-Unform Quantzaton Vector Quantzaton Outlne = log = log 55 Z. L ultmeda Communcaton, 6 Sprng p.5 Z. L ultmeda Communcaton, 6 Sprng p.6 Non-unform Quantzaton Unform quantzer s not optmal f source s not unformly dstruted For gven, to reduce SE, we want narrow n when f() s hgh and wde n when f() s low d k ˆ f ( ) d y k k f() k f ( ) d Z. L, ultmeda Communcaton, 6 Sprng p.7 Lloyd-a Scalar Quantzer Also known as pdf-optmzed quantzer Gven, the optmal and y that mnmze SE satsfy: d d Lagrangan condton :,. y d y d y E X X I f ( ) d f ( ) d y s the centrod of nterval [(-), ()]. (condtonal mean) k ˆ f ( ) d y k k k f ( ) d f() - Z. L, ultmeda Communcaton, 6 Sprng p.8 y

8 d Lloyd-a Scalar Quantzer y f ( ) y y y f ( ) s the mdpont of y and y+ Nearest neghorng quantzer. - + Summary of Lloyd-a condtons: y f ( ) d f ( ) d y y y y+ A specal case Relatonshp to unform quantzer: If f() = c (unform), Lloyd-a quantzer reduces to unform quantzer y f ( ) d f ( ) d c c( d ) ( ) ( If the rate s hgh, f() s close to constant n each n, we also have y ( ) / L- quantzer reduces to unform quantzer. ) Z. L, ultmeda Communcaton, 6 Sprng p.9 Z. L, ultmeda Communcaton, 6 Sprng p. Eample f() - For the gven pdf, desgn the optmal -level md-rse quantzer. Soluton: By symmetry, = -, =, =. ( ) d ( ) d / 6 y /. / / f ( ) d y Lloyd-a Scalar Quantzer Summary of condtons for optmal quantzer: f ( ) d f ( ) d Gven, can fnd the correspondng optmal y Gven y, can fnd the correspondng optmal How to fnd optmal and y smultaneously? A deadlock: o Reconstructon levels depend on decson levels o Decson levels depend on reconstructon levels Soluton: teratve method! y y Z. L, ultmeda Communcaton, 6 Sprng p. Z. L, ultmeda Communcaton, 6 Sprng p.

9 Lloyd Algorthm (wth known f() ) If the pdf f() s known: Performance of Lloyd-a Scalar Quantzer Recall: Upper & lower ounds for D(R) functon:. Intalze all y, let j =, d = (dstorton). P R D( R) R. Update all decson levels. Update all y, 4. Computer SE: 5. If (dj- dj) / dj- < ε, stop, otherwse set j = j +, go to step. d y j y y k y k k k f ( ) d / f ( ) d f ( ) d P h( X ) : Entropy Power (always σ ) e Dstruton Unform Laplacan Gaussan Entropy Power 6.7 e e.865 Z. L, ultmeda Communcaton, 6 Sprng p. Z. L, ultmeda Communcaton, 6 Sprng p.4 Performance of Lloyd-a Scalar Quantzer Let X take values n [-V, V] wth pdf f() and varance. If X s quantzed nto ns y the Lloyd-a quantzer, t can e shown that when s large, the mnmal SE s d ( ) V f d Sgnfcance: drect estmate of the quantzaton error n terms of the pdf and the numer of ns. Proof can e found n the followng places: Panter, Dte, Quantzaton Dstorton n Pulse-Count odulaton wth Nonunform Spacng of Levels, Proceedngs of IRE, 95. Notes:. Ths s only good for fnte range V.. The formula s eact for pecewse constant pdf. Z. L, ultmeda Communcaton, 6 Sprng p.5 Performance of Lloyd-a Scalar Quantzer Rate-Dstorton performance: d ( ) V f d If = ^R where V R R f ( ) d d V f ( ) d. Eample: If X has unform dstruton n [-V, V], then V ( ). f d V V V d R V R. L- quantzer reduces to unform quant. Comparng wth ounds: L quantzer only acheves the upper ound. Z. L ultmeda Communcaton, 6 Sprng p.6

10 Outlne Vector Quantzaton Lecture 6 Re-Cap Scalar Quantzaton Vector Quantzaton n nr f ( ),,..., X n n X Encoder Decoder n Encoder: Represent a sequence X n = {X, X,, Xn} y an nde fn(x n ) n {,,, nr }. Decoder: ap fn(xn) to a reconstructon sequence (codeword). Xˆ Codeook: the collecton of all codewords. Questons to e addressed: Why ths s etter than scalar quantzer? How to generate the codeook? How to fnd the est codeword for each nput lock? Z. L ultmeda Communcaton, 6 Sprng p.7 Z. L ultmeda Communcaton, 6 Sprng p.8 VQ Induced from Scalar Quantzer Consder the quantzaton of two neghorng samples of a source: Bt rate: ts / sample 8 quantzaton ns. If unform scalar quantzaton s used for each sample, the -D samplng space s parttoned nto 64 rectangular regons (Vorono regons) Defcences of Scalar Quantzer. All codewords are dstruted n a cuc: Not effcent for most dstrutons. The optmal codeword arrangement should depend on the pdf: Assgn codewords to the typcal regon. Scalar quantzer of one sample Hgh pro regon of AR() source. Hgh pro regon of IID Gaussan. VQ has etter performance even for IID. Z. L ultmeda Communcaton, 6 Sprng p.9 Z. L ultmeda Communcaton, 6 Sprng p.4

11 Defcences of Scalar Quantzer Lnde-Buzo-Gray (LBG) Algorthm. The Vorono regons nduced from SQ are always cuc: Cuc regon vs sphercal regon: Gven the same volume, the granular error of the sphere s the smallest among dfferent shapes. lm n SE Area: Sde length: a error:.77 Gven the same volumes (.e., rate R), the SE of the sphercal Vorono regon s the mnmum among all shapes: cuc e SE 6 sphere Area: Radus=a error: /.4 SE.56 sphere or.5 db loss for cuc Vorono regons (same for all pdfs). Z. L ultmeda Communcaton, 6 Sprng p.4. Algorthm to select code-words from a tranng set. Also known as Generalzed Lloyd Algorthm (GLA):. Start from an ntal set of recon values {y}, = to, and a set of tranng vectors {Xn}, n =, N.. For each tranng vector Xn, fnd the recon value that s closest to t. Q(Xn)= Yj ff d(xn, Yj) d(xn, Y) for all j.. Compute average dstorton. 4. If dstorton s small enough, stop. Otherwse, replace the recon value y the avg values of all vectors n each quantzaton regon. Go to Step. Z. L ultmeda Communcaton, 6 Sprng p.4 kmeans() % desred rate R=8; [nd, vq_codeook]=kmeans(, ^R); atla Implementaton kd-tree mplementaton [kdt.nd, kdt.leafs, kdt.mo]=uldvsualwordlst(, ^R); [node, pref_code]=searchvsualwordlst(q, kdt.nd, kdt.leafs); Summary Transforms Untary transform preserves energy, angle, lmted DoF KLT/PCA: energy compacton and de-correlaton DCT: a good KLT/PCA appromaton A t of ntro to Genome Info Compresson, more to come Scalar Quantzaton: If sgnal s unform, what s the epected quantzaton error? Non-unform sgnal dstruton, optmal quantzaton desgn (Lloyd- a) Vector Quantzaton: ore effcent Fast algorthm ests lke kd-tree ased A specal case of transform: over-complete ass, very sparse coeffcent (only none zero entry) Shall revst wth coupled dctonary approach n super resoluton Z. L ultmeda Communcaton, 6 Sprng p.4 Z. L ultmeda Communcaton, 6 Sprng p.44