An Efficient Lloyd-Max Quantizer for Matching Pursuit Decompositions

Size: px
Start display at page:

Download "An Efficient Lloyd-Max Quantizer for Matching Pursuit Decompositions"

Transcription

1 BRAZILVI ITERATIOAL TELECOMMUICATIOS SYMPOSIUM (ITS6), SEPTEMBER 3-6, 6, FORTALEZA-CE, BRAZIL A Efficiet Lloyd-Max Quatizer for Matchig Pursuit Decompositios Lisadro Lovisolo, Eduardo A B da Silva ad Paulo S R Diiz Abstract Several applicatios are usig the Matchig Pursuit algorithm for sigal ad video compressio. The Matchig Pursuit approximates sigals iteratively usig liear combiatios of pre-defied atoms of a dictioary. I compressio applicatios Matchig Pursuits coefficiets, which multiply the atoms i the liear combiatio, eed to be quatized. The Lloyd- Max quatizer is kow to be the best quatizer for a give source. However, to desig a Lloyd-Max quatizer the statistics of the source eed to be kow. The statistics of Matchig Pursuit coefficiets are difficult to model. I this paper, startig from the observatio that the statistics of the agles betwee the residues ad the atoms preset little variatio alog Matchig Pursuit iteratios, we propose to use these statistics to model the oes of Matchig Pursuit coefficiets. This permits the desig of Lloyd-Max quatizers for Matchig Pursuit coefficiets. The Lloyd-Max quatize is compared to a state-of-the-art off-loop Matchig Pursuit quatizatio scheme. Results show that the proposed scheme has good rate-distortio performace, specially at low rates. Idex Terms Matchig Pursuit, Quatizatio, Compressio. After M iteratios, the MP obtais the M-term approximatio ˆx, or simply M-term, give by ˆx = γ g i(), (3) = which has distortio x ˆx = r M x,themth residue. The performace of the MP for sigal compressio applicatios depeds heavily o two aspects: i) Dictioary it should iclude atoms that are good matches to the possible compoets of the sigals to compress. I additio, the dictioary cardiality affects the data rate. ii) Quatizatio for compressio the coefficiets γ must be quatized. Usig a coefficiet quatizatio rule Q[ the compressed sigal is retrieved by the quatized M-term I. Itroductio The Matchig Pursuit algorithm [ (MP) approximates sigals iteratively. The approximatio is obtaied usig atoms (pre-defied sigals) g k from a dictioary D (the collectio of possible atoms). The MP is beig used i compressio schemes for -dimesioal [, [3 ad - dimesioal sigals [, [4. The MP works as follows. Let D = {g k } with k [,...,C(D), such that g k = k, C(D) is the dictioary cardiality (umber of elemets i D). At each iteratio, the MP searches for the atom g i(), i() [...C(D), with largest ier product with the residual sigal r x [, [5, which is give by γ = r x, g i(). () Observe that the first residue is equal to the sigal to be decomposed, that is r x = x. The ier product γ,which is a measure of how much of g i() is icluded i r x,is used to compute the ext residue r x = r x γ g i(). () Lisadro Lovisolo (lisadro@uerj.br) is with DETEL-FE-UERJ, Eduardo A B da Silva ad Paulo S R Diiz (eduardo, diiz@lps.ufrj.br) are with POLI/COPPE-UFRJ. ˆx q = Q[γ g i(). (4) = I order to desig efficiet quatizers Q[, oe eeds a statistical model for MP coefficiets. However, MP coefficiets are difficult to model. For example, i [6 it has bee observed that MP residues have a chaotic behavior. Here, istead of searchig for a good model for MP coefficiets, our approach is based o a statistical model for the MP agles. It relies o the observatio that the agles betwee the atom chose ad the residue beig decomposed alog MP iteratios have well behaved statistics. Defie, at each MP iteratio, the agle betwee the residue r x ad the selected atom g i() as ( r θ = arccos x, g i() r x ). (5) We have verified, experimetally (see sectio II), that the statistics of θ are approximately idepedet of. Here, it is cojectured that the agles betwee the residues ad the atoms i MP iteratios ca be statistically modeled as idepedet ad idetically distributed. SBrT 97

2 BRAZILVI ITERATIOAL TELECOMMUICATIOS SYMPOSIUM (ITS6), SEPTEMBER 3-6, 6, FORTALEZA-CE, BRAZIL II. Agles i Matchig Pursuit Iteratios Usig the defiitio of θ, eq. (5), the MP algorithm is such that γ = x cos (θ ), (6) γ = x si (θ )cos(θ ), (7). γ = x si (θ i )cos(θ ). (8) i= However, if D icludes the elemets g k ad g k for ay k, forcig the ier products to be positive, the γ = x si (θ i )cos(θ ). (9) i= This work assumes that g k ad g k belog to D obtaiig always positive coefficiets, ad all subsequet refereces to dictioaries cosider that. That is, if D does ot iclude g k, the g k is icluded i D ad C(D) is updated accordigly. A. Statistics of the Agles i MP Iteratios I [6, it has bee observed that MP residues have a chaotic behavior, thus it would be reasoable to assume that, after some MP iteratios, the residues ca have ay orietatio. I additio, oe may assume that these orietatios may have uiform probability desity fuctio o the uit-ball. This is equivalet to assume that the residues are realizatios from a memoryless idepedet ad idetically distributed (iid) Gaussia source. That is, the sigal source is such that if a outcome is x = [x(),x(),...,x(), the the x(j) have the same Gaussia distributio (,σ ).This source does ot privilege ay sigal orietatio. Therefore, we use this source to ivestigate MP agle statistics. Fig. shows the relative frequecy histograms of the RVs Θ, which correspod to the agles θ that result from decompositios of realizatios of a Gaussia source usig a dictioary composed of 6 ormalized radom atoms also draw from a Gaussia source i R 4.These histograms were obtaied usig a esemble of 5, MP decompositios of radom sigals from a Gaussia source. Such dictioaries, composed of C(D) ormalized sigals draw from a -dimesioal Gaussia source, are referred here as GS D(C(D),) ad thus the former dictioary is deoted by GSD(6, 4). It should be oted that this dictioary has its cardiality doubled i order to obtai just positive coefficiets. I Fig. oe otes that the histograms of the RVs Θ have very similar shape for all. This leads to the cojecture that the pdfs f Θ (θ )are idepedet ad idetically distributed at ay iteratio. The results preseted i [6 corroborate this assumptio, where it is show that, uder specific coditios, MP residues have chaotic behavior. Fig. shows the mea ad the variace of Θ for several. Fig. depicts also the covariaces betwee MP agles i.5..5 ormalized histogram of Θ θ.5..5 ormalized histogram of Θ θ ormalized histogram of Θ θ ormalized histogram of Θ θ.5..5 ormalized histogram of Θ θ ormalized histogram of Θ θ Fig.. Relative frequecy histograms of Θ for a Gaussia source i R 4 usig the GSD(6, 4). 4 Expected value of Θ Cov[Θ Θ 5 5 Cov[Θ 5 Θ Variace of of Θ Cov[Θ Θ 5 5 Cov[Θ Θ 5 5 Fig.. Mea, variace ad covariace of Θ for a Gaussia source i R 4 usig the GSD(6, 4). differet steps. From it oe observes that Cov[Θ i Θ k =, i k, that is, the agles are ucorrelated. I this work we assume that the agles are idepedet RVs. This is ot a ureasoable assumptio, sice it does ot cotradicts the behavior observed i Fig.. The results preseted so far make use of a dictioary of relatively low dimesio ad cardiality C(D). I practice, the MP is commoly used i large dimesioal spaces with dictioaries such as the Gabor oe [5. The elemets of this dictioary are defied by traslatios, modulatios ad dilatios of a prototype sigal. The most commo choice for the prototype f() is the Gaussia widow. We aalyze here a real Gabor dictioary composed of atoms π with predefied phases i multiples of V.Eachatomis the give by [5 8 δ(), j = >< «p j g() = K (j,p,v) f cos kπ j + πv, j (,L), >: j V, j = L () where f() = 4 e π, is the sample, K (j,p,v) provides uit-orm atoms, ad v [,...,V. Above, j defies the atom scale, p defies the time shift, ad k defies the atom modulatio, ad for L =log () scales their rages are [ j [,L, p [, j ),k [, j ), ad v [,V. SBrT 98

3 BRAZILVI ITERATIOAL TELECOMMUICATIOS SYMPOSIUM (ITS6), SEPTEMBER 3-6, 6, FORTALEZA-CE, BRAZIL 3 Fig. 3 shows f Θ (θ ), for some, obtaied for a esemble of 8, decompositios of Gaussia sigals i R 64 usig the Gabor dictioary with four phases. Fig. 4showsf Θ (θ ), for some, obtaied for a esemble of 8, decompositios of sigals drive from a source that has gamma distributed coordiates i R 64 for the same dictioary. ote that the agle statistics show for each sigal source differ strogly oly at the first MP iteratio, beig visually very similar for the other iteratios. It ca be oted that for this dictioary the agles i differet MP steps have similar statistics eve for very differet sources. ote that, although the statistics of Θ are ot exactly equal to the statistics obtaied i the first MP iteratio for a Gaussia source, they are reasoably close to these; therefore f Θ (θ ), for >, ca be reasoably approximated by the f Θ (θ ) obtaied for a memoryless Gaussia source. This is a reasoable assumptio, sice the memoryless Gaussia source does ot privilege ay orietatio, what seem to be the case for the residues r x for >. B. Discussio of Results It was verified that the statistics of the agles, after the first MP iteratio, ca be cosidered to be ivariat with respect to the step umber. For some dictioaries, for example the Gabor oe, the agle pdf, after some steps, is slightly differet from the agle pdf obtaied for a memoryless Gaussia source. However, oe ca still use a Gaussia source to obtai good estimates of the pdf of the agle i MP iteratios. The differece i the statistics meas that, although the residues of MP iteratios have similar orietatios, these orietatios are ot uiformly distributed. evertheless, the iid statistical model is ot too bad a assumptio. I the sequel, this model is used to desig Lloyd-Max quatizers of MP coefficiets. This permits to verify the validity of the statistical model for the agles i MP iteratios. It should be oted that the statistics of the first MP agle, ad therefore of the first coefficiet, are much more source depedet tha they are at further MP steps. Therefore, i order, to make a appropriate use of the preseted agle model, the first coefficiet, γ, will be quatized with egligible error ad ecoded as side iformatio. ote that this value replaces x, the value of the sigal orm which is usually trasmitted as side iformatio i practical MP based compressio schemes [, [4. III. Quatized Matchig Pursuit Decompositios I compressio applicatios, oe ecodes the quatized versios Q[γ of the coefficiets γ of M-terms. Oe way to obtai the Q[γ is to quatize the coefficiets offloop, i.e. first the whole decompositio is obtaied ad the the coefficiets are quatized. Aother strategy is iloop quatizatio [4, i which the quatized coefficiet is used to compute the residue, ad due to that i-loop quatizatio might result i atoms i M-term that deped o the quatizer used. 3 x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ Fig. 3. ormalized histograms of MP agles for a Gaussia source i R 64, usig bi, at = {, 8, 6, 3, 64, 7}, for the 4-phase Gabor dictioary i R x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ x 6 ormalized histogram of Θ Fig. 4. ormalized histograms of MP agles for a source with coordiates drive from a gamma distributio, usig bi, at = {, 8, 6, 3, 64, 7}, for the 4-phase Gabor dictioary i R 64. We choose off-loop quatizatio sice it allows a simple rate distortio (RD) optimizatio procedure, i which differet quatizers are tried out i order to fid oe that meets a prescribed RD criterio. I cotrast, i-loop quatizatio would require, for RD optimizatio, several MP sigal decompositios, what would greatly icrease the computatioal demads. Sice i off-loop quatizatio the residues are computed without kowledge of the quatizer, the quatizatio error i oe step caot be compesated i subsequet steps. However, these quatizatio errors ca be miimized by desigig appropriate quatizers [. I this work, MP coefficiet quatizers desig is achieved by employig the iid statistical model of MP agles to desig Lloyd-Max quatizers for the MP coefficiets. The quatizers desiged make use of dead-zoes because some advatage i RD performace ca be gaied by quatizig small coefficiets to zero [, [4: either the coefficiets quatized to zero or their atoms idices are SBrT 99

4 BRAZILVI ITERATIOAL TELECOMMUICATIOS SYMPOSIUM (ITS6), SEPTEMBER 3-6, 6, FORTALEZA-CE, BRAZIL 4 set, reducig the data rate. For trasmissio, it is importat that if coefficiets ad/or idices are lost, the decoder ca just successfully igore the lost terms whe recostructig the sigal. Therefore, it would be highly desirable for the quatizer for a give γ to be idepedet of the quatized values of other γ m (m ). A. Distortio Due to Off-Loop Quatizatio Whe M-terms are quatized off-loop, each coefficiet γ is replaced by its quatized versio Q[γ ad the sigal approximatio is retrieved usig eq. (4). The distortio criterio employed for MP quatizer desig ca be: i) the eergy of the error relative to the actual sigal d = x ˆx q ; () ii) the eergy of the error relative to the M-term d M = ˆx ˆx q = (γ Q[γ )g i(). () = I this work, the secod oe is used, sice we desig oe Lloyd-Max quatizer for each umber of coded terms M. The distortio per sample of the quatized M-term is give by d M = d M,thatis, d M = (γ Q[γ )(γ m Q[γ m ) g i(), g i(m), = m= (3) where is the sigal legth. D is composed of uit orm vectors ( g i() = ). Thus, defiig the quatizatio error e q (γ )=γ Q[γ, (4) it follows that [ d M = M e q (γ )+ (5) = e q (γ )e q (γ m ) g i(), g i(m). = m=, For a M-term ˆx, the miimizatio of eq. (5) implies also the miimizatio of the distortio per sample (see eq. ()) give by d = d = x ˆx q. (6) ote that if d M is equal to zero the d = r M x /. ) Distortio for a Give Source: For a give sigal source X, oe may cosider the expected value of d M { E[d M = M E [ e q(γ ) + (7) = E [ e q (Γ )e q (Γ m ) g i(), g i(m). = m=,m Each Γ is a radom variable (RV) that correspods to γ,for M, for sigals draw from X. I eq. (7), E [ e q(γ ) stads for the expected value of the squared quatizatio errors of the RV Γ. B. MP Coefficiet Quatizatio Usig MP Agles I eq. (9) the value of x is required to compute the MP coefficiets from the MP agles. Accordig to thediscussioattheedofsectioiiweusethefirst coefficiet γ istead of x, the eq. (9) ca be rewritte as γ = γ δ, δ =ta(θ ) si (θ i )cos(θ ),. (8) i= Thus, the pdfs of the coefficiets γ ca be computed from the pdfs of the agles Θ, assumig that the Θ are idepedet. The pdfs of the θ, >, ca be obtaied as explaied i sectio II. For a kow γ, the pdf of the RV Γ,for, is give by f (γ γ )=f (γ δ γ ), where is the RV whose outcome is δ, see eq. (8). If a optimal quatizer Q is desiged for the RV Y, the the optimal quatizer for Z = cy (c is a costat) is simply a scaled versio of Q. Thus,forγ kow, the quatizatio of δ istead of γ,fromeq.(7),gives { E[d M γ = γ M E [ e q ( ) + (9) = E [ e q ( )e q ( m ) g i(), g i(m). = m=,m Oce the pdfs of the RVs are kow E[d M γ cabe computed for ay quatizatio rule applied to.sice the quatizatio is applied to δ istead of γ, the value of γ is required at the decoder. I this work we use γ to defie γ,for. The use of γ to compute γ ( ) guaratees its correct value at the decoder. For quatizer desig usig eq. (9) the pdfs of are eeded which ca be computed from the RVs Θ i, i [,...,, correspodig to the agles betwee the residues ad the atoms selected i MP steps, as i eq. (8). IV. Lloyd-Max Quatizers for MP Coefficiets For a give first coefficiet γ the quatizatio of MP coefficiets should aim to miimize the distortio i eq. (9). I this case, oe should desig quatizers for,. Therefore, if just oe quatizer is to be used the it shall be desiged for = M =,thervgiveby the uio of the ( ). Sice MP iteratios are disjoit evets the pdf of = M = is give by f (δ) = M M = f (δ ). A. Distortio for a Optimal Quatizer The distortio per sample i eq. (9) has two terms. The first term is the sum of the squared quatizatio errors of, whereas the secod cotais a sum of ier products betwee dictioary atoms weighted by the products of the quatizatio errors of the atoms ivolved i the ier products. As verified, i sectio II, the RVs of the agles Θ ca be assumed to be ucorrelated. Although ad m may be correlated, whe desigig a quatizer for = M = the assumptio that the quatizatio errors e q ( ) are ucorrelated is reasoable. It is also SBrT 9

5 BRAZILVI ITERATIOAL TELECOMMUICATIOS SYMPOSIUM (ITS6), SEPTEMBER 3-6, 6, FORTALEZA-CE, BRAZIL 5 reasoable to assume that the quatizatio errors products e q ( )e q ( m ) are ot correlated to the ier products g i(), g i(m). Usig the assumptios above, the secod term i eq. (9) becomes E [e q ( ) E [e q ( m ) E [ g i(), g i(m). = m=,m () The atoms selected at differet MP steps may be correlated. But due to the ivariat ature of the agles statistics i differet MP steps, oe ca cosider the expected value of the ier product betwee the atoms selected i ay two differet MP steps ad m to be also ivariat. That is E [ g i(), g i(m) = c. Therefore, otig that E [e q ( ) = eq. () yields E [e q( ) = (M )E [e q( ) M m=,m E [e q ( ), () = = m=,m E [e q( m) c = () E [e q( m) c. The optimal quatizatio of leads to E [e q ( ) =, so that the expressio above vaishes. As a result eq. (9) becomes E[d M γ = γ E[( Q[ ), (3) = This result is a sum of terms E[( Q[ ), thus E[d M γ = γ (δ Q[δ ) f (δ )dδ. (4) = Sice the same quatizer Q[ is applied to all, M, oe has that E[d M γ =γ M (δ Q[δ) f (δ)dδ, (5) with f (δ) = f (δ ) (6) M = If = M = ad defiig eq. (5) becomes MSE( ) = E [ ( Q[ ) (7) E[d M γ = γ (M ) MSE( ). (8) The desig of a optimal quatizer for = M =,that miimizes eq. (8), is accomplished by Lloyd-Max quatizers [7 (). ote that sice Lloyd-Max quatizers are ubiased estimates of the iput the E[e q ( ) =, ad thus the term i eq. () vaishes validatig eq. (3). B. Lloyd-Max Quatizer Desig The desig of Lloyd-Max quatizers requires f (δ ), for M; these are estimated from f Θ (θ ). I tur, f Θ (θ ) is estimated by applyig oe MP decompositio step to a large set of sigals draw from a memoryless Gaussia source. As discussed i sectio II, this is a acceptable procedure sice all Θ,, have statistics that are similar to the oes obtaied for a memoryless Gaussia source. The estimated f (δ ) are the used to calculate f (δ), ad f (δ) istheusedtoobtaithe of b coef bits (L = b coef levels). The quatizer thresholds ad recostructio levels are calculated usig a iterative algorithm [7. The f (δ ), that are used to compute f (δ), are obtaied recursively from a estimate of f Θ (θ ) that is i tur obtaied usig the agle i the first MP step for a esemble of C(D) realizatios from a Gaussia source. The same quatizer law is used for all coefficiets γ. ote that f (δ) varies with M, therefore each M leads to a differet quatizer. The quatizer desig is idepedet of γ ad it suffices to desig quatizers for γ =, storig copies of the quatizers i both ecoder ad decoder. The ecoder seds γ, the umber of bits of the quatizer, ad the umber of terms of the decompositio (M), i a header, to the decoder. The parameter γ is used to scale the quatizer i both coder ad decoder, a simple strategy that makes good use of resources. V. Lloyd-Max Quatizers Performace The state-of-the-art for off-loop quatizatio of MP coefficiets is preseted i [. There, uiform quatizers whose umber of quatizatio levels ad rage adapt accordig to the coefficiets of the M-term are used. Here, this quatizatio scheme is referred to as adaptive bouded uiform quatizer (). The implemets a bit-allocatio per coefficiet that relies o the kow result that MP coefficiets magitudes decrease o average at each MP iteratio. I the, previous to quatizatio all the coefficiets eed to be sorted i decreasig magitude ad the is fed with the coefficiets i this order. For each coefficiet, the employs a uiform quatizer of differet rage ad umber of levels; the quatizer rage for the l th coefficiet depeds othequatizedvalueofthe(l ) th coefficiet, ad the umber of levels of each coefficiet quatizer is decided usig a criterio based o a Lagragia multiplier a bitallocatio procedure. For decodig the quatized decompositio the umber of bits used to quatize the secod coefficiet as well as the larger coefficiet are set as side iformatio. The total rate of MP quatized decompositios is give by R = S [log (C(D)) + r coef, (9) where S is the umber of terms that remai after quatizatio, ad r coef is the rate icurred i codig the quatized coefficiets. Therefore, the rate i bits per sample is give by R/, where is the sigal space dimesio. The SBrT 9

6 BRAZILVI ITERATIOAL TELECOMMUICATIOS SYMPOSIUM (ITS6), SEPTEMBER 3-6, 6, FORTALEZA-CE, BRAZIL 6 strategy employed to geerate the coded bitstream is to etropy code the differeces betwee the quatizatio idices of successive coefficiets. I the comparisos betwee Lloyd-Max quatizatio ad preseted here, this strategy is also employed to code the Lloyd-Max quatized M-terms. Fig. 5 shows the RD curves of quatized MP expasios arrivig from three differet radom sources i R (a memoryless Gaussia, a memoryless uiform ad a memoryless Gamma distributed source) usig both the Lloyd- Max quatizatio ad the ad a GSD(8, ). For each distict source the results are averages over a esemble of quatized MP decompositios of sigals from each source. For this experimet the s were desiged with bit-depth ragig from to 8. It ca be see i Fig. 5 that both quatizers have similar performace for the three sigal sources; however the teds to be slightly better at low rates (below 8 ). Fig. 6 shows the RD curves of quatized MP expasios arrivig from three differet radom sources i R 64 for the ad the for the Gabor dictioary of 4 phases i R 64, see eq. (). For that purpose the s were desiged with bit-depth ragig from to 6. The decompositios to be coded allowed a maximum of 56 terms. For each distict source the results are averages over a esemble of quatized MP decompositios of sigals from each source. The decompositios quatized with the require a larger bitrate because its bitallocatio scheme does ot maages to cotrol the rate as the shape of f Θ (θ ) for this dictioary implies a low decay rate of the coefficiets ivolved i M-terms obtaied usig this dictioary. VI. Coclusio The agles betwee the residues ad the selected atoms i Matchig Pursuit iteratios ca be statistically modeled as idepedet ad idetically distributed (iid) at each iteratio. That is, the agles i Matchig Pursuit steps ca be cosidered statistically ivariat with respect to the decompositio step. As a result, the statistics of Matchig Pursuit agles ca be obtaied from the statistics of the first Matchig Pursuit agle for sigals draw from a memoryless Gaussia source. Based o the iid statistical model for Matchig Pursuit agles, Lloyd-Max quatizers for Matchig Pursuit coefficiets were desiged. The Lloyd-Max quatizer is desiged to be the same for all the coefficiets i a M-term, ad its desig requires both the umber of terms to be coded ad the probability desity fuctio of the first Matchig Pursuit agle for a memoryless Gaussia source, resultig i dictioary depedet quatizers. It is importat to poit out that if the source is ot Gaussia the first MP agle has differet statistics tha the MP agles i other steps. However, eve i this case, the Lloyd-Max quatizers obtaied for the Gaussia source ca still be employed sice the first coefficiet is ot quatized but trasmitted with egligible error as side iformatio. E[d E[d E[d Memoryless Gaussia Source Memoryless Uiform Source Memoryless Gamma Distributed Source Fig. 5. ad RDs for three differet radom sources usig the GSD(8, ), the distortio is defied i eq. (6). E[d E[d E[d Memoryless Gaussia Source Memoryless Uiform Source Memoryless Gamma Distributed Source Fig. 6. ad RDs for three differet radom sources usig the 4-phase Gabor dictioary i R 64, the distortio is defied i eq. (6). The Lloyd-Max quatizatio preseted was compared to the state-of-the-art off-loop quatizatio scheme i [ ad both quatizatio schemes showed similar rate distortio performace. Refereces [ S. Mallat ad Z. Zhag, Matchig pursuits with time-frequecy dictioaries, IEEE Tras. o Sigal Processig, vol. 4, o., pp , December 993. [ P.Frossard,P.Vadergheyst,R.M.FiguerasIVetura,ad M. Kut, A posteriori quatizatio of progressive matchig pursuit streams, IEEE Tras. o Sigal Processig, vol. 5, o., pp , February 4. [3 L.Lovisolo,M.A.M.Rodrigues,E.A.B.daSilva,adP.S.R. Diiz, Efficiet coheret decompositios of power systems sigals usig damped siusoids, IEEE Tras. o Sigal Processig, vol. 53, pp , 5. [4 R. eff ad A. Zakhor, Modulus quatizatio for matchigpursuit video codig, IEEE Tras. o Circuits ad Systems for Video Techology, vol., pp ,. [5 S. Mallat, A Wavelet Tour of Sigal Processig, sted. Sa Diego, Califoria, USA: Academic Press, 998. [6 G. Davis, S. Mallat, ad M. Avellaeda, Adaptive greedy approximatios, Joural of Costructive Approximatio, vol. 3, pp , 997. [7 A. K. Jai, Fudametals of Digital Image Processig, 3rded., ser. Pretice Hall Iformatio Ad System Scieces Series. Eglewood Cliffs, J 763: Pretice Hall, 989. SBrT 9

Run-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE

Run-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE Geeral e Image Coder Structure Motio Video (s 1,s 2,t) or (s 1,s 2 ) Natural Image Samplig A form of data compressio; usually lossless, but ca be lossy Redudacy Removal Lossless compressio: predictive

More information

OPTIMAL PIECEWISE UNIFORM VECTOR QUANTIZATION OF THE MEMORYLESS LAPLACIAN SOURCE

OPTIMAL PIECEWISE UNIFORM VECTOR QUANTIZATION OF THE MEMORYLESS LAPLACIAN SOURCE Joural of ELECTRICAL EGIEERIG, VOL. 56, O. 7-8, 2005, 200 204 OPTIMAL PIECEWISE UIFORM VECTOR QUATIZATIO OF THE MEMORYLESS LAPLACIA SOURCE Zora H. Perić Veljo Lj. Staović Alesadra Z. Jovaović Srdja M.

More information

Vector Quantization: a Limiting Case of EM

Vector Quantization: a Limiting Case of EM . Itroductio & defiitios Assume that you are give a data set X = { x j }, j { 2,,, }, of d -dimesioal vectors. The vector quatizatio (VQ) problem requires that we fid a set of prototype vectors Z = { z

More information

Vector Permutation Code Design Algorithm. Danilo SILVA and Weiler A. FINAMORE

Vector Permutation Code Design Algorithm. Danilo SILVA and Weiler A. FINAMORE Iteratioal Symposium o Iformatio Theory ad its Applicatios, ISITA2004 Parma, Italy, October 10 13, 2004 Vector Permutatio Code Desig Algorithm Dailo SILVA ad Weiler A. FINAMORE Cetro de Estudos em Telecomuicações

More information

CEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering

CEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio

More information

Algebra of Least Squares

Algebra of Least Squares October 19, 2018 Algebra of Least Squares Geometry of Least Squares Recall that out data is like a table [Y X] where Y collects observatios o the depedet variable Y ad X collects observatios o the k-dimesioal

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

Sensitivity Analysis of Daubechies 4 Wavelet Coefficients for Reduction of Reconstructed Image Error

Sensitivity Analysis of Daubechies 4 Wavelet Coefficients for Reduction of Reconstructed Image Error Proceedigs of the 6th WSEAS Iteratioal Coferece o SIGNAL PROCESSING, Dallas, Texas, USA, March -4, 7 67 Sesitivity Aalysis of Daubechies 4 Wavelet Coefficiets for Reductio of Recostructed Image Error DEVINDER

More information

Machine Learning for Data Science (CS 4786)

Machine Learning for Data Science (CS 4786) Machie Learig for Data Sciece CS 4786) Lecture & 3: Pricipal Compoet Aalysis The text i black outlies high level ideas. The text i blue provides simple mathematical details to derive or get to the algorithm

More information

An Introduction to Randomized Algorithms

An Introduction to Randomized Algorithms A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis

More information

Lecture 11: Channel Coding Theorem: Converse Part

Lecture 11: Channel Coding Theorem: Converse Part EE376A/STATS376A Iformatio Theory Lecture - 02/3/208 Lecture : Chael Codig Theorem: Coverse Part Lecturer: Tsachy Weissma Scribe: Erdem Bıyık I this lecture, we will cotiue our discussio o chael codig

More information

Formation of A Supergain Array and Its Application in Radar

Formation of A Supergain Array and Its Application in Radar Formatio of A Supergai Array ad ts Applicatio i Radar Tra Cao Quye, Do Trug Kie ad Bach Gia Duog. Research Ceter for Electroic ad Telecommuicatios, College of Techology (Coltech, Vietam atioal Uiversity,

More information

Random Variables, Sampling and Estimation

Random Variables, Sampling and Estimation Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig

More information

UC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 17 Lecturer: David Wagner April 3, Notes 17 for CS 170

UC Berkeley CS 170: Efficient Algorithms and Intractable Problems Handout 17 Lecturer: David Wagner April 3, Notes 17 for CS 170 UC Berkeley CS 170: Efficiet Algorithms ad Itractable Problems Hadout 17 Lecturer: David Wager April 3, 2003 Notes 17 for CS 170 1 The Lempel-Ziv algorithm There is a sese i which the Huffma codig was

More information

ECE 564/645 - Digital Communication Systems (Spring 2014) Final Exam Friday, May 2nd, 8:00-10:00am, Marston 220

ECE 564/645 - Digital Communication Systems (Spring 2014) Final Exam Friday, May 2nd, 8:00-10:00am, Marston 220 ECE 564/645 - Digital Commuicatio Systems (Sprig 014) Fial Exam Friday, May d, 8:00-10:00am, Marsto 0 Overview The exam cosists of four (or five) problems for 100 (or 10) poits. The poits for each part

More information

Module 5 EMBEDDED WAVELET CODING. Version 2 ECE IIT, Kharagpur

Module 5 EMBEDDED WAVELET CODING. Version 2 ECE IIT, Kharagpur Module 5 EMBEDDED WAVELET CODING Versio ECE IIT, Kharagpur Lesso 4 SPIHT algorithm Versio ECE IIT, Kharagpur Istructioal Objectives At the ed of this lesso, the studets should be able to:. State the limitatios

More information

The Maximum-Likelihood Decoding Performance of Error-Correcting Codes

The Maximum-Likelihood Decoding Performance of Error-Correcting Codes The Maximum-Lielihood Decodig Performace of Error-Correctig Codes Hery D. Pfister ECE Departmet Texas A&M Uiversity August 27th, 2007 (rev. 0) November 2st, 203 (rev. ) Performace of Codes. Notatio X,

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

Geometry of LS. LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT

Geometry of LS. LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT OCTOBER 7, 2016 LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT Geometry of LS We ca thik of y ad the colums of X as members of the -dimesioal Euclidea space R Oe ca

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

Convergence of random variables. (telegram style notes) P.J.C. Spreij

Convergence of random variables. (telegram style notes) P.J.C. Spreij Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space

More information

Lecture 19: Convergence

Lecture 19: Convergence Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may

More information

Machine Learning for Data Science (CS 4786)

Machine Learning for Data Science (CS 4786) Machie Learig for Data Sciece CS 4786) Lecture 9: Pricipal Compoet Aalysis The text i black outlies mai ideas to retai from the lecture. The text i blue give a deeper uderstadig of how we derive or get

More information

Element sampling: Part 2

Element sampling: Part 2 Chapter 4 Elemet samplig: Part 2 4.1 Itroductio We ow cosider uequal probability samplig desigs which is very popular i practice. I the uequal probability samplig, we ca improve the efficiecy of the resultig

More information

1 Inferential Methods for Correlation and Regression Analysis

1 Inferential Methods for Correlation and Regression Analysis 1 Iferetial Methods for Correlatio ad Regressio Aalysis I the chapter o Correlatio ad Regressio Aalysis tools for describig bivariate cotiuous data were itroduced. The sample Pearso Correlatio Coefficiet

More information

The multiplicative structure of finite field and a construction of LRC

The multiplicative structure of finite field and a construction of LRC IERG6120 Codig for Distributed Storage Systems Lecture 8-06/10/2016 The multiplicative structure of fiite field ad a costructio of LRC Lecturer: Keeth Shum Scribe: Zhouyi Hu Notatios: We use the otatio

More information

Expectation and Variance of a random variable

Expectation and Variance of a random variable Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio

More information

Optimally Sparse SVMs

Optimally Sparse SVMs A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but

More information

Chapter 6 Principles of Data Reduction

Chapter 6 Principles of Data Reduction Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a

More information

MASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science. BACKGROUND EXAM September 30, 2004.

MASSACHUSETTS INSTITUTE OF TECHNOLOGY Department of Electrical Engineering and Computer Science. BACKGROUND EXAM September 30, 2004. MASSACHUSETTS INSTITUTE OF TECHNOLOGY Departmet of Electrical Egieerig ad Computer Sciece 6.34 Discrete Time Sigal Processig Fall 24 BACKGROUND EXAM September 3, 24. Full Name: Note: This exam is closed

More information

THE SYSTEMATIC AND THE RANDOM. ERRORS - DUE TO ELEMENT TOLERANCES OF ELECTRICAL NETWORKS

THE SYSTEMATIC AND THE RANDOM. ERRORS - DUE TO ELEMENT TOLERANCES OF ELECTRICAL NETWORKS R775 Philips Res. Repts 26,414-423, 1971' THE SYSTEMATIC AND THE RANDOM. ERRORS - DUE TO ELEMENT TOLERANCES OF ELECTRICAL NETWORKS by H. W. HANNEMAN Abstract Usig the law of propagatio of errors, approximated

More information

w (1) ˆx w (1) x (1) /ρ and w (2) ˆx w (2) x (2) /ρ.

w (1) ˆx w (1) x (1) /ρ and w (2) ˆx w (2) x (2) /ρ. 2 5. Weighted umber of late jobs 5.1. Release dates ad due dates: maximimizig the weight of o-time jobs Oce we add release dates, miimizig the umber of late jobs becomes a sigificatly harder problem. For

More information

Orthogonal Gaussian Filters for Signal Processing

Orthogonal Gaussian Filters for Signal Processing Orthogoal Gaussia Filters for Sigal Processig Mark Mackezie ad Kiet Tieu Mechaical Egieerig Uiversity of Wollogog.S.W. Australia Abstract A Gaussia filter usig the Hermite orthoormal series of fuctios

More information

1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable

More information

Basics of Probability Theory (for Theory of Computation courses)

Basics of Probability Theory (for Theory of Computation courses) Basics of Probability Theory (for Theory of Computatio courses) Oded Goldreich Departmet of Computer Sciece Weizma Istitute of Sciece Rehovot, Israel. oded.goldreich@weizma.ac.il November 24, 2008 Preface.

More information

Module 1 Fundamentals in statistics

Module 1 Fundamentals in statistics Normal Distributio Repeated observatios that differ because of experimetal error ofte vary about some cetral value i a roughly symmetrical distributio i which small deviatios occur much more frequetly

More information

Discrete-Time Systems, LTI Systems, and Discrete-Time Convolution

Discrete-Time Systems, LTI Systems, and Discrete-Time Convolution EEL5: Discrete-Time Sigals ad Systems. Itroductio I this set of otes, we begi our mathematical treatmet of discrete-time s. As show i Figure, a discrete-time operates or trasforms some iput sequece x [

More information

ADVANCED DIGITAL SIGNAL PROCESSING

ADVANCED DIGITAL SIGNAL PROCESSING ADVANCED DIGITAL SIGNAL PROCESSING PROF. S. C. CHAN (email : sccha@eee.hku.hk, Rm. CYC-702) DISCRETE-TIME SIGNALS AND SYSTEMS MULTI-DIMENSIONAL SIGNALS AND SYSTEMS RANDOM PROCESSES AND APPLICATIONS ADAPTIVE

More information

Time-Domain Representations of LTI Systems

Time-Domain Representations of LTI Systems 2.1 Itroductio Objectives: 1. Impulse resposes of LTI systems 2. Liear costat-coefficiets differetial or differece equatios of LTI systems 3. Bloc diagram represetatios of LTI systems 4. State-variable

More information

Since X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain

Since X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain Assigmet 9 Exercise 5.5 Let X biomial, p, where p 0, 1 is ukow. Obtai cofidece itervals for p i two differet ways: a Sice X / p d N0, p1 p], the variace of the limitig distributio depeds oly o p. Use the

More information

Lecture 15: Learning Theory: Concentration Inequalities

Lecture 15: Learning Theory: Concentration Inequalities STAT 425: Itroductio to Noparametric Statistics Witer 208 Lecture 5: Learig Theory: Cocetratio Iequalities Istructor: Ye-Chi Che 5. Itroductio Recall that i the lecture o classificatio, we have see that

More information

Reliability and Queueing

Reliability and Queueing Copyright 999 Uiversity of Califoria Reliability ad Queueig by David G. Messerschmitt Supplemetary sectio for Uderstadig Networked Applicatios: A First Course, Morga Kaufma, 999. Copyright otice: Permissio

More information

Infinite Sequences and Series

Infinite Sequences and Series Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet

More information

Statistical Noise Models and Diagnostics

Statistical Noise Models and Diagnostics L. Yaroslavsky: Advaced Image Processig Lab: A Tutorial, EUSIPCO2 LECTURE 2 Statistical oise Models ad Diagostics 2. Statistical models of radom iterfereces: (i) Additive sigal idepedet oise model: r =

More information

Probability and statistics: basic terms

Probability and statistics: basic terms Probability ad statistics: basic terms M. Veeraraghava August 203 A radom variable is a rule that assigs a umerical value to each possible outcome of a experimet. Outcomes of a experimet form the sample

More information

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals

More information

Economics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator

Economics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters

More information

Lecture 3: August 31

Lecture 3: August 31 36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,

More information

OFDM Precoder for Minimizing BER Upper Bound of MLD under Imperfect CSI

OFDM Precoder for Minimizing BER Upper Bound of MLD under Imperfect CSI MIMO-OFDM OFDM Precoder for Miimizig BER Upper Boud of MLD uder Imperfect CSI MCRG Joit Semiar Jue the th 008 Previously preseted at ICC 008 Beijig o May the st 008 Boosar Pitakdumrogkija Kazuhiko Fukawa

More information

1 Hash tables. 1.1 Implementation

1 Hash tables. 1.1 Implementation Lecture 8 Hash Tables, Uiversal Hash Fuctios, Balls ad Bis Scribes: Luke Johsto, Moses Charikar, G. Valiat Date: Oct 18, 2017 Adapted From Virgiia Williams lecture otes 1 Hash tables A hash table is a

More information

1 Introduction to reducing variance in Monte Carlo simulations

1 Introduction to reducing variance in Monte Carlo simulations Copyright c 010 by Karl Sigma 1 Itroductio to reducig variace i Mote Carlo simulatios 11 Review of cofidece itervals for estimatig a mea I statistics, we estimate a ukow mea µ = E(X) of a distributio by

More information

Axioms of Measure Theory

Axioms of Measure Theory MATH 532 Axioms of Measure Theory Dr. Neal, WKU I. The Space Throughout the course, we shall let X deote a geeric o-empty set. I geeral, we shall ot assume that ay algebraic structure exists o X so that

More information

Lecture 7: Density Estimation: k-nearest Neighbor and Basis Approach

Lecture 7: Density Estimation: k-nearest Neighbor and Basis Approach STAT 425: Itroductio to Noparametric Statistics Witer 28 Lecture 7: Desity Estimatio: k-nearest Neighbor ad Basis Approach Istructor: Ye-Chi Che Referece: Sectio 8.4 of All of Noparametric Statistics.

More information

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1 EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum

More information

Quick Review of Probability

Quick Review of Probability Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter 2 & Teachig

More information

ECE 901 Lecture 12: Complexity Regularization and the Squared Loss

ECE 901 Lecture 12: Complexity Regularization and the Squared Loss ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality

More information

Math 216A Notes, Week 5

Math 216A Notes, Week 5 Math 6A Notes, Week 5 Scribe: Ayastassia Sebolt Disclaimer: These otes are ot early as polished (ad quite possibly ot early as correct) as a published paper. Please use them at your ow risk.. Thresholds

More information

Quick Review of Probability

Quick Review of Probability Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter & Teachig Material.

More information

Chapter Vectors

Chapter Vectors Chapter 4. Vectors fter readig this chapter you should be able to:. defie a vector. add ad subtract vectors. fid liear combiatios of vectors ad their relatioship to a set of equatios 4. explai what it

More information

Information-based Feature Selection

Information-based Feature Selection Iformatio-based Feature Selectio Farza Faria, Abbas Kazeroui, Afshi Babveyh Email: {faria,abbask,afshib}@staford.edu 1 Itroductio Feature selectio is a topic of great iterest i applicatios dealig with

More information

Lecture 27. Capacity of additive Gaussian noise channel and the sphere packing bound

Lecture 27. Capacity of additive Gaussian noise channel and the sphere packing bound Lecture 7 Ageda for the lecture Gaussia chael with average power costraits Capacity of additive Gaussia oise chael ad the sphere packig boud 7. Additive Gaussia oise chael Up to this poit, we have bee

More information

Optimum LMSE Discrete Transform

Optimum LMSE Discrete Transform Image Trasformatio Two-dimesioal image trasforms are extremely importat areas of study i image processig. The image output i the trasformed space may be aalyzed, iterpreted, ad further processed for implemetig

More information

Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.

Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n. Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator

More information

Problem Set 4 Due Oct, 12

Problem Set 4 Due Oct, 12 EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios

More information

4.3 Growth Rates of Solutions to Recurrences

4.3 Growth Rates of Solutions to Recurrences 4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES 81 4.3 Growth Rates of Solutios to Recurreces 4.3.1 Divide ad Coquer Algorithms Oe of the most basic ad powerful algorithmic techiques is divide ad coquer.

More information

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak

More information

Discrete Mathematics for CS Spring 2008 David Wagner Note 22

Discrete Mathematics for CS Spring 2008 David Wagner Note 22 CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 22 I.I.D. Radom Variables Estimatig the bias of a coi Questio: We wat to estimate the proportio p of Democrats i the US populatio, by takig

More information

ECEN 655: Advanced Channel Coding Spring Lecture 7 02/04/14. Belief propagation is exact on tree-structured factor graphs.

ECEN 655: Advanced Channel Coding Spring Lecture 7 02/04/14. Belief propagation is exact on tree-structured factor graphs. ECEN 655: Advaced Chael Codig Sprig 014 Prof. Hery Pfister Lecture 7 0/04/14 Scribe: Megke Lia 1 4-Cycles i Gallager s Esemble What we already kow: Belief propagatio is exact o tree-structured factor graphs.

More information

Advanced Stochastic Processes.

Advanced Stochastic Processes. Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.

More information

Session 5. (1) Principal component analysis and Karhunen-Loève transformation

Session 5. (1) Principal component analysis and Karhunen-Loève transformation 200 Autum semester Patter Iformatio Processig Topic 2 Image compressio by orthogoal trasformatio Sessio 5 () Pricipal compoet aalysis ad Karhue-Loève trasformatio Topic 2 of this course explais the image

More information

Problem Set 2 Solutions

Problem Set 2 Solutions CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S

More information

ON POINTWISE BINOMIAL APPROXIMATION

ON POINTWISE BINOMIAL APPROXIMATION Iteratioal Joural of Pure ad Applied Mathematics Volume 71 No. 1 2011, 57-66 ON POINTWISE BINOMIAL APPROXIMATION BY w-functions K. Teerapabolar 1, P. Wogkasem 2 Departmet of Mathematics Faculty of Sciece

More information

Bayesian Methods: Introduction to Multi-parameter Models

Bayesian Methods: Introduction to Multi-parameter Models Bayesia Methods: Itroductio to Multi-parameter Models Parameter: θ = ( θ, θ) Give Likelihood p(y θ) ad prior p(θ ), the posterior p proportioal to p(y θ) x p(θ ) Margial posterior ( θ, θ y) is Iterested

More information

62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +

62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + 62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of

More information

Lecture 1 Probability and Statistics

Lecture 1 Probability and Statistics Wikipedia: Lecture 1 Probability ad Statistics Bejami Disraeli, British statesma ad literary figure (1804 1881): There are three kids of lies: lies, damed lies, ad statistics. popularized i US by Mark

More information

Filter banks. Separately, the lowpass and highpass filters are not invertible. removes the highest frequency 1/ 2and

Filter banks. Separately, the lowpass and highpass filters are not invertible. removes the highest frequency 1/ 2and Filter bas Separately, the lowpass ad highpass filters are ot ivertible T removes the highest frequecy / ad removes the lowest frequecy Together these filters separate the sigal ito low-frequecy ad high-frequecy

More information

Monte Carlo Integration

Monte Carlo Integration Mote Carlo Itegratio I these otes we first review basic umerical itegratio methods (usig Riema approximatio ad the trapezoidal rule) ad their limitatios for evaluatig multidimesioal itegrals. Next we itroduce

More information

Report on Private Information Retrieval over Unsynchronized Databases

Report on Private Information Retrieval over Unsynchronized Databases Report o Private Iformatio Retrieval over Usychroized Databases Lembit Valgma Supervised by Vitaly Skachek May 25, 217 1 Problem Statemet There are may challeges cocerig olie privacy. Private iformatio

More information

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015 ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],

More information

6.867 Machine learning, lecture 7 (Jaakkola) 1

6.867 Machine learning, lecture 7 (Jaakkola) 1 6.867 Machie learig, lecture 7 (Jaakkola) 1 Lecture topics: Kerel form of liear regressio Kerels, examples, costructio, properties Liear regressio ad kerels Cosider a slightly simpler model where we omit

More information

Olli Simula T / Chapter 1 3. Olli Simula T / Chapter 1 5

Olli Simula T / Chapter 1 3. Olli Simula T / Chapter 1 5 Sigals ad Systems Sigals ad Systems Sigals are variables that carry iformatio Systemstake sigals as iputs ad produce sigals as outputs The course deals with the passage of sigals through systems T-6.4

More information

Lossy Compression via Sparse Linear Regression: Computationally Efficient Encoding and Decoding

Lossy Compression via Sparse Linear Regression: Computationally Efficient Encoding and Decoding ossy Compressio via Sparse iear Regressio: Computatioally Efficiet Ecodig ad Decodig Ramji Vekataramaa Dept of Egieerig, Uiv of Cambridge ramjiv@egcamacuk Tuhi Sarkar Dept of EE, IIT Bombay tuhi91@gmailcom

More information

Estimation for Complete Data

Estimation for Complete Data Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of

More information

Random Matrices with Blocks of Intermediate Scale Strongly Correlated Band Matrices

Random Matrices with Blocks of Intermediate Scale Strongly Correlated Band Matrices Radom Matrices with Blocks of Itermediate Scale Strogly Correlated Bad Matrices Jiayi Tog Advisor: Dr. Todd Kemp May 30, 07 Departmet of Mathematics Uiversity of Califoria, Sa Diego Cotets Itroductio Notatio

More information

5.1 Review of Singular Value Decomposition (SVD)

5.1 Review of Singular Value Decomposition (SVD) MGMT 69000: Topics i High-dimesioal Data Aalysis Falll 06 Lecture 5: Spectral Clusterig: Overview (cotd) ad Aalysis Lecturer: Jiamig Xu Scribe: Adarsh Barik, Taotao He, September 3, 06 Outlie Review of

More information

Approximations and more PMFs and PDFs

Approximations and more PMFs and PDFs Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.

More information

6.883: Online Methods in Machine Learning Alexander Rakhlin

6.883: Online Methods in Machine Learning Alexander Rakhlin 6.883: Olie Methods i Machie Learig Alexader Rakhli LECTURES 5 AND 6. THE EXPERTS SETTING. EXPONENTIAL WEIGHTS All the algorithms preseted so far halluciate the future values as radom draws ad the perform

More information

Oblivious Transfer using Elliptic Curves

Oblivious Transfer using Elliptic Curves Oblivious Trasfer usig Elliptic Curves bhishek Parakh Louisiaa State Uiversity, ato Rouge, L May 4, 006 bstract: This paper proposes a algorithm for oblivious trasfer usig elliptic curves lso, we preset

More information

CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5

CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5 CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio

More information

First Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise

First Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >

More information

The Minimum Distance Energy for Polygonal Unknots

The Minimum Distance Energy for Polygonal Unknots The Miimum Distace Eergy for Polygoal Ukots By:Johaa Tam Advisor: Rollad Trapp Abstract This paper ivestigates the eergy U MD of polygoal ukots It provides equatios for fidig the eergy for ay plaar regular

More information

The DOA Estimation of Multiple Signals based on Weighting MUSIC Algorithm

The DOA Estimation of Multiple Signals based on Weighting MUSIC Algorithm , pp.10-106 http://dx.doi.org/10.1457/astl.016.137.19 The DOA Estimatio of ultiple Sigals based o Weightig USIC Algorithm Chagga Shu a, Yumi Liu State Key Laboratory of IPOC, Beijig Uiversity of Posts

More information

On Random Line Segments in the Unit Square

On Random Line Segments in the Unit Square O Radom Lie Segmets i the Uit Square Thomas A. Courtade Departmet of Electrical Egieerig Uiversity of Califoria Los Ageles, Califoria 90095 Email: tacourta@ee.ucla.edu I. INTRODUCTION Let Q = [0, 1] [0,

More information

Statistical Analysis on Uncertainty for Autocorrelated Measurements and its Applications to Key Comparisons

Statistical Analysis on Uncertainty for Autocorrelated Measurements and its Applications to Key Comparisons Statistical Aalysis o Ucertaity for Autocorrelated Measuremets ad its Applicatios to Key Comparisos Nie Fa Zhag Natioal Istitute of Stadards ad Techology Gaithersburg, MD 0899, USA Outlies. Itroductio.

More information

Shannon s noiseless coding theorem

Shannon s noiseless coding theorem 18.310 lecture otes May 4, 2015 Shao s oiseless codig theorem Lecturer: Michel Goemas I these otes we discuss Shao s oiseless codig theorem, which is oe of the foudig results of the field of iformatio

More information

Lecture 11: Decision Trees

Lecture 11: Decision Trees ECE9 Sprig 7 Statistical Learig Theory Istructor: R. Nowak Lecture : Decisio Trees Miimum Complexity Pealized Fuctio Recall the basic results of the last lectures: let X ad Y deote the iput ad output spaces

More information

5. Fast NLMS-OCF Algorithm

5. Fast NLMS-OCF Algorithm 5. Fast LMS-OCF Algorithm The LMS-OCF algorithm preseted i Chapter, which relies o Gram-Schmidt orthogoalizatio, has a compleity O ( M ). The square-law depedece o computatioal requiremets o the umber

More information

Modified Logistic Maps for Cryptographic Application

Modified Logistic Maps for Cryptographic Application Applied Mathematics, 25, 6, 773-782 Published Olie May 25 i SciRes. http://www.scirp.org/joural/am http://dx.doi.org/.4236/am.25.6573 Modified Logistic Maps for Cryptographic Applicatio Shahram Etemadi

More information

Chandrasekhar Type Algorithms. for the Riccati Equation of Lainiotis Filter

Chandrasekhar Type Algorithms. for the Riccati Equation of Lainiotis Filter Cotemporary Egieerig Scieces, Vol. 3, 00, o. 4, 9-00 Chadrasekhar ype Algorithms for the Riccati Equatio of Laiiotis Filter Nicholas Assimakis Departmet of Electroics echological Educatioal Istitute of

More information

Lecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting

Lecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would

More information