Short course A vademecum of statistical pattern recognition techniques with applications to image and video analysis. Agenda

Transcription

1 Short course A vademecum of statistica attern recognition techniques with aications to image and video anaysis Lecture 3 Density estimation Massimo Piccardi University of Technoogy, Sydney, Austraia Massimo Piccardi, UTS Agenda Density estimation Likeihood function Maximum-ikeihood density estimation Gaussian mixture modes on-arametric (Kerne Density Estimation) Exame aers Massimo Piccardi, UTS 2

2 Density estimation A the reviewed cassification criteria use dfs Accurate modeing of dfs from sets of sames is therefore a fundamenta task in Bayesian cassification (robabiity) density (function) estimation Parametric: Gaussians, Gaussian mixtures etc on-arametric: histogram, k-nearest neighbours, KDE, meanshift etc Gaussian mixtures and KDE in the foowing B: sames modes: density estimation modes sames: saming Massimo Piccardi, UTS 3 Density estimation Let us have a set of sames, X = {x i }, i=.., that are a generated from the same distribution and indeendenty of one another (i.i.d. sames) Density estimation is erformed by choosing an aroriate df mode (for exame, Gaussian) and fitting its arameters, θ, to the set of sames The robabiity of the entire set, X, is then given by: ( X θ ) = ( θ ) i= x i Massimo Piccardi, UTS 4

3 Likeihood function We define a function that we ca the ikeihood of the arameters given the set of sames, L(θ X): ( θ X ) ( X θ ) L = ease note that L is a function of θ given X, whereas is a function of X given θ. Often, L(θ X) is noted just as L(θ). The og-ikeihood, LL(θ X), is often used instead of L for these main reasons: og (or n) is a monoticay increasing function of its argument: maxima of the argument are maxima aso of the og og of roduct = sum of og: removes roduct oerator avoids numerica underfow during the evauation of L: = Σ = -000 Massimo Piccardi, UTS 5 ML density estimation Find arameters θ maximising L(θ) or, equivaenty, LL(θ) It is ossibe to undertake direct maximisation by differentiation: comute dl/dθ = 0 and sove for θ If cosed-form soutions are nor ossibe, iterative methods can be used instead (e.g. ewton-rahson) dl/dθ θ n+ is a better aroximation than θ n for the zero of dl/dθ θ n+ θ n Massimo Piccardi, UTS 6

4 ML for the Gaussian Easy case for ML density estimation: Gaussian df. Its arameters are θ = {µ, Σ} µ Σ ML ML = = i= ( xi µ ML )( xi µ ML ) i= x i T The variance is a so-caed biased estimate; it shoud be mutiied by /(-) to unbias; yet, the correction is negigibe for any reasonabe Massimo Piccardi, UTS 7 The EM aroach Direct maximisation of the og-ikeihood is often inconvenient or just difficut A very ouar aternative for ML estimation is given by the Exectation-Maximization (EM) aroach In the EM aroach, instead of maximising function L(θ) (or LL(θ)), we maximise another function, Q(θ) The aroach works since obtaining a maximum for Q(θ) guarantees a maximum or at east an increase in LL(θ) over an initiay arbitrary choice of θ The aroach was roosed by Demster, Laird, Rubin (DLR) in Maximum Likeihood from Incomete Data via the EM Agorithm, JSTOR, 977 a 22-age aer with 23 reviewers Read H. Tagare, A Gente Introduction to the EM agorithm. Part I: Theory, htt://noode.med.yae.edu/hdtag/ubs/em.s for an easy introduction Massimo Piccardi, UTS 8

5 The EM aroach EM osits the existence of atent variabes (n.b it aso works for missing data), Y, and sets a different target for maximisation: Q ( θ, θ ) = E[ n ( X,Y θ ) X, θ ] = n ( X, y θ ) ( y X, θ )dy L(θ) is therefore caed the incomete data ikeihood, whie Q(θ,θ ) is the exected vaue of the comete data ogikeihood, n (X,Y θ), on conditiona robabiity (y X,θ ) For the aroach to make sense, Q(θ,θ ) must be such that its maximisation is easier than that of L(θ) = Massimo Piccardi, UTS 9 The EM aroach Finding an exression for Q(θ,θ ) requires an exression for og (X,y θ) and for density (y X, θ ) y is an instance of a the atent variabes in the mode We assume that the x i are aso indeendent given y; hence, og (X,y θ) = og i (x i,y θ) = Σ i og (x i,y θ) An exression for (x i,y θ) must be simer than for (x i θ); moreover, for exonentia densities, the og goes An exression for (y X, θ ) may be hard to find Q(θ,θ ) must then be differentiated in θ and maxima found Massimo Piccardi, UTS 0

6 The EM aroach Q(θ,θ ) must be evauated iterativey unti convergence Each iteration is guaranteed to increase the incomete og-ikeihood, LL(θ X) - that is what we want The maximum we find in the arameter sace uon convergence is a oca one! Its osition deends on the choice of the initia θ Each iteration consists of two stes: the E ste, where we comute the udated (y X, θ ) the M ste, where better θ are chosen by differentiation of Q(θ,θ ) Massimo Piccardi, UTS The EM agorithm. Choose an initia θ 2. E ste: comute (y X,θ ) 3. M ste: comute Q(θ,θ ) and find its maxima, θ new 4. Check for convergence of either L or θ; if not, θ θ new and return to ste 2 Massimo Piccardi, UTS 2

7 The EM agorithm courtesy of Prof. Ricardo Gutierrez-Osuna, Texas A&M University Massimo Piccardi, UTS 3 Gaussian mixture mode (GMM) A Gaussian mixture mode (GMM) is a mode with a finite number, M, of Gaussian comonents Each -th comonent has a robabiity in the mixture, or weight, α, and its own arameters, µ, Σ, =..M As a generative mode, a GMM can be samed ike this (ancestra saming): first, draw one vaue out of M according to the discrete distribution given by the α ; this icks the comonent second, draw a same from a Gaussian distribution of arameters µ, Σ (for instance, with Box-Muer and Choesky) Massimo Piccardi, UTS 4

8 Gaussian mixture mode (GMM) The df of a GMM is given by: M ( x) = α ( x µ Σ ) =, GMMs are very usefu and ouar modes since they can estimate mutimoda distributions accuratey Massimo Piccardi, UTS 5 GMM: size of arameters With D-dimensiona data, the GMM arameters size are as: for each weight, α (aka P(ω ) or π ): a scaar for each mean, µ : a D x vector for each covariance matrix, Σ : a D x D symmetric matrix with D(D+)/2 dof, if fu D, if diagona, if sherica At times, the covariances are chosen to be the same for a comonents Massimo Piccardi, UTS 6

9 The ikeihood function for a GMM: an exame Just an exame to disay the ikeihood function for a sime GMM We can easiy visuaise functions of 2 arameters, therefore we choose the foowing sime mode: ( x) =. 3 ( x µ, σ =. 6) ( x µ, σ ) = where the ony arameters are µ and µ 2 µ and µ 2 are made vary in range in 0.5 stes Massimo Piccardi, UTS 7 The data 300 uni-dimensiona sames Their histogram: Massimo Piccardi, UTS 8

10 Likeihood surface Two maxima found, at: µ = 0.5, µ 2 = -4.5 (ikeihood: ) µ = -4, µ 2 = (ikeihood: ) Massimo Piccardi, UTS 9 Quaity of fitting One maximum is ceary better than the other maximum at µ = 0.5, µ 2 = -4.5 maximum at µ = -4, µ 2 = µ 2 µ µ µ 2 Massimo Piccardi, UTS 20

11 EM for GMM EM is the main too to find arameters for a GMM with maximum ikeihood EM for GMMs assumes that, for each x i same, there exists a atent discrete r.v., y i, whose vaue is the index, ={..M}, of the Gaussian comonent resonsibe for generating that same It is assumed that each x i deends ony on its y i Therefore: ( xi, yi ) = ( xi yi ) ( yi ) = ( xi µ y, Σ ) i y α i yi ote that (x i, y i ) is much simer than (x i ), as we wanted: M ( xi ) = α ( xi µ, Σ ) = Massimo Piccardi, UTS 2 EM for GMM An exression for (y X, θ ) is derived by assuming that each y i deends ony on x i : For a GMM, Q(θ,θ ) becomes [Bimes 98]: Q + ( y X, θ ) = ( yi xi, θ ) M ( θ, θ ) = n ( α ) ( xi, θ ) M = i= n = i= ( ( x µ, Σ )) ( x, θ ) i i= Q(θ,θ ) is then differentiated to find the maxima; a constraint, Σ =..M α =, needs to be added to find meaningfu weights, α i + Massimo Piccardi, UTS 22

12 EM for GMM: re-estimation formuas E ste: M ste: α G ( ) ( xi µ, Σ ) y = = i xi, θ M αk G( xi µ k, Σk ) α new i= k= ( x θ ) = i, i new i= = µ Σ new = i= x ( x, θ ) ( x, θ ) i i new new T ( xi µ )( xi µ ) ( xi, θ ) i= i= ( x, θ ) i (aka resonsibiity) Massimo Piccardi, UTS 23 EM for mixture modes: caveats Singuarities may arise during training: one comonent modes one datum ony, tighty its covariance tends to 0 (x θ ) tends to (x θ) aso tends to we have reached a maximum of the ikeihood; yet, the mode s arametrisation is not usefu Common trick: add some esion to Σ The oca maximum we reach uon convergence may vary heaviy with the initia arameters If the data are drawn in a sequence, there exist faster, onine/incrementa versions of the re-estimation formuas Massimo Piccardi, UTS 24

13 ML and MAP density estimation ML density estimation: θ ML = arg max θ ( ( X θ )) MAP density estimation: think of arameters θ as r.v. themseves, aowing some rior distribution (θ): θ MAP = arg max θ ( ( θ X ) ( X θ ) ( θ )) MAP is usefu to favour certain vaues of θ B: Do not confuse ML/MAP density estimation with ML/MAP cassification Massimo Piccardi, UTS 25 The evidence function Both ML and MAP are oint estimates of the arameters. It is aso ossibe to marginaise θ: ( X ) ( X θ ) ( θ ) = dθ The margina ikeihood above is known as the evidence function It can be used to comare different modes i.e. different choices of (X θ), (θ) Massimo Piccardi, UTS 26

14 on-arametric estimators GMMs beong to the genera category of arametric density estimators A different aroach to density estimation can be taken by choosing modes with a minima number of arameters Widesread aroaches incude: histograms k-nearest neighbours (k) kerne density estimation (KDE) mean-shift vector Massimo Piccardi, UTS 27 Histogram With the histogram, the data sace is divided in a reguar grid (each eement is caed a bin) (x) is uniform within each bin and given by: (number of sames in the bin)/(tota number of sames) Limitations: shar/non-smooth estimate deends on the size of the bins deends on the aignment of the grid number of bins grows exonentiay with D Usefu for visuaization in or 2D Massimo Piccardi, UTS 28

15 Generic non-arametric estimation (x) ~ k/v, where V is the voume surrounding x k is the number of sames in V is the tota number of sames it rovides a good estimate if is arge, k grows with and V is sma enough for (x) to be constant Two main aroaches, KDE and k in KDE, V is fixed and k comuted from data set in k, k is fixed and V comuted from data set The mean shift vector aroach is a ost rocessing of KDE that finds the distribution s modes exicity; ends u in a mixture mode, but in a non-arametric manner Massimo Piccardi, UTS 29 Kerne Density Estimation A kerne function, K(u) (aka Parzen window), is fit centred on each same Tyica kernes (in D): K u = u 2 Uniform ( ) Triange K ( u) = ( u ) u Eanechnikov Gaussian K K ( u) = 3 ( u 2 ) u 4 ( ) = ( ) 2 u 2π 2 ex u 2 Massimo Piccardi, UTS 30

16 KDE df Kernes are then a added u and sum normaised; this is the KDE df (in D dimensions): h ( x) = D i= x x K h h is caed the bandwidth Kernes are tyicay radiay symmetric, so there is ony one scaar arameter, h, aso in D dimensions; it equates to a sherica covariance matrix Given x, evauation of (x) is comutationay heavy: high run-time execution time i Massimo Piccardi, UTS 3 KDE bandwidth How to choose the bandwidth? The EM agorithm woud ead to a useess soution: h = 0 A seudo-ikeihood can be used in ace of the standard ikeihood: when evauating (x i ) in L(θ), eave the kerne centred on it out; in this way, same x i has to be exained by its cosest neighbours Many other methods to estimate the bandwidth: Maxima Smoothing Princie, Least Squares Cross Vaidation, Biased Cross Vaidation, Smoothed Cross Vaidation, many! B. A. Turach. Bandwidth Seection in Kerne Density Estimation: A Review. Technica Reort Université Cathoique de Louvain, Begium, 993 Massimo Piccardi, UTS 32

17 GMM vs KDE GMM GMM M ( x) = α G( x µ, Σ ) = KDE (Gaussian kerne) ( x) = G( x, Σ) KDE x i i= one comonent er observation! ony one Σ for a comonents equa weights Simiarity ony notationa centred in the observation Massimo Piccardi, UTS 33 GMM vs KDE GMM KDE (Gaussian kerne) Massimo Piccardi, UTS 34

18 GMM vs KDE: exame Exame with 3 modes in 2D Parameters in GMM with different constraints on covariance; how mode changes fu: * * 3 = 7 arameters diagona: * * 2 = 4 arameters sherica: * * = arameters shared fu: * 2 + * 3 = arameters shared diagona: * 2 + * 2 = 0 arameters shared sherica: * 2 + * = 9 arameters Sherica mode not as restrictive for KDE sherica kerne: arameter Massimo Piccardi, UTS 35 The data Massimo Piccardi, UTS 36

19 y y GMM, fu covariance 5 df(obj,[x,y]) x og-ikeihood = Massimo Piccardi, UTS 37 GMM, diagona covariance 5 df(obj,[x,y]) x og-ikeihood = Massimo Piccardi, UTS 38

20 y y GMM, shared fu covariance 5 df(obj,[x,y]) x og-ikeihood = Massimo Piccardi, UTS 39 GMM, shared diagona covariance 5 df(obj,[x,y]) x og-ikeihood = Massimo Piccardi, UTS 40

21 y KDE, sherica Gaussian kerne 5 df(obj,[x,y]) x ony free arameter! og-ikeihood comares with GMM Massimo Piccardi, UTS 4 Exame aers Aication: background subtraction Extracting moving objects in a video Mixture modes: C. Stauffer and W.E.L. Grimson, Adative background mixture modes for rea-time tracking, Proc. IEEE CVPR 999, cites on Googe Schoar (9 ov 08). Many modification aers have foowed. KDE: A. Egamma, D. Harwood, and L.S. Davis, on-arametric mode for background subtraction, Proc. ECCV 2000, Massimo Piccardi, UTS 42

22 Additiona materias A reference for KDE bandwidth seection: B. A. Turach. Bandwidth Seection in Kerne Density Estimation: A Review. Technica Reort Université Cathoique de Louvain, Begium, 993 Massimo Piccardi, UTS 43