Estimating Parameters for a Gaussian pdf

Transcription

1 Pattern Recognition and achine Learning Jaes L. Crowley ENSIAG 3 IS First Seester 00/0 Lesson 5 7 Noveber 00 Contents Estiating Paraeters for a Gaussian pdf Notation... The Pattern Recognition Proble...3 ultivariate Gaussian Density Functions (cont'd...5 ore properties for ultivariate Gaussian Density Functions... 5 The partition of a Gaussian PDF... 6 Conditional Gaussian Density... 6 Likelihood Estiation for the Gaussian Paraeters...7 Sequential Estiation of Gaussian Paraeters...9 Bayesian Inference for the Gaussian Paraeters... 9 Source: Pattern Recognition and achine Learning, C.. Bishop, Springer Verlag, 006.

2 Notation x a variable X a rando variable (unpredictable value x A vector of D variables. X A vector of D rando variables. D The nuber of diensions for the vector x or X E An observation. An event. k Class index K Total nuber of classes C k The kth class. ω k The stateent (assertion that E C k k Nuber of exaples for the class k. (think = ass Total nuber of exaples. = K k = k { k } A set of k exaples for the class k. { } = {X k } k=,k {t } A set of class labels (indicators for the saples µ = E{ } The Expected Value, or Average fro the saples. L = ˆ Estiated Variance True Variance N(x; µ, σ = πσ e (x µ σ Gaussian (Noral Density function.

3 The Pattern Recognition Proble Assue that we have a sensor that produces discrete observations of the world. Each observation is an event, E. Assue that for each observation, the sensor provides a vector of D features, X Observation: (E, X Our proble is to build a box that assigns each observation to one of K classes {C k } labeled k= to K. This proble is known as Decision Theory. ˆ k = decide(e C k We can decopose this into two coponent functions d( and y( X : ˆ k d(y( X Where y( X is a discriinant function that aps RD RK d( is a decision function d(: R K { ˆ k } Generally we choose d( to ake as few istakes as possible. We can express this atheatically using probability theory as: ˆ k = arg ax{p( k X } k In this case, our priary tools are Bayes Rule, that tells us: p( k X = p( X k p( X p( k In general, p( X, p( X k and p( k are estiated fro a set of training data coposed of saple observations { X } labeled with an indicator variable {t } telling the class k for each observation. 3

4 Equivalently, we can partition the training set { } into K subsets { k } each of which contains k saples. Typically p( k is estiated as p( k = k other sources. although this can also be obtained fro The Gaussian density that allows us to estiate p( X k = N ( X µ k, k = D ($ det( k e ( X µ k T k ( X µ k p( X K = p( X K k = N ( X µ k, $ k k= k= Where the paraeters µ k (ean and k (covariance for p( X k, as well as p( k are estiated fro the training data { X } and {t }. Today we look at soe of the different ethods to copute this estiation. 4

5 ultivariate Gaussian Density Functions (cont'd ore properties for ultivariate Gaussian Density Functions Assue a feature vector X of D rando variables p( X = N ( X µ, = D ( det( e ( X µ T ( X µ The classic ethod to estiate the paraeters fro a training set { } as the first and second oents of the training data. and µ = E{ X } = µ & E{X } & % ( % ( µ = % ( = % E{X } ( %... ( %... ( % ( % ( $ ' $ E{X D }' µ D = E{( X E{ X }( X E{ X } T } $... D ' & Where = &... D & & % D D... DD ( ij = E{(X i µ i (X j µ j } = (X i µ i (X j µ j In soe cases, it is convenient to work with an inverse of the covariance: = $ This is called the precision for the training set { X }. For exaple, if each observation is corrupted by a sensor noise with ean 0 and covariance β, then the estiated covariance, ˆ is ˆ = + $ This is ore conveniently expressed with precisions, as precisions add. ˆ = + B where B = $ 5

6 The partition of a Gaussian PDF Suppose we partition the vector X of D rando variables into sub-vectors X a and X b of A and B coponents A+B=D. x % $ ' x X = $ ' X = a % $ ' $ ' X b & $ ' & x D The partition of a Gaussian rando vector is coposed of two Gaussian rando vectors. siilarly µ a % µ = $ ' and = aa ab & T % ( where µ b & $ ba ab = ab bb ' = aa ab & % ( $ ba bb ' Conditional Gaussian Density If two rando vector have Gaussian statistics, then their conditional probability is Gaussian. Suppose that X a and X b are both Gaussian. where: p( X a X b = N ( X a µ a b, a b = A ( µ a b = µ a + ab bb ( X b µ b det( a b e ( X µ a b T a b ( X µ a b and a b = aa ab bb ba The derivation is in Bishop pages

7 Likelihood Estiation for the Gaussian Paraeters There are alternative ethods to define the paraeters for a Gaussian pdf. For exaple, we can copute the ost likely paraeters for the data set as a axiu likelihood estiate. Consider saple observations X = { }. Assuing that the are independent, p( X, X µ, = p( X µ, p( X µ, so that p( X, X,..., X µ, = N ( X µ, we define this as the Likelihood. (recall X = { }. L( µ, X = p(x µ, = N ( X µ, in general is it ore convenient to work with the Log-likelihood: L( µ, X = ln{l( µ, X} = ln{n ( X µ,} L( µ, X = ln{l( The log likelihood for X is µ, X} = ln{n ( X µ,} L( µ, X = ln{p(x µ, } = D ln{$} ln{det(} using algebra we can show that L( µ, X = % $ (X µ $ µ setting this to zero we obtain %( µ T ( µ 7

8 µ L = Siilarly, but setting L( µ, X $ ij = 0 we can obtain L = ( X µ L T ( X µ L Notice that the axiu likelihood gives a biased estiate for Σ. If we evaluate draw our Saples fro a noral density with µ and p( N ( µ, we will discover that µ L = µ but L = The unbiased estiate would be: = $ ( X µ L ( X µ L T L and converge as grows larger. 8

9 Sequential Estiation of Gaussian Paraeters In any on-line applications, new data ust be added to the estiation as it arrives. This can be accoplished with a Bayesian approach to estiation. In Bayesian recognition we are interested in accuulating evidence. Each new saple is evidence for µ L and L. We can see this by reforulating the estiation sequentially, as if the data arrive in teporal sequence. The estiate after points is: ( µ L = we can decopose this to ( µ L = + ( µ L = X + ( µ L = ( µ L + ( ( µ L ( µ L We can interpret this as saying that the influence of the new data decreases as /. Clearly, as increases the contribution fro each data point gets saller. Bayesian Inference for the Gaussian Paraeters Bayesian estiation considers the estiation as a proble of evidence accuulation. To keep the algebra siple, consider that case where D= and suppose that σ is fixed. as before, our saple set is X = { }. p(x µ = N ( µ, = ( N $ % e ( $µ Note that p(x µ is NOT a pdf and does NOT su to. 9

10 If we choose a prior p(µ =N (µ µ o, o asd then the posterieur density is a production of two quadratics, and hence also Gaussian. p(µ X =N (µ µ, Thus p(µ X p(x µp(µ where µ = + µ o + o o + µ L and = + o where µ L = Not that = + is ore conveniently expressed as the precision: λ=/σ o because precision are cobined by addition. = λ =λ o +Μλ Thus we can forulate: % ( p(µ X ' p(µ $ p( µ * p( µ & and p(x = N( µ, - $ ( ' e % & +,( X * %µ % 0

11 which is equivalent to p(x = N(X µ, % $ ( ' * & % e + ( ' & *,(X +µ +