Short course A vademecum of statistical pattern recognition techniques with applications to image and video analysis. Agenda

Transcription

1 Short course A vademecum of statistical pattern recognition techniques with applications to image and video analysis Lecture Recalls of probability theory Massimo Piccardi University of Technology, Sydney, Australia Massimo Piccardi, UTS Agenda Basic probability concepts Joint, conditional, marginal probabilities Bayes theorem Independence Cdf, pdf Mean, variance, moments Epectations Covariance matri, correlation coefficients Sample mean, sample covariance Gaussian distribution Main properties of Gaussian distributions Massimo Piccardi, UTS

2 Basic probability concepts Probability: numbers assigned to events reflecting how likely they are to occur Sample space, S: set of all possible events Probability law: mapping: event p(event) courtesy of Prof. Ricardo Gutierrez-Osuna, Teas A&M University Massimo Piccardi, UTS 3 Basic probability concepts In the eample, we have a random variable that is the outcome of a hypothetical eperiment It has 4 possible outcomes, A i, i=..4 (a discrete r.v.) Aioms on probability Aiom I: 0 p[a i ] Aiom II: p[s] = Aiom III: if A i A j =, then p[a i A j ] = p[a i ] + p[a j ] Many properties Massimo Piccardi, UTS 4

3 Joint probability Let us now consider two discrete random variables: Weather (W) and Temperature (T) Both assumed binary, i.e. only two possible values each: W: rainy (r), sunny (s) T: low (l), high (h) T W l h Take 00 samples of (W,T) and map the joint frequencies r in this table Assuming we have enough samples, we call them joint probabilities s Massimo Piccardi, UTS 5 Joint probability Joint probability of W and T, value by value: p(w = r, T = l) = 5/00 = 0.5 p(w = r, T = h) = 0/00 = 0.0 p(w = s, T = l) = 5/00 = 0.05 p(w = s, T = h) = 60/00 = 0.60 The notation with the variables, p(w,t), means the whole set of joint probabilities Each of the values can be noted as p(r,l) for short, instead of p(w = r, T = l), provided there is no ambiguity Massimo Piccardi, UTS 6

4 Joint probability The joint probabilities add up to, as they cover all possible cases (Aiom II) Thus, in the eample, only 3 of them can be arbitrarily chosen, as the fourth results from: the sum of the other 3. There are 3 independent numbers (degrees of freedom, dof) For two variables with N values each, the joint probabilities have N dof Massimo Piccardi, UTS 7 Conditional probability The concept of conditional probability is simple: calculate the desired frequencies not on all the samples, but on specific sub-sets where certain conditions are true Eample: p(w = r T = l) reads as: the probability of Weather being rainy given that the Temperature is low instead of considering all the 00 samples, one just takes those where the temperature is low (30 samples in total) out of the above, compute the frequency of rainy days: 5 out of 30 = 0.83 Massimo Piccardi, UTS 8

5 Conditional probability Given two r.v., the conditional probability fies one of the two and uses the other as the only variable NB: joint probability: D; each conditional probability: D Let us fi T = l in the eample; then, the only variable is W, with two possible values: p(w = r T = l) = 5/30 = 0.83 p(w = s T = l) = 5/30 = 0.7 they are all the possible cases and as such their sum is ; we have only one dof For variables with N values, the conditional probabilities have N dof Massimo Piccardi, UTS 9 Conditional probability In the eample: p(w = r T = l) = 5/30 =.83 p(w = s T = l) = 5/30 =.7 p(w = r T = h) = 0/70 =.4 p(w = s T = h) = 60/70 =.86 dof dof there are degrees of freedom overall for p(w T), and N(N-) for two N-valued r.v. NB: p(r, l) < p(r l) by definition (the latter has a smaller denominator!) Massimo Piccardi, UTS 0

6 Marginal probability W and T are jointly called a random vector, or, equivalently, a multivariate random variable One can obtain the marginal probability of either variable by adding up the joint probabilities for all possible values of the other (marginalisation) : ( W ) p( W T ) p =, T The above is called the sum rule In the eample: p(w = r) = 35/00 p(t = l) = 30/00 p(w = s) = 65/00 ( dof) p(t = h) = 70/00 ( dof) Massimo Piccardi, UTS Bayes theorem ( W, T ) p( W T ) p( T ) p = joint probability conditional probability of W given T marginal probability of T Always holds! It is called the product rule It is a powerful tool to break down the compleity of the joint probabilities into the product of simpler probabilities Sum rule + product rule: foundations of statistical PR Massimo Piccardi, UTS

7 Independence ( W, T ) p( W ) p( T ) p = joint probability marginal probability of W marginal probability of T If the above holds, the two r.v. are called independent Often a very desirable case Equivalent to p(w T) = p(w) and p(t W) = p(t) Does not hold here! For instance: p(r,l) = 0.5 p(r) = 0.35; p(l) = 0.30 p(r) p(l) = 0.05 Massimo Piccardi, UTS 3 A special eample Given three binary r.v., A, A and S, let us assume e, 90 samples: that p(a, A S) = p(a S) p(a S) #(A,A,S=0): 0 instead of the always true (from Bayes rule): A p(a, A S) = p(a A, S) p(a S), or 0 5 p(a, A S) = p(a A, S) p(a S) #(A The above reads as A and A are independent,a,s=): 0 given S 0 4 A Not equivalent to A and A are independent! 8 It is a relevant case, with S often called a latent p(a variable or a state and the A i being measurements,a ): A S 33 A A A A A Massimo Piccardi, UTS 4

8 Cumulative density function (cdf) The cumulative distribution function of a random variable is defined as the probability of event { t}: F(t) = p[ t] Some properties: -F( + ) = -F( - ) = 0 -F(a) F(b) if a b Matlab, Statistics Toolbo, command plot(-:0.:, cdf('norm',-:0.:,0,0.5)) Massimo Piccardi, UTS 5 Probability density function (pdf) The pdf of a continuous r.v., if it eists, is defined as the derivative of the cdf: p() = df()/d Some properties: -p() 0 - p() can be >! it is not a probability; rather, a density of probability - any p()d - Bayes still applies! Matlab, Statistics Toolbo, command plot(-:0.:, pdf('norm',-:0.:,0,0.5)) Massimo Piccardi, UTS 6

9 Mean, variance and moments The pdf, or cdf, describes the probability distribution fully; yet, sometimes we prefer to describe it in a more synthetic way Mean, or epected value: Variance: VAR µ E ( ) σ E ( µ ) The standard deviation, σ, is its square root VAR() is also = E[ ] - µe[] + µ = E[ ] - µ [ ] p( ) = d [ ] = ( µ ) p( ) d N th moment: E N N [ ] p( ) = d Massimo Piccardi, UTS 7 Epectations An epectation is an averaging operation weighted by p(); it can be etended to any function of, f(): E [ f ( ) ] f ( ) p( ) = d E[f()] is a scalar value the famous Jensen s inequality: E[f()] f(e[]) Consistently, the epectation of f(,y) over : E [ f (, y) ] f (, y) p( ) = d averages out and returns a function of the sole y p() f() Massimo Piccardi, UTS 8

10 Epectations A marginalisation can be seen as a particular epectation: ( y) p( y,) d = p( y ) p( ) d E[ p( y ) ] p = = An epectation can also be computed over a conditional probability: E [ f ( ) y] f ( ) p( y) = d Massimo Piccardi, UTS 9 Mean, variance and moments The same definitions etend to multivariate r.v., X=[,.. D ] T : The mean becomes a D vector: T [ X ] = [ µ ] [ E[ ] E[ ] T µ = E,, µ,, D = The variance becomes a D D covariance matri: COV E E [ ] T ( X ) = Σ = E ( X µ )( X µ ) = [( µ )( µ )] E[ ( µ )( µ )] [( µ )( µ )] E ( µ )( µ ) D D D D D [ ] D D D D Massimo Piccardi, UTS 0

11 Covariance matri The covariance matri is obviously a symmetric matri: only D(D + )/ dof Σ = cov( D, ) = cov(, σ D ) cov(, σ D D ) Terms cov( i, j ) measure how much i and j co-vary A valid covariance matri is also positive definite: X T Σ X > 0 for any X 0 Massimo Piccardi, UTS Correlation coefficients Terms cov( i, j ) are often epressed as correlation coefficients, ρ ij : ρ = ij cov(, ) NB: - ρ ij + (corollary of the Cauchy-Schwarz inequality) i i σ σ j j courtesy of Wkipedia Massimo Piccardi, UTS

12 (Un)correlation vs independence Two r.v., i, j, are uncorrelated iff ρ ij = 0 Two uncorrelated variables are not independent; they are only in terms of linear mutual dependencies For two uncorrelated variables, i, j, it can be easily shown that: E[ i, j ]=E[ i ] E[ j ] Independence p( i, j ) = p( i )p( j ) is a much stronger property and would guarantee: E[ in, jm ]=E[ in ] E[ jm ] for any N, M Massimo Piccardi, UTS 3 Sample mean and sample covariance At times, either p() is not available or the epectation integrals are not easy to compute Assuming a set of samples, i, i= N, is available, it is possible to approimate the mean and the covariance as: N µ E[] i sample mean N Σ E i= N [( µ ) ] ( i µ )( i µ ) N i= Epectations can be approimated in the same way (Monte Carlo methods) T sample covariance Massimo Piccardi, UTS 4

13 Gaussian distribution The Gaussian, or normal, distribution enjoys nice properties making it very popular for pdf modelling Gaussian pdf in dimension: ( µ ) σ p( ) = e πσ with µ=, σ=.5: Massimo Piccardi, UTS 5 Multivariate Gaussian distribution Gaussian pdf in D dimensions (X=[,.. D ] T ): p ( X ) = ( π ) D Σ e T ( X µ ) Σ ( X µ ) with D=, µ =0, µ =0, Σ=[.5.3;.3 ] Massimo Piccardi, UTS 6

14 Properties of Gaussian distributions Mean and variance describe the whole pdf Uncorrelation independence Covariance matri of joint probability becomes diagonal Given and jointly Gaussian, also their marginal and conditional pdfs are Gaussian Linear transformations are Gaussian: given X ~ N(µ, Σ) Y = A X + K Y ~ N(Aµ + K, AΣA T ) Massimo Piccardi, UTS 7 Properties of Gaussian distributions: eample Just an eample: given two scalar Gaussian r.v., ~ N(µ, σ ) and ~ N(µ, σ ) as marginal probabilities, consider y = + This is equivalent to X = [, ] T, A = [ ] and y = AX µ y = µ + µ ; σ y = σ + cov(, ) + σ If, have common variance, σ : σ y = σ + cov(, ) If they are also uncorrelated/independent : σ y = σ If, instead, they have maimal, positive correlation: σ y = 4σ Massimo Piccardi, UTS 8