PROBABILITY DISTRIBUTIONS. J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception

Transcription

1 PROBABILITY DISTRIBUTIONS

2 Credits 2 These slides were sourced and/or modified from: Christopher Bishop, Microsoft UK

3 Parametric Distributions 3 Basic building blocks: Need to determine given Representation: or? Recall Curve Fitting

4 4 Binary Variables Coin flipping: heads=1, tails=0 Bernoulli Distribution

5 END OF LECTURE MON SEPT 20, 2010

6 Guidelines for Paper Presentations 6 Everyone should read the paper prior to the presentation and be prepared to discuss it. What is the objective? What tools from the course are being used? What did you not understand?

7 Guidelines for Paper Presentations 7 For the presenter: Your presentation should be around 10 minutes long no more than 15! (About 10 slides) What is the objective? What tools from the course are being used and how? What are the key ideas? What are the unsolved problems? Be prepared to answer questions from other students.

8 8 Binary Variables N coin flips: Binomial Distribution

9 9 Binomial Distribution

10 Parameter Estimation 10 ML for Bernoulli Given:

11 11 Parameter Estimation Example: Prediction: all future tosses will land heads up Overfitting to D

12 12 Beta Distribution Distribution over. where Γ(x) = Note that 0 u x 1 e u du Γ(x + 1) = xγ(x) Γ(1) = 1 Γ(x + 1) = x! when x is an integer.

13 13 Bayesian Bernoulli The Beta distribution provides the conjugate prior for the Bernoulli distribution.

14 14 Beta Distribution

15 15 Prior Likelihood = Posterior

16 16 Properties of the Posterior As the size N of the data set increases

17 17 Multinomial Variables 1-of-K coding scheme:

18 18 ML Parameter estimation Given: To ensure, use a Lagrange multiplier, λ See Appendix E for a review of Lagrange multipliers.

19 19 The Multinomial Distribution,,, for j k where N m 1,m 2,,m K N! m 1!m 2!,m K!

20 20 The Dirichlet Distribution Since K µ k = 1 k =1 Conjugate prior for the multinomial distribution.

21 21 Bayesian Multinomial

22 22 Bayesian Multinomial

23 23 The Gaussian Distribution

24 Central Limit Theorem 24 The distribution of the sum of N i.i.d. random variables becomes increasingly Gaussian as N grows. Example: N uniform [0,1] random variables.

25 25 Geometry of the Multivariate Gaussian where Δ Mahalanobis distance from µ to x Eigenvector equation: Σu i = λ i u i where (u i,λ i ) are the ith eigenvector and eigenvalue of Σ. Note that Σ real and symmetric λ i real. Proof? See Appendix C for a review of matrices and eigenvectors.

26 26 Geometry of the Multivariate Gaussian Δ = Mahalanobis distance from µ to x where (u i,λ i ) are the ith eigenvector and eigenvalue of Σ. or y = U(x - µ)

27 27 Moments of the Multivariate Gaussian thanks to anti-symmetry of z

28 28 Moments of the Multivariate Gaussian

29 29 Partitioned Gaussian Distributions

30 30 Partitioned Conditionals and Marginals

31 31 Partitioned Conditionals and Marginals

32 Maximum Likelihood for the Gaussian 32 Given i.i.d. data, the log likelihood function is given by Sufficient statistics

33 Maximum Likelihood for the Gaussian 33 Set the derivative of the log likelihood function to zero, and solve to obtain Similarly Recall: If x and a are vectors, then ( x xt a) = ( x at x) = a

34 34 Maximum Likelihood for the Gaussian Under the true distribution Hence define

35 Bayesian Inference for the Gaussian (Univariate Case) 35 Assume σ 2 is known. Given i.i.d. data, the likelihood function for µ is given by This has a Gaussian shape as a function of µ (but it is not a distribution over µ ).

36 Bayesian Inference for the Gaussian (Univariate Case) 36 Combined with a Gaussian prior over µ, this gives the posterior Completing the square over µ, we see that

37 Bayesian Inference for the Gaussian 37 where Note: Shortcut: Get Δ 2 in form aµ 2 2bµ + c = a(µ b / a) 2 + const and identify µ N = b / a 1 σ N 2 = a

38 38 Bayesian Inference for the Gaussian Example: for N = 0, 1, 2 and 10.

39 39 Bayesian Inference for the Gaussian Sequential Estimation The posterior obtained after observing N { 1 data points becomes the prior when we observe the N th data point.

40 40 Bayesian Inference for the Gaussian Now assume µ is known. The likelihood function for λ = 1/ σ 2 is given by This has a Gamma shape as a function of λ.

41 41 Bayesian Inference for the Gaussian The Gamma distribution

42 42 Bayesian Inference for the Gaussian Now we combine a Gamma prior, with the likelihood function for λ to obtain which we recognize as with

43 43 Bayesian Inference for the Gaussian If both µ and λ are unknown, the joint likelihood function is given by We need a prior with the same functional dependence on µ and λ.

44 44 Bayesian Inference for the Gaussian The Gaussian-gamma distribution

45 Bayesian Inference for the Gaussian 45 The Gaussian-gamma distribution

46 Bayesian Inference for the Gaussian 46 Multivariate conjugate priors µ unknown, Λ known: p( µ ) Gaussian. Λ unknown, µ known: p( Λ) Wishart, µ and Λ unknown: p( µ, Λ) Gaussian-Wishart,

47 47 Student s t-distribution where Infinite mixture of Gaussians.

48 48 Student s t-distribution

49 49 Student s t-distribution Robustness to outliers: Gaussian vs t-distribution.

50 50 Student s t-distribution The D-variate case: where. Properties:

51 Periodic variables 51 Examples: time of day, direction, We require

52 52 von Mises Distribution This requirement is satisfied by where is the 0 th order modified Bessel function of the 1 st kind. (The von Mises distribution is the intersection of an isotropic bivariate Gaussian with the unit circle)

53 53 von Mises Distribution

54 54 Maximum Likelihood for von Mises Given a data set, function is given by, the log likelihood Maximizing with respect to µ 0 we directly obtain Similarly, maximizing with respect to m we get which can be solved numerically for m ML.

55 55 Mixtures of Gaussians Old Faithful data set Duration of last eruption (min) Time to next eruption (min) Single Gaussian Mixture of two Gaussians

56 56 Mixtures of Gaussians Combine simple models into a complex model: Component Mixing coefficient K=3

57 57 Mixtures of Gaussians

58 58 Mixtures of Gaussians Determining parameters log likelihood µ, σ and π using maximum Log of a sum; no closed form maximum. Solution: use standard, iterative, numeric optimization methods or the expectation maximization algorithm (Chapter 9).