INTRODUCTION TO BAYESIAN INFERENCE PART 2 CHRIS BISHOP

Transcription

1 INTRODUCTION TO BAYESIAN INFERENCE PART 2 CHRIS BISHOP

2

3 Personal Healthcare Revolution Electronic health records (CFH) Personal genomics (DeCode, Navigenics, 23andMe) X-prize: first $10k human genome technology NIH: $1k by 2014 Microsoft Research Cambridge: PhD Scholarships Internships: 3 months Postdoctoral Fellowships

4 Why Probabilities? Class Image vector cancer or normal

5 Decisions One-step solution train a function to decide the class Two-step solution inference : infer posterior probabilities decision : use probabilities to decide the class

6 Minimum Misclassification Rate

7 Why Separate Inference and Decision? Minimizing risk (loss matrix may change over time) Reject option Unbalanced class priors Combining models

8 Loss Matrix Decision True class

9 Minimum Expected Loss Regions are chosen, at each x, to minimize

10 Reject Option

11 Unbalanced class priors In screening application, cancer is very rare Use balanced data sets to train models, then use Bayes theorem to correct the posterior probabilities

12 Combining models Image data and blood tests Assume independent for each class:

13 Binary Variables (1) Coin flipping: heads=1, tails=0 Bernoulli Distribution

14 Expectation and Variance In general For Bernoulli

15 Likelihood function Data set Likelihood function

16 Prior Distribution Simplification if prior has same functional form as likelihood function Called conjugate prior

17 Beta Distribution

18 Posterior Distribution

19 Posterior Distribution

20 Properties of the Posterior As the size N of the data set increases

21 Predictive Distribution What is the probability that the next coin flip will be heads?

22 The Exponential Family where is the natural parameter We can interpret g( ) as the normalization coefficient

23 Likelihood Function Give a data set, Depends on data through sufficient statistics

24 Expected Sufficient Statistics

25 Conjugate priors For the exponential family Combining with the likelihood function, we get Prior corresponds to º pseudo-observations with statistic Â

26 Bernoulli revisited The Bernoulli distribution Comparing with the general form we see that and so Logistic sigmoid

27 Bernoulli revisited The Bernoulli distribution in canonical form where

28 The Gaussian Distribution

29 Likelihood Function

30 Bayesian Inference unknown mean Assume ¾ 2 is known Data set Likelihood function for ¹

31 Bayesian Inference unknown mean Conjugate prior is a Gaussian which gives a Gaussian posterior

32

33 Bayesian Inference unknown precision Now assume ¹ is known Likelihood function for precision = 1/¾ 2

34 Conjugate prior Gamma distribution

35 Unknown Mean and Precision Likelihood function Gaussian-gamma distribution

36 Gaussian-gamma Distribution

37 Linear Regression (1) Noisy sinusoidal data

38 Linear Regression (2) Linear combination of basis functions Noise model Likelihood function

39 Linear Regression (3) Polynomial basis functions

40 Linear Regression (4) Define a conjugate prior over w Combining with likelihood function gives the posterior where

41 Simple Example (1) Data from straight line with Gaussian noise First order polynomial model

42 Simple Example (2) 0 data points observed Prior Data Space

43 Simple Example (3) 1 data point observed Likelihood Posterior Data Space

44 Simple Example (4) 2 data points observed Likelihood Posterior Data Space

45 Simple Example (5) 20 data points observed Likelihood Posterior Data Space

46 Predictive Distribution (1) Predict t for new values of x by integrating over w: where

47 Predictive Distribution (3) Example: Sinusoidal data, 9 Gaussian basis functions, 1 data point

48 Predictive Distribution (4) Example: Sinusoidal data, 9 Gaussian basis functions, 2 data points

49 Predictive Distribution (5) Example: Sinusoidal data, 9 Gaussian basis functions, 4 data points

50 Predictive Distribution (6) Example: Sinusoidal data, 9 Gaussian basis functions, 25 data points

51 Bayesian Model Comparison (1) Alternative models M i, i=1,,l Predictive distribution is a mixture Model selection: keep only most probable model

52 Bayesian Model Comparison (2) From Bayes theorem posterior model evidence (marginal likelihood) prior For equal priors, models ranked by marginal likelihood

53 Bayesian Model Comparison (4) For a model with parameters w Note that

54 Bayesian Model Comparison (5) Consider model with a single parameter w

55 Bayesian Model Comparison (6) Taking logarithms, we obtain Negative With M parameters, all assumed to have the same ratio, we get

56 Linear Regression revisited Marginal likelihood

57 Linear Regression revisited Noisy sinusoidal data

58 Linear Regression revisited Polynomial of order M,

59 Bayesian Model Comparison Matching data and model complexity