Advanced Introduction to Machine Learning CMU-10715

Transcription

1 Advanced Introduction to Machine Learning CMU Gaussian Processes Barnabás Póczos

2 2 Some of these slides in the intro are taken from D. Lizotte, R. Parr, C. Guesterin

3 Contents Introduction Ridge Regression Gaussian Processes Weight space view Bayesian Ridge Regression + Kernel trick Function space view Prior distribution over functions + calculation posterior distributions 3

4 Contents Introduction Ridge Regression Gaussian Processes Weight space view Bayesian Ridge Regression + Kernel trick Function space view Prior distribution over functions + calculation posterior distributions 4

5 Why GPs for Regression? Here are some data points! What function did they come from? - I have no idea. Oh. Okay. Uh, you think this point is likely in the function, too? - I still have no idea. 5

6 Why GPs for Regression? You can t get anywhere without making some assumptions GPs are a nice way of expressing this prior on functions idea. Can be used in many applications: Regression Classification Optimization 6

7 Why GPs for Regression? Under certain assumptions GPs can answer the following questions Here s where the function will most likely be. (expected function) Here are some examples of what it might look like. (sampling from the posterior distribution) Here is a prediction of what you ll see if you evaluate your function at x, with confidence 7

8 1D Gaussian Distribution Parameters Mean, Variance, 2 8

9 Multivariate Gaussian 9

10 Multivariate Gaussian A 2-dimensional Gaussian is defined by a mean vector = [ 1, 2 ] a covariance matrix: 2 2 1, ,2 where i,j2 = E[ (x i i ) (x j j ) ] is (co)variance Note: is symmetric, positive semi-definite : x: x T x 0 2,1 2,2 10

11 Multivariate Gaussian examples = (0,0)

12 Multivariate Gaussian examples = (0,0)

13 Useful Properties of Gaussians Marginal distributions of Gaussians are Gaussian Given: x ( x, x ), a b (, ) a b aa ba ab bb Marginal Distribution: 13

14 Marginal distributions of Gaussians are Gaussian 14

15 Useful Properties of Gaussians Conditional distributions of Gaussians are Gaussian Notation: aa ba Conditional Distribution: ab bb 1 aa ba ab bb 15

16 Higher Dimensions Visualizing > 3 dimensions is difficult Means and marginals are practical, but then we don t see correlations between those variables Marginals are Gaussian, e.g., f(6) ~ N(µ(6), σ 2 (6)) Visualizing a multivariate Gaussian f: σ 2 (6) µ(6)

17 Why stop there? Yet Higher Dimensions Don t panic: It s just a function 17

18 Getting Ridiculous Why stop there? 18

19 Gaussian Process Definition: Probability distribution indexed by an arbitrary set (integer, real, finite dimensional vector, etc) Each element gets a Gaussian distribution over the reals with mean µ(x) These distributions are dependent/correlated as defined by k(x,z) Any finite subset of indices defines a multivariate Gaussian distribution 19

20 Gaussian Process Distribution over functions Domain (index set) of the functions can be pretty much whatever Reals Real vectors Graphs Strings Sets Most interesting structure is in k(x,z), the kernel. 20

21 Bayesian Updates for GPs How do Bayesians use a Gaussian Process? Start with GP prior Get some data Compute a posterior Ask interesting questions about the posterior 21

22 Samples from the prior distribution 22 Picture is taken from Rasmussen and Williams

23 Samples from the posterior distribution 23 Picture is taken from Rasmussen and Williams

24 Prior 24

25 Data 25

26 Posterior 26

27 Contents Introduction Ridge Regression Gaussian Processes Weight space view Bayesian Ridge Regression + Kernel trick Function space view Prior distribution over functions + calculation posterior distributions 27

28 Ridge Regression Linear regression: Ridge regression: The Gaussian Process is a Bayesian Generalization of the Ridge regression 28

29 Contents Introduction Ridge Regression Gaussian Processes Weight space view Bayesian Ridge Regression + Kernel trick Function space view Prior distribution over functions + calculation posterior distributions 29

30 Weight Space View GP = Bayesian ridge regression in feature space + Kernel trick to carry out computations The training data 30

31 Bayesian Analysis of Linear Regression with Gaussian noise 31

32 Bayesian Analysis of Linear Regression with Gaussian noise The likelihood: 32

33 Bayesian Analysis of Linear Regression with Gaussian noise The prior: Now, we can calculate the posterior: 33

34 Bayesian Analysis of Linear Regression with Gaussian noise Ridge Regression After completing the square MAP estimation 34

35 Bayesian Analysis of Linear Regression with Gaussian noise This posterior covariance matrix doesn t depend on the observations y, A strange property of Gaussian Processes 35

36 Projections of Inputs into Feature Space The reviewed Bayesian linear regression suffers from limited expressiveness To overcome the problem ) go to a feature space and do linear regression there a., explicit features b., implicit features (kernels) 36

37 Explicit Features Linear regression in the feature space 37

38 Explicit Features The predictive distribution after feature map: 38

39 Explicit Features Shorthands: The predictive distribution after feature map: 39

40 Explicit Features The predictive distribution after feature map: (*) A problem with (*) is that it needs an NxN matrix inversion... (*) can be rewritten: 40

41 Mean expression. We need: Proofs Lemma: Variance expression. We need: Matrix inversion Lemma: 41

42 From Explicit to Implicit Features The feature space always enters in the form of: Lemma: 42

43 Contents Introduction Ridge Regression Gaussian Processes Weight space view Bayesian Ridge Regression + Kernel trick Function space view Prior distribution over functions + calculation posterior distributions 43

44 Function Space View An alternative way to get the previous results Inference directly in function space Definition: (Gaussian Processes) GP is a collection of random variables, s.t. any finite number of them have a joint Gaussian distribution 44

45 Function Space View Notations: 45

46 Gaussian Processes: Function Space View 46

47 Function Space View The Bayesian linear regression is an example of GP 47

48 Special case Function Space View 48

49 Function Space View 49 Picture is taken from Rasmussen and Williams

50 Observation Function Space View Explanation 50

51 Prediction with noise free observations noise free observations 51

52 Prediction with noise free observations Goal: 52

53 Prediction with noise free observations Lemma: Proofs: a bit of calculation using the joint (n+m) dim density Remarks: 53

54 Prediction with noise free observations 54 Picture is taken from Rasmussen and Williams

55 Prediction using noisy observations The joint distribution: 55

56 Prediction using noisy observations The posterior for the noisy observations: where In the weight space view we had: 56

57 Prediction using noisy observations Short notations: 57

58 Prediction using noisy observations Two ways to look at it: Linear predictor Manifestation of the Representer Theorem 58

59 Prediction using noisy observations Remarks: 59

60 Inputs: GP pseudo code 60

61 GP pseudo code (continued) Outputs: 61

62 Results using Netlab, Sin function 62

63 Results using Netlab, Sin function Increased # of training points 63

64 Results using Netlab, Sin function Increased noise 64

65 Results using Netlab, Sinc function 65

66 Thanks for the Attention! 66