Nonparmeteric Bayes & Gaussian Processes. Baback Moghaddam Machine Learning Group

Transcription

1 Nonparmeteric Bayes & Gaussian Processes Baback Moghaddam Machine Learning Group

2 Outline Bayesian Inference Hierarchical Models Model Selection Parametric vs. Nonparametric Gaussian Processes regression classification Summary

3 Bayesian Inference ) ( ) ( ), ( ), ( M D p M p M D p M D p θ θ θ = Posterior Likelihood x Prior Evidence θ θ θ d M p M D p M D p ) ( ), ( ) ( = The evidence for our model M is also called Marginal Likelihood

4 Bayesian Nutshell Posterior Likelihood x Prior p( θ D, M ) p( D θ, M ) p( θ M ) x θ θ θ θˆmap θˆml

5 Hierarchical Models α Hyperparameter Prior p( θ α, M ) θ Parameter Likelihood p( D θ, M ) d i N Data

6 Hierarchical Models Hyperprior p( α M ) α Hyperparameter Prior p( θ α, M ) θ Parameter Likelihood p( D θ, M ) d i N Data

7 Level 1: infer parameters Level 2: infer hyper-parameters Level 3: infer models Bayesian Hierarchy ), ( ), ( ), ( ),, ( M D p M p M D p M D p α α θ θ α θ = ) ( ) ( ), ( ), ( M D p M p M D p M D p α α α = ) ( ) ( ) ( M p M D p D M p

8 Model Selection by Zoubin Ghahramani

9 Bayesian Model Selection by Zoubin Ghahramani

10 Bayesian Occam s Razor M 1 p(d = d M ) M 2 all possible data sets d for any model M all d D p( D = d M ) = 1 The law of conservation of belief states that models that explain many possible data sets must necessarily assign each of them a low probability.

11 Bayesian Occam s Razor M 1 M 1 p(d = d M ) M 2 M3 M 2 observed data D M 1 : the too simple model is unlikely to generate this data M 3 M 3 : the too complex model is a little better but still unlikely M 2 : the just right model has the highest marginal likelihood

12 All the Bayesics by Zoubin Ghahramani

13 Approximation Methods for the evidence and posterior integrals Laplace s method Bayesian Information Criteria (BIC = MDL) Akaike Information Criteria (AIC) Variational Bayes (VB) Expectation Propagation (EP) Markov Chain Monte Carlo (MCMC) Exact ( Perfect ) Sampling etc

14 Parametric vs. Nonparametric Parametric the total # of parameters is fixed (property of its distribution) so it doesn t depend (grow) with the # of data points collected for prediction, knowing θ means you can throw away your data Nonparametric p( d D) p( d θ ) p( θ D) new new # of parameters can grow with the number of data so the model can adapt to the data s complexity but this typically means you can not throw away your data future predictions require access to the previous training set p( d D, α) new

15 Parametric p( d D) p( d θ ) p( θ D) n n Nonparametric p( d D, α) n α α θ d 1 d2 dn d d 1 2 dn

16 Nonparametric Bayesian Models Gaussian Process Dirichlet Process Chinese restaurant process Polya urn model Stick-breaking models Pitman-Yor process Indian buffet process Polya trees Dirichlet diffusion trees Infinite Hidden Markov Models

17 Gaussian Process Regression for modeling, prediction, curve fitting y = f (x) + ε some (mis)conceptions: not curve fitting! we want p( y x ) not just regression with Gaussian noise input x can be :,, output y (hence f ) is a scalar n,,, ATGC

18 Early GP History Thiele (1880) Kolmogorov (1941), Wiener (1949) Thompson (1956) : meteorology Matheron (1963) : Kriging in Geostatistics Whittle (1963) : geospatial prediction O Hagan (1978) : general Bayesian regression Ripley (1981): Bayesian spatial models

19 Thorvald Nicolai Thiele Danish Astronomer, Actuarian, Statistician Born 1838, died 1910 General Theory of Observations (1889) Find best predictor for time series x(t) Formulated a Kalman Filter for a GP - rediscovered by Kalman & Bucy (1960)

20 Recent GP History Bar-Shalom & Fortman (1988) : Kalman filters Poggio & Girosi (1989) : Generalized RBFs Wahba (1990) : ARMA models & splines Cressie (1993) : spatial statistics (2D/3D) Neal (1996) : MLP = GP Williams & Rasmussen (1997) : general ML Saunders (1998) : KRR (Kriging rediscovered) etc etc (just see NIPS, UAI, AISTATS, ICML )

21 Two Equivalent Views GP models Weight Space linear parametric finite-dim Random Process nonlinear nonparametric infinite-dim

22 Weight Space View Parametric and finite-dimensional Regression with basis functions φ - e.g., cubic polynomials 2 3 φ( x ) = [1 x x x ] w = w w w w ] [ T T Gaussian prior on weights let K = < f f T > = Cov(f) vector f is jointly Gaussian

23 Random Process View non-parametric and infinite-dimensional imagine increasing the length of that f vector to infinity K = ij k( x, x ) i j Informally speaking the infinite-dimensional vector f becomes a function f(x) its covariance matrix K becomes a kernel function k(x i,x j ) in the limit f(x) becomes a stochastic Gaussian Process mean function m(x) kernel function k(x, x )

24 Our Beloved Gaussian 1D 2D by Carl Rasmussen

25 Gaussians beget Gaussians 1D 2D Also : the product of Gaussians is Gaussian e.g., Gaussian prior x Gaussian likelihood Gaussian posterior by Carl Rasmussen

26 Marginalization y a = mean(x) b = mean(y) A = cov(x,x') B = cov(x,y) C = cov(y,y') x by Carl Rasmussen

27 Conditional Gaussians Jointly Gaussian (sub)vectors x and y Conditional density of x given y Schur complement of C

28 Definition of Gaussian Process f ( x) ~ GP( m( x), k( x, x')) by Carl Rasmussen

29 1D GP sample (RBF)

30 2D GP sample (RBF)

31 Covariances (Kernels) not a function of f by Carl Rasmussen

32 Squared Exponential (RBF) Kernel by Carl Rasmussen

33 Nonstationary Covariances k ( x, x') tanh( a x x' + b) = not a valid kernel (psd)! by Carl Rasmussen

34 Matern Class of Covariances by Carl Rasmussen

35 Matern Class of Covariances by Carl Rasmussen

36 Rational Quadratic Kernels by Carl Rasmussen

37 Rational Quadratic Kernels by Carl Rasmussen

38 Building New Covariances by Carl Rasmussen

39 Matlab Demo 1

40 Sampling from a GP Matlab: % given GP(fmean,K) L = chol(k) ; % K = L*L ; while 1 f = L*randn(n,1) + fmean; plot(f) pause end

41 Demo 1: Sampling from Prior Prior Cov

42 Conditional Gaussians Jointly Gaussian (sub)vectors x and y Conditional density of x given y mean covariance

43 GPR with Gaussian Noise by Carl Rasmussen

44 GPR Prediction by Carl Rasmussen

45 GPR Prediction parameter by Carl Rasmussen

46 posterior mean +/- 2σ

47 posterior mean +/- 2σ

48 Matlab Demo 2

49 Posterior Sampling for GPR % train set: (x,y) L = chol(k) ; a = L \(L\y); % a = inv(k)*y; ML = -y *a/2 sum(log(diag(l))) + const; % test set: xt (no noise) ftmean = Kt*a; V = L\Kt; Kt = K V *V; % Kt = K Kt inv(k)*kt Lt = chol(kt) ; while 1 ft = Lt*randn(m,1) + ftmean; plot(ft) end

50 Demo 2: Sampling from Posterior Prior Cov Posterior Cov

51 All the Bayesics by Zoubin Ghahramani

52 GP Marginal Likelihood by Carl Rasmussen

53 ML for RBF : exp( ( x x') 2 / 2 ) + 2 δ xx' σ by Carl Rasmussen

54 Sparse GPs by Carl Rasmussen

55 Graphical Model of GPs observed unknown by Carl Rasmussen

56 Sparse GP observed unknown

57 Full GPR (all the data) y=kα

58 Sparse GPR (subset of data) y=k ss α s

59 Advantages of GPs uses probability theory (it s not a hack!) yields full predictive distributions can be a building block : p(y x) posterior sampling automatic learning of kernel parameters principled (efficient) model selection ideal for limited training data have good generalization performance

60 The GP Bible (for ML folk) all the chapters are available online!

61 Example: GPR Pseudocode