Deep Gaussian Processes

Transcription

1 Deep Gaussian Processes Neil D. Lawrence 8th April 2015 Mascot Num 2015

2 Outline Introduction Deep Gaussian Process Models Variational Methods Composition of GPs Results

3 Outline Introduction Deep Gaussian Process Models Variational Methods Composition of GPs Results

4 Deep Neural Network x 1 x 2 x 3 x 4 x 5 x 6 input layer h 1 1 h 1 2 h 1 3 h 1 4 h 1 5 h 1 6 h 1 7 h 1 8 hidden layer 1 h 2 1 h 2 2 h 2 3 h 2 4 h 2 5 h 2 6 hidden layer 2 h 3 1 h 3 2 h 3 3 h 3 4 hidden layer 3 y 1 label

5 Deep Neural Network x 1 x 2 x 3 x 4 x 5 x 6 given x h 1 1 h 1 2 h 1 3 h 1 4 h 1 5 h 1 6 h 1 7 h 1 8 h 1 = φ (W 1 x) h 2 1 h 2 2 h 2 3 h 2 4 h 2 5 h 2 6 h 2 = φ (W 2 h 1 ) h 3 1 h 3 2 h 3 3 h 3 4 h 3 = φ (W 3 h 2 ) y 1 y = w 4 h 3

6 Mathematically h 1 = φ (W 1 x) h 2 = φ (W 2 h 1 ) h 3 = φ (W 3 h 2 ) y = w 4 h 3

7 Overfitting Potential problem: if number of nodes in two adjacent layers is big, corresponding W is also very big and there is the potential to overfit. Proposed solution: dropout. Alternative solution: parameterize W with its SVD. W = UΛV or W = UV where if W R k 1 k 2 then U R k 1 q and V R k 2 q, i.e. we have a low rank matrix factorization for the weights.

8 Deep Neural Network x 1 x 2 x 3 x 4 x 5 x 6 input layer latent layer 1 h 3 1 h 3 2 h 3 3 h 3 4 h 3 5 h 3 6 h 3 7 h 3 8 hidden layer 1 latent layer 2 h 2 1 h 2 2 h 2 3 h 2 4 h 2 5 h 2 6 hidden layer 2 latent layer 3 h 1 1 h 1 2 h 1 3 h 1 4 hidden layer 3 y 1 label

9 Deep Neural Network x 1 x 2 x 3 x 4 x 5 x 6 z 1 given x = V 1 x h1 = g (U1z1) h 3 1 h 3 2 h 3 3 h 3 4 h 3 5 h 3 6 h 3 7 h 3 8 z 2 = V 2 h 3 h 2 1 h 2 2 h 2 3 h 2 4 h 2 5 h 2 6 h 2 = g (U 2 z 2 ) z 3 = V 1 h 2 h 1 1 h 1 2 h 1 3 h 1 4 h 3 = g (U 3 z 3 ) y 1 y = w 4 h 3

10 Mathematically z 1 = V 1 x h 1 = φ (U 1 z 1 ) z 2 = V 2 h 1 h 2 = φ (U 2 z 2 ) z 3 = V 3 h 2 h 3 = φ (U 3 z 3 ) y = w 4 h 3

11 A Cascade of Neural Networks z 1 = V 1 x z 2 = V 2 φ (U 1z 1 ) z 3 = V 3 φ (U 2z 2 ) y = w 4 z 3

12 Replace Each Neural Network with a Gaussian Process z 1 = f (x) z 2 = f (z 1 ) z 3 = f (z 2 ) y = f (z 3 ) This is equivalent to Gaussian prior over weights and integrating out all parameters and taking width of each layer to infinity.

13 Gaussian Processes: Extremely Short Overview

14 Gaussian Processes: Extremely Short Overview

15 Gaussian Processes: Extremely Short Overview

16 Gaussian Processes: Extremely Short Overview

17 Outline Introduction Deep Gaussian Process Models Variational Methods Composition of GPs Results

18 Mathematically Composite multivariate function g(x) = f 5 (f 4 (f 3 (f 2 (f 1 (x)))))

19 Why Deep? Gaussian processes give priors over functions. Elegant properties: e.g. Derivatives of process are also Gaussian distributed (if they exist). For particular covariance functions they are universal approximators, i.e. all functions can have support under the prior. Gaussian derivatives might ring alarm bells. E.g. a priori they don t believe in function jumps.

20 Process Composition From a process perspective: process composition. A (new?) way of constructing more complex processes based on simpler components. Note: To retain Kolmogorov consistency introduce IBP priors over latent variables in each layer (Zhenwen Dai).

21 Analysis of Deep GPs Duvenaud et al. (2014) Duvenaud et al show that the derivative distribution of the process becomes more heavy tailed as number of layers increase.

22 Difficulty for Probabilistic Approaches Propagate a probability distribution through a non-linear mapping. Normalisation of distribution becomes intractable. z2 y j = f j (z) z 1 Figure : A three dimensional manifold formed by mapping from a two dimensional space to a three dimensional space.

23 Difficulty for Probabilistic Approaches z y 1 = f 1 (z) y 2 = f 2 (z) y2 y 1 Figure : A string in two dimensions, formed by mapping from one dimension, z, line to a two dimensional space, [y 1, y 2 ] using nonlinear functions f 1 ( ) and f 2 ( ).

24 Difficulty for Probabilistic Approaches y = f (z) + ɛ p(z) p(y) Figure : A Gaussian distribution propagated through a non-linear mapping. y i = f (z i ) + ɛ i. ɛ N ( 0, 0.2 2) and f ( ) uses RBF basis, 100 centres between -4 and 4 and l = 0.1. New distribution over y (right) is multimodal and difficult to normalize.

25 Variational Compression (Sne; Quiñonero Candela and Rasmussen, 2005; Lawrence, 2007; Titsias, 2009) Complexity of standard GP: O(n 3 ) in computation. O(n 2 ) in storage.

26 Variational Compression (Sne; Quiñonero Candela and Rasmussen, 2005; Lawrence, 2007; Titsias, 2009) Complexity of standard GP: O(n 3 ) in computation. O(n 2 ) in storage. Via low rank representations of covariance: O(nm 2 ) in computation. O(nm) in storage. Where m is user chosen number of inducing variables. They give the rank of the resulting covariance.

27 Variational Compression (Sne; Quiñonero Candela and Rasmussen, 2005; Lawrence, 2007; Titsias, 2009) Complexity of standard GP: O(n 3 ) in computation. O(n 2 ) in storage. Via low rank representations of covariance: O(nm 2 ) in computation. O(nm) in storage. Where m is user chosen number of inducing variables. They give the rank of the resulting covariance.

28 Variational Compression Inducing variables are a compression of the real observations. They are like pseudo-data. They can be in space of f or a space that is related through a linear operator (Álvarez et al., 2010) e.g. a gradient or convolution. There are inducing variables associated with each set of hidden variables, z i.

29 Variational Compression II Importantly conditioning on inducing variables renders the likelihood independent across the data. It turns out that this allows us to variationally handle uncertainty on the kernel (including the inputs to the kernel). It also allows standard scaling approaches: stochastic variational inference Hensman et al. (2013), parallelization Gal et al. (2014) and work by Zhenwen Dai on GPUs to be applied: an engineering challenge?

30 Outline Introduction Deep Gaussian Process Models Variational Methods Composition of GPs Results

31 What is Variational Inference? Convert an integral into an optimization. Entered machine learning via statistical physics in 1990s. But there s a classic example from statistics: expectation maximization.

32 Variational Bound Latent variable model: marginal likelihood computed by integrating latent variables. p(y θ) = p(y z, θ)p(z)dz

33 Variational Bound Log marginal likelihood computed by integrating latent variables. log p(y θ) = log p(y z, θ)p(z)dz

34 Variational Bound Jensen s inequality allows us to obtain a lower bound. log p(y)= log p(y z, θ)p(z)dz

35 Variational Bound Jensen s inequality allows us to obtain a lower bound. log p(y) logp(y z, θ)p(z)dz But the bound can be very loose.

36 Variational Bound Modify Jensens by introducing variational distribution, q(z). log p(y)= log q(z) p(y z, θ)p(z) dz q(z)

37 Variational Bound Modify Jensens by introducing variational distribution, q(z). p(y z, θ)p(z) log p(y) q(z) log dz q(z) Bound is tightened through changing q(z).

38 Variational Bound This is the bound behind EM, in E-step set q(z) = p(z y). p(y z, θ)p(z) log p(y) q(z) log dz q(z) Replace variational distribution with...

39 Variational Bound This is the bound behind EM, in E-step set q(z) = p(z y). p(y z, θ)p(z) log p(y θ) = p(z y, θ) log dz p(z y, θ)... true posterior which allows for...

40 Variational Bound This is the bound behind EM, in E-step set q(z) = p(z y). log p(y θ) = p(z y, θ) log p(y θ)dz... a reorganisation via product rule...

41 Variational Bound This is the bound behind EM, in E-step set q(z) = p(z y). log p(y θ) = log p(y θ)... to recover equality (bound is tight).

42 Variational Bound This is the bound behind EM, in M-step ignore fact that q(z) = p(z y, θ) should depend on parameters and maximize bound. p(y z, θ)p(z) log p(y) q(z) log dz q(z)

43 Variational Bound This is the bound behind EM, in M-step ignore fact that q(z) = p(z y, θ) should depend on parameters and maximize bound. log p(y) q(z) log p(y z, θ)dz + Split into expected log likelihood... q(z) log p(z) q(z) dz

44 Variational Bound This is the bound behind EM, in M-step ignore fact that q(z) = p(z y, θ) should depend on parameters and maximize bound. log p(y) log p(y z, θ) q(z) KL ( q(z) p(z) )... and Kullback Leibler divergence term.

45 Variational Bound This gives the variational lower bound... L = log p(y z, θ) q(z) KL ( q(z) p(z) )... which is an information theoretic interpretation of marginalization.

46 How is this a Variational Method? To apply EM we need to compute p(z y, θ) Often this is intractable, in this case we note that: log p(y) = q(z) log p(y z)p(z) dz + q(z) log q(z) q(z) p(z y) dz (dropping conditioning on θ) the difference between the bound and the log likelihood is the Kullback Leibler divergence between the true posterior and the variational distribution q(z).

47 How is this a Variational Method? To apply EM we need to compute p(z y, θ) Often this is intractable, in this case we note that: log p(y) = L + KL ( q(z) p(z y) ) (dropping conditioning on θ) the difference between the bound and the log likelihood is the Kullback Leibler divergence between the true posterior and the variational distribution q(z).

48 Variational Compression Model for our data, y. p(y) y

49 Variational Compression Prior density over f. Likelihood relates data, y, to f. p(y) = p(y f)p(f)df f y

50 Variational Compression Augment standard model with a set of m new inducing variables, u. p(y) = p(y f)p(u f)p(f)dfdu u f y

51 Variational Compression p(y) = p(y f)p(u f)p(f)dfdu u f y

52 Variational Compression p(y) = p(y f)p(f u)dfp(u)du u f y

53 Variational Compression u p(y) = p(y f)p(f u)dfp(u)du f y

54 Variational Compression u p(y u) = p(y f)p(f u)df f y

55 Variational Compression u p(y u) = n p(y i f i )p(f u)df i=1 f i y i i = 1... n

56 Variational Compression Consider the conditional likelihood. n p(y u) = p(y i f i )p(f u)df i=1

57 Variational Compression Consider the conditional log likelihood. log p(y u) = log n p(y i f i )p(f u)df i=1

58 Variational Compression Introduce variational lower bound n i=1 log p(y u) q(f) log p(y i f i )p(f u) df q(f)

59 Variational Compression Set q(f) = p(f u) log p(y u) n p(f u) log p(y i f i )df i=1

60 Variational Compression Set q(f) = p(f u) log p(y u) n log p(yi f i ) p( f i u) i=1

61 Variational Compression Difference between bound and truth is KL divergence: KL ( p(f u) p(f u, y) ) = p(f u) log p(f u) p(f u, y) df This is why we call it variational compression, information in y is compressed into u

62 Gaussian p(y i f i ) For Gaussian likelihoods: log p(yi f i ) p( f i u) = 1 2 log 2πσ2 1 ( yi 2σ 2 ) f 2 i 1 ( f 2 2σ 2 i 2 ) fi

63 Gaussian p(y i f i ) For Gaussian likelihoods: log p(yi f i ) p( f i u) = 1 2 log 2πσ2 1 ( yi 2σ 2 ) f 2 i 1 ( f 2 2σ 2 i 2 ) fi Implying: p(y i u) exp log c i N ( yi f i, σ 2 )

64 Gaussian Process Over f and u Define: q i,i = var p( fi u) ( ) fi = f 2 i f 2 i p( f i u) p( f i u) We can write: ( c i = exp q ) i,i 2σ 2 If joint distribution of p(f, u) is Gaussian then: q i,i = k i,i k i,u K 1 u,uk i,u c i is not a function of u but is a function of X u.

65 Lower Bound on Likelihood Substitute variational bound into marginal likelihood: Note that: p(y) n c i i=1 is linearly dependent on u. N ( y f, σ 2 I ) p(u)du f p(f u) = K f,u K 1 u,uu

66 Deterministic Training Conditional Making the marginalization of u straightforward. In the Gaussian case: p(u) = N ( ) u 0, K u,u n p(y u)p(u)du c i N ( y K f,u K 1 u,uu, σ 2) N ( ) u 0, K u,u du i=1

67 Deterministic Training Conditional Making the marginalization of u straightforward. In the Gaussian case: p(u) = N ( u 0, K u,u ) p(y u)p(u)du n i=1 c i N ( y 0, σ 2 I + K f,u K 1 u,uk u,f )

68 Deterministic Training Conditional Making the marginalization of u straightforward. In the Gaussian case: p(u) = N ( u 0, K u,u ) p(y u)p(u)du n i=1 c i N ( y 0, σ 2 I + K f,u K 1 u,uk u,f ) Maximize log of the bound to find covariance function parameters, L n i=1 log c i + log N ( y 0, σ 2 I + K f,u K 1 u,uk u,f, )

69 Deterministic Training Conditional Making the marginalization of u straightforward. In the Gaussian case: p(u) = N ( u 0, K u,u ) p(y u)p(u)du n i=1 c i N ( y 0, σ 2 I + K f,u K 1 u,uk u,f ) Maximize log of the bound to find covariance function parameters, L n i=1 log c i + log N ( y 0, σ 2 I + K f,u K 1 u,uk u,f, )

70 Deterministic Training Conditional Making the marginalization of u straightforward. In the Gaussian case: p(u) = N ( u 0, K u,u ) p(y u)p(u)du n i=1 c i N ( y 0, σ 2 I + K f,u K 1 u,uk u,f ) Maximize log of the bound to find covariance function parameters, L log N ( ) y 0, σ 2 I + K f,u K 1 u,uk u,f, If the bound is normalized, the c i terms are removed.

71 Deterministic Training Conditional Making the marginalization of u straightforward. In the Gaussian case: p(u) = N ( u 0, K u,u ) p(y u)p(u)du n i=1 c i N ( y 0, σ 2 I + K f,u K 1 u,uk u,f ) Maximize log of the bound to find covariance function parameters, If the bound is normalized, the c i terms are removed. This results in the projected process approximation (Rasmussen and Williams, 2006) or DTC (Quiñonero Candela and Rasmussen, 2005). Proposed by (Smola and Bartlett, 2001; Seeger et al., 2003; Csató and Opper, 2002; Csató, 2002).

72 Relationship to Nyström Approximation Variational lower bound leads to Nyström style approximation (Williams and Seeger, 2001; Seeger et al., 2003). Relations to subset of regressors (Poggio and Girosi, 1990; Williams et al., 2002). K σ 2 I + K fu K 1 uuk uf Has probabilistic interpretation of cf u N (0, K uu ) y u N ( K fu K 1 uuu, σ 2 I ) w N (0, αi) y w N ( Φw, σ 2 I ) y N ( 0, αφφ + σ 2 I )

73 Marginalising Latent Variables Integrating out Z becomes possible variationally, because Gaussian expectations of log N ( f K fu K 1 uuu, σ 2 I ) are now tractable Relies on computing expectations of K fu and K uf K fu under Gaussian density over Z.

74 Apply Variational Inference Before Integration of u p(y u)p(u)du n c i i=1 N ( y K f,u K 1 u,uu, σ 2) N ( u 0, K u,u ) du

75 Apply Variational Inference Before Integration of u p(y u)p(u)p(z)dudz n i=1 q(z) log c in ( y K f,u K 1 u,uu, σ 2) N ( ) u 0, K u,u du q(z)

76 Outline Introduction Deep Gaussian Process Models Variational Methods Composition of GPs Results

77 Structures for Extracting Information from Data z 4 Latent layer 4 z 3 Latent layer 3 z 2 Latent layer 2 z 1 Latent layer 1 y Data space

78 Damianou and Lawrence (2013) Deep Gaussian Processes Andreas C. Damianou Neil D. Lawrence Dept. of Computer Science & Sheffield Institute for Translational Neuroscience, University of Sheffield, UK Abstract In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use the question as to whether deep structures and the learning of abstract structure can be undertaken in smaller data sets. For smaller data sets, questions of generalization arise: to demonstrate such structures are justified it is useful to have an objective measure of the model s applicability. The traditional approach to deep learning is based around binary latent variables and the restricted Boltzmann machine (RBM) [Hinton, 2010]. Deep hierarchies are constructed by stacking these models and various approximate inference techniques (such as contrastive divergence) are used for estimating model parameters. A significant amount of work has then to be done with annealed importance sampling if even the likelihood1 of a data set under

79 Deep Models z 4 1 z 4 2 z 4 3 z 4 4 Latent layer 4 z 3 1 z 3 2 z 3 3 z 3 4 Latent layer 3 z 2 1 z 2 2 z 2 3 z 2 4 z 2 5 z 2 6 Latent layer 2 z 1 1 z 1 2 z 1 3 z 1 4 z 1 5 z 1 6 Latent layer 1 y 1 y 2 y 3 y 4 y 5 y 6 y 7 y 8 Data space

80 Deep Models z 4 Latent layer 4 z 3 Latent layer 3 z 2 Latent layer 2 z 1 Latent layer 1 y Data space

81 Deep Models z 4 Abstract features z 3 z 2 z 1 More combination Combination of low level features Low level features y Data space

82 Deep Gaussian Processes Damianou and Lawrence (2013) I Deep architectures allow abstraction of features (Bengio, 2009; Hinton and Osindero, 2006; Salakhutdinov and Murray, 2008). I We use variational approach to stack GP models.

83 Outline Introduction Deep Gaussian Process Models Variational Methods Composition of GPs Results

84 Motion Capture High five data. Model learns structure between two interacting subjects.

85 Deep hierarchies motion capture Deep Gaussian processes 38

86 Digits Data Set Are deep hierarchies justified for small data sets? We can lower bound the evidence for different depths. For 150 6s, 0s and 1s from MNIST we found at least 5 layers are required.

87 Deep hierarchies MNIST Deep Gaussian processes 37

88 Summary Deep Gaussian Processes allow unsupervised and supervised deep learning. They can be easily adapted to handle multitask learning. Data dimensionality turns out to not be a computational bottleneck. Variational compression algorithms show promise for scaling these models to massive data sets.

89 References I M. A. Álvarez, D. Luengo, M. K. Titsias, and N. D. Lawrence. Efficient multioutput Gaussian processes through variational inducing kernels. In Y. W. Teh and D. M. Titterington, editors, Proceedings of the Thirteenth International Workshop on Artificial Intelligence and Statistics, volume 9, pages 25 32, Chia Laguna Resort, Sardinia, Italy, May JMLR W&CP 9. [PDF]. Y. Bengio. Learning Deep Architectures for AI. Found. Trends Mach. Learn., 2 (1):1 127, Jan ISSN [DOI]. L. Csató. Gaussian Processes Iterative Sparse Approximations. PhD thesis, Aston University, L. Csató and M. Opper. Sparse on-line Gaussian processes. Neural Computation, 14(3): , A. Damianou and N. D. Lawrence. Deep Gaussian processes. In C. Carvalho and P. Ravikumar, editors, Proceedings of the Sixteenth International Workshop on Artificial Intelligence and Statistics, volume 31, AZ, USA, JMLR W&CP 31. [PDF].

90 References II D. Duvenaud, O. Rippel, R. Adams, and Z. Ghahramani. Avoiding pathologies in very deep networks. In S. Kaski and J. Corander, editors, Proceedings of the Seventeenth International Workshop on Artificial Intelligence and Statistics, volume 33, Iceland, JMLR W&CP 33. Y. Gal, M. van der Wilk, and C. E. Rasmussen. Distributed variational inference in sparse Gaussian process regression and latent variable models. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, Cambridge, MA, J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In A. Nicholson and P. Smyth, editors, Uncertainty in Artificial Intelligence, volume 29. AUAI Press, [PDF]. G. E. Hinton and S. Osindero. A fast learning algorithm for deep belief nets. Neural Computation, 18:2006, N. D. Lawrence. Learning for larger datasets with the Gaussian process latent variable model. In M. Meila and X. Shen, editors, Proceedings of the Eleventh International Workshop on Artificial Intelligence and Statistics, pages , San Juan, Puerto Rico, March Omnipress. [PDF].

91 References III T. K. Leen, T. G. Dietterich, and V. Tresp, editors. Advances in Neural Information Processing Systems, volume 13, Cambridge, MA, MIT Press. T. Poggio and F. Girosi. Networks for approximation and learning. Proceedings of the IEEE, 78(9): , J. Quiñonero Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6: , C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, [Google Books]. R. Salakhutdinov and I. Murray. On the quantitative analysis of deep belief networks. In S. Roweis and A. McCallum, editors, Proceedings of the International Conference in Machine Learning, volume 25, pages Omnipress, M. Seeger, C. K. I. Williams, and N. D. Lawrence. Fast forward selection to speed up sparse Gaussian process regression. In C. M. Bishop and B. J. Frey, editors, Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, Key West, FL, 3 6 Jan A. J. Smola and P. L. Bartlett. Sparse greedy Gaussian process regression. In Leen et al. (2001), pages

92 References IV M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In D. van Dyk and M. Welling, editors, Proceedings of the Twelfth International Workshop on Artificial Intelligence and Statistics, volume 5, pages , Clearwater Beach, FL, April JMLR W&CP 5. C. K. I. Williams, C. E. Rasmussen, A. Schwaighofer, and V. Tresp. Observations of the Nyström method for Gaussian process prediction. Technical report, University of Edinburgh, C. K. I. Williams and M. Seeger. Using the Nyström method to speed up kernel machines. In Leen et al. (2001), pages