ASummaryofGaussianProcesses Coryn A.L. Bailer-Jones

Transcription

1 ASummaryofGaussianProcesses Coryn A.L. Baier-Jones Cavendish Laboratory University of Cambridge Introduction A genera prediction probem can be posed as foows. We consider that the variabe of interest, t, is reated to the set of measurabe variabes,, viasomefunctionf, suchthat t = F(). (1) Typicay, the function F wi be unknown. Cast probabiisticay, the probem becomes one of evauating P (t +1 D, +1 )given +1 and a set of training data, D = {{ }, t }. Ishausethefoowingnotation: t +1 is the singe data output corresponding to the L inputs denoted by the vector +1 (i.e. the dimensionaity of the input space is L). t +1 is the vector of the +1vaues of t, forwhichthecorrespondingsetofinputvectorsisthesetofvectors{ +1 }, which can be considered as an ( +1) L matri. The Gaussian Process Mode AgraphicasummaryofhowtheGaussianProcessmodeperforms predictions is given in Figure 1. The Gaussian Process mode is an attempt to sove this probem byassumingthatthesetofvariabes t has a joint Gaussian distribution, P (t { }, C, µ) = 1 ( Z ep 1 ) 2 (t µ) T C 1 (t µ). (2) ote that the distribution is competey determined by µ and C. The covariance matri, C,has eements C ij = C( i, j ). I wi consider µ =0. Theuseofµ 0wibediscussedater. Thus P (t { }, C )= 1 ( ep 1 ) Z 2 tt C 1 t. (3) Let t +1 be the vaue we wish to predict given its corresponding input vector +1. The joint distribution of t +1 is P (t +1 { }, +1, C +1 )= 1 ( ep 1 ) Z +1 2 tt +1C 1 +1 t +1. (4) The predictive probabiity distribution for t +1 is, therefore, P (t +1 t, { }, +1, C +1 ) = P (t +1 { }, +1, C +1 ) P (t { }, C ) = Z Z +1 ep 1 [ 1 2 (tt +1 C 1 +1 t +1 t T C 1 t ) ] (5). (6)

2 t 1. training data t 3. mode prescribes that the probabiity distribution of a new data point is a Gaussian P(t) t 2. make a prediction at this point 5. standard deviation of P(t) gives error bars on prediction 4. mean of P(t) is predicted vaue of t the interpoated function (interpoant) Figure 1: Summary of how predictions are made with a Gaussian Process mode. After some matri manipuation it can be shown that where P (t +1 t, { }, +1, C +1 )= 1 Z ep (t +1 ˆt +1 ) 2 2σ 2ˆt, (7) +1 ˆt +1 = k T C 1 t, σ 2ˆt +1 = k k T C 1 k, (8) (9) and k =[C( 1, +1 ),C( 2, +1 ),...,C(, +1 )], (10) k = C( +1, +1 ) (11) As ˆt +1 is the maimum vaue of the probabiity distribution of interest 1 it is the predicted vaue for t +1. Thus ˆt() istheinterpoantofthedata,i.e.thepredictedfunction. ote that it does not depend on C +1. Therefore we ony have to invert a singe covariance matri once, namey C, in order to make any number of predictions, t +1. A diagramatic representation of the reationship between these vectors and matrices is given in Figure 2. 1 ote that even if P (t )andp (t +1) arezero-mean,theconditionadistributionp (t +1) isnotnecessariyzeromean. 2

3 +1 T k C+1 C k +1 k Figure 2: The reationship between the matrices C and C +1. Covariance Function The eements of the covariance matri, C,aredenotedC ij. By definition, the covariance between t i and t j is defined as C(t i,t j )=E[(t i E[t i ])(t j E[t j ])], where E denotes epectation. But to evauate epectation vaues we need to know the probabiity distribution over t, anditiseactythat which we are trying to find. Therefore we must parameterize the covariancefunction,c ij,andinfer these parameters from the training data. C ij is a function of the training input data, { },because these data determine the correation between the training data outputs, t. Thus instead epicity parameterizing the function, F, andsovingforitsparametersbysomeformofregression,the Gaussian Process approach defines a parameterized probabiistic mode for the correation between different vaues of the function. These parameters are then found using standardoptimisationtechniques. Asuitabeformofthecovariancefunctionis C ij = θ 1 ep 1 =L ( () i () j )2 + θ 2 + θ 3 δ ij + L ij, (12) 2 =1 r 2 where () i discussed. is the th dimension of the i th input vector, i. The four terms in this equation are now 1. The eponentia term specifies that we wish to fit a smooth interpoant to the training data. The form of this term epresses our beief that inputs which are cose to each other give rise to outputs which are cose to each other: It achieves this by yieding a reativey arge contribution to C ij when i and j are simiar. Each input dimension is given a separate ength scae,r, which dictates how rapidy the interpoant varies as a function of input. If r is reativey arge, the eponentia term wi be sma, the contribution to C ij wi be arge, and hence the function wi not vary much as is varied. r is a measure of the ength over which varies significanty. We can therefore aso consider r 1 as a measure of the reevance of the th input in determining the output. If the function hardy varies at a with one of the inputs (r arge) we woud say that this input has itte reevance in determining the output: we coud probaby eave out this input and the mode coud compensate by an adjustment in the vaue of θ 2.ote 3

4 t Figure 3: The constant term in C ij (θ 2 )contributesaconstanttotheinterpoatingfunction. that significance is stricty defined as t/,andisafunctionof. Theparameterθ 1 gives the overa scae of variations in the t space. 2. The constant term, θ 2,providesfordata,t,withanon-zeromean. Consideratwo-dimensiona case. Let ( ) ( ) θ2 ρθ C = 2 1 ρ = θ ρθ 2 θ 2. 2 ρ 1 Therefore, C 1 = 1 ( 1 1 ρ θ 2 1 ρ 2 ρ 1 ). As ρ 1, C 1,i.e.thereisperfectcorreationbetweent i and t j.so,ifθ 2 is the ony term in C ij,thenc ij = const, whichmeanst i = t j or more specificay t +1 = const. Inotherwords, the interpoant woud be a hyperpane of gradient zero (horizonta ine in the one-dimensiona case, see Figure 3). In genera, then, the θ 2 term adds a constant offset to the interpoant, which is a simiar roe to the bias node in neura networks. Another way of thinking about this is to consider the prediction equation, ˆt +1 = k T C 1 t. (13) Both C 1 and t depend ony upon the training data, so are constant for any predictions and when we make predictions far from the data, k T = θ 2 (1, 1,...,1), and hence ˆt +1 = θ 2 const. 3. This is the noise mode for the outputs and therefore ony occurs in C ij when i = j. Inthiscase the noise is assumed to be input independent Gaussian noise with variance θ Without the L ij term in equation 12, C ij θ 2 for i j,whichwoudresutint i and t j assuming the same vaue. The L ij aows us to mode inear trends in the data. The equation of apanein rea (ratherthancovariance)spacecanbewritten t i = L =1 a () i. 4

5 The covariance of any two such functions, t i and t j is Cov[t i,t j ] = E[(t i E(t i ))(t j E(t j ))] (14) = E[t i t j ] t i t j (15) = E a 2 () i () j + ( )( ) a m a n (m) i (n) j a () i a () j (16) m n where E is the epectation operator. In order to evauate these epectations we need to know what the prior probabiity distributions over t i and t j are, which corresponds to needing to know the priors for the parameters a.ifthea are independent then we have Cov[t i,t j ]=E [ ] ( )( ) a 2 () i () j a () i a () j If we put Gaussian priors on the a with zero mean and variances σ,thenweget. Cov[t i,t j ]= σ 2 () i () j. (17) Aternativey, we may want to put a deta function prior on the a,i.e.p (a )=δ(a α ), in which case we get Cov[t i,t j ]= [ ] ( )( ) α 2 () i () j α () i α () j. (18) Thus the epression in either equation 17 or equation 18 can be usedasthel ij term in equation 12. When optimizing the mode we can then put any priors on theα (or aternativey on the σ )thatwewant. We can write the inear term in equation 17 as z.b where z () = () i () j and a =(σ1 2,σ2 2,...,σ2 L ). C = z.b is just the equation of a hyperpane with norma b. (This hyperpane is in C = C(z (1),z (2),...,z (L) )space.) Incusionofthistermmeansthatwecanmodeineartrendsin the function t = F(). If a were negative, then i j impies that C ij woud be arge and negative, meaning that t i and t j woud be very different. If we did not have this term then t const for vaues of t which deviate greaty from the range in the training data, as was discussed in point number 1 in this ist. There are other forms of the covariance function which coud be used, such as a more compe (input dependent) noise mode. The ony restriction is that the covariance matri be positive definite. From comparison with neura network methods, the parameters of a Gaussian Process are often referred to as hyperparameters rather than parameters, and this nomencature wi now be adopted. The reason for this distinction is that the hyperparameters of a Gaussian Process do not parameterize the function in the way that the neura network weights do. Ihaveconsideredµ =0. Itooksasthoughequation2woudthengivethemostprobabe vaue for t as t = 0. However, this assumes that C is not singuar. If we had just the θ 2 (constant) term in C ij,thenc woud be singuar and we find that t = a(1, 1,...,1), where a is a constant. Thus a zero-mean Gaussian Process is competey genera provided we have a consant term in the covariance function. If we use a Gaussian Process with a non-zero mean, µ = θ 0 (1, 1,...,1), then θ 0 is 5

6 another hyperparameter which must be inferred from the data. We woud epect its vaue to be near to the mean of the training data. When I have used a non-zero mean hyperparameter, the particuar impementation of a Gaussian Process I used set θ 0 to some sensibe vaue and set θ 2 to zero. In theory there is no reason to use a mode with both θ 0 and θ 2,athoughtheremaybenumericareasons. Hyperparameter Determination The various hyperparameters of the covariance function are of course not known in advance, and they must be determined using the training data. From the Bayesian point of view we woud ike to integrate over a possibe hyperparameters. These integrations can be achieved in principe using Monte Caro methods. Aternativey we can take the maimum ikeihood approach. Let Θ be the set of a hyperparameters, Θ = {θ 2,θ 1,θ 3,a 1,...,a L,r 1,...,r L }.Theikeihoodofthe parameters, L(Θ), is L(Θ) P (t { }, C ) (19) P (t { }, Θ,C f ), (20) where C f specifies the form of the covariance function. The maimum ikeihood approach is to maimize L(Θ) to yied the optimum hyperparameters. An improvement over this is to incorporate a prior, P (Θ), on the hyperparameters. By Bayes theorem, the posterior probabiity of the hyperparameters given the training data is P (Θ t, { },C f )= P (t { }, Θ,C f )P (Θ { },C f ) P (t { },C f ). (21) Maimisation of P (Θ t, { },C f )isknownasthemaimum a posteriori (MAP) approach, which is abayesianversionofmaimumikeihoodestimation. Thiswi be a good approimation to the fu Bayesian approach (i.e. integrating over a hyperparameters) if the probabiity mass of the probabiity distribution for the hyperparameters is strongy concentrated around the maimum ikeihood soution: P (t { },C f ) = P (t { }, Θ,C f )P (Θ { },C f )dθ (22) P (t { }, Θ MAP,C f ) (23) where Θ MAP is the most probabe vaue of the hyperparameters evauated by the MAP method (i.e. it is the vaue which maimises P (t { }, Θ,C f )). is a voume term which takes into account the finite width of the P (t { }, Θ,C f )distribution. In maimizing equation 21, we can consider the denominator as aconstantbecauseitisindependent of Θ. The term P (Θ { },C f )incorporatesourpriorknowedgeofthehyperparameters. But the prior, P (Θ), is independent of either { } or C f,so P (Θ { },C f )=P (Θ). (24) (ote that the prior appears in equation 23.) In practice it is easier to maimize the ogarithm of (P (Θ t, { },C f ): n P (Θ t, { },C f ) = np (t { }, Θ,C f )+np (Θ { },C f ) (25) n P (t { },C f ). (26) 6

7 Substituting equation 24 into this and coecting a terms independent of Θ into the term c, the optimum hyperparameters are given the maimum of n P (Θ t, { },C f ) = np (t { }, Θ,C f )+np(θ) + c (27) [ ( 1 = n ep 1 )] Z 2 tt C 1 t +np (Θ) + c (28) = 1 2 tt C 1 t 2 n(2π) 1 2 n C +np (Θ) + c (29) where Z =(2π) /2 C. For conciseness, et L =np (Θ t, { },C f ). The derivative of this with respect to one of the hyperparameters, θ, is ( L θ = 1 2 Tr C 1 C θ ) + 1 C 2 tt C 1 θ C 1 t n P (θ) + θ where the prior on θ, P (θ) isassumedtobeindependentoftheotherpriors. Themaimumofthis function can be found by standard optimisation procedures (such as gradient descent or conjugate gradients). ote that C 1 must be evauated at each step of the optimisation agorithm. Directmethods for this incude LU decomposition (so-caed direct and indirect ) and Gauss-Jordan eimination, both of which are O( 3 )methods.if is arge we can use Skiing s approimate inversion methods which are O( 2 ). (30) 7