Global Gaussian approximations in latent Gaussian models
|
|
- Rhoda Fisher
- 5 years ago
- Views:
Transcription
1 Global Gaussan approxmatons n latent Gaussan models Botond Cseke Aprl 9, 2010 Abstract A revew of global approxmaton methods n latent Gaussan models. 1 Latent Gaussan models In ths secton we ntroduce notaton and defne the model under consderaton. Let p y x, θ l be the condtonal probablty of the observatons y = y 1,..., y n gven the latent varables x = x 1,..., x n and the hyper-parameters θ l. We assume that the lkelhood p y x, θ l factorzes over the latent varables as n p y x, θ l = p y x, θ l. =1 he pror p x θ p over the latent varables s taken to be Gaussan wth canoncal parameters hθ p and Qθ p, that s, p x θ p exp x hθ p 1 2 x Qθ p x. Examples for p x θ p nclude Gaussan process models, where Q 1 θ p s the covarance functon evaluated at the nput locatons and Gaussan Markov random felds, where the elements of Qθ p are the nteractons strengths Q j θ p between the latent varables x and x j. he pror p θ l, θ p over the hyper-parameters s typcally taken to be non-nformatve unform for locaton varables and log-unform for scale varables and factorzes w.r.t. the parameters of the lkelhood and the parameters of the pror. In order to smplfy notaton we use a sngle proxy θ = θ l, θ p to denote the hyper-parameters of the model. he jont dstrbuton of the varables n the model we study s n p y, x, θ p y x, θ exp x hθ 1 2 x Qθ x p θ. =1 We take y fxed and we consder the problem of computng accurate approxmatons of the posteror margnal denstes of the latent varables p x y, θ, gven a fxed hyper-parameter value. hen we ntegrate these margnals over the approxmatons of the hyper-parameters posteror p θ y. he exact quanttes are gven by the formulas p x y, θ = 1 p y θ py x, θ dx \ py j x j, θpx θ 1 j p θ y p θ p y θ. 2 he use the word evdence for p y θ = dxp y, x θ. In the followng we omt p y x, θ s and p x θ s dependence on θ whenever t s not relevant and use t x as an alas of p y x, θ and p 0 x as an alas of p x θ. We use p x = Zp 1 t x p 0 x, that s Z q θ p y θ. A Gaussan approxmaton of p wll be denoted as q and Z q wll denote ts normalzaton constant. 1
2 2 Global Gaussan approxmatons 2.1 he Laplace method he Laplace method 1 computes an approxmatng Gaussan that s characterzed by the local propertes of the dstrbuton at ts mode x = argmax x log p x. he mean s m s defned as m = x whle the nverse of the covarance matrx s the Hessan of log p at the mode x. he dea behnd the method s the followng. Let f = log p. Expandng f n second order at an arbtrary value x we get f x = f x + x x x f x x x 2 xxf x x x + R 2 [f] x; x, where R 2 [f] x; x s the resdual term of the expanson at x wth R 2 [f] x; x = 0. By usng a change of varables s = x x, we have log dxe fx = f x 1 2 xf x [ 2 xxf x ] 1 x f x log 2 xxf x + log E s [e R2[f]s+ x; x], where denotes the determnant and the expectaton w.r.t. s s taken over a normal dstrbuton wth canoncal parameters x f x and 2 xxf x. A closer look at 3 and 4 suggests that choosng x = x and usng the approxmaton R 2 [log p] x; x 0 yelds an approxmaton of the log evdence log dxp x log p x 1 2 log 2 xx log p x. 5 Meanwhle, p can be approxmated by the Gaussan q x = N x x, [ 2 xx log p x ] 1. 6 Note that any reasonably good approxmaton of E s [ e R 2[f]s+ x; x ] can mprove the accuracy of the approxmaton n 5. he Laplace method requres the second order dfferentablty of log p at x, thus a necessary condton for the applcablty of ths approxmaton scheme s the second order dfferentablty of log p. A dstrbuton p for whch the method fals to gve any meanngful nformaton about the varances s, for example, when p y x = λ exp λ y x /2. In ths case the Hessan of log p at an arbtrary pont x s ether equal to the nverse varance of the pror or t s undefned. Snce the Laplace approxmaton captures the characterstcs of the modal confguraton, t often gves poor estmates of the normalzaton constant e.g. Kuss and Rasmussen, However, compared to other methods the man advantage of the Laplace method s ts speed the optmzaton log p w.r.t. x requres only a few Newton steps. 2.2 Varatonal approxmaton An approxmaton scheme that goes beyond local characterstcs s the so called varatonal approxmaton. As mentoned above the mnmzaton of D [p q] leads to the true Gaussan approxmaton but the computatons are ntractable. An alternatve approach s to use to mnmze D [q p]. As shown n Opper and Archambeau 2009, ths approach leads to a tractable optmzaton problem. 1 he Laplace method s known n statstcs as the Gaussan approxmaton e.g., Sva,
3 Expressng D [q p] n terms of the moment parameters of m and V of q, one gets D [q p] = [ ] q x dxq x log + log Z p p x = F m, V + log p y θ, where F s the varatonal free energy e.g. Opper and Archambeau, 2009 F m, V 1 2 log V tr QV m Qm m h E q [log t x ] + C, where C s an rrelevant constant. he optmalty condtons for F are E q [log p x] m that s, V 1 2 = Q dag 2 E q [log t x ] m = 0 and V 1 = 2 E q [log p x] m m, 7 and m = Q h 1 + E q [log t x ].8 m As t was ponted out by Opper and Archambeau, 2009, due to the propertes of Gaussan ntegrals see 3.1 of the Appendx these are equvalent to [ ] [ log p x E q = 0 and V 1 2 ] log p x = E q x x x, 9 that s, the statonary condtons for D [q p] w.r.t. m and V resemble that of the Laplace approxmaton. Loosely speakng, the statonary condtons for the varatonal free energy F w.r.t. the average confguraton m are smlar to the statonary condtons of the energy functon w.r.t model parameters x. Intutvely, n contrast to Laplace approxmaton, the the optmalty condtons for the varatonal free energy hold n average. Snce D [q p] = 0 f and only f q s equal to the posteror, the varatonal free energy F s an upper bound on log Z p and one can approxmate log Z p by the mnmum of F e.g. Neal and Hnton, 1998, that s, log Z p mn m,v F m, V. If t depends only on x then a suffcent condton for the convexty of F n m, V s the convexty Im, v = log t x N x m, v dx n m, v. As t s was ponted out by many authors e.g. Kuss and Rasmussen, 2005; Mnka, 2005, the varatonal approxmaton tends to have the same hallmark as the Laplace approxmaton, whch s the underestmaton of the posteror margnal varances. hs can be explaned by the fact that the varatonal approxmaton s a lmt case of expectaton propagaton when usng local α-dvergences wth α Expectaton propagaton Expectaton propagaton EP approxmates the ntegral for the evdence n the followng way. Let us assume that q s an Gaussan approxmaton of p constraned to have the form qx = Zq 1 t j j x j p 0 x. hen the evdence can be approxmated as Z p = dx p 0 x t j x j j = Z q dx qx t j x j t j j x j Z q dx j qx j t jx j t j x j. 10 j 3
4 and we are left wth choosng the approprate t j x j that yeld both a good approxmaton of the ntegral and of px. EP computes the terms t j x j by teratng t new j x j Collapse t j x j t j x j 1 qx t j x j, for all j {1,..., n}, 11 qx where Collapser s the Kullback-Lebler KL projecton of the dstrbuton r nto the famly of Gaussan dstrbutons. In other words, t s the Gaussan dstrbuton that matches the frst two moments of r. Usng the propertes of the KL dvergence, one can check that when the terms t depend only on x then Collapse t j x j t j x j 1 qx /qx = Collapse t j x j t j x j 1 qx /qx, therefore, the teraton n 11 s well defned. At any fxed pont of ths teraton we have a set of t j x j terms such that Collapse t j x j t j x j 1 qx = qx for any j {1,..., n}. By defnng the cavty dstrbuton q \j x t j x j 1 qx and scalng the terms t j, the above statonarty condton can be rewrtten as { dx j 1, xj, x 2 j} t j x j q \j { x j = dx j 1, xj, x 2 j} tj x j q \j x j, for all j {1,..., n}, and so, the approxmaton for Z p has the form Z p dx p 0 x j t j x j. he updates 11 can be vewed as an teratve applcaton of ADF. It turns out that the Gaussan terms t depend on the same subset or lnear transformatons of the parameters as t and the projecton step n Equaton 11 bols down to computng low dmensonal ntegrals see Secton 3 of the Appendx. In practce these ntegrals are typcally one or two dmensonal and are tractable or can be accurately approxmated usng numercal quadrature rules. Expectaton propagaton, as proposed n Mnka 2001, can be vewed as a generalzaton of loopy belef propagaton e.g. Murphy et al., 1999 to probablstc models wth contnuos varables and and also as an teratve applcaton of the ADF procedure e.g. Csató and Opper, As we can see from the Equatons 12, 14, and 15 n Secton 3 the convexty of log t x N x m, V dx w.r.t. m or the concavty of log t j x j Seeger, 2008 s a suffcent condton for t to be normalzable and thus for the exstence of q new. However, ths alone does not guarantee convergence. o our knowledge, the ssue of EP s convergence n case of the models we study n ths paper s stll an open queston. he teraton n 11 can also be derved by usng varatonal free energes and t can be relaxed such that the projectons are taken on t α j x j t j x j 1 qx, wth α 0, 1]. he lmt α 0 corresponds to the varatonal approxmaton of Opper and Archambeau Detals of EP 3.1 Gaussan formulas he frst and second moments of a dstrbuton p x = q x = N x m, V, are gven by 1 Z f x q x, where q s a Gaussan E p [x] = m + V m log Z and V p [x] = V + V 2 mm log ZV. 12 Usng the ntegraton by parts one can show that the moments of p can also be wrtten n the form E p [x] = m + 1 Z V E q [ x f] and V p [x] = V + 1 Z 2 V [ ZE q [ 2 xx f ] E q [ x f] E q [ x f] ] V, 13 provded that f x e x x and fx x x e x vanshes at nfnty and the requred ntegrals exst. 4
5 3.2 Detals of the expectaton propagaton for Gaussan models Assume the dstrbuton has the form p x = p 0 x t U x, where U are lnear transformatons. hs formulaton ncludes both the representatons when t j depend only on a subset of parameters, that s, t x = t x I wth U = I,I and the representaton used n logstc regresson, where U s the th row of the desgn matrx. Computng t new Frst we compute the form of the term approxmatons, and show that t has a low rank representaton. Let q x = N x m, V and let h = V 1 m, Q = V 1 the canoncal parameters of q. We use q \ x = N x m \, V \ to denote the dstrbuton q \ q/ t α. After some calculus one can show that the moment matchng Gaussan q new x = N x m new, V new of q t q \ s gven by 1 m new = m \ + V \ U U V \ U E [z ] U m \ 1 1 V new = V \ + V \ U U V \ U V [z ] U V \ U U V \ U U V \, where z s a random varable dstrbuted accordng to z t z N z U m \, U V \ U. he update for the term approxmaton t s gven by t new q new /q \. he latter dvson yelds V new 1 V \ 1 1 = U V [z ] 1 U V \ U U 14 V new 1 m new V \ 1 1 m \ = U V [z ] 1 E [z ] U V \ U U m \ 15 leadng to t new x = exp U j x h j 1 2 U jx K j U j x, where h and K are gven by the correspondng quanttes n 14 and 15. he approxmatng dstrbuton q s defned by the canoncal parameters Q = Q + j U j K j U j h = b + j U j h j, that s, the sum over the parameters of t h j and the parameters of the pror p 0 x exp x x Qx/2. Computng q \ Now we turn out attenton to the computaton of the dstrbuton q \. he quanttes we are nterested n are U m \ and U V \ Uj. After some calculus, one can show that these are gven by U V \ U = U K αu K U 1 U = U V U I αk U V U 1 U m \ = U K αu K U 1 h αu h = I αk U V U 1 U m α U V U h. herefore, the computatonal bottleneck of EP reduces to the computaton of the quanttes U m and U V U. hese can be computed from the canoncal representaton of q by U Q 1 h and U Q 1 U. 5
6 Computng the margnal lkelhood approxmaton Let us defne log Z m, V 1 2 m V m log det V n log 2π 2 and log Z m, V log t α U x N x m, V dx. Expectaton propagaton approxmates the margnal lkelhood p y θ by Z ep = Z 1 n/α Zα. Usng the above ntroduced notaton ths can be wrtten as log Z EP = 1 log Z j m \, V \ + log Z m \, V \ log Z m, V + log Z m, V 16 α whch n the case when t depends on U x bols down to log Z ep = 1 log Z j U m \, U V \ U + α [ log Z U m \, U V \ U 1 α log Z U m, U V U ] + log Z m, V. One can see that Z ep can be wrtten as the sum of the approxmate leave-one-out errors log Z j U m \, U V \ U note, that these are not the approxmatons of the leave-one-out densty snce t j does depend on t and a term dependng on the approxmatng densty. 3.3 Solvng the akahash equatons he akahash equatons akahash et al., 1973 am to compute certan elements of the nverse of a postve defnte matrx from ts Cholesky factor. he dervaton of the equatons or the algorthm can be found n many papers e.g., Ersman and nney, 1975; Rue et al., In the followng we present the lne of arguments n Rue et al Let Q = LL, z N0, I and L x = z. hen usng the notaton V = Q 1 we fnd that x N0, V. he equatons L x = z can be rewrtten as L x = z L 1 n k=+1 L kx k. Multplyng both sdes wth x j, j n, usng z = L x and takng expectatons we arrve to the akahash equatons V j = δ j L 2 n L 1 k=+1 L kv kj. Snce we only want to compute the dagonal of V or the elements V j for whch L j 0 the algorthm can be wrtten n the followng Matlab frendly form 1: functon V = SolveakahashL 2: for = n : 1 : 1 3: I = {j : L j 0, j > } 4: V I, = V I,I L I, /L, 5: V,I = VI, 6: V, = 1/L 2, V,IL I, /L 7: end he complexty of ths algorthm scales wth non zerosq 2 /n. References Csató, L. and Opper, M Sparse representaton for Gaussan process models. In. K. Leen,. G. Detterch, and V. resp, edtors, Advances n Neural Informaton Processng Systems 13, Cambrdge, MA, USA. MI Press. Ersman, A. M. and nney, W. F On computng certan elements of the nverse of a sparse matrx. Commun. ACM, 183,
7 Kuss, M. and Rasmussen, C. E Assessng approxmate nference for bnary Gaussan process classfcaton. Journal of Machne Learnng Research, 6, Mnka,. P A famly of algorthms for approxmate Bayesan nference. Ph.D. thess, MI. Mnka,. P Dvergence measures and message passng. echncal Report MSR-R , Mcrosoft Research Ltd., Cambrdge, UK. Murphy, K., Wess, Y., and Jordan, M. I Loopy belef propagaton for approxmate nference: An emprcal study. In Proceedngs of the Ffteenth Conference on Uncertanty n Artfcal Intellgence, volume 9, pages , San Francsco, USA. Morgan Kaufman. Neal, R. and Hnton, G A vew of the EM algorthm that justfes ncremental, sparse, and other varants. In M. I. Jordan, edtor, Learnng n Graphcal Models, pages Kluwer Academc Publshers. Opper, M. and Archambeau, C he varatonal Gaussan approxmaton revsted. Neural Comput., 213, Rue, H., Martno, S., and Chopn, N Approxmate Bayesan nference for latent Gaussan models by usng ntegrated nested Laplace approxmatons. Journal Of he Royal Statstcal Socety Seres B, 712, Seeger, M. W Bayesan nference and optmal desgn for the sparse lnear model. Journal of Machne Learnng Research, 9, Sva, D. S Data Analyss: A Bayesan utoral. Clarendon Oxford Unv. Press, Oxford. akahash, K., Fagan, J., and Chn, M.-S Formaton od a sparse mpedance matrx and ts applcaton to short crcut study. In Proceedngs of the 8th PICA Conference. 7
Gaussian process classification: a message-passing viewpoint
Gaussan process classfcaton: a message-passng vewpont Flpe Rodrgues fmpr@de.uc.pt November 014 Abstract The goal of ths short paper s to provde a message-passng vewpont of the Expectaton Propagaton EP
More information1 Motivation and Introduction
Instructor: Dr. Volkan Cevher EXPECTATION PROPAGATION September 30, 2008 Rce Unversty STAT 63 / ELEC 633: Graphcal Models Scrbes: Ahmad Beram Andrew Waters Matthew Nokleby Index terms: Approxmate nference,
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationHidden Markov Models & The Multivariate Gaussian (10/26/04)
CS281A/Stat241A: Statstcal Learnng Theory Hdden Markov Models & The Multvarate Gaussan (10/26/04) Lecturer: Mchael I. Jordan Scrbes: Jonathan W. Hu 1 Hdden Markov Models As a bref revew, hdden Markov models
More informationExpectation propagation
Expectaton propagaton Lloyd Ellott May 17, 2011 Suppose p(x) s a pdf and we have a factorzaton p(x) = 1 Z n f (x). (1) =1 Expectaton propagaton s an nference algorthm desgned to approxmate the factors
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationWhy BP Works STAT 232B
Why BP Works STAT 232B Free Energes Helmholz & Gbbs Free Energes 1 Dstance between Probablstc Models - K-L dvergence b{ KL b{ p{ = b{ ln { } p{ Here, p{ s the eact ont prob. b{ s the appromaton, called
More informationStat260: Bayesian Modeling and Inference Lecture Date: February 22, Reference Priors
Stat60: Bayesan Modelng and Inference Lecture Date: February, 00 Reference Prors Lecturer: Mchael I. Jordan Scrbe: Steven Troxler and Wayne Lee In ths lecture, we assume that θ R; n hgher-dmensons, reference
More informationEcon107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4)
I. Classcal Assumptons Econ7 Appled Econometrcs Topc 3: Classcal Model (Studenmund, Chapter 4) We have defned OLS and studed some algebrac propertes of OLS. In ths topc we wll study statstcal propertes
More informationConjugacy and the Exponential Family
CS281B/Stat241B: Advanced Topcs n Learnng & Decson Makng Conjugacy and the Exponental Famly Lecturer: Mchael I. Jordan Scrbes: Bran Mlch 1 Conjugacy In the prevous lecture, we saw conjugate prors for the
More informationThe Expectation-Maximization Algorithm
The Expectaton-Maxmaton Algorthm Charles Elan elan@cs.ucsd.edu November 16, 2007 Ths chapter explans the EM algorthm at multple levels of generalty. Secton 1 gves the standard hgh-level verson of the algorthm.
More informationRelevance Vector Machines Explained
October 19, 2010 Relevance Vector Machnes Explaned Trstan Fletcher www.cs.ucl.ac.uk/staff/t.fletcher/ Introducton Ths document has been wrtten n an attempt to make Tppng s [1] Relevance Vector Machnes
More informationA quantum-statistical-mechanical extension of Gaussian mixture model
A quantum-statstcal-mechancal extenson of Gaussan mxture model Kazuyuk Tanaka, and Koj Tsuda 2 Graduate School of Informaton Scences, Tohoku Unversty, 6-3-09 Aramak-aza-aoba, Aoba-ku, Senda 980-8579, Japan
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationSTATS 306B: Unsupervised Learning Spring Lecture 10 April 30
STATS 306B: Unsupervsed Learnng Sprng 2014 Lecture 10 Aprl 30 Lecturer: Lester Mackey Scrbe: Joey Arthur, Rakesh Achanta 10.1 Factor Analyss 10.1.1 Recap Recall the factor analyss (FA) model for lnear
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationEM and Structure Learning
EM and Structure Learnng Le Song Machne Learnng II: Advanced Topcs CSE 8803ML, Sprng 2012 Partally observed graphcal models Mxture Models N(μ 1, Σ 1 ) Z X N N(μ 2, Σ 2 ) 2 Gaussan mxture model Consder
More informationFinite Mixture Models and Expectation Maximization. Most slides are from: Dr. Mario Figueiredo, Dr. Anil Jain and Dr. Rong Jin
Fnte Mxture Models and Expectaton Maxmzaton Most sldes are from: Dr. Maro Fgueredo, Dr. Anl Jan and Dr. Rong Jn Recall: The Supervsed Learnng Problem Gven a set of n samples X {(x, y )},,,n Chapter 3 of
More informationDifferentiating Gaussian Processes
Dfferentatng Gaussan Processes Andrew McHutchon Aprl 17, 013 1 Frst Order Dervatve of the Posteror Mean The posteror mean of a GP s gven by, f = x, X KX, X 1 y x, X α 1 Only the x, X term depends on the
More informationWhy Bayesian? 3. Bayes and Normal Models. State of nature: class. Decision rule. Rev. Thomas Bayes ( ) Bayes Theorem (yes, the famous one)
Why Bayesan? 3. Bayes and Normal Models Alex M. Martnez alex@ece.osu.edu Handouts Handoutsfor forece ECE874 874Sp Sp007 If all our research (n PR was to dsappear and you could only save one theory, whch
More informationStatistical pattern recognition
Statstcal pattern recognton Bayes theorem Problem: decdng f a patent has a partcular condton based on a partcular test However, the test s mperfect Someone wth the condton may go undetected (false negatve
More informationOutline. Bayesian Networks: Maximum Likelihood Estimation and Tree Structure Learning. Our Model and Data. Outline
Outlne Bayesan Networks: Maxmum Lkelhood Estmaton and Tree Structure Learnng Huzhen Yu janey.yu@cs.helsnk.f Dept. Computer Scence, Unv. of Helsnk Probablstc Models, Sprng, 200 Notces: I corrected a number
More informationMaximum Likelihood Estimation (MLE)
Maxmum Lkelhood Estmaton (MLE) Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 01 UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models
More informationOn an Extension of Stochastic Approximation EM Algorithm for Incomplete Data Problems. Vahid Tadayon 1
On an Extenson of Stochastc Approxmaton EM Algorthm for Incomplete Data Problems Vahd Tadayon Abstract: The Stochastc Approxmaton EM (SAEM algorthm, a varant stochastc approxmaton of EM, s a versatle tool
More information1 Matrix representations of canonical matrices
1 Matrx representatons of canoncal matrces 2-d rotaton around the orgn: ( ) cos θ sn θ R 0 = sn θ cos θ 3-d rotaton around the x-axs: R x = 1 0 0 0 cos θ sn θ 0 sn θ cos θ 3-d rotaton around the y-axs:
More informationHidden Markov Models
Hdden Markov Models Namrata Vaswan, Iowa State Unversty Aprl 24, 204 Hdden Markov Model Defntons and Examples Defntons:. A hdden Markov model (HMM) refers to a set of hdden states X 0, X,..., X t,...,
More informationChapter 11: Simple Linear Regression and Correlation
Chapter 11: Smple Lnear Regresson and Correlaton 11-1 Emprcal Models 11-2 Smple Lnear Regresson 11-3 Propertes of the Least Squares Estmators 11-4 Hypothess Test n Smple Lnear Regresson 11-4.1 Use of t-tests
More informationThe Geometry of Logit and Probit
The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.
More informationFeb 14: Spatial analysis of data fields
Feb 4: Spatal analyss of data felds Mappng rregularly sampled data onto a regular grd Many analyss technques for geophyscal data requre the data be located at regular ntervals n space and/or tme. hs s
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 16
STAT 39: MATHEMATICAL COMPUTATIONS I FALL 218 LECTURE 16 1 why teratve methods f we have a lnear system Ax = b where A s very, very large but s ether sparse or structured (eg, banded, Toepltz, banded plus
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationLecture 4: Universal Hash Functions/Streaming Cont d
CSE 5: Desgn and Analyss of Algorthms I Sprng 06 Lecture 4: Unversal Hash Functons/Streamng Cont d Lecturer: Shayan Oves Gharan Aprl 6th Scrbe: Jacob Schreber Dsclamer: These notes have not been subjected
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More informationSingular Value Decomposition: Theory and Applications
Sngular Value Decomposton: Theory and Applcatons Danel Khashab Sprng 2015 Last Update: March 2, 2015 1 Introducton A = UDV where columns of U and V are orthonormal and matrx D s dagonal wth postve real
More informationMarkov Chain Monte Carlo (MCMC), Gibbs Sampling, Metropolis Algorithms, and Simulated Annealing Bioinformatics Course Supplement
Markov Chan Monte Carlo MCMC, Gbbs Samplng, Metropols Algorthms, and Smulated Annealng 2001 Bonformatcs Course Supplement SNU Bontellgence Lab http://bsnuackr/ Outlne! Markov Chan Monte Carlo MCMC! Metropols-Hastngs
More information8 : Learning in Fully Observed Markov Networks. 1 Why We Need to Learn Undirected Graphical Models. 2 Structural Learning for Completely Observed MRF
10-708: Probablstc Graphcal Models 10-708, Sprng 2014 8 : Learnng n Fully Observed Markov Networks Lecturer: Erc P. Xng Scrbes: Meng Song, L Zhou 1 Why We Need to Learn Undrected Graphcal Models In the
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationMMA and GCMMA two methods for nonlinear optimization
MMA and GCMMA two methods for nonlnear optmzaton Krster Svanberg Optmzaton and Systems Theory, KTH, Stockholm, Sweden. krlle@math.kth.se Ths note descrbes the algorthms used n the author s 2007 mplementatons
More informationMACHINE APPLIED MACHINE LEARNING LEARNING. Gaussian Mixture Regression
11 MACHINE APPLIED MACHINE LEARNING LEARNING MACHINE LEARNING Gaussan Mture Regresson 22 MACHINE APPLIED MACHINE LEARNING LEARNING Bref summary of last week s lecture 33 MACHINE APPLIED MACHINE LEARNING
More informationEstimating the Fundamental Matrix by Transforming Image Points in Projective Space 1
Estmatng the Fundamental Matrx by Transformng Image Ponts n Projectve Space 1 Zhengyou Zhang and Charles Loop Mcrosoft Research, One Mcrosoft Way, Redmond, WA 98052, USA E-mal: fzhang,cloopg@mcrosoft.com
More informationLecture 3: Probability Distributions
Lecture 3: Probablty Dstrbutons Random Varables Let us begn by defnng a sample space as a set of outcomes from an experment. We denote ths by S. A random varable s a functon whch maps outcomes nto the
More informationP R. Lecture 4. Theory and Applications of Pattern Recognition. Dept. of Electrical and Computer Engineering /
Theory and Applcatons of Pattern Recognton 003, Rob Polkar, Rowan Unversty, Glassboro, NJ Lecture 4 Bayes Classfcaton Rule Dept. of Electrcal and Computer Engneerng 0909.40.0 / 0909.504.04 Theory & Applcatons
More informationLecture 12: Classification
Lecture : Classfcaton g Dscrmnant functons g The optmal Bayes classfer g Quadratc classfers g Eucldean and Mahalanobs metrcs g K Nearest Neghbor Classfers Intellgent Sensor Systems Rcardo Guterrez-Osuna
More informationAPPROXIMATE PRICES OF BASKET AND ASIAN OPTIONS DUPONT OLIVIER. Premia 14
APPROXIMAE PRICES OF BASKE AND ASIAN OPIONS DUPON OLIVIER Prema 14 Contents Introducton 1 1. Framewor 1 1.1. Baset optons 1.. Asan optons. Computng the prce 3. Lower bound 3.1. Closed formula for the prce
More informationInformation Geometry of Gibbs Sampler
Informaton Geometry of Gbbs Sampler Kazuya Takabatake Neuroscence Research Insttute AIST Central 2, Umezono 1-1-1, Tsukuba JAPAN 305-8568 k.takabatake@ast.go.jp Abstract: - Ths paper shows some nformaton
More informationProbabilistic & Unsupervised Learning
Probablstc & Unsupervsed Learnng Convex Algorthms n Approxmate Inference Yee Whye Teh ywteh@gatsby.ucl.ac.uk Gatsby Computatonal Neuroscence Unt Unversty College London Term 1, Autumn 2008 Convexty A convex
More informationUsing T.O.M to Estimate Parameter of distributions that have not Single Exponential Family
IOSR Journal of Mathematcs IOSR-JM) ISSN: 2278-5728. Volume 3, Issue 3 Sep-Oct. 202), PP 44-48 www.osrjournals.org Usng T.O.M to Estmate Parameter of dstrbutons that have not Sngle Exponental Famly Jubran
More informationA Bayes Algorithm for the Multitask Pattern Recognition Problem Direct Approach
A Bayes Algorthm for the Multtask Pattern Recognton Problem Drect Approach Edward Puchala Wroclaw Unversty of Technology, Char of Systems and Computer etworks, Wybrzeze Wyspanskego 7, 50-370 Wroclaw, Poland
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More informationClassification as a Regression Problem
Target varable y C C, C,, ; Classfcaton as a Regresson Problem { }, 3 L C K To treat classfcaton as a regresson problem we should transform the target y nto numercal values; The choce of numercal class
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationC4B Machine Learning Answers II. = σ(z) (1 σ(z)) 1 1 e z. e z = σ(1 σ) (1 + e z )
C4B Machne Learnng Answers II.(a) Show that for the logstc sgmod functon dσ(z) dz = σ(z) ( σ(z)) A. Zsserman, Hlary Term 20 Start from the defnton of σ(z) Note that Then σ(z) = σ = dσ(z) dz = + e z e z
More information3.1 ML and Empirical Distribution
67577 Intro. to Machne Learnng Fall semester, 2008/9 Lecture 3: Maxmum Lkelhood/ Maxmum Entropy Dualty Lecturer: Amnon Shashua Scrbe: Amnon Shashua 1 In the prevous lecture we defned the prncple of Maxmum
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationPHYS 705: Classical Mechanics. Calculus of Variations II
1 PHYS 705: Classcal Mechancs Calculus of Varatons II 2 Calculus of Varatons: Generalzaton (no constrant yet) Suppose now that F depends on several dependent varables : We need to fnd such that has a statonary
More informationLinear Regression Analysis: Terminology and Notation
ECON 35* -- Secton : Basc Concepts of Regresson Analyss (Page ) Lnear Regresson Analyss: Termnology and Notaton Consder the generc verson of the smple (two-varable) lnear regresson model. It s represented
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationIntroduction to Hidden Markov Models
Introducton to Hdden Markov Models Alperen Degrmenc Ths document contans dervatons and algorthms for mplementng Hdden Markov Models. The content presented here s a collecton of my notes and personal nsghts
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More informationComposite Hypotheses testing
Composte ypotheses testng In many hypothess testng problems there are many possble dstrbutons that can occur under each of the hypotheses. The output of the source s a set of parameters (ponts n a parameter
More informationScalable Multi-Class Gaussian Process Classification using Expectation Propagation
Scalable Mult-Class Gaussan Process Classfcaton usng Expectaton Propagaton Carlos Vllacampa-Calvo and Danel Hernández Lobato Computer Scence Department Unversdad Autónoma de Madrd http://dhnzl.org, danel.hernandez@uam.es
More informationThe exam is closed book, closed notes except your one-page cheat sheet.
CS 89 Fall 206 Introducton to Machne Learnng Fnal Do not open the exam before you are nstructed to do so The exam s closed book, closed notes except your one-page cheat sheet Usage of electronc devces
More informationGlobal Sensitivity. Tuesday 20 th February, 2018
Global Senstvty Tuesday 2 th February, 28 ) Local Senstvty Most senstvty analyses [] are based on local estmates of senstvty, typcally by expandng the response n a Taylor seres about some specfc values
More informationA quantum-statistical-mechanical extension of Gaussian mixture model
Journal of Physcs: Conference Seres A quantum-statstcal-mechancal extenson of Gaussan mxture model To cte ths artcle: K Tanaka and K Tsuda 2008 J Phys: Conf Ser 95 012023 Vew the artcle onlne for updates
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationChapter 5. Solution of System of Linear Equations. Module No. 6. Solution of Inconsistent and Ill Conditioned Systems
Numercal Analyss by Dr. Anta Pal Assstant Professor Department of Mathematcs Natonal Insttute of Technology Durgapur Durgapur-713209 emal: anta.bue@gmal.com 1 . Chapter 5 Soluton of System of Lnear Equatons
More informationThe Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s
More informationProbability Theory (revisited)
Probablty Theory (revsted) Summary Probablty v.s. plausblty Random varables Smulaton of Random Experments Challenge The alarm of a shop rang. Soon afterwards, a man was seen runnng n the street, persecuted
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More informationSecond order approximations for probability models
Second order approxmatons for probablty models lbert Kappen Department of Bophyscs Njmegen Unversty Njmegen, the Netherlands bertmbfysunnl Wm Wegernc Department of Bophyscs Njmegen Unversty Njmegen, the
More informationLecture 4: Constant Time SVD Approximation
Spectral Algorthms and Representatons eb. 17, Mar. 3 and 8, 005 Lecture 4: Constant Tme SVD Approxmaton Lecturer: Santosh Vempala Scrbe: Jangzhuo Chen Ths topc conssts of three lectures 0/17, 03/03, 03/08),
More informationLecture 20: Hypothesis testing
Lecture : Hpothess testng Much of statstcs nvolves hpothess testng compare a new nterestng hpothess, H (the Alternatve hpothess to the borng, old, well-known case, H (the Null Hpothess or, decde whether
More informationSparse Gaussian Processes Using Backward Elimination
Sparse Gaussan Processes Usng Backward Elmnaton Lefeng Bo, Lng Wang, and Lcheng Jao Insttute of Intellgent Informaton Processng and Natonal Key Laboratory for Radar Sgnal Processng, Xdan Unversty, X an
More informationCalculating CLs Limits. Abstract
DØNote 4492 Calculatng CLs Lmts Harrson B. Prosper Florda State Unversty, Tallahassee, Florda 32306 (Dated: June 8, 2004) Abstract Ths note suggests how the calculaton of lmts based on the CLs method mght
More informationChapter 2 - The Simple Linear Regression Model S =0. e i is a random error. S β2 β. This is a minimization problem. Solution is a calculus exercise.
Chapter - The Smple Lnear Regresson Model The lnear regresson equaton s: where y + = β + β e for =,..., y and are observable varables e s a random error How can an estmaton rule be constructed for the
More informationWorkshop: Approximating energies and wave functions Quantum aspects of physical chemistry
Workshop: Approxmatng energes and wave functons Quantum aspects of physcal chemstry http://quantum.bu.edu/pltl/6/6.pdf Last updated Thursday, November 7, 25 7:9:5-5: Copyrght 25 Dan Dll (dan@bu.edu) Department
More information8/25/17. Data Modeling. Data Modeling. Data Modeling. Patrice Koehl Department of Biological Sciences National University of Singapore
8/5/17 Data Modelng Patrce Koehl Department of Bologcal Scences atonal Unversty of Sngapore http://www.cs.ucdavs.edu/~koehl/teachng/bl59 koehl@cs.ucdavs.edu Data Modelng Ø Data Modelng: least squares Ø
More informationCIS526: Machine Learning Lecture 3 (Sept 16, 2003) Linear Regression. Preparation help: Xiaoying Huang. x 1 θ 1 output... θ M x M
CIS56: achne Learnng Lecture 3 (Sept 6, 003) Preparaton help: Xaoyng Huang Lnear Regresson Lnear regresson can be represented by a functonal form: f(; θ) = θ 0 0 +θ + + θ = θ = 0 ote: 0 s a dummy attrbute
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationDepartment of Computer Science Artificial Intelligence Research Laboratory. Iowa State University MACHINE LEARNING
MACHINE LEANING Vasant Honavar Bonformatcs and Computatonal Bology rogram Center for Computatonal Intellgence, Learnng, & Dscovery Iowa State Unversty honavar@cs.astate.edu www.cs.astate.edu/~honavar/
More informationErrors for Linear Systems
Errors for Lnear Systems When we solve a lnear system Ax b we often do not know A and b exactly, but have only approxmatons  and ˆb avalable. Then the best thng we can do s to solve ˆx ˆb exactly whch
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationTopic 5: Non-Linear Regression
Topc 5: Non-Lnear Regresson The models we ve worked wth so far have been lnear n the parameters. They ve been of the form: y = Xβ + ε Many models based on economc theory are actually non-lnear n the parameters.
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationClustering gene expression data & the EM algorithm
CG, Fall 2011-12 Clusterng gene expresson data & the EM algorthm CG 08 Ron Shamr 1 How Gene Expresson Data Looks Entres of the Raw Data matrx: Rato values Absolute values Row = gene s expresson pattern
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationChat eld, C. and A.J.Collins, Introduction to multivariate analysis. Chapman & Hall, 1980
MT07: Multvarate Statstcal Methods Mke Tso: emal mke.tso@manchester.ac.uk Webpage for notes: http://www.maths.manchester.ac.uk/~mkt/new_teachng.htm. Introducton to multvarate data. Books Chat eld, C. and
More informationStatistical Foundations of Pattern Recognition
Statstcal Foundatons of Pattern Recognton Learnng Objectves Bayes Theorem Decson-mang Confdence factors Dscrmnants The connecton to neural nets Statstcal Foundatons of Pattern Recognton NDE measurement
More informationThe Multiple Classical Linear Regression Model (CLRM): Specification and Assumptions. 1. Introduction
ECONOMICS 5* -- NOTE (Summary) ECON 5* -- NOTE The Multple Classcal Lnear Regresson Model (CLRM): Specfcaton and Assumptons. Introducton CLRM stands for the Classcal Lnear Regresson Model. The CLRM s also
More informationGaussian Mixture Models
Lab Gaussan Mxture Models Lab Objectve: Understand the formulaton of Gaussan Mxture Models (GMMs) and how to estmate GMM parameters. You ve already seen GMMs as the observaton dstrbuton n certan contnuous
More informationPrimer on High-Order Moment Estimators
Prmer on Hgh-Order Moment Estmators Ton M. Whted July 2007 The Errors-n-Varables Model We wll start wth the classcal EIV for one msmeasured regressor. The general case s n Erckson and Whted Econometrc
More informationEstimation: Part 2. Chapter GREG estimation
Chapter 9 Estmaton: Part 2 9. GREG estmaton In Chapter 8, we have seen that the regresson estmator s an effcent estmator when there s a lnear relatonshp between y and x. In ths chapter, we generalzed the
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far Supervsed machne learnng Lnear models Least squares regresson Fsher s dscrmnant, Perceptron, Logstc model Non-lnear
More informationMATH 829: Introduction to Data Mining and Analysis The EM algorithm (part 2)
1/16 MATH 829: Introducton to Data Mnng and Analyss The EM algorthm (part 2) Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 20, 2016 Recall 2/16 We are gven ndependent observatons
More informationLOW BIAS INTEGRATED PATH ESTIMATORS. James M. Calvin
Proceedngs of the 007 Wnter Smulaton Conference S G Henderson, B Bller, M-H Hseh, J Shortle, J D Tew, and R R Barton, eds LOW BIAS INTEGRATED PATH ESTIMATORS James M Calvn Department of Computer Scence
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationTHE ROYAL STATISTICAL SOCIETY 2006 EXAMINATIONS SOLUTIONS HIGHER CERTIFICATE
THE ROYAL STATISTICAL SOCIETY 6 EXAMINATIONS SOLUTIONS HIGHER CERTIFICATE PAPER I STATISTICAL THEORY The Socety provdes these solutons to assst canddates preparng for the eamnatons n future years and for
More information