Expectation propagation
|
|
- Darlene Freeman
- 6 years ago
- Views:
Transcription
1 Expectaton propagaton Lloyd Ellott May 17, 2011
2 Suppose p(x) s a pdf and we have a factorzaton p(x) = 1 Z n f (x). (1) =1 Expectaton propagaton s an nference algorthm desgned to approxmate the factors f. In dong so, we may recover approxmatons of the margnals and jonts of p, or we may fnd the normalzng constant for p. EP nvolves parametersng an approxmaton f of each factor f and teratvely ncludng each factor nto the approxmaton by mnmsng a KL-dvergence.
3 For each factor f, fx an approxmatng famly of dstrbutons Ω. Gven (1) and Ω, the EP algorthm s as follows: ntalze approxmatons f repeat for = 1,..., n do f argmax ˆf Ω KL 1 B f 1 f C ˆf j end for untl stoppng condton reached Here, B and C are normalsng constants. j f (2)
4 Wrtng p = j f, we see that the update n the EP algorthm sets f to: 1 argmn ˆf Ω B (f p )(x) log Cf (x) dx, such that B ˆf (x) (ˆf p )(x)dx = C. (3) From ths equaton, we see that f ˆf were unconstraned (.e. f Ω were all functons on the range of x), then ˆf = C B f would be a soluton. Unfortunately, the computaton of B and C are often ntractable. Therefore, to make progress n EP, we must place constrants on f so that mnmsng (3) s tractable.
5 There are two man sorts of constrants on f that we wll examne: 1. Exponental famly constrans, 2. Fully factorsed constrants. In what follows we wll see the general mplcaton of these assumptons n detal, makng reference to the formulaton of EP updates as mnmsng (2). Other constrants are possble: any choce of Ω for whch the computaton of (3) s tractable leads to an EP algorthm.
6 Exponental famly constrants Suppose f (x) = h(x) exp(η T u(x) A(η)) and p(x) s any dstrbuton. We want to fnd the suffcent statstc η that mnmses the followng KL-dvergence: KL(p q) = p(x) log p f (x) dx, = E p [p(x)] + E p [h(x)] A(η) + η T E p [u(x)]. We proceed by equatng the dervatve of wth respect to η to zero: η A(η) = E p [u(x)]. (4) But, because f s from an exponental famly, η A(η) = E f [u(x)]. Thus, (9) s mnmsed when E f [u(x)] = E p [u(x)]. Ths s why EP s sometmes called moment matchng.
7 Returnng to the stuaton of EP, suppose we restrct f to be proportonal to a dstrbuton n a a gven exponental famly: Ω = {f (x) : f (x) h (x) exp(η T u(x) A (η)) η}. Wthout loss of generalty, we have assumed the same form of the suffcent statstcs u(x) for each approxmatng dstrbuton. Suppose f exp(η T u(x) A ( η )) are the current ste approxmatons (proportonalty n η ). The EP mnmsaton step for f (2) s: ( ) 1 f argmax KL ˆf Ω B f p 1 C ˆf p. Collectng terms n the exponent, the second argument n the KL-dvergence s exponental famly wth (proportonalty n ˆη ): ˆf p exp (ˆη T + j η T j )u(x) A (ˆη ) j A ( η ). (5)
8 Suppose η j agree gven for all j. We wll use (5) to wrte Eˆf p [u(x)] as a functon of ˆη : Suppose Φ (ˆη ) = Eˆf p [u(x)]. To proceed, we must be able to compute E f p [u(x)] for the fxed η j. In ths case, the update (2) s gven by the followng: ˆη Φ 1 (E f p [u(x)]). (6)
9 Fully factorsed constrants Suppose x = (x 1,..., x k ) and p(x) = 1 B n f (C ), =1 where C 1,..., C n are subsets of x. (N.b. that the C mght overlap.) Ths model has the same expressve power as factor graphs: If G s a factor graph then the terms f (C ) correspond to the factors of G. In partcular, f G s an undrected graphcal model, then we can choose C 1,..., C n so that C s the par of vertces conencted by the -th edge of G.
10 The fully factorsed constrant on f (C ) s: f (C ) = x l C f l (x l ) We wll also assume that f l (x l ) are restrcted to functons proportonal to exponental famles wth base measure, suffcent statstcs, and partton functons h l, η l, A l respectvely. As above: f l (x l ) exp( η l T u l(x l ) A l ( η l )). Note that as f splts, we wrte seperate suffcent statstcs for each component of x. We have constraned Ω to be an exponental famly that splts over the random varables contaned n C.
11 Under these constrants, we fnd factors n the KL-dvergence (3) that depend on ˆf for a fxed : ( ) 1 KL B f p 1 C f p = 1 (f p )(x) log(f /ˆf )(x)dx B = 1 f (C ) f jl (x l ) log(f /ˆf )(x)dx B j x l C j = 1 fjl (x l ) no ˆη dependence B x\c f (C ) C j j x l C j \C x l C j C ( ) 1 =KL B f p C 1 C ˆf p C, f jl (x l ) log(f /ˆf )(x)dx where p C = j,x l :x l C j C f jl (x l ). Expectatons wth respect to the frst argument of ths KL are ntegrals over C whch are tractable.
12 In partcluar, ˆf = ˆ x l C f l (x l ), and so the above KL s optmsed when the followng KL-dvergences are mnmsed for each l: ( ) 1 KL B f p C 1 D ˆf l p C. By the exponental famly dervaton above, (ˆf l p C )(x l ) exp ˆη l T + j :x l C j η T jl u l (x l ) A l (ˆη l ) A jl ( η l ) (7) j :x l C j
13 So the EP update for ˆf l s found as follows: 1. Use equaton (7) above to wrte Eˆfl p C [u l (x l )] as a functon of ˆη l : suppose the functon s Φ l (ˆη l ) = Eˆf l p C [u l (x l )] 2. Compute E fl p C [u l (x l )]. ( ) 3. Set ˆf l Φ 1 l E fl p C [u l (x l )]. These frst two steps nvolve ntegraton over C whch s tractable f the szes of C are small. Every named exponental famly admts an analytc form for Φ 1.
14 Example: Graphcal models on bnary varables Suppose G s an undrected graphcal model on bnary random varables V (G) = {x 1,..., x n }: p(g) 1 Z xy E(G) f xy (x, y). (8) Here, E(G) are the edges of G. We have absorbed the factors nvolvng just one varable nto the factors on the edges. We can wrte f xy as the followng exponental famly wth suffcent statstcs x, y, xy: f xy (xy) = µ (1 x)(1 y) xy;00 µ x(1 y) xy;10 µ(1 x)y xy;01 µxy xy;11 = exp(σ x x + yσ y + σ xy xy + b xy ). (9)
15 In (9), the suffcent statstcs for f xy are: And the partton functon s: σ x = log(µ xy;10 /µ xy;00 ), σ y = log(µ xy;01 /µ xy;00 ), σ xy = log µ xy;11µ xy;00 µ xy;10 ; µ xy;01 b xy = log µ xy;00. We wll apply the fully factorzed constrant to the approxmate ste potentals: f xy (xy) = f xy:x (x)f xy:y, exp(δ xy:x x) exp(δ xy:y y). (10) The suffcent statstcs of ths approxmaton are x and y.
16 We derve the update (6) for ˆf xy assumng that f x y, f x are gven for all x y xy. We must fnd the expected values of the suffcent statstcs of f xy p xy {xy}. As n (7), wth C = {xy}: f xy p xy {xy} (x, y) exp(σ xx + σ y y + σ xy xy + b xy + σ xy ;xx + σ x y,y y). (11) y N(x)\y x N(y)\x
17 We compute the expected value of x under (11). E fxy p xy [x] s: {xy} exp σ x exp(σ y + σ xy + σ x y;y ) y N(x)\y / 1 + exp(σ x + σ xy ;x y N(x)\y + exp(σ x + σ y + σ xy + =ρ x. σ xy ;x) + exp(σ y + x N(y)\x σ x y;y + x N(y)\x y N(x)\y x N(y)\x σ xy ;x) σ x y;y ), (12)
18 The expresson for (12) n the prevous slde can be calculated drectly from (11) by expandng E fxy p xy [x] as: {xy} 0 (f xy p xy {xy} (0, 0) + f xy p xy {xy} (0, 1)) + 1 (f xy p xy {xy} (1, 0) + f xy p xy {xy}(1, 1)) f xy p xy {xy} (0, 0) + f xy p xy {xy} (0, 1) + f xy p xy {xy} (1, 0) + f xy p xy {xy} (1, 1). Next, E f xy [x] = (0 (exp(0 σ xy:x + 0 σ xy:y ) + exp(0 σ xy:x + 1 σ xy:y )) +1 (exp(1 σ xy:x + 0 σ xy:y ) + exp(1 σ xy:x + 1 σ xy:y ))) / (exp(0 σ xy:x + 0 σ xy:y ) + exp(1 σ xy:x + 0 σ xy:y ) + exp(0 σ xy:x + 1 σ xy:y ) + exp(1 σ xy:x + 1 σ xy:y ))) = exp( δ xy;x ) 1 + exp( δ xy;x ). (13)
19 Equatng (12) and (13) yelds the update for δ xy;x : Thus, the update for δ xy;x s: E f xy [x] = E fxy p xy [x], {xy} exp( δ xy;x ) 1 + exp( δ xy;x ) = ρ x, δ xy;x = log δ xy;x log ρ x 1 ρ x, ρ x 1 ρ x. (14) and the update for δ xy;y s by symmetry. Ths completes the EP algorthm for arbtrary undrected graphs of bnary random varables. Note that (??) s found by nvertng the expected value as a functon of the natural parameter. Ths s the Φ 1 functon from (6).
Gaussian process classification: a message-passing viewpoint
Gaussan process classfcaton: a message-passng vewpont Flpe Rodrgues fmpr@de.uc.pt November 014 Abstract The goal of ths short paper s to provde a message-passng vewpont of the Expectaton Propagaton EP
More informationDifferentiating Gaussian Processes
Dfferentatng Gaussan Processes Andrew McHutchon Aprl 17, 013 1 Frst Order Dervatve of the Posteror Mean The posteror mean of a GP s gven by, f = x, X KX, X 1 y x, X α 1 Only the x, X term depends on the
More informationNUMERICAL DIFFERENTIATION
NUMERICAL DIFFERENTIATION 1 Introducton Dfferentaton s a method to compute the rate at whch a dependent output y changes wth respect to the change n the ndependent nput x. Ths rate of change s called the
More informationEM and Structure Learning
EM and Structure Learnng Le Song Machne Learnng II: Advanced Topcs CSE 8803ML, Sprng 2012 Partally observed graphcal models Mxture Models N(μ 1, Σ 1 ) Z X N N(μ 2, Σ 2 ) 2 Gaussan mxture model Consder
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More information8 : Learning in Fully Observed Markov Networks. 1 Why We Need to Learn Undirected Graphical Models. 2 Structural Learning for Completely Observed MRF
10-708: Probablstc Graphcal Models 10-708, Sprng 2014 8 : Learnng n Fully Observed Markov Networks Lecturer: Erc P. Xng Scrbes: Meng Song, L Zhou 1 Why We Need to Learn Undrected Graphcal Models In the
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationLecture 10: Euler s Equations for Multivariable
Lecture 0: Euler s Equatons for Multvarable Problems Let s say we re tryng to mnmze an ntegral of the form: {,,,,,, ; } J f y y y y y y d We can start by wrtng each of the y s as we dd before: y (, ) (
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationHidden Markov Models & The Multivariate Gaussian (10/26/04)
CS281A/Stat241A: Statstcal Learnng Theory Hdden Markov Models & The Multvarate Gaussan (10/26/04) Lecturer: Mchael I. Jordan Scrbes: Jonathan W. Hu 1 Hdden Markov Models As a bref revew, hdden Markov models
More information1 Motivation and Introduction
Instructor: Dr. Volkan Cevher EXPECTATION PROPAGATION September 30, 2008 Rce Unversty STAT 63 / ELEC 633: Graphcal Models Scrbes: Ahmad Beram Andrew Waters Matthew Nokleby Index terms: Approxmate nference,
More informationOnline Appendix to The Allocation of Talent and U.S. Economic Growth
Onlne Appendx to The Allocaton of Talent and U.S. Economc Growth Not for publcaton) Chang-Ta Hseh, Erk Hurst, Charles I. Jones, Peter J. Klenow February 22, 23 A Dervatons and Proofs The propostons n the
More informationMATH 5707 HOMEWORK 4 SOLUTIONS 2. 2 i 2p i E(X i ) + E(Xi 2 ) ä i=1. i=1
MATH 5707 HOMEWORK 4 SOLUTIONS CİHAN BAHRAN 1. Let v 1,..., v n R m, all lengths v are not larger than 1. Let p 1,..., p n [0, 1] be arbtrary and set w = p 1 v 1 + + p n v n. Then there exst ε 1,..., ε
More informationConjugacy and the Exponential Family
CS281B/Stat241B: Advanced Topcs n Learnng & Decson Makng Conjugacy and the Exponental Famly Lecturer: Mchael I. Jordan Scrbes: Bran Mlch 1 Conjugacy In the prevous lecture, we saw conjugate prors for the
More informationStat260: Bayesian Modeling and Inference Lecture Date: February 22, Reference Priors
Stat60: Bayesan Modelng and Inference Lecture Date: February, 00 Reference Prors Lecturer: Mchael I. Jordan Scrbe: Steven Troxler and Wayne Lee In ths lecture, we assume that θ R; n hgher-dmensons, reference
More informationVariational Bayesian Theory
Chapter 2 Varatonal Bayesan Theory 2.1 Introducton Ths chapter covers the majorty of the theory for varatonal Bayesan learnng that wll be used n rest of ths thess. It s ntended to gve the reader a context
More informationThe EM Algorithm (Dempster, Laird, Rubin 1977) The missing data or incomplete data setting: ODL(φ;Y ) = [Y;φ] = [Y X,φ][X φ] = X
The EM Algorthm (Dempster, Lard, Rubn 1977 The mssng data or ncomplete data settng: An Observed Data Lkelhood (ODL that s a mxture or ntegral of Complete Data Lkelhoods (CDL. (1a ODL(;Y = [Y;] = [Y,][
More informationProbabilistic & Unsupervised Learning
Probablstc & Unsupervsed Learnng Convex Algorthms n Approxmate Inference Yee Whye Teh ywteh@gatsby.ucl.ac.uk Gatsby Computatonal Neuroscence Unt Unversty College London Term 1, Autumn 2008 Convexty A convex
More informationOutline. Bayesian Networks: Maximum Likelihood Estimation and Tree Structure Learning. Our Model and Data. Outline
Outlne Bayesan Networks: Maxmum Lkelhood Estmaton and Tree Structure Learnng Huzhen Yu janey.yu@cs.helsnk.f Dept. Computer Scence, Unv. of Helsnk Probablstc Models, Sprng, 200 Notces: I corrected a number
More informationSolutions Homework 4 March 5, 2018
1 Solutons Homework 4 March 5, 018 Soluton to Exercse 5.1.8: Let a IR be a translaton and c > 0 be a re-scalng. ˆb1 (cx + a) cx n + a (cx 1 + a) c x n x 1 cˆb 1 (x), whch shows ˆb 1 s locaton nvarant and
More informationParametric fractional imputation for missing data analysis. Jae Kwang Kim Survey Working Group Seminar March 29, 2010
Parametrc fractonal mputaton for mssng data analyss Jae Kwang Km Survey Workng Group Semnar March 29, 2010 1 Outlne Introducton Proposed method Fractonal mputaton Approxmaton Varance estmaton Multple mputaton
More informationMean Field / Variational Approximations
Mean Feld / Varatonal Appromatons resented by Jose Nuñez 0/24/05 Outlne Introducton Mean Feld Appromaton Structured Mean Feld Weghted Mean Feld Varatonal Methods Introducton roblem: We have dstrbuton but
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationFirst Year Examination Department of Statistics, University of Florida
Frst Year Examnaton Department of Statstcs, Unversty of Florda May 7, 010, 8:00 am - 1:00 noon Instructons: 1. You have four hours to answer questons n ths examnaton.. You must show your work to receve
More informationWhy BP Works STAT 232B
Why BP Works STAT 232B Free Energes Helmholz & Gbbs Free Energes 1 Dstance between Probablstc Models - K-L dvergence b{ KL b{ p{ = b{ ln { } p{ Here, p{ s the eact ont prob. b{ s the appromaton, called
More information= z 20 z n. (k 20) + 4 z k = 4
Problem Set #7 solutons 7.2.. (a Fnd the coeffcent of z k n (z + z 5 + z 6 + z 7 + 5, k 20. We use the known seres expanson ( n+l ( z l l z n below: (z + z 5 + z 6 + z 7 + 5 (z 5 ( + z + z 2 + z + 5 5
More informationMath 426: Probability MWF 1pm, Gasson 310 Homework 4 Selected Solutions
Exercses from Ross, 3, : Math 26: Probablty MWF pm, Gasson 30 Homework Selected Solutons 3, p. 05 Problems 76, 86 3, p. 06 Theoretcal exercses 3, 6, p. 63 Problems 5, 0, 20, p. 69 Theoretcal exercses 2,
More informationA tutorial on variational Bayesian inference. Charles W. Fox & Stephen J. Roberts
A tutoral on varatonal Bayesan nference Charles W. Fox & Stephen J. Roberts Artfcal Intellgence Revew An Internatonal Scence and Engneerng Journal ISSN 0269-2821 Artf Intell Rev DOI 10.1007/ s10462-011-9236-8
More informationGaussian Mixture Models
Lab Gaussan Mxture Models Lab Objectve: Understand the formulaton of Gaussan Mxture Models (GMMs) and how to estmate GMM parameters. You ve already seen GMMs as the observaton dstrbuton n certan contnuous
More informationLecture 3: Probability Distributions
Lecture 3: Probablty Dstrbutons Random Varables Let us begn by defnng a sample space as a set of outcomes from an experment. We denote ths by S. A random varable s a functon whch maps outcomes nto the
More informationSELECTED PROOFS. DeMorgan s formulas: The first one is clear from Venn diagram, or the following truth table:
SELECTED PROOFS DeMorgan s formulas: The frst one s clear from Venn dagram, or the followng truth table: A B A B A B Ā B Ā B T T T F F F F T F T F F T F F T T F T F F F F F T T T T The second one can be
More informationU.C. Berkeley CS294: Spectral Methods and Expanders Handout 8 Luca Trevisan February 17, 2016
U.C. Berkeley CS94: Spectral Methods and Expanders Handout 8 Luca Trevsan February 7, 06 Lecture 8: Spectral Algorthms Wrap-up In whch we talk about even more generalzatons of Cheeger s nequaltes, and
More informationThe Expectation-Maximization Algorithm
The Expectaton-Maxmaton Algorthm Charles Elan elan@cs.ucsd.edu November 16, 2007 Ths chapter explans the EM algorthm at multple levels of generalty. Secton 1 gves the standard hgh-level verson of the algorthm.
More informationStanford University CS359G: Graph Partitioning and Expanders Handout 4 Luca Trevisan January 13, 2011
Stanford Unversty CS359G: Graph Parttonng and Expanders Handout 4 Luca Trevsan January 3, 0 Lecture 4 In whch we prove the dffcult drecton of Cheeger s nequalty. As n the past lectures, consder an undrected
More informationIntegrals and Invariants of Euler-Lagrange Equations
Lecture 16 Integrals and Invarants of Euler-Lagrange Equatons ME 256 at the Indan Insttute of Scence, Bengaluru Varatonal Methods and Structural Optmzaton G. K. Ananthasuresh Professor, Mechancal Engneerng,
More informationNumerical Algorithms for Visual Computing 2008/09 Example Solutions for Assignment 4. Problem 1 (Shift invariance of the Laplace operator)
Numercal Algorthms for Vsual Computng 008/09 Example Solutons for Assgnment 4 Problem (Shft nvarance of the Laplace operator The Laplace equaton s shft nvarant,.e., nvarant under translatons x x + a, y
More informationCHAPTER 5 NUMERICAL EVALUATION OF DYNAMIC RESPONSE
CHAPTER 5 NUMERICAL EVALUATION OF DYNAMIC RESPONSE Analytcal soluton s usually not possble when exctaton vares arbtrarly wth tme or f the system s nonlnear. Such problems can be solved by numercal tmesteppng
More informationChapter 4: Root Finding
Chapter 4: Root Fndng Startng values Closed nterval methods (roots are search wthn an nterval o Bsecton Open methods (no nterval o Fxed Pont o Newton-Raphson o Secant Method Repeated roots Zeros of Hgher-Dmensonal
More informationLeast squares cubic splines without B-splines S.K. Lucas
Least squares cubc splnes wthout B-splnes S.K. Lucas School of Mathematcs and Statstcs, Unversty of South Australa, Mawson Lakes SA 595 e-mal: stephen.lucas@unsa.edu.au Submtted to the Gazette of the Australan
More informationMarginal Effects in Probit Models: Interpretation and Testing. 1. Interpreting Probit Coefficients
ECON 5 -- NOE 15 Margnal Effects n Probt Models: Interpretaton and estng hs note ntroduces you to the two types of margnal effects n probt models: margnal ndex effects, and margnal probablty effects. It
More information% & 5.3 PRACTICAL APPLICATIONS. Given system, (49) , determine the Boolean Function, , in such a way that we always have expression: " Y1 = Y2
5.3 PRACTICAL APPLICATIONS st EXAMPLE: Gven system, (49) & K K Y XvX 3 ( 2 & X ), determne the Boolean Functon, Y2 X2 & X 3 v X " X3 (X2,X)", n such a way that we always have expresson: " Y Y2 " (50).
More informationUsing T.O.M to Estimate Parameter of distributions that have not Single Exponential Family
IOSR Journal of Mathematcs IOSR-JM) ISSN: 2278-5728. Volume 3, Issue 3 Sep-Oct. 202), PP 44-48 www.osrjournals.org Usng T.O.M to Estmate Parameter of dstrbutons that have not Sngle Exponental Famly Jubran
More informationMATH 829: Introduction to Data Mining and Analysis The EM algorithm (part 2)
1/16 MATH 829: Introducton to Data Mnng and Analyss The EM algorthm (part 2) Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 20, 2016 Recall 2/16 We are gven ndependent observatons
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationJoint Statistical Meetings - Biopharmaceutical Section
Iteratve Ch-Square Test for Equvalence of Multple Treatment Groups Te-Hua Ng*, U.S. Food and Drug Admnstraton 1401 Rockvlle Pke, #200S, HFM-217, Rockvlle, MD 20852-1448 Key Words: Equvalence Testng; Actve
More informationGlobal Gaussian approximations in latent Gaussian models
Global Gaussan approxmatons n latent Gaussan models Botond Cseke Aprl 9, 2010 Abstract A revew of global approxmaton methods n latent Gaussan models. 1 Latent Gaussan models In ths secton we ntroduce notaton
More informationComputation of Higher Order Moments from Two Multinomial Overdispersion Likelihood Models
Computaton of Hgher Order Moments from Two Multnomal Overdsperson Lkelhood Models BY J. T. NEWCOMER, N. K. NEERCHAL Department of Mathematcs and Statstcs, Unversty of Maryland, Baltmore County, Baltmore,
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More informationNote 10. Modeling and Simulation of Dynamic Systems
Lecture Notes of ME 475: Introducton to Mechatroncs Note 0 Modelng and Smulaton of Dynamc Systems Department of Mechancal Engneerng, Unversty Of Saskatchewan, 57 Campus Drve, Saskatoon, SK S7N 5A9, Canada
More informationMaximum Likelihood Estimation
Maxmum Lkelhood Estmaton INFO-2301: Quanttatve Reasonng 2 Mchael Paul and Jordan Boyd-Graber MARCH 7, 2017 INFO-2301: Quanttatve Reasonng 2 Paul and Boyd-Graber Maxmum Lkelhood Estmaton 1 of 9 Why MLE?
More informationIntroduction to Vapor/Liquid Equilibrium, part 2. Raoult s Law:
CE304, Sprng 2004 Lecture 4 Introducton to Vapor/Lqud Equlbrum, part 2 Raoult s Law: The smplest model that allows us do VLE calculatons s obtaned when we assume that the vapor phase s an deal gas, and
More informationLecture 21: Numerical methods for pricing American type derivatives
Lecture 21: Numercal methods for prcng Amercan type dervatves Xaoguang Wang STAT 598W Aprl 10th, 2014 (STAT 598W) Lecture 21 1 / 26 Outlne 1 Fnte Dfference Method Explct Method Penalty Method (STAT 598W)
More information1 Matrix representations of canonical matrices
1 Matrx representatons of canoncal matrces 2-d rotaton around the orgn: ( ) cos θ sn θ R 0 = sn θ cos θ 3-d rotaton around the x-axs: R x = 1 0 0 0 cos θ sn θ 0 sn θ cos θ 3-d rotaton around the y-axs:
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationModelli Clamfim Equazione del Calore Lezione ottobre 2014
CLAMFIM Bologna Modell 1 @ Clamfm Equazone del Calore Lezone 17 15 ottobre 2014 professor Danele Rtell danele.rtell@unbo.t 1/24? Convoluton The convoluton of two functons g(t) and f(t) s the functon (g
More informationPROBABILITY PRIMER. Exercise Solutions
PROBABILITY PRIMER Exercse Solutons 1 Probablty Prmer, Exercse Solutons, Prncples of Econometrcs, e EXERCISE P.1 (b) X s a random varable because attendance s not known pror to the outdoor concert. Before
More informationWhat would be a reasonable choice of the quantization step Δ?
CE 108 HOMEWORK 4 EXERCISE 1. Suppose you are samplng the output of a sensor at 10 KHz and quantze t wth a unform quantzer at 10 ts per sample. Assume that the margnal pdf of the sgnal s Gaussan wth mean
More informationCIE4801 Transportation and spatial modelling Trip distribution
CIE4801 ransportaton and spatal modellng rp dstrbuton Rob van Nes, ransport & Plannng 17/4/13 Delft Unversty of echnology Challenge the future Content What s t about hree methods Wth specal attenton for
More informationP exp(tx) = 1 + t 2k M 2k. k N
1. Subgaussan tals Defnton. Say that a random varable X has a subgaussan dstrbuton wth scale factor σ< f P exp(tx) exp(σ 2 t 2 /2) for all real t. For example, f X s dstrbuted N(,σ 2 ) then t s subgaussan.
More informationIntegrals and Invariants of
Lecture 16 Integrals and Invarants of Euler Lagrange Equatons NPTEL Course Varatonal Methods and Structural Optmzaton G. K. Ananthasuresh Professor, Mechancal Engneerng, Indan Insttute of Scence, Banagalore
More informationChapter 12. Ordinary Differential Equation Boundary Value (BV) Problems
Chapter. Ordnar Dfferental Equaton Boundar Value (BV) Problems In ths chapter we wll learn how to solve ODE boundar value problem. BV ODE s usuall gven wth x beng the ndependent space varable. p( x) q(
More informationMarkov Chain Monte Carlo (MCMC), Gibbs Sampling, Metropolis Algorithms, and Simulated Annealing Bioinformatics Course Supplement
Markov Chan Monte Carlo MCMC, Gbbs Samplng, Metropols Algorthms, and Smulated Annealng 2001 Bonformatcs Course Supplement SNU Bontellgence Lab http://bsnuackr/ Outlne! Markov Chan Monte Carlo MCMC! Metropols-Hastngs
More informationA Hybrid Variational Iteration Method for Blasius Equation
Avalable at http://pvamu.edu/aam Appl. Appl. Math. ISSN: 1932-9466 Vol. 10, Issue 1 (June 2015), pp. 223-229 Applcatons and Appled Mathematcs: An Internatonal Journal (AAM) A Hybrd Varatonal Iteraton Method
More informationGaussian Conditional Random Field Network for Semantic Segmentation - Supplementary Material
Gaussan Condtonal Random Feld Networ for Semantc Segmentaton - Supplementary Materal Ravtea Vemulapall, Oncel Tuzel *, Mng-Yu Lu *, and Rama Chellappa Center for Automaton Research, UMIACS, Unversty of
More informationNP-Completeness : Proofs
NP-Completeness : Proofs Proof Methods A method to show a decson problem Π NP-complete s as follows. (1) Show Π NP. (2) Choose an NP-complete problem Π. (3) Show Π Π. A method to show an optmzaton problem
More informationComments on Detecting Outliers in Gamma Distribution by M. Jabbari Nooghabi et al. (2010)
Comments on Detectng Outlers n Gamma Dstrbuton by M. Jabbar Nooghab et al. (21) M. Magdalena Lucn Alejandro C. Frery September 17, 215 arxv:159.55v1 [stat.co] 16 Sep 215 Ths note shows that the results
More informationANSWERS. Problem 1. and the moment generating function (mgf) by. defined for any real t. Use this to show that E( U) var( U)
Econ 413 Exam 13 H ANSWERS Settet er nndelt 9 deloppgaver, A,B,C, som alle anbefales å telle lkt for å gøre det ltt lettere å stå. Svar er gtt . Unfortunately, there s a prntng error n the hnt of
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationCollege of Computer & Information Science Fall 2009 Northeastern University 20 October 2009
College of Computer & Informaton Scence Fall 2009 Northeastern Unversty 20 October 2009 CS7880: Algorthmc Power Tools Scrbe: Jan Wen and Laura Poplawsk Lecture Outlne: Prmal-dual schema Network Desgn:
More informationAnnouncements EWA with ɛ-exploration (recap) Lecture 20: EXP3 Algorithm. EECS598: Prediction and Learning: It s Only a Game Fall 2013.
Lecture 0: EXP3 Algorthm 1 EECS598: Predcton and Learnng: It s Only a Game Fall 013 Prof. Jacob Abernethy Lecture 0: EXP3 Algorthm Scrbe: Zhhao Chen Announcements None 0.1 EWA wth ɛ-exploraton (recap)
More informationFormal solvers of the RT equation
Formal solvers of the RT equaton Formal RT solvers Runge- Kutta (reference solver) Pskunov N.: 979, Master Thess Long characterstcs (Feautrer scheme) Cannon C.J.: 970, ApJ 6, 55 Short characterstcs (Hermtan
More informationarxiv: v2 [stat.me] 26 Jun 2012
The Two-Way Lkelhood Rato (G Test and Comparson to Two-Way χ Test Jesse Hoey June 7, 01 arxv:106.4881v [stat.me] 6 Jun 01 1 One-Way Lkelhood Rato or χ test Suppose we have a set of data x and two hypotheses
More informationInexact Newton Methods for Inverse Eigenvalue Problems
Inexact Newton Methods for Inverse Egenvalue Problems Zheng-jan Ba Abstract In ths paper, we survey some of the latest development n usng nexact Newton-lke methods for solvng nverse egenvalue problems.
More informationStat 543 Exam 2 Spring 2016
Stat 543 Exam 2 Sprng 2016 I have nether gven nor receved unauthorzed assstance on ths exam. Name Sgned Date Name Prnted Ths Exam conssts of 11 questons. Do at least 10 of the 11 parts of the man exam.
More informationON MECHANICS WITH VARIABLE NONCOMMUTATIVITY
ON MECHANICS WITH VARIABLE NONCOMMUTATIVITY CIPRIAN ACATRINEI Natonal Insttute of Nuclear Physcs and Engneerng P.O. Box MG-6, 07725-Bucharest, Romana E-mal: acatrne@theory.npne.ro. Receved March 6, 2008
More informationDr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur
Analyss of Varance and Desgn of Experment-I MODULE VIII LECTURE - 34 ANALYSIS OF VARIANCE IN RANDOM-EFFECTS MODEL AND MIXED-EFFECTS EFFECTS MODEL Dr Shalabh Department of Mathematcs and Statstcs Indan
More informationVQ widely used in coding speech, image, and video
at Scalar quantzers are specal cases of vector quantzers (VQ): they are constraned to look at one sample at a tme (memoryless) VQ does not have such constrant better RD perfomance expected Source codng
More informationLecture 20: Hypothesis testing
Lecture : Hpothess testng Much of statstcs nvolves hpothess testng compare a new nterestng hpothess, H (the Alternatve hpothess to the borng, old, well-known case, H (the Null Hpothess or, decde whether
More informationStat 543 Exam 2 Spring 2016
Stat 543 Exam 2 Sprng 206 I have nether gven nor receved unauthorzed assstance on ths exam. Name Sgned Date Name Prnted Ths Exam conssts of questons. Do at least 0 of the parts of the man exam. I wll score
More informationParametric fractional imputation for missing data analysis
Secton on Survey Research Methods JSM 2008 Parametrc fractonal mputaton for mssng data analyss Jae Kwang Km Wayne Fuller Abstract Under a parametrc model for mssng data, the EM algorthm s a popular tool
More informationErratum: A Generalized Path Integral Control Approach to Reinforcement Learning
Journal of Machne Learnng Research 00-9 Submtted /0; Publshed 7/ Erratum: A Generalzed Path Integral Control Approach to Renforcement Learnng Evangelos ATheodorou Jonas Buchl Stefan Schaal Department of
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationComputing Correlated Equilibria in Multi-Player Games
Computng Correlated Equlbra n Mult-Player Games Chrstos H. Papadmtrou Presented by Zhanxang Huang December 7th, 2005 1 The Author Dr. Chrstos H. Papadmtrou CS professor at UC Berkley (taught at Harvard,
More informationEconomics 101. Lecture 4 - Equilibrium and Efficiency
Economcs 0 Lecture 4 - Equlbrum and Effcency Intro As dscussed n the prevous lecture, we wll now move from an envronment where we looed at consumers mang decsons n solaton to analyzng economes full of
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More informationAdditional Codes using Finite Difference Method. 1 HJB Equation for Consumption-Saving Problem Without Uncertainty
Addtonal Codes usng Fnte Dfference Method Benamn Moll 1 HJB Equaton for Consumpton-Savng Problem Wthout Uncertanty Before consderng the case wth stochastc ncome n http://www.prnceton.edu/~moll/ HACTproect/HACT_Numercal_Appendx.pdf,
More informationOn an Extension of Stochastic Approximation EM Algorithm for Incomplete Data Problems. Vahid Tadayon 1
On an Extenson of Stochastc Approxmaton EM Algorthm for Incomplete Data Problems Vahd Tadayon Abstract: The Stochastc Approxmaton EM (SAEM algorthm, a varant stochastc approxmaton of EM, s a versatle tool
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationAssortment Optimization under MNL
Assortment Optmzaton under MNL Haotan Song Aprl 30, 2017 1 Introducton The assortment optmzaton problem ams to fnd the revenue-maxmzng assortment of products to offer when the prces of products are fxed.
More informationProfessor Terje Haukaas University of British Columbia, Vancouver The Q4 Element
Professor Terje Haukaas Unversty of Brtsh Columba, ancouver www.nrsk.ubc.ca The Q Element Ths document consders fnte elements that carry load only n ther plane. These elements are sometmes referred to
More informationCHAPTER 14 GENERAL PERTURBATION THEORY
CHAPTER 4 GENERAL PERTURBATION THEORY 4 Introducton A partcle n orbt around a pont mass or a sphercally symmetrc mass dstrbuton s movng n a gravtatonal potental of the form GM / r In ths potental t moves
More informationThermodynamics and statistical mechanics in materials modelling II
Course MP3 Lecture 8/11/006 (JAE) Course MP3 Lecture 8/11/006 Thermodynamcs and statstcal mechancs n materals modellng II A bref résumé of the physcal concepts used n materals modellng Dr James Ellott.1
More informationMATH 281A: Homework #6
MATH 28A: Homework #6 Jongha Ryu Due date: November 8, 206 Problem. (Problem 2..2. Soluton. If X,..., X n Bern(p, then T = X s a complete suffcent statstc. Our target s g(p = p, and the nave guess suggested
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More informationLearning Theory: Lecture Notes
Learnng Theory: Lecture Notes Lecturer: Kamalka Chaudhur Scrbe: Qush Wang October 27, 2012 1 The Agnostc PAC Model Recall that one of the constrants of the PAC model s that the data dstrbuton has to be
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for
More informationRadar Trackers. Study Guide. All chapters, problems, examples and page numbers refer to Applied Optimal Estimation, A. Gelb, Ed.
Radar rackers Study Gude All chapters, problems, examples and page numbers refer to Appled Optmal Estmaton, A. Gelb, Ed. Chapter Example.0- Problem Statement wo sensors Each has a sngle nose measurement
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More information