Markov Chain Monte Carlo Lecture 6
|
|
- Ezra Byrd
- 6 years ago
- Views:
Transcription
1 where (x 1,..., x N ) X N, N s called the populaton sze, f(x) f (x) for at least one {1, 2,..., N}, and those dfferent from f(x) are called the tral dstrbutons n terms of mportance samplng. Dfferent ways of specfcaton of the tral dstrbutons and updatng the populaton Markov Chan Monte Carlo Lecture 6 An actvely pursued research drecton for allevatng the local-trap problem suffered by the Metropols-Hastngs (MH) algorthm s the populatonbased MCMC, where a populaton of Markov chans are run n parallel, each equpped wth possbly dfferent but related nvarant dstrbutons. Informaton exchange between dfferent chans provdes a means for the target chans to learn from past samples, and ths n turn mproves the convergence of the target chans. Mathematcally, the populaton-based MCMC may be descrbed as follows. In order to smulate from a target dstrbuton f(x), one smulates of an augmented system wth the nvarant dstrbuton f(x 1,..., x N ) = N =1 f (x ), (1)
2 of Markov chans lead to dfferent algorthms, such as the adaptve drecton samplng (Glks et al., 1994), conjugate gradent Monte Carlo (Lu, Lang and Wong, 2000), parallel temperng (Geyer, 1991; Hukushma and Nemoto, 1996), evolutonary Monte Carlo (Lang and Wong, 2000, 2001), sequental parallel temperng (Lang, 2003), and equ-energy sampler (Kou, Zhou and Wong, 2006).
3 f(r) r d 1 f(x (t) + r e ), (2) Markov Chan Monte Carlo Lecture 6 Adaptve drecton samplng Adaptve drecton samplng (ADS) (Glks et al., 1994) s an early populaton-based MCMC method, n whch each dstrbuton f (x) s dentcal to the target dstrbuton, and at each teraton, one sample s randomly selected from the current populaton to undergo an update along a drecton toward another sample randomly selected from the remanng set of the current populaton. An mportant form of the ADS s the snooker algorthm. 1. Select one ndvdual, say x (t) c, at random from the current populaton x (t). The x (t) c s called the current pont. 2. Select another ndvdual, say x (t) a, from the remanng set of the current populaton,.e., {x (t) : c}, and form a drecton e t = x (t) c x (t) a. The ndvdual x (t) a s called the anchor pont. 3. Set y c = x (t) a +r t e t, where r t s a scalar sampled from the densty
4 where d s the dmenson of x, and the factor r d 1 s derved from a transformaton Jacoban (Roberts and Glks, 1994). 4. Form the new populaton x (t+1) by replacng x (t) c all other ndvduals unchanged (.e., set x (t+1) by y c and leavng = x (t) for c).
5 To show the sampler s proper, we need to show that at the equlbrum the new sample y c s ndependent of the x (t) for a and s dstrbuted as f(x). Ths fact follows drectly from the followng lemma, whch s a generalzed verson of Lemma 3.1 of Roberts and Glks (1994) and was proved by Lu, Lang and Wong (2000). Lemma 0.1 (Lu, Lang and Wong, 2000) Suppose x π(x) and y s any fxed pont n a d-dmensonal space. Let e = x y. If r s drawn from dstrbuton f(r) r d 1 π(y + re), then x = y + re follows the dstrbuton π(x). If y s generated from a dstrbuton ndependent of x, then x s ndependent of y.
6 Conjugate gradent Monte Carlo (Lu, Lang and Wong, 2000) Let x (t) = (x (t) 1,..., x (t) N ) denote the current populaton of samples. One teraton of the CGMC sampler conssts of the followng steps. 1. Select one ndvdual, say x (t) c, at random from the current populaton x (t). 2. Select another ndvdual, say x (t) a, at random from the remanng set of the populaton,.e. {x (t) : c}. Startng wth x (t) a, conduct a determnstc search, usng the conjugate gradent method or the steepest descent method, to fnd a local mode of f(x). Denote the local mode by z a (t), whch s called the anchor pont. 3. Set y c = z (t) a sampled from the densty + r t e t, where e t = x (t) c z a (t), and r t s a scalar f(r) r d 1 f(z (t) a + r t e t ), (3) where d s the dmenson of x, and the factor r d 1 s derved from the transformaton Jacoban.
7 4. Form the new populaton x (t+1) by replacng x (t) c other ndvduals unchanged (.e., set x (t+1) = x (t) by y c and leavng for c). The gradent-based optmzaton procedure performed n step 2 can be replaced by some other optmzaton procedures, for example, a short run of smulated annealng (Krkpatrck et al., 1983). Snce the local optmzaton step s usually expensve n computaton, Lu, Lang and Wong (2000) proposed the multple-try MH algorthm for the lne samplng step, whch enables effectve use of the local modal nformaton of the dstrbuton and thus mprove the convergence of the algorthm.
8 Sample MH Algorthm (Lewandowsk and Lu, 2008) In adaptve drecton samplng and conjugate gradent Monte Carlo, when updatng the populaton, one frst selects an ndvdual from the populaton and then updates the selected ndvdual usng the standard Metropols-Hastngs procedure. If the canddate state s of hgh qualty relatve to the whole populaton, one certanly wants to keep t n the populaton. However, the acceptance of the canddate state depends on the qualty of the ndvdual that s selected for updatng. To mprove the acceptance rate of hgh qualty canddates and to mprove the set {x (t) : = 1,..., N} as a sample of sze N from f(x), Lewandowsk and Lu (2008) proposed the samplng Metropols-Hastngs (SMH) algorthm.
9 Sample MH Algorthm Take one canddate draw x (t) 0 from a proposal dstrbuton g(x) on X, and compute the acceptance probablty α (t) 0 = N =0 g(x (t) ) f(x (t) N =1 g(x (t) ) f(x (t) ) ) mn 0 k N. g(x (t) k ) f(x (t) k ) Draw U Unf0, 1, and set { } S t+1 = x (t+1) 1,..., x (t+1) n { St, f U > α (t) 0 ; = { } x (t) 1,..., x (t) 1, x(t) 0, x (t) +1,..., x(t) n, f U α (t) 0,
10 where s chosen from (1,..., n) wth the probablty weghts ( ) g(x (t) (t) 1 ) g(xn ),...,. f(x (t) 1 ) f(x (t) n ) Thus, x t+1 and x t dffer by one element at most. It s easy to see that n the case of N = 1, SMH reduces to the tradtonal MH wth ndependence proposals. The mert of SMH s that to accept a canddate state, t compares the canddate wth the whole populaton, nstead of a sngle ndvdual randomly selected from the current populaton. Lewandowsk and Lu (2008) show that SMH wll converge under mld condtons to the target dstrbuton N =1 f(x ) for {x 1,..., x N }, and can be more effcent than the tradtonal MH and adaptve drecton samplng.
11 Parallel temperng (Geyer, 1991) Parallel temperng smulates n parallel a sequence of dstrbutons f (x) exp( H(x)/T ), = 1,..., n, (4) where T s the temperature assocated wth the dstrbuton f (x). The temperatures form a ladder T 1 > T 2 > > T n 1 > T n 1, so f n (x) f(x) corresponds to the target dstrbuton. The dea underlyng ths algorthm can be explaned as follows: Rasng temperature flattens the energy landscape of the dstrbuton and thus eases the MH traversal of the sample space, the hgh densty samples generated at the hgh temperature levels can be transmtted to the target temperature level through the exchange operatons, and ths n turn mproves convergence of the target Markov chan.
12 N ) denote the current populaton of samples. One teraton of parallel temperng conssts of the followng steps. Let x (t) = (x (t) 1,..., x (t) usng the MH algo- 1. Parallel MH step: Update each x (t) rthm. to x (t+1) 2. State swappng step: Try to exchange x (t+1) wth ts neghbors: Set j = 1 or + 1 accordng to probabltes q e (, j), where q e (, + 1) = q e (, 1) = 0.5 for 1 < < N and q e (1, 2) = q e (N, N 1) = 1, and accept the swap wth probablty mn { 1, exp ( [ H(x (t+1) ] [ ) H(x (t+1) 1 j ) 1 T T j ])}. (5)
13 Evolutonary Monte Carlo (Lang and Wong, 2000, 2001) The genetc algorthm (Holland, 1975) has been successfully appled to many hard optmzaton problems, such as the travelng salesman problem, proten foldng, machne learnng, among others. It s known that ts crossover operator s the key to the power of the genetc algorthm, whch makes t possble to explore a far greater range of potental solutons to a problem than conventonal optmzaton algorthms. Motvated by the genetc algorthm, Lang and Wong (2000, 2001) proposed the evolutonary Monte Carlo algorthm (EMC), whch ncorporates most attractve features of the genetc algorthm nto the framework of Markov chan Monte Carlo. EMC works n a fashon smlar to parallel temperng: A populaton of Markov chans are smulated n parallel wth each chan havng a dfferent temperature. The dfference between the two algorthms s that EMC ncludes a genetc operator, namely, the crossover operator n ts smulaton. The numercal results ndcate that the crossover operator mproves the convergence of the smulaton and that EMC can outperform parallel temperng n almost all scenaros.
14 Suppose the target dstrbuton of nterest s wrtten n the form f(x) exp{ H(x)}, x X R d, where the dmenson d > 1, and H(x) s called the ftness functon n terms of genetc algorthms. Let x = {x 1,..., x N } denote a populaton of sze N wth x from the dstrbuton wth densty f (x) exp{ H(x)/T }. In terms of genetc algorthms, x s called a chromosome or an ndvdual, each element of x s called a gene, and a realzaton of the element s called a genotype. As n parallel temperng, the temperatures form a decreasng ladder T 1 > T 2 > > T N 1, wth f N (x) beng the target dstrbuton.
15 Mutaton The mutaton operator s defned as an addtve Metropols-Hastngs move. One chromosome, say x k, s randomly selected from the current populaton x. A new chromosome s generated by addng a random vector e k so that y k = x k + e k, (6) where the scale of e k s chosen such that the operaton has a moderate acceptance rate, e.g., 0.2 to 0.5, as suggested by Gelman, Roberts and Glks (1996). The new populaton y = {x 1,, x k 1, y k, x k+1,, x N } s accepted wth probablty mn(1,r m ), where r m = f(y) f(x) T (x y) T (y x) = exp { H(y k) H(x k ) T k } T (x y) T (y x), (7) and T ( ) denotes the transton probablty between populatons.
16 f(r) r d 1 f(x + re). (8) Markov Chan Monte Carlo Lecture 6 Crossover One type of crossover operators that works for the real-coded chromosomes s the so-called real crossover, whch ncludes the k-pont and unform crossover operators. They are called real crossover by Wrght (1991) to ndcate that they are appled to real-coded chromosomes. In addton to the real crossover, Lang and Wong (2001a) proposed the snooker crossover operator, whch works as follows: 1. Randomly select one chromosome, say x, from the current populaton x. 2. Select the other chromosome, say x j, from the sub-populaton x\ {x } wth a probablty proportonal to exp{ H(x j )/T s }, where T s s called the selecton temperature. 3. Let e = x x j, and y = x j + re, where r (, ) s a random varable sampled from the densty
17 4. Construct a new populaton by replacng x wth the offsprng y, and replace x by y.
18 Exchange Ths operaton s the same as that used n parallel temperng (Geyer, 1991; Hukushma and Nemoto, 1996). Gven the current populaton x and the temperature ladder t, (x, t) = (x 1, T 1,, x N, T N ), one tres to make an exchange between x and x j wthout changng the t s. The new populaton s accepted wth probablty mn(1,r e ), ( )} 1 r e = f(x ) f(x) T (x x ) T (x x) = exp { (H(x ) H(x j )) T 1 T j (9) Typcally, the exchange s only performed on neghborng temperature levels.,
19 The Algorthm Based on the operators descrbed above, the algorthm can be summarzed as follows. Gven an ntal populaton x = {x 1,, x N } and a temperature ladder t = {T 1, T 2,, T N }, EMC terates between the followng two steps: 1. Apply ether mutaton or crossover operator to the populaton wth probablty q m and 1 q m, respectvely. The q m s called the mutaton rate. 2. Try to exchange x wth x j for N pars (, j) wth beng sampled unformly on {1,, N} and j = ± 1 wth probablty q e (, j), where q e (, + 1) = q e (, 1) = 0.5 and q e (1, 2) = q e (N, N 1) = 1.
20 Consder smulatng from a 2D mxture normal dstrbuton f(x) = πσ k=1 w k exp{ 1 2σ 2 (x µ k) (x µ k )}, (10) where σ = 0.1, w 1 = = w 20 = The mean vectors µ 1, µ 2,, µ 20 (gven n Table 1) are unformly drawn from the rectangle [0, 10] [0, 10]. Among them, components 2, 4, and 15 are well separated from the others. The dstance between component 4 and ts nearest neghborng component s 3.15, and the dstance between component 15 and ts nearest neghborng component (except component 2) s 3.84, whch are 31.5 and 38.4 tmes of the standard devaton, respectvely. Mxng the components across so long dstances puts a great challenge on EMC.
21 Table 1: Mean vectors of the 20 components of the mxture normal dstrbuton (Lang and Wong, 2001). k µ k1 µ k2 k µ k1 µ k2 k µ k1 µ k2 k µ k1 µ k
22 Table 2: Comparson of EMC and parallel temperng for the mxture normal example. (Lang and Wong, 2001) parameter true value EMC-A EMC-B PT est. SD est. SD est. SD µ µ Σ Σ Σ
23 x y (a) evolutonary samplng x y (b) parallel temperng Fgure 1: The sample path of the frst teratons at temperature t = 1. (a) EMC. (b) Parallel temperng. (Lang and Wong, 2001a)
24 x y (a) evolutonary samplng x y (b) parallel temperng Fgure 2: The plot of whole samples. (a) EMC. (b) Parallel temperng. (Lang and Wong, 2001a)
Markov Chain Monte Carlo (MCMC), Gibbs Sampling, Metropolis Algorithms, and Simulated Annealing Bioinformatics Course Supplement
Markov Chan Monte Carlo MCMC, Gbbs Samplng, Metropols Algorthms, and Smulated Annealng 2001 Bonformatcs Course Supplement SNU Bontellgence Lab http://bsnuackr/ Outlne! Markov Chan Monte Carlo MCMC! Metropols-Hastngs
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationThe Study of Teaching-learning-based Optimization Algorithm
Advanced Scence and Technology Letters Vol. (AST 06), pp.05- http://dx.do.org/0.57/astl.06. The Study of Teachng-learnng-based Optmzaton Algorthm u Sun, Yan fu, Lele Kong, Haolang Q,, Helongang Insttute
More informationReport on Image warping
Report on Image warpng Xuan Ne, Dec. 20, 2004 Ths document summarzed the algorthms of our mage warpng soluton for further study, and there s a detaled descrpton about the mplementaton of these algorthms.
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationCIS526: Machine Learning Lecture 3 (Sept 16, 2003) Linear Regression. Preparation help: Xiaoying Huang. x 1 θ 1 output... θ M x M
CIS56: achne Learnng Lecture 3 (Sept 6, 003) Preparaton help: Xaoyng Huang Lnear Regresson Lnear regresson can be represented by a functonal form: f(; θ) = θ 0 0 +θ + + θ = θ = 0 ote: 0 s a dummy attrbute
More informationInformation Geometry of Gibbs Sampler
Informaton Geometry of Gbbs Sampler Kazuya Takabatake Neuroscence Research Insttute AIST Central 2, Umezono 1-1-1, Tsukuba JAPAN 305-8568 k.takabatake@ast.go.jp Abstract: - Ths paper shows some nformaton
More informationBoostrapaggregating (Bagging)
Boostrapaggregatng (Baggng) An ensemble meta-algorthm desgned to mprove the stablty and accuracy of machne learnng algorthms Can be used n both regresson and classfcaton Reduces varance and helps to avod
More informationLecture 20: November 7
0-725/36-725: Convex Optmzaton Fall 205 Lecturer: Ryan Tbshran Lecture 20: November 7 Scrbes: Varsha Chnnaobreddy, Joon Sk Km, Lngyao Zhang Note: LaTeX template courtesy of UC Berkeley EECS dept. Dsclamer:
More informationOn an Extension of Stochastic Approximation EM Algorithm for Incomplete Data Problems. Vahid Tadayon 1
On an Extenson of Stochastc Approxmaton EM Algorthm for Incomplete Data Problems Vahd Tadayon Abstract: The Stochastc Approxmaton EM (SAEM algorthm, a varant stochastc approxmaton of EM, s a versatle tool
More informationChapter - 2. Distribution System Power Flow Analysis
Chapter - 2 Dstrbuton System Power Flow Analyss CHAPTER - 2 Radal Dstrbuton System Load Flow 2.1 Introducton Load flow s an mportant tool [66] for analyzng electrcal power system network performance. Load
More informationCS 3750 Machine Learning Lecture 6. Monte Carlo methods. CS 3750 Advanced Machine Learning. Markov chain Monte Carlo
CS 3750 Machne Learnng Lectre 6 Monte Carlo methods Mlos Haskrecht mlos@cs.ptt.ed 5329 Sennott Sqare Markov chan Monte Carlo Importance samplng: samples are generated accordng to Q and every sample from
More informationAn adaptive SMC scheme for ABC. Bayesian Computation (ABC)
An adaptve SMC scheme for Approxmate Bayesan Computaton (ABC) (ont work wth Prof. Mke West) Department of Statstcal Scence - Duke Unversty Aprl/2011 Approxmate Bayesan Computaton (ABC) Problems n whch
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationParametric fractional imputation for missing data analysis. Jae Kwang Kim Survey Working Group Seminar March 29, 2010
Parametrc fractonal mputaton for mssng data analyss Jae Kwang Km Survey Workng Group Semnar March 29, 2010 1 Outlne Introducton Proposed method Fractonal mputaton Approxmaton Varance estmaton Multple mputaton
More informationLecture 3: Probability Distributions
Lecture 3: Probablty Dstrbutons Random Varables Let us begn by defnng a sample space as a set of outcomes from an experment. We denote ths by S. A random varable s a functon whch maps outcomes nto the
More informationProbabilistic Graphical Models
School of Computer Scence robablstc Graphcal Models Appromate Inference: Markov Chan Monte Carlo 05 07 Erc Xng Lecture 7 March 9 04 X X 075 05 05 03 X 3 Erc Xng @ CMU 005-04 Recap of Monte Carlo Monte
More informationMATH 829: Introduction to Data Mining and Analysis The EM algorithm (part 2)
1/16 MATH 829: Introducton to Data Mnng and Analyss The EM algorthm (part 2) Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 20, 2016 Recall 2/16 We are gven ndependent observatons
More informationMean Field / Variational Approximations
Mean Feld / Varatonal Appromatons resented by Jose Nuñez 0/24/05 Outlne Introducton Mean Feld Appromaton Structured Mean Feld Weghted Mean Feld Varatonal Methods Introducton roblem: We have dstrbuton but
More informationLinear Regression Analysis: Terminology and Notation
ECON 35* -- Secton : Basc Concepts of Regresson Analyss (Page ) Lnear Regresson Analyss: Termnology and Notaton Consder the generc verson of the smple (two-varable) lnear regresson model. It s represented
More information10.34 Fall 2015 Metropolis Monte Carlo Algorithm
10.34 Fall 2015 Metropols Monte Carlo Algorthm The Metropols Monte Carlo method s very useful for calculatng manydmensonal ntegraton. For e.g. n statstcal mechancs n order to calculate the prospertes of
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationAdaptive evolutionary Monte Carlo algorithm for optimization with applications to sensor placement problems
Stat Comput (2008) 18: 375 390 DOI 10.1007/s11222-008-9079-6 Adaptve evolutonary Monte Carlo algorthm for optmzaton wth applcatons to sensor placement problems Yuan Ren Yu Dng Famng Lang Receved: 28 Aprl
More informationOutline for today. Markov chain Monte Carlo. Example: spatial statistics (Christensen and Waagepetersen 2001)
Markov chan Monte Carlo Rasmus Waagepetersen Department of Mathematcs Aalborg Unversty Denmark November, / Outlne for today MCMC / Condtonal smulaton for hgh-dmensonal U: Markov chan Monte Carlo Consder
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More informationDesign and Optimization of Fuzzy Controller for Inverse Pendulum System Using Genetic Algorithm
Desgn and Optmzaton of Fuzzy Controller for Inverse Pendulum System Usng Genetc Algorthm H. Mehraban A. Ashoor Unversty of Tehran Unversty of Tehran h.mehraban@ece.ut.ac.r a.ashoor@ece.ut.ac.r Abstract:
More informationProblem Points Score Total 100
Physcs 450 Solutons of Sample Exam I Problem Ponts Score 1 8 15 3 17 4 0 5 0 Total 100 All wor must be shown n order to receve full credt. Wor must be legble and comprehensble wth answers clearly ndcated.
More informationHopfield networks and Boltzmann machines. Geoffrey Hinton et al. Presented by Tambet Matiisen
Hopfeld networks and Boltzmann machnes Geoffrey Hnton et al. Presented by Tambet Matsen 18.11.2014 Hopfeld network Bnary unts Symmetrcal connectons http://www.nnwj.de/hopfeld-net.html Energy functon The
More information( ) ( ) ( ) ( ) STOCHASTIC SIMULATION FOR BLOCKED DATA. Monte Carlo simulation Rejection sampling Importance sampling Markov chain Monte Carlo
SOCHASIC SIMULAIO FOR BLOCKED DAA Stochastc System Analyss and Bayesan Model Updatng Monte Carlo smulaton Rejecton samplng Importance samplng Markov chan Monte Carlo Monte Carlo smulaton Introducton: If
More informationj) = 1 (note sigma notation) ii. Continuous random variable (e.g. Normal distribution) 1. density function: f ( x) 0 and f ( x) dx = 1
Random varables Measure of central tendences and varablty (means and varances) Jont densty functons and ndependence Measures of assocaton (covarance and correlaton) Interestng result Condtonal dstrbutons
More informationChecking Pairwise Relationships. Lecture 19 Biostatistics 666
Checkng Parwse Relatonshps Lecture 19 Bostatstcs 666 Last Lecture: Markov Model for Multpont Analyss X X X 1 3 X M P X 1 I P X I P X 3 I P X M I 1 3 M I 1 I I 3 I M P I I P I 3 I P... 1 IBD states along
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationAppendix B: Resampling Algorithms
407 Appendx B: Resamplng Algorthms A common problem of all partcle flters s the degeneracy of weghts, whch conssts of the unbounded ncrease of the varance of the mportance weghts ω [ ] of the partcles
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationA Robust Method for Calculating the Correlation Coefficient
A Robust Method for Calculatng the Correlaton Coeffcent E.B. Nven and C. V. Deutsch Relatonshps between prmary and secondary data are frequently quantfed usng the correlaton coeffcent; however, the tradtonal
More informationCSC321 Tutorial 9: Review of Boltzmann machines and simulated annealing
CSC321 Tutoral 9: Revew of Boltzmann machnes and smulated annealng (Sldes based on Lecture 16-18 and selected readngs) Yue L Emal: yuel@cs.toronto.edu Wed 11-12 March 19 Fr 10-11 March 21 Outlne Boltzmann
More informationSolving of Single-objective Problems based on a Modified Multiple-crossover Genetic Algorithm: Test Function Study
Internatonal Conference on Systems, Sgnal Processng and Electroncs Engneerng (ICSSEE'0 December 6-7, 0 Duba (UAE Solvng of Sngle-objectve Problems based on a Modfed Multple-crossover Genetc Algorthm: Test
More informationTemperature. Chapter Heat Engine
Chapter 3 Temperature In prevous chapters of these notes we ntroduced the Prncple of Maxmum ntropy as a technque for estmatng probablty dstrbutons consstent wth constrants. In Chapter 9 we dscussed the
More informationANSWERS. Problem 1. and the moment generating function (mgf) by. defined for any real t. Use this to show that E( U) var( U)
Econ 413 Exam 13 H ANSWERS Settet er nndelt 9 deloppgaver, A,B,C, som alle anbefales å telle lkt for å gøre det ltt lettere å stå. Svar er gtt . Unfortunately, there s a prntng error n the hnt of
More information8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS
SECTION 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS 493 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS All the vector spaces you have studed thus far n the text are real vector spaces because the scalars
More informationBOOTSTRAP METHOD FOR TESTING OF EQUALITY OF SEVERAL MEANS. M. Krishna Reddy, B. Naveen Kumar and Y. Ramu
BOOTSTRAP METHOD FOR TESTING OF EQUALITY OF SEVERAL MEANS M. Krshna Reddy, B. Naveen Kumar and Y. Ramu Department of Statstcs, Osmana Unversty, Hyderabad -500 007, Inda. nanbyrozu@gmal.com, ramu0@gmal.com
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More information1 The Mistake Bound Model
5-850: Advanced Algorthms CMU, Sprng 07 Lecture #: Onlne Learnng and Multplcatve Weghts February 7, 07 Lecturer: Anupam Gupta Scrbe: Bryan Lee,Albert Gu, Eugene Cho he Mstake Bound Model Suppose there
More informationx = , so that calculated
Stat 4, secton Sngle Factor ANOVA notes by Tm Plachowsk n chapter 8 we conducted hypothess tests n whch we compared a sngle sample s mean or proporton to some hypotheszed value Chapter 9 expanded ths to
More informationIV. Performance Optimization
IV. Performance Optmzaton A. Steepest descent algorthm defnton how to set up bounds on learnng rate mnmzaton n a lne (varyng learnng rate) momentum learnng examples B. Newton s method defnton Gauss-Newton
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationGaussian Mixture Models
Lab Gaussan Mxture Models Lab Objectve: Understand the formulaton of Gaussan Mxture Models (GMMs) and how to estmate GMM parameters. You ve already seen GMMs as the observaton dstrbuton n certan contnuous
More information4DVAR, according to the name, is a four-dimensional variational method.
4D-Varatonal Data Assmlaton (4D-Var) 4DVAR, accordng to the name, s a four-dmensonal varatonal method. 4D-Var s actually a drect generalzaton of 3D-Var to handle observatons that are dstrbuted n tme. The
More informationWeb Appendix B Estimation. We base our sampling procedure on the method of data augmentation (e.g., Tanner and Wong,
Web Appendx B Estmaton Lkelhood and Data Augmentaton We base our samplng procedure on the method of data augmentaton (eg anner and Wong 987) here e treat the unobserved ndvdual choces as parameters Specfcally
More informationIntroduction to Vapor/Liquid Equilibrium, part 2. Raoult s Law:
CE304, Sprng 2004 Lecture 4 Introducton to Vapor/Lqud Equlbrum, part 2 Raoult s Law: The smplest model that allows us do VLE calculatons s obtaned when we assume that the vapor phase s an deal gas, and
More informationChapter 2 Real-Coded Adaptive Range Genetic Algorithm
Chapter Real-Coded Adaptve Range Genetc Algorthm.. Introducton Fndng a global optmum n the contnuous doman s challengng for Genetc Algorthms (GAs. Tradtonal GAs use the bnary representaton that evenly
More informationSDMML HT MSc Problem Sheet 4
SDMML HT 06 - MSc Problem Sheet 4. The recever operatng characterstc ROC curve plots the senstvty aganst the specfcty of a bnary classfer as the threshold for dscrmnaton s vared. Let the data space be
More informationLINEAR REGRESSION ANALYSIS. MODULE IX Lecture Multicollinearity
LINEAR REGRESSION ANALYSIS MODULE IX Lecture - 30 Multcollnearty Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur 2 Remedes for multcollnearty Varous technques have
More informationSampling Self Avoiding Walks
Samplng Self Avodng Walks James Farbanks and Langhao Chen December 3, 204 Abstract These notes present the self testng algorthm for samplng self avodng walks by Randall and Snclar[3] [4]. They are ntended
More informationWinter 2008 CS567 Stochastic Linear/Integer Programming Guest Lecturer: Xu, Huan
Wnter 2008 CS567 Stochastc Lnear/Integer Programmng Guest Lecturer: Xu, Huan Class 2: More Modelng Examples 1 Capacty Expanson Capacty expanson models optmal choces of the tmng and levels of nvestments
More informationEcon107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4)
I. Classcal Assumptons Econ7 Appled Econometrcs Topc 3: Classcal Model (Studenmund, Chapter 4) We have defned OLS and studed some algebrac propertes of OLS. In ths topc we wll study statstcal propertes
More informationSampling Theory MODULE VII LECTURE - 23 VARYING PROBABILITY SAMPLING
Samplng heory MODULE VII LECURE - 3 VARYIG PROBABILIY SAMPLIG DR. SHALABH DEPARME OF MAHEMAICS AD SAISICS IDIA ISIUE OF ECHOLOGY KAPUR he smple random samplng scheme provdes a random sample where every
More informationThe Geometry of Logit and Probit
The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.
More informationWeek3, Chapter 4. Position and Displacement. Motion in Two Dimensions. Instantaneous Velocity. Average Velocity
Week3, Chapter 4 Moton n Two Dmensons Lecture Quz A partcle confned to moton along the x axs moves wth constant acceleraton from x =.0 m to x = 8.0 m durng a 1-s tme nterval. The velocty of the partcle
More informationStructure and Drive Paul A. Jensen Copyright July 20, 2003
Structure and Drve Paul A. Jensen Copyrght July 20, 2003 A system s made up of several operatons wth flow passng between them. The structure of the system descrbes the flow paths from nputs to outputs.
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 12 10/21/2013. Martingale Concentration Inequalities and Applications
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.65/15.070J Fall 013 Lecture 1 10/1/013 Martngale Concentraton Inequaltes and Applcatons Content. 1. Exponental concentraton for martngales wth bounded ncrements.
More informationInductance Calculation for Conductors of Arbitrary Shape
CRYO/02/028 Aprl 5, 2002 Inductance Calculaton for Conductors of Arbtrary Shape L. Bottura Dstrbuton: Internal Summary In ths note we descrbe a method for the numercal calculaton of nductances among conductors
More informationTopic 23 - Randomized Complete Block Designs (RCBD)
Topc 3 ANOVA (III) 3-1 Topc 3 - Randomzed Complete Block Desgns (RCBD) Defn: A Randomzed Complete Block Desgn s a varant of the completely randomzed desgn (CRD) that we recently learned. In ths desgn,
More informationChapter 3 Describing Data Using Numerical Measures
Chapter 3 Student Lecture Notes 3-1 Chapter 3 Descrbng Data Usng Numercal Measures Fall 2006 Fundamentals of Busness Statstcs 1 Chapter Goals To establsh the usefulness of summary measures of data. The
More informationTHEORY OF GENETIC ALGORITHMS WITH α-selection. André Neubauer
THEORY OF GENETIC ALGORITHMS WITH α-selection André Neubauer Informaton Processng Systems Lab Münster Unversty of Appled Scences Stegerwaldstraße 39, D-48565 Stenfurt, Germany Emal: andre.neubauer@fh-muenster.de
More informationUNIVERSITY OF TORONTO Faculty of Arts and Science. December 2005 Examinations STA437H1F/STA1005HF. Duration - 3 hours
UNIVERSITY OF TORONTO Faculty of Arts and Scence December 005 Examnatons STA47HF/STA005HF Duraton - hours AIDS ALLOWED: (to be suppled by the student) Non-programmable calculator One handwrtten 8.5'' x
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationComputation of Higher Order Moments from Two Multinomial Overdispersion Likelihood Models
Computaton of Hgher Order Moments from Two Multnomal Overdsperson Lkelhood Models BY J. T. NEWCOMER, N. K. NEERCHAL Department of Mathematcs and Statstcs, Unversty of Maryland, Baltmore County, Baltmore,
More informationThis model contains two bonds per unit cell (one along the x-direction and the other along y). So we can rewrite the Hamiltonian as:
1 Problem set #1 1.1. A one-band model on a square lattce Fg. 1 Consder a square lattce wth only nearest-neghbor hoppngs (as shown n the fgure above): H t, j a a j (1.1) where,j stands for nearest neghbors
More informationTHE ROBUSTNESS OF GENETIC ALGORITHMS IN SOLVING UNCONSTRAINED BUILDING OPTIMIZATION PROBLEMS
Nnth Internatonal IBPSA Conference Montréal, Canada August 5-8, 2005 THE ROBUSTNESS OF GENETIC ALGORITHMS IN SOLVING UNCONSTRAINED BUILDING OPTIMIZATION PROBLEMS Jonathan Wrght, and Al Alajm Department
More informationEnsemble Methods: Boosting
Ensemble Methods: Boostng Ncholas Ruozz Unversty of Texas at Dallas Based on the sldes of Vbhav Gogate and Rob Schapre Last Tme Varance reducton va baggng Generate new tranng data sets by samplng wth replacement
More informationUsing T.O.M to Estimate Parameter of distributions that have not Single Exponential Family
IOSR Journal of Mathematcs IOSR-JM) ISSN: 2278-5728. Volume 3, Issue 3 Sep-Oct. 202), PP 44-48 www.osrjournals.org Usng T.O.M to Estmate Parameter of dstrbutons that have not Sngle Exponental Famly Jubran
More informationUsing the estimated penetrances to determine the range of the underlying genetic model in casecontrol
Georgetown Unversty From the SelectedWorks of Mark J Meyer 8 Usng the estmated penetrances to determne the range of the underlyng genetc model n casecontrol desgn Mark J Meyer Neal Jeffres Gang Zheng Avalable
More informationQuantifying Uncertainty
Partcle Flters Quantfyng Uncertanty Sa Ravela M. I. T Last Updated: Sprng 2013 1 Quantfyng Uncertanty Partcle Flters Partcle Flters Appled to Sequental flterng problems Can also be appled to smoothng problems
More informationSupporting Information
Supportng Informaton The neural network f n Eq. 1 s gven by: f x l = ReLU W atom x l + b atom, 2 where ReLU s the element-wse rectfed lnear unt, 21.e., ReLUx = max0, x, W atom R d d s the weght matrx to
More informationCHALMERS, GÖTEBORGS UNIVERSITET. SOLUTIONS to RE-EXAM for ARTIFICIAL NEURAL NETWORKS. COURSE CODES: FFR 135, FIM 720 GU, PhD
CHALMERS, GÖTEBORGS UNIVERSITET SOLUTIONS to RE-EXAM for ARTIFICIAL NEURAL NETWORKS COURSE CODES: FFR 35, FIM 72 GU, PhD Tme: Place: Teachers: Allowed materal: Not allowed: January 2, 28, at 8 3 2 3 SB
More informationECE 534: Elements of Information Theory. Solutions to Midterm Exam (Spring 2006)
ECE 534: Elements of Informaton Theory Solutons to Mdterm Eam (Sprng 6) Problem [ pts.] A dscrete memoryless source has an alphabet of three letters,, =,, 3, wth probabltes.4,.4, and., respectvely. (a)
More informationSome modelling aspects for the Matlab implementation of MMA
Some modellng aspects for the Matlab mplementaton of MMA Krster Svanberg krlle@math.kth.se Optmzaton and Systems Theory Department of Mathematcs KTH, SE 10044 Stockholm September 2004 1. Consdered optmzaton
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationDr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur
Analyss of Varance and Desgn of Experment-I MODULE VIII LECTURE - 34 ANALYSIS OF VARIANCE IN RANDOM-EFFECTS MODEL AND MIXED-EFFECTS EFFECTS MODEL Dr Shalabh Department of Mathematcs and Statstcs Indan
More informationPHYS 705: Classical Mechanics. Calculus of Variations II
1 PHYS 705: Classcal Mechancs Calculus of Varatons II 2 Calculus of Varatons: Generalzaton (no constrant yet) Suppose now that F depends on several dependent varables : We need to fnd such that has a statonary
More informationA PROBABILITY-DRIVEN SEARCH ALGORITHM FOR SOLVING MULTI-OBJECTIVE OPTIMIZATION PROBLEMS
HCMC Unversty of Pedagogy Thong Nguyen Huu et al. A PROBABILITY-DRIVEN SEARCH ALGORITHM FOR SOLVING MULTI-OBJECTIVE OPTIMIZATION PROBLEMS Thong Nguyen Huu and Hao Tran Van Department of mathematcs-nformaton,
More informationSupplementary material: Margin based PU Learning. Matrix Concentration Inequalities
Supplementary materal: Margn based PU Learnng We gve the complete proofs of Theorem and n Secton We frst ntroduce the well-known concentraton nequalty, so the covarance estmator can be bounded Then we
More informationECE559VV Project Report
ECE559VV Project Report (Supplementary Notes Loc Xuan Bu I. MAX SUM-RATE SCHEDULING: THE UPLINK CASE We have seen (n the presentaton that, for downlnk (broadcast channels, the strategy maxmzng the sum-rate
More informationModule 2. Random Processes. Version 2 ECE IIT, Kharagpur
Module Random Processes Lesson 6 Functons of Random Varables After readng ths lesson, ou wll learn about cdf of functon of a random varable. Formula for determnng the pdf of a random varable. Let, X be
More informationChapter 13: Multiple Regression
Chapter 13: Multple Regresson 13.1 Developng the multple-regresson Model The general model can be descrbed as: It smplfes for two ndependent varables: The sample ft parameter b 0, b 1, and b are used to
More informationLecture 4. Instructor: Haipeng Luo
Lecture 4 Instructor: Hapeng Luo In the followng lectures, we focus on the expert problem and study more adaptve algorthms. Although Hedge s proven to be worst-case optmal, one may wonder how well t would
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationSimulation and Random Number Generation
Smulaton and Random Number Generaton Summary Dscrete Tme vs Dscrete Event Smulaton Random number generaton Generatng a random sequence Generatng random varates from a Unform dstrbuton Testng the qualty
More informationLecture 7: Boltzmann distribution & Thermodynamics of mixing
Prof. Tbbtt Lecture 7 etworks & Gels Lecture 7: Boltzmann dstrbuton & Thermodynamcs of mxng 1 Suggested readng Prof. Mark W. Tbbtt ETH Zürch 13 März 018 Molecular Drvng Forces Dll and Bromberg: Chapters
More informationXII.3 The EM (Expectation-Maximization) Algorithm
XII.3 The EM (Expectaton-Maxzaton) Algorth Toshnor Munaata 3/7/06 The EM algorth s a technque to deal wth varous types of ncoplete data or hdden varables. It can be appled to a wde range of learnng probles
More information4 Analysis of Variance (ANOVA) 5 ANOVA. 5.1 Introduction. 5.2 Fixed Effects ANOVA
4 Analyss of Varance (ANOVA) 5 ANOVA 51 Introducton ANOVA ANOVA s a way to estmate and test the means of multple populatons We wll start wth one-way ANOVA If the populatons ncluded n the study are selected
More informationANOVA. The Observations y ij
ANOVA Stands for ANalyss Of VArance But t s a test of dfferences n means The dea: The Observatons y j Treatment group = 1 = 2 = k y 11 y 21 y k,1 y 12 y 22 y k,2 y 1, n1 y 2, n2 y k, nk means: m 1 m 2
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationProbability Theory (revisited)
Probablty Theory (revsted) Summary Probablty v.s. plausblty Random varables Smulaton of Random Experments Challenge The alarm of a shop rang. Soon afterwards, a man was seen runnng n the street, persecuted
More informationNumerical Heat and Mass Transfer
Master degree n Mechancal Engneerng Numercal Heat and Mass Transfer 06-Fnte-Dfference Method (One-dmensonal, steady state heat conducton) Fausto Arpno f.arpno@uncas.t Introducton Why we use models and
More informationStatistical Inference. 2.3 Summary Statistics Measures of Center and Spread. parameters ( population characteristics )
Ismor Fscher, 8//008 Stat 54 / -8.3 Summary Statstcs Measures of Center and Spread Dstrbuton of dscrete contnuous POPULATION Random Varable, numercal True center =??? True spread =???? parameters ( populaton
More informationA HYBRID DIFFERENTIAL EVOLUTION -ITERATIVE GREEDY SEARCH ALGORITHM FOR CAPACITATED VEHICLE ROUTING PROBLEM
IJCMA: Vol. 6, No. 1, January-June 2012, pp. 1-19 Global Research Publcatons A HYBRID DIFFERENTIAL EVOLUTION -ITERATIVE GREEDY SEARCH ALGORITHM FOR CAPACITATED VEHICLE ROUTING PROBLEM S. Kavtha and Nrmala
More informationVQ widely used in coding speech, image, and video
at Scalar quantzers are specal cases of vector quantzers (VQ): they are constraned to look at one sample at a tme (memoryless) VQ does not have such constrant better RD perfomance expected Source codng
More information