Steepest descent minimization applied to a simple seismic inversion

Size: px
Start display at page:

Download "Steepest descent minimization applied to a simple seismic inversion"

Transcription

1 Steepest descent minimization applied to a simple seismic inversion Sylvain Barbot Friday, December 10, Problem Definition The problem one tries to address is the estimation of the location of a seismic wave source. One considers the seismic wave to propagate straight from the epicenter to the seismograph without reflection. Besides, one knows the wave propagation velocity v. And the relation 1 holds: (x xi )2 + (y y i )2 t i = v Equation 1 is called the direct model: (1) d = g(m) One has a set of time of wave front arrival measured from different seismographs scattered on the surface (the problem has been reduced to a two-dimensional space). One writes t i the time of arrival measured from the seismograph i d = C d = Where d is data column matrix and C d the corresponding covariance matrix. One finally writes the a priori information on model parameters m prior and its covariance matrix C m. These two last objects are necessary for the gradient formulation but as no real information is available on the seismic source, one defines arbitrarily the seismic source in the middle of the seismograph but with a high uncertainty (the source might be around a 10km circle): m prior = [ ] C m = We first draw the posterior probability density [ ] (2) (3) σ m (m) = exp ( S(m)) (4) 1

2 Figure 1: Probability Density σ m. Left: x and y vary from 0 to 20 km; Right: a zoom, x and y vary from 14 to 16 km. where 2 S(m) = (g(m) d obs ) T C 1 d This σ m as represented in figure 1. (g(m) d obs )+(m m prior ) T C 1 m (m m prior ) (5) 2

3 2 Implementation and Results 2.1 Steepest Descent Algorithm OOne then implements the steepest descent algorithm to minimize S(m). The steepest ascent vector γ is defined by relation 6: γ k = C m G T k C 1 d (g(m k) d obs ) + (m k m prior ).... G k = δf i δf i δx δy.. (6) And we use the following algorithm which converges towards the best estimation in the least square sense fitting the data parameters with respect to our direct model expression: m k+1 = m k εγ k (7) Finally, the a posteriori covariance matrix holds: C m = (G T k C 1 d G k + C 1 m ) 1 (8) One has implemented the steepest descent algorithm on IDL to find numerically the solution of this simple seismic inversion and to display the probability density shape. One has chosen ε arbitrarily with the trial and error method. Values close to 1 fail because too large to allow the convergence: we have chosen the value because we have verified in a first computation that the steepest ascent vector has components of the order of 10 4, and we want our jumps to be of the order of one km. Should the value have not ensures convergence, we would have taken half the value. After iterations, the computer answers: Least Squares Solution: Covariance Matrix:

4 2.2 Quasi-Newton Algorithm One has then implemented the quasi-newton algorithm. The difference holding in the calculation of the hessian matrix ξ k at each iteration: The algorithm becomes: ξ k = (G T k C 1 D G k + C 1 D ) 1 (9) γ k = G T k C 1 d (g(m k) d obs ) + C 1 M (m k m prior ) (10) m k+1 = m k ξ k γ k After 5 iterations, the programs displays: Least Squares Solution: Covariance Matrix:

5 3 Appendix FUNCTION seismic_proba_density d_obs = TRANSPOSE([3.12, 3.26, 2.98, 3.12, 2.84, 2.98]) data_cov = DIAG_MATRIX(REPLICATE(.1^2, 6)) m_prior = TRANSPOSE([4, 15.5]) prior_cov = DIAG_MATRIX(REPLICATE(10.^2, 2)) S = FLTARR(100, 100) FOR l = 0, DO BEGIN y = l / 5. FOR c = 0, DO BEGIN x = c / 5. S[c, l] = 1./2. * (TRANSPOSE(seismic_direct([x, y]) - d_obs) ## $ INVERT(data_cov) ## (seismic_direct([x, y]) - d_obs)) S[c, l]+= 1./2. * (TRANSPOSE([[x], [y]] - m_prior) ## $ INVERT(prior_cov) ## ([[x], [y]] - m_prior)) ENDFOR ENDFOR distribution = EXP(- S) END Table 1: IDL code calculating the probability density ρ M = exp ( S(m)) 5

6 PRO seismic_inversion iteration = & eps = 1e-5 d_obs = TRANSPOSE([3.12, 3.26, 2.98, 3.12, 2.84, 2.98]) data_cov = DIAG_MATRIX(REPLICATE(.1^2, 6)) m_prior = TRANSPOSE([10, 2]) prior_cov = [[1d, 0d], [0, 100d^2]] Mk = m_prior FOR k = 0, iteration - 1 DO BEGIN B = seismic_direct(mk) - d_obs LUDC, data_cov, index matrice = TRANSPOSE(LUSOL(data_cov, index, TRANSPOSE(B))) Gk = seismic_gradient(mk) vector = prior_cov ## TRANSPOSE(Gk) ## matrice + (Mk - m_prior) mk = mk - eps * vector ENDFOR post_cov = INVERT(TRANSPOSE(Gk) ## $ INVERT(data_cov) ## Gk + INVERT(data_cov)) PRINT, Least Squares Solution:, mk PRINT, Covariance Matrix:, post_cov Table 2: Procedure running the steepest descent algorithm core. 6

7 FUNCTION seismic_direct, m x = [3., 3., 4., 4., 5., 5.] y = [15., 16., 15., 16., 15., 16.] v = 5 RETURN, TRANSPOSE(SQRT((m[0] - x)^2 + (m[1] - y)^2) / v) END FUNCTION seismic_gradient, m x = [3., 3., 4., 4., 5., 5.] y = [15., 16., 15., 16., 15., 16.] v = 5 result = [[(m[0] - x) / (v * SQRT((m[0] - x)^2 + (m[1] - y)^2))],$ [(m[1] - y) / (v * SQRT((m[0] - x)^2 + (m[1] - y)^2))]] RETURN, TRANSPOSE(result) END Table 3: Ancillary functions calculating the direct model and the gradient matrix. 7

8 PRO seismic_inversion_quasi_newton iteration = 5 d_obs = TRANSPOSE([3.12, 3.26, 2.98, 3.12, 2.84, 2.98]) data_cov = DIAG_MATRIX(REPLICATE(.1^2, 6)) m_prior = TRANSPOSE([10, 2]) prior_cov = [[1d, 0d], [0, 100d^2]] Mk = m_prior FOR k = 0, iteration - 1 DO BEGIN B = seismic_direct(mk) - d_obs LUDC, data_cov, index matrice = TRANSPOSE(LUSOL(data_cov, index, TRANSPOSE(B))) Gk = seismic_gradient(mk) hessian = INVERT(TRANSPOSE(Gk) ## idata_cov ## Gk + iprior_cov) vector = TRANSPOSE(Gk) ## matrice + iprior_cov ## (Mk - m_prior) mk = mk - hessian ## vector ENDFOR PRINT, Least Squares Solution:, mk PRINT, Covariance Matrix:, hessian Table 4: Procedure running the Quasi-Newton algorithm 8

Problem Set 5: Iterative methods with generalized least squares

Problem Set 5: Iterative methods with generalized least squares Problem Set 5: Iterative methods with generalized least squares GEOS 627: Inverse Problems and Parameter Estimation, Carl Tape Assigned: February 20, 2017 Due: March 6, 2017 Last compiled: July 7, 2017

More information

nonrobust estimation The n measurement vectors taken together give the vector X R N. The unknown parameter vector is P R M.

nonrobust estimation The n measurement vectors taken together give the vector X R N. The unknown parameter vector is P R M. Introduction to nonlinear LS estimation R. I. Hartley and A. Zisserman: Multiple View Geometry in Computer Vision. Cambridge University Press, 2ed., 2004. After Chapter 5 and Appendix 6. We will use x

More information

Application of new Monte Carlo method for inversion of prestack seismic data. Yang Xue Advisor: Dr. Mrinal K. Sen

Application of new Monte Carlo method for inversion of prestack seismic data. Yang Xue Advisor: Dr. Mrinal K. Sen Application of new Monte Carlo method for inversion of prestack seismic data Yang Xue Advisor: Dr. Mrinal K. Sen Overview Motivation Introduction Bayes theorem Stochastic inference methods Methodology

More information

ECE 680 Modern Automatic Control. Gradient and Newton s Methods A Review

ECE 680 Modern Automatic Control. Gradient and Newton s Methods A Review ECE 680Modern Automatic Control p. 1/1 ECE 680 Modern Automatic Control Gradient and Newton s Methods A Review Stan Żak October 25, 2011 ECE 680Modern Automatic Control p. 2/1 Review of the Gradient Properties

More information

The Inversion Problem: solving parameters inversion and assimilation problems

The Inversion Problem: solving parameters inversion and assimilation problems The Inversion Problem: solving parameters inversion and assimilation problems UE Numerical Methods Workshop Romain Brossier romain.brossier@univ-grenoble-alpes.fr ISTerre, Univ. Grenoble Alpes Master 08/09/2016

More information

Course on Inverse Problems

Course on Inverse Problems California Institute of Technology Division of Geological and Planetary Sciences March 26 - May 25, 2007 Course on Inverse Problems Albert Tarantola Institut de Physique du Globe de Paris Lesson XVI CONCLUSION

More information

Course on Inverse Problems Albert Tarantola

Course on Inverse Problems Albert Tarantola California Institute of Technology Division of Geological and Planetary Sciences March 26 - May 25, 27 Course on Inverse Problems Albert Tarantola Institut de Physique du Globe de Paris CONCLUSION OF THE

More information

LAB. Balboa Heights, Panama. Boulder, Colorado. Mexico City, Mexico. Data Table. Difference Between P-wave and S-wave. S-wave Arrival Time

LAB. Balboa Heights, Panama. Boulder, Colorado. Mexico City, Mexico. Data Table. Difference Between P-wave and S-wave. S-wave Arrival Time Name: Date: Lab Period: Locating the Epicenter of an Earthquake Introduction: The epicenter is the point on Earth's surface directly above an earthquake. Seismic stations detect earthquakes by the tracings

More information

Numerical Optimization Algorithms

Numerical Optimization Algorithms Numerical Optimization Algorithms 1. Overview. Calculus of Variations 3. Linearized Supersonic Flow 4. Steepest Descent 5. Smoothed Steepest Descent Overview 1 Two Main Categories of Optimization Algorithms

More information

Multivariate Newton Minimanization

Multivariate Newton Minimanization Multivariate Newton Minimanization Optymalizacja syntezy biosurfaktantu Rhamnolipid Rhamnolipids are naturally occuring glycolipid produced commercially by the Pseudomonas aeruginosa species of bacteria.

More information

Optimization method examples in Matlab Carl Tape, Qinya Liu, Albert Tarantola July 2, The forward problem 3

Optimization method examples in Matlab Carl Tape, Qinya Liu, Albert Tarantola July 2, The forward problem 3 Optimization method examples in Matlab Carl Tape, Qinya Liu, Albert Tarantola July 2, 27 This file : /home/carltape/latex/notes/tromp/optimization_guide.tex Matlab code : /opt/seismo-util/source/optim_matlab/optimization.m

More information

Note that, when perturbing c(x) instead of m(x), an additional Taylor approximation is necessary: 1 c 2 (x) 1. c 2 0(x) 2εc 1(x).

Note that, when perturbing c(x) instead of m(x), an additional Taylor approximation is necessary: 1 c 2 (x) 1. c 2 0(x) 2εc 1(x). Chapter 3 Scattering series In this chapter we describe the nonlinearity of the map c u in terms of a perturbation (Taylor) series. To first order, the linearization of this map is called the Born approximation.

More information

Quasi-Newton Methods

Quasi-Newton Methods Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications

More information

Optimization II: Unconstrained Multivariable

Optimization II: Unconstrained Multivariable Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1

More information

MA/OR/ST 706: Nonlinear Programming Midterm Exam Instructor: Dr. Kartik Sivaramakrishnan INSTRUCTIONS

MA/OR/ST 706: Nonlinear Programming Midterm Exam Instructor: Dr. Kartik Sivaramakrishnan INSTRUCTIONS MA/OR/ST 706: Nonlinear Programming Midterm Exam Instructor: Dr. Kartik Sivaramakrishnan INSTRUCTIONS 1. Please write your name and student number clearly on the front page of the exam. 2. The exam is

More information

Optimization II: Unconstrained Multivariable

Optimization II: Unconstrained Multivariable Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization II: Unconstrained

More information

Zero-Order Methods for the Optimization of Noisy Functions. Jorge Nocedal

Zero-Order Methods for the Optimization of Noisy Functions. Jorge Nocedal Zero-Order Methods for the Optimization of Noisy Functions Jorge Nocedal Northwestern University Simons Institute, October 2017 1 Collaborators Albert Berahas Northwestern University Richard Byrd University

More information

4. DATA ASSIMILATION FUNDAMENTALS

4. DATA ASSIMILATION FUNDAMENTALS 4. DATA ASSIMILATION FUNDAMENTALS... [the atmosphere] "is a chaotic system in which errors introduced into the system can grow with time... As a consequence, data assimilation is a struggle between chaotic

More information

Methods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent

Methods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived

More information

Logistic Regression. Seungjin Choi

Logistic Regression. Seungjin Choi Logistic Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/

More information

Augmented Reality VU numerical optimization algorithms. Prof. Vincent Lepetit

Augmented Reality VU numerical optimization algorithms. Prof. Vincent Lepetit Augmented Reality VU numerical optimization algorithms Prof. Vincent Lepetit P3P: can exploit only 3 correspondences; DLT: difficult to exploit the knowledge of the internal parameters correctly, and the

More information

Nonlinear Optimization: What s important?

Nonlinear Optimization: What s important? Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

More information

Optimization. Totally not complete this is...don't use it yet...

Optimization. Totally not complete this is...don't use it yet... Optimization Totally not complete this is...don't use it yet... Bisection? Doing a root method is akin to doing a optimization method, but bi-section would not be an effective method - can detect sign

More information

FALL 2018 MATH 4211/6211 Optimization Homework 4

FALL 2018 MATH 4211/6211 Optimization Homework 4 FALL 2018 MATH 4211/6211 Optimization Homework 4 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution

More information

Approximate Message Passing

Approximate Message Passing Approximate Message Passing Mohammad Emtiyaz Khan CS, UBC February 8, 2012 Abstract In this note, I summarize Sections 5.1 and 5.2 of Arian Maleki s PhD thesis. 1 Notation We denote scalars by small letters

More information

GEOPHYSICAL INVERSE THEORY AND REGULARIZATION PROBLEMS

GEOPHYSICAL INVERSE THEORY AND REGULARIZATION PROBLEMS Methods in Geochemistry and Geophysics, 36 GEOPHYSICAL INVERSE THEORY AND REGULARIZATION PROBLEMS Michael S. ZHDANOV University of Utah Salt Lake City UTAH, U.S.A. 2OO2 ELSEVIER Amsterdam - Boston - London

More information

Lab Activity Locating Epicenters

Lab Activity Locating Epicenters Lab Activity Locating Epicenters Key Features: Analyzing Seismograms P-waves & S-waves Using ESRT page 11 Drawing Epicenter Circles By Kaitlin Coder https://www.teacherspayteachers.com/store/nys-earth-science-and-living-environment-regents

More information

Vasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks

Vasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks C.M. Bishop s PRML: Chapter 5; Neural Networks Introduction The aim is, as before, to find useful decompositions of the target variable; t(x) = y(x, w) + ɛ(x) (3.7) t(x n ) and x n are the observations,

More information

ECE580 Exam 1 October 4, Please do not write on the back of the exam pages. Extra paper is available from the instructor.

ECE580 Exam 1 October 4, Please do not write on the back of the exam pages. Extra paper is available from the instructor. ECE580 Exam 1 October 4, 2012 1 Name: Solution Score: /100 You must show ALL of your work for full credit. This exam is closed-book. Calculators may NOT be used. Please leave fractions as fractions, etc.

More information

5 Quasi-Newton Methods

5 Quasi-Newton Methods Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min

More information

Regression with Numerical Optimization. Logistic

Regression with Numerical Optimization. Logistic CSG220 Machine Learning Fall 2008 Regression with Numerical Optimization. Logistic regression Regression with Numerical Optimization. Logistic regression based on a document by Andrew Ng October 3, 204

More information

Nonlinear equations and optimization

Nonlinear equations and optimization Notes for 2017-03-29 Nonlinear equations and optimization For the next month or so, we will be discussing methods for solving nonlinear systems of equations and multivariate optimization problems. We will

More information

1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:

1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by: Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion

More information

Optimization Methods for Machine Learning

Optimization Methods for Machine Learning Optimization Methods for Machine Learning Sathiya Keerthi Microsoft Talks given at UC Santa Cruz February 21-23, 2017 The slides for the talks will be made available at: http://www.keerthis.com/ Introduction

More information

CoE 3SK3 Computer Aided Engineering Tutorial: Unconstrained Optimization

CoE 3SK3 Computer Aided Engineering Tutorial: Unconstrained Optimization CoE 3SK3 Computer Aided Engineering Tutorial: Unconstrained Optimization Jie Cao caoj23@grads.ece.mcmaster.ca Department of Electrical and Computer Engineering McMaster University Feb. 2, 2010 Outline

More information

, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are

, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues

More information

Truncated Conjugate Gradient Method for History Matching in Reservoir Simulation

Truncated Conjugate Gradient Method for History Matching in Reservoir Simulation Trabalho apresentado no CNAC, Gramado - RS, 2016. Proceeding Series of the Brazilian Society of Computational and Applied athematics Truncated Conjugate Gradient ethod for History atching in Reservoir

More information

Uncertainty quantification for inverse problems with a weak wave-equation constraint

Uncertainty quantification for inverse problems with a weak wave-equation constraint Uncertainty quantification for inverse problems with a weak wave-equation constraint Zhilong Fang*, Curt Da Silva*, Rachel Kuske** and Felix J. Herrmann* *Seismic Laboratory for Imaging and Modeling (SLIM),

More information

Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems

Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems International Journal of Scientific and Research Publications, Volume 3, Issue 10, October 013 1 ISSN 50-3153 Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming

More information

MATH 4211/6211 Optimization Quasi-Newton Method

MATH 4211/6211 Optimization Quasi-Newton Method MATH 4211/6211 Optimization Quasi-Newton Method Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 Quasi-Newton Method Motivation:

More information

σ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =

σ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) = Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,

More information

Uncertainty quantification for Wavefield Reconstruction Inversion

Uncertainty quantification for Wavefield Reconstruction Inversion Uncertainty quantification for Wavefield Reconstruction Inversion Zhilong Fang *, Chia Ying Lee, Curt Da Silva *, Felix J. Herrmann *, and Rachel Kuske * Seismic Laboratory for Imaging and Modeling (SLIM),

More information

Strong Lens Modeling (II): Statistical Methods

Strong Lens Modeling (II): Statistical Methods Strong Lens Modeling (II): Statistical Methods Chuck Keeton Rutgers, the State University of New Jersey Probability theory multiple random variables, a and b joint distribution p(a, b) conditional distribution

More information

Data Fitting and Uncertainty

Data Fitting and Uncertainty TiloStrutz Data Fitting and Uncertainty A practical introduction to weighted least squares and beyond With 124 figures, 23 tables and 71 test questions and examples VIEWEG+ TEUBNER IX Contents I Framework

More information

Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23

Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23 Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is

More information

Nonlinear Least Squares Curve Fitting

Nonlinear Least Squares Curve Fitting Nonlinear Least Squares Curve Fitting Jack Merrin February 12, 2017 1 Summary There is no general solution for the minimization of chi-squared to find the best estimates of the parameters when the model

More information

Chapter 3 Numerical Methods

Chapter 3 Numerical Methods Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2

More information

Epicenter - a point, directly above the true center of disturbance, from which the shock waves of an earthquake apparently radiate.

Epicenter - a point, directly above the true center of disturbance, from which the shock waves of an earthquake apparently radiate. Name: Date: Period: Earthquakes Epicenter - a point, directly above the true center of disturbance, from which the shock waves of an earthquake apparently radiate. Define each of the following terms in

More information

Earthquake information

Earthquake information Name: Date: Block Earthquake information The graph below shows travel time in minutes and distance traveled for primary and secondary waves. Primary and secondary waves start at the same time but do not

More information

Trajectory-based optimization

Trajectory-based optimization Trajectory-based optimization Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 2012 Emo Todorov (UW) AMATH/CSE 579, Winter 2012 Lecture 6 1 / 13 Using

More information

K/U /39 T/I /50 C /102 A

K/U /39 T/I /50 C /102 A Name: Partner: K/U /39 T/I /50 C /102 A Purpose: What is the relationship between the magnitude of the force causing the acceleration and the frequency of revolution of an object in uniform circular motion?

More information

A Robot that Learns an Evaluation Function for Acquiring of Appropriate Motions

A Robot that Learns an Evaluation Function for Acquiring of Appropriate Motions A Robot that Learns an Evaluation Function for Acquiring of Appropriate Motions Katsunari Shibata and Yoich Okabe Research Center for Advanced Science and Technology, Univ. of Tokyo -6-1 Komaba, Meguro-ku,

More information

Introduction. New Nonsmooth Trust Region Method for Unconstraint Locally Lipschitz Optimization Problems

Introduction. New Nonsmooth Trust Region Method for Unconstraint Locally Lipschitz Optimization Problems New Nonsmooth Trust Region Method for Unconstraint Locally Lipschitz Optimization Problems Z. Akbari 1, R. Yousefpour 2, M. R. Peyghami 3 1 Department of Mathematics, K.N. Toosi University of Technology,

More information

Machine Learning Brett Bernstein. Recitation 1: Gradients and Directional Derivatives

Machine Learning Brett Bernstein. Recitation 1: Gradients and Directional Derivatives Machine Learning Brett Bernstein Recitation 1: Gradients and Directional Derivatives Intro Question 1 We are given the data set (x 1, y 1 ),, (x n, y n ) where x i R d and y i R We want to fit a linear

More information

Quantitative Information Flow. Lecture 7

Quantitative Information Flow. Lecture 7 Quantitative Information Flow Lecture 7 1 The basic model: Systems = Information-Theoretic channels Secret Information Observables s1 o1... System... sm on Input Output 2 Probabilistic systems are noisy

More information

Machine Learning for Signal Processing Expectation Maximization Mixture Models. Bhiksha Raj 27 Oct /

Machine Learning for Signal Processing Expectation Maximization Mixture Models. Bhiksha Raj 27 Oct / Machine Learning for Signal rocessing Expectation Maximization Mixture Models Bhiksha Raj 27 Oct 2016 11755/18797 1 Learning Distributions for Data roblem: Given a collection of examples from some data,

More information

Chapter 9. Derivatives. Josef Leydold Mathematical Methods WS 2018/19 9 Derivatives 1 / 51. f x. (x 0, f (x 0 ))

Chapter 9. Derivatives. Josef Leydold Mathematical Methods WS 2018/19 9 Derivatives 1 / 51. f x. (x 0, f (x 0 )) Chapter 9 Derivatives Josef Leydold Mathematical Methods WS 208/9 9 Derivatives / 5 Difference Quotient Let f : R R be some function. The the ratio f = f ( 0 + ) f ( 0 ) = f ( 0) 0 is called difference

More information

Engineering Part IIB: Module 4F10 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers

Engineering Part IIB: Module 4F10 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers Engineering Part IIB: Module 4F0 Statistical Pattern Processing Lecture 5: Single Layer Perceptrons & Estimating Linear Classifiers Phil Woodland: pcw@eng.cam.ac.uk Michaelmas 202 Engineering Part IIB:

More information

Convex Optimization CMU-10725

Convex Optimization CMU-10725 Convex Optimization CMU-10725 Quasi Newton Methods Barnabás Póczos & Ryan Tibshirani Quasi Newton Methods 2 Outline Modified Newton Method Rank one correction of the inverse Rank two correction of the

More information

Multi-layer Neural Networks

Multi-layer Neural Networks Multi-layer Neural Networks Steve Renals Informatics 2B Learning and Data Lecture 13 8 March 2011 Informatics 2B: Learning and Data Lecture 13 Multi-layer Neural Networks 1 Overview Multi-layer neural

More information

Approximate- vs. full-hessian in FWI: 1D analytical and numerical experiments

Approximate- vs. full-hessian in FWI: 1D analytical and numerical experiments Approximate- vs. full-hessian in FWI: 1D analytical and numerical experiments Raul Cova and Kris Innanen ABSTRACT Feasibility of using Full Waveform Inversion (FWI) to build velocity models has been increasing

More information

Bayesian Computation in Color-Magnitude Diagrams

Bayesian Computation in Color-Magnitude Diagrams Bayesian Computation in Color-Magnitude Diagrams SA, AA, PT, EE, MCMC and ASIS in CMDs Paul Baines Department of Statistics Harvard University October 19, 2009 Overview Motivation and Introduction Modelling

More information

Optimization Methods

Optimization Methods Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available

More information

Variational data assimilation

Variational data assimilation Background and methods NCEO, Dept. of Meteorology, Univ. of Reading 710 March 2018, Univ. of Reading Bayes' Theorem Bayes' Theorem p(x y) = posterior distribution = p(x) p(y x) p(y) prior distribution

More information

Training Schro dinger s Cat

Training Schro dinger s Cat Training Schro dinger s Cat Quadratically Converging algorithms for Optimal Control of Quantum Systems David L. Goodwin & Ilya Kuprov d.goodwin@soton.ac.uk comp-chem@southampton, Wednesday 15th July 2015

More information

4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion:

4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: Unconstrained Convex Optimization 21 4 Newton Method H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: f(x + p) f(x)+p T f(x)+ 1 2 pt H(x)p ˆf(p) In general, ˆf(p) won

More information

Lecture 10. Neural networks and optimization. Machine Learning and Data Mining November Nando de Freitas UBC. Nonlinear Supervised Learning

Lecture 10. Neural networks and optimization. Machine Learning and Data Mining November Nando de Freitas UBC. Nonlinear Supervised Learning Lecture 0 Neural networks and optimization Machine Learning and Data Mining November 2009 UBC Gradient Searching for a good solution can be interpreted as looking for a minimum of some error (loss) function

More information

HOMEWORK #4: LOGISTIC REGRESSION

HOMEWORK #4: LOGISTIC REGRESSION HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2019 Due: 11am Monday, February 25th, 2019 Submit scan of plots/written responses to Gradebook; submit your

More information

Adaptive Filter Theory

Adaptive Filter Theory 0 Adaptive Filter heory Sung Ho Cho Hanyang University Seoul, Korea (Office) +8--0-0390 (Mobile) +8-10-541-5178 dragon@hanyang.ac.kr able of Contents 1 Wiener Filters Gradient Search by Steepest Descent

More information

How Do Scientists Find the Epicenter of an Earthquake?

How Do Scientists Find the Epicenter of an Earthquake? 3.4 Explore How Do Scientists Find the Epicenter of an Earthquake? Seismograph data says that the earthquake is 100 km (62 mi) away, but at which point on the circle is the earthquake located? EE 116 3.4

More information

Numerical Methods for PDE-Constrained Optimization

Numerical Methods for PDE-Constrained Optimization Numerical Methods for PDE-Constrained Optimization Richard H. Byrd 1 Frank E. Curtis 2 Jorge Nocedal 2 1 University of Colorado at Boulder 2 Northwestern University Courant Institute of Mathematical Sciences,

More information

Midterm. Introduction to Machine Learning. CS 189 Spring Please do not open the exam before you are instructed to do so.

Midterm. Introduction to Machine Learning. CS 189 Spring Please do not open the exam before you are instructed to do so. CS 89 Spring 07 Introduction to Machine Learning Midterm Please do not open the exam before you are instructed to do so. The exam is closed book, closed notes except your one-page cheat sheet. Electronic

More information

Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr

Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters

More information

NonlinearOptimization

NonlinearOptimization 1/35 NonlinearOptimization Pavel Kordík Department of Computer Systems Faculty of Information Technology Czech Technical University in Prague Jiří Kašpar, Pavel Tvrdík, 2011 Unconstrained nonlinear optimization,

More information

Programming, numerics and optimization

Programming, numerics and optimization Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428

More information

Regression. Oscar García

Regression. Oscar García Regression Oscar García Regression methods are fundamental in Forest Mensuration For a more concise and general presentation, we shall first review some matrix concepts 1 Matrices An order n m matrix is

More information

Managing Uncertainty

Managing Uncertainty Managing Uncertainty Bayesian Linear Regression and Kalman Filter December 4, 2017 Objectives The goal of this lab is multiple: 1. First it is a reminder of some central elementary notions of Bayesian

More information

Numerical Methods I Solving Nonlinear Equations

Numerical Methods I Solving Nonlinear Equations Numerical Methods I Solving Nonlinear Equations Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 16th, 2014 A. Donev (Courant Institute)

More information

Gradient Descent. Dr. Xiaowei Huang

Gradient Descent. Dr. Xiaowei Huang Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,

More information

Optimization. Yuh-Jye Lee. March 21, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 29

Optimization. Yuh-Jye Lee. March 21, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 29 Optimization Yuh-Jye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 21, 2017 1 / 29 You Have Learned (Unconstrained) Optimization in Your High School Let f (x) = ax

More information

Mathematical optimization

Mathematical optimization Optimization Mathematical optimization Determine the best solutions to certain mathematically defined problems that are under constrained determine optimality criteria determine the convergence of the

More information

Iterative linearized migration and inversion

Iterative linearized migration and inversion Stanford Exploration Project, Report 123, October 31, 2005, pages 101 128 Iterative linearized migration and inversion Huazhong Wang 1 ABSTRACT The objective of seismic imaging is to obtain an image of

More information

Parallel Algorithms for Four-Dimensional Variational Data Assimilation

Parallel Algorithms for Four-Dimensional Variational Data Assimilation Parallel Algorithms for Four-Dimensional Variational Data Assimilation Mie Fisher ECMWF October 24, 2011 Mie Fisher (ECMWF) Parallel 4D-Var October 24, 2011 1 / 37 Brief Introduction to 4D-Var Four-Dimensional

More information

2.098/6.255/ Optimization Methods Practice True/False Questions

2.098/6.255/ Optimization Methods Practice True/False Questions 2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence

More information

Introduction to unconstrained optimization - direct search methods

Introduction to unconstrained optimization - direct search methods Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the

More information

8 Numerical methods for unconstrained problems

8 Numerical methods for unconstrained problems 8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields

More information

Lecture 4: Types of errors. Bayesian regression models. Logistic regression

Lecture 4: Types of errors. Bayesian regression models. Logistic regression Lecture 4: Types of errors. Bayesian regression models. Logistic regression A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting more generally COMP-652 and ECSE-68, Lecture

More information

Non-linear least-squares inversion with data-driven

Non-linear least-squares inversion with data-driven Geophys. J. Int. (2000) 142, 000 000 Non-linear least-squares inversion with data-driven Bayesian regularization Tor Erik Rabben and Bjørn Ursin Department of Petroleum Engineering and Applied Geophysics,

More information

Numerical Optimization Prof. Shirish K. Shevade Department of Computer Science and Automation Indian Institute of Science, Bangalore

Numerical Optimization Prof. Shirish K. Shevade Department of Computer Science and Automation Indian Institute of Science, Bangalore Numerical Optimization Prof. Shirish K. Shevade Department of Computer Science and Automation Indian Institute of Science, Bangalore Lecture - 13 Steepest Descent Method Hello, welcome back to this series

More information

SQUAREM. An R package for Accelerating Slowly Convergent Fixed-Point Iterations Including the EM and MM algorithms.

SQUAREM. An R package for Accelerating Slowly Convergent Fixed-Point Iterations Including the EM and MM algorithms. An R package for Accelerating Slowly Convergent Fixed-Point Iterations Including the EM and MM algorithms Ravi 1 1 Division of Geriatric Medicine & Gerontology Johns Hopkins University Baltimore, MD, USA

More information

Constrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.

Constrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method. Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization

More information

Inverse Kinematics. Mike Bailey.

Inverse Kinematics. Mike Bailey. Inverse Kinematics This work is licensed under a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License Mike Bailey mjb@cs.oregonstate.edu inversekinematics.pptx Inverse Kinematics

More information

Numerical Optimization

Numerical Optimization Numerical Optimization Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Spring 2010 Emo Todorov (UW) AMATH/CSE 579, Spring 2010 Lecture 9 1 / 8 Gradient descent

More information

Maximum Likelihood Estimation

Maximum Likelihood Estimation Maximum Likelihood Estimation Guy Lebanon February 19, 2011 Maximum likelihood estimation is the most popular general purpose method for obtaining estimating a distribution from a finite sample. It was

More information

Numerical Optimization: Basic Concepts and Algorithms

Numerical Optimization: Basic Concepts and Algorithms May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some

More information

Optimization Part 1 P. Agius L2.1, Spring 2008

Optimization Part 1 P. Agius L2.1, Spring 2008 Optimization Part 1 Contents Terms and definitions, formulating the problem Unconstrained optimization conditions FONC, SONC, SOSC Searching for the solution Which direction? What step size? Constrained

More information

. α β γ δ ε ζ η θ ι κ λ μ Aμ ν(x) ξ ο π ρ ς σ τ υ φ χ ψ ω. Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω. Friday April 1 ± ǁ

. α β γ δ ε ζ η θ ι κ λ μ Aμ ν(x) ξ ο π ρ ς σ τ υ φ χ ψ ω. Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω. Friday April 1 ± ǁ . α β γ δ ε ζ η θ ι κ λ μ Aμ ν(x) ξ ο π ρ ς σ τ υ φ χ ψ ω. Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω Friday April 1 ± ǁ 1 Chapter 5. Photons: Covariant Theory 5.1. The classical fields 5.2. Covariant

More information

Machine Learning CS 4900/5900. Lecture 03. Razvan C. Bunescu School of Electrical Engineering and Computer Science

Machine Learning CS 4900/5900. Lecture 03. Razvan C. Bunescu School of Electrical Engineering and Computer Science Machine Learning CS 4900/5900 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Machine Learning is Optimization Parametric ML involves minimizing an objective function

More information

ECE580 Exam 2 November 01, Name: Score: / (20 points) You are given a two data sets

ECE580 Exam 2 November 01, Name: Score: / (20 points) You are given a two data sets ECE580 Exam 2 November 01, 2011 1 Name: Score: /100 You must show ALL of your work for full credit. This exam is closed-book. Calculators may NOT be used. Please leave fractions as fractions, etc. I do

More information

From the propagation of uncertainties to the propagation of distributions

From the propagation of uncertainties to the propagation of distributions From the propagation of uncertainties to the propagation of distributions Carlo Carobbi Dipartimento di Elettronica e Telecomunicazioni Università degli Studi di Firenze Istituto di Fisica Applicata "Nello

More information