Recent Advances in SPSA at the Extremes: Adaptive Methods for Smooth Problems and Discrete Methods for Non-Smooth Problems

Size: px
Start display at page:

Download "Recent Advances in SPSA at the Extremes: Adaptive Methods for Smooth Problems and Discrete Methods for Non-Smooth Problems"

Transcription

1 Recent Advances in SPSA at the Extremes: Adaptive Methods for Smooth Problems and Discrete Methods for Non-Smooth Problems SGM2014: Stochastic Gradient Methods IPAM, February 24 28, 2014 James C. Spall The Johns Hopins University Applied Physics Laboratory and Department of Applied Mathematics and Statistics

2 Organization We present two ey extensions to basic simultaneous perturbation stochastic approximation (SPSA) algorithm While SPSA in basic form is not formally a standard stochastic gradient method, it is in same general family of first-order SA methods We present recent results that deal with problems at the extremes in some sense: 1. SPSA for adaptive estimation in smooth problems, where we wish to obtain a Hessian estimate, and 2. SPSA-type ideas in fully discrete problems. Acnowledgment: The discrete wor is joint with Dr. Qi Wang of Johns Hopins University and Barclays 2

3 Extension 1 Adaptive Methods for Smooth Problems 3

4 Bacground Interested in finding θ = θ such that g(θ) = 0 where θ is vector of adjustables Common special case of minimization: find root θ = θ to L( θ) g( θ) = = 0 θ where L(θ) is scalar-valued loss function Assume only (possibly noisy) measurements of g(θ) and/or L(θ) available Noisy measurements arise in areas such as Monte Carlo simulation, real-time control/estimation, machine learning, system identification, etc. Stochastic approximation (SA) widely used for above rootfinding and optimization with noisy measurements, including basic SPSA (Note: Google James Spall NIPS 2012 for video of tutorial) 4

5 Brief Diversion (3 slides): Basic SPSA Algorithm (NIPS 2012 tutorial; Spall, 1987, 1992) Let ĝ Let ˆθ SPSA algorithm has form (θ) denote SP estimate of g(θ) at th iteration denote estimate for θ at th iteration θˆ + 1 = θˆ ag ˆ ˆ ( θ) where {a } is nonnegative gain sequence Generic iterative form above is standard in SA; stochastic analogue to steepest descent Under conditions, ˆθ θ in some stochastic sense as 5

6 Computation of ĝ ( ) (Heart of SPSA) Let be vector of p independent random variables at th iteration = T 1, 2,..., p typically generated by Monte Carlo Let {c } be sequence of positive scalars For iteration +1, tae measurements at design levels: ˆθ ± c ( ± ) y ( θˆ c ) L( θˆ c ) ( + ) + = + +ε = ˆ ( ) +ε y ( θˆ c ) L( θ c ) where ε are measurement noise terms ( ± ) Common special case is when ε = 0 (e.g., system identification with perfect measurements of the lielihood function) 6

7 Computation of The standard SP form for ( ): ĝ g ˆ ( ) (cont d) gˆ ( θˆ ) y ( θˆ + c ) y ( θˆ c ) 2c 1 = y ( θˆ + c ) y ( θˆ c ) 2c p ĝ Note that ( ) only requires two measurements of L( ) independent of p Above SP form contrasts with standard finite-difference approximations taing 2p (or p+1) measurements Intuitive reason why ĝ ( ) is appropriate is that E[ gˆ ( θˆ ) θˆ ] g( θˆ ); formalized in Spall (1987, 1992) 7

8 Bac to Bacground (cont d) Standard SA algorithms (Robbins-Monro, etc.) show 1storder behavior Sharp initial decline (in optimization case) Slow convergence in final phase Sensitivity to units/scaling for elements of θ Long-standing interest in capturing 2nd-order (Newtonlie) effects in SA setting Adaptive SA algorithms Iterate averaging Shortcomings in both of above in implementation difficulty and practical efficiency 8

9 Standard Adaptive SPSA Algorithm Let G (θ) denote direct (noisy) measurement of g(θ) or simultaneous perturbation estimate of g(θ) at th iteration Let ˆθ denote estimate for θ at th iteration Standard adaptive SPSA algorithm has parallel recursions form + = 1 θ ˆ ˆ ˆ 1 θ ah G( θ), = 1 H H + Hˆ where {a } is nonnegative gain sequence and Ĥ is efficient per-iteration Jacobian estimate Convergence of θˆ and H can be shown (Spall, 2000) 9

10 Cost of Implementation For any p, the cost per iteration of adaptive method is Four L measurements or Three g measurements Above costs compare very favorably with previous methods: O(p 2 ) loss measurements per iteration in finite-difference setting (e.g., Fabian, 1971) O(p) g measurements per iteration in root-finding setting (e.g., Ruppert, 1985) 7

11 Enhanced Adaptive SPSA Algorithm (Spall, 2009, IEEE Trans. Auto. Contr.) Standard adaptive algorithm can be improved: Introduce feedbac term that helps remove error in periteration H estimates Optimal weighting to account for noisy measurements of loss function or of g(θ) Enhanced Adaptive SPSA algorithm has same general parallel recursions form: θ ˆ θ ˆ θ ˆ, + = 1 1 ah G 1 ( ) H = (1 w ) H + w Hˆ Ψˆ ( ) Items highlighted in red represent new expressions here: ˆΨ is feedbac term w is optimal weighting 11

12 Enhanced Algorithm (cont d) Recursion for θ is unchanged from basic adaptive method Critical aspect of recursion for H is the averaging of periteration Jacobian estimates Hˆ Each per-iteration H estimate formed by simultaneous perturbation of G (θ) values Per-iteration estimate formed from only two G (θ) values (vs. 2 dim(θ) values in former adaptive methods) Two changes to recursion for H (feedbac and weighting) Feedbac term uses current (cumulative) estimate of H as true H: subtracts out error in Hˆ that depends on true H Weighting accounts for increasing effective noise contribution across iterations Special (non-optimal) case of weighting is original adaptive SPSA algorithm where all Hˆ are given equal weight Feedbac and weighting briefly discussed in slides to follow 12

13 Feedbac Contribution to Enhanced Algorithm Per-iteration Jacobian estimate can be decomposed into four parts: ˆ ˆ 2 H = H( θ ) + Ψ + noise +O( c ) where Ψ = Ψ ( ˆ ) H( θ) is error term and c is positive scalar such that c 0 as (c is the difference interval for the per-iteration estimates) Feedbac term is best estimate of error term (using current estimate of H) at each iteration: ( H ) ˆ Ψ 1 Overall (cumulative) estimate of H (i.e., based on weighted sums of Hˆ Ψˆ Ψ H n, = 0,1,..., n) is 13

14 Recall: Optimal Weighting 2 Hˆ = H θˆ + Ψ + + ( ) noise O( c ) 2 The noise term is of stochastic order O( c ) in the case of 1 noisy loss measurements (optimization) and order O( c ) in the case of noisy g measurements (root-finding) Above implies that noise variance grows with Optimal weighting is such that noise growth is damped down as gets large Can solve for optimal weights via Lagrange multipliers 14

15 Comments on Theory Former theory on convergence of θ continues to apply New theory required for convergence of H due to modified form Under various standard smoothness and bounded moments conditions, can show H H( θ ) a.s. For each entry in H have In special case of noise-free measurements, can show that rate of convergence of to H(θ ) is almost O(e ) Feedbac is critical to fast convergence, < variance for enhanced adaptive SPSA 1 variance for standard adaptive SPSA H Applies in case of direct loss or direct g measurements ˆ 15

16 Numerical Study Consider minimization problem based on 4th-order polynomial loss function; dim(θ) = 10 Compare standard adaptive and enhanced adaptive SPSA in terms of terminal loss function values and H estimates Used the stochastic gradient setting (direct noisy measurements of gradient g) (2SG) and setting with only loss values (2SPSA see next slide) Considered 50 independent runs for each experiment; normalized loss is from 0 (best) to 1(initial condition) Number of iterations Normalized loss from standard 2SG Normalized loss from enhanced 2SG P-values for comparing loss functions ,

17 Relative Convergence Rates for Hessian Estimate in 2SPSA in Noise-Free Problem Λ H true H (for quadratic loss functions) 17

18 Concluding Remars for Extension 1 Design of efficient and (relatively) easy to use adaptive search algorithms is long-standing problem Especially difficult in setting of noisy measurements Knowledge of Jacobian/Hessian matrix also useful in contexts outside of improved search (e.g., Cramér-Rao bound) Simultaneous perturbation idea can be used to dramatically enhance efficiency in multivariate problems Improvements here to standard adaptive SPSA are simple to use and further enhance efficiency Feedbac to reduce error in per-iteration Jacobian/Hessian estimates Optimal weighting to reduce effects of noise Using methods for efficient estimation of model of U.S. Navy system 18

19 Extension 2 Optimization with Discrete Version of SPSA Using Noisy Loss Function Measurements (Joint wor with Qi Wang of JHU and Barclays) 19

20 Discrete Problem Description Consider real-valued loss function L( θ): where is set of integers So, θ assumed to lie on p-dimensional integer grid Want to solve problem But we only now the noisy measurements p min L( θ) p θ y( θ) = L( θ) +ε( θ) E ( θ ) ε ( ) = 0 θ 20

21 Algorithm Description DSPSA: discrete SPSA (Wang and Spall, 2011) Step 0: Pic initial guess ˆθ 0 Step 1: Generate = [ 1, 2,, p ] T, where has userspecified distribution satisfying conditions. Special case is when i are independent Bernoulli random variables ±1 with probability ½ Step 2: πθ ( ˆ = ˆ ( ) + 1 ˆ = θˆ θˆ ) θ p 2,where θ 1,..., p Step 3: Evaluate y at πθ ( ˆ + ˆ ) 2 and πθ ( ) 2 Step 4: Construct gradient approximation: 1 1 ˆ ˆ 1 1 ( ) ( ˆ ) ( ˆ ) T g θ = 1,..., y π θ + Δ y π θ Δ 2 2 p Step 5: After M iterations of recursion, θˆ + 1 = θˆ agˆ ( θˆ ) set ˆ θ M as the solution;

22 Algorithm Description (Cont d) Remars: Simple implementation Two loss function measurements in each iteration Mae use of the function structure implicitly (gradientlie quantity) Handle noisy measurements Convergence property: under some general conditions, the sequence generated by DSPSA converges almost surely to the optimal solution. 22

23 Rate of Convergence Results for DSPSA We discuss MSE of DSPSA (Wang and Spall, 2013) Under some general conditions, we have * 2 * 2 ˆ µ + E θ ˆ θ (1 2 a ) E θ θ a pb (*) where m is determined by the loss function and b is a uniform upper bound for ˆ 1 ˆ 1 E ( ) ( ) L πθ + Δ L πθ Δ + ε ε 2 2 E Note: (*) is a difference-equation-lie recursion 2 ( + ) 2 23

24 Rate of Convergence Results for DSPSA (Cont d) Solving the above difference-equation-lie recursion, ( 1 α ) c * 2 * 2 1 Eθˆ ˆ θ Oe Eθ0 θ + O α where c > 0 and 0.5 < a < 1 The first big-o term is a function of m, a, a, A, and the second big-o term is a function of m, a, a, A, b,. 24

25 Rate of Convergence Results for DSPSA (Cont d) The result on MSE can produce rate of convergence * of P([ θˆ ] θ ) to 0 in the big-o sense, where θˆ indicates the nearest multivariate-integer point of ˆ * P([ θ ] θ ) [ ] ˆθ Convergence rate of can be used to compare DSPSA with other standard discrete stochastic algorithms such as stochastic ruler (SR) algorithm (Yan and Muai, 1992) and stochastic comparison (SC) algorithm (Gong et al, 1999). 25

26 Comparison of SR, SC and DSPSA We find the rate of convergence of SR and SC could not be better than DSPSA. The parameters (coefficients) involved in SR and SC are harder to tune than the coefficients in DSPSA. 26

27 Numerical Comparisons (one of many, with similar results) 27 Sewed quartic loss function (Spall 2003,Ex 6.6) with additive noise N(0,1) pb is an upper triangular matrix of 1 s, p = 200, domain { 10, 9,, 9, 10} 200 : T T p 3 p 4 i= 1 i= 1 i L ( θ) = θ B B θ+ 01. ( B θ) i ( B θ) Stochastic Ruler Stochastic Comparison DSPSA Stochastic Ruler Stochastic Comparison DSPSA [ θˆ ] θ [ θˆ ] θ 0 * * * L([ θˆ ]) L( θ ) ˆ * L([ θ0]) L( θ ) Number of Measurements x Number of Measurements x 10 4

28 Application of DSPSA in Resource Allocation in Public Health Application of DSPSA towards developing optimal public health strategies for containing the spread of influenza given limited societal resources. Open source software for intervention strategies (FluTE: Chao et al., 2010). θˆ DSPSA (perturbation) π( θˆ ) + Δ 2 π( θˆ ) Δ 2 FluTE Simulator y ( π( θˆ ) + Δ 2) y( π( θˆ ) Δ 2) DSPSA (calculation) θˆ +1 28

29 Concluding Remars for Extension 2 We introduce DSPSA for discrete stochastic optimization problem and show the almost sure convergence property. We discuss the rate of convergence of DSPSA, which is O(1/ a ). Rate of convergence results allow for objective comparison with other stochastic discrete optimization methods (e.g. stochastic ruler and stochastic comparison). We consider the application of DSPSA in resource allocation in public health. 29

30 Extra Slides 30

31 Per-Iteration Jacobian (Hessian) Estimate T δg δg = 2 p p 2 c + 2 c Hˆ 1,,,,,, where δg = G(θ + c ) G(θ c ) and G( ) is direct noisy measurement of g(θ) or the SP estimate of g(θ) built from noisy L(θ) measurements and [ 1, 2,, p ] T is mean-zero random vector such that the { j } satisfy standard SPSA conditions (mean zero, symmetrically distributed random variables with finite inverse moments) 31

32 Optimal Weighting Recall: H = (1 w ) H + w Hˆ Ψˆ ( ) 1 For the case of noisy loss functions, optimal weights are: c 4 w = 4 0 c For the case of noisy root-finding (g) measurements, optimal weights are: c 2 w = 2 0 c i = i = Both of above based on downweighting later per-iteration Jacobian estimates to compensate for increased noise contribution i i 32

75th MORSS CD Cover Slide UNCLASSIFIED DISCLOSURE FORM CD Presentation

75th MORSS CD Cover Slide UNCLASSIFIED DISCLOSURE FORM CD Presentation 75th MORSS CD Cover Slide UNCLASSIFIED DISCLOSURE FORM CD Presentation 712CD For office use only 41205 12-14 June 2007, at US Naval Academy, Annapolis, MD Please complete this form 712CD as your cover

More information

OPTIMIZATION WITH DISCRETE SIMULTANEOUS PERTURBATION STOCHASTIC APPROXIMATION USING NOISY LOSS FUNCTION MEASUREMENTS. by Qi Wang

OPTIMIZATION WITH DISCRETE SIMULTANEOUS PERTURBATION STOCHASTIC APPROXIMATION USING NOISY LOSS FUNCTION MEASUREMENTS. by Qi Wang OPTIMIZATION WITH DISCRETE SIMULTANEOUS PERTURBATION STOCHASTIC APPROXIMATION USING NOISY LOSS FUNCTION MEASUREMENTS by Qi Wang A dissertation submitted to Johns Hopins University in conformity with the

More information

Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions

Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions International Journal of Control Vol. 00, No. 00, January 2007, 1 10 Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions I-JENG WANG and JAMES C.

More information

Efficient Monte Carlo computation of Fisher information matrix using prior information

Efficient Monte Carlo computation of Fisher information matrix using prior information Efficient Monte Carlo computation of Fisher information matrix using prior information Sonjoy Das, UB James C. Spall, APL/JHU Roger Ghanem, USC SIAM Conference on Data Mining Anaheim, California, USA April

More information

1216 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 54, NO. 6, JUNE /$ IEEE

1216 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 54, NO. 6, JUNE /$ IEEE 1216 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 54, NO 6, JUNE 2009 Feedback Weighting Mechanisms for Improving Jacobian Estimates in the Adaptive Simultaneous Perturbation Algorithm James C Spall, Fellow,

More information

CONSTRAINED OPTIMIZATION OVER DISCRETE SETS VIA SPSA WITH APPLICATION TO NON-SEPARABLE RESOURCE ALLOCATION

CONSTRAINED OPTIMIZATION OVER DISCRETE SETS VIA SPSA WITH APPLICATION TO NON-SEPARABLE RESOURCE ALLOCATION Proceedings of the 200 Winter Simulation Conference B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer, eds. CONSTRAINED OPTIMIZATION OVER DISCRETE SETS VIA SPSA WITH APPLICATION TO NON-SEPARABLE

More information

FDSA and SPSA in Simulation-Based Optimization Stochastic approximation provides ideal framework for carrying out simulation-based optimization

FDSA and SPSA in Simulation-Based Optimization Stochastic approximation provides ideal framework for carrying out simulation-based optimization FDSA and SPSA in Simulation-Based Optimization Stochastic approximation provides ideal framework for carrying out simulation-based optimization Rigorous means for handling noisy loss information inherent

More information

EFFICIENT CALCULATION OF FISHER INFORMATION MATRIX: MONTE CARLO APPROACH USING PRIOR INFORMATION. Sonjoy Das

EFFICIENT CALCULATION OF FISHER INFORMATION MATRIX: MONTE CARLO APPROACH USING PRIOR INFORMATION. Sonjoy Das EFFICIENT CALCULATION OF FISHER INFORMATION MATRIX: MONTE CARLO APPROACH USING PRIOR INFORMATION by Sonjoy Das A thesis submitted to The Johns Hopkins University in conformity with the requirements for

More information

17 Solution of Nonlinear Systems

17 Solution of Nonlinear Systems 17 Solution of Nonlinear Systems We now discuss the solution of systems of nonlinear equations. An important ingredient will be the multivariate Taylor theorem. Theorem 17.1 Let D = {x 1, x 2,..., x m

More information

/97/$10.00 (c) 1997 AACC

/97/$10.00 (c) 1997 AACC Optimal Random Perturbations for Stochastic Approximation using a Simultaneous Perturbation Gradient Approximation 1 PAYMAN SADEGH, and JAMES C. SPALL y y Dept. of Mathematical Modeling, Technical University

More information

Allocation of Resources in CB Defense: Optimization and Ranking. NMSU: J. Cowie, H. Dang, B. Li Hung T. Nguyen UNM: F. Gilfeather Oct 26, 2005

Allocation of Resources in CB Defense: Optimization and Ranking. NMSU: J. Cowie, H. Dang, B. Li Hung T. Nguyen UNM: F. Gilfeather Oct 26, 2005 Allocation of Resources in CB Defense: Optimization and Raning by NMSU: J. Cowie, H. Dang, B. Li Hung T. Nguyen UNM: F. Gilfeather Oct 26, 2005 Outline Problem Formulation Architecture of Decision System

More information

Condensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C.

Condensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C. Condensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C. Spall John Wiley and Sons, Inc., 2003 Preface... xiii 1. Stochastic Search

More information

Q-Learning and Stochastic Approximation

Q-Learning and Stochastic Approximation MS&E338 Reinforcement Learning Lecture 4-04.11.018 Q-Learning and Stochastic Approximation Lecturer: Ben Van Roy Scribe: Christopher Lazarus Javier Sagastuy In this lecture we study the convergence of

More information

EE 381V: Large Scale Optimization Fall Lecture 24 April 11

EE 381V: Large Scale Optimization Fall Lecture 24 April 11 EE 381V: Large Scale Optimization Fall 2012 Lecture 24 April 11 Lecturer: Caramanis & Sanghavi Scribe: Tao Huang 24.1 Review In past classes, we studied the problem of sparsity. Sparsity problem is that

More information

Improved Methods for Monte Carlo Estimation of the Fisher Information Matrix

Improved Methods for Monte Carlo Estimation of the Fisher Information Matrix 008 American Control Conference Westin Seattle otel, Seattle, Washington, USA June -3, 008 ha8. Improved Methods for Monte Carlo stimation of the isher Information Matrix James C. Spall (james.spall@jhuapl.edu)

More information

ECE521 lecture 4: 19 January Optimization, MLE, regularization

ECE521 lecture 4: 19 January Optimization, MLE, regularization ECE521 lecture 4: 19 January 2017 Optimization, MLE, regularization First four lectures Lectures 1 and 2: Intro to ML Probability review Types of loss functions and algorithms Lecture 3: KNN Convexity

More information

Cramér Rao Bounds and Monte Carlo Calculation of the Fisher Information Matrix in Difficult Problems 1

Cramér Rao Bounds and Monte Carlo Calculation of the Fisher Information Matrix in Difficult Problems 1 Cramér Rao Bounds and Monte Carlo Calculation of the Fisher Information Matrix in Difficult Problems James C. Spall he Johns Hopkins University Applied Physics Laboratory Laurel, Maryland 073-6099 U.S.A.

More information

ESTIMATION ALGORITHMS

ESTIMATION ALGORITHMS ESTIMATIO ALGORITHMS Solving normal equations using QR-factorization on-linear optimization Two and multi-stage methods EM algorithm FEL 3201 Estimation Algorithms - 1 SOLVIG ORMAL EQUATIOS USIG QR FACTORIZATIO

More information

arxiv: v3 [math.oc] 8 Jan 2019

arxiv: v3 [math.oc] 8 Jan 2019 Why Random Reshuffling Beats Stochastic Gradient Descent Mert Gürbüzbalaban, Asuman Ozdaglar, Pablo Parrilo arxiv:1510.08560v3 [math.oc] 8 Jan 2019 January 9, 2019 Abstract We analyze the convergence rate

More information

Mixture Models & EM. Nicholas Ruozzi University of Texas at Dallas. based on the slides of Vibhav Gogate

Mixture Models & EM. Nicholas Ruozzi University of Texas at Dallas. based on the slides of Vibhav Gogate Mixture Models & EM icholas Ruozzi University of Texas at Dallas based on the slides of Vibhav Gogate Previously We looed at -means and hierarchical clustering as mechanisms for unsupervised learning -means

More information

Monte Carlo Computation of the Fisher Information Matrix in Nonstandard Settings

Monte Carlo Computation of the Fisher Information Matrix in Nonstandard Settings Monte Cao Computation of the Fisher Information Matrix in Nonstandard Settings James C. SPALL The Fisher information matrix summarizes the amount of information in the data relative to the quantities of

More information

Combine Monte Carlo with Exhaustive Search: Effective Variational Inference and Policy Gradient Reinforcement Learning

Combine Monte Carlo with Exhaustive Search: Effective Variational Inference and Policy Gradient Reinforcement Learning Combine Monte Carlo with Exhaustive Search: Effective Variational Inference and Policy Gradient Reinforcement Learning Michalis K. Titsias Department of Informatics Athens University of Economics and Business

More information

Mixture Models & EM. Nicholas Ruozzi University of Texas at Dallas. based on the slides of Vibhav Gogate

Mixture Models & EM. Nicholas Ruozzi University of Texas at Dallas. based on the slides of Vibhav Gogate Mixture Models & EM icholas Ruozzi University of Texas at Dallas based on the slides of Vibhav Gogate Previously We looed at -means and hierarchical clustering as mechanisms for unsupervised learning -means

More information

Super-Resolution. Dr. Yossi Rubner. Many slides from Miki Elad - Technion

Super-Resolution. Dr. Yossi Rubner. Many slides from Miki Elad - Technion Super-Resolution Dr. Yossi Rubner yossi@rubner.co.il Many slides from Mii Elad - Technion 5/5/2007 53 images, ratio :4 Example - Video 40 images ratio :4 Example Surveillance Example Enhance Mosaics Super-Resolution

More information

Machine Learning Ensemble Learning I Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi Spring /

Machine Learning Ensemble Learning I Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi Spring / Machine Learning Ensemble Learning I Hamid R. Rabiee Jafar Muhammadi, Alireza Ghasemi Spring 2015 http://ce.sharif.edu/courses/93-94/2/ce717-1 / Agenda Combining Classifiers Empirical view Theoretical

More information

Numerical Optimization

Numerical Optimization Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function

More information

Answers to Selected Exercises in Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C.

Answers to Selected Exercises in Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C. Answers to Selected Exercises in Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C. Spall This section provides answers to selected exercises in the chapters

More information

Stochastic gradient descent and robustness to ill-conditioning

Stochastic gradient descent and robustness to ill-conditioning Stochastic gradient descent and robustness to ill-conditioning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France ÉCOLE NORMALE SUPÉRIEURE Joint work with Aymeric Dieuleveut, Nicolas Flammarion,

More information

Discrete Variables and Gradient Estimators

Discrete Variables and Gradient Estimators iscrete Variables and Gradient Estimators This assignment is designed to get you comfortable deriving gradient estimators, and optimizing distributions over discrete random variables. For most questions,

More information

Lecture 8: Bayesian Estimation of Parameters in State Space Models

Lecture 8: Bayesian Estimation of Parameters in State Space Models in State Space Models March 30, 2016 Contents 1 Bayesian estimation of parameters in state space models 2 Computational methods for parameter estimation 3 Practical parameter estimation in state space

More information

Machine Learning Basics: Maximum Likelihood Estimation

Machine Learning Basics: Maximum Likelihood Estimation Machine Learning Basics: Maximum Likelihood Estimation Sargur N. srihari@cedar.buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 Topics 1. Learning

More information

Computer Intensive Methods in Mathematical Statistics

Computer Intensive Methods in Mathematical Statistics Computer Intensive Methods in Mathematical Statistics Department of mathematics johawes@kth.se Lecture 16 Advanced topics in computational statistics 18 May 2017 Computer Intensive Methods (1) Plan of

More information

Supplementary Note on Bayesian analysis

Supplementary Note on Bayesian analysis Supplementary Note on Bayesian analysis Structured variability of muscle activations supports the minimal intervention principle of motor control Francisco J. Valero-Cuevas 1,2,3, Madhusudhan Venkadesan

More information

OPTIMAL DESIGN INPUTS FOR EXPERIMENTAL CHAPTER 17. Organization of chapter in ISSO. Background. Linear models

OPTIMAL DESIGN INPUTS FOR EXPERIMENTAL CHAPTER 17. Organization of chapter in ISSO. Background. Linear models CHAPTER 17 Slides for Introduction to Stochastic Search and Optimization (ISSO)by J. C. Spall OPTIMAL DESIGN FOR EXPERIMENTAL INPUTS Organization of chapter in ISSO Background Motivation Finite sample

More information

Asymptotic and finite-sample properties of estimators based on stochastic gradients

Asymptotic and finite-sample properties of estimators based on stochastic gradients Asymptotic and finite-sample properties of estimators based on stochastic gradients Panagiotis (Panos) Toulis panos.toulis@chicagobooth.edu Econometrics and Statistics University of Chicago, Booth School

More information

Chapter 6: Derivative-Based. optimization 1

Chapter 6: Derivative-Based. optimization 1 Chapter 6: Derivative-Based Optimization Introduction (6. Descent Methods (6. he Method of Steepest Descent (6.3 Newton s Methods (NM (6.4 Step Size Determination (6.5 Nonlinear Least-Squares Problems

More information

Kostas Triantafyllopoulos University of Sheffield. Motivation / algorithmic pairs trading. Model set-up Detection of local mean-reversion

Kostas Triantafyllopoulos University of Sheffield. Motivation / algorithmic pairs trading. Model set-up Detection of local mean-reversion Detecting Mean Reverted Patterns in Statistical Arbitrage Outline Kostas Triantafyllopoulos University of Sheffield Motivation / algorithmic pairs trading Model set-up Detection of local mean-reversion

More information

AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET. Questions AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET

AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET. Questions AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET The Problem Identification of Linear and onlinear Dynamical Systems Theme : Curve Fitting Division of Automatic Control Linköping University Sweden Data from Gripen Questions How do the control surface

More information

Super-Resolution. Shai Avidan Tel-Aviv University

Super-Resolution. Shai Avidan Tel-Aviv University Super-Resolution Shai Avidan Tel-Aviv University Slide Credits (partial list) Ric Szelisi Steve Seitz Alyosha Efros Yacov Hel-Or Yossi Rubner Mii Elad Marc Levoy Bill Freeman Fredo Durand Sylvain Paris

More information

CLOSE-TO-CLEAN REGULARIZATION RELATES

CLOSE-TO-CLEAN REGULARIZATION RELATES Worshop trac - ICLR 016 CLOSE-TO-CLEAN REGULARIZATION RELATES VIRTUAL ADVERSARIAL TRAINING, LADDER NETWORKS AND OTHERS Mudassar Abbas, Jyri Kivinen, Tapani Raio Department of Computer Science, School of

More information

Lecture 10. Neural networks and optimization. Machine Learning and Data Mining November Nando de Freitas UBC. Nonlinear Supervised Learning

Lecture 10. Neural networks and optimization. Machine Learning and Data Mining November Nando de Freitas UBC. Nonlinear Supervised Learning Lecture 0 Neural networks and optimization Machine Learning and Data Mining November 2009 UBC Gradient Searching for a good solution can be interpreted as looking for a minimum of some error (loss) function

More information

Recursive Least Squares for an Entropy Regularized MSE Cost Function

Recursive Least Squares for an Entropy Regularized MSE Cost Function Recursive Least Squares for an Entropy Regularized MSE Cost Function Deniz Erdogmus, Yadunandana N. Rao, Jose C. Principe Oscar Fontenla-Romero, Amparo Alonso-Betanzos Electrical Eng. Dept., University

More information

Lecture 3 September 1

Lecture 3 September 1 STAT 383C: Statistical Modeling I Fall 2016 Lecture 3 September 1 Lecturer: Purnamrita Sarkar Scribe: Giorgio Paulon, Carlos Zanini Disclaimer: These scribe notes have been slightly proofread and may have

More information

AN EMPIRICAL SENSITIVITY ANALYSIS OF THE KIEFER-WOLFOWITZ ALGORITHM AND ITS VARIANTS

AN EMPIRICAL SENSITIVITY ANALYSIS OF THE KIEFER-WOLFOWITZ ALGORITHM AND ITS VARIANTS Proceedings of the 2013 Winter Simulation Conference R. Pasupathy, S.-H. Kim, A. Tolk, R. Hill, and M. E. Kuhl, eds. AN EMPIRICAL SENSITIVITY ANALYSIS OF THE KIEFER-WOLFOWITZ ALGORITHM AND ITS VARIANTS

More information

On sampled-data extremum seeking control via stochastic approximation methods

On sampled-data extremum seeking control via stochastic approximation methods On sampled-data extremum seeing control via stochastic approximation methods Sei Zhen Khong, Ying Tan, and Dragan Nešić Department of Electrical and Electronic Engineering The University of Melbourne,

More information

Behavior Policy Gradient Supplemental Material

Behavior Policy Gradient Supplemental Material Behavior Policy Gradient Supplemental Material Josiah P. Hanna 1 Philip S. Thomas 2 3 Peter Stone 1 Scott Niekum 1 A. Proof of Theorem 1 In Appendix A, we give the full derivation of our primary theoretical

More information

HOMEWORK #4: LOGISTIC REGRESSION

HOMEWORK #4: LOGISTIC REGRESSION HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2019 Due: 11am Monday, February 25th, 2019 Submit scan of plots/written responses to Gradebook; submit your

More information

Gaussian Mixture Model

Gaussian Mixture Model Case Study : Document Retrieval MAP EM, Latent Dirichlet Allocation, Gibbs Sampling Machine Learning/Statistics for Big Data CSE599C/STAT59, University of Washington Emily Fox 0 Emily Fox February 5 th,

More information

Non-asymptotic bound for stochastic averaging

Non-asymptotic bound for stochastic averaging Non-asymptotic bound for stochastic averaging S. Gadat and F. Panloup Toulouse School of Economics PGMO Days, November, 14, 2017 I - Introduction I - 1 Motivations I - 2 Optimization I - 3 Stochastic Optimization

More information

Perturbation. A One-measurement Form of Simultaneous Stochastic Approximation* Technical Communique. h,+t = 8, - a*k?,(o,), (1) Pergamon

Perturbation. A One-measurement Form of Simultaneous Stochastic Approximation* Technical Communique. h,+t = 8, - a*k?,(o,), (1) Pergamon Pergamon PII: sooo5-1098(%)00149-5 Aufomorica,'Vol. 33,No. 1.p~. 109-112, 1997 6 1997 Elsevier Science Ltd Printed in Great Britain. All rights reserved cmls-lo!%/97 517.00 + 0.00 Technical Communique

More information

SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS

SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 2005, DR RAPHAEL HAUSER 1. The Quasi-Newton Idea. In this lecture we will discuss

More information

Latent Variable Models and EM algorithm

Latent Variable Models and EM algorithm Latent Variable Models and EM algorithm SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic 3.1 Clustering and Mixture Modelling K-means and hierarchical clustering are non-probabilistic

More information

Beyond stochastic gradient descent for large-scale machine learning

Beyond stochastic gradient descent for large-scale machine learning Beyond stochastic gradient descent for large-scale machine learning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with Eric Moulines - October 2014 Big data revolution? A new

More information

arxiv: v1 [cs.it] 31 Dec 2018

arxiv: v1 [cs.it] 31 Dec 2018 arxiv:181211891v1 [csit] 31 Dec 2018 How did Donald Trump Surprisingly Win the 2016 United States Presidential Election? an Information-Theoretic Perspective ie Clean Sensing for Big Data Analytics: Optimal

More information

STOCHASTIC PERTURBATION METHODS FOR ROBUST OPTIMIZATION OF SIMULATION EXPERIMENTS

STOCHASTIC PERTURBATION METHODS FOR ROBUST OPTIMIZATION OF SIMULATION EXPERIMENTS The Pennsylvania State University The Graduate School Department of Industrial and Manufacturing Engineering STOCHASTIC PERTURBATION METHODS FOR ROBUST OPTIMIZATION OF SIMULATION EXPERIMENTS A Thesis in

More information

An Evolving Gradient Resampling Method for Machine Learning. Jorge Nocedal

An Evolving Gradient Resampling Method for Machine Learning. Jorge Nocedal An Evolving Gradient Resampling Method for Machine Learning Jorge Nocedal Northwestern University NIPS, Montreal 2015 1 Collaborators Figen Oztoprak Stefan Solntsev Richard Byrd 2 Outline 1. How to improve

More information

EM Algorithm II. September 11, 2018

EM Algorithm II. September 11, 2018 EM Algorithm II September 11, 2018 Review EM 1/27 (Y obs, Y mis ) f (y obs, y mis θ), we observe Y obs but not Y mis Complete-data log likelihood: l C (θ Y obs, Y mis ) = log { f (Y obs, Y mis θ) Observed-data

More information

10-701/15-781, Machine Learning: Homework 4

10-701/15-781, Machine Learning: Homework 4 10-701/15-781, Machine Learning: Homewor 4 Aarti Singh Carnegie Mellon University ˆ The assignment is due at 10:30 am beginning of class on Mon, Nov 15, 2010. ˆ Separate you answers into five parts, one

More information

Estimators as Random Variables

Estimators as Random Variables Estimation Theory Overview Properties Bias, Variance, and Mean Square Error Cramér-Rao lower bound Maimum likelihood Consistency Confidence intervals Properties of the mean estimator Introduction Up until

More information

HOMEWORK #4: LOGISTIC REGRESSION

HOMEWORK #4: LOGISTIC REGRESSION HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2018 Due: Friday, February 23rd, 2018, 11:55 PM Submit code and report via EEE Dropbox You should submit a

More information

COMP 551 Applied Machine Learning Lecture 2: Linear regression

COMP 551 Applied Machine Learning Lecture 2: Linear regression COMP 551 Applied Machine Learning Lecture 2: Linear regression Instructor: (jpineau@cs.mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551 Unless otherwise noted, all material posted for this

More information

Optimization Tutorial 1. Basic Gradient Descent

Optimization Tutorial 1. Basic Gradient Descent E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

More information

APPLYING QUANTUM COMPUTER FOR THE REALIZATION OF SPSA ALGORITHM Oleg Granichin, Alexey Wladimirovich

APPLYING QUANTUM COMPUTER FOR THE REALIZATION OF SPSA ALGORITHM Oleg Granichin, Alexey Wladimirovich APPLYING QUANTUM COMPUTER FOR THE REALIZATION OF SPSA ALGORITHM Oleg Granichin, Alexey Wladimirovich Department of Mathematics and Mechanics St. Petersburg State University Abstract The estimates of the

More information

Estimating Gaussian Mixture Densities with EM A Tutorial

Estimating Gaussian Mixture Densities with EM A Tutorial Estimating Gaussian Mixture Densities with EM A Tutorial Carlo Tomasi Due University Expectation Maximization (EM) [4, 3, 6] is a numerical algorithm for the maximization of functions of several variables

More information

Topic 3 Random variables, expectation, and variance, II

Topic 3 Random variables, expectation, and variance, II CSE 103: Probability and statistics Fall 2010 Topic 3 Random variables, expectation, and variance, II 3.1 Linearity of expectation If you double each value of X, then you also double its average; that

More information

Local Strong Convexity of Maximum-Likelihood TDOA-Based Source Localization and Its Algorithmic Implications

Local Strong Convexity of Maximum-Likelihood TDOA-Based Source Localization and Its Algorithmic Implications Local Strong Convexity of Maximum-Likelihood TDOA-Based Source Localization and Its Algorithmic Implications Huikang Liu, Yuen-Man Pun, and Anthony Man-Cho So Dept of Syst Eng & Eng Manag, The Chinese

More information

Constrained Optimization

Constrained Optimization 1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange

More information

Lecture 9: Policy Gradient II 1

Lecture 9: Policy Gradient II 1 Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John

More information

LECTURE 5 NOTES. n t. t Γ(a)Γ(b) pt+a 1 (1 p) n t+b 1. The marginal density of t is. Γ(t + a)γ(n t + b) Γ(n + a + b)

LECTURE 5 NOTES. n t. t Γ(a)Γ(b) pt+a 1 (1 p) n t+b 1. The marginal density of t is. Γ(t + a)γ(n t + b) Γ(n + a + b) LECTURE 5 NOTES 1. Bayesian point estimators. In the conventional (frequentist) approach to statistical inference, the parameter θ Θ is considered a fixed quantity. In the Bayesian approach, it is considered

More information

Fisher Information Matrix-based Nonlinear System Conversion for State Estimation

Fisher Information Matrix-based Nonlinear System Conversion for State Estimation Fisher Information Matrix-based Nonlinear System Conversion for State Estimation Ming Lei Christophe Baehr and Pierre Del Moral Abstract In practical target tracing a number of improved measurement conversion

More information

Autoencoders and Score Matching. Based Models. Kevin Swersky Marc Aurelio Ranzato David Buchman Benjamin M. Marlin Nando de Freitas

Autoencoders and Score Matching. Based Models. Kevin Swersky Marc Aurelio Ranzato David Buchman Benjamin M. Marlin Nando de Freitas On for Energy Based Models Kevin Swersky Marc Aurelio Ranzato David Buchman Benjamin M. Marlin Nando de Freitas Toronto Machine Learning Group Meeting, 2011 Motivation Models Learning Goal: Unsupervised

More information

Factor Analysis and Kalman Filtering (11/2/04)

Factor Analysis and Kalman Filtering (11/2/04) CS281A/Stat241A: Statistical Learning Theory Factor Analysis and Kalman Filtering (11/2/04) Lecturer: Michael I. Jordan Scribes: Byung-Gon Chun and Sunghoon Kim 1 Factor Analysis Factor analysis is used

More information

The Expectation-Maximization Algorithm

The Expectation-Maximization Algorithm The Expectation-Maximization Algorithm Francisco S. Melo In these notes, we provide a brief overview of the formal aspects concerning -means, EM and their relation. We closely follow the presentation in

More information

Deep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C.

Deep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C. Chapter 4: Numerical Computation Deep Learning Authors: I. Goodfellow, Y. Bengio, A. Courville Lecture slides edited by 1 Chapter 4: Numerical Computation 4.1 Overflow and Underflow 4.2 Poor Conditioning

More information

DEVELOPMENTS IN STOCHASTIC OPTIMIZATION ALGORITHMS WITH GRADIENT APPROXIMATIONS BASED ON FUNCTION MEASUREMENTS. James C. Span

DEVELOPMENTS IN STOCHASTIC OPTIMIZATION ALGORITHMS WITH GRADIENT APPROXIMATIONS BASED ON FUNCTION MEASUREMENTS. James C. Span Proceedings of the 1994 Winter Simulaiaon Conference eel. J. D. Tew, S. Manivannan, D. A. Sadowski, and A. F. Seila DEVELOPMENTS IN STOCHASTIC OPTIMIZATION ALGORITHMS WITH GRADIENT APPROXIMATIONS BASED

More information

SOLVING POWER AND OPTIMAL POWER FLOW PROBLEMS IN THE PRESENCE OF UNCERTAINTY BY AFFINE ARITHMETIC

SOLVING POWER AND OPTIMAL POWER FLOW PROBLEMS IN THE PRESENCE OF UNCERTAINTY BY AFFINE ARITHMETIC SOLVING POWER AND OPTIMAL POWER FLOW PROBLEMS IN THE PRESENCE OF UNCERTAINTY BY AFFINE ARITHMETIC Alfredo Vaccaro RTSI 2015 - September 16-18, 2015, Torino, Italy RESEARCH MOTIVATIONS Power Flow (PF) and

More information

Cognitive Cyber-Physical System

Cognitive Cyber-Physical System Cognitive Cyber-Physical System Physical to Cyber-Physical The emergence of non-trivial embedded sensor units, networked embedded systems and sensor/actuator networks has made possible the design and implementation

More information

SIMULTANEOUS STATE AND PARAMETER ESTIMATION USING KALMAN FILTERS

SIMULTANEOUS STATE AND PARAMETER ESTIMATION USING KALMAN FILTERS ECE5550: Applied Kalman Filtering 9 1 SIMULTANEOUS STATE AND PARAMETER ESTIMATION USING KALMAN FILTERS 9.1: Parameters versus states Until now, we have assumed that the state-space model of the system

More information

Gaussian Processes for Machine Learning

Gaussian Processes for Machine Learning Gaussian Processes for Machine Learning Carl Edward Rasmussen Max Planck Institute for Biological Cybernetics Tübingen, Germany carl@tuebingen.mpg.de Carlos III, Madrid, May 2006 The actual science of

More information

Direct Simulation Methods #2

Direct Simulation Methods #2 Direct Simulation Methods #2 Econ 690 Purdue University Outline 1 A Generalized Rejection Sampling Algorithm 2 The Weighted Bootstrap 3 Importance Sampling Rejection Sampling Algorithm #2 Suppose we wish

More information

Using Multiple Kernel-based Regularization for Linear System Identification

Using Multiple Kernel-based Regularization for Linear System Identification Using Multiple Kernel-based Regularization for Linear System Identification What are the Structure Issues in System Identification? with coworkers; see last slide Reglerteknik, ISY, Linköpings Universitet

More information

A Few Notes on Fisher Information (WIP)

A Few Notes on Fisher Information (WIP) A Few Notes on Fisher Information (WIP) David Meyer dmm@{-4-5.net,uoregon.edu} Last update: April 30, 208 Definitions There are so many interesting things about Fisher Information and its theoretical properties

More information

An Introduction to Expectation-Maximization

An Introduction to Expectation-Maximization An Introduction to Expectation-Maximization Dahua Lin Abstract This notes reviews the basics about the Expectation-Maximization EM) algorithm, a popular approach to perform model estimation of the generative

More information

Stochastic Proximal Gradient Algorithm

Stochastic Proximal Gradient Algorithm Stochastic Institut Mines-Télécom / Telecom ParisTech / Laboratoire Traitement et Communication de l Information Joint work with: Y. Atchade, Ann Arbor, USA, G. Fort LTCI/Télécom Paristech and the kind

More information

Numerical solutions of nonlinear systems of equations

Numerical solutions of nonlinear systems of equations Numerical solutions of nonlinear systems of equations Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan E-mail: min@math.ntnu.edu.tw August 28, 2011 Outline 1 Fixed points

More information

Chapter 1: A Brief Review of Maximum Likelihood, GMM, and Numerical Tools. Joan Llull. Microeconometrics IDEA PhD Program

Chapter 1: A Brief Review of Maximum Likelihood, GMM, and Numerical Tools. Joan Llull. Microeconometrics IDEA PhD Program Chapter 1: A Brief Review of Maximum Likelihood, GMM, and Numerical Tools Joan Llull Microeconometrics IDEA PhD Program Maximum Likelihood Chapter 1. A Brief Review of Maximum Likelihood, GMM, and Numerical

More information

Lecture 4 September 15

Lecture 4 September 15 IFT 6269: Probabilistic Graphical Models Fall 2017 Lecture 4 September 15 Lecturer: Simon Lacoste-Julien Scribe: Philippe Brouillard & Tristan Deleu 4.1 Maximum Likelihood principle Given a parametric

More information

Prediction-based adaptive control of a class of discrete-time nonlinear systems with nonlinear growth rate

Prediction-based adaptive control of a class of discrete-time nonlinear systems with nonlinear growth rate www.scichina.com info.scichina.com www.springerlin.com Prediction-based adaptive control of a class of discrete-time nonlinear systems with nonlinear growth rate WEI Chen & CHEN ZongJi School of Automation

More information

CONVERGENCE RATE OF LINEAR TWO-TIME-SCALE STOCHASTIC APPROXIMATION 1. BY VIJAY R. KONDA AND JOHN N. TSITSIKLIS Massachusetts Institute of Technology

CONVERGENCE RATE OF LINEAR TWO-TIME-SCALE STOCHASTIC APPROXIMATION 1. BY VIJAY R. KONDA AND JOHN N. TSITSIKLIS Massachusetts Institute of Technology The Annals of Applied Probability 2004, Vol. 14, No. 2, 796 819 Institute of Mathematical Statistics, 2004 CONVERGENCE RATE OF LINEAR TWO-TIME-SCALE STOCHASTIC APPROXIMATION 1 BY VIJAY R. KONDA AND JOHN

More information

1 Introduction 198; Dugard et al, 198; Dugard et al, 198) A delay matrix in such a lower triangular form is called an interactor matrix, and almost co

1 Introduction 198; Dugard et al, 198; Dugard et al, 198) A delay matrix in such a lower triangular form is called an interactor matrix, and almost co Multivariable Receding-Horizon Predictive Control for Adaptive Applications Tae-Woong Yoon and C M Chow y Department of Electrical Engineering, Korea University 1, -a, Anam-dong, Sungbu-u, Seoul 1-1, Korea

More information

An interior-point stochastic approximation method and an L1-regularized delta rule

An interior-point stochastic approximation method and an L1-regularized delta rule Photograph from National Geographic, Sept 2008 An interior-point stochastic approximation method and an L1-regularized delta rule Peter Carbonetto, Mark Schmidt and Nando de Freitas University of British

More information

February 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM

February 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM February 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Abstract. The Rao-Blacwell theorem told us how to improve an estimator. We will discuss conditions on when the Rao-Blacwellization of an estimator

More information

Lecture 9: Policy Gradient II (Post lecture) 2

Lecture 9: Policy Gradient II (Post lecture) 2 Lecture 9: Policy Gradient II (Post lecture) 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver

More information

Static Parameter Estimation using Kalman Filtering and Proximal Operators Vivak Patel

Static Parameter Estimation using Kalman Filtering and Proximal Operators Vivak Patel Static Parameter Estimation using Kalman Filtering and Proximal Operators Viva Patel Department of Statistics, University of Chicago December 2, 2015 Acnowledge Mihai Anitescu Senior Computational Mathematician

More information

Quadratic Extended Filtering in Nonlinear Systems with Uncertain Observations

Quadratic Extended Filtering in Nonlinear Systems with Uncertain Observations Applied Mathematical Sciences, Vol. 8, 2014, no. 4, 157-172 HIKARI Ltd, www.m-hiari.com http://dx.doi.org/10.12988/ams.2014.311636 Quadratic Extended Filtering in Nonlinear Systems with Uncertain Observations

More information

Machine Learning and Data Mining. Linear regression. Kalev Kask

Machine Learning and Data Mining. Linear regression. Kalev Kask Machine Learning and Data Mining Linear regression Kalev Kask Supervised learning Notation Features x Targets y Predictions ŷ Parameters q Learning algorithm Program ( Learner ) Change q Improve performance

More information

Why should you care about the solution strategies?

Why should you care about the solution strategies? Optimization Why should you care about the solution strategies? Understanding the optimization approaches behind the algorithms makes you more effectively choose which algorithm to run Understanding the

More information

Generalized Attitude Determination with One Dominant Vector Observation

Generalized Attitude Determination with One Dominant Vector Observation Generalized Attitude Determination with One Dominant Vector Observation John L. Crassidis University at Buffalo, State University of New Yor, Amherst, New Yor, 460-4400 Yang Cheng Mississippi State University,

More information

Artificial Intelligence

Artificial Intelligence Artificial Intelligence Dynamic Programming Marc Toussaint University of Stuttgart Winter 2018/19 Motivation: So far we focussed on tree search-like solvers for decision problems. There is a second important

More information

Sampling Controlled Stochastic Recursions: Applications to Simulation Optimization and Stochastic Root Finding.

Sampling Controlled Stochastic Recursions: Applications to Simulation Optimization and Stochastic Root Finding. Sampling Controlled Stochastic Recursions: Applications to Simulation Optimization and Stochastic Root Finding. Fatemeh Sadat Hashemi Dissertation submitted to the Faculty of the Virginia Polytechnic Institute

More information