# Rank-Constrainted Optimization: A Riemannian Manifold Approach

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Ran-Constrainted Optimization: A Riemannian Manifold Approach Guifang Zhou1, Wen Huang2, Kyle A. Gallivan1, Paul Van Dooren2, P.-A. Absil2 1- Florida State University - Department of Mathematics 1017 Academic Way, Tallahassee, FL , US 2- Universite catholique de Louvain - ICTEAM Institute Avenue G. Lemaı tre 4, 148 Louvain-la-Neuve, Belgium Abstract. This paper presents an algorithm that solves optimization problems on a matrix manifold M Rm n with an additional ran inequality constraint. New geometric objects are defined to facilitate efficiently finding a suitable ran. The convergence properties of the algorithm are given and a weighted low-ran approximation problem is used to illustrate the efficiency and effectiveness of the algorithm. 1 Introduction We consider low-ran optimization problems that combine a ran inequality constraint with a matrix manifold constraint: min f (X), X M (1) where min(m, n), M = {X M ran(x) } and M is a submanifold of Rm n. Typical choices for M are: the entire set Rm n, a sphere, symmetric matrices, elliptopes, spectrahedrons. Recently, optimization on manifolds has attracted significant attention. Attempts have been made to understand problem (1) by considering a related but simpler problem minx Rm n f (X), where Rm n = {X Rm n ran(x) = }, (see e.g., [1, 2]). Since Rm n is a submanifold of Rm n of dimension (m+n ), the simpler problem can be solved using techniques from Riemannian optimization [] applied to matrix manifolds. However, a disadvantage is that the manifold Rm n is not closed in Rm n, which complicates considerably the conver gence analysis and performance of an iteration. Very recently a more global view of a projected line-search method on Rm n = m n {X R ran(x) } along with a convergence analysis has been developed in [4]. In [], the results of [4] are exploited to propose an algorithm that successively increases the ran by a given constant. Its convergence result to critical This research was supported by the National Science Foundation under grant NSF This paper presents research results of the Belgian Networ DYSCO (Dynamical Systems, Control, and Optimization), funded by the Interuniversity Attraction Poles Programme initiated by the Belgian Science Policy Office. This wor was supported by grant FNRS PDR T

3 for X M but is not for any point X M 1. The tangent cone TX M contains the tangent space TX Mr and also the curves approaching X by points of ran T greater than r, but not beyond, i.e., TX M = TX Mr + {η r NX Mr TX M ran(η r ) r}. The characterization of TX M for M = Rm n has been given in [4]. To obtain the next iteration on M M, a general line-search method is considered: Xn+1 = R Xn (tn ηxn,r ), where tn is a step size and R is a ran-related retraction defined below in Definition 1. Definition 1 (Ran-related retraction) Let X Mr. A mapping R X : TX M M is a ran-related retraction if, ηx TX M, (i) R X (0) = X, (ii) δ > 0 such that [0, δ) t 7 R X (tηx ) is smooth and R X (tηx ) Mr for all t [0, δ), where r is the integer such that r r, ηx TX M r, and d ηx / TX Mr 1, (iii) dt R X (tηx ) t=0 = ηx. It can be shown that a ran-related retraction always exists. Note R X is not necessarily S a retraction on M since it may not be smooth on the tangent bundle TM := X M TX M. The modified Riemannian optimization algorithm is setched as Alg. 1. Generally speaing, η in Step 7 in Alg. 1 is not unique and any one can be used to change to ran r. The ey to efficiency of the algorithm is to choose r correctly and then choose an appropriate η. In [4], a similar idea is developed for M = Rm n and r =. The authors define the search direction as the projection of the negative gradient onto the tangent cone. If M = Rm n and r =, the choice of η in Step 7 is equivalent to the definition in [4, Corollary.]. Main Theoretical Results Suppose the cost function f is continuously differentiable and the algorithm chosen in Step 2 of Alg. 1 has the property of global convergence to critical points of fr, we give the following main results. The proofs are omitted due to space constraints (see [1]). Theorem 1 (Global Result) The sequence q {Xn } generated by Alg. 1 satisfies 1 + ǫ12 ǫ2. lim inf n PTXn M (gradff (Xn )) 1 2 Theorem 2 (Local Result) Let ff be a C function and X be a nondegenerate minimizer of ff with ran r. Suppose ǫ2 > 0. Denote the sequence of iterates generated by Algorithm 1 by {Xn }. There exists a neighborhood of X, UX, such that if D = {X M f (X) f (X0 )} UX ; D is compact; fˆ : TM R : ξ 7 ff R (ξ) is a radially L-C 1 function with sufficient large βrl defined in [, Definition 7.4.1] such that for 1 any X, Y D, R X (Y ) < βrl, then there exists N > 0 such that n > N ran(xn ) = r and Xn UX. 21

4 Algorithm 1 Modified Riemannian Optimization Algorithm 1: 2: : 4: : 6: 7: 8: 9: 10: 11: 12: 1: 14: 1: 16: 17: 4 for n = 0, 1, 2,... do Apply a Riemannian algorithm (e.g. one of GenRTR [7], RBFGS [8, 9, 11], RTR-SR1 [9, 12]) to approximately optimize f over Mr with initial point Xn and stop at X n Mr, where either gradfr (X n ) < ǫ (flag 1) or X n is close to M r 1 (flag 0); if flag = 1 then if Both Conditions I and II are satisfied then Set r and η to be r and gradfr (X n ) respectively; while gradff (X n ) η > ǫ21 η do Set r to be r + 1 and η argminη TX M r gradff (X n ) ηf ; end while Obtain Xn+1 by applying an Armijo-type line search algorithm along η using a ran-related retraction; else If ǫ is small enough, stop. Otherwise, ǫ τ ǫ, where τ (0, 1); end if else {flag = 0} If the ran has not been increased on any previous iteration, reduce the ran of X n based on truncation while eeping the function value decrease, update r, obtain next iterate Xn+1 ; Else reduce the ran of X n such that the next iterate Xn+1 satisfies f (Xn+1 ) f (Xi ) c(f (Xi+1 ) f (Xi )), where i is such that the latest ran increase was from Xi to Xi+1, 0 < c < 1. Set r to be the ran of Xn+1 ; end if end for Application We illustrate the effectiveness of Alg. 1 for the weighted low-ran approximation problem: min A X2W, f (X) = A X2W = vec{a X}T W vec{a X} X M (2) where M = R80 10, A is given, W R is a positive definite symmetric weighting matrix and vec{a} denotes the vectorized form of A, i.e., a vector constructed by stacing the consecutive columns of A in one vector. The matrix A is generated by A1 AT2 Rm n, where A1 R80, A2 R10. W = U ΣU T, where U R is a random orthogonal matrix generated by Matlab s QR and RAND. The mn singular values of the weighting matrix are generated by Matlab function LOGSPACE with condition number 100 and multiplying, element-wise, by a uniform distribution matrix on the interval [0., 1.]. Three values of are considered, one less than the true ran, one equal to the 22

5 = = =7 method Alg. 1 DMM SULS APM Alg. 1 DMM SULS APM Alg. 1 DMM SULS APM ran.02(99/100) 7(0/100) 7(0/100) f W Rel Err( A X AW ) time(sec) Table 1: Method comparisons. The number in the parenthesis indicates the fraction of experiments where the numerical ran (number of singular values greater than 10 8 ) found by the algorithm equals the true ran. The subscript ± indicates a scale of 10±. Alg. 1 and SULS, are stopped when the norm of the final gradient on the fixed-ran manifold over the norm of initial full gradient is less than 10 8 while DMM and APM are stopped when the norm of final gradient over the norm of initial gradient is less than true ran and one greater than the true ran. Four algorithms are compared: Alg. 1, DMM [14], SULS [4] and APM [6]. The initial point of Alg. 1 and SULS is a randomly generated ran-1 matrix and the initial points of DMM and APM are randomly generated n-by-(n ) and m-by-(m ) matrices respectively. Results shown in Table 1 are the average of 100 runs for different data matrices R, weighting matrices W and initial points. The results show that Alg. 1 with ǫ1 =, ǫ2 = 10 4 (GenRTR is used as the inner algorithm, when the sizes become larger, limited-memory RBFGS [9, 11], for example, may be a better choice) has significant advantages compared with the other methods. It achieves good accuracy in the approximation with less computational time. Furthermore, for Alg. 1, all Riemannian objects, their updates and Xn are based on an efficient three factor, U, D, V, representation, and the singular values are immediately available while, for the other three methods, an additional SVD is required. References [1] B. Mishra, G. Meyer, F. Bach, and R. Sepulchre. Low-ran optimization with trace norm penalty. SIAM Journal on Optimization, 2(4): , 201. [2] M. Journe e, F. Bach, P.-A. Absil, and R. Sepulchre. Low-ran optimization on the cone of positive semidefinite matrices. SIAM Journal on Optimiza- 2

6 tion, 20():227 21, [] P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, [4] R. Schneider and A. Uschmajew. Convergence results for projected linesearch methods on varieties of low-ran matrices via Lojasiewicz inequality. ArXiv e-prints, February [] A. Uschmajew and B. Vandereycen. Line-search methods and ran increase on low-ran matrix varieties. In Proceedings of the 2014 International Symposium on Nonlinear Theory and its Applications (NOLTA2014), [6] W.-S. Lu, S.-C. Pei, and P.-H. Wang. Weighted low-ran approximation of general complex matrices and its application in the design of 2-d digital filters. IEEE Transactions on Circuits and Systems I, 44:60 6, [7] P.-A. Absil, C. G. Baer, and Kyle A. Gallivan. Trust-region methods on Riemannian manifolds. Foundations of Computational Mathematics, 7():0 0, [8] W. Ring and B. Wirth. Optimization methods on Riemannian manifolds and their application to shape space. SIAM Journal on Optimization, 22(2):96 627, January [9] W. Huang. Optimization algorithms on Riemannian manifolds with applications. PhD thesis, Florida State University, 201. [10] Donal B. O Shea and Leslie C. Wilson. Limits of tangent spaces to real surfaces. American Journal of Mathematics, 126():pp , [11] Wen Huang, K. A. Gallivan, and P.-A. Absil. A Broyden class of quasi-newton methods for Riemannian optimization. UCL-INMA , U.C.Louvain, Submitted for publication, 201. [12] W. Huang, P.-A. Absil, and K. A. Gallivan. A Riemannian symmetric ran-one trust-region method. Mathematical Programming, February doi: /s [1] Guifang Zhou. Ran-contrained optimization: A Riemannian approach. PhD thesis, Florida State University. in preparation. [14] I. Brace and J. H. Manton. An improved BFGS-on-manifold algorithm for computing low-ran weighted approximations. In Proceedings of 17th International Symposium on Mathematical Theory of Networs and Systems, pages ,

### Optimization Algorithms on Riemannian Manifolds with Applications

Optimization Algorithms on Riemannian Manifolds with Applications Wen Huang Coadvisor: Kyle A. Gallivan Coadvisor: Pierre-Antoine Absil Florida State University Catholic University of Louvain November

### The nonsmooth Newton method on Riemannian manifolds

The nonsmooth Newton method on Riemannian manifolds C. Lageman, U. Helmke, J.H. Manton 1 Introduction Solving nonlinear equations in Euclidean space is a frequently occurring problem in optimization and

### H 2 -optimal model reduction of MIMO systems

H 2 -optimal model reduction of MIMO systems P. Van Dooren K. A. Gallivan P.-A. Absil Abstract We consider the problem of approximating a p m rational transfer function Hs of high degree by another p m

### Max-Planck-Institut für Mathematik in den Naturwissenschaften Leipzig

Max-Planck-Institut für Mathematik in den Naturwissenschaften Leipzig Convergence analysis of Riemannian GaussNewton methods and its connection with the geometric condition number by Paul Breiding and

### A variable projection method for block term decomposition of higher-order tensors

A variable projection method for block term decomposition of higher-order tensors Guillaume Olikier 1, P.-A. Absil 1, and Lieven De Lathauwer 2 1- Université catholique de Louvain - ICTEAM Institute B-1348

### Generalized Shifted Inverse Iterations on Grassmann Manifolds 1

Proceedings of the Sixteenth International Symposium on Mathematical Networks and Systems (MTNS 2004), Leuven, Belgium Generalized Shifted Inverse Iterations on Grassmann Manifolds 1 J. Jordan α, P.-A.

### Quasi-Newton Methods

Quasi-Newton Methods Werner C. Rheinboldt These are excerpts of material relating to the boos [OR00 and [Rhe98 and of write-ups prepared for courses held at the University of Pittsburgh. Some further references

### CONVEX OPTIMIZATION OVER POSITIVE POLYNOMIALS AND FILTER DESIGN. Y. Genin, Y. Hachez, Yu. Nesterov, P. Van Dooren

CONVEX OPTIMIZATION OVER POSITIVE POLYNOMIALS AND FILTER DESIGN Y. Genin, Y. Hachez, Yu. Nesterov, P. Van Dooren CESAME, Université catholique de Louvain Bâtiment Euler, Avenue G. Lemaître 4-6 B-1348 Louvain-la-Neuve,

### Online Learning in the Manifold of Low-Rank Matrices

Online Learning in the Manifold of Low-Ran Matrices Uri Shalit, Daphna Weinshall Computer Science Dept. and ICNC The Hebrew University of Jerusalem uri.shalit@mail.huji.ac.il daphna@cs.huji.ac.il Gal Chechi

### Navigation and Obstacle Avoidance via Backstepping for Mechanical Systems with Drift in the Closed Loop

Navigation and Obstacle Avoidance via Backstepping for Mechanical Systems with Drift in the Closed Loop Jan Maximilian Montenbruck, Mathias Bürger, Frank Allgöwer Abstract We study backstepping controllers

### Introduction to Nonlinear Optimization Paul J. Atzberger

Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,

### arxiv: v4 [math.oc] 5 Jan 2016

Restarted SGD: Beating SGD without Smoothness and/or Strong Convexity arxiv:151.03107v4 [math.oc] 5 Jan 016 Tianbao Yang, Qihang Lin Department of Computer Science Department of Management Sciences The

### You should be able to...

Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set

### Ranking from Crowdsourced Pairwise Comparisons via Matrix Manifold Optimization

Ranking from Crowdsourced Pairwise Comparisons via Matrix Manifold Optimization Jialin Dong ShanghaiTech University 1 Outline Introduction FourVignettes: System Model and Problem Formulation Problem Analysis

### Linear Regression (continued)

Linear Regression (continued) Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 6, 2017 1 / 39 Outline 1 Administration 2 Review of last lecture 3 Linear regression

### NONLINEAR CLASSIFICATION AND REGRESSION. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition

NONLINEAR CLASSIFICATION AND REGRESSION Nonlinear Classification and Regression: Outline 2 Multi-Layer Perceptrons The Back-Propagation Learning Algorithm Generalized Linear Models Radial Basis Function

### Supplementary Notes for W. Rudin: Principles of Mathematical Analysis

Supplementary Notes for W. Rudin: Principles of Mathematical Analysis SIGURDUR HELGASON In 8.00B it is customary to cover Chapters 7 in Rudin s book. Experience shows that this requires careful planning

### Line Search Methods for Unconstrained Optimisation

Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic

### HAIYUN ZHOU, RAVI P. AGARWAL, YEOL JE CHO, AND YONG SOO KIM

Georgian Mathematical Journal Volume 9 (2002), Number 3, 591 600 NONEXPANSIVE MAPPINGS AND ITERATIVE METHODS IN UNIFORMLY CONVEX BANACH SPACES HAIYUN ZHOU, RAVI P. AGARWAL, YEOL JE CHO, AND YONG SOO KIM

### New Inexact Line Search Method for Unconstrained Optimization 1,2

journal of optimization theory and applications: Vol. 127, No. 2, pp. 425 446, November 2005 ( 2005) DOI: 10.1007/s10957-005-6553-6 New Inexact Line Search Method for Unconstrained Optimization 1,2 Z.

### Numerical optimization

Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

### Optimization Methods for Machine Learning

Optimization Methods for Machine Learning Sathiya Keerthi Microsoft Talks given at UC Santa Cruz February 21-23, 2017 The slides for the talks will be made available at: http://www.keerthis.com/ Introduction

### Math 411 Preliminaries

Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector

### Multilayer Perceptron

Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4

### The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1

October 2003 The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1 by Asuman E. Ozdaglar and Dimitri P. Bertsekas 2 Abstract We consider optimization problems with equality,

### A Descent Method for Equality and Inequality Constrained Multiobjective Optimization Problems

A Descent Method for Equality and Inequality Constrained Multiobjective Optimization Problems arxiv:1712.03005v2 [math.oc] 11 Dec 2017 Bennet Gebken 1, Sebastian Peitz 1, and Michael Dellnitz 1 1 Department

### Fast Nonnegative Matrix Factorization with Rank-one ADMM

Fast Nonnegative Matrix Factorization with Rank-one Dongjin Song, David A. Meyer, Martin Renqiang Min, Department of ECE, UCSD, La Jolla, CA, 9093-0409 dosong@ucsd.edu Department of Mathematics, UCSD,

### Chap. 1. Some Differential Geometric Tools

Chap. 1. Some Differential Geometric Tools 1. Manifold, Diffeomorphism 1.1. The Implicit Function Theorem ϕ : U R n R n p (0 p < n), of class C k (k 1) x 0 U such that ϕ(x 0 ) = 0 rank Dϕ(x) = n p x U

### Cubic regularization of Newton s method for convex problems with constraints

CORE DISCUSSION PAPER 006/39 Cubic regularization of Newton s method for convex problems with constraints Yu. Nesterov March 31, 006 Abstract In this paper we derive efficiency estimates of the regularized

### A NUMERICAL METHOD TO SOLVE A QUADRATIC CONSTRAINED MAXIMIZATION

A NUMERICAL METHOD TO SOLVE A QUADRATIC CONSTRAINED MAXIMIZATION ALIREZA ESNA ASHARI, RAMINE NIKOUKHAH, AND STEPHEN L. CAMPBELL Abstract. The problem of maximizing a quadratic function subject to an ellipsoidal

### Steepest descent method on a Riemannian manifold: the convex case

Steepest descent method on a Riemannian manifold: the convex case Julien Munier Abstract. In this paper we are interested in the asymptotic behavior of the trajectories of the famous steepest descent evolution

### Piecewise-Bézier C 1 interpolation on Riemannian manifolds with application to 2D shape morphing

Piecewise-Bézier C 1 interpolation on Riemannian manifolds with application to 2D shape morphing Pierre-Yves Gousenbourger Department of Mathematical Engineering ICTEAM Institute Université catholique

### RESEARCH ARTICLE. A strategy of finding an initial active set for inequality constrained quadratic programming problems

Optimization Methods and Software Vol. 00, No. 00, July 200, 8 RESEARCH ARTICLE A strategy of finding an initial active set for inequality constrained quadratic programming problems Jungho Lee Computer

### Robotics 1 Inverse kinematics

Robotics 1 Inverse kinematics Prof. Alessandro De Luca Robotics 1 1 Inverse kinematics what are we looking for? direct kinematics is always unique; how about inverse kinematics for this 6R robot? Robotics

### AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING

AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING XIAO WANG AND HONGCHAO ZHANG Abstract. In this paper, we propose an Augmented Lagrangian Affine Scaling (ALAS) algorithm for general

### A PROJECTED HESSIAN GAUSS-NEWTON ALGORITHM FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS AND INEQUALITIES

IJMMS 25:6 2001) 397 409 PII. S0161171201002290 http://ijmms.hindawi.com Hindawi Publishing Corp. A PROJECTED HESSIAN GAUSS-NEWTON ALGORITHM FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS AND INEQUALITIES

### Single layer NN. Neuron Model

Single layer NN We consider the simple architecture consisting of just one neuron. Generalization to a single layer with more neurons as illustrated below is easy because: M M The output units are independent

### Part 4: IIR Filters Optimization Approach. Tutorial ISCAS 2007

Part 4: IIR Filters Optimization Approach Tutorial ISCAS 2007 Copyright 2007 Andreas Antoniou Victoria, BC, Canada Email: aantoniou@ieee.org July 24, 2007 Frame # 1 Slide # 1 A. Antoniou Part4: IIR Filters

### Exercise Solutions to Functional Analysis

Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n

### Finite Convergence for Feasible Solution Sequence of Variational Inequality Problems

Mathematical and Computational Applications Article Finite Convergence for Feasible Solution Sequence of Variational Inequality Problems Wenling Zhao *, Ruyu Wang and Hongxiang Zhang School of Science,

### Affine iterations on nonnegative vectors

Affine iterations on nonnegative vectors V. Blondel L. Ninove P. Van Dooren CESAME Université catholique de Louvain Av. G. Lemaître 4 B-348 Louvain-la-Neuve Belgium Introduction In this paper we consider

### Comparing Two Matrices by Means of Isometric Projections

Comparing Two Matrices by Means of Isometric Projections T. Cason, P.-A. Absil, P. Van Dooren Université catholique de Louvain, Department of Mathematical Engineering Bâtiment Euler, avenue Georges Lemaître

### DISCUSSION PAPER 2011/70. Stochastic first order methods in smooth convex optimization. Olivier Devolder

011/70 Stochastic first order methods in smooth convex optimization Olivier Devolder DISCUSSION PAPER Center for Operations Research and Econometrics Voie du Roman Pays, 34 B-1348 Louvain-la-Neuve Belgium

### Visual SLAM Tutorial: Bundle Adjustment

Visual SLAM Tutorial: Bundle Adjustment Frank Dellaert June 27, 2014 1 Minimizing Re-projection Error in Two Views In a two-view setting, we are interested in finding the most likely camera poses T1 w

### Local convexity on smooth manifolds 1,2,3. T. Rapcsák 4

Local convexity on smooth manifolds 1,2,3 T. Rapcsák 4 1 The paper is dedicated to the memory of Guido Stampacchia. 2 The author wishes to thank F. Giannessi for the advice. 3 Research was supported in

### ON THE CONSTRUCTION OF DUALLY FLAT FINSLER METRICS

Huang, L., Liu, H. and Mo, X. Osaka J. Math. 52 (2015), 377 391 ON THE CONSTRUCTION OF DUALLY FLAT FINSLER METRICS LIBING HUANG, HUAIFU LIU and XIAOHUAN MO (Received April 15, 2013, revised November 14,

### On the convergence properties of the modified Polak Ribiére Polyak method with the standard Armijo line search

ANZIAM J. 55 (E) pp.e79 E89, 2014 E79 On the convergence properties of the modified Polak Ribiére Polyak method with the standard Armijo line search Lijun Li 1 Weijun Zhou 2 (Received 21 May 2013; revised

### E5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization

E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained

### Computing Laser Beam Paths in Optical Cavities:

arxiv manuscript No. (will be inserted by the editor) Computing Laser Beam Paths in Optical Cavities: An Approach based on Geometric Newton Method Davide Cuccato Alessandro Saccon Antonello Ortolan Alessandro

### The Nearest Doubly Stochastic Matrix to a Real Matrix with the same First Moment

he Nearest Doubly Stochastic Matrix to a Real Matrix with the same First Moment William Glunt 1, homas L. Hayden 2 and Robert Reams 2 1 Department of Mathematics and Computer Science, Austin Peay State

### DUAL REGULARIZED TOTAL LEAST SQUARES SOLUTION FROM TWO-PARAMETER TRUST-REGION ALGORITHM. Geunseop Lee

J. Korean Math. Soc. 0 (0), No. 0, pp. 1 0 https://doi.org/10.4134/jkms.j160152 pissn: 0304-9914 / eissn: 2234-3008 DUAL REGULARIZED TOTAL LEAST SQUARES SOLUTION FROM TWO-PARAMETER TRUST-REGION ALGORITHM

### Image Registration Lecture 2: Vectors and Matrices

Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this

### Optimal Scaling of Companion Pencils for the QZ-Algorithm

Optimal Scaling of Companion Pencils for the QZ-Algorithm D Lemonnier, P Van Dooren 1 Introduction Computing roots of a monic polynomial may be done by computing the eigenvalues of the corresponding companion

### McMaster University. Advanced Optimization Laboratory. Title: A Proximal Method for Identifying Active Manifolds. Authors: Warren L.

McMaster University Advanced Optimization Laboratory Title: A Proximal Method for Identifying Active Manifolds Authors: Warren L. Hare AdvOl-Report No. 2006/07 April 2006, Hamilton, Ontario, Canada A Proximal

### Vector Space Concepts

Vector Space Concepts ECE 174 Introduction to Linear & Nonlinear Optimization Ken Kreutz-Delgado ECE Department, UC San Diego Ken Kreutz-Delgado (UC San Diego) ECE 174 Fall 2016 1 / 25 Vector Space Theory

### Subset Selection. Ilse Ipsen. North Carolina State University, USA

Subset Selection Ilse Ipsen North Carolina State University, USA Subset Selection Given: real or complex matrix A integer k Determine permutation matrix P so that AP = ( A 1 }{{} k A 2 ) Important columns

### Second Order Optimization Algorithms I

Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The

### Optimal Newton-type methods for nonconvex smooth optimization problems

Optimal Newton-type methods for nonconvex smooth optimization problems Coralia Cartis, Nicholas I. M. Gould and Philippe L. Toint June 9, 20 Abstract We consider a general class of second-order iterations

### Optimization Methods for Circuit Design

Technische Universität München Department of Electrical Engineering and Information Technology Institute for Electronic Design Automation Optimization Methods for Circuit Design Compendium H. Graeb Version

### ANALYSIS OF NONLINEAR PARTIAL LEAST SQUARES ALGORITHMS

ANALYSIS OF NONLINEAR PARIAL LEAS SQUARES ALGORIHMS S. Kumar U. Kruger,1 E. B. Martin, and A. J. Morris Centre of Process Analytics and Process echnology, University of Newcastle, NE1 7RU, U.K. Intelligent

### Solving A Low-Rank Factorization Model for Matrix Completion by A Nonlinear Successive Over-Relaxation Algorithm

Solving A Low-Rank Factorization Model for Matrix Completion by A Nonlinear Successive Over-Relaxation Algorithm Zaiwen Wen, Wotao Yin, Yin Zhang 2010 ONR Compressed Sensing Workshop May, 2010 Matrix Completion

### 2.13 Linearization and Differentials

Linearization and Differentials Section Notes Page Sometimes we can approimate more complicated functions with simpler ones These would give us enough accuracy for certain situations and are easier to

### A Reservoir Sampling Algorithm with Adaptive Estimation of Conditional Expectation

A Reservoir Sampling Algorithm with Adaptive Estimation of Conditional Expectation Vu Malbasa and Slobodan Vucetic Abstract Resource-constrained data mining introduces many constraints when learning from

### An improved generalized Newton method for absolute value equations

DOI 10.1186/s40064-016-2720-5 RESEARCH Open Access An improved generalized Newton method for absolute value equations Jingmei Feng 1,2* and Sanyang Liu 1 *Correspondence: fengjingmeilq@hotmail.com 1 School

### SOME REMARKS ON THE TOPOLOGY OF HYPERBOLIC ACTIONS OF R n ON n-manifolds

SOME REMARKS ON THE TOPOLOGY OF HYPERBOLIC ACTIONS OF R n ON n-manifolds DAMIEN BOULOC Abstract. This paper contains some more results on the topology of a nondegenerate action of R n on a compact connected

### A Line search Multigrid Method for Large-Scale Nonlinear Optimization

A Line search Multigrid Method for Large-Scale Nonlinear Optimization Zaiwen Wen Donald Goldfarb Department of Industrial Engineering and Operations Research Columbia University 2008 Siam Conference on

### Deformation groupoids and index theory

Deformation groupoids and index theory Karsten Bohlen Leibniz Universität Hannover GRK Klausurtagung, Goslar September 24, 2014 Contents 1 Groupoids 2 The tangent groupoid 3 The analytic and topological

### Nonlinear Coordinate Transformations for Unconstrained Optimization I. Basic Transformations

Nonlinear Coordinate Transformations for Unconstrained Optimization I. Basic Transformations TIBOR CSENDES Kalmár Laboratory, József Attila University, Szeged, Hungary and TAMÁS RAPCSÁK Computer and Automation

### Reweighted Nuclear Norm Minimization with Application to System Identification

Reweighted Nuclear Norm Minimization with Application to System Identification Karthi Mohan and Maryam Fazel Abstract The matrix ran minimization problem consists of finding a matrix of minimum ran that

### Strictly convex functions on complete Finsler manifolds

Proc. Indian Acad. Sci. (Math. Sci.) Vol. 126, No. 4, November 2016, pp. 623 627. DOI 10.1007/s12044-016-0307-2 Strictly convex functions on complete Finsler manifolds YOE ITOKAWA 1, KATSUHIRO SHIOHAMA

### Newton s Method. Ryan Tibshirani Convex Optimization /36-725

Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x

### Chapter 3 Numerical Methods

Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2

### Low rank matrix completion by alternating steepest descent methods

SIAM J. IMAGING SCIENCES Vol. xx, pp. x c xxxx Society for Industrial and Applied Mathematics x x Low rank matrix completion by alternating steepest descent methods Jared Tanner and Ke Wei Abstract. Matrix

### Characterization of Gradient Dominance and Regularity Conditions for Neural Networks

Characterization of Gradient Dominance and Regularity Conditions for Neural Networks Yi Zhou Ohio State University Yingbin Liang Ohio State University Abstract zhou.1172@osu.edu liang.889@osu.edu The past

### ECS550NFB Introduction to Numerical Methods using Matlab Day 2

ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Root-finding: find x that solves

### FIXED POINT ITERATIONS

FIXED POINT ITERATIONS MARKUS GRASMAIR 1. Fixed Point Iteration for Non-linear Equations Our goal is the solution of an equation (1) F (x) = 0, where F : R n R n is a continuous vector valued mapping in

### A class of Smoothing Method for Linear Second-Order Cone Programming

Columbia International Publishing Journal of Advanced Computing (13) 1: 9-4 doi:1776/jac1313 Research Article A class of Smoothing Method for Linear Second-Order Cone Programming Zhuqing Gui *, Zhibin

### Low-Rank Factorization Models for Matrix Completion and Matrix Separation

for Matrix Completion and Matrix Separation Joint work with Wotao Yin, Yin Zhang and Shen Yuan IPAM, UCLA Oct. 5, 2010 Low rank minimization problems Matrix completion: find a low-rank matrix W R m n so

### A New Trust Region Algorithm Using Radial Basis Function Models

A New Trust Region Algorithm Using Radial Basis Function Models Seppo Pulkkinen University of Turku Department of Mathematics July 14, 2010 Outline 1 Introduction 2 Background Taylor series approximations

### On Dwell Time Minimization for Switched Delay Systems: Free-Weighting Matrices Method

On Dwell Time Minimization for Switched Delay Systems: Free-Weighting Matrices Method Ahmet Taha Koru Akın Delibaşı and Hitay Özbay Abstract In this paper we present a quasi-convex minimization method

### Stratification of 3 3 Matrices

Stratification of 3 3 Matrices Meesue Yoo & Clay Shonkwiler March 2, 2006 1 Warmup with 2 2 Matrices { Best matrices of rank 2} = O(2) S 3 ( 2) { Best matrices of rank 1} S 3 (1) 1.1 Viewing O(2) S 3 (

### Computational Methods. Least Squares Approximation/Optimization

Computational Methods Least Squares Approximation/Optimization Manfred Huber 2011 1 Least Squares Least squares methods are aimed at finding approximate solutions when no precise solution exists Find the

### The non-convex Burer Monteiro approach works on smooth semidefinite programs

The non-convex Burer Monteiro approach works on smooth semidefinite programs Nicolas Boumal Department of Mathematics Princeton University nboumal@math.princeton.edu Vladislav Voroninski Department of

### CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS

MATHEMATICS OF COMPUTATION Volume 68, Number 228, Pages 165 1613 S 25-5718(99)1135-7 Article electronically published on March 1, 1999 CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS BENEDETTA MORINI Abstract.

### Chapter 4. Unconstrained optimization

Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file

### Multidisciplinary System Design Optimization (MSDO)

Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential

### Dynamic Programming with Hermite Interpolation

Dynamic Programming with Hermite Interpolation Yongyang Cai Hoover Institution, 424 Galvez Mall, Stanford University, Stanford, CA, 94305 Kenneth L. Judd Hoover Institution, 424 Galvez Mall, Stanford University,

### Optimization methods

Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,

### Generalized line criterion for Gauss-Seidel method

Computational and Applied Mathematics Vol. 22, N. 1, pp. 91 97, 2003 Copyright 2003 SBMAC Generalized line criterion for Gauss-Seidel method M.V.P. GARCIA 1, C. HUMES JR. 2 and J.M. STERN 2 1 Department

### Exam February h

Master 2 Mathématiques et Applications PUF Ho Chi Minh Ville 2009/10 Viscosity solutions, HJ Equations and Control O.Ley (INSA de Rennes) Exam February 2010 3h Written-by-hands documents are allowed. Printed

### SOLVING SYSTEMS OF NONLINEAR EQUATIONS USING A GLOBALLY CONVERGENT OPTIMIZATION ALGORITHM

Transaction on Evolutionary Algorithm and Nonlinear Optimization ISSN: 2229-87 Online Publication, June 2012 www.pcoglobal.com/gjto.htm NG-O21/GJTO SOLVING SYSTEMS OF NONLINEAR EQUATIONS USING A GLOBALLY

### Newton s Method. Javier Peña Convex Optimization /36-725

Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and

Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties

### Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

### WHEN studying distributed simulations of power systems,

1096 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL 21, NO 3, AUGUST 2006 A Jacobian-Free Newton-GMRES(m) Method with Adaptive Preconditioner and Its Application for Power Flow Calculations Ying Chen and Chen

### Secrets of Matrix Factorization: Supplementary Document

Secrets of Matrix Factorization: Supplementary Document Je Hyeong Hong University of Cambridge jhh37@cam.ac.uk Andrew Fitzgibbon Microsoft, Cambridge, UK awf@microsoft.com Abstract This document consists

### LECTURE 22: THE CRITICAL POINT THEORY OF DISTANCE FUNCTIONS

LECTURE : THE CRITICAL POINT THEORY OF DISTANCE FUNCTIONS 1. Critical Point Theory of Distance Functions Morse theory is a basic tool in differential topology which also has many applications in Riemannian

### Neural Networks Lecture 4: Radial Bases Function Networks

Neural Networks Lecture 4: Radial Bases Function Networks H.A Talebi Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Winter 2011. A. Talebi, Farzaneh Abdollahi