RankConstrainted Optimization: A Riemannian Manifold Approach

 Damian Dawson
 9 months ago
 Views:
Transcription
1 RanConstrainted Optimization: A Riemannian Manifold Approach Guifang Zhou1, Wen Huang2, Kyle A. Gallivan1, Paul Van Dooren2, P.A. Absil2 1 Florida State University  Department of Mathematics 1017 Academic Way, Tallahassee, FL , US 2 Universite catholique de Louvain  ICTEAM Institute Avenue G. Lemaı tre 4, 148 LouvainlaNeuve, Belgium Abstract. This paper presents an algorithm that solves optimization problems on a matrix manifold M Rm n with an additional ran inequality constraint. New geometric objects are defined to facilitate efficiently finding a suitable ran. The convergence properties of the algorithm are given and a weighted lowran approximation problem is used to illustrate the efficiency and effectiveness of the algorithm. 1 Introduction We consider lowran optimization problems that combine a ran inequality constraint with a matrix manifold constraint: min f (X), X M (1) where min(m, n), M = {X M ran(x) } and M is a submanifold of Rm n. Typical choices for M are: the entire set Rm n, a sphere, symmetric matrices, elliptopes, spectrahedrons. Recently, optimization on manifolds has attracted significant attention. Attempts have been made to understand problem (1) by considering a related but simpler problem minx Rm n f (X), where Rm n = {X Rm n ran(x) = }, (see e.g., [1, 2]). Since Rm n is a submanifold of Rm n of dimension (m+n ), the simpler problem can be solved using techniques from Riemannian optimization [] applied to matrix manifolds. However, a disadvantage is that the manifold Rm n is not closed in Rm n, which complicates considerably the conver gence analysis and performance of an iteration. Very recently a more global view of a projected linesearch method on Rm n = m n {X R ran(x) } along with a convergence analysis has been developed in [4]. In [], the results of [4] are exploited to propose an algorithm that successively increases the ran by a given constant. Its convergence result to critical This research was supported by the National Science Foundation under grant NSF This paper presents research results of the Belgian Networ DYSCO (Dynamical Systems, Control, and Optimization), funded by the Interuniversity Attraction Poles Programme initiated by the Belgian Science Policy Office. This wor was supported by grant FNRS PDR T
2 points can be deduced from [4, Th..9]; it relies on the assumption, often satisfied in practice, that the limit points have ran. Under this assumption, a linesearch method on Rm n is ultimately the same as a linesearch method on m n R. In this paper, we develop an efficient algorithm for the optimization problems. The main contributions are as follows: First, we generalize the admissible set from Rm n to M. Second, the proposed algorithm solves a ran inequal ity constrained problem while finding a suitable ran for approximation of the locally optimal solution. Third, the proposed algorithm successively increases or decreases the ran by an adaptivelychosen amount; therefore, the ran is updating appropriately, which avoids excessive increase of ran in order to save computations and storage. Finally, theoretical convergence results are given. The rest of this paper is organized as follows. A tangent cone descent algorithm is given in Section 2. Some theoretical results are presented in Section without proofs. Finally, a performance study is provided on the weighted lowran problem [6] in Section 4. 2 A Tangent Cone Descent Algorithm The following assumptions are made to hold throughout this paper: (i) The closure of Mr := {X M ran(x) = r} is a subset of or equal to M r. (ii) Mr is a manifold. (iii) The extension of cost function f in (1) on M is welldefined and it is continuously differentiable. The basic idea of the new algorithm is applying Riemannian optimization methods on a fixedran manifold Mr while efficiently and effectively updating the ran r. For each fixedran manifold Mr, efficient and wellunderstood Riemannian optimization algorithms, see e.g., [7, 8, 9], can be used directly. The main issue is to move from one fixedran manifold to another efficiently and effectively. A ran adjustment decision considers the following two functions: one is an extension of f on M, i.e., ff : M R : X 7 ff (X), such that f = ff M, and the other is the restriction of f on a fixedran manifold Mr, i.e., fr : Mr R : X 7 fr (X), such that fr = f Mr. The ran is increased at x Mr if r < and the following two conditions hold simultaneously: Condition I (angle threshold ǫ1 ): tan( (gradff (X), gradfr (X))) > ǫ1 ; Condition II (difference threshold ǫ2 ): gradff (X) gradfr (X) ǫ2, where gradff (X) and gradfr (X) denote the Riemannian gradients of ff and fr at a point X Mr. The parameters ǫ1 and ǫ2 are important for finding the exact/approximate solutions and controlling the computational efficiency of the method. A smaller ǫ1 value maes it easier to increase ran per iteration. The smaller ǫ2 is the stricter the accuracy of the approximate local minimizer required. In particular, convergence to critical points of (1) is obtained if ǫ2 is set to 0. Next, we discuss the details about updating the ran. The tangent cone to M at a point X M, TX M, is equal to the set of all smooth admissible directions for M at X (see e.g., [10]). Note that TX M is a linear space 20
3 for X M but is not for any point X M 1. The tangent cone TX M contains the tangent space TX Mr and also the curves approaching X by points of ran T greater than r, but not beyond, i.e., TX M = TX Mr + {η r NX Mr TX M ran(η r ) r}. The characterization of TX M for M = Rm n has been given in [4]. To obtain the next iteration on M M, a general linesearch method is considered: Xn+1 = R Xn (tn ηxn,r ), where tn is a step size and R is a ranrelated retraction defined below in Definition 1. Definition 1 (Ranrelated retraction) Let X Mr. A mapping R X : TX M M is a ranrelated retraction if, ηx TX M, (i) R X (0) = X, (ii) δ > 0 such that [0, δ) t 7 R X (tηx ) is smooth and R X (tηx ) Mr for all t [0, δ), where r is the integer such that r r, ηx TX M r, and d ηx / TX Mr 1, (iii) dt R X (tηx ) t=0 = ηx. It can be shown that a ranrelated retraction always exists. Note R X is not necessarily S a retraction on M since it may not be smooth on the tangent bundle TM := X M TX M. The modified Riemannian optimization algorithm is setched as Alg. 1. Generally speaing, η in Step 7 in Alg. 1 is not unique and any one can be used to change to ran r. The ey to efficiency of the algorithm is to choose r correctly and then choose an appropriate η. In [4], a similar idea is developed for M = Rm n and r =. The authors define the search direction as the projection of the negative gradient onto the tangent cone. If M = Rm n and r =, the choice of η in Step 7 is equivalent to the definition in [4, Corollary.]. Main Theoretical Results Suppose the cost function f is continuously differentiable and the algorithm chosen in Step 2 of Alg. 1 has the property of global convergence to critical points of fr, we give the following main results. The proofs are omitted due to space constraints (see [1]). Theorem 1 (Global Result) The sequence q {Xn } generated by Alg. 1 satisfies 1 + ǫ12 ǫ2. lim inf n PTXn M (gradff (Xn )) 1 2 Theorem 2 (Local Result) Let ff be a C function and X be a nondegenerate minimizer of ff with ran r. Suppose ǫ2 > 0. Denote the sequence of iterates generated by Algorithm 1 by {Xn }. There exists a neighborhood of X, UX, such that if D = {X M f (X) f (X0 )} UX ; D is compact; fˆ : TM R : ξ 7 ff R (ξ) is a radially LC 1 function with sufficient large βrl defined in [, Definition 7.4.1] such that for 1 any X, Y D, R X (Y ) < βrl, then there exists N > 0 such that n > N ran(xn ) = r and Xn UX. 21
4 Algorithm 1 Modified Riemannian Optimization Algorithm 1: 2: : 4: : 6: 7: 8: 9: 10: 11: 12: 1: 14: 1: 16: 17: 4 for n = 0, 1, 2,... do Apply a Riemannian algorithm (e.g. one of GenRTR [7], RBFGS [8, 9, 11], RTRSR1 [9, 12]) to approximately optimize f over Mr with initial point Xn and stop at X n Mr, where either gradfr (X n ) < ǫ (flag 1) or X n is close to M r 1 (flag 0); if flag = 1 then if Both Conditions I and II are satisfied then Set r and η to be r and gradfr (X n ) respectively; while gradff (X n ) η > ǫ21 η do Set r to be r + 1 and η argminη TX M r gradff (X n ) ηf ; end while Obtain Xn+1 by applying an Armijotype line search algorithm along η using a ranrelated retraction; else If ǫ is small enough, stop. Otherwise, ǫ τ ǫ, where τ (0, 1); end if else {flag = 0} If the ran has not been increased on any previous iteration, reduce the ran of X n based on truncation while eeping the function value decrease, update r, obtain next iterate Xn+1 ; Else reduce the ran of X n such that the next iterate Xn+1 satisfies f (Xn+1 ) f (Xi ) c(f (Xi+1 ) f (Xi )), where i is such that the latest ran increase was from Xi to Xi+1, 0 < c < 1. Set r to be the ran of Xn+1 ; end if end for Application We illustrate the effectiveness of Alg. 1 for the weighted lowran approximation problem: min A X2W, f (X) = A X2W = vec{a X}T W vec{a X} X M (2) where M = R80 10, A is given, W R is a positive definite symmetric weighting matrix and vec{a} denotes the vectorized form of A, i.e., a vector constructed by stacing the consecutive columns of A in one vector. The matrix A is generated by A1 AT2 Rm n, where A1 R80, A2 R10. W = U ΣU T, where U R is a random orthogonal matrix generated by Matlab s QR and RAND. The mn singular values of the weighting matrix are generated by Matlab function LOGSPACE with condition number 100 and multiplying, elementwise, by a uniform distribution matrix on the interval [0., 1.]. Three values of are considered, one less than the true ran, one equal to the 22
5 = = =7 method Alg. 1 DMM SULS APM Alg. 1 DMM SULS APM Alg. 1 DMM SULS APM ran.02(99/100) 7(0/100) 7(0/100) f W Rel Err( A X AW ) time(sec) Table 1: Method comparisons. The number in the parenthesis indicates the fraction of experiments where the numerical ran (number of singular values greater than 10 8 ) found by the algorithm equals the true ran. The subscript ± indicates a scale of 10±. Alg. 1 and SULS, are stopped when the norm of the final gradient on the fixedran manifold over the norm of initial full gradient is less than 10 8 while DMM and APM are stopped when the norm of final gradient over the norm of initial gradient is less than true ran and one greater than the true ran. Four algorithms are compared: Alg. 1, DMM [14], SULS [4] and APM [6]. The initial point of Alg. 1 and SULS is a randomly generated ran1 matrix and the initial points of DMM and APM are randomly generated nby(n ) and mby(m ) matrices respectively. Results shown in Table 1 are the average of 100 runs for different data matrices R, weighting matrices W and initial points. The results show that Alg. 1 with ǫ1 =, ǫ2 = 10 4 (GenRTR is used as the inner algorithm, when the sizes become larger, limitedmemory RBFGS [9, 11], for example, may be a better choice) has significant advantages compared with the other methods. It achieves good accuracy in the approximation with less computational time. Furthermore, for Alg. 1, all Riemannian objects, their updates and Xn are based on an efficient three factor, U, D, V, representation, and the singular values are immediately available while, for the other three methods, an additional SVD is required. References [1] B. Mishra, G. Meyer, F. Bach, and R. Sepulchre. Lowran optimization with trace norm penalty. SIAM Journal on Optimization, 2(4): , 201. [2] M. Journe e, F. Bach, P.A. Absil, and R. Sepulchre. Lowran optimization on the cone of positive semidefinite matrices. SIAM Journal on Optimiza 2
6 tion, 20():227 21, [] P.A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, [4] R. Schneider and A. Uschmajew. Convergence results for projected linesearch methods on varieties of lowran matrices via Lojasiewicz inequality. ArXiv eprints, February [] A. Uschmajew and B. Vandereycen. Linesearch methods and ran increase on lowran matrix varieties. In Proceedings of the 2014 International Symposium on Nonlinear Theory and its Applications (NOLTA2014), [6] W.S. Lu, S.C. Pei, and P.H. Wang. Weighted lowran approximation of general complex matrices and its application in the design of 2d digital filters. IEEE Transactions on Circuits and Systems I, 44:60 6, [7] P.A. Absil, C. G. Baer, and Kyle A. Gallivan. Trustregion methods on Riemannian manifolds. Foundations of Computational Mathematics, 7():0 0, [8] W. Ring and B. Wirth. Optimization methods on Riemannian manifolds and their application to shape space. SIAM Journal on Optimization, 22(2):96 627, January [9] W. Huang. Optimization algorithms on Riemannian manifolds with applications. PhD thesis, Florida State University, 201. [10] Donal B. O Shea and Leslie C. Wilson. Limits of tangent spaces to real surfaces. American Journal of Mathematics, 126():pp , [11] Wen Huang, K. A. Gallivan, and P.A. Absil. A Broyden class of quasinewton methods for Riemannian optimization. UCLINMA , U.C.Louvain, Submitted for publication, 201. [12] W. Huang, P.A. Absil, and K. A. Gallivan. A Riemannian symmetric ranone trustregion method. Mathematical Programming, February doi: /s [1] Guifang Zhou. Rancontrained optimization: A Riemannian approach. PhD thesis, Florida State University. in preparation. [14] I. Brace and J. H. Manton. An improved BFGSonmanifold algorithm for computing lowran weighted approximations. In Proceedings of 17th International Symposium on Mathematical Theory of Networs and Systems, pages ,
Optimization Algorithms on Riemannian Manifolds with Applications
Optimization Algorithms on Riemannian Manifolds with Applications Wen Huang Coadvisor: Kyle A. Gallivan Coadvisor: PierreAntoine Absil Florida State University Catholic University of Louvain November
More informationThe nonsmooth Newton method on Riemannian manifolds
The nonsmooth Newton method on Riemannian manifolds C. Lageman, U. Helmke, J.H. Manton 1 Introduction Solving nonlinear equations in Euclidean space is a frequently occurring problem in optimization and
More informationH 2 optimal model reduction of MIMO systems
H 2 optimal model reduction of MIMO systems P. Van Dooren K. A. Gallivan P.A. Absil Abstract We consider the problem of approximating a p m rational transfer function Hs of high degree by another p m
More informationMaxPlanckInstitut für Mathematik in den Naturwissenschaften Leipzig
MaxPlanckInstitut für Mathematik in den Naturwissenschaften Leipzig Convergence analysis of Riemannian GaussNewton methods and its connection with the geometric condition number by Paul Breiding and
More informationA variable projection method for block term decomposition of higherorder tensors
A variable projection method for block term decomposition of higherorder tensors Guillaume Olikier 1, P.A. Absil 1, and Lieven De Lathauwer 2 1 Université catholique de Louvain  ICTEAM Institute B1348
More informationGeneralized Shifted Inverse Iterations on Grassmann Manifolds 1
Proceedings of the Sixteenth International Symposium on Mathematical Networks and Systems (MTNS 2004), Leuven, Belgium Generalized Shifted Inverse Iterations on Grassmann Manifolds 1 J. Jordan α, P.A.
More informationQuasiNewton Methods
QuasiNewton Methods Werner C. Rheinboldt These are excerpts of material relating to the boos [OR00 and [Rhe98 and of writeups prepared for courses held at the University of Pittsburgh. Some further references
More informationCONVEX OPTIMIZATION OVER POSITIVE POLYNOMIALS AND FILTER DESIGN. Y. Genin, Y. Hachez, Yu. Nesterov, P. Van Dooren
CONVEX OPTIMIZATION OVER POSITIVE POLYNOMIALS AND FILTER DESIGN Y. Genin, Y. Hachez, Yu. Nesterov, P. Van Dooren CESAME, Université catholique de Louvain Bâtiment Euler, Avenue G. Lemaître 46 B1348 LouvainlaNeuve,
More informationOnline Learning in the Manifold of LowRank Matrices
Online Learning in the Manifold of LowRan Matrices Uri Shalit, Daphna Weinshall Computer Science Dept. and ICNC The Hebrew University of Jerusalem uri.shalit@mail.huji.ac.il daphna@cs.huji.ac.il Gal Chechi
More informationNavigation and Obstacle Avoidance via Backstepping for Mechanical Systems with Drift in the Closed Loop
Navigation and Obstacle Avoidance via Backstepping for Mechanical Systems with Drift in the Closed Loop Jan Maximilian Montenbruck, Mathias Bürger, Frank Allgöwer Abstract We study backstepping controllers
More informationIntroduction to Nonlinear Optimization Paul J. Atzberger
Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,
More informationarxiv: v4 [math.oc] 5 Jan 2016
Restarted SGD: Beating SGD without Smoothness and/or Strong Convexity arxiv:151.03107v4 [math.oc] 5 Jan 016 Tianbao Yang, Qihang Lin Department of Computer Science Department of Management Sciences The
More informationYou should be able to...
Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set
More informationRanking from Crowdsourced Pairwise Comparisons via Matrix Manifold Optimization
Ranking from Crowdsourced Pairwise Comparisons via Matrix Manifold Optimization Jialin Dong ShanghaiTech University 1 Outline Introduction FourVignettes: System Model and Problem Formulation Problem Analysis
More informationLinear Regression (continued)
Linear Regression (continued) Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 6, 2017 1 / 39 Outline 1 Administration 2 Review of last lecture 3 Linear regression
More informationNONLINEAR CLASSIFICATION AND REGRESSION. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition
NONLINEAR CLASSIFICATION AND REGRESSION Nonlinear Classification and Regression: Outline 2 MultiLayer Perceptrons The BackPropagation Learning Algorithm Generalized Linear Models Radial Basis Function
More informationSupplementary Notes for W. Rudin: Principles of Mathematical Analysis
Supplementary Notes for W. Rudin: Principles of Mathematical Analysis SIGURDUR HELGASON In 8.00B it is customary to cover Chapters 7 in Rudin s book. Experience shows that this requires careful planning
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationHAIYUN ZHOU, RAVI P. AGARWAL, YEOL JE CHO, AND YONG SOO KIM
Georgian Mathematical Journal Volume 9 (2002), Number 3, 591 600 NONEXPANSIVE MAPPINGS AND ITERATIVE METHODS IN UNIFORMLY CONVEX BANACH SPACES HAIYUN ZHOU, RAVI P. AGARWAL, YEOL JE CHO, AND YONG SOO KIM
More informationNew Inexact Line Search Method for Unconstrained Optimization 1,2
journal of optimization theory and applications: Vol. 127, No. 2, pp. 425 446, November 2005 ( 2005) DOI: 10.1007/s1095700565536 New Inexact Line Search Method for Unconstrained Optimization 1,2 Z.
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of nonrigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationOptimization Methods for Machine Learning
Optimization Methods for Machine Learning Sathiya Keerthi Microsoft Talks given at UC Santa Cruz February 2123, 2017 The slides for the talks will be made available at: http://www.keerthis.com/ Introduction
More informationMath 411 Preliminaries
Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector
More informationMultilayer Perceptron
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Single Perceptron 3 Boolean Function Learning 4
More informationThe Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1
October 2003 The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1 by Asuman E. Ozdaglar and Dimitri P. Bertsekas 2 Abstract We consider optimization problems with equality,
More informationA Descent Method for Equality and Inequality Constrained Multiobjective Optimization Problems
A Descent Method for Equality and Inequality Constrained Multiobjective Optimization Problems arxiv:1712.03005v2 [math.oc] 11 Dec 2017 Bennet Gebken 1, Sebastian Peitz 1, and Michael Dellnitz 1 1 Department
More informationFast Nonnegative Matrix Factorization with Rankone ADMM
Fast Nonnegative Matrix Factorization with Rankone Dongjin Song, David A. Meyer, Martin Renqiang Min, Department of ECE, UCSD, La Jolla, CA, 90930409 dosong@ucsd.edu Department of Mathematics, UCSD,
More informationChap. 1. Some Differential Geometric Tools
Chap. 1. Some Differential Geometric Tools 1. Manifold, Diffeomorphism 1.1. The Implicit Function Theorem ϕ : U R n R n p (0 p < n), of class C k (k 1) x 0 U such that ϕ(x 0 ) = 0 rank Dϕ(x) = n p x U
More informationCubic regularization of Newton s method for convex problems with constraints
CORE DISCUSSION PAPER 006/39 Cubic regularization of Newton s method for convex problems with constraints Yu. Nesterov March 31, 006 Abstract In this paper we derive efficiency estimates of the regularized
More informationA NUMERICAL METHOD TO SOLVE A QUADRATIC CONSTRAINED MAXIMIZATION
A NUMERICAL METHOD TO SOLVE A QUADRATIC CONSTRAINED MAXIMIZATION ALIREZA ESNA ASHARI, RAMINE NIKOUKHAH, AND STEPHEN L. CAMPBELL Abstract. The problem of maximizing a quadratic function subject to an ellipsoidal
More informationSteepest descent method on a Riemannian manifold: the convex case
Steepest descent method on a Riemannian manifold: the convex case Julien Munier Abstract. In this paper we are interested in the asymptotic behavior of the trajectories of the famous steepest descent evolution
More informationPiecewiseBézier C 1 interpolation on Riemannian manifolds with application to 2D shape morphing
PiecewiseBézier C 1 interpolation on Riemannian manifolds with application to 2D shape morphing PierreYves Gousenbourger Department of Mathematical Engineering ICTEAM Institute Université catholique
More informationRESEARCH ARTICLE. A strategy of finding an initial active set for inequality constrained quadratic programming problems
Optimization Methods and Software Vol. 00, No. 00, July 200, 8 RESEARCH ARTICLE A strategy of finding an initial active set for inequality constrained quadratic programming problems Jungho Lee Computer
More informationRobotics 1 Inverse kinematics
Robotics 1 Inverse kinematics Prof. Alessandro De Luca Robotics 1 1 Inverse kinematics what are we looking for? direct kinematics is always unique; how about inverse kinematics for this 6R robot? Robotics
More informationAN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING
AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING XIAO WANG AND HONGCHAO ZHANG Abstract. In this paper, we propose an Augmented Lagrangian Affine Scaling (ALAS) algorithm for general
More informationA PROJECTED HESSIAN GAUSSNEWTON ALGORITHM FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS AND INEQUALITIES
IJMMS 25:6 2001) 397 409 PII. S0161171201002290 http://ijmms.hindawi.com Hindawi Publishing Corp. A PROJECTED HESSIAN GAUSSNEWTON ALGORITHM FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS AND INEQUALITIES
More informationSingle layer NN. Neuron Model
Single layer NN We consider the simple architecture consisting of just one neuron. Generalization to a single layer with more neurons as illustrated below is easy because: M M The output units are independent
More informationPart 4: IIR Filters Optimization Approach. Tutorial ISCAS 2007
Part 4: IIR Filters Optimization Approach Tutorial ISCAS 2007 Copyright 2007 Andreas Antoniou Victoria, BC, Canada Email: aantoniou@ieee.org July 24, 2007 Frame # 1 Slide # 1 A. Antoniou Part4: IIR Filters
More informationExercise Solutions to Functional Analysis
Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n
More informationFinite Convergence for Feasible Solution Sequence of Variational Inequality Problems
Mathematical and Computational Applications Article Finite Convergence for Feasible Solution Sequence of Variational Inequality Problems Wenling Zhao *, Ruyu Wang and Hongxiang Zhang School of Science,
More informationAffine iterations on nonnegative vectors
Affine iterations on nonnegative vectors V. Blondel L. Ninove P. Van Dooren CESAME Université catholique de Louvain Av. G. Lemaître 4 B348 LouvainlaNeuve Belgium Introduction In this paper we consider
More informationComparing Two Matrices by Means of Isometric Projections
Comparing Two Matrices by Means of Isometric Projections T. Cason, P.A. Absil, P. Van Dooren Université catholique de Louvain, Department of Mathematical Engineering Bâtiment Euler, avenue Georges Lemaître
More informationDISCUSSION PAPER 2011/70. Stochastic first order methods in smooth convex optimization. Olivier Devolder
011/70 Stochastic first order methods in smooth convex optimization Olivier Devolder DISCUSSION PAPER Center for Operations Research and Econometrics Voie du Roman Pays, 34 B1348 LouvainlaNeuve Belgium
More informationVisual SLAM Tutorial: Bundle Adjustment
Visual SLAM Tutorial: Bundle Adjustment Frank Dellaert June 27, 2014 1 Minimizing Reprojection Error in Two Views In a twoview setting, we are interested in finding the most likely camera poses T1 w
More informationLocal convexity on smooth manifolds 1,2,3. T. Rapcsák 4
Local convexity on smooth manifolds 1,2,3 T. Rapcsák 4 1 The paper is dedicated to the memory of Guido Stampacchia. 2 The author wishes to thank F. Giannessi for the advice. 3 Research was supported in
More informationON THE CONSTRUCTION OF DUALLY FLAT FINSLER METRICS
Huang, L., Liu, H. and Mo, X. Osaka J. Math. 52 (2015), 377 391 ON THE CONSTRUCTION OF DUALLY FLAT FINSLER METRICS LIBING HUANG, HUAIFU LIU and XIAOHUAN MO (Received April 15, 2013, revised November 14,
More informationOn the convergence properties of the modified Polak Ribiére Polyak method with the standard Armijo line search
ANZIAM J. 55 (E) pp.e79 E89, 2014 E79 On the convergence properties of the modified Polak Ribiére Polyak method with the standard Armijo line search Lijun Li 1 Weijun Zhou 2 (Received 21 May 2013; revised
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equalityconstrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equalityconstrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationComputing Laser Beam Paths in Optical Cavities:
arxiv manuscript No. (will be inserted by the editor) Computing Laser Beam Paths in Optical Cavities: An Approach based on Geometric Newton Method Davide Cuccato Alessandro Saccon Antonello Ortolan Alessandro
More informationThe Nearest Doubly Stochastic Matrix to a Real Matrix with the same First Moment
he Nearest Doubly Stochastic Matrix to a Real Matrix with the same First Moment William Glunt 1, homas L. Hayden 2 and Robert Reams 2 1 Department of Mathematics and Computer Science, Austin Peay State
More informationDUAL REGULARIZED TOTAL LEAST SQUARES SOLUTION FROM TWOPARAMETER TRUSTREGION ALGORITHM. Geunseop Lee
J. Korean Math. Soc. 0 (0), No. 0, pp. 1 0 https://doi.org/10.4134/jkms.j160152 pissn: 03049914 / eissn: 22343008 DUAL REGULARIZED TOTAL LEAST SQUARES SOLUTION FROM TWOPARAMETER TRUSTREGION ALGORITHM
More informationImage Registration Lecture 2: Vectors and Matrices
Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this
More informationOptimal Scaling of Companion Pencils for the QZAlgorithm
Optimal Scaling of Companion Pencils for the QZAlgorithm D Lemonnier, P Van Dooren 1 Introduction Computing roots of a monic polynomial may be done by computing the eigenvalues of the corresponding companion
More informationMcMaster University. Advanced Optimization Laboratory. Title: A Proximal Method for Identifying Active Manifolds. Authors: Warren L.
McMaster University Advanced Optimization Laboratory Title: A Proximal Method for Identifying Active Manifolds Authors: Warren L. Hare AdvOlReport No. 2006/07 April 2006, Hamilton, Ontario, Canada A Proximal
More informationVector Space Concepts
Vector Space Concepts ECE 174 Introduction to Linear & Nonlinear Optimization Ken KreutzDelgado ECE Department, UC San Diego Ken KreutzDelgado (UC San Diego) ECE 174 Fall 2016 1 / 25 Vector Space Theory
More informationSubset Selection. Ilse Ipsen. North Carolina State University, USA
Subset Selection Ilse Ipsen North Carolina State University, USA Subset Selection Given: real or complex matrix A integer k Determine permutation matrix P so that AP = ( A 1 }{{} k A 2 ) Important columns
More informationSecond Order Optimization Algorithms I
Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The
More informationOptimal Newtontype methods for nonconvex smooth optimization problems
Optimal Newtontype methods for nonconvex smooth optimization problems Coralia Cartis, Nicholas I. M. Gould and Philippe L. Toint June 9, 20 Abstract We consider a general class of secondorder iterations
More informationOptimization Methods for Circuit Design
Technische Universität München Department of Electrical Engineering and Information Technology Institute for Electronic Design Automation Optimization Methods for Circuit Design Compendium H. Graeb Version
More informationANALYSIS OF NONLINEAR PARTIAL LEAST SQUARES ALGORITHMS
ANALYSIS OF NONLINEAR PARIAL LEAS SQUARES ALGORIHMS S. Kumar U. Kruger,1 E. B. Martin, and A. J. Morris Centre of Process Analytics and Process echnology, University of Newcastle, NE1 7RU, U.K. Intelligent
More informationSolving A LowRank Factorization Model for Matrix Completion by A Nonlinear Successive OverRelaxation Algorithm
Solving A LowRank Factorization Model for Matrix Completion by A Nonlinear Successive OverRelaxation Algorithm Zaiwen Wen, Wotao Yin, Yin Zhang 2010 ONR Compressed Sensing Workshop May, 2010 Matrix Completion
More information2.13 Linearization and Differentials
Linearization and Differentials Section Notes Page Sometimes we can approimate more complicated functions with simpler ones These would give us enough accuracy for certain situations and are easier to
More informationA Reservoir Sampling Algorithm with Adaptive Estimation of Conditional Expectation
A Reservoir Sampling Algorithm with Adaptive Estimation of Conditional Expectation Vu Malbasa and Slobodan Vucetic Abstract Resourceconstrained data mining introduces many constraints when learning from
More informationAn improved generalized Newton method for absolute value equations
DOI 10.1186/s4006401627205 RESEARCH Open Access An improved generalized Newton method for absolute value equations Jingmei Feng 1,2* and Sanyang Liu 1 *Correspondence: fengjingmeilq@hotmail.com 1 School
More informationSOME REMARKS ON THE TOPOLOGY OF HYPERBOLIC ACTIONS OF R n ON nmanifolds
SOME REMARKS ON THE TOPOLOGY OF HYPERBOLIC ACTIONS OF R n ON nmanifolds DAMIEN BOULOC Abstract. This paper contains some more results on the topology of a nondegenerate action of R n on a compact connected
More informationA Line search Multigrid Method for LargeScale Nonlinear Optimization
A Line search Multigrid Method for LargeScale Nonlinear Optimization Zaiwen Wen Donald Goldfarb Department of Industrial Engineering and Operations Research Columbia University 2008 Siam Conference on
More informationDeformation groupoids and index theory
Deformation groupoids and index theory Karsten Bohlen Leibniz Universität Hannover GRK Klausurtagung, Goslar September 24, 2014 Contents 1 Groupoids 2 The tangent groupoid 3 The analytic and topological
More informationNonlinear Coordinate Transformations for Unconstrained Optimization I. Basic Transformations
Nonlinear Coordinate Transformations for Unconstrained Optimization I. Basic Transformations TIBOR CSENDES Kalmár Laboratory, József Attila University, Szeged, Hungary and TAMÁS RAPCSÁK Computer and Automation
More informationReweighted Nuclear Norm Minimization with Application to System Identification
Reweighted Nuclear Norm Minimization with Application to System Identification Karthi Mohan and Maryam Fazel Abstract The matrix ran minimization problem consists of finding a matrix of minimum ran that
More informationStrictly convex functions on complete Finsler manifolds
Proc. Indian Acad. Sci. (Math. Sci.) Vol. 126, No. 4, November 2016, pp. 623 627. DOI 10.1007/s1204401603072 Strictly convex functions on complete Finsler manifolds YOE ITOKAWA 1, KATSUHIRO SHIOHAMA
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36725
Newton s Method Ryan Tibshirani Convex Optimization 10725/36725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationLow rank matrix completion by alternating steepest descent methods
SIAM J. IMAGING SCIENCES Vol. xx, pp. x c xxxx Society for Industrial and Applied Mathematics x x Low rank matrix completion by alternating steepest descent methods Jared Tanner and Ke Wei Abstract. Matrix
More informationCharacterization of Gradient Dominance and Regularity Conditions for Neural Networks
Characterization of Gradient Dominance and Regularity Conditions for Neural Networks Yi Zhou Ohio State University Yingbin Liang Ohio State University Abstract zhou.1172@osu.edu liang.889@osu.edu The past
More informationECS550NFB Introduction to Numerical Methods using Matlab Day 2
ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Rootfinding: find x that solves
More informationFIXED POINT ITERATIONS
FIXED POINT ITERATIONS MARKUS GRASMAIR 1. Fixed Point Iteration for Nonlinear Equations Our goal is the solution of an equation (1) F (x) = 0, where F : R n R n is a continuous vector valued mapping in
More informationA class of Smoothing Method for Linear SecondOrder Cone Programming
Columbia International Publishing Journal of Advanced Computing (13) 1: 94 doi:1776/jac1313 Research Article A class of Smoothing Method for Linear SecondOrder Cone Programming Zhuqing Gui *, Zhibin
More informationLowRank Factorization Models for Matrix Completion and Matrix Separation
for Matrix Completion and Matrix Separation Joint work with Wotao Yin, Yin Zhang and Shen Yuan IPAM, UCLA Oct. 5, 2010 Low rank minimization problems Matrix completion: find a lowrank matrix W R m n so
More informationA New Trust Region Algorithm Using Radial Basis Function Models
A New Trust Region Algorithm Using Radial Basis Function Models Seppo Pulkkinen University of Turku Department of Mathematics July 14, 2010 Outline 1 Introduction 2 Background Taylor series approximations
More informationOn Dwell Time Minimization for Switched Delay Systems: FreeWeighting Matrices Method
On Dwell Time Minimization for Switched Delay Systems: FreeWeighting Matrices Method Ahmet Taha Koru Akın Delibaşı and Hitay Özbay Abstract In this paper we present a quasiconvex minimization method
More informationStratification of 3 3 Matrices
Stratification of 3 3 Matrices Meesue Yoo & Clay Shonkwiler March 2, 2006 1 Warmup with 2 2 Matrices { Best matrices of rank 2} = O(2) S 3 ( 2) { Best matrices of rank 1} S 3 (1) 1.1 Viewing O(2) S 3 (
More informationComputational Methods. Least Squares Approximation/Optimization
Computational Methods Least Squares Approximation/Optimization Manfred Huber 2011 1 Least Squares Least squares methods are aimed at finding approximate solutions when no precise solution exists Find the
More informationThe nonconvex Burer Monteiro approach works on smooth semidefinite programs
The nonconvex Burer Monteiro approach works on smooth semidefinite programs Nicolas Boumal Department of Mathematics Princeton University nboumal@math.princeton.edu Vladislav Voroninski Department of
More informationCONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS
MATHEMATICS OF COMPUTATION Volume 68, Number 228, Pages 165 1613 S 255718(99)11357 Article electronically published on March 1, 1999 CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS BENEDETTA MORINI Abstract.
More informationChapter 4. Unconstrained optimization
Chapter 4. Unconstrained optimization Version: 28102012 Material: (for details see) Chapter 11 in [FKS] (pp.251276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDFfile
More informationMultidisciplinary System Design Optimization (MSDO)
Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology  Prof. de Weck and Prof. Willcox Today s Topics Sequential
More informationDynamic Programming with Hermite Interpolation
Dynamic Programming with Hermite Interpolation Yongyang Cai Hoover Institution, 424 Galvez Mall, Stanford University, Stanford, CA, 94305 Kenneth L. Judd Hoover Institution, 424 Galvez Mall, Stanford University,
More informationOptimization methods
Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on firstorder information,
More informationGeneralized line criterion for GaussSeidel method
Computational and Applied Mathematics Vol. 22, N. 1, pp. 91 97, 2003 Copyright 2003 SBMAC Generalized line criterion for GaussSeidel method M.V.P. GARCIA 1, C. HUMES JR. 2 and J.M. STERN 2 1 Department
More informationExam February h
Master 2 Mathématiques et Applications PUF Ho Chi Minh Ville 2009/10 Viscosity solutions, HJ Equations and Control O.Ley (INSA de Rennes) Exam February 2010 3h Writtenbyhands documents are allowed. Printed
More informationSOLVING SYSTEMS OF NONLINEAR EQUATIONS USING A GLOBALLY CONVERGENT OPTIMIZATION ALGORITHM
Transaction on Evolutionary Algorithm and Nonlinear Optimization ISSN: 222987 Online Publication, June 2012 www.pcoglobal.com/gjto.htm NGO21/GJTO SOLVING SYSTEMS OF NONLINEAR EQUATIONS USING A GLOBALLY
More informationNewton s Method. Javier Peña Convex Optimization /36725
Newton s Method Javier Peña Convex Optimization 10725/36725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationConditional Gradient (FrankWolfe) Method
Conditional Gradient (FrankWolfe) Method Lecturer: Aarti Singh Coinstructor: Pradeep Ravikumar Convex Optimization 10725/36725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Postdoctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More informationWHEN studying distributed simulations of power systems,
1096 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL 21, NO 3, AUGUST 2006 A JacobianFree NewtonGMRES(m) Method with Adaptive Preconditioner and Its Application for Power Flow Calculations Ying Chen and Chen
More informationSecrets of Matrix Factorization: Supplementary Document
Secrets of Matrix Factorization: Supplementary Document Je Hyeong Hong University of Cambridge jhh37@cam.ac.uk Andrew Fitzgibbon Microsoft, Cambridge, UK awf@microsoft.com Abstract This document consists
More informationLECTURE 22: THE CRITICAL POINT THEORY OF DISTANCE FUNCTIONS
LECTURE : THE CRITICAL POINT THEORY OF DISTANCE FUNCTIONS 1. Critical Point Theory of Distance Functions Morse theory is a basic tool in differential topology which also has many applications in Riemannian
More informationNeural Networks Lecture 4: Radial Bases Function Networks
Neural Networks Lecture 4: Radial Bases Function Networks H.A Talebi Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Winter 2011. A. Talebi, Farzaneh Abdollahi
More informationII. DIFFERENTIABLE MANIFOLDS. Washington Mio CENTER FOR APPLIED VISION AND IMAGING SCIENCES
II. DIFFERENTIABLE MANIFOLDS Washington Mio Anuj Srivastava and Xiuwen Liu (Illustrations by D. Badlyans) CENTER FOR APPLIED VISION AND IMAGING SCIENCES Florida State University WHY MANIFOLDS? Nonlinearity
More informationLowrank matrix completion via preconditioned optimization on the Grassmann manifold
Lowrank matrix completion via preconditioned optimization on the Grassmann manifold Nicolas Boumal a, P.A. Absil b a Inria & D.I., UMR 8548, Ecole Normale Supérieure, Paris, France. b Department of Mathematical
More information