Rank-constrained Optimization: A Riemannian Manifold Approach

Transcription

1 Introduction Rank-constrained Optimization: A Riemannian Manifold Approach Guifang Zhou 1, Wen Huang 2, Kyle A. Gallivan 1, Paul Van Dooren 2, P.-A. Absil 2 1- Florida State University 2- Université catholique de Louvain 23 April 2015 Rank-constrained Optimization: A Riemannian Approach 1

2 Introduction Problem Statements Finding an optimum of a rank-constrained problem min f (X ), X M k M k = {X M rank(x ) k} M is R m n or a submanifold of R m n Roughly speaking, a manifold is a set endowed with coordinate patches which overlap smoothly, e.g., sphere: {x R n x 2 = 1}. Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 2

3 Motivations Introduction E.g. Compression of an image Cost function: f (X ) = X M 2 F, where M R is the matrix representation of the image Figure : Size : Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 3

4 Motivations Introduction Figure : Low rank approximation of the image Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 4

5 Motivations Introduction Figure : Singular values of the image Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 5

6 Motivations Introduction Rank-constrained optimization problems have appeared in many areas M = R m n Signal and image processing System identification Computational finance Low-dimensional embedding M R m n Low-rank approximation of graph similarity matrix Low-rank similarity measure for role model extraction Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 6

7 Introduction Existing Methods SVD-based algorithms Random multistart-type algorithms, e.g. Alternating Projection method, Double Minimization method, EW-TLS, etc. Method of alternating between fixed-rank optimization and a simple update to the rank Projected line-search method Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 7

8 Challenges Introduction Efficiency (both time and space) Rigorous way for rank updating (explain on the next slide) Find the rank independent of k Exact solution Approximate solution Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 8

9 Introduction Framework of Modified Riemannian Optimization Method M is a submanifold of R m n In general, M k = {X M rank(x ) k} is not a manifold M k = M 0 M 1 M 2 M k, where M r = {X M rank(x ) = r} Assume M r is a manifold General Riemannian optimization algorithms, e.g., RTR-Newton, can be applied on (M r, g) Question: When and how to move from one fixed-rank manifold to another one? Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 9

10 Introduction When to increase from one fixed-rank manifold to another one? Consider two functions f F : M R : X f F (X ), f r : M r R : X f r (X ), such that f = f M k F, f r = f Mr. Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 10

11 Introduction When to increase from one fixed-rank manifold to another one? Condition I (angle threshold, θ 0 ): (gradf F (X ), gradf r (X )) = θ > θ 0 Condition II (difference threshold, ɛ): gradf F (X ) gradf r (X ) ɛ Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 11

12 Introduction How to increase from one fixed-rank manifold to another one? Line-search method X i+1 = R Xi (t i η i ), where t i 0 is a step-size, η i is a search direction and R Xi is a retraction, i.e., R Xi : T Xi M M and d dt R X i (tη) = η How to choose η i? Rank-related Direction Vector How to choose retraction R? Rank-related Retraction Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 12

13 Introduction Rank-related Objects η x, r T x M r and η x, r / T x M r 1 is a rank- r-related direction vector M gradf F (x) T x M η x, r argmin η Tx M r gradf F (x) η F M r x M r η x, r R x (η x, r ) Rank-related retraction R x satisfies R x (tη x, r ) M r, t (0, δ) Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 13

14 Introduction Rank Reduction Truncate rank and retract to M May increase the function value Do not destroy the progress that is made in the previous rank increasing step, i.e., f (X i+1 ) f (X s ) c(f (X s+1 ) f (X s )), where s is such that the latest rank increase was from X s to X s+1, 0 < c < 1. Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 14

15 Introduction Main Theoretical Results Parameter ɛ is used to adjust the accuracy of the approximate solutions. (Global Convergence) Suppose some reasonable assumptions hold. Then the sequence {X i } generated by modified Riemannian Optimization method satisfies ( lim inf i P T Xi M k (gradf F (X i )) 1 + ) 1 tan(θ 0 ) 2 ɛ. Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 15

16 Introduction Weighted Low-rank Approximation Problem Formulation argmin B X 2 W, B X 2 W = vec{b X }T W vec{b X } X M k M = R m n B R m n is a given data matrix W R mn mn is a positive definite symmetric weighting matrix vec{a} denotes the vectorized form of A Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 16

17 Experiment Introduction Comparison of different methods. The data matrix B = B 1 B2 T is a random generated matrix with rank 5, B 1 R 80 5, B 2 R The weighting matrix W R is a positive definite symmetric matrix. k = 3, 5, 7. Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 17

18 Experiment Introduction Four algorithms are compared. MROM: modified Riemannian optimization method SULS: Projected line-search method (Schneider and Uschmajew [SU14]) DMM: Double minimization method (Brace and Manton [BM06]) APM: Alternating projections method (Lu et. al [LPW97] or Wentzell et. al [WAK97]) Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 18

19 Experiment Introduction k method rank f Rel Err( A X W A W ) t k = 3 MROM SULS DMM APM k = 5 MROM SULS DMM APM k = 7 MROM SULS 7(0/100) DMM APM 7(0/100) Table : The number in the parenthesis indicates the fraction of experiments where the numerical rank (number of singular values greater than 10 8 ) found by the algorithm equals the true rank. The subscript ±k indicates a scale of 10 ±k. Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 19

20 Conclusion Introduction min f (X ) X M k Generalize the admissible set from R m n k to M k ; Define an algorithm solving a rank inequality constrained problem while finding a suitable rank for approximation; Prove theoretical convergence results; Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 20

21 Thank you! Introduction Thank you for your attention! Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 21

22 Introduction I. Brace and J. H. Manton. An improved BFGS-on-manifold algorithm for computing low-rank weighted approximations. In Proceedings of 17th International Symposium on Mathematical Theory of Networks and Systems, pages , W.-S. Lu, S.-C. Pei, and P.-H. Wang. Weighted low-rank approximation of general complex matrices and its application in the design of 2-d digital filters. IEEE Transactions on Circuits and SystemsI, 44: , R. Schneider and A. Uschmajew. Convergence results for projected line-search methods on varieties of low-rank matrices via {\L}ojasiewicz inequality. ArXiv e-prints, pages 1 1, feb Peter D. Wentzell, Darren T. Andrews, and Bruce R. Kowalski. Maximum likelihood multivariate calibration. Analytical Chemistry, 69(13): , Rank-constrained Optimization: A Riemannian Approach Problem: min f (X ), s. t. X M k 22