Solving Constrained Rayleigh Quotient Optimization Problem by Projected QEP Method

Size: px

Start display at page:

Download "Solving Constrained Rayleigh Quotient Optimization Problem by Projected QEP Method"

Deirdre Stone
5 years ago
Views:

1 Solving Constrained Rayleigh Quotient Optimization Problem by Projected QEP Method Yunshen Zhou Advisor: Prof. Zhaojun Bai University of California, Davis June 15, 2017 Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 1 / 44

2 1 Introduction 2 Theory Problem Decomposition The Lagrange Equations The Equivalent QEP Overview 3 Algorithm Lanczos Algorithm Projected QEP Method Compute the Solution of CRQopt 4 Convergence Analysis 5 Application Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 2 / 44

3 1 Introduction 2 Theory Problem Decomposition The Lagrange Equations The Equivalent QEP Outline 3 Algorithm Lanczos Algorithm Projected QEP Method Compute the Solution of CRQopt 4 Convergence Analysis 5 Application Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 3 / 44

4 The Constrained Rayleigh Quotient Optimization Problem Consider the problem CRQopt min g(v) = v T Av, s.t. v T v = 1, C T v = t. A R n n, A = A T, C R n m is full column rank, t R m and m n. We are interested in the case where A is large and sparse and t 0. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 4 / 44

5 Applications Ridge regression. Trust region subproblems. Constrained least square problems. Transductive learning. Graph cut. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 5 / 44

6 Methods: Previous Work Comments: Secular equations [Gander, Golub and von Matt, 1989]. Inner-outer iteation of Lanczos method [Golub, Zhang and Zha, 2000]. Projected power method [Xu, Li and Schuurmans, 2009]. Only works for small matrices. Only works for the case t = 0. The convergence is slow, the shifting parameter is hard to determine. Parametric eigenvalue problem [Eriksson, Olsson and Kahl, 2011]. The eigenvalue computation involved in each iteration of line search results in a huge amount of calculation. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 6 / 44

7 1 Introduction 2 Theory Problem Decomposition The Lagrange Equations The Equivalent QEP Outline 3 Algorithm Lanczos Algorithm Projected QEP Method Compute the Solution of CRQopt 4 Convergence Analysis 5 Application Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 7 / 44

8 Let Vector Decomposition v = n 0 + P v. where n 0 = (C T ) t is the minimum norm solution of C T v = t and u = P v, where P = I CC. We are interested in the case where n 0 < 1. There are multiple feasible vectors. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 8 / 44

9 Problem Transformation By the vector decomposition, u is the solution of the following CQ optimization problem (CQopt): min f(u) = u T P AP u + 2u T b 0, s.t. u = γ, u N (C T ), where b 0 = P An 0 and γ = 1 n 0 2. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem by June Projected 15, 2017 QEP Method 9 / 44

10 The Lagrange Equations By Lagrange multiplier theory, the Lagrange equations for CQopt is (P AP λi)u = b 0, u = γ, u N (C T ). Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 10 / 44

11 λ 1 < λ 2 f(u 1 ) < f(u 2 ). The Lagrange Optimization Solving CQopt is equivalent to solving the following Lagrange optimization problem (LGopt): (P AP λi)u = b 0, u = γ, u N (C T ), λ = min. When λ = min, the second order necessary condition of Lagrange equations is satisfied. For standard trust-region subproblems, P AP λi is positive semidefinite. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 11 / 44

12 The Equivalent QEP Back to the Lagrange equations, assuming that λ / λ(p AP ), define z = (P AP λi) 2 b 0, then we have { (P AP λi) 2 z = b 0, So γ 2 = b T 0 z. (P AP λi) 2 z = 1 γ 2 b 0b T 0 z, z N (C T ). Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 12 / 44

13 Equivalency of LGopt and QEPmin LGopt: (P AP λi)u = b 0, u = γ, Theorem 1 u N (C T ), λ = min. QEPmin: (P AP λi) 2 z = 1 γ 2 b 0b T 0 z, z N (C T ), λ = min, real. If (λ, z) is the solution of QEPmin, there exists u such that (λ, u) is the solution of LGopt. Conversely, let (λ, u) be the solution of LGopt, there exists z such that (λ, z) is the solution of QEPmin. Therefore, LGopt is equivalent to QEPmin. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 13 / 44

14 Distribution of the Eigenvalues Proposition 1 The smallest real eigenvalue of the QEP (P AP λi) 2 z = 1 γ 2 b 0b T 0 z, z N (C T ), is the leftmost eigenvalue. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 14 / 44

15 Distribution of the Eigenvalues Imag Real Figure 1: Plot of the eigenvalues of QEP. Problem size: n = 100, m = 10. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 15 / 44

16 1 Introduction 2 Theory Problem Decomposition The Lagrange Equations The Equivalent QEP Outline 3 Algorithm Lanczos Algorithm Projected QEP Method Compute the Solution of CRQopt 4 Convergence Analysis 5 Application Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 16 / 44

17 Lanczos Algorithm Given a real symmetric matrix M, the Lanczos process generates an orthogonal matrix Q k whose column space is the Krylov subspace K k (M, r 0 ) = R(r 0, Mr 0,, M k 1 r 0 ). The first k steps of Lanczos algorithm take the matrix form MQ k = Q k T k + β k+1 q k+1 e T k, where T k is a diagonal symmetric matrix with diagonal entries α 1,, α k and subdiagonal entries β 2,, β k. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 17 / 44

18 Lanczos Algorithm Require: M, r 0 Ensure: α, β, Q = [q 1, q 2,..., q k ], k 1: β 1 r 0 2: if β 1 = 0 then 3: stop 4: end if 5: q 1 r 0 /β 1, q 0 0 6: for k 1, 2,... do 7: p Mq k 8: r p β k q k 1 9: α k qk T p 10: r r α k q k 11: β k+1 r 12: if β k+1 = 0 then 13: stop 14: end if 15: q k+1 r/β k+1 16: end for Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 18 / 44

19 Matrix-Vector Multiplication P AP x Apply Lanczos to M = P AP. If r 0 N (C T ), then Mq k = P AP q k = P Aq k. p = P Aq k can be computed by P Aq k = (I CC )Aq k = Aq k CC Aq k = Aq k Cy, where y = arg min y R m Cy Aq k. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 19 / 44

20 Approximate QEPmin by Projection Let b 0 N (C T ) be the starting vector of Lanczos iteration, then the projection of QEPmin is ( (T k λi) 2 b0 2 ) y = e 1 e T 1 βk+1 2 e ke T k y, y R k, γ 2 λ = min, real. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 20 / 44

21 QEP Existence of Real Eigenvalues ( (T k λi) 2 b0 2 y = e 1 e T 1 βk+1 2 e ke T k γ 2 may not have any real eigenvalues. Example: ) y (1) Figure 2: Plot of the eigenvalues of QEP (1). Problem: A = diag(1, 2, 3, 4, 5), C = [4, 8, 3, 1, 1] T, t = 3 and k = 3. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 21 / 44

22 Existence of Real Eigenvalues In order to make sure that the reduced QEP has at least one real eigenvalue, we can solve the following QEP (rqepmin): (T k λi) 2 w = b 0 2 γ 2 e 1 e T 1 w, w R k, λ = min, real. Let (µ (k) 1, w) be the eigenpair of rqepmin, then (µ(k) 1, Q kw) is the approximation of eigenpair of QEPmin. Note that Q k w N (C T ). Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 22 / 44

23 Approximation of the Eigenvalues 0.03 Original QEP Projected QEP Imag Real Figure 3: Plot of the eigenvalues of QEP and projected QEP. Problem size: n = 100, m = 10, k = 25. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 23 / 44

24 Solving rqepmin by Linearization Let y = (T k λi)w, then rqepmin can be written as Let s = [ ] y, then w (T k λi) 2 w = b 0 2 e 1 e T 1 w γ 2 T k y b 0 2 γ 2 e 1 e T 1 w = λy, T k w y = λw. [T k b 0 2 I γ 2 e 1 e T 1 T k ] s = λs. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 24 / 44

25 Stopping Condition Let µ (k) 1 be the smallest real eigenvalue of rqepmin and w be the corresponding eigenvector. Let ŵ (k) = Q k w and the residual of the QEP be ( r k = P AP µ (k)2 2 1 I) ŵ(k) γ 2 b 0 b T 0 ŵ (k). Stopping condition: r k < ε. Observation 1: r k = [ 2µ (k) 1 β k+1q k+1 + β 2 k+1 q k + α k β k+1 q k+1 + β k+1 (α k+1 q k+1 + β k+2 q k+2 )]w k + q k+1 β k β k+1 w k 1. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 25 / 44

26 10 2 w k Observation 2: r k w k. Stopping Condition 10 0 r k Figure 4: Plot of r k and w k. Problem size: n = 100, m = 10. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 26 / 44

27 Compute the Solution of CRQopt Let the approximated solution of CRQopt be γ2 v (k) = Q k y + n 0, b 0 w 1 where w 1 is the first component of w and y = (T k µ (k) 1 I)w. v (k) is the best approximation of the solution of CRQopt in n 0 + K k (P AP, b 0 ), namely, v (k) = arg min s n 0 +K k (P AP,b 0 ), s =1 st As. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 27 / 44

28 Numerical Results: Running Time Explicit Projected Power Projected QEP Running TIme n Figure 5: The running time of different algorithms. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 28 / 44

29 1 Introduction 2 Theory Problem Decomposition The Lagrange Equations The Equivalent QEP Outline 3 Algorithm Lanczos Algorithm Projected QEP Method Compute the Solution of CRQopt 4 Convergence Analysis 5 Application Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 29 / 44

30 Convergence Analysis Let ˆθ be the eigenvalue of QEPmin and suppose ˆθ / D, D = {λ P AP y = λy, y N (C T )}, 1 the sequence {g( v (k) )} is nonincreasing, 2 let k max be the smallest k such that β kmax+1 = 0, then g( v (kmax) ) = g(v), and 3 let θ = min(d) and θ = max(d), then 0 g( v (k) ) g(v) 4 P AP ˆθI ( γ p k+1 (η) )2, where η = θ ˆθ θ θ, and p k (x) is the k-th Chebyshev polynomial of the first kind. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 30 / 44

31 Chebyshev Polynomials p k (x) 0 5 p 1 (x) p 2 (x) p 3 (x) 10 p 4 (x) p 5 (x) x Figure 6: Chebyshev polynomials of different degrees. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 31 / 44

32 Numerical Results: Rate of Convergence η= η= η= Error of Objective Function k Figure 7: The error of objective function for different η. Problem size: n = 500, m = 50. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 32 / 44

33 Numerical Results: Tightness of the Bound Actual Error Error Bound Actual Error Error Bound Error of Objective Function Error of Objective Function k k Figure 8: The actual error and the error bound of the objective function for different k. The left plot is the case where η = and the right plot is the case where η = Problem size: n = 500, m = 50. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 33 / 44

34 1 Introduction 2 Theory Problem Decomposition The Lagrange Equations The Equivalent QEP Outline 3 Algorithm Lanczos Algorithm Projected QEP Method Compute the Solution of CRQopt 4 Convergence Analysis 5 Application Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 34 / 44

35 Application: Constrained Normalized Cut Let i 1,, i k be the indexes of training data. The CRQ form of constrained normalized cut can be written as min z T Lz, s.t. z T z = 1, C T z = t, where and C = [D 1 2 1, ei1,, e ik ], t = [0, ± 1 n,, ± 1 n ] T. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 35 / 44

Example of Application: Image Cut (a) Normalized cut (b) Constrained normalized cut Figure 9: Numerical results of normalized cut with and without constraints.

36 Example of Application: Image Cut (a) Normalized cut (b) Constrained normalized cut Figure 9: Numerical results of normalized cut with and without constraints. Problem size: n = 41616, m = 20. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 36 / 44

37 Example of Application: Image Cut (a) Normalized cut (b) Constrained normalized cut 1 (c) Constrained normalized cut 2 Figure 10: Numerical results of normalized cut with and without constraints. Problem size for constrained normalized cut 1: n = 5184, m = 18. Problem size for constrained normalized cut 2: n = 5184, m = 24. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 37 / 44

38 Future Work Tighter bound for the error of objective function. Compute v (k) when ˆθ D. Convergence analysis when matrix-vector multiplication has error. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 38 / 44

39 References [1] Anders Eriksson, Carl Olsson, and Fredrik Kahl. Normalized cuts revisited: A reformulation for segmentation with linear grouping constraints. Journal of Mathematical Imaging and Vision, 39(1):45-61, [2] Walter Gander, Gene H. Golub, and Urs von Matt. A constrained eigenvalue problem. Linear Algebra and its Applications, 114C115: , [3] Gene H Golub, Zhenyue Zhang, and Hongyuan Zha. Large sparse symmetric eigenvalue problems with homogeneous linear constraints: the Lanczos process with innerouter iterations. Linear Algebra and its Applications, 309(1): , [4] Linli Xu, Wenye Li, and Dale Schuurmans. Fast normalized cut with linear constraints. In the Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pages IEEE, [5] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8): , [6] Chungen Shen, Lei-Hong Zhang, and Ren-Cang Li. On the generalized Lanczos trust-region method. Technical report, DOI: /RG /1. Available at Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 39 / 44

40 Thank you! Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 40 / 44

41 LSQR Consider the least square problem Let min Cy x 2. y R m [ ] 0 C B = C T, 0 [ ] u1 0 u W = k 0. 0 v 1 0 v k We apply Lanczos algorithm to matrix B with starting vector [ ] u1. 0 Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 41 / 44

42 Then where LSQR U k+1 (β 1 e 1 ) = b, AV k = U k+1 B k, A T U k+1 = V k Bk T + α k+1v k+1 e T k+1, α β 2 α B k = 0 β 3 α 3 0 α k. 0 0 β k+1 Then x k = V k y k is the approximation of the solution of the least square problem, where y k is the solution of min β 1e 1 B k y k. y k R k Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 42 / 44

43 Application: Constrained Normalized Cut Let x be an indicator vector, x i = 1 if i A and 1 otherwise. The normalized cut can be simplified to Ncut(A, B) = yt (D W )y y T, Dy where D is a diagonal matrix with the row sums of W on the diagonal, y is a linear transformation of x with the property that y T D1 = 0. Set z = D 1 2 y, L = D 1 2 (D W )D 1 2 and rewrite the problem of minimizing the normalized cut: min z T Lz, s.t. z T z = 1, z T D = 0. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 43 / 44

44 Application: Constrained Normalized Cut Linear constraints: information for cluster z i = ± 1 n and the constraint z T D = 0. Weight Matrix: F (i) F (j) 2 2 w ij = e δ F 2 if X(i) X(j) 2 < r, 0 otherwise. where F (i) is the brightness of pixel i and X(i) is the location of pixel i. Yunshen Zhou Advisor: Prof. Zhaojun Bai Solving Constrained Rayleigh Quotient Optimization Problem June by Projected 15, 2017QEP Method 44 / 44

The Lanczos and conjugate gradient algorithms

The Lanczos and conjugate gradient algorithms Gérard MEURANT October, 2008 1 The Lanczos algorithm 2 The Lanczos algorithm in finite precision 3 The nonsymmetric Lanczos algorithm 4 The Golub Kahan bidiagonalization