Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG. Limited memory BFGS (L-BFGS)

Transcription

1 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno (BFGS) Limited memory BFGS (L-BFGS) February 6, 2014 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

2 Roadmap Line-search Trust-region CG p k = f k + β k p k 1 Newton p k = [ 2 f k ] 1 f k m k (p) = f k + fk T 2 f k p q-newton p k = B 1 k f k m k (p) = f k + fk T B k p Today: How to select B k 2 f k for fast convergence of algorithms? Tomorrow: How to cheaply approximately solve Newton s subproblem & preserve fast convergence? Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

3 Idea Taylor: f (x k+1 ) f (x k ) = 2 f (x k + t(x k+1 x k ))[x }{{} k+1 x k ] B k+1 Secant equation s k := x k+1 x k y k := f (x k+1 ) f (x k ) B k+1 s k = y k n equations in (n + 1)n/2 unknowns! Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

4 SR1 method Ansatz: B k+1 = B k + σvv T, σ = ±1 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

5 SR1 method B k+1 = B k + (y k B k s k )(y k B k s k ) T (y k B k s k ) T s k, when (y k B k s k ) T s k 0 Sherman Morrison à = A + ab T = à 1 = A 1 A 1 ab T A b T A 1 a, provided A, à are non-singular. H k+1 = B 1 k+1 = H k + (s k H k y k )(s k H k y k ) T (s k H k y k ) T y k Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

6 SR1: cases 1 If (y k B k s k ) T s k 0 use SR1 update formula 2 If y k = B k s k = B k+1 = B k 3 If y k B k s k and (y k B k s k ) T s k = 0 = no SR1 update! Skip the update in this case (B k+1 = B k ) Note: B k does not have to be positive definite = Good Hessian approximation for non-convex problems Better suited for TR than LS approach Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

7 SR1: convergence Theorem 6.1, N&W Exact in n steps on convex quadratic functions (if updates never skipped) Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

8 SR1: convergence Theorem 6.2, N&W x k x Then 2 f is bounded & Lipschitz (y k B k s k ) T s k r y k B k s k s k (updates never skipped) {s k }-uniformly linearly independent lim B k 2 f (x ) = 0 k Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

9 SR1: convergence Theorem 6.7, N&W Under restrictive assumptions (unique stationary point in the vicinity of the algorithmic iterations; updates never skipped; B k bounded; etc) = convergence with n-step superlinear rate. Good convergence in practice. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

10 Rank-two update algorithms Idea: minimize B B k, B R n n subject to Bs k = y k, or B = B T. minimize H H k, H R n n subject to Hy k = s k, H = H T. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

11 Rank-two update algorithms Idea: Take scaled Frobenius matrix norm: W = W T, W 0, Wy k = s k. minimize B B k, B R n n subject to Bs k = y k, B = B T. B B k 2 = Tr[W (B B k )W (B B k ) T ], Result: Davidon Fletcher Powell (DFP) algorithm ρ k = (y T k s k) 1. B k+1 = (I ρ k y k s T k )B k(i ρ k s k y T k ) + ρ ky k y T k, Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

12 Davidon Fletcher Powell algorithm: B k+1 = (I ρ k y k s T k )B k(i ρ k s k y T k ) + ρ ky k y T k, H k+1 = H k H ky k y T k H k y T k H ky k + s ksk T yk T s, k ρ k = (y T k s k) 1. Historically: first quasi-newton algorithm. Sometimes gets stuck. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

13 Rank-two update algorithms Idea: Take scaled Frobenius matrix norm: minimize H H k, H R n n subject to Hs k = y k, H = H T. H H k 2 = Tr[W (H H k )W (H H k ) T ], W = W T, W 0, Ws k = y k. Result: Broyden Fletcher Goldfarb Shanno (BFGS) algorithm ρ k = (y T k s k) 1. H k+1 = (I ρ k s k y T k )H k(i ρ k y k s T k ) + ρ ks k s T k, Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

14 Broyden Fletcher Goldfarb Shanno algorithm: H k+1 = (I ρ k s k y T k )H k(i ρ k y k s T k ) + ρ ks k s T k, ρ k = (y T k s k) 1. B k+1 = B k B ks k s T k B k s T k B ks k + y kyk T yk T s, k Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

15 Linesearch framework: Wolfe curvature condition: [ f T k+1 f k] T } {{ } =yk T f T k+1 p k c 2 f T k p k p k }{{} =α 1 k T (c 2 1) f s k k p k > 0. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

16 BFGS/DFP+Wolfe linesearch: Suppose H k 0. Wolfe curvature condition = y T k s k > 0. z R n, z 0: z T H k+1 z = [(I ρ k y k s T k )z k] T H k [(I ρ k y k s T k )z k] + [s T k z]2 /(y T k s k) 0. If s T k z 0 = zt H k+1 z T [s T k z]2 /(y T k s k) > 0. If s T k z = 0 = zt H k+1 z = z T H k z > 0. Therefore H k+1 0! (Same for DFP) Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

17 BFGS/DFP+Wolfe linesearch: B 0 0 (H 0 0) + Wolfe linesearch B k 0 (H k 0), k p k = B 1 k f k = H k f k is a descent direction ( f k 0). Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

18 BFGS: global convergence Theorem 6.4, N&W Exact in n steps on convex quadratic functions; if B 0 = I = same iterates as CG. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

19 BFGS: global convergence Theorem 6.5, N&W 1 B 0 0, B 0 = B T 0 2 f C 2 3 m, M > 0: m z 2 z T 2 f (x)z M z 2 Then x k x : f (x ) = 0 & k=1 x k x <. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

20 BFGS: local convergence Theorem 6.6, N&W 1 f C 2 2 f - Lipschitz 3 k=1 x k x <. Then lim k x k+1 x / x k x = 0. Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

21 Presented quasi-newton algorithms: Good for small to medium size optimization problems with dense (or costly to compute) Hessian matrices Factorization/system solve cost for [ 2 f k ] 1 f k : O(n 3 ) Update + multiplication with H k : O(n 2 ) Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

22 Limited-memory quasi-newton algorithms: Idea Storage of B k (or H k ): O(n 2 ) Instead: store m << n pairs (s k, y k ): O(2mn) Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

23 Example: L-BFGS H k+1 q = [I ρ k y k s T k ]T H k [I ρ k y k s T k ]q + ρ ks k s T k q = [I ρ k s k y T k ]H k[q {ρ k s T k q}y k] + {ρ k s T k q}s k. Algorithm (only one pair is stored) 1: α k := ρ k s T k q 2: q := q α k y k 3: r := H k q 4: β k := ρ k y T k r 5: r := r + (α k β k )s k Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

24 Example: L-BFGS 1: q := f k+1 2: for i = k, k 1,..., k m + 1 do 3: α i := ρ i s T i q 4: q := q α i y i 5: end for 6: r := Hk 0q 7: for i = k m + 1,..., k 1, k do 8: β i := ρ i y T r 9: r := r + (α i β i )s i 10: end for 11: Result: r H k+1 f k+1 i Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG

25 Storage and step complexity revisited: Method Storage Step complexity Convergence rate CG O(n) O(n) linear BFGS/DFP/SR1 O(n 2 ) O(n 2 ) superlinear L-BFGS O(mn) O(mn)??? Newton (full Hess) O(n 2 ) O(n 3 ) quadratic Newton (sparse Hess) O(n) O(n 2 ) O(n) O(n 3 ) quardatic Note: step complexity does not account for complexities in evaluating function/derivatives!!! Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG