Interior-point methods Optimization Geoff Gordon Ryan Tibshirani

Transcription

1 Interior-point methods Optimization Geoff Gordon Ryan Tibshirani

2 SVM duality Review min v T v/2 + 1 T s s.t. Av yd + s 1 0 s 0 max 1 T α α T Kα/2 s.t. y T α = 0 0 α 1 Gram matrix K Interpretation support vectors & complementarity reconstruct primal solution from dual 2

3 Review Kernel trick high-dim feature spaces, fast positive definite function Examples polynomial homogeneous polynomial linear Gaussian RBF

4 Review: LF problem Ax + b 0 Ball center bad summary of LF problem Max-volume ellipsoid / ellipsoid center good summary (1/n of volume), but expensive Analytic center of LF problem maximize product of distances to constraints min ln(ai T x + bi) Dikin analytic center: not quite as good (just 1/m < 1/n), but much cheaper 4

5 Force-field interpretation of analytic center Pretend constraints are repelling a particle normal force for each constraint force 1/distance Analytic center = equilibrium = where forces balance 5

6 Newton for analytic center f(x) = ln(a i T x + bi) df/dx = d 2 f/df 2 = 6

7 Dikin ellipsoid E(x 0) = { x (x x0) T H(x x0) 1 } H = Hessian of log barrier at x0 unit ball of Hessian norm at x0 E(x 0) X for any strictly feasible x0 affine constraints can be just feasible E(x0): as above, but intersected w/ affine constraints vol(e(x ac)) vol(x)/m weaker than ellipsoid center, but still very useful 7

8 E(x0) X E(x 0) = { x (x x0) T H(x x0) 1 } H = A T S -2 A S = diag(s) = diag(ax0 + b) 8

9 me(x0) X Feasible point x: Ax + b 0 Analytic center x ac: A T y = 0 Let Y = diag(y ac), H = A T Y 2 A; show: (x xac) T H(x xac) m 2 [+ m] y = 1./(Axac+b) 9

10 Combinatorics v. analysis Two ways to find a feasible point of Ax+b 0 find analytic center minimize a smooth function find a feasible basis combinatorial search 10

11 Bad conditioning? No problem. Analytic center & Dikin ellipsoids invariant to affine xforms w = Mx+q W = { w AM -1 (w q) + b 0 } Can always xform so that a ball takes up vol(y)/m Dikin sphere 11

12 LF LP: the central path Analytic center was for: find x st Ax + b 0 Now: min ct x st Ax + b 0 Same trick: min ft(x) = c T x (1/t) ln(ai T x + bi) parameter t > 0 central path = t 0: t : 12

13 Force-field interpretation of central path Force along objective; normal forces for each constraint ments c PSfrag replacements 3c t=1 t=3 13

14 Newton for central path min f t(x) = c T x (1/t) ln(ai T x + bi) df/dx = d 2 f/dx 2 = 14

15 Central path example objective t 0 t 15

16 New LP algorithm? Set t=1012. Find corresponding point on central path by Newton s method. worked for example on previous slide! but has convergence problems in general Alternatives? 16

17 Constraint form of central path min ln si st Ax + b 0 c T x λ a 1-1 mapping λ(t) w/ x(λ(t)) = x(t) t>0 but this form is slightly less convenient since we don t know minimal feasible value of λ or maximal nontrivial value of λ 17

18 Dual of central path min ct x (1/t) ln si st Ax + b = s 0 minx,s maxy L(x,s,y) = c T x (1/t) ln si + y T (s Ax b) 18

19 Primal-dual correspondence Primal and dual for central path: min c T x (1/t) ln si st Ax + b = s 0 max (m ln t)/t + m/t + (1/t) ln yi y T b st A T y = c y 0 L(x,s,y) = ct x (1/t) ln si + y T (s Ax b) grad wrt s: to get x: 19

20 At optimum: Duality gap primal value c T x (1/t) ln si = dual value (m ln t)/t + m/t + (1/t) ln yi y T b s y = te 20

21 Primal-dual constraint form Primal-dual pair: min c T x st Ax + b 0 max b T y st A T y = c y 0 KKT: Ax + b 0 (primal feasibility) y 0 A T y = c (dual feasibility) c T x + b T y 0 (strong duality) or, c T x + b T y λ (relaxed strong duality) 21

22 Analytic center of relaxed KKT Relaxed KKT conditions: Ax + b = s 0 y 0 A T y = c c T x + b T y λ Central path = {analytic centers of relaxed KKT} 22

23 Algorithm t := 1, y := 1m, x := 0 n [s := 1 m ] Repeat Use infeasible-start Newton to find point on dual central path Recover primal (s,x); gap c T x + b T y = m/t s = 1./ty x = A\(s b) [have already (Newton)] t := αt (α > 1) 23

24 Example ty gap 10 4 m/t 10 2 duality gap m = 50 m = 500 m = Newton iterations 24