School of Mathematics T H E U N I V E R S I T Y O H Using interior point methods for optimization in training very large scale Support Vector Machines F E D I N U R Part : Interior Point Methods for QP Jacek ondzio Email: J.ondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio joint work with Kristian Woodsend Montreal, 8 June 2009 Montreal, 8 June 2009 3 Outline Interior Point Methods for QP logarithmic barrier function complementarity conditions linear algebra Support Vector Machine training Quadratic Programming formulation specific features IPMs for SVM training separable formulations!!! linear and nonlinear kernels in SVMs Challenges remaining nonlinear kernels in SVMs indefinite kernels in SVMs Conclusions Montreal, 8 June 2009 2 Elements of the IPM What do we need to derive the Interior Point Method? logarithmic barriers. duality theory: Lagrangian function; first order optimality conditions. Newton method. Wright, Primal-Dual Interior-Point Methods, SIAM, 997. Andersen, ondzio, Mészáros and Xu, Implementation of Interior Point Methods for Large Scale Linear Programming, in: Interior Point Methods in Mathematical Programming, T Terlaky (ed.), Kluwer Academic, 996, pp. 89 252. Montreal, 8 June 2009 4
Logarithmic barrier ln v i replaces the inequality v i 0. Observe that min e n i= ln v i max n ln x The minimization of n i= ln v i is equivalent to the maximization of the product of distances from all hyperplanes defining the positive orthant: it prevents all v i from approaching zero. Montreal, 8 June 2009 5 i= v i x Conditions for a stationary point of the Lagrangian v L(v, λ, µ) = c A T λ + Qv µv e = 0 λ L(v, λ, µ) = Av b = 0, where V = diag{v, v 2,, v n }. Let us denote s = µv e, i.e. V Se = µe. The First Order Optimality Conditions are: Av = b, A T λ + s Qv = c, V Se = µe, (v, s) > 0. Montreal, 8 June 2009 7 Logarithmic barrier Replace the primal QP min c T v + 2 vt Qv s.t. Av = b, v 0, with the primal barrier program min c T v + 2 vt Qv µ n s.t. Av = b. j= ln v j First Order Optimality Conditions Active-set Method: Av = b A T λ + s Qv = c V Se = 0 v, s 0. Interior Point Method: Av = b A T λ + s Qv = c V Se = µe v, s 0. Lagrangian: L(v, λ, µ) = c T v + n 2 vt Qv λ T (Av b) µ ln v j. j= Montreal, 8 June 2009 6 Montreal, 8 June 2009 8
Complementarity v i s i = 0 i =, 2,..., n. Active-set Method makes a guess of optimal partition: A I = {, 2,..., n}. For active constraints (i A), v i = 0 and v i s i = 0 i A. For inactive constraints (i I), s i = 0 hence v i s i = 0 i I. Interior Point Method uses ε-mathematics: Replace v i s i = 0 i =, 2,..., n by v i s i = µ i =, 2,..., n. Force convergence µ 0. Montreal, 8 June 2009 9 Newton Method (cont d) Note that f(v, λ, s) = A 0 0 Q A T I. S 0 V Thus, for a given point (v, λ, s) we find the Newton direction ( v, λ, s) by solving the system of linear equations: A 0 0 [ ] v Q A T I λ = b Av c A T λ s + Qv. S 0 V s µe V Se Montreal, 8 June 2009 Apply Newton Method to the FOC The first order optimality conditions for the barrier problem form a large system of nonlinear equations f(v, λ, s) = 0, where f : R 2n+m R 2n+m is a mapping defined as follows: Av b f(v, λ, s) = A T λ + s Qv c. V Se µe Actually, the first two terms of it are linear; only the last one, corresponding to the complementarity condition, is nonlinear. Montreal, 8 June 2009 0 Linear Algebra of IPM for QP A 0 0 Q A T I S 0 V Use the third equation to eliminate [ ] [ ] v ξp λ = ξ d = s ξ µ s = V (ξ µ S v) = V S v + V ξ µ, from the second equation and get [ Q Θ A T ] [ ] v A 0 λ = where Θ = V S is a diagonal scaling matrix. Θ is always very ill-conditioned. b Av c A T λ s + Qv. µe V Se [ ξd V ] ξ µ. ξ p Montreal, 8 June 2009 2
Augmented system [ Q Θ A T ] [ ] [ ] v r A 0 λ = h = Symmetric but indefinite linear system. In general, it may be difficult to solve. Separable Quadratic Programs [ ξd V ] ξ µ. ξ p When matrix Q is diagonal (Q = D), the augmented system can be further reduced. Eliminate v = (D + Θ ) (A T λ r), to get normal equations (symmetric, positive definite system) (A(D + Θ ) A T ) λ = g = A(D + Θ ) r + h. Interior Point Methods Theory: IPMs converge in O( n) or O(n) iterations Practice: IPMs converge in O(log n) iterations... but one iteration may be expensive! Suppose A R m n is a dense matrix. Major computational effort when solving separable QP (separable QP means that Q = D, diagonal). build H = A(Q + Θ ) A T O(nm 2 ) compute Cholesky H = LΛL T O(m 3 ) Recall n m. Montreal, 8 June 2009 3 Montreal, 8 June 2009 5 Sparsity Issues in QP Observation: the inverse of the sparse matrix may be dense. Example 2 2 2 2 = 5 4 3 2 = = 4 4 3 2 3 3 3 2 2 2 2 2. IPMs for QP: Do not explicitly invert the matrix Q + Θ to form A(Q + Θ ) A T unless Q is diagonal. Montreal, 8 June 2009 4 Ill-conditioning of Θ = V S For active constraints: Θ j = v j /s j 0 Θ j ; For inactive constraints: Θ j = v j /s j Θ j 0. oldfarb and Scheinberg, A product form Cholesky factorization for handling dense columns in IPMs for linear programming, Mathematical Programming, 99(2004) -34. Although Θ = (Q + Θ ) behaves badly, the Cholesky factorization H = LΛL T behaves well: Λ captures instability (variability) of Θ; L is well conditioned (bounded independently of Θ). Represent L = L L 2... L m, where L i has entries only in column i. Drawback: PFCF is sequential by nature. Montreal, 8 June 2009 6
Interior Point Methods: Summary Interior Point Methods for QP polynomial algorithms excellent practical behaviour competitive for small problems (,000,000 vars) beyond competition for large problems (,000,000 vars) Opportunities for SVM training with IPMs dense data very large size well-suited to parallelism Classification We consider a set of points X = {x, x 2,..., x n }, x i R m to be classified into two subsest of good and bad ones. X = and =. We look for a function f : X R such that f(x) 0 if x and f(x) < 0 if x. Usually n m. Montreal, 8 June 2009 7 Montreal, 8 June 2009 9 Linear Classification We consider a case when f is a linear function: Part 2: Support Vector Machine training f(x) = w T x + b, where w R m and b R. In other words we look for a hyperplane which separates good points from bad ones. In such case the decision rule is given by y = sgn(f(x)). If f(x i ) 0, then y i = + and x i. If f(x i ) < 0, then y i = and x i. We say that there is a linearly separable training sample S = ((x, y ), (x 2, y 2 ),..., (x n, y n )). Montreal, 8 June 2009 8 Montreal, 8 June 2009 20
How does it work? iven a linearly separable database (training sample) S = ((x, y ), (x 2, y 2 ),..., (x n, y n )) find a separating hyperplane w T x + b = 0, which satisfies y i (w T x i + b), i =, 2,..., n. iven a new (unclassified) point x 0, compute y 0 = sgn(w T x 0 + b) to decide whether x 0 is good or bad. Montreal, 8 June 2009 2 QP Formulation Finding a separating hyperplane can be formulated as a quadratic programming problem: min 2 w T w s.t. y i (w T x i + b), i =, 2,..., n. In this formulation the Euclidean norm of w is minimized. This is clearly a convex optimization problem. (We can minimize w or w and then the problem can be reformulated as an LP.) Two major difficulties: Clusters may not be separable at all minimize the error of misclassifications; Clusters may be separable by a nonlinear manifold find the right feature map. Montreal, 8 June 2009 23 Separating Hyperplane To guarantee a nonzero margin of separation we look for a hyperplane w T x + b = 0, such that w T x i + b for good points; w T x i + b for bad points. This is equivalent to: w T x i w + w b w w T x i w + w b w for good points; for bad points. In this formualtion the normal vector of the separating hyperplane has unit length. In this case the margin between good and w w bad points is measured by 2 w. This margin should be maximised. This can be achieved by minimising the norm w. Montreal, 8 June 2009 22 Difficult Cases Nonseparable clusters: ξ2 ξ Use nonlinear feature map: Φ Errors when defining clusters of good and bad points. Minimize the global error of misclassifications: ξ + ξ 2. Montreal, 8 June 2009 24
Linearly nonseparable case If perfect linear separation is impossible then for each misclassified data we introduce a slack variable ξ i which measures the distance between the hyperplane and misclassified data. Finding the best hyperplane can be formulated as a quadratic programming problem: min 2 w T w + C n ξ i i= s.t. y i (w T x i + b) + ξ i, i =, 2,..., n, ξ i 0 i =, 2,..., n, where C (C > 0) controls the penalisation for misclassifications. Dual Quadratic Problem Stationarity conditions (with respect to all primal variables): w L(w, b, ξ, z, s) = w n y i x i z i = 0 i= ξi L(w, b, ξ, z, s) = C z i s i = 0 b L(w, b, ξ, z, s) = n i= y i z i = 0. Substituting these equations into the Lagrangian function we get n L(w, b, ξ, z, s) = z i n y 2 i y j (x T i x j)z i z j. i= i,j= Montreal, 8 June 2009 25 Montreal, 8 June 2009 27 We will derive the dual quadratic problem. We associate Lagrange multipliers z R n (z 0) and s R n (s 0) with the constraints y i (w T x i + b) + ξ i and ξ 0, and write the Lagrangian L(w, b, ξ, z, s) = 2 wt w + C n ξ i i= n z i (y i (w T x i + b)+ ξ i ) s T ξ. i= SVM community uses α instead of z. Hence the dual problem has the form: n max z i n 2 y i y j (x T i x j)z i z j s.t. i= i,j= n y i z i = 0, i= SVM community uses α instead of z. 0 z i C, i =, 2,..., n, Montreal, 8 June 2009 26 Montreal, 8 June 2009 28
Dual Quadratic Problem (continued) Observe that the dual problem has a neat formulation in which only dual variables z are present. (The primal variables (w, b, ξ) do not appear in the dual.) Define a dense matrix Q R n n such that q ij = y i y j (x T i x j). Rewrite the (dual) quadratic program: Part 3: IPMs for Support Vector Machine training max e T z 2 zt Qz, s.t. y T z = 0, 0 z Ce, where e is the vector of ones in R n. The matrix Q corresponds to a specific linear kernel function. Montreal, 8 June 2009 29 Montreal, 8 June 2009 3 Dual Quadratic Problem (continued) The primal problem is convex hence the dual problem must be well defined too. The dual problem is to maximise the concave function. We can prove it directly. Lemma: Proof: The matrix Q is positive semidefinite. Define = [y x y 2 x 2... y n x n ] T R n m and observe that Q = T (i.e., q ij = y i y j (x T i x j)). For any z R n we have z T Qz = (z T )( T z) = T z 2 0 hence Q is positive semidefinite. Montreal, 8 June 2009 30 Interior Point Methods in SVM Context Fine and Scheinberg, Efficient SVM training using low-rank kernel representations, J. of Machine Learning Res., 2(2002) 243-264. Ferris and Munson, Interior point methods for massive support vector machines, SIAM J. on Optimization, 3(2003) 783-804. Woodsend and ondzio, Exploiting separability in large-scale linear SVM training, Tech Rep MS-07-002, Edinburgh 2007. http://www.maths.ed.ac.uk/~gondzio/reports/wgsvm.html Unified framework which includes: Classification (l and l 2 error) Universum SVM Ordinal Regression Regression Reformulate QPs as separable. Montreal, 8 June 2009 32
IPMs for SVMs: Exploit separability Key trick: represent Q = F T DF, where F R k n, k n. Introduce new variable u = F z. Observe: z T Qz = z T F T DF z = u T Du. min c T z + 2 zt Qz s.t. Az = b, z 0. min c T z + 2 ut Du s.t. Az = b, F z u = 0, z 0. Comparison: SVM-HOPDM vs other algorithms Data with 255 attributes. C = 00, 0% misclassified. Training time (s) 000 00 0 SVM-HOPDM SVMlight SVMPerf LibLinear LibSVM SVMTorch SVM-QP SVM-QP Presolve non-separable QP separable QP m constraints m + k constraints n variables n + k variables Montreal, 8 June 2009 33 0. 000 0000 Size of training data set Montreal, 8 June 2009 35 Comparison: SVM-HOPDM vs other algorithms Data with 255 attributes. C =, 0% misclassified. Training time (s) 000 00 0 0. SVM-HOPDM SVMlight SVMPerf LibLinear LibSVM SVMTorch SVM-QP SVM-QP Presolve 000 0000 Size of training data set Montreal, 8 June 2009 34 Comparison: of training times using real-world data sets. Each data set was trained using C =, 0 and 00. NC indicates that the method did not converge to a solution. Data set C SVM- SVM light SVM perf Lib- LibSVM SVMTorch SVM-QP SVM-QP (n m) HOPDM Linear presolve Adult 6.5 87.7 280.7.6 92.4 62.8 64.5 88.8 3256 23 0 26.5 043.3 3628.0 9.3 857.7 5046.0 284. 206.8 00 27.9 0447.4 2947.2 64.2 5572. 44962.5 544.8 26.9 Covtype 47.7 992.4 795.6 8.5 2085.8 287.9 73.8 405.6 50000 54 0 52.7 602.2 2274.5 34.3 256.7 0880.6 97.6 44.3 00 55.4 66263.8 58699.8 235.2 6588.0 7448. 58.8 457.4 MNIST 79.6 262.9 754. 9.3 97. 660. 233.0 09. 0000 780 0 83.4 3425.5 8286.8 65.4 275.2 5748. 349.4 04.4 00 86.2 NC 96789.0 NC 456.4 54360.6 602.5 267. SensIT 55.2 93.5 848.3 53.6 2542.0 284.4 535.2 456.7 78823 00 0 60. 7797.4 > 25000 369. 7867.8 227.8 875.4 470.7 00 63.6 NC > 25000 NC 49293.7 204642.6 650. 489.3 USPS 3.2 5.0 40.9 4.4 0.4 7.7 5.2 7.4 729 256 0 4.2 47.4 346.6 27.7 20.9 23.9 64.7 27.4 00 4.3 345.2 2079.5 NC 93.8 42.4 86.9 43.8 Montreal, 8 June 2009 36
Computational effort of IPM-based SVM implementation: build H = A(Q + Θ ) A T O(nm 2 ) compute Cholesky H = LΛL T O(m 3 ) Attempts to reduce this effort ertz and riffin, SVM classifiers for large datasets, Tech Rep ANL/MCS-TM-289, Argonne National Lab, 2005. use iterative method (preconditioned conjugate gradient). Jung, O Leary and Tits, Adaptive constraint reduction for training SVMs, Electronic Trans on Num Analysis 3(2008) 56-77. use a subset of points n n mimic active-set strategy within IPM. Montreal, 8 June 2009 37 Parallelism (continued) Cholesky factor preserves block-structure: H i = L i Λ i L T i, L i = I, Λ i = H i, i =..p L Ai = A i L T i Λ i = A i Hi, i =..p H 0 = p i= A ihi A T i = L 0 Λ 0 L T 0 [ ] [ ] v r And the system H λ = h is solved by t i = L i r i, i =..p t 0 = L 0 (h L Ai t i ) q i = Λ i t i, i = 0..p λ = L T 0 q 0 v i = L T i (q i L T A i λ), i =..p Operations (Cholesky, Solve, Product) performed on sub-blocks Montreal, 8 June 2009 39 Parallelism Exploit bordered block-diagonal structure in augm. system reak H into blocks: H A T H 2 A T 2 H =.... H p A T, p A A 2... A p 0 and decompose L Λ...... H = L p L A... L Ap L 0 Λ p Λ0 L T... L T A. L T p L T A p L T 0 Montreal, 8 June 2009 38 Comparison: Parallel software PASCAL Large Scale Learning Challenge http://largescale.first.fraunhofer.de/about/ Data set n m Alpha 500000 500 eta 500000 500 amma 500000 500 Delta 500000 500 Espilon 500000 2000 Zeta 500000 2000 FD 2560000 900 OCR 3500000 56 DNA 6000000 800 OOPS Object-Oriented Parallel Solver http://www.maths.ed.ac.uk/~gondzio/parallel/solver.html Montreal, 8 June 2009 40
Dataset # cores C OOPS PPDT PSVM Milde Alpha 6 39 3673 684 (806) 0.0 50 4269 4824 (8520) eta 6 20 5003 2390 (83407) 0.0 48 4738 486 (8494) amma 6 44 685 (8375) 0.0 49 795 480 (84445) Delta 6 40 6 (5763) 0.0 46 9492 4865 (8442) Epsilon 32 730 7436 (58488) 0.0 293 3639 (56984) Zeta 32 544 4368 (2284) 0.0 297 37283 (68059) FD 32 399 (39227) 0.0 252 (52408) OCR 32 36 (58307) 0.0 330 (36523) DNA 48 2668 0.0 6557 482 Montreal, 8 June 2009 4 Accuracy measured using area under precision recall curve. Evaluation results taken from PASCAL Challenge website. Dataset OOPS LibLinear LaRank Alpha 0.345 0.60 0.606 eta 0.4988 0.4988 0.500 amma 0.74 0.85 0.87 Delta 0.344 0.346 0.355 Epsilon 0.034 0.4935 0.493 Zeta 0.05 0.493 0.4875 FD 0.2274 0.2654 0.308 OCR 0.595 0.660 0.68 Montreal, 8 June 2009 43 Dataset C OOPS LibLinear LaRank n # cores Time n Time n Time Alpha 500,000 6 39 500,000 47 500,000 3354 0.0 50 2 2474 eta 500,000 6 20 500,000 35 500,000 6372 0.0 48 2 880 amma 500,000 6 44 500,000 (8845) 500,000 0.0 49 348 2038 Delta 500,000 6 40 500,000 (3266) 500,000 0.0 46 429 Epsilon 500,000 32 730 250,000 36 500,000 5599 0.0 293 265 240 Zeta 500,000 32 544 250,000 278 500,000 0.0 297 248 FD 2,560,000 32 399 500,000 23 500,000 537 0.0 252 93 332 OCR 3,500,000 32 36 250,000 8 500,000 5695 0.0 330 2 4266 DNA 6,000,000 48 2668 600,000 44 600,000 300 0.0 6557 30 407 Montreal, 8 June 2009 42 Nonlinear and/or Indefinite Kernels: A kernel is a function K, such that for all x i, x j X K(x i, x j ) = φ(x i ), φ(x j ), where φ is a mapping from X to an (inner product) feature space F. We use.,. to denote a scalar product. Linear Kernel K(x i, x j ) = x T i x j. Polynomial Kernel K(x i, x j ) = (x T i x j + ) d. aussian Kernel K(x i, x j ) = e x i x j 2 σ 2. Montreal, 8 June 2009 44
Nonlinear and/or Indefinite Kernels: Kernels are in general dense matrices making large-scale SVM training computationally demanding (or impossible). Challenge: How to approximate kernels? Find Q with desirable properties such that distance(q, Q) is minimized. The distance may be measured with a matrix norm (say, Frobenius) or a regman divergence. Dhillon and Tropp, Matrix nearness problems with regman divergences, SIAM J. on Matrix An. & Appl., 29(2007) 20-46. Lanckriet, Cristianini, artlett, haoui and Jordan, Learning the kernel matrix with semidefinite programming, Journal of Machine Learning Research 5 (2004), 27-72. Montreal, 8 June 2009 45 Conclusions: Interior Point Methods are well-suited to large scale optimization Support Vector Machine training requires a solution of very large optimization problem IPMs provide an attractive approach to solve SVM training problems Montreal, 8 June 2009 47 IPM perspective: nonlinear/indefinite Kernels: Approximate kernel matrix Q, Q ij = K(x i, x j ) using low rank outer product Q LΛL T + D, where L R n k, k n. Exploit separability within IPMs Augmented system becomes: H = Montreal, 8 June 2009 46 L T L Thank you for your attention! Woodsend and ondzio, Exploiting separability in large-scale linear SVM training, Tech Rep MS-07-002, Edinburgh 2007. http://www.maths.ed.ac.uk/~gondzio/reports/wgsvm.html Woodsend and ondzio, Hybrid MPI/OpenMP parallel linear SVM training, Tech Rep ERO-09-00, Edinburgh, 2009. http://www.maths.ed.ac.uk/~gondzio/reports/wghybridsvm.html Montreal, 8 June 2009 48