Using interior point methods for optimization in training very large scale Support Vector Machines. Interior Point Methods for QP. Elements of the IPM

Transcription

1 School of Mathematics T H E U N I V E R S I T Y O H Using interior point methods for optimization in training very large scale Support Vector Machines F E D I N U R Part : Interior Point Methods for QP Jacek ondzio J.ondzio@ed.ac.uk URL: joint work with Kristian Woodsend Montreal, 8 June 2009 Montreal, 8 June Outline Interior Point Methods for QP logarithmic barrier function complementarity conditions linear algebra Support Vector Machine training Quadratic Programming formulation specific features IPMs for SVM training separable formulations!!! linear and nonlinear kernels in SVMs Challenges remaining nonlinear kernels in SVMs indefinite kernels in SVMs Conclusions Montreal, 8 June Elements of the IPM What do we need to derive the Interior Point Method? logarithmic barriers. duality theory: Lagrangian function; first order optimality conditions. Newton method. Wright, Primal-Dual Interior-Point Methods, SIAM, 997. Andersen, ondzio, Mészáros and Xu, Implementation of Interior Point Methods for Large Scale Linear Programming, in: Interior Point Methods in Mathematical Programming, T Terlaky (ed.), Kluwer Academic, 996, pp Montreal, 8 June

2 Logarithmic barrier ln v i replaces the inequality v i 0. Observe that min e n i= ln v i max n ln x The minimization of n i= ln v i is equivalent to the maximization of the product of distances from all hyperplanes defining the positive orthant: it prevents all v i from approaching zero. Montreal, 8 June i= v i x Conditions for a stationary point of the Lagrangian v L(v, λ, µ) = c A T λ + Qv µv e = 0 λ L(v, λ, µ) = Av b = 0, where V = diag{v, v 2,, v n }. Let us denote s = µv e, i.e. V Se = µe. The First Order Optimality Conditions are: Av = b, A T λ + s Qv = c, V Se = µe, (v, s) > 0. Montreal, 8 June Logarithmic barrier Replace the primal QP min c T v + 2 vt Qv s.t. Av = b, v 0, with the primal barrier program min c T v + 2 vt Qv µ n s.t. Av = b. j= ln v j First Order Optimality Conditions Active-set Method: Av = b A T λ + s Qv = c V Se = 0 v, s 0. Interior Point Method: Av = b A T λ + s Qv = c V Se = µe v, s 0. Lagrangian: L(v, λ, µ) = c T v + n 2 vt Qv λ T (Av b) µ ln v j. j= Montreal, 8 June Montreal, 8 June

3 Complementarity v i s i = 0 i =, 2,..., n. Active-set Method makes a guess of optimal partition: A I = {, 2,..., n}. For active constraints (i A), v i = 0 and v i s i = 0 i A. For inactive constraints (i I), s i = 0 hence v i s i = 0 i I. Interior Point Method uses ε-mathematics: Replace v i s i = 0 i =, 2,..., n by v i s i = µ i =, 2,..., n. Force convergence µ 0. Montreal, 8 June Newton Method (cont d) Note that f(v, λ, s) = A 0 0 Q A T I. S 0 V Thus, for a given point (v, λ, s) we find the Newton direction ( v, λ, s) by solving the system of linear equations: A 0 0 [ ] v Q A T I λ = b Av c A T λ s + Qv. S 0 V s µe V Se Montreal, 8 June 2009 Apply Newton Method to the FOC The first order optimality conditions for the barrier problem form a large system of nonlinear equations f(v, λ, s) = 0, where f : R 2n+m R 2n+m is a mapping defined as follows: Av b f(v, λ, s) = A T λ + s Qv c. V Se µe Actually, the first two terms of it are linear; only the last one, corresponding to the complementarity condition, is nonlinear. Montreal, 8 June Linear Algebra of IPM for QP A 0 0 Q A T I S 0 V Use the third equation to eliminate [ ] [ ] v ξp λ = ξ d = s ξ µ s = V (ξ µ S v) = V S v + V ξ µ, from the second equation and get [ Q Θ A T ] [ ] v A 0 λ = where Θ = V S is a diagonal scaling matrix. Θ is always very ill-conditioned. b Av c A T λ s + Qv. µe V Se [ ξd V ] ξ µ. ξ p Montreal, 8 June

4 Augmented system [ Q Θ A T ] [ ] [ ] v r A 0 λ = h = Symmetric but indefinite linear system. In general, it may be difficult to solve. Separable Quadratic Programs [ ξd V ] ξ µ. ξ p When matrix Q is diagonal (Q = D), the augmented system can be further reduced. Eliminate v = (D + Θ ) (A T λ r), to get normal equations (symmetric, positive definite system) (A(D + Θ ) A T ) λ = g = A(D + Θ ) r + h. Interior Point Methods Theory: IPMs converge in O( n) or O(n) iterations Practice: IPMs converge in O(log n) iterations... but one iteration may be expensive! Suppose A R m n is a dense matrix. Major computational effort when solving separable QP (separable QP means that Q = D, diagonal). build H = A(Q + Θ ) A T O(nm 2 ) compute Cholesky H = LΛL T O(m 3 ) Recall n m. Montreal, 8 June Montreal, 8 June Sparsity Issues in QP Observation: the inverse of the sparse matrix may be dense. Example = = = IPMs for QP: Do not explicitly invert the matrix Q + Θ to form A(Q + Θ ) A T unless Q is diagonal. Montreal, 8 June Ill-conditioning of Θ = V S For active constraints: Θ j = v j /s j 0 Θ j ; For inactive constraints: Θ j = v j /s j Θ j 0. oldfarb and Scheinberg, A product form Cholesky factorization for handling dense columns in IPMs for linear programming, Mathematical Programming, 99(2004) -34. Although Θ = (Q + Θ ) behaves badly, the Cholesky factorization H = LΛL T behaves well: Λ captures instability (variability) of Θ; L is well conditioned (bounded independently of Θ). Represent L = L L 2... L m, where L i has entries only in column i. Drawback: PFCF is sequential by nature. Montreal, 8 June

5 Interior Point Methods: Summary Interior Point Methods for QP polynomial algorithms excellent practical behaviour competitive for small problems (,000,000 vars) beyond competition for large problems (,000,000 vars) Opportunities for SVM training with IPMs dense data very large size well-suited to parallelism Classification We consider a set of points X = {x, x 2,..., x n }, x i R m to be classified into two subsest of good and bad ones. X = and =. We look for a function f : X R such that f(x) 0 if x and f(x) < 0 if x. Usually n m. Montreal, 8 June Montreal, 8 June Linear Classification We consider a case when f is a linear function: Part 2: Support Vector Machine training f(x) = w T x + b, where w R m and b R. In other words we look for a hyperplane which separates good points from bad ones. In such case the decision rule is given by y = sgn(f(x)). If f(x i ) 0, then y i = + and x i. If f(x i ) < 0, then y i = and x i. We say that there is a linearly separable training sample S = ((x, y ), (x 2, y 2 ),..., (x n, y n )). Montreal, 8 June Montreal, 8 June

6 How does it work? iven a linearly separable database (training sample) S = ((x, y ), (x 2, y 2 ),..., (x n, y n )) find a separating hyperplane w T x + b = 0, which satisfies y i (w T x i + b), i =, 2,..., n. iven a new (unclassified) point x 0, compute y 0 = sgn(w T x 0 + b) to decide whether x 0 is good or bad. Montreal, 8 June QP Formulation Finding a separating hyperplane can be formulated as a quadratic programming problem: min 2 w T w s.t. y i (w T x i + b), i =, 2,..., n. In this formulation the Euclidean norm of w is minimized. This is clearly a convex optimization problem. (We can minimize w or w and then the problem can be reformulated as an LP.) Two major difficulties: Clusters may not be separable at all minimize the error of misclassifications; Clusters may be separable by a nonlinear manifold find the right feature map. Montreal, 8 June Separating Hyperplane To guarantee a nonzero margin of separation we look for a hyperplane w T x + b = 0, such that w T x i + b for good points; w T x i + b for bad points. This is equivalent to: w T x i w + w b w w T x i w + w b w for good points; for bad points. In this formualtion the normal vector of the separating hyperplane has unit length. In this case the margin between good and w w bad points is measured by 2 w. This margin should be maximised. This can be achieved by minimising the norm w. Montreal, 8 June Difficult Cases Nonseparable clusters: ξ2 ξ Use nonlinear feature map: Φ Errors when defining clusters of good and bad points. Minimize the global error of misclassifications: ξ + ξ 2. Montreal, 8 June

7 Linearly nonseparable case If perfect linear separation is impossible then for each misclassified data we introduce a slack variable ξ i which measures the distance between the hyperplane and misclassified data. Finding the best hyperplane can be formulated as a quadratic programming problem: min 2 w T w + C n ξ i i= s.t. y i (w T x i + b) + ξ i, i =, 2,..., n, ξ i 0 i =, 2,..., n, where C (C > 0) controls the penalisation for misclassifications. Dual Quadratic Problem Stationarity conditions (with respect to all primal variables): w L(w, b, ξ, z, s) = w n y i x i z i = 0 i= ξi L(w, b, ξ, z, s) = C z i s i = 0 b L(w, b, ξ, z, s) = n i= y i z i = 0. Substituting these equations into the Lagrangian function we get n L(w, b, ξ, z, s) = z i n y 2 i y j (x T i x j)z i z j. i= i,j= Montreal, 8 June Montreal, 8 June We will derive the dual quadratic problem. We associate Lagrange multipliers z R n (z 0) and s R n (s 0) with the constraints y i (w T x i + b) + ξ i and ξ 0, and write the Lagrangian L(w, b, ξ, z, s) = 2 wt w + C n ξ i i= n z i (y i (w T x i + b)+ ξ i ) s T ξ. i= SVM community uses α instead of z. Hence the dual problem has the form: n max z i n 2 y i y j (x T i x j)z i z j s.t. i= i,j= n y i z i = 0, i= SVM community uses α instead of z. 0 z i C, i =, 2,..., n, Montreal, 8 June Montreal, 8 June

8 Dual Quadratic Problem (continued) Observe that the dual problem has a neat formulation in which only dual variables z are present. (The primal variables (w, b, ξ) do not appear in the dual.) Define a dense matrix Q R n n such that q ij = y i y j (x T i x j). Rewrite the (dual) quadratic program: Part 3: IPMs for Support Vector Machine training max e T z 2 zt Qz, s.t. y T z = 0, 0 z Ce, where e is the vector of ones in R n. The matrix Q corresponds to a specific linear kernel function. Montreal, 8 June Montreal, 8 June Dual Quadratic Problem (continued) The primal problem is convex hence the dual problem must be well defined too. The dual problem is to maximise the concave function. We can prove it directly. Lemma: Proof: The matrix Q is positive semidefinite. Define = [y x y 2 x 2... y n x n ] T R n m and observe that Q = T (i.e., q ij = y i y j (x T i x j)). For any z R n we have z T Qz = (z T )( T z) = T z 2 0 hence Q is positive semidefinite. Montreal, 8 June Interior Point Methods in SVM Context Fine and Scheinberg, Efficient SVM training using low-rank kernel representations, J. of Machine Learning Res., 2(2002) Ferris and Munson, Interior point methods for massive support vector machines, SIAM J. on Optimization, 3(2003) Woodsend and ondzio, Exploiting separability in large-scale linear SVM training, Tech Rep MS , Edinburgh Unified framework which includes: Classification (l and l 2 error) Universum SVM Ordinal Regression Regression Reformulate QPs as separable. Montreal, 8 June

9 IPMs for SVMs: Exploit separability Key trick: represent Q = F T DF, where F R k n, k n. Introduce new variable u = F z. Observe: z T Qz = z T F T DF z = u T Du. min c T z + 2 zt Qz s.t. Az = b, z 0. min c T z + 2 ut Du s.t. Az = b, F z u = 0, z 0. Comparison: SVM-HOPDM vs other algorithms Data with 255 attributes. C = 00, 0% misclassified. Training time (s) SVM-HOPDM SVMlight SVMPerf LibLinear LibSVM SVMTorch SVM-QP SVM-QP Presolve non-separable QP separable QP m constraints m + k constraints n variables n + k variables Montreal, 8 June Size of training data set Montreal, 8 June Comparison: SVM-HOPDM vs other algorithms Data with 255 attributes. C =, 0% misclassified. Training time (s) SVM-HOPDM SVMlight SVMPerf LibLinear LibSVM SVMTorch SVM-QP SVM-QP Presolve Size of training data set Montreal, 8 June Comparison: of training times using real-world data sets. Each data set was trained using C =, 0 and 00. NC indicates that the method did not converge to a solution. Data set C SVM- SVM light SVM perf Lib- LibSVM SVMTorch SVM-QP SVM-QP (n m) HOPDM Linear presolve Adult Covtype MNIST NC NC SensIT > NC > NC USPS NC Montreal, 8 June

10 Computational effort of IPM-based SVM implementation: build H = A(Q + Θ ) A T O(nm 2 ) compute Cholesky H = LΛL T O(m 3 ) Attempts to reduce this effort ertz and riffin, SVM classifiers for large datasets, Tech Rep ANL/MCS-TM-289, Argonne National Lab, use iterative method (preconditioned conjugate gradient). Jung, O Leary and Tits, Adaptive constraint reduction for training SVMs, Electronic Trans on Num Analysis 3(2008) use a subset of points n n mimic active-set strategy within IPM. Montreal, 8 June Parallelism (continued) Cholesky factor preserves block-structure: H i = L i Λ i L T i, L i = I, Λ i = H i, i =..p L Ai = A i L T i Λ i = A i Hi, i =..p H 0 = p i= A ihi A T i = L 0 Λ 0 L T 0 [ ] [ ] v r And the system H λ = h is solved by t i = L i r i, i =..p t 0 = L 0 (h L Ai t i ) q i = Λ i t i, i = 0..p λ = L T 0 q 0 v i = L T i (q i L T A i λ), i =..p Operations (Cholesky, Solve, Product) performed on sub-blocks Montreal, 8 June Parallelism Exploit bordered block-diagonal structure in augm. system reak H into blocks: H A T H 2 A T 2 H =.... H p A T, p A A 2... A p 0 and decompose L Λ H = L p L A... L Ap L 0 Λ p Λ0 L T... L T A. L T p L T A p L T 0 Montreal, 8 June Comparison: Parallel software PASCAL Large Scale Learning Challenge Data set n m Alpha eta amma Delta Espilon Zeta FD OCR DNA OOPS Object-Oriented Parallel Solver Montreal, 8 June

11 Dataset # cores C OOPS PPDT PSVM Milde Alpha (806) (8520) eta (83407) (8494) amma (8375) (84445) Delta (5763) (8442) Epsilon (58488) (56984) Zeta (2284) (68059) FD (39227) (52408) OCR (58307) (36523) DNA Montreal, 8 June Accuracy measured using area under precision recall curve. Evaluation results taken from PASCAL Challenge website. Dataset OOPS LibLinear LaRank Alpha eta amma Delta Epsilon Zeta FD OCR Montreal, 8 June Dataset C OOPS LibLinear LaRank n # cores Time n Time n Time Alpha 500, , , eta 500, , , amma 500, ,000 (8845) 500, Delta 500, ,000 (3266) 500, Epsilon 500, , , Zeta 500, , , FD 2,560, , , OCR 3,500, , , DNA 6,000, , , Montreal, 8 June Nonlinear and/or Indefinite Kernels: A kernel is a function K, such that for all x i, x j X K(x i, x j ) = φ(x i ), φ(x j ), where φ is a mapping from X to an (inner product) feature space F. We use.,. to denote a scalar product. Linear Kernel K(x i, x j ) = x T i x j. Polynomial Kernel K(x i, x j ) = (x T i x j + ) d. aussian Kernel K(x i, x j ) = e x i x j 2 σ 2. Montreal, 8 June

12 Nonlinear and/or Indefinite Kernels: Kernels are in general dense matrices making large-scale SVM training computationally demanding (or impossible). Challenge: How to approximate kernels? Find Q with desirable properties such that distance(q, Q) is minimized. The distance may be measured with a matrix norm (say, Frobenius) or a regman divergence. Dhillon and Tropp, Matrix nearness problems with regman divergences, SIAM J. on Matrix An. & Appl., 29(2007) Lanckriet, Cristianini, artlett, haoui and Jordan, Learning the kernel matrix with semidefinite programming, Journal of Machine Learning Research 5 (2004), Montreal, 8 June Conclusions: Interior Point Methods are well-suited to large scale optimization Support Vector Machine training requires a solution of very large optimization problem IPMs provide an attractive approach to solve SVM training problems Montreal, 8 June IPM perspective: nonlinear/indefinite Kernels: Approximate kernel matrix Q, Q ij = K(x i, x j ) using low rank outer product Q LΛL T + D, where L R n k, k n. Exploit separability within IPMs Augmented system becomes: H = Montreal, 8 June L T L Thank you for your attention! Woodsend and ondzio, Exploiting separability in large-scale linear SVM training, Tech Rep MS , Edinburgh Woodsend and ondzio, Hybrid MPI/OpenMP parallel linear SVM training, Tech Rep ERO-09-00, Edinburgh, Montreal, 8 June