Intel Math Kernel Library (Intel MKL) LAPACK

Transcription

1 Intel Math Kernel Library (Intel MKL) LAPACK Linear equations Victor Kostin Intel MKL Dense Solvers team manager

2 LAPACK Systems of Linear Equations Linear Least Squares Eigenvalue Problems Singular Value Decomposition Matrix factorizations LU QR Cholesky Schur decomposition

3 LAPACK: 3 parts 3

4 ScaLAPACK: Linear Algebra on clusters 4

5 Systems of linear equations a 11 x 1 + a 12 x 2 + a 13 x a 1,N 1 x N 1 + a 1N x N = f 1 a 21 x 1 + a 22 x 2 + a 23 x a 2,N 1 x N 1 + a 2N x N = f 2 a 31 x 1 + a 32 x 2 + a 33 x a 3,N 1 x N 1 + a 3N x N = f 3 a M1 x 1 + a M2 x 2 + a M3 x a M,N 1 x N 1 + a M,N x N = f M Represent this system of M equations for N unknowns in matrix-vector form a 11 a 12 a 13 a 1,N 1 a 1N a 21 a 22 a 23 a 2,N 1 a 2N a 31 a 32 a 33 a 3,N 1 a 3N a M 1,1 a M 1,2 a M 1,3 a M 1,N 1 a M 1,N a M1 a M2 a M3 a M,N 1 a MN x 1 x 2 x 3 x N 1 x N = f 1 f 2 f 3 f M 1 f M or even shorter Ax = f Given coefficient matrix A and right hand side vector f we have to find vector x. 5

6 Algorithm sketch Roughly speaking, solving system of linear equations Ax = f consists of Factorization step: Given an N-by-N matrix A find a lower triangular matrix L and an upper triangular matrix U such that A = L U Matrix A must be nonsingular Solving steps: Find solutions of 2 systems of linear equations Ly = f Ux = y. 6

7 Factors L and U L = 1 l 21 1 l 31 l 32 1 l N 1,1 l N 1,2 l N 1,3 1 l N1 l N2 l N3 l N,N 1 1, U = u 11 u 12 u 13 u 1,N 1 u 1N u 22 u 23 u 2,N 1 u 2N u 33 u 3,N 1 u 3N u N 1,N 1 u N 1,N u NN Blocking A 11 A 12 A 21 A 22 = L 11 L 21 L 22 U 11 U 12 U 22 = L 11 U 11 L 11 U 12 L 21 U 11 L 21 U 12 + L 22 U Factorize L 11 U 11 = A 11 (see next slide) 2. Find L 21 by solving a system with triangular coefficient matrix L 21 U 11 = A Find U 12 by solving a system with triangular coefficient matrix L 11 U 12 = A Update the block A 22 A 22 L 21 U 12 Factorize updated L 22 U 22 = A 22 GEMM TRSM 7

8 LU factorization formulas a 11 a 12 a 13 a 21 a 22 a 23 = a 31 a 32 a 33 = l l 31 l 32 1 u 11 u 12 u 13 0 u 22 u u 33 u 11 u 12 u 13 l 21 u 11 l 21 u 12 + u 22 l 21 u 13 + u 23 l 31 u 11 l 31 u 12 + l 32 u 22 l 31 u 13 + l 32 u 23 + u 33 u 11 = a 11 l 21 = a 21 u 11 u 22 = a 22 l 21 u 12 u 12 = a 12 l 31 = a 31 u 11 u 23 = a 22 l 21 u 13 u 13 = a 13 l 32 = a 32 l 31 u 12 u 22 u 33 = a 33 l 31 u 13 l 32 u 23 8

9 LU in MKL LAPACK If using Intel MKL, to find triangular factors just call?getrf().?getrf() factorizes matrix A as A = P L U Here P is permutation matrix (partial pivoting) chosen to improve numerical stability of computations. Action P on a matrix from the left is just permuting the rows of the matrix. Syntax call DGETRF(M, N, A, LDA, IPIV, INFO) A N P is returned via IPIV Matrix can be rectangular (M N)! A is stored as an MxN-submatrix in LDAxN array (LDA M). Factors L and U replace elements of A (A is destroyed). No need to store unit diagonal of L. INFO: =0, computation successful =-i, i-th argument has illegal value =i, u ii is exactly zero. Factorization completed but U is singular. M LDA 9

10 Solving systems of linear equations with triangular coefficient matrices: formulas Ux = y x N = y N u NN u 11 u 12 u 13 u 1,N 1 u 1N u 22 u 23 u 2,N 1 u 2N u 33 u 3,N 1 u 3N u N 1,N 1 u N 1,N u NN x 1 x 2 x 3 x N 1 x N = y 1 y 2 y 3 y N 1 y N x N 1 = y N 1 u N 1,N x N u N 1,N 1 x 1 = y 1 N j=2 u 11 u 1j x j Solving system of linear equations Ly = f (which must be solved first) is even simpler due to unit values on the diagonal of L (no division). 10

11 Solving systems of linear equations with triangular coefficient matrices: Intel MKL function Assuming A is factorized the solution can be obtained by calling DGETRS(TRANS, N, NRHS, A, LDA, IPIV, B, LDB, INFO) TRANS= T or N or C depending on the system to solve: Ax = f or A T x = f or A H x = f respectively. C is applicable for complex matrices only. NRHS is the Number of Right Hand Side vectors. Several systems can be solved at once. Right-hand-side columns form NxNRHS-matrix stored in array B(LDB, NRHS), LDB N INFO =0, execution was successful =-i, the i-th parameter had an illegal value 11

12 Driver routines?gesv() can do both (factorization + solving) steps in one call CALL DGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO ) A will be replaced with factors B will be replaced with solution INFO =0, execution is successful =-i, the i-th parameter has an illegal value =i, i-th diagonal element u ii is exactly zero. Computation cannot be completed.?gesvx() computes the solution and provides error bounds on the solution and condition number estimation Forward error estimate FERR= x x comp Backward error estimate BERR=min( δa A smallest to satisfy A + δa x comp = f + δf x., δf f Condition number estimate cond(a)= A A 1 ) where δf, δa are the 12

13 More Factorization Routines LU factorization of band and tridiagonal matrix:?gbtrf(),?gttrf() A = P L U Cholesky factorization of real symmetric or complex Hermitian positive definite matrix:?potrf(),?pftrf(),?pptrf() A = L L T or A = L L H Cholesky factorization with complete pivoting of a real symmetric (complex Hermitian) positive semidefinite matrix:?pstrf() P T A P = L L T or P T A P = L L H LDLT factorization of real symmetric or complex Hermitian matrix:?sytrf(),?hetrf(),?sptrf(),?hptrf() A = P L D L T P And even more (see MKL reference manual). 13

14 More triangular solvers LU-factored band matrices LU-factored tridiagonal matrices Cholesky-factored symmetric (Hermitian) positive definite matrices LDLT-factored symmetric matrices LDLT-factored Hermitian matrices Triangular matrices 14

15 Estimating errors (1/2) Consider 2 systems of equations Ax = f and A x + x the same matrix but different RHS. x = A 1 f A 1 f = A 1 A 1 f f Ax A 1 This implies x x f f A x ( A A 1 ) Condition number of matrix A: cond(a)= f f f f f = A A 1 if A is nonsingular if A is singular = f + f with Relative error of solution is estimated by relative perturbation of RHS and proportional to the condition number of the matrix. This estimate is reachable (the inequality may become an equality for specially chosen RHS Practically, a matrix is called well-conditioned if cond(a) is not very big. Otherwise, it is called ill-conditioned. 15

16 Estimating errors (2/2) Let x comp denote the computed solution of the system Ax = f. How can we estimate x x comp? x Compute the residual Ax comp = f + f. Accurate multiplication of the vector x comp by the matrix A is needed. Applying the estimate from slide #16 we get x x comp f cond(a). x f But we need to know the condition number cond(a). In fact, we have to be sure the matrix is well-conditioned because otherwise factorization can be inaccurate and computation unstable. 16

17 Condition number estimators For matrices stored in different formats the listed functions estimate cond 1 A = A 1 A 1 1 cond A = A A 1 Norms A 1 = max j i a ij or A = max i j a ij are to be provided and the matrix factored. Norms can be computed via call?lange(norm,m,n,a,lda,work) or its equivalent for other matrix types. 17

18 Routine Naming Convention LAPACK names have the structure?yyzzz? Data type s c d z zzz trf trs con rfs rfsx tri equ, equb real, single precision complex, single precision real, double precision complex, double precision Computation performed perform a triangular matrix factorization solve the linear system with a factored matrix estimate the matrix condition number refine the solution and compute error bounds refine the solution and compute error bounds using extra-precise iterative refinement compute the inverse matrix using the factorization equilibrate a matrix. yy ge gb gt dt po pp pf pb pt sy sp he hp tr tp tf tb Matrix type and storage scheme general general band general tridiagonal diagonally dominant tridiagonal symmetric or Hermitian positive-definite symmetric or Hermitian positive-definite (packed storage) symmetric or Hermitian positive-definite (RFP storage) symmetric or Hermitian positive-definite band symmetric or Hermitian positive-definite tridiagonal symmetric indefinite symmetric indefinite (packed storage) Hermitian indefinite Hermitian indefinite (packed storage) triangular triangular (packed storage) triangular (RFP storage) triangular band E.g., DPOTRF = Triangular (Cholesky) factorization of double precision symmetric positive definite matrix 18

19 Matrix Storage Schemes Full storage - a matrix is stored in a two-dimensional array Packed storage scheme symmetric, Hermitian or triangular matrices are stored more compactly (the upper or lower triangle is packed by columns in one-dimensional array) Band storage: a band matrix is stored an a twodimensional array compactly (columns are in columns, diagonals are in rows) Rectangular Full Packed storage(rfp) combining the full and packed storage to maintain efficiency of BLAS3. Bi/tridiagonal matrices can be stored in 2 (or 3) onedimensional arrays. 19

20 Q & A 20

21 Legal Disclaimer & Optimization Notice INFORMATION IN THIS DOCUMENT IS PROVIDED AS IS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Copyright, Intel Corporation. All rights reserved. Intel, the Intel logo, Intel Core, Intel Inside, the Intel Inside logo, Itanium, Itanium Inside, Pentium, Pentium Inside, Xeon, Xeon Phi, Core, VTune, and Cilk are trademarks of Intel Corporation in the U.S. and other countries. Optimization Notice Intel s compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #

22

23 Condition number: general case (1/2) More complicated case A + A x + x = f + f Assuming nonsingularity of A investigate A + A. A + A 1 = I + A A 1 A 1 = A 1 I A A 1 + A A 1 2 Neumann series converges if A A 1 < 1. Assume stronger condition A A 1 A A 1 = A A A 1 A = cond A A A < 1 23

24 Condition number: general case (2/2) Under this assumption we get A + A 1 One can also derive x x cond(a) A 1 1 cond A f f + A A 1 cond A A A A A Provided matrix A is nonsingular the condition cond A A A < 1 guarantees nonsingularity of the perturbed matrix A + A 24