Department of Computer Science, University of Illinois at Urbana-Champaign Probing for Schur Complements and Preconditioning Generalized Saddle-Point Problems Eric de Sturler, sturler@cs.uiuc.edu, http://www-faculty.cs.uiuc.edu/~sturler Optimization Technology Center, Northwestern University, October 6, 24
Collaborators Jörg Liesen, TU Berlin Chris Siefert, Mike Parks, UIUC Armand Beaudoin, UIUC: metal deformation Lihua Zhu, UIUC: metal deformation Alla Sheffer, UBC: surface parameterization
Overview Crash course on Krylov subspace methods Saddle-point problems Preconditioners Numerical Example: Surface Parameterization Preconditioning Matrices with Nonzero (2,2) Block Approximating the Schur complement Probing Numerical results Oseen Problem (lid driven cavity) Metal Deformation Conclusions
Krylov Methods Crash Course Consider Ax = b. Given x and r = b Ax, compute optimal m m 1 K A, r span r, Ar,, A r : update from ( ) { } Let min ( ) ( ) b A x + z min r Az m z K Ar, 2 z K Ar, 2 ( ) m K m r Ar A r A r 2 m 1 =, then z = Kmζ, and we must solve the following least squares problem AKmζ r Ar A r A r ζ r 2 m Do this accurately and efficiently every iteration for increasing m. Arnoldi recurrence: H AVm = Vm+ 1H m, where v1 = r / r, V 2 m+ 1Vm+ 1 = Im+ 1, and range ( V ) = range( K ) m+ 1 m+ 1 r AV y = V e r V H y = e r H y m m 2 m+ 1 1 2 m+ 1 m m 2 1 2 m m 2
Krylov Methods Crash Course = b, and relate convergence to polynomials. 2 m 1 xm = x + zm where zm span{ r, Ar, A r,, A r} m r = r Az span{ r, Ar,, A r } Consider Ax m Assume z K min m m A= UΛ U 1 (diagonalizable), then residual at step m 1 = min ( ) min max ( λ) m ( Ar, ) pm( ) = 1 pm( ) = 1λ Λ( A) r Az p A r r U U p For normal matrix this bound is sharp. For highly nonnormal matrix this bound may not be useful. κ ( U ) small: convergence determined by minimal polynomial Clustered eigenvalues yield fast convergence: preconditioning Eigenvalues surrounding origin yields very poor convergence. m
Krylov Methods Crash Course 1. A = 2. 3. diag( 1,2, 3,,1) A = diag( 1, 1,1,2,, 49, 52,,1) A = diag( 99, 97,, 1,1, 3,, 97, 99) log1 r 2-2 -4 GMRES/MINRES -6-8 1 2 3-1 -12-14 2 4 6 8 1 12 #iterations
Generalized Saddle-point Problems We consider systems of the type T A B x f C y = g and T A B x f C D y = g Systems of this type arise in a variety of problems: Constrained optimization problems o FETI (type) methods (Mike Parks) o Surface parameterization Systems of PDEs with continuity constraints o Navier-Stokes o Potential flow in porous media o Polycrystal plasticity metal deformation o Electrostatics / electromagnetics
Preconditioners Various preconditioners have been proposed Block diagonal preconditioner Constraint preconditioner Block upper triangular Ideal version leads to small number of nonzero eigenvalues, and hence fast convergence for Krylov methods In general we use an approximation to these ideal versions (otherwise too expensive) Typically involve Schur complement type matrix
Preconditioners Significant body of work by Elman, Golub, Wathen, Benzi, Silvester, Gould, Nocedal, Hribar, Simoncini, Perugia, BP, Use splitting A= F E (ds&l): 1 T F F E B I S N 1 ( 1 T = CF B ) C M 1 T ( ) 1 M CF B C =, N F B 1 T =, MN I =, ( NM ) 2 NM = U U I U U Oblique projection: ( ) ( ) 1 1 2 1 2 = NM Principal angles ( ωi = cosϕi) between null( NM ) and range( NM ) play important role in eigenvalue bounds.
Preconditioners Preconditioned system: Eigenvalues: λ S I S N x fˆ M y = gˆ 1 + ω 2 max 1 λ 1.5 1 ω max S { 1, 1 5 / 2}, λ ( ± ) Further splitting gives fixed point iteration that depends only on x : ( ) x I NM Sx f = + k+ 1 k ( ) Solves related system I ( I NM) S x = f Eigenvalues ( ) 1 1 λ 1 ω R 2 2 max S
Preconditioners This approach is applicable to general problems; for details and analysis see ds&l 22. Eigenvalue bounds for general problems Regarding related system o Smaller system to solve o Good bounds on eigenvalues, single cluster o Shows bounds always better than for block diag. prec. o Satisfies constraints o Extends constraint preconditioners to general matrices and is more efficient (CPU and memory). CF B which may be expensive to compute and iterate with. 1 T Problem is that (this type of) iteration involves ( ) 1 In that case, we must look for alternatives.,
Invariance Property (also nonzero (2,2)) Aˆ T A B R 2 = RC 1, Preconditioner: F 1 1 T ( RCF B R ) 1 2 1 I S N 1 T M, where N = F B R2 1 MN = R MNR = I,NM = NM 2 2 M = R CF B C 1 1 T and ( ) 1 (oblique) projection. This matrix has exactly the same eigenvalues (and bounds) as the original matrix with block diagonal preconditioner. Also leads to the same fixed point iteration/related system I I NM S x = I I NM S x = f ( ( ) ) ( ) ( ) 2
Surface Parameterization
Example: Texture Mapping
Angle-based Flattening Solve mesh flattening as constrained optimization problem. Minimize relative deformation of angles, ( ) 2 2 αij, φij, / φij,, Subject to constraints on valid 2D mesh 1. Angles between and π (orientation), hardly ever a problem (dealt with algorithmically) 2. Angles in triangle sum to π, 3. Angles around interior node sum to 2π, 4. Triangles at an interior node need to agree on edge lengths: nonlinear constraint. ( ) 2 αij, φij, φij, subject to T ( ) ( ) ( ) g2 α g3 α g4 α = ij, min / T Critical point of Lagrangian L( αλ, ) = F( α) + λ g( α) ij,
Nonlinear System Critical point of : L(, ) = F( ) + T g( ), L(, ) = Newton iteration: 2, L(, ) =, L(, ) Jacobian: 2 [F( ) + T g( )] [ g( )] T g( ) = A + G k B T T C k B C k (symmetric and indefinite) A = diag 2w i j and depends only on G k g ( 4) ( ) B depends only on g ( 2) ( ) and hence is constant (zero,one) C k depends on g ( 3) and g ( 4) and is partially constant ( g ( 3) )
Preconditioning Consider the following choice of blocks F Ek C Ck T k with F T A B = B and E k G k =. Explicit inverse of F known (very good splitting). I Sk Nk x f Mk y = g I I N M S x = f ( ) and ( ) k k k λ S 1 + ω 2 max 1 λ 1.5 1 ω max S and ( ) 1 1 λ 1 ω R 2 2 max S
Scaling Constraints Scale constraints ( ) 3 g α and g ( ) 4 α by ε and precondition: I εsk εnk x f ε Mk y = ε g Block diagonal preconditioner: 1 1 Related system (after splitting): ε( ) I I N M S x = f ( ) k k k Eigenvalue bounds for the scaled system: Block diagonal preconditioner: Related ( ) 1 1 λ ε 1 ω R 2 2 max 1 + ω 2 max 1 λs λ 1.5ε 1 ω S max S Very fast convergence even for modest ε, (but Newton )
Convergence Three Balls 1 1-2 1-4 GMRES convergence (no scaling) no prec right prec. B(S) (I-F) 1-6 1-8 1-1 2 4 6 8 1
Convergence Three Balls Computed r n / r 1 1-2 1-4 1-6 1-8 =.1 1-1 GMRES convergence with scaling = 1 =.1 = 1 1 2 3 4 5 no prec right prec. B(S) (I-F)
Eigenvalues of B(S) = 1-1.5-1 -.5.5 1 1.5 2 2.5 3 3.5 =.1-1.5-1 -.5.5 1 1.5 2 2.5 3 3.5 =.1-1.5-1 -.5.5 1 1.5 2 2.5 3 3.5
Eigenvalues of Related System = 1.7.8.9 1 1.1 1.2 1.3 1.4 =.1.7.8.9 1 1.1 1.2 1.3 1.4 =.1.7.8.9 1 1.1 1.2 1.3 1.4
Preconditioners nonzero (2,2) First, we consider saddle-point problems with nonzero (2,2) block, such as stabilized finite element discretizations. Block-diagonal preconditioner from splitting (1,1) block T 1 F E B F 1 T 1 C D ( D CF B ) I S N x fˆ M Q = gˆ, Preconditioned system: ( y) where MN = I + Q and ( NM ) 2 = NM + NQM Further splitting leads to fixed point iteration and a smaller ( ) related system: I ( I NM) S x = f
Preconditioners nonzero (2,2) I S N Eigenvalues of B( S) = M Q? I N B () = 1 1 M Q 2 4 δ is an eigenvalue of Q and typically small. where j : λ 1, ( 1 + δ ) 1 ( 1 ) 2 j ± + + δj Let null( NM ) U1 and other eigenvectors NM U 2 = U2Θ Eigenvalues of B( S ) from perturbation of B ( ) : 1 + ω 2 max 1 λs λ 2γκ Θ 1 ω max S (typically γκ Θ modest) : ( )( ) 1 ( I ( I NM) S) 1 λ 1+ ω δ + ω δκ 1 ω R 2 2 1 1 Θ max S
Approximating the Schur complement Replace Schur complement type matrix by approximation: F 1 1 SF by F 1 1 SF, where 1 S F is approximated directly Preconditioned system I S N M2 or I S N M2 Q 2 where N F B 1 T =, 2 ( ) ( E) M 1 = S C, MN= S S I+ E 1 2 F NM = N I + M = NM + NEM 2 F F 2 2 2 2, and Bounds: I S N I N S M2 Q = 2 M Q EM EQ
Approximating the Schur complement B( S, E) I S N I N S = M2 Q = 2 M Q EM EQ Eigenvalues of B( S,E) from perturbation of (, ) B : 1 2 1 + ω max λs, E λ γκ Θ S + γκ ˆ V E 1 ω max (typically γκ Θ and γκ ˆ V modest) When D = (and hence QQ, 2 = ) λ S, E 1 1 ω 2 max 2 2 + S 1 ω max 5 λ + E
Approximating the Schur complement Splitting as before leads to exactly the same related system. New splitting: I S N I N S M2 Q = 2 M2 M2N I E Linear system: I ( I NM ) E ( x ) 2 S N f MS 2 I E y = + g Eigenvalues cluster around 1. 2 2 2 ( E ) 1 λ R 1+ N 1+ M max S, We lose advantages of smaller system and satisfying the constraints.
Probing for the Schur Complement 1 T Approximate ( D CF B ) 1 1 T without having ( D CF B ) Probing (Chan&Mathew 92): reconstruct matrix with small bandwidth from products with selected vectors and invert. Based on rapid decay from diagonal in matrix (1D interface). 1 We use decay of F T on mesh with structure of CB,, D to select pattern of large coefficients in matrix to probe for Construct set of vectors to minimize lumping other coeff.s Cheap to use multiple orderings and combine inverses of matrices with small bandwidth (convergence not great) Better to approximate matrix structure for multidimensional problem directly and use incomplete decomposition
Oseen problem, driven cavity Software by Elman, Silvester, Wathen, Ramage System (A nonsymmetric): T A B x f B βc y = g viscosity ν=1/1 and stabilization β=1/4
Basic Splittings of (1,1) Block.6.4 Unpreconditioned Diagonal Prec. Tridiagonal Prec..2 Imag..2.4.6.8.5.5 1 1.5 2 Real
Oseen problem Not constrained to structured splittings A=F-E Multigrid cycle is a (complicated) splitting Use number of V-cycles for F -1. Useful because we can study convergence and eigenvalue (bounds) for sequence of problems with decreasing S. Likewise, we can analyze the effect of approximations to the Schur complement (not as easy)
1 V-cycle, Related System: Eigenvalues.1.5.5.1.15.2.7.75.8.85.9.95 1 1.5
2 V-cycles, Related System: Eigenvalues.1.5.5.1.15.2.7.75.8.85.9.95 1 1.5
4 V-cycles, Related System: Eigenvalues.1.5.5.1.15.2.7.75.8.85.9.95 1 1.5
6 V-cycles, Related System: Eigenvalues.1.5.5.1.15.2.7.75.8.85.9.95 1 1.5
Exact Schur Complement Convergence for block-diagonal preconditioner for various numbers of V-cycles for the splitting 1 1 2 1 V Cycle 2 V Cycles 3 V Cycles 4 V Cycles 5 V Cycles 6 V Cycles 1 4 r k / r 1 6 1 8 1 1 5 1 15 2 25 3 35 4 45 Iteration
Inexact Schur Complement - ILU(1.e-4) Convergence for block-diagonal preconditioner for various numbers of V-cycles for the splitting 1 1 2 1 V Cycle 2 V Cycles 3 V Cycles 4 V Cycles 5 V Cycles 6 V Cycles 1 4 r k / r 1 6 1 8 1 1 5 1 15 2 25 3 35 4 45 5 Iteration
Exact Schur Complement Convergence for related system using various numbers of V-cycles for the splitting 1 1 2 1 V Cycle 2 V Cycles 3 V Cycles 4 V Cycles 5 V Cycles 6 V Cycles 1 4 r k / r 1 6 1 8 1 1 2 4 6 8 1 12 14 Iteration
Inexact Schur Complement - ILU(1.e-4) Convergence for related system using various numbers of V-cycles for the splitting 1 1 2 1 V Cycle 2 V Cycles 3 V Cycles 4 V Cycles 5 V Cycles 6 V Cycles 1 4 r k / r 1 6 1 8 1 1 2 4 6 8 1 12 14 Iteration
Exact Schur Complement Eigenvalue perturbations and bounds for block-diagonal preconditioner using various numbers of V-Cycles for splitting
Exact Schur Complement Eigenvalue perturbations and bounds for related system using various numbers of V-Cycles for splitting without κ ( Θ)
Inexact Schur Complement ILU Convergence for block-diagonal preconditioner using various accuracies for Schur Complement ILU 1 1 2 ILU(1e 3) ILU(1e 4) ILU(1e 5) ILU(1e 6) 1 4 r k / r 1 6 1 8 1 1 5 1 15 2 25 3 Iteration
Inexact Schur Complement - ILU Convergence for related system using various accuracies for Schur Complement ILU 1 1 2 ILU(1e 3) ILU(1e 4) ILU(1e 5) ILU(1e 6) 1 4 r k / r 1 6 1 8 1 1 1 2 3 4 5 6 7 Iteration