Principle Component Analysis and Model Reduction for Dynamical Systems D.C. Sorensen Virginia Tech 12 Nov 2004
Protein Substate Modeling and Identification PCA Dimension Reduction Using the SVD Tod Romo George Phillips T.D. Romo, J.B. Clarage, D.C. Sorensen and G.N. Phillips, Jr., Automatic Identification of Discrete Substates in Proteins: Singular Value Decomposition Analysis of Time Averaged Crystallographic Refinements, Proteins: Structure, Function, and Genetics22,311-321,(1995).
Outline Brief Intro to Model Reduction for Dynamical Systems Reduced Basis Trajectory Time Integration for MD The Symmetric SVD: Reduced Dimension MD Simulation D.C. Sorensen 3
LTI Systems and Model Reduction Time Domain ẋ = Ax + Bu y = Cx A R n n, B R n m, C R p n, n >> m, p Frequency Domain sx = Ax + Bu y = Cx Transfer Function H(s) C(sI A) 1 B, y(s) = H(s)u(s) D.C. Sorensen 4
Model Reduction Construct a new system {Â, ˆB, Ĉ} with LOW dimension k << n ˆx = ˆx + ˆBu ŷ = Ĉˆx Goal: Preserve system response ŷ should approximate y Projection: x(t) = V ˆ x(t) and V ˆx = AVˆx + Bu D.C. Sorensen 5
Model Reduction by (Krylov) Projection Approximate x S V = Range(V) k-diml. subspace i.e. Put x = Vˆx, and then force W T [V ˆx (AVˆx + Bu)] = 0 ŷ = CVˆx If W T V = I k, then the k dimensional reduced model is ˆx = ˆx + ˆBu ŷ = Ĉˆx where  = W T AV, ˆB = W T B, Ĉ = CV. D.C. Sorensen 6
Moment Matching Krylov Subspace Projection Padé via Lanczos (PVL) Freund, Feldmann Bai Multipoint Rational Interpolation Grimme Gallivan, Grimme, Van Dooren Gugercin, Antoulas, Beattie D.C. Sorensen 7
Gramian Based Model Reduction Proper Orthogonal Decomposition (POD) Principle Component Analysis (PCA) The gramian ẋ(t) = f(x(t), u(t)), y = g(x(t), u(t)) Eigenvectors of P Orthogonal Basis P = o x(τ)x(τ) T dτ P = VS 2 V T x(t) = VSw(t) D.C. Sorensen 8
PCA or POD Reduced Basis Low Rank Approximation x V kˆx k (t) Galerkin condition Global Basis ˆx k = Vk T f(v kˆx k (t), u(t)) Global Approximation Error (H 2 bound for LTI) x V kˆx k 2 σ k+1 Snapshot Approximation to P P 1 m x(t j )x(t j ) T m j=1 D.C. Sorensen 9
SVD of Snapshot Trajectory (Conformations) SVD of X: X = [x(t 1 ), x(t 2 ),..., x(t m )] where X = VSW T V k S k W T k V T V = W T W = I n S = diag(σ 1, σ 2,, σ n ) with σ 1 σ 2 σ n. D.C. Sorensen 10
SVD Compression n 100 SVD of Clown m k ( m + n) v.s. m x n Storage 90 80 70 60 50 40 30 k 20 Advantage of SVD Compression 10 0 0 20 40 60 80 100 120 140 160 180 200 D.C. Sorensen 11
Image Compression - Feature Detection original rank = 10 rank = 30 rank = 50
POD in CFD Extensive Literature Karhunen-Loéve, L. Sirovich Burns, King Kunisch and Volkwein Many, many others Incorporating Observations Balancing Lall, Marsden and Glavaski K. Willcox and J. Peraire D.C. Sorensen 13
POD for LTI systems Impulse Response: H(t) = C(tI A) 1 B, t 0 Input to State Map: Controllability Gramian: x(t) = e At B P = o x(τ)x(τ) T dτ = o e Aτ BB T e AT τ dτ State to Output Map: Observability Gramian: y(t) = Ce At x(0) Q = o e AT τ C T Ce Aτ dτ D.C. Sorensen 14
Balanced Reduction (Moore 81) Lyapunov Equations for system Gramians AP + PA T + BB T = 0 A T Q + QA + C T C = 0 With P = Q = S : Want Gramians Diagonal and Equal States Difficult to Reach are also Difficult to Observe Reduced Model A k = W T k AV k, B k = W T k B, C k = C k V k PVk = W k S k QW k = V k S k Reduced Model Gramians Pk = S k and Q k = S k. D.C. Sorensen 15
Hankel Norm Error estimate (Glover 84) Why Balanced Realization? Hankel singular values = λ(pq) Model reduction H error (Glover) y ŷ 2 2 (sum neglected singular values) u 2 Extends to MIMO Preserves Stability Key Challenge Approximately solve large scale Lyapunov Equations in Low Rank Factored Form D.C. Sorensen 16
CD Player Impluse Response n = 120 k = 11, tol = 5e-3 Sigma Plot: n = 120 CDplayer 50 Singular Values (db) 0 50 100 150 10 0 10 1 10 2 10 3 10 4 10 5 Frequency (rad/sec) Error plot 20 Singular Values (db) 30 40 50 60 70 80 90 10 0 10 1 10 2 10 3 10 4 10 5 Frequency (rad/sec)
CD Player Impluse Response k = 17, tol = 5e-4 Sigma Plot: n = 120 CDplayer 50 Singular Values (db) 0 50 100 150 10 0 10 1 10 2 10 3 10 4 10 5 Frequency (rad/sec) Error plot 30 Singular Values (db) 40 50 60 70 80 10 0 10 1 10 2 10 3 10 4 10 5 Frequency (rad/sec)
CD Player Impluse Response k = 31, tol = 5e-5 Sigma Plot: n = 120 CDplayer 50 Singular Values (db) 0 50 100 150 10 0 10 1 10 2 10 3 10 4 10 5 Frequency (rad/sec) Error plot 40 Singular Values (db) 50 60 70 80 90 100 10 0 10 1 10 2 10 3 10 4 10 5 Frequency (rad/sec)
CD Player - Hankel Singular Values 10 2 Hankel Singular Values 10 0 10 2 10 4 10 6 10 8 10 10 10 12 10 14 0 20 40 60 80 100 120
Reduction of Second Order Systems Mẍ + Gẋ + Kx = Bu y(t) = Cx(t) ˆM ˆx + Ĝ ˆx + ˆKˆx = ˆBu ŷ(t) = Ĉˆx(t) where ˆM = V T MV, etc. with V T V = I. Key Point: Preserve Second Order Form DO NOT convert to First Order Sys. Keeps Physical Meaning - can be built D.C. Sorensen 21
Applications Mechanical Systems Electrical Systems MEMS devices e.g. Building Model N = 26394, k = 200 (ROM) Frequency Response α =.67, β =.0032 5 reduced original 4.5 4 3.5 Response G( jω) 3 2.5 2 1.5 1 0.5 0 0 5 10 15 20 25 30 Frequency ω D.C. Sorensen 22
Error Bound for Second Order Systems A. Antoulas, C. Teng Controllability Gramian - Impulse Response P := 0 x(t)x(t) dt. Reduce with Dominant Eigenspace P: PV 1 = V 1 S 1 Bounded H 2 norm of error system E = Σ ˆΣ E 2 H 2 C o tr{s 2 } Key: Expression for P in frequency domain. D.C. Sorensen 23
PCA Model Reduction for Molecular Dynamics Rachel Vincent Monte Pettitt D.C. Sorensen 24
Classical Equations of Motion Molecular dynamics (MD) simulation is a computational tool used to study a molecular system as it evolves through time. Newton s second law of motion governs atomic motion in MD: M r(t) = V(r(t)). r(t) = vector of atomic coordinates at time t = [x 1t y 1t z 1t x Nat y Nat z Nat] T M = diagonal matrix of atomic masses V(r(t)) = potential energy function D.C. Sorensen 25
Time Step Barrier fs Example: DHFR (dihydrofolate reductase), 23,558 atoms To realize a microsecond simulation with a time step of 2 fs would require about 13 months of simulation time when utilizing 126 processors. Time with respect to simulation using the NAMD program (Not Another Molecular Dynamics program) on an Origin 2000 R10000/250. D.C. Sorensen 26
PCA Reduced Basis Simulation MVÿ(t) = V(Vy(t)) ˆMÿ(t) = V T V(Vy(t)) 1. Initial Basis V: truncated SVD of short traditional MD trajectory using ARPACK. 2. Approximate the reduced basis potential energy with Radial Basis Fit. 3. Update reduced basis positions y and velocities ẏ in k dimensions using the approximate potential. 4. Reconstruct 3ND trajectory r = Vy. 5. Update and truncate reduced basis and perform full space correction as needed. D.C. Sorensen 27
Remarks Butane (n = 42): 80% to 90% of the total motion with 10-15 LSVs (24% - 36% DOF) Reduced Simulation times order of seconds Traditional MD simulation took several minutes. BPTI(n = 2700): 80% to 90% of the total motion with 300-500 LSVs (11% - 19% DOF) Reduced Simulation times order of minutes Traditional MD simulation took several hours. D.C. Sorensen 28
Symmetry Preserving SVD (Mili Shah) Collaboration with the Physical and Biological Computing Group Lydia Kavraki Mark Moll David Schwarz Amarda Shehua Allison Heath D.C. Sorensen 29
Symmetry in HIV-1 protease Backbone representation of HIV-1 protease (from M. Moll) bound to an inhibitor (shown in orange) Uses PCA dimension reduction of Molecular Dynamics Simulations Symmetry across a plane should be present D.C. Sorensen 30
Animation: Symmetric SVD Approximation Comparison of Regular and Symmetric Preserving SVD 1.5 Comparison of Regular and Symmetric Preserving SVD 1.5 1 1 0.5 0.5 0 0 0.5 0.5 1 1 1.5 1.5 1 0.5 0 0.5 1 1.5 1.5 1.5 1 0.5 0 0.5 1 1.5 click below figures for movies D.C. Sorensen 31
Finding the Plane of Symmetry Suppose X = [x 1, x 2,..., x n ] and Y = [y 1, y 2,..., y n ] are two sets of points symmetric across a plane Exact symmetry condition: Y = (I 2ww T )X, where w is the normal to the (hyper-) Plane of Symmetry H = {x : w T x = 0} Remark: In Numerical Linear Algebra (I 2ww T ) is a Householder Transformation or Elementary Reflector D.C. Sorensen 32
Best Approximate Plane of Symmetry Symmetry condition with Noise: Y = (I 2w o w T o )X + E, Problem: Compute a unit vector w that gives the best Approximate Plane of Symmetry Solution: min w Y (I 2wwT )X F, (XY T + YX T )v = vλ min, w = v gives the normal w to the best approximate plane of symmetry D.C. Sorensen 33
Supressing Outlier Effects Iteratively determine diagonal weighting matrix D w The i-th diagonal of D w is 1/discrepancy, discrepancy = y i (I 2ww T )x i Problem: Compute a unit vector w that gives the best Weighted Approximate Plane of Symmetry Solution: min v [Y (I 2vv T )X]D w F, (XD 2 w Y T + YD 2 w X T )v = vλ min, w v gives the normal w to the best weighted approximate plane of symmetry D.C. Sorensen 34
Finding Normal to Best Plane of Symmetry 1 0.5 0 0.5 1 1.5 1 0.5 0 0.5 1 click below for movie D.C. Sorensen 35
The Symmetric SVD Approximation If WX 2 = X 1 + E where W = blockdiag(i 2ww T ) min WˆX 2 =ˆX 1 Solved by: with ( X1 X 2 U = 1 2 ( U1 U 2 ) ( ) ˆX 1 2 ˆX 2 F and ( ˆX 1 ˆX 2 ) = USV T ), S = 2S 1, V = V 1. and U 2 = WU 1, U 1 S 1 V T 1 = 1 2 (X 1 + WX 2 ) D.C. Sorensen 36
Symmetric Major Modes: HIV-1 protease Major mode regular SVD is red Major mode SYMMETRIC SVD is blue 3120 atoms (3*3120=9360 degrees of freedom) MD trajectory consisted of 10000 conformations (NAMD) SVD and SymSVD used P ARPACK on a Linux cluster dual-processor nodes; 1600MHz AMD Athlon processors, 1GB RAM per node. 1GB/s Ethernet connection. 12 Processors = 6 nodes. First 10 standard singular vectors: 88 secs. First 10 symmetric singular vectors: 131 secs. D.C. Sorensen 37
Animation: Symmetric SVD on HIV1 Protease click for movie Red = Unsymmetric Blue = Symmetric First SVD mode Symmetric vs. Unsymmetric D.C. Sorensen 38
Rotational Symmetry 1.5 Perfect Rotational Symmetry No Noise 1 0.5 0 0.5 1 1.5 1.5 1 0.5 0 0.5 1 1.5 X j = WX j 1, j = 1 : k 1, where W = I QGQ T I p G is a rotation X k = X 0. D.C. Sorensen 39
Finding the Axis of Rotation q is an axis of rotation iff Q T q = 0 q T W = q T (I QGQ T ) = q T q T X 0 = q T X j Let M = (k 1)X 0 k 1 j=1 X j min q =1 MT q (= 0 if exact symmetry holds) Good for noisy data (for another condition see Minovic,Ishikawa and Kato ) D.C. Sorensen 40
Best Rotationally Symmetric Approximation If W k j X j = X 0 + E j, j = 1 : k 1 min bx j+1 =W b X j X 0 : X k 1 X 0 : X k 1 2 F = 1 k 1 E j 2 F k, j=0 [ XT 0... X ] T T k 1 = USV T with U = 1 [U T 0... U T k 1 k ] T U 0 S 0 V T 0 = 1 k (X 0 + W k 1 X 1 + W k 2 X 2 +... + WX k 1 ). S = ks 0 V = V 0 U j = W j U 0, j = 0, 1, 2,..., k 1 D.C. Sorensen 41
Animation: Rotationally Symmetric SVD Approximation Comparison of Symmetric and Regular SVD 1.5 Comparison of Symmetric and Regular SVD 1.5 1 1 0.5 0.5 0 0 0.5 0.5 1 1 1.5 1.5 1 0.5 0 0.5 1 1.5 1.5 1.5 1 0.5 0 0.5 1 1.5 click below figures for movies D.C. Sorensen 42
Animation: Rotationally Symmetric SVD on HIV1 click for movie Red = Unsymmetric Blue = Symmetric Second SVD mode Rotationally Symmetric vs. Unsymmetric D.C. Sorensen 43
Potential for Symmetric SVD Obtain a Symmetric PCA reduced dimension approximate trajectory Test Hypothesis of Symmetry in an Unknown Protein Locate Symmeteric Sub-Structures Things to Do: Improve convergence rate for finding w Give a complete analysis of convergence Give a complete analysis of discrepancy weighting Extend to more complex symmetries Find New Applications D.C. Sorensen 44
Contact Info. e-mail: sorensen@rice.edu web page: www.caam.rice.edu/ sorensen/ ARPACK: www.caam.rice.edu/software/arpack/