Principle Component Analysis and Model Reduction for Dynamical Systems

Transcription

1 Principle Component Analysis and Model Reduction for Dynamical Systems D.C. Sorensen Virginia Tech 12 Nov 2004

2 Protein Substate Modeling and Identification PCA Dimension Reduction Using the SVD Tod Romo George Phillips T.D. Romo, J.B. Clarage, D.C. Sorensen and G.N. Phillips, Jr., Automatic Identification of Discrete Substates in Proteins: Singular Value Decomposition Analysis of Time Averaged Crystallographic Refinements, Proteins: Structure, Function, and Genetics22, ,(1995).

3 Outline Brief Intro to Model Reduction for Dynamical Systems Reduced Basis Trajectory Time Integration for MD The Symmetric SVD: Reduced Dimension MD Simulation D.C. Sorensen 3

4 LTI Systems and Model Reduction Time Domain ẋ = Ax + Bu y = Cx A R n n, B R n m, C R p n, n >> m, p Frequency Domain sx = Ax + Bu y = Cx Transfer Function H(s) C(sI A) 1 B, y(s) = H(s)u(s) D.C. Sorensen 4

5 Model Reduction Construct a new system {Â, ˆB, Ĉ} with LOW dimension k << n ˆx = ˆx + ˆBu ŷ = Ĉˆx Goal: Preserve system response ŷ should approximate y Projection: x(t) = V ˆ x(t) and V ˆx = AVˆx + Bu D.C. Sorensen 5

6 Model Reduction by (Krylov) Projection Approximate x S V = Range(V) k-diml. subspace i.e. Put x = Vˆx, and then force W T [V ˆx (AVˆx + Bu)] = 0 ŷ = CVˆx If W T V = I k, then the k dimensional reduced model is ˆx = ˆx + ˆBu ŷ = Ĉˆx where  = W T AV, ˆB = W T B, Ĉ = CV. D.C. Sorensen 6

7 Moment Matching Krylov Subspace Projection Padé via Lanczos (PVL) Freund, Feldmann Bai Multipoint Rational Interpolation Grimme Gallivan, Grimme, Van Dooren Gugercin, Antoulas, Beattie D.C. Sorensen 7

8 Gramian Based Model Reduction Proper Orthogonal Decomposition (POD) Principle Component Analysis (PCA) The gramian ẋ(t) = f(x(t), u(t)), y = g(x(t), u(t)) Eigenvectors of P Orthogonal Basis P = o x(τ)x(τ) T dτ P = VS 2 V T x(t) = VSw(t) D.C. Sorensen 8

9 PCA or POD Reduced Basis Low Rank Approximation x V kˆx k (t) Galerkin condition Global Basis ˆx k = Vk T f(v kˆx k (t), u(t)) Global Approximation Error (H 2 bound for LTI) x V kˆx k 2 σ k+1 Snapshot Approximation to P P 1 m x(t j )x(t j ) T m j=1 D.C. Sorensen 9

10 SVD of Snapshot Trajectory (Conformations) SVD of X: X = [x(t 1 ), x(t 2 ),..., x(t m )] where X = VSW T V k S k W T k V T V = W T W = I n S = diag(σ 1, σ 2,, σ n ) with σ 1 σ 2 σ n. D.C. Sorensen 10

11 SVD Compression n 100 SVD of Clown m k ( m + n) v.s. m x n Storage k 20 Advantage of SVD Compression D.C. Sorensen 11

12 Image Compression - Feature Detection original rank = 10 rank = 30 rank = 50

13 POD in CFD Extensive Literature Karhunen-Loéve, L. Sirovich Burns, King Kunisch and Volkwein Many, many others Incorporating Observations Balancing Lall, Marsden and Glavaski K. Willcox and J. Peraire D.C. Sorensen 13

14 POD for LTI systems Impulse Response: H(t) = C(tI A) 1 B, t 0 Input to State Map: Controllability Gramian: x(t) = e At B P = o x(τ)x(τ) T dτ = o e Aτ BB T e AT τ dτ State to Output Map: Observability Gramian: y(t) = Ce At x(0) Q = o e AT τ C T Ce Aτ dτ D.C. Sorensen 14

15 Balanced Reduction (Moore 81) Lyapunov Equations for system Gramians AP + PA T + BB T = 0 A T Q + QA + C T C = 0 With P = Q = S : Want Gramians Diagonal and Equal States Difficult to Reach are also Difficult to Observe Reduced Model A k = W T k AV k, B k = W T k B, C k = C k V k PVk = W k S k QW k = V k S k Reduced Model Gramians Pk = S k and Q k = S k. D.C. Sorensen 15

16 Hankel Norm Error estimate (Glover 84) Why Balanced Realization? Hankel singular values = λ(pq) Model reduction H error (Glover) y ŷ 2 2 (sum neglected singular values) u 2 Extends to MIMO Preserves Stability Key Challenge Approximately solve large scale Lyapunov Equations in Low Rank Factored Form D.C. Sorensen 16

17 CD Player Impluse Response n = 120 k = 11, tol = 5e-3 Sigma Plot: n = 120 CDplayer 50 Singular Values (db) Frequency (rad/sec) Error plot 20 Singular Values (db) Frequency (rad/sec)

18 CD Player Impluse Response k = 17, tol = 5e-4 Sigma Plot: n = 120 CDplayer 50 Singular Values (db) Frequency (rad/sec) Error plot 30 Singular Values (db) Frequency (rad/sec)

19 CD Player Impluse Response k = 31, tol = 5e-5 Sigma Plot: n = 120 CDplayer 50 Singular Values (db) Frequency (rad/sec) Error plot 40 Singular Values (db) Frequency (rad/sec)

20 CD Player - Hankel Singular Values 10 2 Hankel Singular Values

21 Reduction of Second Order Systems Mẍ + Gẋ + Kx = Bu y(t) = Cx(t) ˆM ˆx + Ĝ ˆx + ˆKˆx = ˆBu ŷ(t) = Ĉˆx(t) where ˆM = V T MV, etc. with V T V = I. Key Point: Preserve Second Order Form DO NOT convert to First Order Sys. Keeps Physical Meaning - can be built D.C. Sorensen 21

22 Applications Mechanical Systems Electrical Systems MEMS devices e.g. Building Model N = 26394, k = 200 (ROM) Frequency Response α =.67, β = reduced original Response G( jω) Frequency ω D.C. Sorensen 22

23 Error Bound for Second Order Systems A. Antoulas, C. Teng Controllability Gramian - Impulse Response P := 0 x(t)x(t) dt. Reduce with Dominant Eigenspace P: PV 1 = V 1 S 1 Bounded H 2 norm of error system E = Σ ˆΣ E 2 H 2 C o tr{s 2 } Key: Expression for P in frequency domain. D.C. Sorensen 23

24 PCA Model Reduction for Molecular Dynamics Rachel Vincent Monte Pettitt D.C. Sorensen 24

25 Classical Equations of Motion Molecular dynamics (MD) simulation is a computational tool used to study a molecular system as it evolves through time. Newton s second law of motion governs atomic motion in MD: M r(t) = V(r(t)). r(t) = vector of atomic coordinates at time t = [x 1t y 1t z 1t x Nat y Nat z Nat] T M = diagonal matrix of atomic masses V(r(t)) = potential energy function D.C. Sorensen 25

26 Time Step Barrier fs Example: DHFR (dihydrofolate reductase), 23,558 atoms To realize a microsecond simulation with a time step of 2 fs would require about 13 months of simulation time when utilizing 126 processors. Time with respect to simulation using the NAMD program (Not Another Molecular Dynamics program) on an Origin 2000 R10000/250. D.C. Sorensen 26

27 PCA Reduced Basis Simulation MVÿ(t) = V(Vy(t)) ˆMÿ(t) = V T V(Vy(t)) 1. Initial Basis V: truncated SVD of short traditional MD trajectory using ARPACK. 2. Approximate the reduced basis potential energy with Radial Basis Fit. 3. Update reduced basis positions y and velocities ẏ in k dimensions using the approximate potential. 4. Reconstruct 3ND trajectory r = Vy. 5. Update and truncate reduced basis and perform full space correction as needed. D.C. Sorensen 27

28 Remarks Butane (n = 42): 80% to 90% of the total motion with LSVs (24% - 36% DOF) Reduced Simulation times order of seconds Traditional MD simulation took several minutes. BPTI(n = 2700): 80% to 90% of the total motion with LSVs (11% - 19% DOF) Reduced Simulation times order of minutes Traditional MD simulation took several hours. D.C. Sorensen 28

29 Symmetry Preserving SVD (Mili Shah) Collaboration with the Physical and Biological Computing Group Lydia Kavraki Mark Moll David Schwarz Amarda Shehua Allison Heath D.C. Sorensen 29

30 Symmetry in HIV-1 protease Backbone representation of HIV-1 protease (from M. Moll) bound to an inhibitor (shown in orange) Uses PCA dimension reduction of Molecular Dynamics Simulations Symmetry across a plane should be present D.C. Sorensen 30

31 Animation: Symmetric SVD Approximation Comparison of Regular and Symmetric Preserving SVD 1.5 Comparison of Regular and Symmetric Preserving SVD click below figures for movies D.C. Sorensen 31

32 Finding the Plane of Symmetry Suppose X = [x 1, x 2,..., x n ] and Y = [y 1, y 2,..., y n ] are two sets of points symmetric across a plane Exact symmetry condition: Y = (I 2ww T )X, where w is the normal to the (hyper-) Plane of Symmetry H = {x : w T x = 0} Remark: In Numerical Linear Algebra (I 2ww T ) is a Householder Transformation or Elementary Reflector D.C. Sorensen 32

33 Best Approximate Plane of Symmetry Symmetry condition with Noise: Y = (I 2w o w T o )X + E, Problem: Compute a unit vector w that gives the best Approximate Plane of Symmetry Solution: min w Y (I 2wwT )X F, (XY T + YX T )v = vλ min, w = v gives the normal w to the best approximate plane of symmetry D.C. Sorensen 33

34 Supressing Outlier Effects Iteratively determine diagonal weighting matrix D w The i-th diagonal of D w is 1/discrepancy, discrepancy = y i (I 2ww T )x i Problem: Compute a unit vector w that gives the best Weighted Approximate Plane of Symmetry Solution: min v [Y (I 2vv T )X]D w F, (XD 2 w Y T + YD 2 w X T )v = vλ min, w v gives the normal w to the best weighted approximate plane of symmetry D.C. Sorensen 34

35 Finding Normal to Best Plane of Symmetry click below for movie D.C. Sorensen 35

36 The Symmetric SVD Approximation If WX 2 = X 1 + E where W = blockdiag(i 2ww T ) min WˆX 2 =ˆX 1 Solved by: with ( X1 X 2 U = 1 2 ( U1 U 2 ) ( ) ˆX 1 2 ˆX 2 F and ( ˆX 1 ˆX 2 ) = USV T ), S = 2S 1, V = V 1. and U 2 = WU 1, U 1 S 1 V T 1 = 1 2 (X 1 + WX 2 ) D.C. Sorensen 36

37 Symmetric Major Modes: HIV-1 protease Major mode regular SVD is red Major mode SYMMETRIC SVD is blue 3120 atoms (3*3120=9360 degrees of freedom) MD trajectory consisted of conformations (NAMD) SVD and SymSVD used P ARPACK on a Linux cluster dual-processor nodes; 1600MHz AMD Athlon processors, 1GB RAM per node. 1GB/s Ethernet connection. 12 Processors = 6 nodes. First 10 standard singular vectors: 88 secs. First 10 symmetric singular vectors: 131 secs. D.C. Sorensen 37

38 Animation: Symmetric SVD on HIV1 Protease click for movie Red = Unsymmetric Blue = Symmetric First SVD mode Symmetric vs. Unsymmetric D.C. Sorensen 38

39 Rotational Symmetry 1.5 Perfect Rotational Symmetry No Noise X j = WX j 1, j = 1 : k 1, where W = I QGQ T I p G is a rotation X k = X 0. D.C. Sorensen 39

40 Finding the Axis of Rotation q is an axis of rotation iff Q T q = 0 q T W = q T (I QGQ T ) = q T q T X 0 = q T X j Let M = (k 1)X 0 k 1 j=1 X j min q =1 MT q (= 0 if exact symmetry holds) Good for noisy data (for another condition see Minovic,Ishikawa and Kato ) D.C. Sorensen 40

41 Best Rotationally Symmetric Approximation If W k j X j = X 0 + E j, j = 1 : k 1 min bx j+1 =W b X j X 0 : X k 1 X 0 : X k 1 2 F = 1 k 1 E j 2 F k, j=0 [ XT 0... X ] T T k 1 = USV T with U = 1 [U T 0... U T k 1 k ] T U 0 S 0 V T 0 = 1 k (X 0 + W k 1 X 1 + W k 2 X WX k 1 ). S = ks 0 V = V 0 U j = W j U 0, j = 0, 1, 2,..., k 1 D.C. Sorensen 41

42 Animation: Rotationally Symmetric SVD Approximation Comparison of Symmetric and Regular SVD 1.5 Comparison of Symmetric and Regular SVD click below figures for movies D.C. Sorensen 42

43 Animation: Rotationally Symmetric SVD on HIV1 click for movie Red = Unsymmetric Blue = Symmetric Second SVD mode Rotationally Symmetric vs. Unsymmetric D.C. Sorensen 43

44 Potential for Symmetric SVD Obtain a Symmetric PCA reduced dimension approximate trajectory Test Hypothesis of Symmetry in an Unknown Protein Locate Symmeteric Sub-Structures Things to Do: Improve convergence rate for finding w Give a complete analysis of convergence Give a complete analysis of discrepancy weighting Extend to more complex symmetries Find New Applications D.C. Sorensen 44

45 Contact Info. web page: sorensen/ ARPACK: