Ger sgorin Circle Chaper 9 Approimaing Eigenvalues Per-Olof Persson persson@berkeley.edu Deparmen of Mahemaics Universiy of California, Berkeley Mah 128B Numerical Analysis (Ger sgorin Circle) Le A be an n n mari, C he comple plane, and R i = z C n z a ii a ij Then j=1,j i 1 The eigenvalues of A are inside R = n i=1 R i 2 Union of any k circles no inersecing he oher (n k) conains precisely k of he eigenvalues (couning mulipliciies) Linear Algebra The se {v (1), v (2),..., v (k) } is linearly independen if whenever 0 = α 1 v (1) + α 2 v (2) + + α k v (k), hen α i = 0 for i = 1,..., k. Oherwise i is linearly dependen. For n linearly independen vecors {v (1), v (2),..., v (n) } in R n and any vecor R n, unique consans β 1, β 2,..., β n eiss wih = β 1 v (1) + β 2 v (2) + + β n v (n) n linearly independen vecors in R n is called a basis for R n. Orhogonal Vecors The se {v (1), v (2),..., v (k) } is orhogonal is (v (i) ) v (j) = 0 for all i j. If also (v (i) ) v (j) = 1 for i = 1,..., n, i is orhonormal. An orhogonal se of nonzero vecors is linearly independen. If A has disinc eigenvalues λ 1,..., λ k, hen he corresponding se of eigenvecors { (1),..., (k) } is linearly independen. Gram-Schmid Orhogonal Marices (Gram-Schmid) Le { (1), v (2),..., (k) } be linearly independen vecors in R n. Then a se of k orhogonal vecors {v (1), v (2),..., v (k) } in R n is given by v 1 = 1 ( v v 2 = 2 1 2 v1 v 1 ( v v 3 = 3 1 3 v1 v 1 i=1 ) v 1, ) v 1. n 1 ( ) v v k = k i k vi v v i i ( v 2 3 v 2 v 2 ) v 2, A mari Q is orhogonal if is columns {q 1, q 2,..., q n} form an orhonormal se in R n. If Q is an orhogonal n n mari, hen (i) Q is inverible wih Q 1 = Q (ii) For any, y R n, (Q) Qy = y (iii) For any R n, Q 2 = 2
Similar Marices Two marices A and B are similar if a nonsingular mari S eiss wih A = S 1 BS. If A, B are similar, A = S 1 BS, and λ, is an eigenvalue/vecor pair of A. Then λ, S is an eigenvalue/vecor pair of B. An n n mari A is similar o a diagonal mari D if and only if A has n linearly independen eigenvecors. Then, D = S 1 AS when he columns of S are eigenvecors and he diagonal elemens of D are corresponding eigenvalues Corollary An n n mari A wih n disinc eigenvalues is similar o a diagonal mari. Similariy Transformaions (Schur) For any mari A, a nonsingular mari U eiss wih T = U 1 AU where T is an upper-riangular mari wih eigenvalues of A on he diagonal. U saisfies U 2 = 2 for any (uniary). The mari A is symmeric here eiss a diagonal mari D and an orhogonal mari Q wih A = QDQ. Corollary If A is a symmeric n n mari, i has n orhonormal eigenvecors and real eigenvalues. Symmeric mari posiive definie all eigenvalues are posiive. Eigenvalue Algorihms Eigenvalue Algorihms The mos obvious mehod is ill-condiioned: Find roos of p A (λ) Insead, compue Schur facorizaion A = UT U 1 by inroducing zeros However, his can no be done in a finie number of seps: Any eigenvalue solver mus be ieraive To see his, consider a general polynomial of degree m p(z) = z m + a m 1 z m 1 + + a 1 z + a 0 There is no closed-form epression for he roos of p: (Abel, 1842) In general, he roos of polynomial equaions higher han fourh degree canno be wrien in erms of a finie number of operaions (coninued) However, he roos of p are he eigenvalues of he companion mari 0 a 0 1 0 a 1 1 0 a 2 A =. 1...... 0 am 2 1 a m 1 Therefore, in general we canno find he eigenvalues of a mari in a finie number of seps (even in eac arihmeic) In pracice, bes algorihms available converge in jus a few ieraions Power Ieraion Convergence of Power Ieraion Consider he of R n : r() = A For an eigenvecor, he corresponding eigenvalue is r() = λ Simple power ieraion for larges eigenvalue: Algorihm: Power Ieraion Av(k) v (k+1) = Av (k) v (0) = some vecor wih v (0) = 1 w = Av (k 1) v (k) = w/ w λ (k) = (v (k) ) Av (k) apply A normalize Assume linearly independen eigenvecors {q 1,..., q n } Epand iniial v (0) in eigenvecors q i, apply A k : v (0) = a 1 q 1 + a 2 q 2 + + a m q m v (k) = c k A k v (0) = c k (a 1 λ k 1q 1 + a 2 λ k 2q 2 + + a m λ k mq m ) = c k λ k 1(a 1 q 1 + a 2 (λ 2 /λ 1 ) k q 2 + + a m (λ m /λ 1 ) k q m ) If λ 1 > λ 2 λ m 0 and q1 v(0) 0, his gives: ( v (k) (±q 1 ) = O λ 2 /λ 1 k), λ (k) λ 1 = O ( λ 2 /λ 1 k) Finds he larges eigenvalue (unless eigenvecor o v (0) ) Linear convergence, facor λ 2 /λ 1 a each ieraion Convergence can be acceleraed by Aiken s 2 mehod For symmeric A, wice as fas convergence: O( λ 2 /λ 1 2k )
Aiken s 2 Mehod Inverse Ieraion Acceleraing linearly convergen sequences Suppose {p n } n=0 linearly convergen wih limi p Assume ha p n+1 p p n p p n+2 p p n+1 p Solving for p gives p p n+2p n p 2 n+1 (p n+1 p n ) 2 = = p n p n+2 2p n+1 + p n p n+2 2p n+1 + p n Use his for new more rapidly converging sequence {ˆp n } n=0 : (p n+1 p) 2 ˆp n = p n p n+2 2p n+1 + p n or, using dela noaion, ˆp n = p n ( p n) 2 2, for n 0 p n Apply power ieraion on (A µi) 1, wih eigenvalues (λ j µ) 1 Converges o eigenvecor q J if he parameer µ is close o λ J : ( v (k) µ λ J k) ( (±q j ) = O, λ (k) µ λ J ) k λ J = O µ λ K µ λ K Algorihm: Inverse Ieraion v (0) = some vecor wih v (0) = 1 Solve (A µi)w = v (k 1) for w v (k) = w/ w λ (k) = (v (k) ) Av (k) apply (A µi) 1 normalize Rayleigh Quoien Ieraion Reducion o Tridiagonal Form Parameer µ is consan in inverse ieraion, bu convergence is beer for µ close o he eigenvalue Improvemen: A each ieraion, se µ o las compued Algorihm: Rayleigh Quoien Ieraion v (0) = some vecor wih v (0) = 1 λ (0) = (v (0) ) Av (0) = corresponding Solve (A λ (k 1) I)w = v (k 1) for w apply mari v (k) = w/ w normalize λ (k) = (v (k) ) Av (k) Consider symmeric marices A Two phases of eigenvalue compuaions: Firs reducing he mari o ridiagonal form, hen o diagonal A = A Phase 1 T Phase 2 Phase 1 can be done using Householder s mehod Phase 2 easier and faser wih ridiagonal form T han wih original mari A D Householder Transformaions Householder Reflecors Le w R n wih w w = 1. The n n mari P = I 2ww is called a Householder ransformaion A Householder ransformaion, P = I 2ww, is symmeric and orhogonal, so P 1 = P. Le Q k be of he form Q k = where I is (k 1) (k 1) F is (n k + 1) (n k + 1) [ I 0 0 F Creae Householder reflecor F ha inroduces zeros: 2 =. F = 0. = 2e 1 0
Householder Reflecors Choice of Reflecor Idea: Reflec across hyperplane H orhogonal o v = 2 e 1, by he orhogonal mari F = I 2 vv v v We can reflec o any muliple z of 2 e 1 wih z = 1 Beer numerical properies wih large v 2, for eample v = sign( 1 ) 2 e 1 + Compare wih projecor P v = I vv v v H v F = e 1 Noe: sign(0) = 1, bu in MATLAB, sign(0)==0 e 1 e 1 H + H + e 1 + e 1 Inroducing Zeros by Similariy Transformaions Try compuing he diagonal facorizaion A = QDQ by applying Householder reflecors from lef and righ ha inroduce zeros: A Q 1 0 0 0 0 Q 1 A Q 1 Q 1 AQ 1 The firs row changes by he lef muliplicaion, and he righ muliplicaion desroys he zeros previously inroduced We already knew his would no work, because of Abel s heorem oherwise we could compue eigenvalues in finie number of operaions, and herefore also roos of polynomial equaions The Tridiagonal Form Insead, ry compuing a ridiagonal mari T similar o A: 0 0 0 Q 1 Q 0 1 0 0 A Q 1 A Q 1 AQ 1 This ime he zeros we inroduce are no desroyed Coninue in a similar way wih column 2: Q 1 Q 1 0 0 0 0 Q 1 AQ 1 Q 2 Q 1 AQ 1 Q 2 Q 1 AQ 1Q 2 Afer n 2 seps, we obain he ridiagonal form: Q n 2 Q 2Q 1 A Q 1 Q 2 Q n 2 = T = }{{}}{{} Q Q Householder Reducion o Tridiagonal The Hessenberg Form Algorihm: Householder ridiagonal for k = 1 o n 2 = A k+1:n,k v k = sign( 1 ) 2 e 1 + v k = v k / v k 2 A k+1:n,k:n = A k+1:n,k:n 2v k (v k A k+1:n,k:n) A k:n,k+1:n = A k:n,k+1:n 2(A k:n,k+1:n v k )v k For general marices A, compue an upper Hessenberg mari H similar o A: Q 1 Q 0 1 0 0 A Q 1 A Q 1 AQ 1 Afer n 2 seps, we obain he Hessenberg form: Q n 2 Q 2Q 1 A Q 1 Q 2 Q n 2 = H = }{{}}{{} Q Q
The QR Facorizaion QR Facorizaion using Gram-Schmid Find orhonormal vecors ha span he successive spaces spanned by he columns of A: span({a 1 }) span({a 1, a 2 }) span({a 1, a 2, a 3 })... This means ha (for full rank A), span({q 1, q 2,..., q j }) = span({a 1, a 2,..., a j }), j = 1,..., n Find new q j orhogonal o q 1,..., q j 1 by subracing componens along previous vecors v j = a j (q 1a j )q 1 (q 2a j )q 2 (q j 1a j )q j 1 Normalize o ge q j = v j / v j We hen obain a QR facorizaion A = QR, wih In mari form, his becomes A = QR: r [ [ 11 r 12 r 1n a 1 a 2 a n = q 1 q 2 q r 22. n.... r nn and r ij = q ia j, (i j) j 1 r jj = a j r ij q i 2 i=1 Classical Gram-Schmid Givens Roaions Sraigh-forward applicaion of Gram-Schmid orhogonalizaion Numerically unsable Algorihm: Classical Gram-Schmid for j = 1 o n v j = a j for i = 1 o j 1 r ij = q i a j v j = v j r ij q i r jj = v j 2 q j = v j /r jj [ cos θ sin θ A Givens roaion R = roaes R sin θ cos θ 2 by θ To se an elemen o zero, choose cos θ and sin θ so ha [ [ [ cos θ sin θ i = 2 i + 2 j sin θ cos θ j 0 or cos θ = i 2 i + 2 j QR facorize by inroducing zeros: [ (2,3) [ 0, sin θ = (1,2) [ 0 j 2 i + 2 j (2,3) [ = R 0 The QR Algorihm for Eigenvalue Calculaion The Shifed QR Algorihm Remarkably simple algorihm for compuing eigenvalues: QR facorize and muliply in reverse order Algorihm: Pure QR Algorihm A (0) = A Q (k) R (k) = A (k 1) A (k) = R (k) Q (k) QR facorizaion of A (k 1) Recombine facors in reverse order Wih some assumpions, A (k) converge o a Schur form for A (diagonal if A symmeric) Similariy ransformaions of A: A (k) = R (k) Q (k) = (Q (k) ) T A (k 1) Q (k) The QR algorihm behaves like inverse ieraion = inroduce shifs µ (k) o accelerae he convergence: A (k 1) µ (k) I = Q (k) R (k) A (k) = R (k) Q (k) + µ (k) I The shif uses µ (k) = A (k) mm The QR algorihm wih shif migh fail, e.g. wih wo symmeric eigenvalues Break symmery by he Wilkinson shif µ = a n sign(δ)bn 1/ ( ) 2 δ + δ 2 + b 2 n 1 [ an 1 b where δ = (a n 1 a n )/2 and B = n 1 is he lower-righ submari of A (k) b n 1 a n
A Pracical Shifed QR Algorihm Algorihm: Pracical QR Algorihm (Q (0) ) T A (0) Q (0) = A A (0) is ridiagonalizaion of A Pick a shif µ (k) e.g., choose µ (k) = A mm (k 1) Q (k) R (k) = A (k 1) µ (k) I QR of A (k 1) µ (k) I A (k) = R (k) Q (k) + µ (k) I Recombine in reverse order If any off-diagonal elemen A (k) j,j+1 is close o zero, se [ A j,j+1 = A j+1,j = 0 o obain A1 0 = A 0 A (k) 2 and now apply he QR algorihm o A 1 and A 2