THE INVERSE POWER METHOD (or INVERSE ITERATION) -- applcaton of the Power method to A some fxed constant ρ (whch s called a shft), x λ ρ If the egenpars of A are { ( λ, x ) } ( ), or (more usually) to, then the egenpars of ( ( A ρ I), for A ρ I) are that Let (0) be the startng vector and (assumng that A s nondefectve) suppose ( 0) () () ( n) α x + α x + + α n x = L Then () = ( A ρi ) α = x () (0) α + x () α n + L+ x n ( n) If ρ some λ, then >> for, whch mples that the teraton wll converge very fast NOTE Shfts are much more effectve for the nverse Power method than for the Power method They enable you to compute any egenvector, not ust those assocated wth the largest or the smallest egenvalue 96
97 EXAMPLE = 5 0 0 4 0 A has egenpars 4,,, 4,
Use of the INVERSE POWER METHOD You need a reasonably good approxmaton to some egenvalue -- ths s obtaned by other means You do not explctly compute lnear system ( A ρ I) Instead, for example, you solve the () (0) ( A ρ I) = () for You need to compute the LU factorzaton of A ρi only once, and then use t to solve a lnear system n each teraton If A s a full matrx, the cost s ( / ) n flops () for the LU factorzaton and n flops to compute each vector These costs are much cheaper f A s upper Hessenberg or trdagonal As ρ some egenvalue λ, A ρi becomes very close to a sngular matrx, whch suggests that ( + ) ( ) ( A ρ I) = may be dffcult to solve; A ρi wll lely be very ll-condtoned However, ths algorthm wors well n practce -- see the dscusson on page 4 Usng, for example, Gaussan elmnaton wth partal pvotng, the computed ( +) approxmaton to s the exact soluton of some perturbed lnear system δa ( + ) ( ) ( A + δ A ρi ) ˆ =, where s very small In ths case, ρ s very lely A ρi ( ) also a good approxmaton to an egenvalue of A + δa, and thus ˆ + should be very close to an egenvector of A + δa, whch should then also be very close to an egenvector of A Thus, nverse teraton s usually very effectve From above, how effectve t s depends on whether or not a small perturbaton δ A n A ρi results n only a small perturbaton n ts egenvalue and egenvector That s, ths depends on the condton number of ths egenvalue and egenvector of A ρi and not on the condton number of A ρi (wth respect to solvng the above lnear system) In other words, the effectveness of nverse teraton does not depend on the condton number κ ( A ρi) THE RAYLEIGH QUOTIENT (page 4) Gven a (complex) vector x wth n entres and a (complex) Raylegh uotent s n n matrx A, the 98
T ρ x Ax x Ax = or n the real case T Theorem 54 (page 5) Gven A and x, let r ( µ ) = Ax µ x Then r (µ) s mnmzed when x Ax µ =, the Raylegh uotent Proof mn µ Ax µ x s a least-suares problem n varable, namely µ Its assocated lnear system s the n over-determned lnear system x µ = Ax The soluton can be obtaned from the normal euatons (a lnear system), whch s ( x x) µ = x Ax, from whch t follows that x Ax µ = Alternatve proof: Gven A and x, { Ax µx} s a lne (a -dmensonal subspace) n the vector space of complex vectors wth n entres Ax {Ax - µx} x 99
mn µ Therefore, Ax µ x s obtaned when µ s such that Ax µ x s orthogonal to x ( Ax µ x, x) = 0 x Ax µ = 0 x Ax µ = NOTE If x s an egenvector of A (suppose that sdes by x, Ax = λx ), then on multplyng both x Ax = λ λ = x Ax That s, the Raylegh uotent wth respect to an egenvector s eual to an egenvalue Thus, Ax µ x = 0 f and only f ( µ, x) s an egenpar of A, and f x s not an egenvector of A, then Ax µ x 0 but x Ax mn Ax µ x occurs when µ = µ The pont of the above dscusson: gven A and x (and thnng of x as an approxmaton to an egenvector), the Raylegh uotent s the best approxmaton (n the above sense) to an egenvalue Gven a vector, the followng result ndcates how good an approxmaton to an egenvalue the Raylegh uotent s, n terms of how close s to an egenvector Theorem 55 (page 6) Suppose Av = λv, where, and let ρ = A = A Then = A v = 00
Proof v Av Av = λ v λ = = v Av v v Therefore, = v Av A = v Av v A + v A A = v A( v ) + ( v ) A whch mples that v A( v ) + ( v ) A Recall the Cauchy-Schwarz neualty: ( x, y) x y, where ( x, y) = y x Ths gves Smlarly, v A( v ) v A( v ) = A( v ) A v ( v ) A v A Thus, from above, A v EXAMPLE If v < ε, then A ε That s, f v s O (ε ), then s also O (ε ) 0
THE RAYLEIGH QUOTIENT ITERATION -- a varaton of the nverse Power method -- a dfferent shft s used at each step, namely the Raylegh uotent of the current approxmate egenvector Algorthm Choose a startng vector x such that = (that s, x ) Repeat untl convergence: ρ x Ax apply the nverse Power method: solve ( A ρ I) xˆ = x for xˆ xˆ x (that s, normalze xˆ ) xˆ Because a dfferent shft s used at every step, ths algorthm s not guaranteed to converge to an egenvector However, n practce t very seldom fals, and usually converges very rapdly to an egenvector of ( A ρ I) correspondng to the domnant egenvalue of (A ρi), for some ρ Snce { ρ } computed n Step are Raylegh uotents, {} x an egenvector of A correspondng to an egenvalue λ ρ of A The Raylegh uotent of ths computed egenvector then gves the approxmaton to λ = 0
CONVERGENCE OF THE RAYLEIGH QUOTIENT ALGORITHM (pages 7-8) If convergent, then the Raylegh Quotent algorthm converges uadratcally (order ) The proof asumes that the egenvalue λ (to whch t converges) s smple (multplcty s ) A setch of the argument: Suppose that the egenpars of A are ( λ t, vt ) The Raylegh Quotent algorthm computes { ρt } λ (assumed to be a smple egenvalue) { t } egenvector v Suppose that λ s the closest egenvalue to λ Analyss of the Inverse Power method gves v + r v, where r s the rato of the largest egenvalues of error of the r ) ( A ρ I) st ( + approxmaton to + error of the th approxmaton to v s to v That s, For suffcently large, snce s the largest egenvalue of ( A ρ I), r = = λ λ A v λ λ by Thm 55 Therefore, v A + λ λ v v, whch s uadratc convergence NOTE If A s a real symmetrc (or Hermtan) matrx, the Raylegh Quotent algorthm s cubcally convergent: 0
v constant + v Reason: for Hermtan matrces A, the result of Theorem 55 s stronger -- f v = O( ) and ρ = A, then = O( ε ) For a dscusson of ths, see the ε mddle of page 8 COST OF THE RAYLEIGH QUOTIENT ALGORITHM (page 9) O ( n ) flops per teraton for a full matrx, snce the LU factorzaton must be recomputed for each teraton However, the cost s only O ( n ) flops per teraton f A s upper Hessenberg See Exercse 5 04