arxiv:cs/ v4 [cs.na] 21 Nov 2005

Transcription

1 arxiv:cs/0300v4 [cs.na] Nov 005 Smoothed Analysis of the Condition Numbers and Growth Factors of Matrices Arvind Sankar Department of Mathematics Massachusetts Institute of Technology Shang-Hua Teng Department of Computer Science Boston University and Akamai Technologies Inc. June, 08 Abstract Daniel A. Spielman Department of Computer Science Yale University Let A be an arbitrary matrix and let A be a slight random perturbation of A. We prove that it is unlikely that A has large condition number. Using this result, we prove it is unlikely that A has large growth factor under Gaussian elimination without pivoting. By combining these results, we show that the smoothed precision necessary to solve Ax = b, for any b, using Gaussian elimination without pivoting is logarithmic. Moreover, when A is an all-zero square matrix, our results significantly improve the average-case analysis of Gaussian elimination without pivoting performed by Yeung and Chan (SIAM J. Matrix Anal. Appl., 997). Partially supported by NSF grant CCR-0487 Partially supported by an Alfred P. Sloan Foundation Fellowship, and NSF grants CCR-0487 and CCR Partially supported by an Alfred P. Sloan Foundation Fellowship, NSF grant CCR-99753, and NSF grants CCR and ITR CCR

2 Introduction Spielman and Teng [ST04], introduced the smoothed analysis of algorithms to explain the success of algorithms and heuristics that could not be well understood through traditional worst-case and average-case analyses. Smoothed analysis is a hybrid of worst-case and average-case analyses in which one measures the maximum over inputs of the expected value of a measure of the performance of an algorithm on slight random perturbations of that input. For example, the smoothed complexity of an algorithm is the maximum over its inputs of the expected running time of the algorithm under slight perturbations of that input. If an algorithm has low smoothed complexity and its inputs are subject to noise, then it is unlikely that one will encounter an input on which the algorithm performs poorly. (See also the Smoothed Analysis Homepage [Smo]) Smoothed analysis is motivated by the existence of algorithms and heuristics that are known to work well in practice, but which are known to have poor worst-case performance. Average-case analysis was introduced in an attempt to explain the success of such heuristics. However, averagecase analyses are often unsatisfying as the random inputs they consider may bare little resemblance to the inputs actually encountered in practice. Smoothed analysis attempts to overcome this objection by proving a bound that holds in every neighborhood of inputs. In this paper, we prove that perturbations of arbitrary matrices are unlikely to have large condition numbers or large growth factors under Gaussian Elimination without pivoting. As a consequence, we conclude that the smoothed precision necessary for Gaussian elimination is logarithmic. We obtain similar results for perturbations that affect only the non-zero and diagonal entries of symmetric matrices. We hope that these results will be a first step toward a smoothed analysis of Gaussian elimination with partial pivoting an algorithm that is widely used in practice but known to have poor worst-case performance. Intherestofthissection, werecall thedefinitionsoftheconditionnumbersandgrowthfactors of matrices, and review prior work on their average-case analysis. In Section 3, we perform a smoothed analysis of thecondition numberof amatrix. InSection 4, we usetheresultsof Section 3toobtain a smoothed analysis of the growth factors of Gaussian elimination without pivoting. In Section 5, we combine these results to obtain a smoothed bound on the precision needed by Gaussian elimination without pivoting. Definitions of zero-preserving perturbations and our results on perturbations that only affect the non-zero and diagonal entries of symmetric matrices appear in Section 6. In the conclusion section, we explain how our results may be extended to larger families of perturbations, present some counter-examples, and suggest future directions for research. Other conjectures and open questions appear in the body of the paper. The analysis in this paper requires many results from probability. Where reasonable, these have been deferred to the appendix.

3 . Condition numbers and growth factors We use the standard notation for the, and -norms of matrices and column vectors, and define A Ñ Ü = Ñ Ü A i,j. i,j Definition. (Condition Number). ÓÖ ÕÙ Ö Ñ ØÖ Ü A Ø ÓÒ Ø ÓÒ ÒÙÑ Ö Ó A Ò Ý A κ(a) = A. The condition number measures how much the solution to a system Ax = b changes as one makes slight changes to A and b. A consequence is that if ones solves the linear system using fewer than ÐÓ (κ(a)) bits of precision, one is likely to obtain a result far from a solution. For more informationontheconditionnumberofamatrix, wereferthereadertooneof[gl83,tb97,dem97]. The simplest and most often implemented method of solving linear systems is Gaussian elimination. Natural implementations of Gaussian elimination use O ( n 3) arithmetic operations to solve a system of n linear equations in n variables. If the coefficients of these equations are specified using b bits, in the worst case it suffices to perform the elimination usingo(bn) bits of precision [GLS9]. This high precision may be necessary because the elimination may produce large intermediate entries [TB97]. However, in practice one usually obtains accurate answers using much less precision. In fact, it is rare to find an implementation of Gaussian elimination that uses anything more than double precision, and high-precision solvers are rarely used or needed in practice [TB97, TS90] (for example, LAPACK uses 64 bits [ABB + 99]). One of the main results of this paper is that O(b+ÐÓ n) bits of precision usually suffice for Gaussian elimination in the smoothed analysis framework. Since Wilkinson s seminal work [Wil6], it has been understood that it suffices to carry out Gaussian elimination with b+ðó (5nκ(A) L U / A +3) bits of accuracy to obtain a solution that is accurate to b bits. In this formula, L and U are the LU-decomposition of A; that is, U is the upper-triangular matrix and L is the lower-triangular matrix with s on the diagonal for which A = LU.. Prior work The average-case behaviors of the condition numbers and growth factors of matrices have been studied both analytically and experimentally. In his paper, The probability that a numerical analysis problem is difficult, Demmel [Dem88] proved that it is unlikely that a Gaussian random matrix centered at the origin has large condition number. Demmel s bounds on the condition number were improved by Edelman [Ede88]. Average-case analysis of growth factors began with the experimental work of Trefethen and Schreiber [TS90], who found that Gaussian random matrices rarely have large growth factors under partial or full pivoting.

4 Definition. (Gaussian Matrix). Ñ ØÖ Ü G Ù Ò Ö Ò ÓÑ Ñ ØÖ Ü Ó Ú Ö Ò σ ÒØÖÝ Ó G Ò Ò Ô Ò ÒØ ÙÒ Ú Ö Ø Ù Ò Ú Ö Ð Û Ø Ñ Ò 0 Ò Ø Ò Ö Ú Ø ÓÒ σº Yeung and Chan [YC97] study the growth factors of Gaussian elimination without pivoting on Gaussian random matrices of variance. They define ρ U and ρ L by ρ U (A) = U / A, and ρ L (A) = L, where A = LU is the LU-factorization of A obtained without pivoting. They prove Theorem.3 (Yeung-Chan). Ì Ö Ü Ø ÓÒ Ø ÒØ c > 0 Ò 0 < b < Ù Ø Ø G Ò n n Ù Ò Ö Ò ÓÑ Ñ ØÖ Ü Ó Ú Ö Ò Ò G = LU Ø Ä͹ ØÓÖ Þ Ø ÓÒ Ó G Ø Ò [ρ L (G) > x] cn3 x, Ò ( ) cn 7/ [ρ U (G) > x] Ñ Ò x, n + cn5/ x +b n. As it is generally believed that partial pivoting is better than no pivoting, their result provides some intuition for the experimental results of Trefethen and Schreiber demonstrating that random matrices rarely have large growth factors under partial pivoting. However, we note that it is difficult to make this intuition rigorous as there are matrices A for which no pivoting has L Ñ Ü U Ñ Ü / A Ñ Ü = while partial pivoting has growth factor n. (See also [Hig90]) The running times of many numerical algorithms depend on the condition numbers of their inputs. For example, the number of iterations taken by the method of conjugate gradients can be bounded in terms of the square root of the condition number. Similarly, the running times of interior-point methods can be bounded in terms of condition numbers [Ren95]. Blum [Blu89] suggested that a complexity theory of numerical algorithms should be parameterized by the condition number of an input in addition to the input size. Smale [Sma97] proposed a complexity theory of numerical algorithms in which one:. proves aboundon therunningtimeof analgorithm solvingaproblem intermsof its condition number, and then. proves that it is unlikely that a random problem instance has large condition number. This program is analogous to the average-case complexity of Theoretical Computer Science. 3

5 .3 Our results To better model the inputs that occur in practice, we propose replacing step of Smale s program with. prove that for every input instance it is unlikely that a slight random perturbation of that instance has large condition number. That is, we propose to bound the smoothed value of the condition number. Our first result in this program is presented in Section 3, where we improve upon Demmel s [Dem88] and Edelman s [Ede88] average-case results to show that a slight Gaussian perturbation of an arbitrary matrix is unlikely to have large condition number. Definition.4 (Gaussian Perturbation). Ä Ø A Ò Ö ØÖ ÖÝ n n Ñ ØÖ Üº Ì Ñ ØÖ Ü A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ A Ò ÛÖ ØØ Ò A = A+G Û Ö G Ù Ò Ö Ò ÓÑ Ñ ØÖ Ü Ó Ú Ö Ò σ º Ï Ð Ó Ö Ö ØÓ A Ù Ò Ñ ØÖ Ü Ó Ú Ö Ò σ ÒØ Ö Ø Aº In our smoothed analysis of the condition number, we consider an arbitrary n n matrix A of norm at most n, and we bound the probability that κ( A + G), the condition number of its Gaussian perturbation, is large, where G is a Gaussian random matrix of variance σ. We bound this probability in terms of σ and n. In contrast with the average-case analysis of Demmel and Edelman, our analysis can be interpreted as demonstrating that if there is a little bit of imprecision or noise in the entries of a matrix, then it is unlikely it is ill-conditioned. On the other hand, Edelman [Ede9] writes of random matrices: What is a mistake is to psychologically link a random matrix with the intuitive notion of a typical matrix or the vague concept of any old matrix. The reader might also be interested in recent work on the smoothed analysis of the condition numbers of linear programs [BD0, DST0, ST03]. In Section 4, we use results from Section 3 to perform a smoothed analysis of the growth factors of Gaussian elimination without pivoting. If one specializes our results to perturbations of an allzero square matrix, then one obtains a bound on ρ U that improves the bound obtained by Yeung and Chan by a factor of n and which agrees with their experimental observations. The result obtained for ρ L also improves the bound of Yeung and Chan [YC97] by a factor of n. However, while Yeung and Chan compute the density functions of the distribution of the elements in L and U, such precise estimates are not immediately available in our model. As a result, the techniques we develop are applicable to a wide variety of models of perturbations beyond the Gaussian. For example, one could use our techniques to obtain results of a similar nature if G were a matrix of random variables chosen uniformly in [, ]. We comment further upon this in the conclusions section of the paper. 4

6 The less effect a perturbation has, the more meaningful the results of smoothed analysis are. As many matrices encountered in practice are sparse or have structure, it would be best to consider perturbations that respect their sparsity pattern or structure. Our first result in this direction appears in Section 6, in which we consider the condition numbers and growth factors of perturbations of symmetric matrices that only alter their non-zero and diagonal elements. We prove results similar to those proved for dense perturbations of arbitrary matrices. Notation and Mathematical Preliminaries We use bold lower-case Roman letters such as x, a, b j to denote vectors in R?. Whenever a vector, say a R n is present, its components will be denoted by lower-case Roman letters with subscripts, such as a,...,a n. Matrices are denoted by bold upper-case Roman letters such as A and scalars are denoted by lower-case roman letters. Indicator random variables and random event variables are denoted by upper-case Roman letters. Random variables taking real values are denoted by upper-case Roman letters, except when they are components of a random vector or matrix. The probability of an event A is written [A], and the expectation of a variable X is written [X]. The indicator random variable for an event A is written [A]. We write ÐÒ to denote the natural logarithm, base e, and explicitly write the base for all other logarithms. For integers a b, we let a : b denote the set of integers {x: a x b}. For a matrix A we let A a:b,c:d denote the submatrix of A indexed by rows in a : b and columns in c : d. We will bound many probabilities by applying the following proposition. Proposition. (Minimum Average Maximum). Ä Ø µ(x,y) ÒÓÒ¹Ò Ø Ú ÒØ ¹ Ö Ð ÙÒØ ÓÒ Ò Ð Ø X Ò Y Ö Ò ÓÑ Ú Ö Ð ØÖ ÙØ ÓÖ Ò ØÓ µ(x,y)º Á A(X,Y) Ò Ú ÒØ Ò F(X,Y) ÙÒØ ÓÒ Ø Ò Ñ Ò [A(X,Y)] [A(X,Y)] Ñ Ü [A(X,Y)],and X Y X,Y X Y Ñ Ò Y [F(X,Y)] X,Y [F(X,Y)] Ñ Ü Y[F(X,Y)], X X Û Ö Ò Ø Ð Ø¹ Ò Ò Ö Ø¹ Ò Ø ÖÑ Y ØÖ ÙØ ÓÖ Ò ØÓ Ø Ò Ù ØÖ ÙØ ÓÒ ÓÒ µ(x,y)º We recall that a matrix Q is an orthonormal matrix if its inverse is equal to its transpose, that is, Q T Q = I. In Section 3 we will use the following proposition. Proposition. (Orthonormal Transformation of Gaussian). Ä Ø A Ñ ØÖ Ü Ò R n n Ò Q Ò ÓÖØ ÓÒÓÖÑ Ð Ñ ØÖ Ü Ò R n n º Á A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ Ø Ò QA Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó Q A Ó Ú Ö Ò σ º We will also use the following extension of Proposition.7 of [ST04]. 5

7 Proposition.3 (Gaussian Measure of Halfspaces). Ä Ø t ÒÝ ÙÒ Ø Ú ØÓÖ Ò R n Ò r ÒÝ Ö Ðº Ä Ø b Ú ØÓÖ Ò R n Ò b Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó b Ó Ú Ö Ò σ º Ì Ò b [ ] t b T r t=r e t /σ dt. πσ t= r In this paper we will use the following properties of matrix norms and vector norms. Proposition.4 (Product). ÓÖ ÒÝ Ô Ö Ó Ñ ØÖ A Ò B Ù Ø Ø AB Ò Ò ÓÖ Ú ÖÝ p AB p A p B p. Proposition.5 (Vector Norms). ÓÖ ÒÝ ÓÐÙÑÒ Ú ØÓÖ a Ò R n a / n a a º Proposition.6 (-norm). ÓÖ ÒÝ Ñ ØÖ Ü A A = A T, ÓØ Ö ÕÙ Ð ØÓ Ø Ð Ö Ø ÒÚ ÐÙ Ó A T Aº Proposition.7 ( A : the maximum absolute row sum norm). ÓÖ Ú ÖÝ Ñ ØÖ Ü A A = Ñ Üa T i, i Û Ö a,...,a n Ö Ø ÖÓÛ Ó Aº Ì Ù ÓÖ ÒÝ Ù Ñ ØÖ Ü D Ó A D A. Proposition.8 ( A : the maximum absolute column sum norm). ÓÖ Ú ÖÝ Ñ ØÖ Ü A Û Ö a,...,a n Ö Ø ÓÐÙÑÒ Ó Aº Ì Ù A = Ñ Ü a i, i A = A T. 3 Smoothed analysis of the condition number of a matrix In this section, we will prove the following theorem which shows that for every matrix it is unlikely that a slight perturbation of that matrix has large condition number. 6

8 Theorem 3. (Smoothed Analysis of Condition number). Ä Ø A Ò n n Ñ ØÖ Ü Ø ¹ Ý Ò A n Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò x [κ(a) x] ( 4.n + ) ÐÒ(x)/9n. xσ As bounds on the norm of a random matrix are standard, we focus on the norm of the inverse. Recall that / A = Ñ Ò x Ax / x. The first step in the proof is to bound the probability that A v is small for a fixed unit vector v. This result is also used later (in Section 4.) in studying the growth factor. Using this result and an averaging argument, we then bound the probability that A is large. Lemma 3. (Projection of A ). Ä Ø A Ò Ö ØÖ ÖÝ ÕÙ Ö Ñ ØÖ Ü Ò R n n Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ä Ø v Ò Ö ØÖ ÖÝ ÙÒ Ø Ú ØÓÖº Ì Ò ] A [ v > x < πxσ Proof. Let Q be an orthonormal matrix such that Q T e = v. Let B = QA and B = QA. By Proposition., B is a Gaussian perturbation of B of variance σ. We have A v = A Q T e = (QA) e = B e. Thus, to prove the lemma it is sufficient to show [ B e > x] < B We observe that B e = πxσ. (B ) :,, thelengthofthefirstcolumnofb. ThefirstcolumnofB,bythedefinitionofthematrixinverse, is the vector that is orthogonal to every row of B but the first and that has inner product with the first row of B. Hence its length is the reciprocal of the length of the projection of the first row of B onto the subspace orthogonal to the rest of the rows. Let b,...,b n be the rows of B and b,..., bn be the rows of B. Note that bi is a Gaussian perturbation of bi of variance σ. Let t be the unit vector that is orthogonal to the span of b,...,b n. Then (B ) :, = t T b. 7

9 Thus, B ] t T b > x [ ] t Ñ Ü T b < /x b,...,b n b < πxσ, [ ] [ B v > x = b,...,b n where the first inequality follows from Proposition. and the second inequality follows from Lemma A.. Theorem 3.3 (Smallest singular value). Ä Ø A Ò Ö ØÖ ÖÝ ÕÙ Ö Ñ ØÖ Ü Ò R n n Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò [ ] A n x.35 A xσ Proof. Let v be a uniformly distributed random unit vector in R n. It follows from Lemma 3. that A,v [ ] A v x π Since A is a Gaussian perturbation of A, with probability there is a unique pair (u, u) of unit vectors such that A u = A. From the inequality A v A u T v, we know that for every c > 0, [ A v x ] c/n A,v A,v = A,v = A A A [ A [ ] A x A,v [ ] A x [ A x [ A x xσ (3.) x and u T v ] c/n [ u T v ] c/n A x [ u T v ] c/n A x A,v ] [ u Ñ Ò T v ] c/n (by Proposition.) A: A x v ] [ ] G c, (by Lemma B.) G where G is a Gaussian random variable with mean 0 and variance. To prove this last inequality, we first note that that v is a random unit vector and is independent from u. Thus, in a basis of 8

10 R n in which u is the first vector, v is a uniformly distributed random unit [ vector with the first coordinate equal to u T v, and so we may apply Lemma B. to bound v u T v ] c/n from below by G [ G c ]. So, A [ A [ ] A A,v v x ] c/n x [ ] G G c n πxσ [ ] (by (3.)). c G G c Because this inequality is true for every c, we will choose a value for c that almost maximizes [ ] c G G c and which in turn almost minimizes the right hand side. Choosing c = 0.57, and evaluating the error function numerically, we determine A [ ] A x.35 n xσ. Note that Theorem 3.3 gives a smoothed analogue of the following bound of Edelman [Ede88] on Gaussian random matrices. Theorem 3.4 (Edelman). Ä Ø G R n n Ù Ò Ö Ò ÓÑ Ñ ØÖ Ü Û Ø Ú Ö Ò σ Ø Ò G [ ] G n x xσ. As Gaussian random matrices can be viewed as Gaussian random perturbations of the n n all-zero square matrix, Theorem 3.3 extends Edelman s theorem to Gaussian random perturbations of an arbitrary matrix. The constant.35 in Theorem 3.3 is bigger than Edelman s for Gaussian random matrices. We conjecture that it is possible to reduce.35 in Theorem 3.3 to as well. Conjecture (Smallest Singular Value). Ä Ø A Ò Ö ØÖ ÖÝ ÕÙ Ö Ñ ØÖ Ü Ò R n n Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò [ ] A n x A xσ We now apply Theorem 3.3 to prove Theorem 3.. 9

11 Proof of Theorem 3.. As observed by Davidson and Szarek [DS0, Theorem II.7], one can apply inequality (.4) of [LT9] to show that for all k 0, [ A A σ ( n+k )] e k /. A Replacing σ by its upper bound of and setting ǫ = e k /, we obtain [ A A n+ ÐÒ(/ǫ)] ǫ, A for all ǫ. By assumption, A n; so, A From the result of Theorem 3.3, we have [ A 3 n+ ÐÒ(/ǫ) A [ A.35 n ǫσ ] ǫ. ] ǫ. Combining these two bounds, we find [ ] A A 7.05n+.35 n ÐÒ(/ǫ) ǫ. A ǫσ So that we can express this probability in the form of A [ A A x ], for x, we let x = 7.05n+.35 n ÐÒ(/ǫ). (3.) ǫσ It follows Equation (3.) and the assumption σ that xǫ, implying ÐÒ(/ǫ) ÐÒx. From Equation (3.), we derive ( 7.05n+.35 ) ( n ÐÒ(/ǫ) 7.05n+.35 ) ( n ÐÒx 4.n + ) ÐÒ(x)/9n ǫ =. xσ xσ xσ Therefore, we conclude [ ] A A x ( 4.n + ) ÐÒ(x)/9n xσ. We conjecture that the + ÐÒ(x)/9n term should be unnecessary because those matrices for which A is large are less likely to have A large as well. 0

12 4 Growth Factor of Gaussian Elimination without Pivoting Wenowturntoprovingaboundonthegrowthfactor. WewillconsideramatrixA R n n obtained from a Gaussian perturbation of variance σ of an arbitrary matrix A satisfying A. With probability, none of the diagonal entries that occur during elimination will be 0. So, in the spirit of Yeung and Chan [YC97], we analyze the growth factor of Gaussian elimination without pivoting. When we specialize our smoothed analyses to the case A = 0, we improve the bounds of Yeung and Chan (see Theorem.3) by a factor of n. Our improved bound on ρ U agrees with their experimental analyses. 4. Growth in U We recall that ρ U (A) = U A. In this section, we give two bounds on ρ U (A). The first will have a better dependence on σ, and second will have a better dependence on n. It is the later bound, Theorem 4.3, that agrees with the experiments of Yeung and Chan [YC97] when specialized to the average-case by setting A = 0 and σ =. 4.. First bound Theorem 4. (First bound on ρ U (A)). Ä Ø A Ò n n Ñ ØÖ Ü Ø Ý Ò A Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò Proof. By Proposition.7. [ρ U (A) > +x] < n(n+). π xσ ρ U (A) = U (Ui,: ) T = Ñ Ü. A i A So, we need to bound the probability that the -norm of the vector defined by each row of U is large and then apply a union bound to bound the overall probability. Fix for now a k between and n. We denote the upper triangular segment of the kth row of U by u T = U k,k:n, and observe that u can be obtained from the formula: u T = a T b T C D (4.) where a T = A k,k:n b T = A k,:k C = A :k,:k D = A :k,k:n.

13 This expression for u follows immediately from ( ) ( C D L:k,:k 0 A :k,: = b T a T = L k,:k From (4.), we derive T u = (b a T C D) )( U:k,:k U :k,k:n 0 u T ( T a + b T C D) a T ( + C T) b D by Propositions.4 and.8 ) A (+ (C T) b by Proposition.7 We now bound the probability ( C T) b is large. By Proposition.5, ( C T) b ( k C T) b. Note that b and C are independent of each other. Therefore, [ ( ] [ C T) ( b > x C T) b > x/ ] k b,c b,c k (k )σ + k < π xσ πxσ, (4.3) where the second inequality follows from Lemma 4. below and the last inequality follows from the assumption σ. We now apply a union bound over the choices of k to obtain n k [ρ U (A) > +x] < πxσ n(n+). π xσ k= ). (4.) Lemma 4.. Ä Ø C Ò Ö ØÖ ÖÝ ÕÙ Ö Ñ ØÖ Ü Ò R d d Ò C Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó C Ó Ú Ö Ò σ º Ä Ø b ÓÐÙÑÒ Ú ØÓÖ Ò R d Ù Ø Ø b Ò Ð Ø b Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó b Ó Ú Ö Ò σ º Á b Ò C Ö Ò Ô Ò ÒØ Ó ÓØ Ö Ø Ò b,c [ ] C b x π σ d+ xσ

14 Proof. Let ^b be the unit vector in the direction of b. By applying Lemma 3., we obtain for all b, [ ] [ C C b > x = ^b > x ] C C b πxσ b. Let µ(b) denote the density according to which b is distributed. Then, we have [ ] [ ] C b C > x = b > x µ(b)db b,c b R d C ( ) b R d πxσ b µ(b)db = πxσ b[ b ]. ] It is known [KJ8, p. 77] that b [ b σ d+ b. As [X] [X ] for every positive random variable X, we have b [ b ] σ d+ b σ d Second Bound for ρ U (A) In this section, we establish an upper bound on ρ U (A) which dominates the bound in Theorem 4. for σ n 3/. If we specialize the parameters in this bound to A = 0 and σ =, we improve the average-case bound proved by Yeung and Chan [YC97] (see Theorem.3) by a factor of n. Moreover, the resulting bound agrees with their experimental results. Theorem 4.3 (Second bound on ρ U (A)). Ä Ø A Ò n n Ñ ØÖ Ü Ø Ý Ò A Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º ÓÖ n ( [ρ U (A) > +x] πx 3 n3/ + n σ + 4 ) n 3 σ Proof. As in the proof of Theorem 4., we will separately consider the kth row of U for each k n. For any such k, define u, a, b, C and D as in the proof of Theorem 4.. In the case when k = n, we may apply (4.3) in the proof of Theorem 4., to show ] u n [ > +x A πxσ. (4.4) We now turn to the case k n. By (4.) and Proposition.5, we have ( T u a + b T C D) a + ( T k b T C D) = a + k b T C D. 3

15 The last equation follows from Proposition.6. Therefore, for all k n, u a + k b T C D k b T C D + (by Proposition.7). A A A k b T C D + (A n,: ) T (also by Proposition.7). We now observe that for fixed b and C, (b T C )D is a Gaussian random row vector of variance b T C σ centered at (b T C ) D, where D is the center of D. We have D A, by the assumptions of the theorem; so, b,c,d b T C D b T C D b T C. Thus, if we let t T = (b T C D)/ b T C, then for every fixed b and C, t is a Gaussian random column vector in R n k+ of variance σ centered at a vector of -norm at most. We also have [ ] [ b T C D b x T C t x]. (4.5) It follows from Lemma 4. that [ b T C x] b,c = b,c,t σ (k )+. π xσ Hence, we may apply Corollary C.5 to show [ b T C σ t x] (k )+ σ (n k+)+ b,c,t π ( ) xσ + nσ. (4.6) π xσ Note that A n,: is a Gaussian perturbation of variance σ of a row vector in R n. As A n,: is independent of b, C and D, we can apply (4.5), (4.6) and Lemma C.4 to show k b T C D ) (A n,: ) T x k (+ nσ π xσ (A n,: ) T ) k (+ nσ by Lemma A.4. π 4 xσ nσ,

16 Applying a union bound over the choices for k, we obtain [ρ U (A) > +x] = n k= π ( πx 3 πx ) k (+ nσ xσ ( ) n σ +n + n ) σ ( 3 n3/ + n σ ) n σ, n + nσ πxσ where the second inequality follows from n k 3 n3/. k= 4. Growth in L Let L be the lower-triangular part of the LU-factorization of A. We have / L (k+):n,k = A (k ) (k+):n,k A (k ) k,k, where we let A (k) denote the matrix remaining after the first k columns have been eliminated. So, A (0) = A. Recall ρ L (A) = L, which is equal to the maximum absolute row sum of L (Proposition.7). We will show that it is unlikely that L (k+):n,k is large by proving that it is unlikely that is large while A (k ) is small. A (k ) (k+):n,k k,k Theorem 4.4 (ρ L (A)). Ä Ø A Ò n¹ ݹn Ñ ØÖ Ü ÓÖ Û A Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Á n Ø Ò ( ) n [ρ L (A) > x] π x σ + ÐÒn+ π ÐÒn 5

17 Proof. For each k between and n, we have L (k+):n,k = A(k ) (k+):n,k A (k ) k,k = A (k+):n,k A (k+):n,:(k ) A :(k ),:(k ) A :(k ),k A k,k A k,:(k ) A :(k ),:(k ) A :(k ),k = A (k+):n,k A (k+):n,:(k ) v, A k,k A k,:(k ) v whereweletv = A :(k ),:(k ) A :(k ),k. Since A,andallthetermsA (k+):n,k, A (k+):n,:(k ), A k,k, A k,:(k ) and v are independent, we can apply Lemma 4.5 to show that L(k+):n,k ] ( ) [ > x πx σ + ÐÒ(Ñ Ü(n k,))+ π ÐÒ(Ñ Ü(n k,)) ( ) πx σ + ÐÒn+, π ÐÒn wherethe last inequality follows the facts that z+ πz is an increasing function whenz π /3, and ÐÒ π /3. The theorem now follows by applying a union bound over the n choices for k and observing that L is at most n times the largest entry in L. Lemma 4.5 (Vector Ratio). Ä Ø d Ò n ÔÓ Ø Ú ÒØ Ö º Ä Ø a b x Ò Y Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó a R b Rd x R n Ò Y R n d Ö Ô Ø Ú ÐÝ Ó Ú Ö Ò σ Ù Ø Ø a b x Ò Y º Ä Ø v Ò Ö ØÖ ÖÝ Ú ØÓÖ Ò R d º Á a b x Ò Y Ö Ò Ô Ò ÒØ Ò σ Ø Ò [ x+yv a+b T v ] > x πx ( σ + ÐÒmax(n,)+ ), π ÐÒmax(n,) Proof. We begin by observing that a+b T v and each component of x+yv is a Gaussian random variable of variance σ ( + v ) whose mean has absolute value at most + v, and that all these variables are independent. By Lemma A.3, ( )( ) x,y [ x+yv ] + v + σ (+ v ) ÐÒmax(n,)+. π ÐÒmax(n,) 6

18 On the other hand, Lemma A. implies [ a,b a+b T v > x ] π xσ + v. (4.7) Thus, we can apply Corollary C.4 to show ( ( ) ] + v x+yv + σ + v ) [ ÐÒmax(n,)+ πðòmax(n,) a+b T > x v π xσ + v ( ( σ + v = + v ) ÐÒmax(n,)+ πx + σ + v σ + v ( ) πx σ + ÐÒmax(n,)+, π ÐÒmax(n,) where the last inequality follows from (+z) (+z ), z 0. 5 Smoothed Analysis of Gaussian Elimination πðòmax(n,) ) We now combine the results from the previous sections to bound the smoothed precision needed in the application of Gaussian elimination without pivoting to obtain solutions to linear systems accurate to b bits. Theorem 5. (Smoothed precision of Gaussian elimination). ÓÖ n > e 4 Ð Ø A Ò n¹ ݹn Ñ ØÖ Ü ÓÖ Û A Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ /4º Ì Ò Ø ÜÔ Ø ÒÙÑ Ö Ó Ø Ó ÔÖ ÓÒ Ò ÖÝ ØÓ ÓÐÚ Ax = b ØÓ b Ø Ó ÙÖ Ý Ù Ò Ù Ò Ð Ñ Ò Ø ÓÒ Û Ø ÓÙØ Ô ÚÓØ Ò Ø ÑÓ Ø b+ ( ) ÐÓ n+3 ÐÓ + ÐÓ σ (+ nσ)+ ÐÓ ÐÓ n+6.83 Proof. By Wilkinson s theorem, we need the machine precision, ǫ mach, to satisfy 5 b nρ L (A)ρ U (A)κ(A)ǫ mach =.33+b+ÐÓ n+ðó (ρ L (A))+Ñ Ü(0, ÐÓ (ρ U (A)))+ÐÓ (κ(a)) ÐÓ (/ǫ mach ). We will apply Lemma C.6 to bound these log terms. Theorem 4. tells us that [ρ U (A) > +x] n(n+). π xσ 7

19 To put this inequality into a form to which Lemma C.6 may be applied, we set ( ) πσ y = x +, n(n+) to obtain [ρ U (A) > y] ( ) n(n+) + π σ y. By Lemma C.6, ( ) n(n+) [Ñ Ü(0, ÐÓ ρ U (A))] ÐÓ + π σ ÐÓ (n(n+)+σ ) π ( ÐÓ.0n ) ( + ÐÓ σ ÐÓ n+ðó ( σ ) +0.5, + ÐÓ e ) ( + ÐÓ σ ) ( ) e + ÐÓ π ( ) e + ÐÓ π where in the second-to-last inequality, we used the assumptions n e 4 and σ /. In the last inequality, we numerically computed ÐÓ (.0e/ π) < 0.5. Theorem 4.4 and Lemma C.6 imply ( ( [ÐÓ ρ L (A)] ÐÓ π n σ + ÐÒn+ ))+ÐÓ e π ÐÒn ( ( )) ( ) ÐÓ n+ðó σ + e ÐÒn + + ÐÓ π ÐÒn π ( ) ( ( )) ( ) e = ÐÓ n+ðó + ÐÓ σ ÐÒn+ÐÓ +σ + ÐÒn + ÐÓ π ÐÒn π using σ and n > e4, ÐÓ n+ðó ( σ ÐÓ n+ðó ( σ )+ ÐÓ ÐÓ n+ðó ( ) + ÐÓ ÐÓ n+.67, + ) ( ) e 6 + ÐÓ π π as ÐÓ (+/6 π)+ðó (e/ π) <.67. Theorem3.3andLemmaC.6, alongwiththeobservation that ÐÓ (.35e) <.68, imply [ ] A ÐÓ ( ) ÐÓ n+ðó σ 8

20 Finally, [ÐÓ ( A )] ÐÓ (+ nσ) follows from the well-known facts that the expectation of A A is at most nσ (c.f., [Seg00]) and that [ÐÓ (X)] ÐÓ [X] for every positive random variable X. Thus, the expected number of digits of precision needed is at most b+ ( ) ÐÓ n+3 ÐÓ + ÐÓ σ (+ nσ)+ ÐÓ ÐÓ n The following conjecture would further improve the coefficient of ÐÓ (/σ). Conjecture. Ä Ø A n¹ ݹn Ñ ØÖ Ü ÓÖ Û A Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò ÓÖ ÓÑ ÓÒ Ø ÒØ c Ò c º [ρ L (A)ρ U (A)κ(A) > x] nc ÐÓ c (x), xσ 6 Zero-preserving perturbations of symmetric matrices with diagonals Many matrices that occur in practice are symmetric and sparse. Moreover, many matrix algorithms take advantage of this structure. Thus, it is natural to study the smoothed analysis of algorithms under perturbations that respect symmetry and non-zero structure. In this section, we study the condition numbers and growth factors of Gaussian elimination without pivoting of symmetric matrices under perturbations that only alter their diagonal and non-zero entries. Definition 6. (Zero-preserving perturbations). Ä Ø T Ñ ØÖ Üº Ï Ò Ø Þ ÖÓ¹ ÔÖ ÖÚ Ò Ô ÖØÙÖ Ø ÓÒ Ó T Ó Ú Ö Ò σ ØÓ Ø Ñ ØÖ Ü T Ó Ø Ò Ý Ò Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ñ Ò ¼ Ò Ú Ö Ò σ ØÓ Ø ÒÓÒ¹Þ ÖÓ ÒØÖ Ó T º Throughout this section, when we express a symmetric matrix A as T +D+T T, we mean that T is lower-triangular with zeros on the diagonal and D is a diagonal matrix. By making a zeropreserving perturbation to T, we preserve the symmetry of the matrix. The main results of this section are that the smoothed condition number and growth factors of symmetric matrices under zero-preserving perturbations to T and diagonal perturbations to D have distributions similar those proved in Sections 3 and 4 for dense matrices under dense perturbations. 9

21 6. Bounding the condition number We begin by recalling that the singular values and vectors of symmetric matrices are the eigenvalues and eigenvectors. Lemma 6.. Ä Ø A = T + D + T T Ò Ö ØÖ ÖÝ n¹ ݹn ÝÑÑ ØÖ Ñ ØÖ Üº Ä Ø T Þ ÖÓ¹ ÔÖ ÖÚ Ò Ô ÖØÙÖ Ø ÓÒ Ó T Ó Ú Ö Ò σ Ð Ø G D ÓÒ Ð Ñ ØÖ Ü Ó Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò σ Ò Ñ Ò ¼ Ø Ø Ö Ò Ô Ò ÒØ Ó T Ò Ð Ø D = D + GD º Ì Ò ÓÖ A = T +D+T T [ ] A n 3/ x π xσ. Proof. By Proposition., T,G D [ (T +D+T T ) x ] Ñ Ü T [ ] ((T + D+T T )+G D ) x. G D The proof now follows from Lemma 6.3, taking T + D+T T as the base matrix. Lemma 6.3. Ä Ø A Ò Ö ØÖ ÖÝ n¹ ݹn ÝÑÑ ØÖ Ñ ØÖ Ü Ð Ø GD ÓÒ Ð Ñ ØÖ Ü Ó Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò σ Ò Ñ Ò ¼ Ò Ð Ø A = A+GD º Ì Ò [ ] A n 3/ x π xσ. Proof. Let x,...,x n be the diagonal entries of G D, and let g = n x i, and n Then, y,...,y n,g [ ( A+GD ) x] i= y i = x i g. [ ( = A+diag(y,...,y n )+gi) x] y,...,y n,g Ñ Ü y,...,y n g [ ( A+diag(y,...,y n )+gi) x where the last inequality follows from Proposition.. The proof now follows from Proposition 6.4 and Lemma 6.5. Proposition 6.4. Ä Ø X,...,X n Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò σ Û Ø Ñ Ò a,...,a n Ö Ô Ø Ú Ðݺ Ä Ø G = n X i, Ò n i= Y i = X i G. 0 ],

22 Ì Ò G Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò σ /n Û Ø Ñ Ò (/n) a i Ò Ô Ò ÒØ Ó Y,...,Y n º Lemma 6.5. Ä Ø A Ò Ö ØÖ ÖÝ n¹ ݹn ÝÑÑ ØÖ Ñ ØÖ Ü Ò Ð Ø G Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ñ Ò 0 Ò Ú Ö Ò σ /nº Ä Ø A = A+GIº Ì Ò A [ ] A n 3/ x π xσ. Proof. Let λ,...,λ n be the eigenvalues of A. Then, ( A+GI) = Ñ Ò λ i +G. i Thus, A [ ] [ A x = Ñ Ò λ i G < ] G i x i [ λ i G < ] G x i n π xσ n 3/ π xσ, where the second-to-last inequality follows from Lemma A. for R. As in Section 3, we can now prove: Theorem 6.6 (Condition number of symmetric matrices). Ä Ø A = T + D + T T Ò Ö ØÖ ÖÝ n¹ ݹn ÝÑÑ ØÖ Ñ ØÖ Ü Ø Ý Ò A nº Ä Ø σ Ð Ø T Þ ÖÓ¹ÔÖ ÖÚ Ò Ô ÖØÙÖ Ø ÓÒ Ó T Ó Ú Ö Ò σ Ð Ø G D ÓÒ Ð Ñ ØÖ Ü Ó Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò σ Ò Ñ Ò 0 Ø Ø Ö Ò Ô Ò ÒØ Ó T Ò Ð Ø D = D +GD º Ì Ò ÓÖ A = T +D+T T [κ(a) x] 6 π n 7/ xσ ( + ) ÐÒ(x)/9n Proof. As in the proof of Theorem 3., we can apply the techniques used in the proof of [DS0, Theorem II.7], to show [ A A n+k ] < e k /. The rest of the proof follows the outline of the proof of Theorem 3., using Lemma 6. instead of Theorem Bounding entries in U In this section, we will prove:

23 Theorem 6.7 (ρ U (A) of symmetric matrices). Ä Ø A = T + D+ T T Ò Ö ØÖ ÖÝ n¹ ݹn ÝÑÑ ØÖ Ñ ØÖ Ü Ø Ý Ò A º Ä Ø σ Ð Ø T Þ ÖÓ¹ÔÖ ÖÚ Ò Ô ÖØÙÖ Ø ÓÒ Ó T Ó Ú Ö Ò σ Ð Ø G D ÓÒ Ð Ñ ØÖ Ü Ó Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò σ Ò Ñ Ò 0 Ø Ø Ö Ò Ô Ò ÒØ Ó T Ò Ð Ø D = D+GD º Ì Ò ÓÖ A = T +D+T T [ρ U (A) > +x] n 3 7 πxσ Proof. We proceed as in the proof of Theorem 4.. For k between and n, we define u, a, b and C as in the proof of Theorem 4.. By (4.) u ( + A C T) b + k b T C + C k b. Hence ] u [ > +x A [ [ b ] C b > (k ) π xσ ] x k, by Lemmas 6. and C.4, +jσ (k ), where j is the number of non-zeros in b, π xσ k(k ). π xσ Applying a union bound over k, [ρ U (A) > x] πxσ n k= k(k ) 7 n 7/ π xσ. 6.3 Bounding entries in L As in Section 4., we derive a bound on the growth factor of L. As before, we will show that it is unlikely that A (k ) j,k is large while A (k ) k,k is small. However, our techniques must differ from those used in Section 4., as the proof in that section made critical use of the independence of A k,:(k ) and A :(k ),k. Theorem 6.8 (ρ L (A) of symmetric matrices). Ä Ø σ Ò n º Ä Ø A = T+ D+ T T Ò Ö ØÖ ÖÝ n¹ ݹn ÝÑÑ ØÖ Ñ ØÖ Ü Ø Ý Ò A º Ä Ø T Þ ÖÓ¹ÔÖ ÖÚ Ò Ô ÖØÙÖ Ø ÓÒ Ó T Ó Ú Ö Ò σ Ð Ø G D ÓÒ Ð Ñ ØÖ Ü Ó Ò Ô Ò ÒØ Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó

24 Ú Ö Ò σ Ò Ñ Ò 0 Ø Ø Ö Ò Ô Ò ÒØ Ó T Ò Ð Ø D = D+GD º Ä Ø A = T +D+T T º Ì Ò ( ) x πσ, [ρ L(A) > x] 3.n4 π xσ ÐÒ3/ e xσ. Proof. Using Lemma 6.9, we obtain for all k L(k+):n,k ] ( ) [ j > k : L j,k > x] [ > x 3.n π xσ ÐÒ3/ e xσ. Applying a union bound over the choices for k, we then have ( ) [ j,k : L j,k > x] 3.n3 π xσ ÐÒ3/ e xσ. The result now follows from the fact that L is at most n times the largest entry in L. Lemma 6.9. ÍÒ Ö Ø ÓÒ Ø ÓÒ Ó Ì ÓÖ Ñ º [ L(k+):n,k x πσ, ] > x 3.n Proof. We recall that xσ ÐÒ3/ ( ) π e xσ. L k+:n,k = A k+:n,k A k+:n,:k A :k,:k A :k,k A k,k A k,:k A :k,:k A :k,k Because of the symmetry of A, A k,:k is the same as A :k,k, so we can no longer use the proof technique that worked in Section 4.. Instead, we will bound the tails of the numerator and denominator separately, exploiting the fact that only the denominator depends upon A k,k. Consider the numerator first. Setting v = A :k,:k A :k,k, the numerator can be written A k+:n,:k ( v Let ). We will now prove that for all x /σ, [ A k+:n,:k A k+:n,:k A :k,:k ( ) ] ( v > x π c = n (+σ ) ÐÒ(xσ))+n. (6.) xσ +σ ÐÒ(xσ), (6.) which implies c cσ = ÐÒ(xσ). It suffices to prove (6.) for all x for which the right-hand side is less than. Given that x /σ, it suffices to consider x for which cx and xσ. 3

25 We use the parameter c to divide the probability as follows: [ ( ) ] v A k+:n,:k > x A k+:n,:k A :k,:k [( ) ] v > cx A :(k ),:k [ ( ) v + A k+:n,:k A k+:n,:k > ( ) v c ( ) v cx ] (6.3) (6.4) ( To evaluate (6.4), we note that once v is fixed, each component of A v ) k+:n,:k is a Gaussian random variable of variance ( v) σ and mean at most ( Ak+:n,:k v ) ( v). So, ( ) v A k+:n,:k > ( ) v c implies one of the Gaussian random variables differs from its mean by more than (/c )/σ times it standard deviation, and we can therefore apply Lemma A. and a union bound to derive (6.4) π ne ( c cσ ) c cσ n = πxσ ÐÒ(xσ). To bound (6.3), we note that Lemma 6. and Corollary C.5 imply [ ] A A :k,:k A n :k,k > y :(k ),:k πyσ, and so A :(k ),:k [( ) ] v > cx [ A A :(k ),:k n π(cx )σ n = π(cxσ( /cx)) n (+σ ÐÒ(xσ)) = π xσ( /cx) π :k,:k A :k,k n (+σ ÐÒ(xσ)), by cx. xσ ] > cx 4

26 So, [ A k+:n,:k A k+:n,:k A :k,:k ( ) ] ( v > x π ) n xσ ÐÒ(xσ) + n (+σ ÐÒ(xσ)) xσ +σ ) ÐÒ(xσ) +n ( n π xσ, (6.5) by the assumption xσ, which proves (6.). As for the denominator, we note that A k,k is independent of all other terms, and hence [ ] Ak,k A k,:k A :k,:k A :k,k < /x πxσ, (6.6) by Lemma A.. Applying Corollary C.3 with ( ) α = n +n π to combine (6.5) with (6.6), we derive the bound β = 4n σ γ = π π ( (( πxσ n +n+ +4 ) ) σ/3 n +n ÐÒ 3/( )) π/xσ n ( πxσ 3+4 )( σ/3 ÐÒ 3/( ) ) π/xσ + as σ. 7 Conclusions and open problems 7. Generality of results 3.n xσ ÐÒ3/( e π/xσ ), In this paper, we have presented bounds on the smoothed values of the condition number and growth factors assuming the input matrix is subjected to a slight Gaussian perturbation. We would like to point out here that our results can be extended to some other families of perturbations. With the exception of the proof of Theorem 3.3, the only properties of Gaussian random vectors that we used in Sections 3 and 4 are. there is a constant c for which the probability that a Gaussian random vector has distance less than ǫ to a hyperplane is at most cǫ, and 5

27 . it is exponentially unlikely that a Gaussian random vector lies far from its mean. Moreover, a result similar to Theorem 3.3 but with an extra factor of d could be proved using just fact. In fact, results of a character similar to ours would still hold if the second condition were reduced to a polynomial probability. Many other families of perturbations share these properties. For example, similar results would hold if we let A = A + U, where U is a matrix of variables independently uniformly chosen in [ σ,σ], or if A = A+SS, where the columns of SS are chosen uniformly among those vectors of norm at most σ. 7. Counter-Examples The results of sections 3 and 4 do not extend to zero-preserving perturbations for non-symmetric matrices. For example, the following matrix remains ill-conditioned under zero-preserving perturbations A symmetric matrix that remains ill-conditioned under zero-preserving perturbations that do not alter the diagonal can be obtained by locating the above matrix in the upper-right quadrant, and its transpose in the lower-left quadrant: The following matrix maintains large growth factor under zero-preserving perturbations, re- 6

28 gardless of whether partial pivoting or no pivoting is used These examples can be easily normalized to so that their -norms are equal to. 7.3 Open Problems Questions that naturally follow from this work are: What is the probability that the perturbation of an arbitrary matrix has large growth factors under Gaussian elimination with partial pivoting? What is the probability that the perturbation of an arbitrary matrix has large growth factors under Gaussian elimination with complete pivoting? Can zero-preserving perturbations of symmetric matrices have large growth factors under partial pivoting or under complete pivoting? Can zero-preserving perturbations of arbitrary matrices have large growth factors under complete pivoting? For the first question, we point out that experimental data of Trefethen and Bau [TB97, p. 68] suggest that the probability that the perturbation of an arbitrary matrix has large growth factor under partial pivoting may be exponentially smaller than without pivoting. This leads us to conjecture: Conjecture 3. Ä Ø A Ò n¹ ݹn Ñ ØÖ Ü ÓÖ Û A Ò Ð Ø A Ù Ò Ô Ö¹ ØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ä Ø U Ø ÙÔÔ Ö¹ØÖ Ò ÙÐ Ö Ñ ØÖ Ü Ó Ø Ò ÖÓÑ Ø Ä͹ ØÓÖ Þ Ø ÓÒ Ó A Û Ø Ô ÖØ Ð Ô ÚÓØ Ò º Ì Ö Ü Ø ÓÐÙØ ÓÒ Ø ÒØ k k Ò α ÓÖ Û [ U Ñ Ü / A Ñ Ü > x+] n k e αxk σ Finally, we ask whether similar analyses can be performed for other algorithms of Numerical Analysis. One might start by extending Smale s program by analyzing the smoothed values of other condition numbers. 7

29 7.4 Recent Progress Since the announcement of our result, Wschebor [Wsc04] improved the smoothed bound on the condition number. Theorem 7. (Wschebor). Ä Ø A Ò n n Ñ ØÖ Ü Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò [κ(a) x] n (5+ x 4 πn +7 4 A (+ÐÓ n) ) / σ n When A n, his result implies ( ) n ÐÓ n [κ(a) x] O. xσ We conjecture Conjecture 4. Ä Ø A Ò n n Ñ ØÖ Ü Ø Ý Ò A n Ò Ð Ø A Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó A Ó Ú Ö Ò σ º Ì Ò ( n [κ(a) x] O. xσ) 8 Acknowledgments We thank Alan Edelman for suggesting the name smoothed analysis, for suggesting we examine growth factors, and for his continuing support of our efforts. We thank Juan Cuesta and Mario Wschebor for pointing out some mistakes in an early draft of this paper. We thank Felipe Cucker for bringing Wschebor s paper [Wsc04] to our attention. Finally, we thank the referees for their extraordinary efforts and many helpful suggestions. References [ABB + 99] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov, and D. Sorensen. LAPACK Users Guide, Third Edition. SIAM, Philadelphia, 999. [AS64] Milton Abramowitz and Irene A. Stegun, editors. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, volume 55 of Applied mathematics series. U. S. Department of Commerce, Washington, DC, USA, 964. Tenth printing, with corrections (December 97). 8

30 [BD0] [Blu89] Avrim Blum and John Dunagan. Smoothed analysis of the perceptron algorithm for linear programming. In SODA 0, pages , 00. Lenore Blum. Lectures on a theory of computation and complexity over the reals (or an arbitrary ring). In Erica Jen, editor, The Proceedings of the 989 Complex Systems Summer School, Santa Fe, New Mexico, volume, pages 47, June 989. [Dem88] James Demmel. The probability that a numerical analysis problem is difficult. Math. Comp., pages , 988. [Dem97] James Demmel. Applied Numerical Linear Algebra. SIAM, 997. [DS0] K. R. Davidson and S. J. Szarek. In W. B. Johnson and J. Lindenstrauss, editors, Handbook on the Geometry of Banach spaces, chapter Local operator theory, random matrices, and Banach spaces, pages Elsevier Science, 00. Smoothed anal- available at [DST0] John Dunagan, Daniel A. Spielman, and Shang-Hua Teng. ysis of renegar s condition number for linear programming. spielman/smoothedanalysis, 00. [Ede88] [Ede9] [GL83] Alan Edelman. Eigenvalues and condition numbers of random matrices. SIAM J. Matrix Anal. Appl., 9(4): , 988. Alan Edelman. Eigenvalue roulette and random test matrices. In Marc S. Moonen, Gene H. Golub, and Bart L. R. De Moor, editors, Linear Algebra for Large Scale and Real-Time Applications, NATO ASI Series, pages Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins Series in the Mathematical Sciences. The Johns Hopkins University Press and North Oxford Academic, Baltimore, MD, USA and Oxford, England, 983. [GLS9] Martin Grotschel, Laszlo Lovasz, and Alexander Schrijver. Geometric Algorithms and Combinatorial Optimization. Springer-Verlag, 99. [Hig90] Nick Higham. How accurate is gaussian elimination? In Numerical Analysis 989, Proceedings of the 3th Dundee Conference, volume 8 of Pitman Research Notes in Mathematics, pages 37 54, 990. [JKB95] N. Johnson, S. Kotz, and N. Balakrishnan. Continuous Univariate Distributions, volume. Wiley-Interscience, 995. [KJ8] Samuel Kotz and Norman L. Johnson, editors. Encyclopedia of Statistical Sciences, volume 6. John Wiley & Sons, 98. 9

31 [LT9] [Ren95] [Seg00] Michel Ledoux and Michel Talagrand. Probability in Banach Spaces. Springer-Verlag, 99. J. Renegar. Incorporating condition measures into the complexity theory of linear programming. SIAM J. Optim., 5(3):506 54, 995. Yoav Seginer. The expected norm of random matrices. Combinatorics, Probability and Computing, 9:49 66, 000. [Sma97] Steve Smale. Complexity theory and numerical analysis. Acta Numerica, pages 53 55, 997. [Smo] [ST03] [ST04] spielman/smoothedanalysis. Daniel Spielman and Shang-Hua Teng. Smoothed analysis of termination of linear programming algorithms. Mathematical Programming, Series B, 97: , 003. Daniel A. Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. J. ACM, 5(3): , 004. [TB97] L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, Philadelphia, PA, 997. [TS90] [Wil6] Lloyd N. Trefethen and Robert S. Schreiber. Average-case stability of Gaussian elimination. SIAM Journal on Matrix Analysis and Applications, (3): , 990. J. H. Wilkinson. Error analysis of direct methods of matrix inversion. J. Assoc. Comput. Mach., 8:6 330, 96. [Wsc04] M. Wschebor. Smoothed analysis of κ(a). J. of Complexity, 0():97 07, February 004. [YC97] Man-Chung Yeung and Tony F. Chan. Probabilistic analysis of gaussian elimination without pivoting. SIAM J. Matrix Anal. Appl., 8():499 57, 997. A Gaussian random variables Lemma A.. Ä Ø X ÙÒ Ú Ö Ø Ù Ò Ö Ò ÓÑ Ú Ö Ð Û Ø Ñ Ò 0 Ò Ø Ò Ö Ú Ø ÓÒ º Ì Ò ÓÖ ÐÐ k [X k] e k. π k 30

32 Proof. We have [X k] = π k e x dx putting t = x, = e t dt π k t e t π k k dt = e k. π k Lemma A.. Ä Ø x d¹ Ñ Ò ÓÒ Ð Ù Ò Ö Ò ÓÑ Ú ØÓÖ Ó Ú Ö Ò σ Ð Ø t ÙÒ Ø Ú ØÓÖ Ò Ð Ø λ Ö Ðº Ì Ò ] t [ T ǫ x λ ǫ πσ. Lemma A.3. Ä Ø g,...,g n Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ñ Ò ¼ Ò Ú Ö Ò º Ì Ò [ ] Ñ Ü g i ÐÒ(Ñ Ü(n,))+. i π ÐÒ(Ñ Ü(n,)) Proof. For any a, [ ] Ñ Ü g i = i t=0 a [ t=0 a+ Ñ Ü dt + a i = a+ n π a a+ n π a ] g i t dt a n [ g t] dt n e t dt π t ( t d e t a = a+ n π a e a. e t d t ) ( ) t (applying Lemma A.,) 3

33 Setting a = ÐÒ(Ñ Ü(n,)), which is greater than for all n, we obtain the following upper bound on the expectation: ÐÒ(Ñ Ü(n,))+ n π ÐÒ(Ñ Ü(n,)) Ñ Ü(n,) ÐÒ(Ñ Ü(n,))+ π ÐÒ(Ñ Ü(n,)). Lemma A.4 (Expectation of reciprocal of the -norm of a Gaussian vector). Ä Ø a Ò Ö ØÖ ÖÝ ÓÐÙÑÒ Ú ØÓÖ Ò R n ÓÖ n º Ä Ø a Ù Ò Ô ÖØÙÖ Ø ÓÒ Ó a Ó Ú Ö Ò σ º Ì Ò [ ] a nσ Proof. Let a = (a,...,a n ). It is clear that the expectation of / a is maximized if a = 0, so we will make this assumption. Without loss of generality, we also assume σ =. For general σ, we can simply scale the bound by the factor /σ. Recall that the Laplace transform of a positive random variable X is defined by [ L[X](t) = X e tx] and the expectation of the reciprocal of a random variable is simply the integral of its Laplace transform. Let X be the absolute value of a standard normal random variable. The Laplace transform of X is given by L[X](t) = e tx e x dx π 0 = e (x+t) dx = π e t π e t = e t Ö 0 t e x dx ( t ). Taking second derivatives, and applying the inequality (c.f. [AS64, 6..3]) π t ) we find that e Ö ( t t is convex. e x dx e x π x+/x, 3

34 We now set a constant c =.4 and set α to satisfy ( ) c/π = e c/π (c/π) Ö. α Numerically, we find) that α.9857 <. As e Ö ( t t is convex, we have the upper bound ( ) e t t Ö t α, for 0 t c/π. For t > c/π, we apply the upper bound which follows from Lemma A.. We now have [ ] = a 0 c/π 0 ( ) e t t Ö π t, ( e t Ö (t/ )) n dt α n+ + π < n+ + n, ( α) t n dt + (/c) (n )/ n (/c) (n )/ π n c/π ( ) n dt π t for n. To verify this last equality, one can multiply through by (n+)(n ) to obtain π (n+)(/c)(n )/ 4, which one can verify by taking the derivitive of the left-hand side to find the point where it is maximized, n = (+ÐÒ(5/6))/ ÐÒ(6/5). 33

35 B Random point on a sphere Lemma B.. Ä Ø d Ò Ð Ø (u,...,u d ) ÙÒ Ø Ú ØÓÖ Ó Ò ÙÒ ÓÖÑÐÝ Ø Ö Ò ÓÑ Ò R d º Ì Ò ÓÖ c [ ] c u [ G c ], d Û Ö G Ù Ò Ö Ò ÓÑ Ú Ö Ð Ó Ú Ö Ò Ò Ñ Ò 0º Proof. We may obtain a random unit vector by choosing d independent Gaussian random variables of variance and mean 0, x,...,x d, and setting u i = x i. x + +x d We have We now note that [ u c ] [ x = d x c ] + +x d d (d )x = [ x (d )c + +x d c d (d )x ] [ x c, since c. + +x d t d = (d )x x + +x d ] is a random variable distributed according to the t-distribution with d degrees of freedom. The lemma now follows from the fact (c.f. [JKB95, Chapter 8, Section ] or [AS64, 6.7.5]) that, for c > 0, [ t d > c ] [ G > c ], and that the distributions of t d and G are symmetric about the origin. C Combination Lemmas Lemma C.. Ä Ø A Ò B ØÛÓ ÔÓ Ø Ú Ö Ò ÓÑ Ú Ö Ð º ÙÑ ½º [A x] f(x)º ¾º [B x A] g(x)º 34

36 Û Ö g ÑÓÒÓØÓÒ ÐÐÝ Ö Ò Ò Ð Ñ x g(x) = 0º Ì Ò [AB x] 0 ( x f ( g t) (t))dt Proof. Let µ A denote the probability measure associated with A. We have [AB x] = 0 0 [B x/s A] dµ A(s) B ( x g dµ A (s), s) integrating by parts, setting t = x/s = = [A s] d ( x ) ds g ds s f(s) d ( x ) ds g ds, s ( x f ( g t) (t))dt. Corollary C. (linear-linear). Ä Ø A Ò B ØÛÓ ÔÓ Ø Ú Ö Ò ÓÑ Ú Ö Ð º ÙÑ ½º [A x] α x Ò ¾º [B x A] β x ÓÖ ÓÑ α,β > 0º Ì Ò [AB x] αβ x ( ( ( ))) x +Ñ Ü 0, ÐÒ αβ Proof. As the probability of an event can be at most, ( α ) [A x] Ñ Ò x, = f(x), and ( ) β [B x] Ñ Ò x, = g(x). Applying Lemma C. while observing 35

37 g (t) = 0 for t [0,β], and f(x/t) = for t x/α, we obtain [AB x] β 0 = Ñ Ü = αβ x ( αt 0dt + Ñ Ü 0, x ( 0, αβ x/α x ( +Ñ Ü β x/α β ) dt + αβ t x ( 0, ÐÒ ( x αβ αt x ))), ) β t dt β + x/α t dt where the Ñ Ü appears in case x/α < β. Corollary C.3. Ä Ø A Ò B ØÛÓ ÔÓ Ø Ú Ö Ò ÓÑ Ú Ö Ð º Á ( ) ½º x /σ [A x] Ñ Ò Ò ¾º [B x A] γ xσ, α+β ÐÒxσ σx ÓÖ ÓÑ α Ò β,γ,σ > 0 Ø Ò x γ/σ, [AB x] αγ ( ( ( )) β xσ xσ + )ln 3α + 3/. γ Proof. Define f and g by f(x) = g(x) = Applying Lemma C. while observing g (t) = 0 for t [ 0, γ σ], and f(x/t) = for t xσ/α, { for x α σ α+β ÐÒxσ xσ for x > α σ { for x γ σ γ xσ for x > γ σ 36

38 we obtain [AB x] = xσ/α γ/σ xσ/α γ/σ α+β ÐÒ(xσ/t) xσ/t γ t σ dt + α+β ÐÒ(xσ/t) γ αγ xσ dt + t xσ xσ/α γ σt dt (substituting s = ÐÒ(xσ/t),t = xσe s, which is defined as x γ/σ, ) as α. = ÐÒα α+βs γ ÐÒ(xσ /γ) xσ )ds + αγ xσe sxσ( se s xσ ÐÒ(xσ /γ) = γ xσ = αγ xσ αγ xσ ÐÒα ( +ÐÒ ( + s(α+βs)ds + αγ xσ ( ) xσ + β ( xσ (ÐÒ 3/ αγ 3α γ ( ( )) β xσ )ln 3α + 3/, γ ) )) ÐÒ 3/ α Lemma C.4 (linear-bounded expectation). Ä Ø A B Ò C ÔÓ Ø Ú Ö Ò ÓÑ Ú Ö Ð Ù Ø Ø [A x] α x, ÓÖ ÓÑ α > 0 Ò Ì Ò A, [B x A] [C x]. [AB x] α x [C]. Proof. Let g(x) be the distribution function of C. By Lemma C., we have ( ) αt [AB x] ( ( g) (t)) dt x = α x 0 0 = α x [C]. t(g (t)) dt 37