Notes on Numerical Analysis

Transcription

1 Notes on Numerical Analysis Alejandro Cantarero This set of notes covers topics that most commonly show up on the Numerical Analysis qualifying exam in the Mathematics department at UCLA. Each section covers a specific area of numerical analysis. At the end of each section, we provide a list of key theorems. If the proof of the theorem is directly relevant to the qualifying exam, it will be included in this document. Otherwise, we simply include the statement of the theorem. At the end of each section, we include a listing of problems on old qualifying exams that fall under the given topic. 1

2 Contents 1 Root Finding Bisection Method Fixed Point Iteration Newton s Method Solving Systems of Non-Linear Equations Qual Problems Interpolation Theory 5.1 Polynomial Interpolation Lagrange Polynomials Newton Divided Differences Hermite Interpolation Trigonometric Interpolation Key Theorems Qual Problems Numerical Differentiation Method of Undetermined Coefficients Common Formulae Richardson Extrapolation Qual Problems Numerical Integration Newton-Cotes Formulas Composite Integration Gaussian Quadrature Key Theorems Qual Problems Iterative Methods Relaxation Methods Key Theorems Qual Problems Direct Methods for Solving Linear Systems LU Factorization Tridiagonal Systems Qual Problems Approximation of Functions Least Squares Qual Problems Numerical Methods for Ordinary Differential Equations One Step Methods Stability Analysis Common Methods and Their Properties Linear Multistep Methods Predictor-Corrector Methods Qual Problems

3 9 Finite Differences for PDEs Order of Accuracy Using Symbols Stability Analysis Well-Posedness of Equations Hyperbolic PDEs Common Methods Parabolic PDEs Common Methods Variable Diffusion Coefficients Key Theorems Qual Problems Finite Element Methods 10.1 Weak Formulations Well-Posedness of the Weak Form Formulating a Finite Element Approximation The Stiffness Matrix Error Estimates Key Theorems Qual Problems References 8 3

4 1 Root Finding In this section, we discuss methods used to find the roots of an equation f(x) = 0. (1.1) The following definition will give us some information on how fast a rootfinding method converges. Definition 1.1. A sequence of iterates, {x n } converges with order p 1 to α if for some c > 0. α x n+1 c α x n p (1.) 1.1 Bisection Method We begin by assuming that f(x) is continuous on [a, b] and f(a)f(b) < 0 (this guarantees the existence of a solution to (1.1) ). The bisection method will also require a tolerance, ɛ. The method is then given by Algorithm 1 Bisection Method Require: f, a, b, ɛ repeat c (a + b)/ 3: if sign(f(b)) sign(f(c)) 0 then a c else 6: b c end if until b c ɛ 9: return c Note that the bisection algorithm will always converge to a root in [a, b]. Let α be the root that the bisection method is converging to and let c n be the nth value of c produced by the method. Then we have Note that in definition 1.1, if p = 1 we can write α c n So the bisection method has linear convergence with rate 1, given by c. 1. Fixed Point Iteration (b a) n. (1.3) α x n c n α x 0. (1.4) Now we consider solving a problem of the form x = g(x) where the fixed point iteration is given by x n+1 = g(x n ). (1.5) Note that the rootfinding problem, equation (1.1), can be reformulated as a fixed point problem. Lemma 1.1. (Existence) Let g(x) be continuous on [a, b] and assume that g([a, b]) [a, b]. Then x = g(x) has at least one solution in [a, b]. Lemma 1.. (Uniqueness I) Let g(x) be continuous on [a, b] and assume that g([a, b]) [a, b]. Further, assume there is a λ (0, 1) such that g(x) g(y) λ x y, x, y [a, b]. (1.6) 4

5 Then x = g(x) has a unique solution p [a, b] and the iterates will converge for any x 0 [a, b] with p x n λn 1 λ x 1 x 0. (1.7) Theorem 1.1. Let g(x) C 1 ([a, b]) such that g([a, b]) [a, b] and max a x b g (x) = λ < 1 then (i) x = g(x) has a unique solution p [a, b]. (ii) The iteration (1.5) converges for any x 0 [a, b]. (iii) 1.3 Newton s Method Newton s method is given by p x n λ n p x 0 λn 1 λ x 1 x 0 (1.8) p x n+1 lim = g (p) n p x n (1.9) x n+1 = x n f(x n) f (x n ). (1.10) We can derive Newton s method by considering a Taylor expansion of f about x n. f(x) = f(x n ) + (x x n )f (x n ) + (x x n) f (ξ) (1.11) where ξ between x and x n. Let p be our root. Then we find 0 = f(x n ) + (p x n )f (x n ) + (p x n) f (ξ) (1.1) p = x n f(x n) f (x n ) (p x n) f (ξ) f (x n ). (1.13) Dropping the error term, we find an approximation to the root, p, that is indeed Newton s method. 1.4 Solving Systems of Non-Linear Equations Newton s method for systems of non-linear equations can be derived in a similar method as shown in 1.3. Let F : R n R n and x, x n, p R n. If f i is the i th equation of F, we can write the Taylor expansion for F about x n as f i (x) = f i (x n ) + n (x j x j,n ) f i(x n ) + H.O.T. (1.14) x j j=1 If p is a root of f i and G is the Jacobian matrix for F, then if we drop the higher order terms in (1.14) we find 0 = F (x n ) + G(x n )(p x n ) (1.15) G(x n )p = F (x n ) G(x n )x n (1.16) p = x n G 1 (x n )F (x n ). (1.17) 5

6 1.5 Qual Problems Winter 003, #3 Fall 004, #1 Fall 005, #1 Bisection Winter 005, #1 Fall 006, #1 Newton s Method Fall 000, # Winter 00, #1 Spring 00, #1 Fixed Point Iteration Spring 001, # Fall 001, #1 Winter 004, #1 Spring 006, #1 Interpolation Theory Given a set of data points, {(x i, y i )} n i=1, select a function f(x) from some class of functions such that y i = f(x i ) for i = 1,..., n..1 Polynomial Interpolation For these problems, we choose our class of functions to be polynomials. interpolation problems found on the numerical qualifying exams. These are the most common.1.1 Lagrange Polynomials A simple formula for constructing interpolating polynomials is given by Lagrange s formula. p n (x) =.1. Newton Divided Differences n i=0 y i l i (x), l i (x) = j i x x j x i x j (.1) Newton divided differences are useful because they provide an easy method for generating interpolating polynomials. The polynomial is computed from where and p n (x) = p n 1 (x) + (x x 0 ) (x x n 1 )f[x 0,..., x n ] (.) p 0 (x) = f(x 0 ) (.3) f[x 0, x 1,..., x n ] = f[x 1,..., x n ] f[x 0,..., x n 1 ] x n x 0. (.4) To compute interpolating polynomials using divided differences, we build the table shown below. 6

7 x i f(x i ) f[x i, x i+1 ] f[x i, x i+1, x i+ ] x 0 f 0 f[x 0, x 1 ] f[x 0, x 1, x ] x 1 f 1 f[x 1, x ] f[x 1, x, x 3 ] x f f[x, x 3 ] f[x, x 3, x 4 ] x 3 f 3 f[x 3, x 4 ] x 4 f Now let s look at the error in the interpolating polynomial. Let t R be distinct from the node points x 0, x 1,..., x n. Using divided differences to interpolate f(x) at the node points and t, we find p n+1 (x) = p n (x) + (x x 0 ) (x x n )f[x 0,..., x n, t]. (.5) We have that p n+1 (t) = f(t), so let x = t and we find f(t) p n (t) = (t x 0 ) (t x n )f[x 0,..., x n, t]. (.6) From the formula given in Theorem., we can conclude that.1.3 Hermite Interpolation f[x 0,..., x n, t] = f (n+1) (ξ) (n + 1)!. (.7) Hermite polynomials are used when we wish to interpolate both the function (p(x) interpolates f(x)) and its first derivative (p (x) interpolates f (x)). The general problem here is given by: Find a polynomial p(x) such that p (i) (x 1 ) = y (i) 1 i = 0, 1,..., α 1 1. (.8) p (i) (x n ) = y (i) n i = 0, 1,..., α n 1 Here the y (i) j are given data and α j are the number of conditions imposed on p(x) at x j for j = 1,..., n. If we define N = α α n then we have a unique polynomial of degree at most N 1 that satisfies the above equations. To show this, we basically follow the proof for Theorem.1, noting that if a polynomial q(x) has its first α derivatives equal to zero at x j, where x j is a root, then x j is a root of order α 1 and we can write. Trigonometric Interpolation [Incomplete] q(x) = r(x)(x x j ) α 1. (.9) 7

8 .3 Key Theorems Theorem.1. Given n + 1 distinct points x 0,..., x n and associated function values y 0,..., y n, there is a polynomial, p(x), of degree at most n that interpolates y i at x i. Further, this polynomial is unique. Proof. The existence of a polynomial, p(x), of degree at most n is given by the construction in equation (.1). To show uniqueness, let p 1 (x) and p (x) be such that p j (x i ) = y i for j = 1, and i = 0,..., n (that is p 1 and p are both interpolating polynomials). Let r(x) = p 1 (x) p (x). Clearly, r(x) has degree less than or equal to n. Further, r(x i ) = p 1 (x i ) p (x i ) = y i y i = 0. So, r(x) has n + 1 zeros which implies that r(x) 0 p 1 (x) = p (x) and hence, the interpolating polynomial is unique. Theorem.. Let x 0, x 1,..., x n be distinct real numbers contained in the interval I and let f C n+1 (I). Then, for each x I, there exists ξ I such that.4 Qual Problems Polynomial Interpolation Spring 00, # Fall 003, #3 Winter 004, #3 Spring 006, #3 f(x) p n (x) = (x x 0) (x x n ) f (n+1) (ξ). (.10) (n + 1)! Trigonometric Interpolation Fall 001, # 3 Numerical Differentiation Numerical differentiation formulas can be found by first constructing an interpolating polynomial p n (x) for your function f(x) at a set of node points and then computing the derivative of p n (x). Recall from.1.1, the interpolating polynomial is given by Now n p n (x) = f(x i )l i (x). (3.1) i=0 Further, recall the formula for the error of the interpolating polynomial n p n(x) = f(x i )l i(x) D h f(x). (3.) i=0 Then f(x) p n (x) = ψ n (x)f[x 0,..., x n, x]. (3.3) f (x) D h f(x) = ψ n(x)f[x 0,..., x n, x] + ψ n (x)f[x 0,..., x n, x, x] (3.4) = ψ n(x) f (n+1) (ξ 1 ) (n + 1)! + ψ n (x) f (n+) (ξ ) (n + )! To obtain equations for higher order derivatives, we simply continue to differentiate the above equations. To find higher order formulas, we use a higher order interpolating polynomial. (3.5) 8

9 3.1 Method of Undetermined Coefficients This method provides both a simple way to derive formulas for numerical differentiation as well as a simple method for finding the error for a difference formula. The main idea of this method is to Taylor expand all terms about the node at which the derivative is being evaluated. We illustrate with an example. Say we wish to find a formula to approximate f (x) using function values at x h, x, and x + h. Then, we write our method as D () h f(x) = c 1f(x + h) + c f(x) + c 3 f(x h). (3.6) Now, we need Taylor expansions for f(x + h) and f(x h), given by f(x ± h) = f(x) ± hf (x) + h f (x) ± h3 6 f (3) (x) + h4 4 f (4) (ξ ± ) (3.7) with ξ ± [x ± h, x]. Plugging these expansions into (3.6) and grouping terms by powers of h, we find c 1 f(x + h)+c f(x) + c 3 f(x h) = (c 1 + c + c 3 )f(x) + (c 1 c 3 )hf(x) + (c 1 + c 3 ) h f (x) + (c 1 c 3 ) h3 6 f (3) (x) Note that in order to obtain f (x), we need Solving for the coefficients, we find that + h4 4 [c 1f (4) (ξ + ) + c f (4) (ξ )]. (3.8) c 1 + c + c 3 = 0, c 1 c 3 = 0, c 1 + c 3 = h. (3.9) c 1 = 1 h, c = h, c 3 = 1 h. (3.10) Using the Taylor expansions and the coefficients we found, we can find the error, with ξ [x h, x + h]. f (x) D () h f(x) = h 1 f (4) (ξ) (3.11) 3. Common Formulae In this section, we give some of the most common finite difference equations. Table 1 has the equations for one-sided differences and Table lists equations for centered differences. 9

10 Table 1: Weights for common one-sided finite differences. The derivative is at x = 0 and the nodes are numbered. Order of Nodes Accurarcy st Derivative nd Derivative rd Derivative Table : Weights for common centered finite differences. The derivative is at x = 0 and the nodes are numbered. Order of Nodes Accurarcy st Derivative nd Derivative rd Derivative Richardson Extrapolation Richardson extrapolation can be used any time where we have a formula N(h) that approximates an unknown value, M, where the truncation error has the form M N(h) = K 1 h + K h + K 3 h 3 +, (3.1) K i constants. The idea here is simple. Find a way to combine lower-order approximations in order to obtain a formula with a higher-order truncation error. This process is most easily illustrated with a simple example. Consider the approximation of M given by and M = N(h) + K 1 h + K h + K 3 h 3 + (3.13) M = N ( ) h h + K 1 + K h 4 + K h (3.14) Now, look at (3.14) (3.13). Letting N 1 (h) N(h), we find ( ) ( ( ) ) ( ) h h h M = N 1 + N 1 N 1 (h) + K h +. (3.15) 10

11 If we define N (h) = N 1 ( h ) ( ( ) ) h + N 1 N 1 (h) (3.16) we now have an O(h ) approximation for M given by N (h). This process can be continued to generate an approximation of any order. In general, for j =, 3,... m we have an O(h j ) approximation given by ( ) h N j (h) = N j 1 + N j 1(h/) N j 1 (h) j 1. (3.17) Qual Problems Fall 000, #1 Spring 001, #3 Winter 00, # Winter 003, #1 Winter 004, # Fall 005, # Extrapolation Spring 00, #3 4 Numerical Integration (Incomplete) 4.1 Newton-Cotes Formulas 4. Composite Integration 4.3 Gaussian Quadrature Gauss-Legendre Linear change of variables 4.4 Key Theorems 4.5 Qual Problems b Gaussian Quadrature Spring 001, #1 (Gauss-Legendre) Fall 004, # Fall 006, # (Gauss-Laguerre) Newton-Cotes Fall 00, #1 (Composite) Fall 003, #1 (Composite) Fall 00, # (Extrapolation) a ( ) b a 1 f(t)dt = f 1 ( a + b + x(b a) ) dx (4.1) 11

12 5 Iterative Methods In this section, we are interested in solving Ax = b by generating a sequence of approximate solutions, {x (k) }, that converge to the solution x. A few useful quantities for anaylizing the convergence of iterative methods include the residual, r = b Aˆx where ˆx is a numerical approximation to the solution x, and the condition number of a matrix, cond(a) = A A Relaxation Methods These methods depend on a splitting, A = M N, of the matrix A. The iteration is given by Mx (k+1) = Nx (k) + b. (5.1) We start by writting A = D + U + L where D, U, and L are the diagonal, strictly upper triangular, and strictly lower triangular parts of the matrix A. The most common splittings are: Jacobi M J = D N J = (L + U) Gauss-Seidel M G = (D + L) N G = U Successive Over-Relaxation M ω = D + ωl N ω = (1 ω)d ωu Note that for SOR we have the addition of the relaxation parameter ω R. characterization of these methods which may provide some insight. We also have a different Jacobi Gauss-Seidel SOR x (k+1) i = 1 b i a ii x (k+1) i = 1 b i a ii x (k+1) i = 1 ω a ii i 1 j=1 i 1 j=1 i 1 b i j=1 a ij x (k) j a ij x (k+1) j a ij x (k+1) j n j=i+1 n j=i+1 n j=i+1 a ij x (k) j a ij x (k) j a ij x (k) j (5.) (5.3) +(1 ω)x (k) i (5.4) Note that the difference between Jacobi and the other methods is simply using the most recently available values for the x i Key Theorems For these theorems, we assume that b R n, A = M N R n n, and A, M are non-singular. The spectral radius of A is denoted by ρ(a). Theorem 5.1. If ρ(m 1 N) < 1 then the iterates x (k) defined by (5.1) converge to x = A 1 b for any initial guess, x (0). Proof. We start by looking at the error found by subtracting the iterative scheme from the actual solution, M(x x (k+1) ) = N(x x (k) ) + b b Me (k+1) = Ne (k) e (k+1) = M 1 Ne (k). 1

13 Now, consider e (k+1) = (M 1 N)e (k) = (M 1 N) k+1 e (0) M 1 N k+1 e (0). If ρ(m 1 N) < 1 then M 1 N k 0 as k and so e (k) 0 and the method converges. Theorem 5.. If A is symmetric positive definite, then the Gauss-Seidel iteration converges for any x (0). 5. Qual Problems Relaxation Methods Fall 001, #3 Fall 00, #3 Fall 003, # Winter 005, #3 Error Analysis Spring 006, # 6 Direct Methods for Solving Linear Systems Here, we are again interested in solving linear systems Ax = b. 6.1 LU Factorization Often, a matrix can be factored into the form A = LU (6.1) where L is a lower triangular matrix and U is an upper triangular matrix. This is commonly done to solve linear systems numerically. Below, we give conditions on the existence of this factorization. Theorem 6.1. A R n n has an LU factorization if det(a(1 : k, 1 : k)) 0 for k = 1,..., n 1. If the LU factorization exists and A is nonsingular, then the LU factorization is unique and det(a) = u 11 u nn. Theorem 6.. If A T is strictly diagonally dominant, then A has an LU factorization and l ij 1. We then solve the system of equations first with a forward substitution step, Ly = b, and then backsubstitution Ux = y. 6. Tridiagonal Systems Special algorithms exist for solving tridiagonal systems in O(n) operations. As before, we will perform an LU factorization of the matrix A and are again given the choice of which matrix has ones on the diagonal. The two factorizations, A = LU, are illustrated below. 13

14 a 1 c 1 1 e 1 c 1 b a c d = e c b n 1 a n 1 c n e n 1 c n 1 b n a n d n 1 e n α 1 1 γ b α = γ γ n 1 (6.) (6.3) b n α n For each of these factorizations, we can obtain a recurrence relation to determine the unknown values by simply multiplying the L and U matrices together. Note that e 1 and α 1 must be handled differently. The resulting relations for (6.) are then e 1 = a 1, d i = e i 1 b i, e i = a i d i c i 1 (6.4) for i =,..., n. Now we need to solve LUx = f by solving Ly = f and then Ux = y. In this case, we end up with the following equations y 1 = f 1, y i = f i d i y i 1 (6.5) x n = y n e n, x j = y j c j x j+1 e j (6.6) for i =,... n and j = n 1,..., 1. Note the operation counts in this problem. We have 3n 3 floating point operations in finding L and U. The forward solve contains other n operations and 4n 4 operations for the back-substitution. Note that we have obtained a solution using order n floating point operations. 6.3 Qual Problems LU Factorization Fall 000, #3 Tri-Diagonal Systems Winter 00, #3 Fall 005, #3 7 Approximation of Functions (Incomplete) 7.1 Least Squares Here we are given a set of data points {(x i, y i ) i = 1,..., n} and wish to fit some function f to the data such that it minimizes E = ( 1 n ) 1 n (y i f(x i )). (7.1) i=1 14

15 We are only going to consider the case where f = P n (x), a polynomial of degree n. To find this minimum, we need to compute the normal equations, found by differentiating (7.1) with respect to each of the unknown coefficients of P n (x) and setting these equal to zero. Normal equations 7. Qual Problems Winter 003, # Fall 006, #3 A T Ax = A T b (7.) 8 Numerical Methods for Ordinary Differential Equations In this section we look at methods for solving y (t) = f(t, y(t)), y(t 0 ) = y 0. (8.1) The techniques developed in this section can also be applied to systems of ODEs, y = f(t, y), y(x 0 ) = y 0. (8.) This is of particular interest since we can easily convert higher order equations into systems of first order equations, as seen in the following example: Now let y (m) = f(t, y, y,..., y (m 1) ) (8.3) y(t 0 ) = y 0,..., y (m 1) (t 0 ) = y (m 1) 0 (8.4) Then we have y 1 = y, y = y,..., y m = y (m 1). (8.5) y 1 = y y 1 (t 0 ) = y 0 y = y 3 y (t 0 ) = y 0. y m 1 = y m y m = f(t, y 1,..., y m ). (8.6) y m 1 (t 0 ) = y (m ) 0 y m (t 0 ) = y (m 1) 0. Also, by using systems we can convert non-autonomous systems into autonomous systems. An autonomous system has no explicit t dependence in the right-hand side, y = f( y), y(x 0 ) = y 0. (8.7) Suppose we have an n-dimensional non-autonomous system, y = f( y, t). Define a new vector, v = (y 1,..., y n, t) t. Then ( ) d v dt = f(v1,..., v n+1 ) 1 (8.8) with initial conditions v 1 = y 1 0,..., v n = y n 0, v n+1 = t 0 is an autonomous sytem of dimension n

16 8.1 One Step Methods A general explicit one step method (Euler, Runge-Kutta) can be written in the form y k+1 = y k + hφ(y k, h). (8.9) Now let y k = y(t k ) be values of the exact solution to the differential equation. The local truncation error, τ k, is then given by τ k = y k+1 y k hφ(y k, h). (8.10) Specifically, we are interested in finding the leading order term of the local trunctation error, τ k = c(t k )h p+1 + O(h p+ ) (8.11) where the method is said to be order p. To find the local truncation error, we Taylor expand all terms about y k and cancel terms until we can determine the order of the method. Definition 8.1. (Consistency) A method is said to be consistent if τ k 0 as h 0. (8.1) Stability Analysis In this section, we concern ourselves with determining the region of absolute stability of an ODE method. To find the region of absolute stability, we consider the model equation dy = λy. (8.13) dt Solutions to (8.13) go to zero for all initial conditions when λ < 0. In the following definition, h is the step size in the method. Definition 8.. (Region of Absolute Stability) The region of absolute stability is the set of all hλ C such that the numerical solution {y n } 0 as t n when applied to equation (8.13). Definition 8.3. (A-Stability) A method is called A-Stable when the region of absolute stability includes the entire left-half (complex) plane Common Methods and Their Properties We use the notation S to denote the interval of absolute stability in this section. Forward Euler (Explicit, Order 1, S = (, 0)) Backward Euler (Implicit, Order 1, S = (, 0) (, )) Trapezoidal Method (Implicit, Order, S = (, 0)) Theta Method y k+1 = y k + hf(t k, y k ) (8.14) y k+1 = y k + hf(t k+1, y k+1 ) (8.15) y k+1 = y k + 1 h [f(t n, y n ) + f(t n+1, y n+1 )] (8.16) y k+1 = y k + h [θf(t k, y k ) + (1 θ)f(t k+1, y k+1 )] (8.17) 16

17 order θ method 1/ Trapezoidal 1 0 Backward Euler 1 1 Forward Euler 1 otherwise Runge-Kutta (Explicit, Maximum Order, S = (, 0)) y = y k + αhf(y k ) (8.18) y k+1 = y k + hβ 1 f(y k ) + hβ f(y ) (8.19) Second order if α = β 1 = β = 1/. The interval of absolute stability depends on the choice of coefficients. Runge-Kutta 3 (Explicit, Order 3) k 1 = f(t k, y k ), (8.0) ( k = f t k + h, y k + h ) k 1, (8.1) k 3 = f(t k + h, y k + h( k 1 + k )), (8.) y k+1 = y k + h 6 (k 1 + 4k + k 3 ) (8.3) 8. Linear Multistep Methods The general form of a linear multistep method (LMM) is y n+1 = p α j y n j + h j=0 p β j f n j. (8.4) One way to derive these methods is via numerical integration. Here, the general idea is to reformulate the differential equation by integrating over some interval, [t n r, t n+1 ] yielding j= 1 tn+1 y(t n+1 ) = y(t n r ) + f(t, y(t))dt (8.5) t n r Now we can construct an interpolant for the integrand f(t, y(t)) and integrate it over the interval. If we drop the error term from the numerical integration, we are then left with a LMM. For an example of deriving a LMM using the trapezoidal rule to approximate the integral, see the solutions set, Spring 006, #4. As with single step methods, we are now interested in looking at the local truncation error and the stability of the method. The local truncation error analysis proceeds exactly as it did for one step methods, although now we will generally choose to Taylor expand about t n+1 rather than t n in order to minimize the number of Taylor expansions. The following theorem gives us an easy method for characterizing the consistency of a LMM. Theorem 8.1. A linear multistep method is consistent if and only if p α j = 1, j=0 p jα j + j=0 For stability anaylsis, we look at solving the problem p β j = 1. (8.6) j= 1 Consider the polynomial y 0, y(0) = 1. (8.7) 17

18 and let r 0,..., r p be the roots of ρ(r). p ρ(r) = r p+1 α j r p j (8.8) j=0 Definition 8.4. (Root Condition) A linear multistep method satisfies the root condition if 1. r j 1, j = 0, 1,... p. if r j = 1 ρ (r j ) 0 (i.e. the root is simple) Consistency and the root condition allow us to discuss the stability and convergence of the method through the following two theorems. Theorem 8.. (Stability for LMMs) Assume the consistency condition. Then the multistep method is stable if and only if the root condition is satisfied. Theorem 8.3. (Convergence of LMMs) condition is satisfied. A consistent multistep method is convergent if and only if the root 8.3 Predictor-Corrector Methods A predictor-corrector method consists of two steps. First, we use an explicit predictor formula and then an implicit corrector step. It is very common to use an Adams-Bashforth formula as a predictor and an Adams-Moulton formula as the corrector. The simplest case (and most likely to occur on a qualifying exam) would be using AB1 (Forward Euler) as the predictor and AM (the Trapezoidal Rule) as the corrector, Note that in this case, we have actually obtained an RK method. 8.4 Qual Problems There is one ODE question on each numerical analysis qualifying exam. 9 Finite Differences for PDEs y i+1 = y i + hf i (8.9) y i+1 = y i + h (F i + F i ). (8.30) In this section we are primarily concerned with proposing finite difference methods for solving various PDEs and then verifying certain properties of those methods, namely the order of accuracy and stability. By finding the order of accuracy, we will be able to verify whether the method is consistent and stability will be verified through Theorem 9.. Then via the Lax-Richtmeyer Equivalence Theorem (Theorem 9.1), we will have that the method is convergent. Definition 9.1. (Consistency) Let P u = f be a partial differential equation and P k,h v = f be a finite difference scheme for solving the PDE. We say that the finite difference scheme is consistent with the PDE if for any smooth function φ(t, x) P φ P k,h φ 0 as k, h 0. (9.1) 18

19 9.1 Order of Accuracy There are two approaches we can use to find the order of accuracy for finite difference schemes for PDEs. The first is to expand all the terms in Taylor series about a common point and cancel terms in the truncation error until we find the leading order terms. This can also be done through symbols of the difference operator and the differential operator, which we will describe in more detail here. In this section, we write our differential equation as P u = f and the difference method in the form P k,h v = R k,h f. Now we have that Definition 9.. A scheme P k,h v = R k,h f that is consistent with the PDE P u = f is accurate of order p in time and q in space if for any smooth function φ(t, x), We say the method is accurate of order (p, q) Using Symbols P k,h φ R k,h P φ = O(k p ) + O(h q ). (9.) We begin by defining the symbols of the difference operator and the differential operator. Definition 9.3. (Symbol of the Difference Operator) given by The symbol p k,h (s, ξ) of a difference operator P k,h is Definition 9.4. (Symbol of the Differential Operator) by p k,h (s, ξ) = P k,h(e skn e imhξ ) e skn e imhξ. (9.3) The symbol of the differential operator P is given Now from Theorem 9.3, we look at p(s, ξ) = P (est e iξx ) e st e iξx. (9.4) p k,h (s, ξ) r k,h (s, ξ)p(s, ξ) = O(k p ) + O(h q ). (9.5) We then Taylor expand as needed on the left-hand side and cancel terms until we find the appropriate order. For a specific example, see the solutions set. 9. Stability Analysis To determine the stability of the method, we look at the homogeneous equation (that is, we are only interested in the difference operator). To carry out this analysis, we make the substitution g n e imθ for v n m where θ = hξ in the homogeneous equation. We then solve for g, the amplification factor. In particular, if we have found the symbol of the difference scheme, p k,h (s, ξ), we can simply substitute e sk = g We then take the resulting equation and manipulate it to obtain a form such that we can use Theorem 9.. Examples of this analysis are included in the solutions set. There are two useful trigonometric identities when working with these equations that we note here. 9.3 Well-Posedness of Equations 1 cos θ = sin 1 θ (9.6) sin θ = sin 1 θ cos 1 θ (9.7) Through the symbol of the differential operator, we have an easy method for showing whether or not a problem is well posed. We look at p(s, ξ, η) = 0 and find the roots of the symbol in terms of s. We then have a theorem that states the problem is well posed as long as the real part of the roots is bounded for all values of ξ and η. For an example, see Fall 006, #5 in the solutions set. 19

20 9.4 Hyperbolic PDEs Common Methods In this section, we give some methods, and their properties, for solving hyperbolic PDEs, using the onedimensional transport equation as our prototypical example, u t + au x = f. (9.8) After listing some of the common methods, we will give the derivation of some of these methods which is needed to adapt it to problems of a slightly different form than (9.8) and which will also help with remembering the method. Upwind differencing is used on the spatial derivative, au x. Upwind Differencing ( Order 1 in space ) { 1 h (vn m v n m 1), when a > 0 1 h (vn m+1 v n m), if a < 0 (9.9) Lax-Friedrichs ( Order O(k) + O(h ) + O(k 1 h ), Stable for aλ 1 ) v n+1 m 1 (vn m+1 + v n m 1) k + a vn m+1 v n m 1 h = f n m (9.10) Lax-Wendroff ( Order (,), Stable if aλ 1 ) v n+1 m vm n k + a vn m+1 v n m 1 h a k vm+1 n vm n + vm 1 n h = 1 n+1 (fm + f n m) ak 4h (f m+1 n fm 1) n (9.11) Crank-Nicolson ( Order (,), Unconditionally Stable ) v n+1 m vm n k + a vn+1 m+1 vn+1 m 1 + vn m+1 v n m 1 4h = f m n+1 + fm n (9.1) Box Scheme ( Order (,), Unconditionally Stable ) 1 [ (v n+1 m + vm+1 n+1 k ) (vn m + vm+1) n ] + a [ (v n+1 m+1 h vn+1 m ) + (vm+1 n vm) n ] = 1 ( f n+1 m f m n+1 + fm+1 n + fm n ) (9.13) We now give a derivation of the Lax-Wendroff method for equation (9.8). We start by Taylor expanding the solution in time u(x, t + k) = u(x, t) + ku t (x, t) + k u tt + O(k 3 ). (9.14) Now we use the following relations u t = au x + f (9.15) u tt = a(u t ) x + f t (9.16) = a( au xx + f x ) + f t (9.17) = a u xx af x + f t (9.18) 0

21 Now, we plug these into the Taylor expansion u(x, t + k) = u(x, t) + k( au x + f) + k (a u xx af x + f t ) + O(k 3 ) (9.19) Here we switch to difference notation and replace all the derivatives with second order accurate centered finite difference approximations in space and a forward difference in time, yielding u n+1 m = u n m ak h (un m+1 u n m 1) + a k h (un m+1 u n m + u n m 1) + kfm n ak 4h (f m+1 n fm 1) n + k n+1 (fm f n m) (9.0) = u n m ak h (un m+1 u n m 1) + a k h (un m+1 u n m + u n m 1) ak 4h (f m+1 n fm 1) n + k n+1 (fm + f n m) (9.1) 9.5 Parabolic PDEs Common Methods We give the form of these methods for solving the one-dimensional non-homogeneous heat equation, Adpating them to other methods will hopefully be straightforward. u t = bu xx + f(t, x). (9.) Backward-Time Central-Space ( Order (1,), Implicit, Unconditionally Stable ) v n+1 m vm n k = b vn+1 m+1 vn+1 m + vm 1 n+1 h + fm n+1 (9.3) Crank-Nicolson ( Order (,), Implicit, Unconditionally Stable ) v n+1 m vm n k = 1 bvn+1 m+1 vn+1 m + vm 1 n+1 h + 1 bvn m+1 vm n + vm 1 n h + 1 n+1 (fm + f n m) (9.4) Du Fort-Frankel ( Order O(h ) + O(k ) + O(k h ), Explicit, Unconditionally Stable ) v n+1 m vm n 1 k = b vn m+1 (vm n+1 + vm n 1 ) + vm 1 n h + fm n (9.5) ADI (Peaceman-Rachford) ( Order (,), Implicit, Unconditionally Stable ) ADI methods have the advantage that we are only inverting one-dimensional operators. In two dimensions, the method is given by: We can also write this method as a single equation ( I k ) ( A 1h ṽ n+1/ = I + k ) A h v n (9.6) ( I k ) ( A h v n+1 = I + k ) A 1h ṽ n+1/ (9.7) 1

22 ( I k ) ( A 1h I k ) ( A h v n+1 = I + k ) ( A 1h I + k ) A h v n. (9.8) Note that for this method, we have assumed a homogeneous problem. The Peaceman-Rachford method also works in three dimensions. In this case the method is given by which can also be written as a single equation, ( I k ) ( A 1h ṽ n+1/3 = I + k ) A 3h v n (9.9) ( I k ) ( A h ṽ n+/3 = I + k ) A h ṽ n+1/3 (9.30) ( I k ) ( A 3h v n+1 = I + k ) A 1h ṽ n+/3 (9.31) ( I k ) ( A 1h I k ) ( A h I k ) ( A 3h v n+1 = I + k ) ( A 1h I + k ) ( A h I + k ) A 3h v n (9.3) assuming that A 1h and A h commute Variable Diffusion Coefficients When the diffusivity is a function of time or space, the discretization is space is slightly different. Generally, a term of this form is written A second order (in space) discretization of this term is given by 9.6 Key Theorems (b(t, x)u x ) x. (9.33) b(t n, x m+1/ )(v n m+1 v n m) b(t n, x m 1/ )(v n m v n m 1) h. (9.34) Theorem 9.1. (Lax-Richtmyer Equivalence Theorem) A consistent finite difference scheme for a partial differential equation for which the initial value problem is well-posed is convergent if and only if it is stable. Theorem 9.. (Stability) A one step finite difference scheme (with constant coefficients) is stable in a stability region Λ if and only if there is a constant K (independent of θ, k, and h) such that with (k, h) Λ. If g(θ, k, h) is independent of k and h, we have g(θ, k, h) 1 + Kk (9.35) g(θ) 1. (9.36) Theorem 9.3. (Symbols and Order of Accuracy) A scheme P k,h v = R k,h f that is consistent with P u = f is accurate of order (p, q) if and only if for each value of s and ξ we have p k,h (s, ξ) r k,h (s, ξ)p(s, ξ) = O(k p ) + O(h q ). (9.37)

23 9.7 Qual Problems There are two finite difference problems for PDEs on each qualifying exam. Here we list a general category for the problems along with some specific information of the problem to give an idea of topics that have been asked in recent years. Generally, for every problem we either have to give a method or demonstrate that the given method is of a certain order of accuracy and is stable. Problems marked with (bc s) indicate the question asked about well-posedness in relation to the choice of boundary conditions. Convection-Diffusion Equation Fall 006, #6, (Non-linear) Spring 006, #6 (Max Norm) Fall 004, #6 Fall 00, #6 (Max Principle) Spring 001, #7 (L and max norms) Heat Equation Fall 005, #5 (Varying Diffusivity, ADI) Winter 004, #6 (Fourth-Order) Fall 003, #6 (Du Fort-Frankel) Spring 00, #7 Winter 00, #6 Wave Equation with Mixed Partial Term Fall 006, #5 (Well-Posedness) Spring 006, #5 (Well-Posedness) General nd Order Equations Fall 004, #5 (Well-Posedness) Fall 00, #5 (Well-Posedness) Transport Equation Fall 005, #6 (Lax-Wendroff, bc s) Winter 004, #5 (D, bc s) Winter 003, #5 (Box Scheme) Spring 00, #6 (Non-linear) Fall 001, #5 (D) Heat Equation with Mixed Partial Term Winter 003, #6 (Well-Posedness) Fall 001, #6 (Well-Posedness) Wave Equation Spring 001, #6 (bc s) Wave Equation plus First Order Terms Winter 005, #5 (bc s) Winter 00, #5 10 Finite Element Methods In this section, we look at developing finite element methods for scalar elliptic PDEs. We now describe the general form of these problems. Let be a domain in R d and let L be a differential operator of the form Lu = (σ u) + β u + µu (10.1) where σ, β and µ are functions defined over taking values in R d,d, R d and R, respectively. Let f be a function, f : R. We wish to find a function u : R such that { Lu = f in, (10.) Bu = g on where B is an operator that describes the boundary conditions for the problem. In the following sections we will derive the weak formulations of this problem corresponding to different boundary conditions, construct a finite element method, and demonstrate the existence of solutions Weak Formulations To find a weak formulation of the problem, we multiply the equation by a test function v, chosen in an appropriate space, and then integrate over. Using Green Formula (Theorem 10.1), we will then obtain a weak formulation of the problem. 3

24 Dirichlet Boundary Conditions case, the equation (10.) becomes We start with homogeneous Dirichlet boundary conditions. In this { Lu = f in, u = 0 on. We choose our space of test functions to be H 1 0 () since we have the condition u = 0 on. Now we have (σ u)v + v (β u) + µuv = v σ u + v (β u) + µuv Let s define the following bilinear form a(u, v) = 0 (n σ u)v = The weak formulation is then given by { Seek u H0 1 () such that a(u, v) = fv, v H1 0 (). (10.3) fv (10.4) fv. (10.5) v σ u + v (β u) + µuv. (10.6) (10.7) In the case where the Dirichlet boundary conditions are not homogeneous, say we have u = g on where g : R, we have to handle the equation slightly differently. We first assume g is sufficiently smooth so that there exists a lifting, u g of u in H 1 (). That is, u g H 1 () and u g = g on. Then, we can write u = u g + φ where φ H0 1 (). Following the same method as above, we obtain the weak form Seek u H 1 () such that u = u g + φ, φ H0 1 (), a(φ, v) = (10.8) fv a(u g, v), v H0 1 (). Neumann Boundary Conditions Now we consider boundary conditions of the form n σ u = g on where g : R. Now, we take our test functions in H 1 () since we no longer are looking for functions that are zero on. Deriving the weak form, we find giving (σ u) + β u + µu = v σ u + v (β u) + µuv (n σ u)v = v σ u + v (β u) + µuv = { Seek u H 1 () such that a(u, v) = fv + gv, v H1 (). fv (10.9) fv (10.10) fv + gv (10.11) (10.1) Mixed Dirichlet-Neumann Boundary Conditions Suppose we have a partition of our boundary, = D N, where we impose a Dirichlet boundary condition on D and a Neumann condition on N, { u = 0, on D (10.13) n σ u = g, on N 4

25 Table 3: Weak formulations for different boundary conditions for second-order, scalar, elliptic PDEs (10.). Problem V a(u, v) f(v) Homogeneous Dirichlet H0 1 () a(u, v) fv Neumann H 1 () a(u, v) fv + gv Dirichlet-Neumann H 1 D () a(u, v) fv + N gv Robin H 1 () a(u, v) + γuv fv + gv for a function g : N R. The appropriate space for the test functions is now given by H 1 D () = {u H 1 () : u = 0 on D }. (10.14) When finding the weak form, we now split the integral on into two parts on the different portions of the boundary, yielding the following weak form { Seek u H 1 D () such that a(u, v) = fv + N gv, v H 1 (10.15) D (). Robin Boundary Conditions A Robin boundary condition has the form γu + n σ u = g on where γ, g : R. If we compute the integral in equation (10.4), we find the weak formulation { Seek u H 1 () such that a(u, v) + γuv = fv + gv, v (10.16) H1 (). General Form We can summarize these results very simply in a table (given in Table 3). All the weak forms we have considered fit in this framework except for the non-homogeneous Dirichlet problem. The generic form of these methods is given by { Seek u V such that (10.17) a(u, v) = f(v), v V, where V is a Hilbert space satisfying H 1 0 () V H 1 (). (10.18) We have a bilnear form a(, ) on V V and a linear form, f( ) defined on V. 10. Well-Posedness of the Weak Form Here we consider the well-posedness of the problem (10.17). We begin with some definitions. Definition (H 1 -Norm) For u H 1 (), its norm is given by ( ) 1 u H 1 () = u + u. (10.19) We specifically note that from this definition, we can conclude that u L () u H 1 (). This will be useful in proving the following properties: 5

26 Definition 10.. (Coercivity) The bilinear form a(, ) is coercive if α > 0 such that a(u, u) α u V, u V. (10.0) Definition (Continuity of the bilinear form) a(, ) is continuous if γ > 0 such that a(u, v) γ u V v V, u, v V. (10.1) Definition (Continuity of the linear form) f( ) is continuous if δ > 0 such that f(v) δ v V, v V. (10.) In order to demonstrate the well-posedness of the weak variational form, we need to verify the conditions in the Lax-Milgram Lemma (Lemma 10.1), i.e. the bilinear form is continuous (bounded) and coercive and that the linear form is continuous (bounded). In 10.6, we give several inequalities that are useful for finding the bounds needed to verify the Lax-Milgram Lemma. For specific examples, see the solutions set, available as a separate document Formulating a Finite Element Approximation To begin, we need to divide our computational domain into elements. In one dimension, these elements will be (disjoint) intervals and in two dimensions we generally choose to use a set of triangles, K i. We call the set of non-overlapping triangles, T h, a triangulation of the domain = K 1 K K m. The triangles are chosen such that no vertex of one triangle lies on the edge of another. Now, given a triangulation of the domain, we construct the subspace on which our solution exists. Generally, we construct a solution that is a piecewise-polynomial on each element. As an example, we can define V h V such that V h = {v : v is continuous, v K is linear K T h }. We adopt the following notation for r N P r (K) = {v : v is a polynomial of degree r on K}. (10.3) Note that q = dim P r (K) = 1 (r + 1)(r + ). Let n i, i = 1,..., N be the set of nodes for T h. Our solution, v V h will then be described by the values v(n i ). These are the global degrees of freedom. We also have element degrees of freedom, associated with each node belonging to a single element, K T h. Next we construct a set of nodal basis functions on each element, φ i P r (K) such that for n j a node belonging to K giving us the representation v(x) = φ i (n j ) = δ ij (10.4) q v(n i )φ i (x), x K (10.5) i=1 for v(x) P r (K). Now, we have V h = {v : v K P r (K) K T h with v continuous at the nodes}. Functions v V h will be such that v C 0 (). Note that to construct higher order polynomials on the elements, it will be necessary to add additional element degrees of freedom. We can specify additional values at mid-points of the elements or add derivative information at the nodes we already have, resulting in different basis functions. Further, we can construct a basis for the space V h, {ϕ i }, from the nodal basis functions. The important observation here is that ϕ i is only non-zero on elements K such that n i K. Our finite element method is then given by { Find u h V h such that (10.6) a(u h, v) = (f, v) v V h. 6

27 10.4 The Stiffness Matrix Now we briefly look at actually solving (10.6). Let {ϕ 1,..., ϕ M } be a basis for V h. Then for v V h we can write Now, we can write M v = η i ϕ i, η i R. (10.7) i=1 a(u h, ϕ i ) = L(ϕ i ), i = 1,... M (10.8) where L( ) is our linear form, (f, ). Given that the solution can be written as we can write (10.6) as M u h = ξ i ϕ i, ξ i R (10.9) i=1 which can be written in matrix form M a(ϕ i, ϕ j )ξ i = L(ϕ j ), j = 1,... M (10.30) i=1 Aξ = b (10.31) where ξ, b R M with a ij = a(ϕ i, ϕ j ) and b i = L(ϕ i ). Now we make two important observations. Using equation (10.7) we have M M a(v, v) = a η i ϕ i, η j ϕ j = i=1 Since the bilinear form is coercive, we obtain j=1 M η i a(ϕ i, ϕ j )η j = η Aη. (10.3) i,j=1 η Aη = a(v, v) α v V > 0 (10.33) showing that the matrix is positive definite. Further, we often have that the bilinear form is symmetric a(ϕ i, ϕ j ) = a(ϕ j, ϕ i ). This property is easy to verify if it is indeed true for our problem. If it is, then we have that A is a symmetric positive definite matrix which implies that the matrix problem has a unique solution (and in some sense, is easy to solve). Further, we can see that the matrix A will be sparse since a(ϕ i, ϕ j ) = 0 unless n i and n j both belong to the same triangle K. By selecting the proper ordering of the nodes, we can ensure that A is both banded and sparse Error Estimates In order to obtain error estimates, we need to enforce a condition on the triangulation of the domain. Particularly, we cannot allow the triangles to become arbitrarily thin. Let h K be the diameter of K (the length of the longest side of K) and ρ K be the diameter of the largest circle we can inscribe in K. If h = max K Th h K, let β > 0 be a constant independent of h. Then if ρ K h K β K T h (10.34) we can apply some simple error estimates. Now, let π h u V h be an interpolant of u in the appropriate finite element space. We then have the following estimates where the interpolant consist of piecewise polynomials of degree r 1. 7

28 u π h u L () Ch r+1 u H r+1 (), (10.35) u π h u H 1 () Ch r u H r+1 (). (10.36) Now, for elliptic problems, we have the following abstract error estimate, u u h V C u v V v V h. (10.37) Letting v = π h u V h, we can then bound the finite element error by the interpolant error 10.6 Key Theorems Theorem (Green s Formula) (σ u)v = Theorem 10.. (Poincaré s Inequality) c p, > 0 such that v W 1,p 0 () u u h V C u π h u V. (10.38) v σ u (n σ u)v (10.39) Let 1 p < and let be a bounded open set. Then there exists c p, v L p () v L p (). (10.40) Theorem Let be a bounded connected open set in R n, with sufficiently regular boundary. Then we have for u H 1 (), such that u(x)dx = 0, u L () P () u L (). (10.41) Corollary u H1 () = u L () is a norm equivalent with the norm u H1 () on the subspace V 0 (closed in H 1 ()) defined by V 0 = {u H 1 () : u(x)dx = 0}. (10.4) Corollary 10.. Let be a bounded connected open set in R n with sufficiently regular boundary Γ. Suppose Γ = Γ 1 Γ with length (area) of Γ > 0. Let V Γ = {u H 1 () : u Γ = 0}. (10.43) Then V Γ is a closed subspace of H 1 () and u H 1 () = u L () is a norm equivalent with the norm u H1 () on the subspace V Γ. Corollary Let be an open and bounded domain with Lipschitz-continuous boundary Γ =. Then there is a positive constant C such that u Γ L (Γ) C u H 1 (). (10.44) Lemma (Lax-Milgram) Let a be a coercive, bounded bilinear form on a Hilbert space V and f be a bounded linear form on V, then there exists a unique u V such that Further, for all f V, we have a(u, v) = f(v) v V. (10.45) where α is the constant from the coercivity bound. u V 1 α f V (10.46) 8

29 10.7 Qual Problems There is one finite element problem on every qualifying exam. Generally it is the last problem, #7. 11 References The material for these notes was taken primarily from the sources listed below. General Numerical Analysis K. E. Atkinson, An Introduction to Numerical Analysis, John Wiley & Sons, second ed., R. L. Burden and J. D. Faires, Numerical Analysis, Brooks/Cole, seventh ed., 001. Numerical Linear Algebra G. H. Golub and C. F. Van Loan, Matrix Computations, The Johns Hopkins University Press, third ed., Numerical Differentiation B. Fornberg, A Practical Guide to Psuedospectral Methods, Cambridge University Press, 005. Numerical Methods for ODEs K. E. Atkinson, An Introduction to Numerical Analysis, John Wiley & Sons, second ed., L. Shampine, Numerical Solution of Ordinary Differential Equations, Chapman & Hall, Finite Differences for PDEs J. Strikwerda, Finite Difference Schemes and Partial Differential Equations, SIAM, second ed., 004. Finite Element Method A. Ern and J.-L. Guermond, Theory and Practice of Finite Elements, Springer-Verlag, 004. C. Johnson, Numerical Solution of Partial Differential Equations by the Finite Element Method, Studentlitteratur,