1 What is numerical analysis and scientific computing?

Transcription

1 Mathematical preliminaries 1 What is numerical analysis and scientific computing? Numerical analysis is the study of algorithms that use numerical approximation (as opposed to general symbolic manipulations) for the problems of mathematical analysis. (from wiki) The scientific computing/computational mathematics developing and analyzing numerical techniques for solving algebraic or differential equations use computers to analyze/solve scientific/engineering problems. based on mathematical modeling of practical problems 2 Calculus KEY WORDS. Limit, Convergence, Continuous Function, derivative, Intermedia/Mean Value Theorem, Taylor Expansion, Big O Notation. GOAL. Understand the Intermedia/Mean Value Theorem; understand the big O notation; be able to perform the basic Taylor expansion. Definition 2.1. (Limit) A function f defined on a set X of real numbers has the limit L at x 0, written lim f(x) = L, x x 0 if for any given ɛ > 0, there exists a real number δ > 0, such that f(x) L < ɛ, whenever x X and 0 < x x 0 < δ Definition 2.2. (Continuous) Let f be a function defined on a set X of real numbers and x 0. Then f is continuous at x 0 if lim x x 0 f(x) = f(x 0 ). The function f is continuous on the set X if it is continuous at each number in X. Definition 2.3. (Convergence of sequence) Let {x 0 } n=1 be an infinite sequence of real or complex numbers. The sequence {x 0 } n=1 has the limit x (converges to x), if for any ɛ > 0, there exists a positive integer N(ɛ) such that x x n < ɛ, whenever n > N(ɛ). The notation means that the sequence {x 0 } n=1 converges to x. lim x n = x, or x n x, as n, n Theorem 2.4. (Continuous) If f is a function defined on a set X of real numbers and x 0 X, then the following statement is equivalent f is continuous at x 0 If {x n } n=1 is any sequence in X converging to x 0, then lim n f(x n ) = f(x 0 ). Definition 2.5. (Differentiable) Let f be a function defined in an open interval containing x 0. The function f is differentiable at x 0 if f f(x) f(x 0 ) (x 0 ) = lim x x 0 x x 0 exists. The number f (x 0 ) is called the derivative of f at x 0. A function that has a derivative at each number in a set X is differentiable on X. 1

2 Theorem 2.6. (Intermediate Value Theorem.) Let f(x) be a continuous function on the closed interval [a, b]. Then for every value α in between f(a) and f(b), there is at least one point ξ [a, b] for which f(ξ) = α. Theorem 2.7. (Mean Value Theorem.) If f(x) is a continuous function on the closed interval [a, b] and differentiable on the open interval (a, b), then there is at least one number ξ (a, b) such that f(b) f(a) b a = f (ξ). Theorem 2.8. (Rolle s Theorem.) Let f(x) be a continuous function on the closed interval [a, b] and differentiable on the open interval (a, b). If f(a) = f(b) = 0, then there is at least one number ξ (a, b) such that f (ξ) = 0. Theorem 2.9. (Taylor s Theorem.) If f(x) has n + 1 continuous derivatives on [a, b] and c is some point in [a, b], then for any c and x [a, b], where f(x) = f(c) + f (c)(x c) + f (c)(x c) 2 E n+1 = f (n+1) (ξ)(x c) n f (n) (c)(x c) n + E n+1 (1) = O((x c) n+1 ), (2) and ξ lies between c and x and depends on both. Equation (1) is the Taylor expansion of the function f(x) around the point c. Notations (the big O notation O) E n+1 = O((x c) n+1 ) is shorthand for the inequality E n+1 C x c n+1, where C is a constant. For example, in equation (2), C can be f (n+1) (ξ) (n+1)!. Remark Taylor expansion is useful in estimating the function value around x = c when 1. much is known about the function and its derivatives at x = c (however, little is know around the neighborhood of x = c) 2. x is close to c, i.e. E n+1 = O((x c) n+1 ) is very small (within our tolerance for error). Example Taylor expand the following function. 1. f(x) = e x around x = 0 Solution. Since (e x ) i = e x, for any i = 1, 2, 2. sin(x) around x = 0 e x = f(0) + f (0)x + f (0)x 2 = e 0 + e 0 x + e0 x e0 x n f (n) (0)x n + eξ x n+1 = 1 + x + x xn + eξ x n+1 + f (n+1) (ξ)x n+1 = 1 + x + x xn + O(xn+1 ) (3) 2

3 Solution. Since sin(x) 2i+1 = ( 1) i cos(x), for any i = 0, 1, 2,, and sin(x) 2i = ( 1) i sin(x), for any i = 1, 2, sin(x) = f(0) + f (0)x + f (0)x 2 Reading Assignments/References: = sin(0) + cos(0)x + sin(0)x f (n) (0)x n sin(n) (x)x n + f (n+1) (ξ)x n+1 + sin(n+1) (ξ)x n+1 = x x3 3! + x5 5! + + ( 1)k x 2k+1 (2k + 1)! + O(x2k+3 ) (4) 1. About Intermedia Value Theorem: follow the link value_theorem. 2. About Mean Value Theorem: follow the link 3. About the Big O Notation: follow the link 4. About Taylor Series: follow the link or Chapter 1.2 of Cheney & Kincaid Exercise If the following statements are true. 1. If f(x) is a continuous function on the closed interval [a, b]. Let M = max a x b f(x), m = min a x b f(x). Then for every value α [m, M], there is at least one point ξ [a, b] for which f(ξ) = α. 2. If f(x) be a continuous function on the closed interval [a, b] with f(a)f(b) < 0, then there is at least one point ξ [a, b] for which f(ξ) = If f(x) be a continuous function on the closed interval [a, b] and differentiable on the open interval (a, b). If f(a) = f(b) = 0, then there is at least one number ξ (a, b) such that f (ξ) = 0. Exercise Taylor Expand the following function 1. cos(x) around x = log(x) around x = 1. Exercise (Big O Notation) In mathematics, big O notation (so called because it uses the symbol O) describes the limiting behavior of a function for very small or very large arguments, usually in terms of simpler functions. Although developed as a part of pure mathematics, it is now frequently also used in computational complexity theory to describe how the size of the input data affects an algorithm s usage of computational resources (usually running time or memory). Please see for the formal definition. By the definition of Big O notation, if the following statements are true x = O(x), as x x = O(x), as x x 2 = O(x), as x x 2 = O(x), as x N 3 = O(N 4 ), as N N 5 = O(N 4 ), as N. 3

4 3 Linear Algebra KEY WORDS. Matrix, vector, transpose, inverse, determinant, linear system of equations. GOAL. Be able to perform the basic matrix operation, such as transpose, inverse, product, determinant etc; be able to represent a linear system of equations in a matrix vector form. 3.1 Matrix Definition 3.1. (Matrix) A matrix is a rectangular collection of numbers arranged in rows and columns. A matrix with n rows and m columns is said to be of dimension n m. The element at the i th row and j th column is denoted as a ij. Definition 3.2. (Transpose) The transpose of an n m matrix A, denoted as A T, is an m n matrix where a T ij = a ji. Definition 3.3. (Matrix Product) Let A be an n m matrix, and B be an m p matrix. The product of A and B is an n p matrix C whose elements are defined by Example 3.4. A = AB = m c ij = a ik b kj. k=1 is a 3 by 2 matrix, B = [ ] is a 2 by 2 matrix. The product Definition 3.5. (Inverse) If A is an n n matrix. If there exists an n n matrix B, such that AB = BA = I, then B is the inverse of A. Definition 3.6. (Determinant) The determinant of a square n n matrix A, det(a), is defined recursively as follows 1. When n = 1, det(a) = a When n > 1, for any choice of row i, or det(a) = det(a) = n ( 1) i+j a ij m ij, j=1 n ( 1) i+j a ij m ij, i=1 for any choice of column j, where m ij is the determinant of the (n 1) (n 1) matrix obtained by deleting the i th row and j th column from A. a11 a Example 3.7. The determinant of an 2 2 matrix A = 12 is a a 21 a 11 a 22 a 12 a 21. The determinant of A = is a11 a Exercise 3.8. Check that the inverse of a 2 2 matrix A = 12 is A a 21 a 1 = 1 a22 a 12 det(a) 22 a 21 a 11 Exercise 3.9. A 2 2 matrix A = [ ]. What are A T, det(a), A 1? 4

5 4 Computer representation of numbers Definition 4.1. (Floating point representation in decimal system.) In the decimal system, any real number x can be represented in a normalized floating point form as x = ±p 10 n = ±(0.d 1 d 2 d 3...) n. where d 1 0, hence 0.1 q < 1. n is an integer. The numbers d 1, d 2,... are decimal digits 0, 1, 9. For example, 37.2 = Remark 4.2. The requirement of d 1 0 is to ensure a unique representation of real number. For example, 37.2 can not be represented as What is the smallest unit of information in computer? A bit. Information/numbers in computer chips are stored by a series of on and off states (bit), representing 0 and 1. Therefore, numbers wants to be represented in a binary form. Definition 4.3. (Floating point representation in binary system.) In the binary system, any real number x can be represented as x = ±q 2 m = ±(0.b 1 b 2 b 3...) 2 2 m, where b 1 0, hence b 1 = 1 and q 1/2. For example, (0.1) 2 = 2 1 = 0.5, (1.1) 2 = = 1.5, (11) 2 = = 3. (0.1) = 0.5, (0.1) = 0.25 (0.1) = 4. There are only a finite number of bits in computer to represent a number. A single precision floating point number is stored in 32 bits; a double precision floating point number is stored in 64 bits. The real numbers that can be represented in a computer are called its machine numbers. Not all real numbers are machine numbers. Definition 4.4. (Standard single-precision floating point representation) A machine number in standard single-precision floating point representation is, The number is represented by 32-bit. x = ( 1) s (1.f) 2 2 c 127. The leftmost bit is assigned to s for the sign. s = 0/1 corresponds to +/ respectively The next 8 bits are assigned to c. The value of c in representing a floating point number is restricted by 0 < c < ( ) 2 = 255. The values 0 and 255 are reserved for special case of 0 and ±. The range of representable machine numbers is mainly determined by the number of bits assigned to c. Therefore, it can be concluded that the largest single precision number is The smallest single precision number is (1.f) ; (1.0) 2 = The last 23 bits are assigned to f. The value of (1.f) 2 is restricted by 1 (1.f) 2 ( ) 2 = The significant decimal digits of accuracy is mainly determined by the number of bits assigned to f. The number of significant decimal digits of a single precision number is approximately 6, as 2 23 =

6 Definition 4.5. (Standard double-precision floating point representation) A machine number in standard double-precision floating point representation is, The number is represented by 64-bit. x = ( 1) s 2 c 1023 (1.f) 2. The leftmost bit is assigned to s for the sign. s = 0/1 corresponds to +/ respectively The next 11 bits are assigned to c. The value of c in representing a floating point number is restricted by 0 < c < ( ) 2 = The values 0 and 2047 are reserved for special case of 0 and ±. The last 52 bits are assigned to f. The value of (1.f) 2 is restricted by 1 (1.f) 2 ( ) 2 = The significant decimal digits of accuracy is mainly determined by the number of bits assigned to f. Exercise 4.6. Give an estimate on the largest and smallest double precision number. Given an estimate on the number of significant decimal digits of a double precision number. 6