MATH20602 Numerical Analysis 1

Transcription

1 M\cr NA Manchester Numerical Analysis MATH20602 Numerical Analysis 1 Martin Lotz School of Mathematics The University of Manchester Manchester, February 1, 2016

2 Outline General Course Information Introduction to Numerical Analysis Prerequisites from Calculus

3 Outline General Course Information Introduction to Numerical Analysis Prerequisites from Calculus

4 Organisation The course website can be found under 1 / 27

5 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). 1 / 27

6 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). Problems are divided in Part A and Part B. 1 / 27

7 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). Problems are divided in Part A and Part B. Part A comes with advance solutions and should be attempted at home. Part B will be worked on in tutorials. 1 / 27

8 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). Problems are divided in Part A and Part B. Part A comes with advance solutions and should be attempted at home. Part B will be worked on in tutorials. All the material presented in the lecture will be made available online as the course progresses. 1 / 27

9 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). Problems are divided in Part A and Part B. Part A comes with advance solutions and should be attempted at home. Part B will be worked on in tutorials. All the material presented in the lecture will be made available online as the course progresses. Lecture notes will appear on the website towards the end of each week. 1 / 27

10 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). Problems are divided in Part A and Part B. Part A comes with advance solutions and should be attempted at home. Part B will be worked on in tutorials. All the material presented in the lecture will be made available online as the course progresses. Lecture notes will appear on the website towards the end of each week. Lecture podcasts are available on Blackboard. 1 / 27

11 Organisation The course website can be found under Tutorials start on Thursday, February 4 (not tutorials today). Problems are divided in Part A and Part B. Part A comes with advance solutions and should be attempted at home. Part B will be worked on in tutorials. All the material presented in the lecture will be made available online as the course progresses. Lecture notes will appear on the website towards the end of each week. Lecture podcasts are available on Blackboard. Contact: martin.lotz@manchester.ac.uk 1 / 27

12 Example classes Example classes are there to deepen the understanding of the course material. 2 / 27

13 Example classes Example classes are there to deepen the understanding of the course material. Problems sheets should ideally be looked at before the example class. 2 / 27

14 Example classes Example classes are there to deepen the understanding of the course material. Problems sheets should ideally be looked at before the example class. In the example classes, you will be given some time to work on the problems, you should ask questions if some parts are not clear, you will get feedback on your attempts at the problems, we will go through a selection of problems and their solutions. 2 / 27

15 Example classes Example classes are there to deepen the understanding of the course material. Problems sheets should ideally be looked at before the example class. In the example classes, you will be given some time to work on the problems, you should ask questions if some parts are not clear, you will get feedback on your attempts at the problems, we will go through a selection of problems and their solutions. Eman Almoalim and Mante Zemaityte will help in the tutorials. 2 / 27

16 Important dates Reserve these dates: Wed, March 9: Revision class for midterm test Mon, March 14: Midterm (coursework) test (1 hour) Last class of term: Revision for final exam Date of final exam will be announced when available! 3 / 27

17 MATLAB 4 / 27

18 MATLAB Matlab, or an equivalent system, is an indispensable tool for this course. 4 / 27

19 MATLAB Matlab, or an equivalent system, is an indispensable tool for this course. Availability: On campus computers Student version available at low cost 4 / 27

20 MATLAB Matlab, or an equivalent system, is an indispensable tool for this course. Availability: On campus computers Student version available at low cost The course page contains links to Matlab documentation 4 / 27

21 MATLAB Matlab, or an equivalent system, is an indispensable tool for this course. Availability: On campus computers Student version available at low cost The course page contains links to Matlab documentation Alternatives: Python, Julia 4 / 27

22 MATLAB Matlab, or an equivalent system, is an indispensable tool for this course. Availability: On campus computers Student version available at low cost The course page contains links to Matlab documentation Alternatives: Python, Julia An easy to parse introduction to Matlab is available here: 4 / 27

23 Course Structure The lecture consists of the following parts 5 / 27

24 Course Structure The lecture consists of the following parts 1 Introduction and foundations (Lectures 1-2) 5 / 27

25 Course Structure The lecture consists of the following parts 1 Introduction and foundations (Lectures 1-2) 2 Interpolation and Numerical Integration (Lectures 3-9) 5 / 27

26 Course Structure The lecture consists of the following parts 1 Introduction and foundations (Lectures 1-2) 2 Interpolation and Numerical Integration (Lectures 3-9) 3 Numerical Linear Algebra (Lectures 10-15) 5 / 27

27 Course Structure The lecture consists of the following parts 1 Introduction and foundations (Lectures 1-2) 2 Interpolation and Numerical Integration (Lectures 3-9) 3 Numerical Linear Algebra (Lectures 10-15) 4 Nonlinear Equations (Lectures 16-19) 5 / 27

28 Learning Outcomes After the lecture you should have 6 / 27

29 Learning Outcomes After the lecture you should have practical knowledge of a range of iterative techniques for solving linear and nonlinear systems of equations, theoretical knowledge of their convergence properties; 6 / 27

30 Learning Outcomes After the lecture you should have practical knowledge of a range of iterative techniques for solving linear and nonlinear systems of equations, theoretical knowledge of their convergence properties; an appreciation of the problems of numerical computation, in particular how small changes in the data affect the solutions and experience with key examples arising in the solution of differential equations; 6 / 27

31 Learning Outcomes After the lecture you should have practical knowledge of a range of iterative techniques for solving linear and nonlinear systems of equations, theoretical knowledge of their convergence properties; an appreciation of the problems of numerical computation, in particular how small changes in the data affect the solutions and experience with key examples arising in the solution of differential equations; practical knowledge of polynomial interpolation and numerical integration, its numerical implementation and theoretical knowledge of associated approximation properties; 6 / 27

32 Learning Outcomes After the lecture you should have practical knowledge of a range of iterative techniques for solving linear and nonlinear systems of equations, theoretical knowledge of their convergence properties; an appreciation of the problems of numerical computation, in particular how small changes in the data affect the solutions and experience with key examples arising in the solution of differential equations; practical knowledge of polynomial interpolation and numerical integration, its numerical implementation and theoretical knowledge of associated approximation properties; acquired numerical problem solving skills that can be applied to problems from the whole range of applied mathematics. 6 / 27

33 Outline General Course Information Introduction to Numerical Analysis Prerequisites from Calculus

34 What is Numerical Analysis? Numerical analysis is the study of algorithms for solving problems of continuous mathematics. Nick Trefethen, The definition of numerical analysis (1992) 7 / 27

35 What is Numerical Analysis? Numerical analysis is the study of algorithms for solving problems of continuous mathematics. Nick Trefethen, The definition of numerical analysis (1992) An algorithm is a sequence of instructions designed to solve a computational problem. 7 / 27

36 What is Numerical Analysis? Numerical analysis is the study of algorithms for solving problems of continuous mathematics. Nick Trefethen, The definition of numerical analysis (1992) An algorithm is a sequence of instructions designed to solve a computational problem. Continuous mathematics refers to problems involving the real or complex numbers: computing integrals, solving differential equations, approximating functions based on data samples, or solving large systems of linear and nonlinear equations. 7 / 27

37 A simple numerical problem Approximate 2 by computing a sequence x n+1 = x n x n for every n 0, starting with a guess x 0. 8 / 27

38 A simple numerical problem Approximate 2 by computing a sequence x n+1 = x n x n for every n 0, starting with a guess x 0. f x ) ( 0. 5 x+1/x ) ; x = 2 ; f o r i = 1 : 5 x = f ( x ) end Matlab code Back in the day ( 3800 years ago) 8 / 27

39 A simple numerical problem Approximate 2 by computing a sequence x n+1 = x n x n for every n 0, starting with a guess x 0. f x ) ( 0. 5 x+1/x ) ; x = 2 ; f o r i = 1 : 5 x = f ( x ) end / 27

40 A simple numerical problem Approximate 2 by computing a sequence x n+1 = x n x n for every n 0, starting with a guess x 0. f x ) ( 0. 5 x+1/x ) ; x = 2 ; f o r i = 1 : 5 x = f ( x ) end In the lecture we will prove that sequence of numbers generated converges to 2, and determine how fast. 9 / 27

41 Top Ten Algorithms 1 Monte Carlo method or Metropolis algorithm, devised by John von Neumann, Stanislaw Ulam, and Nicholas Metropolis; 2 simplex method of linear programming, developed by George Dantzig; 3 Krylov Subspace Iteration method, developed by Magnus Hestenes, Eduard Stiefel, and Cornelius Lanczos; 4 Householder matrix decomposition, developed by Alston Householder; 5 Fortran compiler, developed by a team lead by John Backus; 6 QR algorithm for eigenvalue calculation, developed by J Francis; 7 Quicksort algorithm, developed by Anthony Hoare; 8 Fast Fourier Transform, developed by James Cooley and John Tukey; 9 Integer Relation Detection Algorithm, developed by Helaman Ferguson and Rodney Forcade; 10 fast Multipole algorithm, developed by Leslie Greengard and Vladimir Rokhlin; (List assembled by Dongarra and Sullivan) 10 / 27

42 The Challenges of Numerical Analysis A particular challenge for Numerical Analysis is the fact that computers are finite devices. 11 / 27

43 The Challenges of Numerical Analysis A particular challenge for Numerical Analysis is the fact that computers are finite devices. Most numerical problems can t be solved in a finite amount of time. 11 / 27

44 The Challenges of Numerical Analysis A particular challenge for Numerical Analysis is the fact that computers are finite devices. Most numerical problems can t be solved in a finite amount of time. Most real numbers can t be represented in a finite amount of space. 11 / 27

45 The Challenges of Numerical Analysis A particular challenge for Numerical Analysis is the fact that computers are finite devices. Most numerical problems can t be solved in a finite amount of time. Most real numbers can t be represented in a finite amount of space. Since none of the numbers which we take out from logarithmic and trigonometric tables admit of absolute precision, but are all to a certain extent approximate only, the results of all calculations performed by the aid of these numbers can only be approximately true. - C.F. Gauss (1809) 11 / 27

46 The E word An unfortunate fact in numerical computation is that we have accept the presence of errors. The are many types of errors to consider / 27

47 The E word An unfortunate fact in numerical computation is that we have accept the presence of errors. The are many types of errors to consider... Modelling errors; 12 / 27

48 The E word An unfortunate fact in numerical computation is that we have accept the presence of errors. The are many types of errors to consider... Modelling errors; Measurement errors; 12 / 27

49 The E word An unfortunate fact in numerical computation is that we have accept the presence of errors. The are many types of errors to consider... Modelling errors; Measurement errors; Rounding errors (most numbers cannot be stored exactly to computer precision); 12 / 27

50 The E word An unfortunate fact in numerical computation is that we have accept the presence of errors. The are many types of errors to consider... Modelling errors; Measurement errors; Rounding errors (most numbers cannot be stored exactly to computer precision); Truncation or approximation errors. 12 / 27

51 The E word An unfortunate fact in numerical computation is that we have accept the presence of errors. The are many types of errors to consider... Modelling errors; Measurement errors; Rounding errors (most numbers cannot be stored exactly to computer precision); Truncation or approximation errors / 27

52 The E word A further source of error... photocopiers! 13 / 27

53 The E word A further source of error... photocopiers! Scan on a Xerox Workcentre More information can be found on Prof. Nick Higham s blog. 13 / 27

54 Types of errors 14 / 27

55 Types of errors Modelling and measurement errors fall outside the scope of numerical analysis, but we need to be aware of their presence! 14 / 27

56 Types of errors Modelling and measurement errors fall outside the scope of numerical analysis, but we need to be aware of their presence! Rounding errors arise due to the fact that computers operate with limited storage space, and results of calculations are always rounded to the neares representable number (see floating point numbers). 14 / 27

57 Types of errors Modelling and measurement errors fall outside the scope of numerical analysis, but we need to be aware of their presence! Rounding errors arise due to the fact that computers operate with limited storage space, and results of calculations are always rounded to the neares representable number (see floating point numbers). Truncation errors arise when replacing functions or equations with an approximations. For example, the Taylor approximation of a function gives f(x) f(x 0 ) + f (x 0 )(x x 0 ) f (x 0 )(x x 0 ) f (n) (x 0 ) (x x 0 ) n. n! 14 / 27

58 Types of errors Modelling and measurement errors fall outside the scope of numerical analysis, but we need to be aware of their presence! Rounding errors arise due to the fact that computers operate with limited storage space, and results of calculations are always rounded to the neares representable number (see floating point numbers). Truncation errors arise when replacing functions or equations with an approximations. For example, the Taylor approximation of a function gives f(x) f(x 0 ) + f (x 0 )(x x 0 ) f (x 0 )(x x 0 ) f (n) (x 0 ) (x x 0 ) n. n! Errors can accumulate! 14 / 27

59 Measuring errors In order to quantify errors in our solutions we need to define a measure for the error If ˆx is an approximation to a quantity x then the absolute error is defined by x ˆx 15 / 27

60 Measuring errors In order to quantify errors in our solutions we need to define a measure for the error If ˆx is an approximation to a quantity x then the absolute error is defined by x ˆx The relative error is defined by ˆx x x, x 0 15 / 27

61 Relative errors Example Relative vs absolute measures An error of 1cm makes a big difference for small objects, but is not considered an error at all on a cosmic scale. 16 / 27

62 Significant Digits When doing calculations by hand, we will be using the concept of significant figures. 17 / 27

63 Significant Digits When doing calculations by hand, we will be using the concept of significant figures. Starting with the first non-zero digit on the left, count all the figures on the right of it, including trailing zeros if they are to the right of the decimal point. 17 / 27

64 Significant Digits When doing calculations by hand, we will be using the concept of significant figures. Starting with the first non-zero digit on the left, count all the figures on the right of it, including trailing zeros if they are to the right of the decimal point , and all have 5 significant figures. 17 / 27

65 Significant Digits When doing calculations by hand, we will be using the concept of significant figures. Starting with the first non-zero digit on the left, count all the figures on the right of it, including trailing zeros if they are to the right of the decimal point , and all have 5 significant figures. Concept can be made more precise using the normalised scientific notation. 17 / 27

66 Significant Digits When doing calculations by hand, we will be using the concept of significant figures. Starting with the first non-zero digit on the left, count all the figures on the right of it, including trailing zeros if they are to the right of the decimal point , and all have 5 significant figures. Concept can be made more precise using the normalised scientific notation. An approximation ˆx of a number x is correct to n significant figures, if both number round to the same number to k significant digits. 17 / 27

67 Floating Point Arithmetic 18 / 27

68 Floating Point Arithmetic On a computer, numbers are stored as a bounded sequence of bits, or 0 and 1 digits, usually using 32 bits (single precision) or 64 bits (double precision) per number. 18 / 27

69 Floating Point Arithmetic On a computer, numbers are stored as a bounded sequence of bits, or 0 and 1 digits, usually using 32 bits (single precision) or 64 bits (double precision) per number. In double precision, a number is represented as x = ±f 2 e, where f is a number in [0, 1], represented using 52 bits, e is an exponent, represented using 11 bits. The remaining bit is used as a sign. 18 / 27

70 Floating Point Arithmetic On a computer, numbers are stored as a bounded sequence of bits, or 0 and 1 digits, usually using 32 bits (single precision) or 64 bits (double precision) per number. In double precision, a number is represented as x = ±f 2 e, where f is a number in [0, 1], represented using 52 bits, e is an exponent, represented using 11 bits. The remaining bit is used as a sign. The largest representable number is of the order ±10 308, the smallest of the order / 27

71 Floating Point Arithmetic On a computer, numbers are stored as a bounded sequence of bits, or 0 and 1 digits, usually using 32 bits (single precision) or 64 bits (double precision) per number. In double precision, a number is represented as x = ±f 2 e, where f is a number in [0, 1], represented using 52 bits, e is an exponent, represented using 11 bits. The remaining bit is used as a sign. The largest representable number is of the order ±10 308, the smallest of the order Floating point numbers form a finite subset of the real numbers and are not uniformly spaced! 18 / 27

72 Big O notation Our measure of running time of an algorithm is the number of arithmetic operations needed to perform a calculation. 19 / 27

73 Big O notation Our measure of running time of an algorithm is the number of arithmetic operations needed to perform a calculation. Running time is measured as a function of a parameter n that describes the problem. 19 / 27

74 Big O notation Our measure of running time of an algorithm is the number of arithmetic operations needed to perform a calculation. Running time is measured as a function of a parameter n that describes the problem. We are usually more interested in the order of magnitude than exact operation count. 19 / 27

75 Big O notation Our measure of running time of an algorithm is the number of arithmetic operations needed to perform a calculation. Running time is measured as a function of a parameter n that describes the problem. We are usually more interested in the order of magnitude than exact operation count. We normally don t care if an algorithm uses 0.5n 2 or 20n 2 operations, but will consider a difference between n log(n) or n 4 operation significant. 19 / 27

76 Big O notation Our measure of running time of an algorithm is the number of arithmetic operations needed to perform a calculation. Running time is measured as a function of a parameter n that describes the problem. We are usually more interested in the order of magnitude than exact operation count. We normally don t care if an algorithm uses 0.5n 2 or 20n 2 operations, but will consider a difference between n log(n) or n 4 operation significant. Given function f(n) and g(n), we say that f(n) O(g(n)) or f(n) = O(g(n)), if there exists C > 0 and n 0, such that for n n 0 we have f(n) < C g(n). 19 / 27

77 Big O notation Our measure of running time of an algorithm is the number of arithmetic operations needed to perform a calculation. Running time is measured as a function of a parameter n that describes the problem. We are usually more interested in the order of magnitude than exact operation count. We normally don t care if an algorithm uses 0.5n 2 or 20n 2 operations, but will consider a difference between n log(n) or n 4 operation significant. Given function f(n) and g(n), we say that f(n) O(g(n)) or f(n) = O(g(n)), if there exists C > 0 and n 0, such that for n n 0 we have f(n) < C g(n). Examples: n log(n) = O(n 2 ), n n O(n 3 ). 19 / 27

78 Numerical Analysis and Manchester: The Baby The world s first stored-program computer, the Small Scale Experimental Machine (SSEM), also called The Baby, was developed at Manchester University in (Picture credit: MOSI) 20 / 27

79 Numerical Analysis and Manchester: Alan Turing Alan Turing not only developed the mathematical foundations of programming, he also did pioneering work in numerical analysis by introducing the concept of Condition Number. 21 / 27

80 Outline General Course Information Introduction to Numerical Analysis Prerequisites from Calculus

81 Intermediate Value Theorem Let f be continuous on [a, b]. Then f is bounded on [a, b] and if y satisfies inf f(x) y sup f(x), x [a,b] x [a,b] then there exists ξ [a, b] such that f(ξ) = y. In particular, the infimum and supremum are achieved. 22 / 27

82 Taylor s Theorem Let f be a function on [a, b], such that n derivatives of f exist and are continuous on [a, b]. Assume further that f (n) is differentiable on (a, b). Let x, x 0 be in [a, b]. Then there exists ξ (a, b) such that f(x) = f(x 0 ) + f (x 0 )(x x 0 ) f (x 0 )(x x 0 ) f (n) (x 0 ) (x x 0 ) n + f (n+1) (ξ) n! (n + 1)! (x x 0) n+1 23 / 27

83 Taylor Expansion: Example 4 sin(x) = x x3 3! + x5 5! Taylor expansion of sin(x). 24 / 27

84 Mean Value Theorem A special case is the Mean Value Theorem: Let f be continuous on [a, b] and differentiable on (a, b). Then there exists a number ξ (a, b) such that This can also be written as f(x) = f(x 0 ) + f (ξ)(x x 0 ). f (ξ) = f(x) f(x 0) x x / 27

85 Rolle s Theorem Yet another special case is Rolle s Theorem: If you end up at the same height at which you started, then the path wasn t all up or all down. Let f be continuous on [a, b] and differentiable on (a, b), and such that f(a) = f(b). Then there exists a number ξ (a, b) such that f (ξ) = / 27

86 Outlook Next class: 27 / 27

87 Outlook Next class: How to avoid rounding errors by clever algorithm design; 27 / 27

88 Outlook Next class: How to avoid rounding errors by clever algorithm design; Horner s method, fast evaluation of polynomials; 27 / 27

89 Outlook Next class: How to avoid rounding errors by clever algorithm design; Horner s method, fast evaluation of polynomials; Introduction to the Interpolation Problem. 27 / 27