Analysis/Calculus Review Day 3

Size: px

Start display at page:

Download "Analysis/Calculus Review Day 3"

Shana Allen
5 years ago
Views:

1 Analysis/Calculus Review Day 3 Arvind Saibaba arvindks@stanford.edu Institute of Computational and Mathematical Engineering Stanford University September 15, 2010

2 Big- Oh and Little- Oh Notation We write f = O(g) if g(x) > 0 for x sufficiently large and if f(x)/g(x) is bounded for x sufficiently large. Write f = o(g) if f/g goes to zero as x goes to +. Also, write f g (f is asymptotic to g) if f/g 1 as x. Note that f g implies that f = O(g). Examples Show that x 2 +x = O(x 2 ) First note that (x 2 +x)/x = 1+1/x goes to 1 as x. Therefore, x 2 +x x 2 and it follows that x 2 +x = O(x 2 ).

3 Taylor s Series Suppose f is a real valued function on [a,b], n is a positive integer, f (n 1) is continuous on [a,b] and f (n) (t) exists for every t ]a,b[. Let α,β be distinct points in [a,b], and define P(t) = n 1 k=0 f (k) (α) (t α) k k! Then there exists a point x between α and beta such that f(β) = P(β)+ f (n) (β α)n n! For n = 1 this is nothing but the mean value theorem. In general the theorem shows that f can be approximated by a polynomial of degree n 1 and the above equation allows us to estimate the error, if we know a prioiri, bounds on f (n) (x).

4 Taylor Series The most common way of stating the Taylor s theorem is where, f(x +h) = f(x)+f (x)h+ f (x)h 2 2! + + f (n) (x)h n n! R n = f (n+1) (c) (n+1)! hn+1 for some x c x +h +R n (h) The function R n is called the remainder of order n or the error term. If R n 0 as n, we say that the Taylor series generated by f converges and write f(x +h) = k=0 f (k) (x) h k k!

5 Maclaurin series The Maclaurin series generated by f is k=0 That is the Taylor series at x = 0. f (k) (0) x k k! Example Compute the Maclaurin series generated by e x. Solution We observe that f (x) = e x, f (x) = e x,..., f (k) = e x Plugging these expressions into the above formula and noting that e 0 = 1, we have that e x x k = k! k=0

6 Common Maclaurin series For 1 x < 1 sinx = x x3 3! + x5 5! = k=0 cosx = 1 x2 2! + x4 4! = k=0 ( 1) k x 2k+1 (2k +1)! ( 1) k x 2k (2k)! log(1+x) = x x2 2 + x3 3 = k=0 ( 1) k+1 x k sinhx = x + x3 3! + x5 5! + = x 2k+1 (2k +1)! k=0 k

7 Max-Min Theorem Theorem If f is a continuous real function on a closed set X, then there exist p,q X such that f(p) = supf(x) and f(q) = inf f(x) x X x X Theorem If f :]a,b[ R is differentiable at c ]a,b[ and f has a maximum (respectively minimum) at c, then f (c) = 0. Proof. Let f have a maximum at c. Then for h 0, [f(c +h) f(c)]/h 0 So, letting h 0,h 0 we get f (c) 0. Similarly, for h 0 we obtain f (c) 0. Hence, f (c) = 0.

8 Rolle s Theorem Theorem If f : [a,b] R is continuous, f is differentiable on ]a, b[ and f(a) = f(b) = 0 then there is a number c ]a,b[ such that f (c) = 0.

9 Proof of Rolle s theorem Theorem If f : [a,b] R is continuous, f is differentiable on ]a,b[ and f(a) = f(b) = 0 then there is a number c ]a,b[ such that f (c) = 0. Proof. If f(x) = 0 for all x [a,b] we can choose any c and we are done. So assume that f is not identically zero. For the min-max theorem we know that there is a point c 1 where f assumes its maximum and a point c 2 where f assumes its minimum. By our assumption and the fact that f(a) = f(b) = 0 at least one of c 1,c 2 ]a,b[. If c 1 ]a,b[ we have f (c 1 ) = 0; similarly for c 2.

10 Mean Value Theorem Theorem If f : [a,b] R is continuous, f is differentiable on ]a,b[, there is a point c ]a,b[ such that f(b) f(a) = f (c)(b a) Apply Rolle s theorem on Corollary If f is differentiable on ]a,b[ and f (x) = 0 for all x ]a,b[ then f is a constant on ]a,b[. φ(x) = f(x) f(a) (x a) f(b) f(a) b a

11 First derivative test

12 Necessary and Sufficient conditions

13 Gradient and Hessian Gradient: f(x) = f x 1 f x 2. f x n f def f(x +ǫe i ) f(x) = lim x i ǫ 0 ǫ Hessian: 2 f(x) = 2 f x1 2 2 f x 2 x 1. 2 f x n x 1 2 f x 1 x f x 1 x n 2 f... 2 f x2 2 x 2 x n f x n x f xn 2

14 Directional Derivative Let f be real valued, defined in a neighbourhood of x 0 R n and e R n be a unit vector. Then d dt f(x f(x 0 +te) f(x 0 ) 0 +te) t=0 = lim t 0 t is called the directional derivative of f at x 0 in the direction e.

15 Let f : A R be twice differentiable on the open set A with 2 f x i x j are continuous then 2 f = 2 f x i x j x j x i which implies that the Hessian is symmetric. Example Suppose f : R 3 R = e xy sinx +x 2 y 4 cos 2 z. Then f x f y = = e xy cosx +ye xy sinx +2xy 4 cos 2 z = = xe xy sinx +e xy sinx +4x 2 y 3 cos 2 z and 2 f x y = 2 f y x = xexy cosx +e xy sinx +xye xy sinx +8xy 3 cos 2 z

16 Implicit Functions Suppose that x and y are related by an equation F(x,y) = 0. We would like to say that this defines a function y = f(x) which is defined implicitly and be able to compute derivatives dy/dx. Consider the circle F(x,y) = x 2 +y 2 1 = 0. A function f(x) is a solution if f(x,f(x)) = 0 for all x in the domain. Clearly, in this case, f(x) = ± 1 x 2. We note that f need not be unique. Also, f is not differentiable at ±1. In general, consider F : R n R m R m and consider F(x,y) = 0 F 1 (x 1,...,x n,y 1,...,y m ) = 0.. F m (x 1,...,x n,y 1,...,y m ) = 0

17 Implicit function theorem Let A R n R m be an open subset and let f : A R m have p continuous derivatives. Suppose that (x 0,y 0 ) A and F(x 0,y 0 ) = 0. Form, F 1 y 1... =. F m y 1... F 1 y m. F m y m evaluated at (x 0,y 0 ). Suppose 0. There is an open neighbourhood U of x 0 and V of y 0 and a unique function f : U V such that F(x,f(x) = 0 for all x U. Further, it has p continuous derivatives.

18 From the equation F(x,f(x)) = 0, if m = n = 1, using Chain rule 0 = F F(x,f(x)) = x x + F f y x Therefore, we obtain In general, f 1 x f n x 1... f 1 x n. f m x n = f x = F/ x F/ y F 1 y F m y 1... F 1 y m. F m y m 1 F 1 x F m x 1... F 1 x n. F m x n

19 Notation Notation: min x R nf(x) subject to c i (x) = 0, i E, c i (x) 0, i I x unknowns, vector of variables f(x) Objective function c i Constraint functions I Index set of inequality constraints E Index set of equality constraints

20 Unconstrained Optimization min x R nf(x) f : Rn R A point x, with a neighbourhood N, is called Global minimizer, if f(x ) f(x) for all x Local minimizer, if f(x ) f(x), for all x N Strong local minimizer, if f(x ) < f(x), for all x N, with x x A neighbourhood N of x is an open set that contains x.

21 Linear Algebra recap Definition (Postive Definite Matrices) A square matrix A is said to be positive definite if x T Ax > 0 for all x R n,x 0 An equivalent definition for a positive definite matrix, is a matrix that has only positive eigenvalues. Practical way to test positive definiteness is to compute eigenvalues, or perform Cholesky factorization. Definition (Positive Semidefinite Matrices) A square matrix A is said to be positive semidefinite if x T Ax 0 for all x R n

22 Consider the 2 2 system Example A = Positive definite means that [ ][ ][ ] x a b x y b d y [ ] a b c d > 0 (x,y) (0,0) That is ax 2 +2bxy +dy 2 > 0. Supposing that this true for all (x,y) (0,0). Setting x = 1,y = 0 implies a > 0. Now set y = 1 we have ax 2 +2bx +d > 0 for all x. This function is a parabola with a minimum (since a > 0) at 2ax +2b = 0, that is x = b/a. Hence, ( a b 2 ( +2b a) b ) +d > 0 a that is, ad b 2 > 0. The converse maybe proved similarly.

23 Taylor s Theorem Theorem (Taylor s theorem) Suppose that f : R n R is continuously differentiable and that p R n. Then we have that f(x +p) = f(x)+ f(x +tp) T p, for some t (0,1). Moreover, if f is twice continuously differentiable, we have that and that f(x +p) = f(x) f(x +tp)p dt, for some t (0,1). f(x +p) = f(x)+ f(x) T p pt 2 f(x +tp)p

24 Necessary Conditions Theorem (First Order Necessary Conditions) If x is a local minimizer and f is continuously differentiable in an open neighbourhood of x, then f(x ) = 0. Theorem (Second Order Necessary Conditions) If x is a local minimizer and 2 f exists in an open neighbourhood of x, then f(x ) = 0 and 2 f(x ) is positive semidefinite. We call x a stationary point if f(x ) = 0.

25 Example Consider the function f(x,y) = x 2 +y 2. The domain of f is the entire plane and the partial derivatives f x = 2x and f y = 2y exist everywhere and the Hessian [ ] f = 0 2 The local extreme values can only occur when f x = 2x = 0 and f y = 2y = 0 that is at the origin. Also the Hessian is positive definite everywhere.

26 Example Consider the function f(x,y) = xy. The domain of f is the entire plane and the partial derivatives f x = y and f y = x exist everywhere and the Hessian [ ] f = 1 0 The local extreme values can only occur when f x = y = 0 and f y = x = 0 that is at the origin. The Hessian is indefinite and therefore no extreme

27 Lagrange Multipliers Theorem Let f : U R n R and g : U R n R be continuously differentiable functions. Let x 0 U, g(x 0 ) = c 0 and let S = g 1 (c 0 ), the level set for g with the value c 0. Assume g(x 0 ) 0. If f S has a maximum or minimum at x 0 and there is a real number λ such that f(x 0 ) = λ g(x 0 ) Corollary If f when constrained to the surface S has a maximum or minimum at x 0, then f(x 0 ) is perpendicular at x 0

Analysis/Calculus Review Day 2

Analysis/Calculus Review Day 2 AJ Friend ajfriend@stanford.edu Arvind Saibaba arvindks@stanford.edu Institute of Computational and Mathematical Engineering Stanford University September 20, 2011 Continuity