Math Camp II. Calculus. Yiqing Xu. August 27, 2014 MIT

Transcription

1 Math Camp II Calculus Yiqing Xu MIT August 27, 2014

2 1 Sequence and Limit 2 Derivatives 3 OLS Asymptotics 4 Integrals

3 Sequence Definition A sequence {y n } = {y 1, y 2, y 3,..., y n } is an ordered set of real numbers, where y 1 is the first term in the sequence and y n is the nth term. Generally, a sequence extends to n =. We can also write the sequence as {y n } n=1. Sequences are similar to functions. Before, we had y = f (x) with x specified over some domain. Now we have {y n } = {f (x)} with each value of x having its own index, n = 1, 2, 3,.... Thus the first number we put into our function gives us y 1. Yiqing Xu (MIT) Calculus August 27, / 55

4 Sequence There are three kinds of sequences: 1 Sequences that converge to a limit. 2 Sequences that increase or decrease without bound. 3 Sequences like that neither converge nor increase without bound alternating over the number line. Yiqing Xu (MIT) Calculus August 27, / 55

5 The Limit of a Sequence Definition The sequence {y n } has the limit L, that is lim n y n = L, if for any ɛ > 0 there is an integer N (which depends on ɛ) with the property that y n L < ɛ for each n > N. {y n } is said to converge to L. If the above does not hold, then {y n } diverges. Uniqueness: If {y n } converges, then the limit L is unique. Yiqing Xu (MIT) Calculus August 27, / 55

6 Properties of Limits of Sequences Let lim n y n = A and lim n z n = B. Then lim [αy n + βz n ] = αa + βb n lim y nz n = AB n lim n y n z n = A B, provided B 0 Yiqing Xu (MIT) Calculus August 27, / 55

7 Limit Does a function f approach some number L as its input variable x goes to some number c (often 0 or ± )? If so f (x) approaches L as x approaches c. Formally we say lim x c f (x) = L. Definition (Limits of Functions) Given any ɛ > 0, δ > 0 s.t. for x in some domain D and within the neighborhood of x 0 of radius δ (except possibly x 0 itself), f (x) c < ɛ Uniqueness: lim x c f (x) = L and lim x c f (x) = M L = M Yiqing Xu (MIT) Calculus August 27, / 55

8 Properties of Limits of Functions Let f and g be functions with lim x c f (x) = A and lim x c g(x) = B. lim [f (x) + g(x)] = lim f (x) + lim g(x) = A + B x c x c x c What property does this look like that we saw before? lim αf (x) = α lim f (x) = αa x c x c lim x c f (x)g(x) = [ lim x c f (x) lim x c g(x) = lim f (x) x c lim f (x)][ lim g(x)] = AB x c x c g(x) = A B, provided B 0 Yiqing Xu (MIT) Calculus August 27, / 55

9 1 Sequence and Limit 2 Derivatives 3 OLS Asymptotics 4 Integrals

10 Derivative Definition The derivative of a function f (x) is simply the slope of the secant of f(x) at a pair of points very close to (x, f (x)): f f (x) f (a) f (a + h) f (a) (a) = lim = lim x a x a h 0 h Definition A straight line is tangent to a curve, at some point, if both line and curve pass through the point with the same direction; such a line is the best straight-line approximation to the curve at that point. Yiqing Xu (MIT) Calculus August 27, / 55

11 Calculating Derivatives Proposition f (x) = x k f (x) = kx k 1 Lemma (Binomial expansion) For any positive integer k, (x + h) k = x k + a 1 x k 1 h a k 1 x 1 h k 1 + a k h k, where a j = k! j!(k j)!, for j = 1,...k Yiqing Xu (MIT) Calculus August 27, / 55

12 Algebraic Operations of Derivatives Theorem Let f, g : X R be differentiable at c X and X R. Then, 1 (kf ) (c) = kf (c) for all k R, 2 (f + g) (c) = f (c) + g (c), 3 (fg) (c) = f (c)g(c) + f (c)g (c) (Product Rule), ( ) 4 f g (c) = f (c)g(c) f (c)g (c) for g(c) 0 (Quotient Rule). g(c) 2 Yiqing Xu (MIT) Calculus August 27, / 55

13 Chain Rule The Chain Rule is a formula for the derivative of the composite of two functions. Theorem Suppose h(x) = f g. Then h (x) = (f (g(x))) = f (g(x))g (x). or in the Leibniz notation: df dx = df du du dx. (Note: There should be some conditions for this theorem to work such as continuity of f and the existence of f. For now, let s assume all the necessary conditions are met.) Yiqing Xu (MIT) Calculus August 27, / 55

14 Characterizing a Continuously Differentiable Function 1 Calculate the first derivative 2 Find where derivative is equal to 0 by solving f (x) = 0. 3 Check the signs of f (x) at x to the left and right of these point(s) 4 If positive then increasing in that region, if negative then decreasing in that region Yiqing Xu (MIT) Calculus August 27, / 55

15 Increasing and Decreasing Functions Definition A function is increasing at x if its derivative is positive at x. That is if f (x) > 0 Definition A function is decreasing at x if its derivative is negative at x. That is if f (x) < 0 Yiqing Xu (MIT) Calculus August 27, / 55

16 The Second Derivative The derivative of the first derivative is the second derivative. We often use the notation f (x) or d2 y dx 2. The second derivative is the slope of the line tangent to the first derivative at the point x. We can think of this as the change in change of the function. In physics, the first derivative is the speed of an object while the second derivative is the acceleration of an object. Yiqing Xu (MIT) Calculus August 27, / 55

17 Convex and Concave Functions Definition A function is convex (or concave up) in a region if a secant line in any two points of the region is above f. Formally, the function f : A R, defined on the convex set A R n is convex if f (αx + (1 α)x) αf (x ) + (1 α)f (x) x and x A and all α [0, 1]. Definition A function is concave in a region if a secant line in any two points of the region is below f. Formally, the function f : A R, defined on the convex set A R n is concave if f (αx + (1 α)x) αf (x ) + (1 α)f (x) x and x A and all α [0, 1]. Yiqing Xu (MIT) Calculus August 27, / 55

18 Convex and Concave Functions To know if a function is convex, we do not need to graph it or figure out the slope of all secant lines through any two of its points. We just check if the second derivative is positive. Definition A function is convex in a region if f (x) > 0 in that region. Definition A function is concave in a region if f (x) < 0 in that region. Yiqing Xu (MIT) Calculus August 27, / 55

19 Graphing a Function 1 First find the points at which f (x) = 0 or f is not defined. Such points are called critical points of f. 2 Evaluate the function at each of these critical points and plot them in the graph. 3 Then, check the sign of f for each of the intervals defined by these critical points. 4 If f > 0 then draw the graph increasing over I, if f < 0 then draw the graph decreasing over I. 5 Find the points at which f (x) = 0 or f is not defined. Such points are called second order critical points of f, or if the second derivative actually changes sign there, inflection points of f. 6 Then, check the sign of f for each of the intervals defined by these critical points. 7 If f > 0 then draw the graph concave up (or convex) over I, if f < 0 then draw the graph concave down (or concave) over I. Yiqing Xu (MIT) Calculus August 27, / 55

20 Finding Maximums and Minimums All the (interior) maximums and minimums are found at critical points. The second derivative helps determine if a critical point is a maximum, a minimum, or neither. If f (x 0 ) = 0 and f (x 0 ) < 0 then x 0 is a maximum of f. If f (x 0 ) = 0 and f (x 0 ) > 0 then x 0 is a minimum of f. If f (x 0 ) = 0 and f (x 0 ) = 0 then we do not know, it might be a max, a min or neither (These are called saddle points). Yiqing Xu (MIT) Calculus August 27, / 55

21 Finding Maximums and Minimums 1 Take derivative 2 Find the x such that the derivative function= 0 3 Evaluate the second derivative at those critical points to determine if at that x there is a minimum or a maximum. Yiqing Xu (MIT) Calculus August 27, / 55

22 Derivatives of the Exponential Functions 1 d dx ln x = 1 x 2 d dx ln x k = d dx k ln x = k x 3 d dx ln u(x) = u (x) u(x) (by the chain rule) 4 d dx αex = αe x 5 d dx eu(x) = e u(x) u (x) Yiqing Xu (MIT) Calculus August 27, / 55

23 L Hospital s Rule Theorem Suppose f and g are differentiable on a < x < b and that either 1 lim x a 2 lim x a x a f (x) = 0 and lim g(x) = 0, or + + x a f (x) = ± and lim g(x) = ± + + Suppose further that g (x) is never zero on a < x < b and that f (x) lim x a + g (x) = L then f (x) lim x a + g(x) = L Yiqing Xu (MIT) Calculus August 27, / 55

24 Taylor s Expansion Theorem If f has derivatives of all orders in an open interval I containing a, then for each positive integer n and for each x in I, where f (x) = f (a) + f (a)(x a) + f (a) + 2! f n (a) (x a) n + R n (x) n! R n (x) = f (n+1) (c) (n + 1)! (x a)n+1 for some c between a and x. Yiqing Xu (MIT) Calculus August 27, / 55

25 Derivatives of Trigonometric Functions d d (sinx) = cosx dx d d (cosx) = sinx dx d dx (sinx) = sec2 x (cscx) = cscx cotx dx (cscx) = secx tanx dx d dx (cscx) = csc2 x Yiqing Xu (MIT) Calculus August 27, / 55

26 Calculus and Matrix Algebra A is n n matrix, x is a n 1 vector Quadratic form Ax x = A x Ax x = (A + A )x = 2Ax (if A is symmetric) Yiqing Xu (MIT) Calculus August 27, / 55

27 Positive-definite Matrices A symmetric n n matrix M is said to be positive definite if z Mz is positive for every non-zero column vector z of n real numbers Positive semi-definite matrices are defined in the same way, except that the expression z Mz is required to be always non-negative M = [ 1 ] Yiqing Xu (MIT) Calculus August 27, / 55

28 Jacobian Matrix A Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function. Suppose F : R n R m, F 1 F 1 x 1 x n J =..... F m x 1 F m x n F (x) = F (p) + J F (p)(x p) + o( x p ). Yiqing Xu (MIT) Calculus August 27, / 55

29 Hessian Matrix A Hessian matrix or Hessian is a square matrix of second-order partial derivatives of a function. Suppose F : R n R 1, 2 F 2 F 2 F x 2 1 x 1 x 2 x 1 x n 2 F 2 F 2 F H(F ) = x 2 x 1 x2 2 x 2 x n F 2 F 2 F x n x 1 x n x 2 xn 2 H(F )(x) = J( F )(x) y = F (x + x) F (x) + J(x) x xt H(x) x Yiqing Xu (MIT) Calculus August 27, / 55

30 1 Sequence and Limit 2 Derivatives 3 OLS Asymptotics 4 Integrals

31 OLS Setup Let X be our n (k + 1) data matrix for k predictors (plus the constant), y be the n 1 vector of the outcome variable, β be the (k + 1) 1 vector of true coefficient (intercept and k slopes), and u be the n 1 vector of error terms: Y = Xβ + u y = y 1 y 2. y n n 1 X = 1 x 11 x x 1k 1 x 21 x x 2k x n1 x n2... x nk n (k+1) Yiqing Xu (MIT) Calculus August 27, / 55

32 OLS Estimator S(b 0 ) = e e = (y Xb) (y Xb) = y y b 0X y y Xb 0 + b 0X Xb 0 = y y 2y Xb 0 + b 0X Xb 0 S(b 0 ) b 0 = 2X y + 2X Xb 0 = 0 ˆb = (X X) 1 X y Yiqing Xu (MIT) Calculus August 27, / 55

33 Properties of the OLS Estimator Properties Unbiasedness Consistency Asymptotic normality Preparation The Weak Law of Large Numbers (WLLN) Lindeberg-Levy Central Limit Theorem (CLT) Convergence in probability and in distribution Slutskty s Theorem Yiqing Xu (MIT) Calculus August 27, / 55

34 The Weak Law of Large Numbers (WLLN) Suppose {x i } is an infinite sequence of i.i.d. random variables with finite expected value E[x i ] = µ <. Then as n approaches infinity, the random variable x n = 1 n n i=1 x i converges in probability to µ. x n p µ, as n. Yiqing Xu (MIT) Calculus August 27, / 55

35 Lindeberg-Levy Central Limit Theorem (CLT) Suppose {x i } is an infinite sequence of i.i.d. random variables with finite expected value and variance, E[x i ] = µ < and Var(x i ) = σ 2 <. Then as n approaches infinity, the random variable n( x n µ) converges in distribution to a normal distribution N(0, σ 2 ): n( xn µ) d N(0, σ 2 ), as n. Yiqing Xu (MIT) Calculus August 27, / 55

36 Convergence in Probability Definition A sequence {X n } of random variables converges in probability towards the random variable X if for all ɛ > 0 lim n Pr ( X n X ε ) = 0, denoted as X n p X or plim X n = X. n Pick any ɛ > 0 and any δ > 0. Let P n be the probability that X n is outside the ball of radius δ centered at X. Then for X n to converge in probability to X there should exist a number N (which will depend on ɛ and δ) such that for all n N, P n < δ. Yiqing Xu (MIT) Calculus August 27, / 55

37 Convergence in Distribution Definition A sequence X 1, X 2,... of random variables is said to converge in distribution (or converge weakly, or converge in law) to a random variable X if lim n F n(x) = F (x), for every number x R at which F is continuous. Here F n and F are the cumulative distribution functions (CDFs) of random variables X n and X, respectively. Yiqing Xu (MIT) Calculus August 27, / 55

38 Slutskty s Theorem Let {x n }, {y n } be two sequences of random variables (or matrices), if {x n } converges in probability to a constant c and {y n } converges in distribution d to a random variable (or matrix) y, then x n y n cy. Yiqing Xu (MIT) Calculus August 27, / 55

39 Unbiasedness E[ˆb X] = E[ ( X X ) 1 X (Xb + u) X] (linearity ) = E[b + ( X X ) 1 X u X] (no perfect collinearity) = E[b X] + E[ ( X X ) 1 X u X] = b + ( X X ) 1 X E[u X] = b (zero conditional mean) E[ˆb] = b (random sampling) Assumptions: 1 linearity y = X b + u 2 random sampling of {X i, y i } 3 no perfect collinearity in X (therefore (X X) 1 exists) 4 zero conditional mean E[u X] = 0. Yiqing Xu (MIT) Calculus August 27, / 55

40 Consistency ˆb = b + ( X X ) 1 X u (linearity and no perfect collinearity) ( n ) 1 ( n ) = b + i=1 x ix i i=1 x iu i ( 1 ) 1 ( n 1 ) n = b + n i=1 x ix i n i=1 x iu i Applying the LLN to the sample means (random sampling): ( ) 1 n x p n ix i E[x ix i ] Q (k+1) (k+1) (positive definite Q exist) i=1 ( ) 1 n x p n iu i E[x iu i ] = 0 (zero conditional mean) i=1 Yiqing Xu (MIT) Calculus August 27, / 55

41 Consistency plim(ˆb) = b + Q 1 0 = b Assumptions: 1 linearity 2 random sampling 3 no perfect collinearity 4 zero conditional mean 5 the second moment of x i exists Note: If Q is positive definite, it is non-singular; therefore, Q 1 exist. E[x i u i ] = 0 since E[x i E[u i x i ]] = 0 by the Law of Iterated Expectations. Yiqing Xu (MIT) Calculus August 27, / 55

42 Asymptotic Normality ( 1 ) 1 ( n 1 ) n ˆb = b + n i=1 x ix i n i=1 x iu i (linearity and no perfect collinearity) ( 1 ) 1 [ ( n n 1 )] n n(ˆb b) = n i=1 x ix i n i=1 x iu i E[x ix i ] = Q E[x iu i ] = 0 (zero conditional mean) Var(x iu i ) = E x [Var(x iu i x i )] = E x [E[x iu i u ix i x i ]] E[Q 1 x iu i ] = 0 = σ 2 E x [x ix i ] (homoskedasticity) = σ 2 Q (a positive definite matrix Q exist) Var(Q 1 x iu i ) = σ 2 Q 1 QQ 1 = σ 2 Q 1 Yiqing Xu (MIT) Calculus August 27, / 55

43 Asymptotic Normality Applying the LLN to the sample mean of x i x i (random sampling): ( ) 1 n x p n ix i E[x ix i ] = Q i=1 Applying the CLT to the sample mean of x i u i (random sampling): ( ) 1 n n x d n iu i 0 N(0, σ 2 Q) i=1 Finally, applying the Slutsky s Theorem: n(ˆb b) d N(0, σ 2 Q 1 ). Yiqing Xu (MIT) Calculus August 27, / 55

44 Asymptotic Normality Assumptions: linearity random sampling no perfect collinearity no perfect collinearity zero conditional mean the second moment of x i exists homoscedasticity Yiqing Xu (MIT) Calculus August 27, / 55

45 1 Sequence and Limit 2 Derivatives 3 OLS Asymptotics 4 Integrals

46 Integration Integration is about calculating the area contained between a function and the axis Figure: Graph of f (x) = 2x + 1 Yiqing Xu (MIT) Calculus August 27, / 55

47 Riemann Integral If for a given function f the Riemann sum approaches a limit as x 0, then that limit is called the Riemann integral of f from a to b. Formally, b a f (x)dx = lim x 0 n f (x i ) x i=1 Yiqing Xu (MIT) Calculus August 27, / 55

48 Notations for Integral Yiqing Xu (MIT) Calculus August 27, / 55

49 Indefinite Integral Definition Indefinite Integral of a function f (x) is a function F (x) such that its derivative is f (x). ie, (F (x) = f (x)). It is denoted as: f (x)dx = F (x) + C Theorem (Uniqueness) If F and G are antiderivatives of f on some interval I, then there is a constant C such that F (x) = G(x) + C for all x in I. As a consequence of this theorem, we add the constant C to an indefinite integral. Yiqing Xu (MIT) Calculus August 27, / 55

50 Indefinite Integral Figure: Graph of f (x) = x and g(x) = x Yiqing Xu (MIT) Calculus August 27, / 55

51 First Fundamental Theorem of Calculus Definition Definite Integral in the interval [a,b] is the Indefinite Integral evaluated at a and b. Mathematically: b a f (x)dx Theorem Let the function f be bounded on [a, b] and continuous on (a, b). Let F be any function that is continuous on [a, b] such that F (x) = f (x) on (a, b). Then b a f (x)dx = F (b) F (a) Yiqing Xu (MIT) Calculus August 27, / 55

52 Second Fundamental Theorems of Calculus Theorem First Fundamental Theorem of Calculus: Let the function f be bounded on [a, b] and continuous on (a, b). Then the function x F (x) = f (s)ds, a a x b has a derivative at each point in (a, b) and F (x) = f (x), a < x < b Yiqing Xu (MIT) Calculus August 27, / 55

53 Computing integrals 1 Find an antiderivative of f : a function F such that F =f. 2 By the fundamental theorem of calculus, b f (x) dx = F (b) F (a). 3 Therefore the value of the integral is F (b) F (a). a Yiqing Xu (MIT) Calculus August 27, / 55

54 Rules Satisfied by Definite Integrals 1 Order of Integration: 2 Constant Multiple 3 Sum and Difference 4 Additivity b a a b b f (x)dx = f (x)dx a b a b (f (x) ± g(x))dx = b a a kf (x)dx = k f (x)dx = b a b a b a f (x)dx f (x)dx b f (x)dx ± g(x)dx a c c f (x)dx + f (x)dx = f (x)dx b a Yiqing Xu (MIT) Calculus August 27, / 55

55 Integration by Substitution Theorem If u = g(x) is a differentiable function whose range is an interval I and f is continuous on I, then f (g(x))g (x)dx = f (u)du Yiqing Xu (MIT) Calculus August 27, / 55

56 Substitution in Definite Integrals Theorem If g is continuous on the interval [a, b] and f is continuous on the range of g, then b a f (g(x)) g (x)dx = g(b) g(a) f (u)du Yiqing Xu (MIT) Calculus August 27, / 55

57 Integration by Parts Theorem (Integration by Parts) Let f, g : [a, b] R be differentiable on [a, b] with f, g being Riemann-integrable on [a, b]. Then, f g and g f are also Riemann-integrable on [a, b], and b a b f (x)g(x) dx = [f (b)g(b) f (a)g(a)] g (x)f (x) dx. a For indefinite integrals, we have f (x)g(x) dx = f (x)g(x) g (x)f (x) dx. Yiqing Xu (MIT) Calculus August 27, / 55

58 Leibniz s Rule Theorem If f is continuous on [a, b] and if u(x) and v(x) are differentiable functions of x whose values lie in [a, b], then d dx v(x) u(x) f (t)dt = f (v(x)) dv dx f (u(x))du dx Yiqing Xu (MIT) Calculus August 27, / 55

59 Yiqing Xu (MIT) Calculus August 27, / 55

60 Thank you! Hope you enjoyed it! Yiqing Xu (MIT) Calculus August 27, / 55