Recitation 3: Applications of the Derivative. 1 Higher-Order Derivatives and their Applications

Transcription

1 Mth 1c TA: Pdric Brtlett Recittion 3: Applictions of the Derivtive Week 3 Cltech Higher-Order Derivtives nd their Applictions Another thing we could wnt to do with the derivtive, motivted by wht we were ble to do in R 1, is the concept of higher-order derivtives These re reltively esy to define for prtil derivtives: Definition Given function f : R n R, we cn define its second-order prtil derivtives s the following: = ( ) f x i x j x i x j In other words, the second-order prtil derivtives re simply ll of the functions you cn get by tking two consecutive prtil derivtives of your function f A useful theorem for clculting these prtil derivtives is the following: Theorem 1 A function f : R n R is clled C t some point if ll of its second-order prtil derivtives re continuous t tht point If function is C, then the order in which second-order prtil derivtives re clculted doesn t mtter: ie for ny i, j x i x j = f x j x i, It bers noting tht if the conditions of this theorem re not met, then the order for computing second-order prtil derivtives my ctully mtter! One such exmple is the function { x 3 y xy 3, (x, y) (0, 0) f(x, y) = x +y 0, (x, y) = (0, 0) At (0, 0), you cn clculte tht f x y = 1 1 = f y x : result tht occurs becuse the second-order prtils of this function re not continuous However, the interesting spects of higher-order prtil derivtives re not relly in their clcultion; rther, the pplictions of higher-order prtil derivtives re the things worth studying In R, for exmple, we could turn the second derivtive of function into lot of informtion bout tht function: in prticulr, we could use this second derivtive to determine whether given criticl point ws locl minim or mxim, 1

2 whether function is concve up or down t given point, nd wht the second-order Tylor pproximtion to tht function ws t point Cn we do the sme for functions from R n to R? As it turns out, the nswer is yes! The tool with which we do this is clled the Hessin, which we define here: Definition The Hessin of function f : R n R t some point is the following mtrix: H(f) x 1 x 1 () f x 1 x n () = x n x 1 () f x n x n () The min useful property of the Hessin is the following: Theorem Let f : R n R be function with well-defined second-order prtils t some point Using the Hessin of f, construct the following function: H(x) = 1 (x 1, x n ) H(f) (x 1, x n ) T Pick ny two co ordintes x i, x j in R n : then x i x j () = H h i h j In other words, H s second-order prtil derivtive re precisely the second-order prtil derivtives of f t! o H is bsiclly function designed to hve the sme second-order prtils s f t One quick thing this theorem suggests is tht we could use H(f) to crete secondorder pproximtion to f t, in similr fshion to how we used the derivtive to crete liner (ie first-order) pproximtion to f We define this below: Theorem 3 If f : R n R is function with continuous second-order prtils, we define the second-order Tylor pproximtion to f t s the function T (f) ( + h) = f() + ( f)() h + 1 (h 1, h n ) H(f) (h 1, h n ) T You cn think of f() s the constnt, or zero-th order prt, ( f)() h s the liner prt, nd H(f) (h) s the second-order prt of this pproximtion To illustrte how this process ctully cretes pretty decent pproximtion to f, we clculte n exmple: Exmple Clculte the second-order Tylor pproximtion to the function f(x, y) = e xy t the origin Answer Clculting the second derivtives of f is pretty strightforwrd: f x = yexy, f y = xexy x = y e xy, f y = x e xy, x y = xyexy + e xy = f y x

3 If we evlute these prtils t 0 nd plug them into the definition bove for T (f) (0,0), we get T (f) (0,0) ((0, 0) + (h 1, h )) = f(0, 0) + ( f)(0, 0) (h 1, h ) + H(f) (0,0) (h 1, h ) = 1 + (0, 0) (h 1, h ) + 1 ( ) ( ) 0 1 (h h1 1, h ) 1 0 h = ( ) (h h 1, h ) h 1 = (h 1h ) = 1 + h 1 h o, t the origin, the second-order Tylor pproximtion for f is just T (f)(x, y) = 1 + xy The following grph, with e xy in solid red nd T in dshed blue, shows tht it s ctully somewht decent pproximtion t (0, 0): z y x As well, we cn use the second derivtives to serch for nd find locl minim nd mxim! We define these terms here: Definition A point R n is clled locl mxim of function f : R n R iff there is some smll vlue r such tht for ny point x in B (r) not equl to, we hve f(x) f() A similr definition holds for locl minim o: how cn we use the derivtive to find such locl mxim? Well, it s cler tht (if our function is differentible in neighborhood round this point) tht no mtter how we move to leve this point, our function must not increse in other words, for ny direction v R n, the directionl derivtive f (, v) must be 0 But this mens tht in fct ll of 3

4 the directionl derivtives must be equl to 0!, becuse if f (, v) ws < 0, then f (, v) would be > 0 This motivtes the following definitions, nd bsiclly proves the following theorem: Definition A point is clled sttionry point of some function f : R n R iff (f) = (0,, 0) A point is clled criticl point if it is sttionry point, or f is not differentible in ny neighborhood of Theorem 4 A function f : R n R ttins its locl mxim nd minim only t criticl points However, it bers noting tht not every criticl or sttionry point is locl mxim or minim! A trivil exmple would be the function f(x, y) = x y : the origin is sttionry point, yet neither locl minim or mxim (s f(0, ɛ) < 0 < f(ɛ, 0), nd thus there re positive nd negtive vlues of f ttined in ny bll round the origin, where it is 0) How cn we tell which sttionry points do wht? Well, in one-vrible clculus, we used the ide of the second derivtive to determine wht ws going on! In specific, we knew tht if the second derivtive of function f t some point ws negtive, then tiny increses in our vrible t tht point would cuse the first derivtive to decrese, nd tiny decreses in our vrible t tht point would cuse the negtive of the first derivtive to increse ie cuse the first derivtive to decrese, nd therefore mke the function itself decrese! Therefore, the second derivtive being negtive t sttionry point implied tht tht point ws locl mxim In higher dimensions, things re tricker t given point, we no longer hve this ide of single second derivtive, but insted hve mny different second derivtives, like x y () nd f () Yet, we cn still use the sme ides s before to figure out wht s going z on! In prticulr, in one dimension, we sid tht we wnted tiny positive chnges of our vribles to mke the first functions decrese In other words, given ny of the prtils f x i, we wnt ny positive chnges in the direction of this prtil to mke our function decrese ie we wnt the directionl derivtive of f x i to be negtive in ny direction v, where ll of the coördintes of v re positive (Positivity here stems from the sme reson tht in one dimension, we hve tht the first derivtive is incresing for ll of the points to the left of mxim nd decresing for ll of the points to the right of mxim) o: this condition, if we write it out, is just sking tht for every i nd nonzero v, tht ( x 1 x i (), x x i (), x n x i () ) (v1, v, vn) is negtive If you choose to write this out s mtrix, this ctully becomes the clim tht for ny v 0, we hve x v T 1 x 1 () f x 1 x n () v < 0 x n x 1 () f x n x n () 4

5 liner lgebr, you my hopefully remember tht ny mtrix stisfying this condition is clled being negtive-definite, nd is equivlent to hving ll n of your eigenvlues existing nd being negtive imilrly, if we were looking for locl minim, we would be sking tht the bove mtrix product is lwys positive: ie tht the mtrix is positive-definite, which is equivlent to ll of its eigenvlues being positive But we ve seen this construction before! In prticulr, this mtrix thing bove is just the Hessin! Bsed on these observtions, we mke the following definitions nd observtions: Definition The Hessin H(f) is positive-definite if nd only if the mtrix x 1 x 1 () x 1 x n () x n x 1 () f x n x n () is positive-definite (The sme reltion holds for being negtive-definite) Recll from Mth 1 tht mtrix is positive-definite if nd only if ll of its eigenvlues re rel nd positive imilrly, mtrix is negtive-definite if nd only if ll of its eigenvlues re rel nd negtive If some of mtrix s eigenvlues re 0, some re negtive nd others re positive, or if there re less rel eigenvlues thn the rnk of the mtrix (ie some eigenvlues re complex,) then the mtrix is neither positive-definite or negtive-definite Note lso tht becuse the Hessin is symmetric whenever the mixed prtils of our function re equl, nd symmetric mtrices hve only rel eigenvlues, you relly should never get complex-vlued eigenvlues Theorem 5 A function f : R n R hs locl mxim t sttionry point if ll of its second-order prtils exist nd re continuous in neighborhood of, nd the Hesssin of f is negtive-definite t imilrly, it hs locl minim if the Hessin is positive-definite t If the Hessin tkes on both positive nd negtive vlues there, it s sddle point: there re directions you cn trvel where your function increse, nd others where it will decrese Finlly, if the Hessin is identiclly 0, you hve no informtion s to wht your function my be up to: you could be in ny of the three bove cses A quick exmple, to illustrte how this gets used: Exmple For f(x, y) = x + y, g(x, y) = x y, nd h(x, y) = x y, find locl minim nd mxim olution First, by tking prtils, it is cler tht the only point t which the grdient of these functions is (0, 0) is the origin There, we hve tht H(f) = (0,0) [ 0 0 ], H(g) = (0,0) [ 0 0 ], H(h) = (0,0) [ 0 0 nd thus tht f is positive-definite t (0,0), g is negtive-definite t (0,0), nd h is neither t (0,0) by exmining the eigenvlues Thus f hs locl minim t (0, 0), g hs locl mxim t (0,0), nd h hs sddle point t (0,0) ], 5

6 Lgrnge Multipliers 1 ttement of the method In the section bove, we tlked bout how to use derivtives to find nd clssify the criticl points of functions R n R This llows us to find the globl minim nd mxim of functions over ll of R n, if we wnt! Often, however, we won t just be looking to find the mximum of some function on ll of R n : sometimes, we ll wnt to mximize function given set of constrints For exmple, we might wnt to mximize the function f(x, y, z) = x+y subject to the constrint tht we re looking t points where x + y = 1 How cn we do this? Initilly, you might be tempted to just try to use our erlier methods: ie look for plces where Df is 0, nd try to clssify these extrem The problem with this method, when we hve set of constrints, is tht it usully won t find the mxim or minim on this constrint: becuse it s only looking for locl mxim or minim over ll of R n, it will ignore points tht could be mxim or minim on our constrined surfce! Ie for the f, g we mentioned bove, we know tht (f) = (1, 1), which is never 0; however, we cn esily see by grphing tht f(x, y) = x + y should hve mximum vlue on the set x + y = 1, specificlly t x = y = 1 o: how cn we find these mxim nd minim in generl? The nswer is the method of Lgrnge multipliers, which we outline here: Proposition 6 uppose tht f : R n R is function whose extreml vlues {x} we would like to find, given the constrints g(x) = c, for some constrining function g(x) Then, we hve the following result: if is n extreml vlue of f restricted to the set = {x : i, g(x) = c}, then either one of (f) is 0, doesn t exist, or there is some constnt λ such tht (f) = λ (g) Why? In this cse, it s worth tlking little bit bout why this result hppens to work, s understnding the proof of the bove proposition is remrkbly useful for using it Consider, gin, the exmple we discussed erlier, where we hve f(x, y) = x + y g(x, y) = x + y = 1 (the function we would like to mximize), (our constrining function) Let = {(x, y) : g(x, y) = 1} In this nottion, we re looking to mximize the function f restricted to the set, which we denote f Wht do we know bout f? Well: if is mximum, we would expect to be criticl point of f The only issue is tht we don t hve ny wy to esily refer to just f : we cn tlk bout f in generl, but if we don t restrict it to we wouldn t expect to still be mximum One wy round this is to think bout pths pecificlly, pick ny pth γ such tht γ s imge is constined entirely within, nd γ(0) = Then, if we look t f γ, we know 6

7 tht this is function from R R; therefore, if f hs mximum t, f γ lso must hve mximum, s it s just pth contined entirely in tht goes through this supposed mximum point In other words, we hve (f γ) t=0 = 0 (f) γ (0) = 0; ie (f) is orthogonl to γ (0), for ny pth γ in, going through 0 But these γ (0) s re just ll of the possible tngent vectors to t : so we hve tht (f) is orthogonl to ll of these tngent vectors! imilrly, we know tht for ny such pth γ, we hve tht g γ is constnt, becuse g is constnt on But this mens tht (becuse the derivtive of ny constnt is 0) (g γ) t=0 = 0 (g) γ (0) = 0 In other words, (g) is lso orthogonl to ll of s tngent vectors t! T() = (f)= y x But is spce formed by plcing one constrint on function of n vribles: in other words, it s n 1-dimensionl spce! Therefore, t the point, the collection of tngent vectors to t is n 1-dimensionl spce, contined in R n But this mens tht the spce of ll vectors orthogonl to this (n 1)-dimensionl spce is 1-dimensionl spce! In specific, we ve just shown tht both (f) nd (g) re contined in the sme 1- dimensionl spce: ie tht one of them is multiple of the other! In other words, we ve shown tht becuse they re both orthogonl to the entire tngent spce to t, there is some λ such tht (Or one of them is 0, or undefined) (f) = λ (g) 7

8 In the very specific cse we re working with where we hve f(x, y) = x + y, g(x, y) = x + y = 1, (f(x, y)) = (1, 1), (g(x, y)) = (x, y) nd we re looking for points where either of these grdients re 0, or where there is some λ such tht (f(x, y)) = (1, 1) = λ (g(x, y)) = (λx, λy) 1 = λx, 1 = λy 1 λ = x, 1 λ = y x = y o: we hve either (0, 0), s this forces (x, y) = (0, 0), or points (x, y) where x = y The first is impossible if we re looking t points where g(x, y) = x + y = 1; for the second, we would hve x + x = 1, ie x = y = ± 1 ( ) ( We ve therefore discovered the two possible extreml points of f: 1 1,, 1, 1 ) In other words, we know tht these points re the possible locl mxim nd minim of f How do we tell whether these points re ctully globl minim nd mxim? The nswer is in the following brief definitions nd theorem: Definition A set R n is clled closed if it contins ll of its limit points: ie {x n } n=1,, nd lim n x n = L, then L if Definition A set R n is clled bounded if there is some M such tht x < M, for every x Lemm 7 If g : R n R is continuous function nd c is ny constnt, then the set = {x : g(x) = c} is closed Theorem 8 If f is continuous function nd we restrict f to closed nd bounded set, then f will hit its globl minim nd mxim on, nd furthermore do this t criticl points: ie plces where D(f ) is 0 Corollry 9 uppose tht f : R n R is some function we wnt to mximize/minimize, g : R n R is some constrint function, is the constrined set {x : g(x) = c}, nd is bounded set Then the bsolute mxim nd minim of g cn ll be found vi the method of Lgrnge multipliers: ie the mxim nd minim of f will come up in the extreml points tht the method of Lgrnge multipliers finds 8

9 ( As ) result ( of this theorem, ) we know tht in our exmple erlier, one of the two points 1 1,, 1, 1 is the mximum of f, while the other must be its minimum By plugging in both vlues to f, we cn see tht the first is the mximum, nd the second is the minimum To illustrte the power nd verstility of the method of Lgrnge multipliers, nd to help you get better feel for how they work in prctice, we work two exmples using the tools we ve just developed: Exmple Consider the stroid, curve in R formed by the eqution x /3 + y /3 = 1 Wht points on this curve re the closest to the origin? olution We wnt to minimize the distnce function given the constrint f(x, y) = x + y g(x, y) = x /3 + y /3 = 1 Using the method of Lgrnge multipliers, we know tht these miniml points will be those for which either (f) or (g) re undefined, or such tht there is some λ such tht o: clculting, we cn see tht (f) = (f) = λ (g) ( ) x x + y, y, x + y which is defined whenever (x, y) (0, 0), nd ( (g) = 3 x 1/3, ) 3 y 1/3,, which is defined whenever x 0 nd y 0 When either x or y = 0, we know tht (in order to stisfy x /3 +y /3 = 1) the other vlue hs to be ±1; so we immeditely know tht we should look t the four points (±1, 0), (0, ±1) when we re looking for our extreml points Aprt from these loctions, we know tht both of these grdients re well-defined nd nonzero; so we re looking for vlues x, y, λ such tht ( ) ( x (f) = x + y, y λ = λ (g) = x + y 3 x 1/3, λ ) 3 y 1/3, 9

10 By equting these two vectors, we re just trying to solve the two equtions x x + y = λ 3 x 1/3, y x + y = λ 3 y 1/3 1 x + y = λ 3 x 4/3, 1 x + y = λ 3 y 4/3 x 4/3 = y 4/3 x = ±y ( ) The only points tht stisfy x = ±y nd lso x /3 + y /3 re the four points ± 1, ± 1 Combining these with the (±1, 0), (0, ±1) points we discovered erlier, we hve eight possible extreml points Plugging these into f(x) gives us 1 for the points with x = ±y nd 1 for the points with one of x, y = 0 Becuse the mximum nd minimum vlues of f( occur on these ) points, we know tht the closest points to the origin re precisely the points ± 1, ± 1 Exmple Consider the following rough model for the economics of pie-bking: Your ingredients for pie re pples (), butter (b), flour (f), nd sugr (s) uppose tht pples cost $/unit, butter costs $3/unit, flour costs $1/unit, nd sugr costs $1/unit uppose tht if you hve units of pples, b units of butter, f units of flour, nd s units of sugr, you cn mke roughly 4 bfs- mny pies (This is not n entirely implusible guess for function tht tells you how mny pies you cn mke: in prticulr, you wnt function tht (1) is 0 whenever you don t hve one of your ingredients, which tking the product of ll of your ingredients does for you, nd () grows linerly if you increse the quntity of ech of your ingredients linerly [ie if you hve k units of ech ingredient, this sys you cn mke k pies, which seems ccurte] The formul lso llows you to slightly skew the ingredient proportions of your pies: if pples re relly expensive, you cn hve pies tht hve more dough to pples, wheres if sugr gets relly expensive you cn just increse the pple rtio) Finlly, suppose you strt with 100 units of currency, nd tht you cnnot hve negtive mount of ny of our ingredients (ie, b, f, s 1) Wht is the mximum number of pies you cn mke? olution This is bit different thn our erlier problems In prticulr, insted of just optimizing the function F (, b, f, s) = 4 bfs 10

11 on one constrint, we re optimizing it over the inequlities + 3b + f + s 100, 0, b 0, f 0, s 0 How cn we do this with Lgrnge multipliers? Well: to do this, we just need to consider severl cses pecificlly, suppose we hve some mximum point (, b, f, s) There re two possibilities: 1 This mximum point occurs on the interior of the set formed by our constrints + 3b + f + s 100, 0, b 0, f 0, s 0 Therefore, this point cn be found by looking t D(f), s it s locl mximum of f without ny constrints! Otherwise, this mximum point occurs on the boundry of the set formed by the constrints + 3b + f + s 100, 0, b 0, f 0, s 0 In other words, this mximum point occurs when we hve either g(, b, f, s) = + 3b + f + s = 100, or when one of the four quntities, b, f, s re 0 We cn eliminte the cses where we hve 0 of ny of our quntity by just noticing tht this trivilly restricts us to mking 0 pies, which is clerly not mximum; this leves us with just the bove single constrint But this is exctly the sitution tht Lgrnge multipliers re set up to del with! In prticulr, we cn use Lgrnge multipliers to mximize 4 bfs with respect to the constrint g(, b, f, s) = + 3b + f + s = 100 Compring ll of the criticl points we find in these wys will yield the overll mximum of 4 bfs on our entire set We perform these clcultions here First, becuse ( (F ) = bfs 4(bfs) 3/4, fs 4(bfs) 3/4, bs 4(bfs) 3/4, ) bf 4(bfs) 3/4, we cn see tht the grdient of F is only undefined or zero t plces where some of its coördintes re zero or negtive Becuse our conditions require tht, b, f, s 0, nd in the cse tht ny quntity is 0 we know tht no pies re mde, we know tht this is impossible: therefore, we don t hve to worry bout f hving ny mxim on the interior of our set of constrints Now, we turn to the constrint g(, b, f, s) = + 3b + f + s = 100 Using Lgrnge multipliers, we know tht criticl points will occur where ( ) bfs (F ) = 4(bfs) 3/4, fs 4(bfs) 3/4, bs 4(bfs) 3/4, bf 4(bfs) 3/4 = λ (g) = (λ, 3λ, λ, λ) This occurs t points tht stisfy the four equtions bfs fs = λ, 4(bfs) 3/4 bs = 3λ, 4(bfs) 3/4 bf = λ, 4(bfs) 3/4 = λ 4(bfs) 3/4 11

12 By combining the first two equtions, we cn see tht bfs fs = λ, = 3λ 4(bfs) 3/4 4(bfs) 3/4 ( b = 4λ 1 ) ( fs (bfs)3/4, = 3 4λ 1 ) fs (bfs)3/4 b = 3 imilrly, we cn combine the middle two equtions to get fs bs = 3λ, 4(bfs) 3/4 4(bfs) 3/4 = λ f = 3 (4λ 1s ) (bfs)3/4, b = (4λ 1s ) (bfs)3/4 f 3 = b, nd(similrly) the lst two equtions to get f = s Combining these results, we cn write our point s (, 3,, ), nd get tht + 3b + f + s = = 100 = 15, (, 3,, ) = (15, 83, 5, 5) With these ingredient rtios, we cn mke ) = 16 pies; s this is the only criticl point tht gives nonzero vlue, we know tht it must be our mximum 1