Sherif Khalifa Sherif Khalifa () Optimization 1 / 50
Y f(x 0 ) Y=f(X) X 0 X Sherif Khalifa () Optimization 2 / 50
Y Y=f(X) f(x 0 ) X 0 X Sherif Khalifa () Optimization 3 / 50
A necessary condition for x 0 to be an interior max or min of y = f (x) is that the first derivative of f at x 0 be zero. The definitions of maximum and minimum for a function of several variables are the same as the definitions for a function of one variable. Let F : U R 1 be a real valued function of n variables, whose domain U is a subset of R n. A point x U is a max or min of F on U if F (x ) > F (x) for all x = x in U. Sherif Khalifa () Optimization 4 / 50
Theorem Let F : U R 1 be a function defined on a subset of R n. If x is a local max or min of F in U and if x is an interior point of U, then F x i (x ) = 0 for i = 1,..., n. The first order condition for a point x to be a max or a min of a function f of one variable is that f (x ) = 0. The same first order condition applies for a function F of n variables. The n dimensional analogue of f (x ) = 0 is that each F x i = 0. Sherif Khalifa () Optimization 5 / 50
Maximize F (x, y) = x 3 y 3 + 9xy F x = 3x 2 + 9y = 0 F y = 3y 2 + 9x = 0 Sherif Khalifa () Optimization 6 / 50
From the first equation 3x 2 + 9y = 0 Substitute into the second equation 9y = 3x 2 y = 1 3 x 2 3y 2 + 9x = 0 ( 3 1 ) 2 3 x 2 + 9x = 0 1 3 x 4 + 9x = 0 x 4 + 27x = 0 x ( x 3 + 27 ) = 0 Sherif Khalifa () Optimization 7 / 50
Solutions are either x = 0 or x = 3. This corresponds to y = 1 3 (0)2 = 0 and y = 1 3 (3)2 = 3. The candidates for max or min of F are the two points (0, 0) and (3, 3). These points are called the critical points. We are unable to determine which is a max and which is a min. Sherif Khalifa () Optimization 8 / 50
To determine whether a critical point is a max or a min, we need to use a condition on the second derivatives of F. The second order condition for a critical point x of a function f on R 1 to be a max is that the second derivative f (x ) be negative. The corresponding condition for a function F on n variables is that the second derivative with repect to all its arguments be negative definite. The second order condition for a critical point x of a function f on R 1 to be a min is that the second derivative f (x ) be positive. The corresponding condition for a function F on n variables is that the second derivative with repect to all its arguments be positive definite. Sherif Khalifa () Optimization 9 / 50
Theorem Let F : U R 1 be a function whose domain is an open set U in R 1. Suppose that F x i (x ) = 0 for i = 1,..., n. In addition, the n leading principal minors of D 2 F (x ) alternate in sign. Then, x is a strict local max of F. [F xx ] < 0 [ Fxx F Determinant of yx F xy F yy ] = (F xx ) (F yy ) (F yx ) (F xy ) > 0 Sherif Khalifa () Optimization 10 / 50
Theorem Let F : U R 1 be a function whose domain is an open set U in R 1. Suppose that F x i (x ) = 0 for i= 1,..., n. In addition, the n leading principal minors of D 2 F (x ) are all positive. Then, x is a strict local min of F. [F xx ] > 0 [ Fxx F Determinant of yx F xy F yy ] = (F xx ) (F yy ) (F yx ) (F xy ) > 0 Sherif Khalifa () Optimization 11 / 50
Theorem Let F : U R 1 be a function whose domain is an open set U in R 1. Suppose that F x i (x ) = 0 for i= 1,..., n. In addition, the n leading principal minors of D 2 F (x ) violate the sign patterns in the previous hypotheses. Then, x is neither a strict local min or local max of F. x is called a saddle point. Sherif Khalifa () Optimization 12 / 50
Maximize F (x, y) = x 3 y 3 + 9xy F x = F x = 3x 2 + 9y = 0 F y = F y = 3y 2 + 9x = 0 Sherif Khalifa () Optimization 13 / 50
[ Fxx F yx F xy F yy [ Fxx F det yx F xy F yy ] [ 6x 9 = 9 6y F xx = 6x ] = 36xy 81 ] Sherif Khalifa () Optimization 14 / 50
At (0, 0) F xx = 6x = 0 [ ] Fxx F det yx = 36xy 81 = 81 < 0 F xy F yy Saddle point Sherif Khalifa () Optimization 15 / 50
At (3, 3) F xx = 6x = 18 > 0 [ ] Fxx F det yx = 36xy 81 = 243 > 0 F xy F yy Min Sherif Khalifa () Optimization 16 / 50
Maximize F (x, y) = x 4 + x 2 6xy + 3y 2 F x = F x = 4x 3 + 2x 6y = 0 F y = F = 6x + 6y = 0 y Sherif Khalifa () Optimization 17 / 50
From the second equation 6x + 6y = 0 x = y Substitute into the first equation 4x 3 + 2x 6y = 0 4x 3 + 2x 6x = 0 4x 3 4x = 0 4x ( x 2 1 ) = 0 Sherif Khalifa () Optimization 18 / 50
Solutions are x = 0, x = 1 and x = 1. This corresponds to y = 0, y = 1, and y = 1. The candidates for max or min of F are the points (0, 0), (1, 1) and ( 1, 1). [ ] [ Fxx F yx 12x = 2 ] + 2 6-6 6 F xy F yy F xx = 12x 2 + 2 [ ] Fxx F det yx = [ 12x 2 + 2 ] (6) 36 F xy F yy Sherif Khalifa () Optimization 19 / 50
At (0, 0) F xx = 2 > 0 [ ] Fxx F det yx = 12 36 = 24 < 0 F xy F yy Saddle point Sherif Khalifa () Optimization 20 / 50
At (1, 1) F xx = 14 > 0 [ ] Fxx F det yx = (14) (6) 36 = 48 > 0 F xy F yy Min Sherif Khalifa () Optimization 21 / 50
At ( 1, 1) F xx = 14 > 0 [ ] Fxx F det yx = (14) (6) 36 > 0 F xy F yy Min Sherif Khalifa () Optimization 22 / 50
Maximize F (x, y) = xy 2 + x 3 y xy F x = F x = y 2 + 3x 2 y y = 0 F y = F y = 2xy + x 3 x = 0 Sherif Khalifa () Optimization 23 / 50
From the first equation y 2 + 3x 2 y y = 0 y [ y + 3x 2 1 ] = 0 Solutions are either y = 0 or y = 1 3x 2. Sherif Khalifa () Optimization 24 / 50
If y = 0, substitute into the second equation 2x (0) + x 3 x = 0 x 3 x = 0 x ( x 2 1 ) = 0 Solutions are either x = 0 or x = 1 or x = 1. The candidates for max or min of F are the points (0, 0), (1, 0) and ( 1, 0). Sherif Khalifa () Optimization 25 / 50
If y = 1 3x 2, substitute into the second equation 2x ( 1 3x 2) + x 3 x = 0 2x 6x 3 + x 3 x = 0 x ( 1 5x 2) = 0 Solutions are either x = 0 or x = 1 5 or x = 1 5. This corresponds to y = 1 3 (0) 2 = 1, y = 1 3 ( 1 5 ) 2 = 2 y = 1 3 ( 1 5 ) 2 = 2 5. The candidates for max or min of F are the points (0, 1), ( ) 1 5, 2 5. 5, ( ) 5 1, 2 5 and Sherif Khalifa () Optimization 26 / 50
The candidates ( ) for max ( or min ) of F are the points (0, 0), (1, 0), ( 1, 0), (0, 1), 5 1, 2 5 and 1 5, 2 5. [ ] [ Fxx F yx 6xy 2y + 3x = 2 ] 1 F xy F yy 2y + 3x 2 1 2x F xx = 6xy [ ] Fxx F det yx = 12x 2 y ( 2y + 3x 2 1 ) 2 F xy F yy Sherif Khalifa () Optimization 27 / 50
At (0, 0) F xx = 6 (0) (0) = 0 [ ] Fxx F det yx = 1 < 0 F xy F yy Saddle point Sherif Khalifa () Optimization 28 / 50
At (1, 0) F xx = 6 (1) (0) = 0 [ ] Fxx F det yx = 4 < 0 F xy F yy Saddle point Sherif Khalifa () Optimization 29 / 50
At ( 1, 0) F xx = 6 ( 1) (0) = 0 [ ] Fxx F det yx = 4 < 0 F xy F yy Saddle point Sherif Khalifa () Optimization 30 / 50
At (0, 1) F xx = 6 (0) (1) = 0 [ ] Fxx F det yx = 1 < 0 F xy F yy Saddle point Sherif Khalifa () Optimization 31 / 50
[ Fxx F det yx F xy F yy ( 1 At 5, 2 ) 5 ( ) ( ) 1 2 F xx = 6 5 > 0 5 ] = 12x 2 y ( 2y + 3x 2 1 ) 2 ( ) 1 2 ( ) ( 2 = 12 5 2 5 ( ) ( ) ) 2 1 2 2 + 3 5 1 5 = 24 ( 4 25 5 + 3 ) 2 5 1 = 24 25 16 25 > 0 Min Sherif Khalifa () Optimization 32 / 50
[ Fxx F det yx F xy F yy ( At 1, 2 ) 5 5 ( F xx = 6 1 ) ( ) 2 5 5 ] < 0 = 12x 2 y ( 2y + 3x 2 1 ) 2 ( = 12 1 ) 2 ( ) ( 2 2 5 5 = 24 ( 4 25 5 + 3 ) 2 5 1 = 24 25 16 25 > 0 Max ( ) ( 2 + 3 1 ) ) 2 2 1 5 5 Sherif Khalifa () Optimization 33 / 50
Definition The constrained optimization problem is that of maximizing a function of several variables, where these variables are bound by some constraining equations. Examples Utility function is maximized subject to the budget constraint, the profit function is maximized subject to the cost of production. Sherif Khalifa () Optimization 34 / 50
Maximize f (x 1, x 2,..., x n ) subject to h 1 (x 1, x 2,..., x n ) = c 1, h 2 (x 1, x 2,..., x n ) = c 2,...,..., h m (x 1, x 2,..., x n ) = c m Sherif Khalifa () Optimization 35 / 50
Maximize f (x 1, x 2 ) subject to h (x 1, x 2 ) = p 1 x 1 + p 2 x 2 = I Sherif Khalifa () Optimization 36 / 50
f (x 1, x 2 = f [x 1, x 2 (x 1 )] ( ) ( ) ( ) f f x2 + = 0 x 1 x 2 x 1 ( ) ( ) ( ) f x2 f = x 2 x 1 x 1 ( ) ( x2 = f ) ( ) f / x 1 x 1 x 2 ( Slope of f = f ) ( ) f / x 1 x 2 ( Slope of h = h ) ( ) h / x 1 x 2 Sherif Khalifa () Optimization 37 / 50
( f x 1 ( f x 1 ( f x 1 ) ( f / x 2 ) / ) / ) = ( h x 1 ) ( h / x 2 ) ( h ) ( ) ( ) f h = / x 1 x 2 x 2 ( ) ( ) ( ) h f h = / = µ x 1 x 2 x 2 Sherif Khalifa () Optimization 38 / 50
( f x 1 ( f x 2 ) ( h µ x 1 ) µ ) = 0 ( ) h = 0 x 2 h (x 1, x 2 ) c = 0 Sherif Khalifa () Optimization 39 / 50
L (x 1, x 2, µ) = f (x 1, x 2 ) µ [h (x 1, x 2 ) c] ( ) ( ) L f h = µ = 0 x 1 x 1 x 1 ( ) ( ) L f h = µ = 0 x 2 x 2 x 2 L µ = h (x 1, x 2 ) c = 0 Sherif Khalifa () Optimization 40 / 50
Maximize f (x 1, x 2 ) = x 1 x 2 subject to h (x 1, x 2 ) = x 1 + 4x 2 = 16 L (x 1, x 2, µ) = x 1 x 2 µ [x 1 + 4x 2 16] L x 1 = x 2 µ = 0 L x 2 = x 1 4µ = 0 L µ = [x 1 + 4x 2 16] = 0 Sherif Khalifa () Optimization 41 / 50
From the first and second equations x 2 = µ µ = 1 4 x 1 x 1 = 4x 2 Substitute into the third equation x 1 + 4x 2 = 16 (4x 2 ) + 4x 2 = 16 x 2 = 2 µ = 2 x 1 = 8 Sherif Khalifa () Optimization 42 / 50
Maximize f (x 1, x 2 ) = 2x 1 x 2 subject to h (x 1, x 2 ) = x 1 + 2x 2 = 4 L (x 1, x 2, µ) = 2x 1 x 2 µ [x 1 + 2x 2 4] L x 1 = 2x 2 µ = 0 L x 2 = 2x 1 2µ = 0 L µ = [x 1 + 2x 2 4] = 0 Sherif Khalifa () Optimization 43 / 50
From the first equation 2x 2 µ = 0 x 2 = µ 2 From the second equation 2x 1 2µ = 0 x 1 = µ Sherif Khalifa () Optimization 44 / 50
From the third equation [x 1 + 2x 2 4] = 0 [ ( µ ) ] µ + 2 4 = 0 2 2µ = 4 µ = 2 x 2 = 1 x 1 = 2 Sherif Khalifa () Optimization 45 / 50
Maximize f (x 1, x 2 ) = x1 2 x 2 subject to h (x 1, x 2 ) = 2x1 2 + x2 2 = 3 L (x 1, x 2, µ) = x1 2 x 2 µ [ 2x1 2 + x2 2 3 ] L x 1 = 2x 1 x 2 4µx 1 = 0 L x 2 = x 2 1 2µx 2 = 0 L µ = [ 2x 2 1 + x 2 2 3 ] = 0 Sherif Khalifa () Optimization 46 / 50
From the first equation 2x 1 x 2 4µx 1 = 0 2x 1 [x 2 2µ] = 0 Solutions are either x 1 = 0 or x 2 = 2µ. Sherif Khalifa () Optimization 47 / 50
If x 1 = 0, substitute into the third equation [ 0 + x2 2 3 ] = 0 x 2 = (3) 1 2 or (3) 1 2 Substitute into the second equation Solutions are [ ] 0, (3) 1 2, 0 (0) 2 2µx 2 = 0 µ = 0 [ ] and 0, (3) 2 1, 0. Sherif Khalifa () Optimization 48 / 50
If x 2 = 2µ, substitute into the second equation x1 2 2µx 2 = 0 ( x1 2 x2 ) 2 x 2 = 0 2 x 1 = x 2 Substitute into the third equation [ ] 2 (x 1 ) 2 + (x 1 ) 2 3 = 0 [ 3x 2 1 3 ] = 0 x 1 = 1 or x 1 = 1 x 2 = 1 or x 2 = 1 Sherif Khalifa () Optimization 49 / 50
Substitute into the second equation 1 2µ (1) = 0 µ = 0.5 1 2µ ( 1) = 0 µ = 0.5 1 2µ ( 1) = 0 µ = 0.5 1 2µ (1) = 0 µ = 0.5 Solutions are (1, 1, 0.5), ( 1, 1, 0.5), (1, 1, 0.5), ( 1, 1, 0.5). Sherif Khalifa () Optimization 50 / 50