MATH2070/2970 Optimisation Introduction Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart
Course Information Lecture Information Optimisation: Weeks 1 7 Contact Information Email: ivan.guo@sydney.edu.au Office: 807, Extension: 41274 Consultation Wednesday 11am in Room 707A. Assessment Assignment 10% Quiz 10% Project (Fin Maths) 10% Final Exam 70%
Course Readings Lecture Slides. Split into relevant topics Available at the course website. Website Lecture Notes Notes for both Optimisation & Financial Mathematics. Available at Kopystop Website Shop 3 / 55 Mountain St Broadway NSW 2007 Google Maps Link Map
Optimisation Outline Introduction and Motivation Linear Programming Non-linear Optimisation without constraints Non-linear Optimisation with constraints Dynamic Programming (At the end of the course)
Review of Introductory Material Motivation Optimising Differentiable Functions of One Variable Optimising Differentiable Functions of Several Variables System of Equations Pivot Operations and variable simplification
Motivation Optimising Differentiable Functions of One Variable Optimising Differentiable Functions of Several Variables System of Equations Pivot Operations and variable simplification
How are optimisation techniques used? Motivation, In most basic format either, Minimise costs/time. Maximise output. Examples: Timetabling (Cityrail, Sydney Buses) HR: Staff allocations. Manufacturing: Blending of raw materials Retail: Determine optimal prices.
Terminology Definition (Parameters) A set of values (fixed or variable) that are used to describe the relationship between quantities. More Definition (Objective Function) A function f(x) of a set of parameters x that one wishes to maximise/minimise will be termed the objective function. More Definition (Constraints) A set of conditions that the parameters need to be satisfied during the optimisation. More
Water tank design example Wish to minimise heat loss through surface area from an open rectangular water storage tank that has a fixed volume, V. Dimensions: x, y and z. Formulate Problem Minimise, such that, S = 2xy + 2yz + xz V = xyz constant and x, y, z > 0. Terminology Parameters : x, y and z. Constraints : V = xyz and x, y, z > 0. Objective function : S(x, y, z) = 2xy + 2xz + xz
Evaluation Simplification obtained by eliminating z variable using the constraint. Note, V = xyz z = V xy which gives, Simplified Problem Minimise, S(x, y) = 2xy + 2V x + V y such that, x, y > 0 and V constant. How do we optimise such a function?
Motivation Optimising Differentiable Functions of One Variable Optimising Differentiable Functions of Several Variables System of Equations Pivot Operations and variable simplification
Review of Univariate optimisation Single Variable Calculus Univariate = Single variable x [a, b] R. Objective function f : [a, b] R. Location of Extrema At which values of x do the extrema (maxima or minima) occur for the function f(x)? Two types of Extrema Local Extrema Global Extrema
Formal definition of the two types of Extrema Let x [a, b] and f : [a, b] R. Definition (Local maximum/minimum) A value x 0 is said to be a local maximum (minimum) if f(x 0 ) f(x) (f(x 0 ) f(x)) for all x in a neighbourhood of x 0 f(x) x 0 x x Figure 1: Plot of a function that has one maximum and two minima.
Formal definition of the two types of Extrema Let x [a, b] and f : [a, b] R. Definition (Global maximum/minimum) A value x is said to be a global maximum (minimum) if f(x ) f(x) (f(x ) f(x)) for all x [a, b] f(x) x 0 x x Figure 2: Plot of a function that has global minimum at x.
f(x) 1 1 x Figure 3: Plot of a function that has global extrema at the endpoints.
f(x) 1 x x 1 x Figure 4: Plot of a function that has global extrema inside the domain.
Linear vs Non-linear Extrema Linear functions Extremum always attained at constraint boundaries A local extremum is also a global extremum Non-linear functions Extrema may be in the interior as well as at boundaries. A local extremum is not necessarily a global extremum. Linear Nonlinear
Finding the local extrema If f is differentiable on (a, b). That is, f (x) exists and is well defined. Previous calculus methods can be used. Find stationary points by checking the first derivative Definition A point x 0 (a, b) is said to be a stationary point of f if f (x 0 ) = 0 A stationary point is also referred to as a critical point. The behaviour of the stationary point can be determined by the..
Finding the local extrema (cont.) Basic second derivative test If f (x 0 ) = 0 for some x 0 (a, b) and If f (x 0 ) < 0, then x 0 is the location of a local maximum. If f (x 0 ) > 0, then x 0 is the location of a local minimum. If f (x 0 ) = 0, then test fails, x 0 is possibly a point of inflection. f(x) f (t) > 0 t g (s) < 0 s g(x)
Finding the local extrema (cont.) Generalised higher derivative test Let m be a positive integer and assume that there exists an x 0 (a, b) such that f (1) (x 0 ) = f (2) (x 0 ) =... = f (2m 1) (x 0 ) = 0. Then the following holds If f (2m) (x 0 ) < 0, then x 0 is the location of a local maximum. If f (2m) (x 0 ) > 0, then x 0 is the location of a local minimum. If f (2m) (x 0 ) = 0, then test fails, x 0 is possibly a point of inflection?
Motivation Optimising Differentiable Functions of One Variable Optimising Differentiable Functions of Several Variables System of Equations Pivot Operations and variable simplification
Review of Multivariate optimisation Multivariate Calculus Multivariate = Many variables x = (x 1, x 2,..., x n ) D R n. f = f(x 1, x 2,..., x n ) = Objective function of those n variables Necessary condition for stationary points f x 1 = f x 2 = = f x n = 0. Sufficient conditions for maxima/minima of multivariate functions considered later in the course.
Non-linear functions Extrema may be in the interior as well as at boundaries. A local extremum is not necessarily a global extremum. Definition (Local maximum/minimum) A point x 0 = (x 1, x 2,..., x d ) D is said to be a local maximum (minimum) if f(x 0 ) f(x) (f(x 0 ) f(x)) for all x in a neighbourhood of x 0 Definition (Global maximum/minimum) A value x D is said to be a global maximum (minimum) if f(x ) f(x) (f(x ) f(x)) for all x D
Multivariate example 1 0 1 0 2 4 6 0 2 4 6 Figure 5: 3-dimensional function with stationary points.
Other type of multivariate Optimisation Linear problems Linear programming problem Maximise/Minimise f(x 1, x 2,..., x n ) = c 1 x 1 + c 2 x 2 + + c n x n, such that, a 1 x 1 + a 2 x 2 +... + a n x n C, where c i, a i and C are constants,
Linear Programming Solution? In a similar vein to the univariate case, Optimal solution for Linear problems Extremum are always attained at constraint corner points To see this note that if partial derivatives are set to zero: f x 1 = 0 c 1 = 0... f = 0 c n = 0. x n Thus need to consider the boundaries for optimal solution Solutions can be difficult in high dimensional problems.
Example : Manufacturing problem A company manufactures two types of drugs by using three different resources. The resources have limited supply. The company wishes to maximise its profit. Each unit of drug earns the following profit: Drug 1 earns a profit of $3,000. Drug 2 earns a profit of $5,000. Each unit of drug uses the following resources: Drug 1 uses 1 gm of Resource 1 and 3 gm of Resource 2. Drug 2 uses 2 gm of Resource 2 and 2 gm of Resource 3. Each resource has the following supply limit. Only 4 gm of Resource 1 are available. Only 18 gm of Resource 2 are available. Only 12 gm of Resource 3 are available.
Model formulation. Let x 1, x 2 represent the number of units of Drug 1 and Drug 2 produced. Write problem mathematically with: Maximize: Z = 3x 1 + 5x 2 subject to: x 1 4 3x 1 + 2x 2 18 2x 2 12 with: x 1 0, x 2 0. Problem formulation Very important to be able to formulate problem mathematically.
Motivation Optimising Differentiable Functions of One Variable Optimising Differentiable Functions of Several Variables System of Equations Pivot Operations and variable simplification
Elementary Row Operations Consider the following system of linear equations: x 1 x 2 x 3 = 2 2x 1 +x 2 x 3 = 5 x 1 +2x 2 3x 3 = 0 (1) Written in augmented matrix notation, 1 1 1 2 2 1 1 5 1 2 3 0 Solution vector x = (x 1, x 2, x 3 ) exists that satisfies (1). Method uses Elementary Row Operations.
Elementary Row Operations The three elementary row operations used are 1. Interchange two rows (denoted R i R j ); 2. Multiply (or divide) any row by a non-zero constant (denoted R i ar i ); 3. Add a (non-zero) multiple of one row to any other row (denoted R i R i + ar j ). Interchange of rows Interchange of rows is not used in linear programming methods. Solution is unchanged Elementary row operations do not change the solution (x 1, x 2, x 3 ) Return to example.
Return to the previous matrix: T 1 = x 1 x 2 x 3 RHS 1 1 1 2 R 1 2 1 1 5 R 2 1 2 3 0 R 3 then the row-operations: R 3 R 3 + R 1 and R 2 R 2 2R 1 transform T 1 to the equivalent tableau: T 2 = x 1 x 2 x 3 RHS 1 1 1 2 0 3 3 1 0 1 4 2
Motivation Optimising Differentiable Functions of One Variable Optimising Differentiable Functions of Several Variables System of Equations Pivot Operations and variable simplification
Pivot operations Let a ij be any non-zero element (called the pivot element) of the coefficient matrix A of a given system of linear equations: i.e. a ij is the element in row i and column j of the corresponding tableau. Then to pivot on a ij 0, denoted P ij, 1. Divide row i by a ij ; 2. Transform to zero all elements a kj, k i (i.e. the elements in the same column j as a ij except row i) by adding suitable multiples of row i. Result, a 11 a 12 a 1n b 11 b 12 0 b 1n a 21 a 22 a 2n P ij b 21 b 22 0 b 2n.. a ij... 1. a n1 a n2 a nn b n1 b n2 0 b nn
Pivot Example Recall example tableau x 1 x 2 x 3 RHS 1 1 1 2 2 1 1 5 1 2 3 0. Pivot the tableau T 1 on the element a 21 = 2 (row 2, column 1). T 2 = P 21 T 1 = x 1 x 2 x 3 RHS 0 3/2 3/2 9/2 1 1/2 1/2 5/2 0 5/2 5/2 5/2 T 2 is also equivalent to T 1 since the pivot operation does not change the solution set (x 1, x 2, x 3 ) (Only uses ERO s).
Example: Pivot solution Pivot on the example matrix to obtain identity. x 1 x 2 x 3 RHS 1 1 1 2 2 1 1 5 1 2 3 0 1 1 1 2 P 11 : 0 3 3 1 0 1 4 2 1 0 0 7/3 P 22 : 0 1 1 1/3 0 0 5 5/3 1 0 0 7/6 P 33 : 0 1 0 2/3 0 0 1 1/3
More variables than equations Consider now the following system of three equations in five variables, x 1 x 2 +x 3 x 4 = 2 2x 1 +x 2 x 3 +x 5 = 5 x 1 +2x 2 +3x 3 +x 4 +2x 5 = 0 Express (x 1, x 2, x 3 ) in terms of (x 4, x 5 ). The variables we solve for (x 1, x 2, x 3 ), are called the basic variables. They must be linearly independent. The remaining variables (x 4, x 5 ) are called non-basic variables:
Method of solving new system Basic Variables Non Basic x 1 x 2 x 3 x 4 x 5 RHS 1 1 1 1 0 2 2 1 1 0 1 5 1 2 3 1 2 0 1 1 1 1 0 2 0 3 3 2 1 9 0 1 4 0 2 2 1 0 0 1/3 1/3 1 0 1 1 2/3 1/3 3 0 0 5 2/3 5/3 5 1 0 0 1/3 1/3 1 0 1 0 8/15 2/3 2 0 0 1 2/15 1/3 1
Solution for new equations Solution given by: Basic Variables Non Basic x 1 x 2 x 3 x 4 x 5 RHS 1 0 0 1/3 1/3 1 0 1 0 8/15 2/3 2 0 0 1 2/15 1/3 1 x 1 = 1 + 1 3 x 4 1 3 x 5 x 2 = 2 8 15 x 4 2 3 x 5 x 3 = 1 + 2 15 x 4 1 3 x 5