1 Introduction to Optimization

Transcription

1 Unconstrained Convex Optimization 2 1 Introduction to Optimization Given a general optimization problem of te form min x f(x) (1.1) were f : R n R. Sometimes te problem as constraints (we are only interested in x tat satisfy) c(x) 0 (1.2) were c : R n R m. We can also write te constraints as: c i (x) 0, i =1,...,m. f and c i can be linear or nonlinear functions. In tis class we assume tat tey are continuous. In tis course, te vector x is not restricted to a discrete set of d values (combinatorial optimization). Te dimension of x (te number of variables) n and te number of constraints m can range from 1 to tousands. 1.1 Examples of Optimization Problems Example 1: Resources allocation and financial planning We ave m different resources, suc as steel, paper, labor time on te punc macine, etc, to manufacture n different products. To produce 1 unit of Product 1, we use: a 11 units of steel a 21 units of paper a 31 units on te punc macine and we make a profit of c 1 dollars. To produce 2 unit of Product 2, we use: a 12 units of steel

2 Unconstrained Convex Optimization 3 a 22 units of paper a 32 units on te punc macine and we make a profit of c 2 dollars. Similarly for Product 3,,n. We ave b 1 units of steel available, b 2 units of paper, b 3 units on te punc macine, etc. Problem: Maximize te profit using te resources available. subject to max c 1 x 1 + c 2 x c n x n = c T x x x 0 a 11 x 1 + a 12 x a 1n x n b 1 a 21 x 1 + a 22 x a 2n x n b 2. a m1 x 1 + a m2 x a mn x n b m or Ax b Bot te objective function and te constraints are linear. Example 2: Data fitting Suppose we measure some data (t i,b i ) i =1,,m and we want to fit a matematical model to te data: b = x p a p + x p 1 a p x 1 a 1 + x 0 Ten we may coose to determine te parameters by least square criterion: m min (b i x T a i ) 2 =min Ax b 2 2 x x i=1 were a i is te basis expanse on t i. In practical settings, tere may be some constraints to te problem, for example, we migt want te parameters to be positive or witin certain range.

3 Unconstrained Convex Optimization Convex Optimization In real world applications, it is easy to write down an optimization problem in te form like (1.1,1.2), but witout careful design we may end up writing down optimization problems tat are essentially impossible to solve. Convex optimization is a subset of optimizations tat studies te problem of optimizing convex functions over convex sets. Rougly speaking, it is a subset of optimizations tat we can solve. Convex set: A set X is convex is for any x 1, x 2 X, θx 1 +(1 θ)x 2 X, 0 θ 1. Geometrically, tis means te line segment joining any two points in te set must be in te set (Fig 1.1): Figure 1.1: Geometrical interpretation of convex sets Convex function: A function f(x) is convex over a convex set X if for x 1, x 2 X, f(θx 1 +(1 θ)x 2 ) θf(x 1 )+(1 θ)f(x 2 ), 0 θ 1. Figure 1.2: Geometrical interpretation of convex function Geometrically tis means te line joining any two function values must lie above te functions itself.

4 Unconstrained Convex Optimization 5 Example: Is te function f(x) =x 2 convex? f(θx 1 + (1 θ)x 2 ) θf(x 1 ) (1 θ)f(x 2 ) = (1 θ) 2 x θ 2 x (1 θ)θx 1 x 2 (1 θ)x 2 2 θx 2 1 = (θ 2 θ)x (1 θ)θx 1 x 2 +((1 θ) 2 (1 θ))x 2 2 = θ(θ 1)x 2 1 +(1 θ)θx 1 x 2 + θ(θ 1)x 2 2 = θ(θ 1)(x 2 1 2x 1 x 2 + x 2 2) = θ(θ 1)(x 1 x 2 ) 2 0. A function is called strictly convex is te inequality is satisfied for all points not at te end (0 <θ<1): f(θx 1 +(1 θ)x 2 ) <θf(x 1 )+(1 θ)f(x 2 ), 0 <θ<1. A function f is concave if f is convex. Example: Is a affine function convex? Is a linear function strictly convex? Is a affine function concave? Assume f(x) =a T x + b f(θx 1 + (1 θ)x 2 ) θf(x 1 ) (1 θ)f(x 2 ) = a T (θx 1 +(1 θ)x 2 )+b θ(a T x 1 + b) (1 θ)(a T x 2 + b) = b θb (1 θ)b =0. Affine functions are convex and concave. It is not strictly convex. Similarly wit linear functions. Any local minimum of a convex function is also a global minimum. A strictly convex function will ave at most one global minimum. Te gradient (first-derivative) of f at x is a vector defined as: f(x) = df dx = f x 1 f x 2. f x n Te Hessian (second-derivative) of f at x is a matrix tat consists te derivative of te gradient: d2 f H(x) = 2 f(x) = dxdx T

5 Unconstrained Convex Optimization 6 were ij = 2 f x i x j Note tat te Hessian is symmetric: H = H T. Example: Analytical gradient and Hessian of function y = x x 2 2 dy dx = 2x1 20x 2 H(x) = y = e x 1+3x e x 1 3x e x dy e dx = x 1 +3x e x 1 3x e x e x 1+3x e x 1 3x e x 1 +3x 2 H(x) = e x 1 3x e x e x 1+3x e x 1 3x e x 1+3x e x 1 3x e x 1+3x e x 1 3x How to calculate gradient numerically? If f(x) is a scalar input function: f (x) =lim 0 f(x + ) f(x) Central difference formula of order O( 1 ): f (x) f(x + ) f(x) Proof: First-degree Taylor expansions f(x + ) =f(x) +f (x) + f (2) (c) 2 So f (x) = f(x+) f(x) f (2) (c) 2! Central difference formula of order O( 2 ): 2!.

6 Unconstrained Convex Optimization 7 f f(x + ) f(x ) (x) 2 Proof: Second-dgree Taylor expansions for f(x + ) and f(x ) Terefore f(x + ) =f(x)+f (x) + f (2) (x) 2 2! f(x ) =f(x) f (x) + f (2) (x) 2 2! + f (3) (c 1 ) 3 f (3) (c 2 ) 3 Namley Terefore f(x + ) f(x ) =2f (x) + (f (3) (c 1 )+f (3) (c 2 )) 3 f (x) = f(x + ) f(x ) 2 Central difference formula of order O( 4 ): f (3) (c) 2 f f(x +2)+8f(x + ) 8f(x )+f(x 2) (x) 12 Proof: Similarly, start wit f(x +2) f(x 2) and f(x + ) f(x ) f(x + ) f(x ) =2f (x) + 2f (3) (x) 3 + 2f (5) (c 2 ) 5 5! Terefore f(x +2) f(x 2) =4f (x) + 16f (3) (x) f (5) (c 2 ) 5 5! f(x+2)+8f(x+) 8f(x )+f(x 2) =12f (x)+ (16f (5) (c 1 ) 64f (5) (c 2 )) 5 Namley Terefore 120 f (x) = f(x +2)+8f(x + ) 8f(x )+f(x 2) 12 f (5) (c) 4 30 If f(x) is a vector input function:

7 Unconstrained Convex Optimization 8 Central difference formula of order O( 1 ): f(x) f(x 1 +,x 2,,x n) f(x) f(x 1,x 2 +,,x n) f(x) ḥ. f(x 1,x 2,,x n+) f(x) Central difference formula of order O( 2 ): f(x) = f(x+e 1 ) f(x e 1 ) 2 f((x+e 2 ) f(x e 2 ) 2ḥ. (x+e n) f(x e n) 2 f(x+e 1 ) f(x) f((x+e 2 ) f(x) ḥ. (x+e n) f(x) were e i is te i t unit vector (only te i t element is nonzero and equal to 1). Two important questions: 1. How to coose? Is te smaller te better? If is large, we ave large truncation error. If is small, we ave large round-off error. A coice of tat is small witout introducing large round-off error is x. 2. Does iger order always means smaller error? Te truncation error for te iger-order formula will go to zero faster tan te lower-order formula, terefore permit te use of a larger step size. Exercise: implement order 1, 2 and 4 central difference formula for computing te gradient of function. Test it on te following two functions and compare tem wit teir true gradient functions y = x x 2 2 y = e x 1+3x e x 1 3x e x How to calculate Hessian numerically?

8 Unconstrained Convex Optimization 9 Finite difference approximation: ij = ( f i(x)) x j f i(x + e j ) f i (x) ij = ( f i(x)) f i(x + e j ) f i (x e j ) x j 2 Exercise: Implement and compare te performance of finite-difference Hessian in te following two versions using te finite difference metod of gradient calculation using te true gradient functions. Practical issues of calculating gradient and Hessian of a function: Analytical evaluation is usually complex, especially wit iger derivative. Not always available Numerical evaluation is error prone, especially wit iger derivative. Numerical evaluation is also expensive in computation, especially wit ig-dimensional functions. Automatic differentiation is a way to numerically evaluate te derivatives of a function specified by a computer program (Furter reading).