Chapter 1. Introduction. 1.1 Problem statement and examples

Transcription

1 Chapter 1 Introduction 1.1 Problem statement and examples In this course we will deal with optimization problems. Such problems appear in many practical settings in almost all aspects of science and engineering. Mathematically, we can write the problem as f(x) (1.1a) subject to c i (x) = 0 i = 1,..., n E (1.1b) subject to h i (x) 0 i = 1,..., n I (1.1c) Where x is an n dimensional vector and the function f : R n R 1 is called the objective function. The c i are called equality constraints and h i are inequality constraints. Our goal is to find the vector x that solves (1.1) assug that such solution exist. For most applications we assume that f, c i and h i are twice differentiable. Example 1 Minimization of a function in 1D Let f(x) = x then an obvious solution is x = 0. If we add the inequality constraint h(x) = x 1 0 then the solution is x = 1. If on the other hand we add the inequality constraint h(x) = x then, the solution is the same us the solution of the problem with no constraints and we obtain x = 0. We see that given an inequality constraint, it can be active or inactive. Finally, consider the case h 1 (x) = x 1 0 and h (x) = 1 x 0. In this case we see that the constraints are inconsistent and the problem has no solution. Example Data fitting 1

2 Assume that the data d is obtained by the model Ax + ɛ = d where A is an n m matrix with n < m and ɛ is a noise vector assumed Gaussian iid with variance 1. Since the number of the data is smaller than the number of parameters we seek there are infinite solutions that fit the data, that is, we can easily find ˆx such that Aˆx = d. Nevertheless, such x will overfit the data. To obtain a meaningful x we introduce the following optimization problem x ρ(x) (1.) Ax d n (1.3) The function ρ(x) is often called a penalty function. Various choices are possible and we review a few next. Example 3 MRI image processing When generating MRI images of the brain it is possible to obtain a sequence of images I 1, I,..., I n. The intensity of each pixel in the image decays at a different rate. A model for the intensity decay of a pixel located at x j is I(t, x j ) = k a i (x j ) exp( λ i (x j )t) (1.4) i=1 where a i (x) and λ i (x) are space dependant coefficients. Given s time measurements Î(t n, x j ), n = 1,... s we would like to evaluate the coefficients a i (x j ) and λ i (x j ) as they indicate possible pathology. This can be done by solving the following optimization problem a,λ 1 ( s k a i exp( λ i t n ) n)) Î(t (1.5) n=1 i=1 a i 0 i = 1,..., k λ i 0 i = 1,..., k Note the nonegativity assumption on the coefficients. Example 4 Portfolio optimization Assume that there are n assets and that you have T dollars. The question is how to invest your T dollars within the given assets.

3 Figure 1.1: a sequence of MRI images. Note the lesion (white blob) in the top of the brain To do that you look at historic gains of each of these assets. Assume that you find that asset i historically gain on average p i, that is, at the end of the period it was worth p i x i. Thus, the total portfolio will worth p x. Unfortunately, life are not that easy. The problem is that this gain was only on average. Each gain has also a standard deviation. As you may know from your own experience, the highest earning stocks are often the most risky one. We assume that A is the covariance matrix associated with the assets. Then, the risk can be written as x Ax. There are a few problems which are associated with finding an optimal portfolio. Here we consider imizing the risk while making some money. This can be written as x Ax (1.6a) p x ηt (1.6b) x e = T x 0 (1.6c) (1.6d) Where η > 1 detere what is the imal earning we can live with reducing the risk as much as possible. This is a classical quadratic programg problem that can be solved for the optimal investment strategy. 3

4 1. Reformulation For many optimization problems simple tricks can transform a difficult problem into a much simpler one. Thus, before solving a particular problem it is worth while checking if there is a different equivalent problem which is easier to solve Adding constraints to avoid nondifferentiability May be the most common transformation is from problems with non-differentiable objective to one with differentiable objective and inequality constraints. Example 5 L 1 imization Consider the following optimization problem which arises in signal processing x 1 = i x i (1.7) Ax = b The objective function is not differentiable at 0. To obtain a differentiable objective we set x = p q; p, q 0 It is then easy to check that the problem is equivalent to p i + q i (1.8) i A(p q) = b p, q 0 (1.9) The new objective function is obviously differentiable but we have two inequality constraints. Example 6 L imization Consider a similar problem to the previous x = max x i (1.10) Ax = b In this case the max function is non-differentiable. Once again it is easy to convert the problem to a differentiable one by introducing a new scalar variable t t (1.11) Ax = b t x i t (1.1) 4

5 1.. Regularizing nondifferentiability For many other problems it is difficult to obtain an exact equivalent differentiable formulation. On the other hand, it is easy to obtain a regularized formulation, that is, a differentiable formulation that yields a similar result to the nondifferentiable one. Example 7 L 1 imization again We approximate the L 1 imization ρ(x i ; θ) (1.13) i Ax = b where { 1 ρ(t; θ) = θ t + θ if t θ t otherwise (1.14) The function ρ is known as the Huber function. It is easy to see that the function ρ(t; θ) is continuously differentiable with respect to t and as θ 0 one obtains the original L 1 imization 1..3 Sequential imization In some problems it is possble to divide the unknowns into two groups of variables, p and q. If we assume that the second group, q is known then it is easy to imize with respect to p. Therefore, it is possible to solve the problem in two stages p,q f(p, q) = q ( ) f(p, q) p Example 8 Combination of linear and nonlinear unknowns Let f(p, q) = (p exp(q) 1) + p + q Then, it is easy to verify that the imum with respect to p is p = And therefore, the problem is eqivalent to f(p(q), q) = exp(q) exp(q) + 1 ( ) ( ) exp(q) exp(q) exp(q) q exp(q) + 1 5

6 1..4 Eliation of equality constraints In many optimization problems especially with linear constraints one could obtain an unconstraint optimization problem by eliating the equality constraints. Example 9 Assume we need to solve the following problem x 1 + x 1 x 1 x = 0 It is easy to see that the equality constraint can be written as x 1 constraint optimization problem can be writtten as x 1 1 = x 1 and therefore the 1..5 Change of variables In many cases we are able to obtain equivalent problems by changing variables. We have to be careful that the map is one to one. That is f(x) = f(ϕ(z)) if the map x = ϕ(z) has an inverse for all admisible x s Example 10 Using the exponent to replace strictly positive variables Consider the optimization problem By setting exp(t i ) = x i, 1.3 Problems (x 1 + 1) + (x + 3) 0.01(log(x 1 ) + log(x )) i = 1, we obtain (exp(t 1 ) + 1) + (exp(t ) + 3) 0.01(t 1 + t ) 1. Choose any field of study and find an optimization problem from that field. Explain the objective function and the constraints.. Plot the function h(x) = max(0, x), comment on its differentiability. Design a smooth, continuously differentiable function that is similar to the function h(x) = max(0, x). 3. Reformulate the following problem as an unconstrained optimization problem x 1 + x + x 3 ( ) x x 1 3 = x 3 ( ) 3 6 6