NatSciLab - Numerical Software Introduction to MATLAB

Transcription

1 Outline NatSciLab - Numerical Software Introduction to MATLAB Onur Oktay Jacobs University Bremen Spring 2010

2 Outline 1 Optimization with Matlab 2 Least Squares problem Linear Least Squares Method Nonlinear Least Squares Method 3 1 norm minimization

3 Outline 1 Optimization with Matlab 2 Least Squares problem Linear Least Squares Method Nonlinear Least Squares Method 3 1 norm minimization

4 Optimization Toolbox - Function list bintprog fgoalattain fminbnd fmincon fminimax fminsearch fminunc fseminf linprog quadprog lsqcurvefit lsqlin lsqnonlin lsqnonneg Minimization Binary integer programming Multiobjective goal attainment Minimize a single-variable function on fixed interval Constrained minimization Minimax constraint problem Derivative-free unconstrained minimization Unconstrained minimization Semi-infinitely constrained minimization Solve linear programming problems Solve quadratic programming problems Least Squares (Curve Fitting) Nonlinear least-squares data fitting Constrained linear least-squares data fitting Nonlinear least-squares data-fitting Least-squares with nonnegative constraint

5 Optimization - Constrained minimization minimize E(u) subject to A 1 u = b 1 (linear equality constraint) A 2 u b 2 (linear inequality constraint) G(u) = 0 (nonlinear equality constraint) H(u) 0 (nonlinear inequality constraint) r 1 u r 2 (domain constraint) E : R N R objective function to be minimized Linear constraints: - u is a solution to the linear equation A 1 u = b - Each entry of the vector A 2 u b 2 is < 0. Nonlinear constraints: - n equality constraints G(u) = (g 1 (u), g 2 (u),..., g n(u)), g 1 (u) = 0, g 2 (u) = 0,..., g n(u) = 0. - m inequality constraints H(u) = (h 1 (u), h 2 (u),..., h m(u)), h 1 (u) 0, h 2 (u) 0,..., h m(u) 0.

6 Optimization - fminunc fminunc finds a minimum of a E : R N R, starting at an initial point x 0 R N. This is generally referred to as unconstrained nonlinear optimization. Usage x = fminunc(f,x0,options) f is a function handle x0 is the starting point options [OPTIONAL] optimization options (see optimset) x is a local minimum of the input function. Simple Example: >> f + x(2)ˆ2); % anonymous function handle >> x0 = [1,1]; >> x = fminunc(f,x0);

7 Optimization - optimset We use optimset to set the optimization options. For example, If the gradient Df and the Hessian D 2 f are available, >> options = optimset( GradObj, on, Hessian, on ) both of them are set to off by default. When Gradient and Hessian are on, the input function f.m must return, - function value f as first output, - gradient value Df as the second output, - Hessian value D 2 f as the third output. This will speed up the calculations. Remember that, for a function F : R N R, - the gradient DF = ( F/ x 1, F/ x 2,..., F/ x N ) is the vector of partial derivatives of F, - the Hessian D 2 F = [ 2 F/ x i x j ] is the N N matrix of second partial derivatives of F.

8 Optimization - optimset Some (not all) of the other uses of optimset. Display FunValCheck MaxFunEvals MaxIter TolFun TolX off displays no output. iter displays output at each iteration. notify displays output only if the algorithm diverges. final (default) displays just the final output. on displays an error when the objective function returns a complex number, Inf, NaN. off (default) displays no error. Maximum number of function evaluations. Default = 200*numberOfVariables. Maximum number of iterations. Default = 200*numberOfVariables. Termination tolerance on the function value. Default = Termination tolerance on x. Default = 10 4.

9 Optimization - fminunc Example - sumsin.m function [ f, Df, D2f ] = sumsin(x) % for simplicity, for the moment we assume that the input is a vector N = length(x); % number of variables f = sum( sin(x).ˆ2 ); if nargout == 2; % Compute Df only if it is called Df = 2 sin(x). cos(x); elseif nargout == 3; % Compute D2f only if it is called D2F = diag( 2 cos(2 x) ); end >> options = optimset( GradObj, on, Hessian, on ); >> x0 = [4,1]; >> x = fminunc(@sumsin,x0,options);

10 Optimization - fminsearch Use fminsearch if E : R N R is not differentiable, e.g., E(x) = sum(abs(x)). fminsearch finds a minimum of a E : R N R, starting at a x 0 R N. Usage x = fminsearch( f, x0, options ) f is a function handle x0 is the starting point options [OPTIONAL] optimization options (see optimset) x is a local minimum of the input function. Example: >> f + x(2)ˆ2); % anonymous function handle >> x0 = [4,0]; >> x = fminsearch(f,x0); >> options = optimset( Tolfun, 1e-6 ); >> xx = fminsearch(@sumsin, x0, options);

11 Outline 1 Optimization with Matlab 2 Least Squares problem Linear Least Squares Method Nonlinear Least Squares Method 3 1 norm minimization

12 Description of the Least Squares problem We start with Data: {(x k, y k ), k = 1, 2,..., r}. Goal: Fit the data into a model Y = f(x, c). - f is the modeling function - c = (c 1, c 2,..., c N ) set of parameters to be determined by LS Least squares problem Find c = (c 1, c 2,..., c N ), which minimizes the sum of the squares E(c) = r ( yk f(x k, c) ) 2 k=1 among all possible choices (unconstrained) of parameters c R N. We say that the LS method is linear if f is of the form f(x, c) = N c n f n (X) n=1

13 Example1 - Data: week3 example1.mat - Model: f(x, c) = c 1 + c 2 X + c 3 X 2 LS method gave c 1 = , c 2 = , c 3 = data (x k,y k ) 2.6 model Y = X 2 2X+3 y x

14 Linear Least Squares Method LS method is linear f is of the form N f(x, c) = c n f n (X). n=1 Linear LS can be written in the matrix form. Designate - x = [x 1 ; x 2 ;... ; x r], y = [y 1 ; y 2 ;... ; y r], c = [c 1 ; c 2 ;... ; c N ] as column vectors. - A = [f 1 (x), f 2 (x),..., f N (x)] as an r N matrix. f n applies to x entrywise: f n(x) = [f n(x 1 ); f n(x 2 );... ; f n(x r)]. - Now, f(x, c) = A c, and E(c) = r k=1 ( yk f(x k, c) )2 = y A c 2 2 Then, the linear LS problem is Find c = (c 1, c 2,..., c N ), which minimizes E(c) = y Ac 2 = norm(y A c, 2) among all possible choices (unconstrained) of parameters c R r.

15 Linear Least Squares Method y f(x, c) = y 1 y 2 y 3. y r f 1 (x 1 ) f 2 (x 1 )... f N (x 1 ) f 1 (x 2 ) f 2 (x 2 )... f N (x 2 ) f 1 (x 3 ) f 2 (x 3 )... f N (x 3 ) f 1 (x r ) f 2 (x r )... f N (x r ) c 1 c 2. c N Best method to solve linear LS is mldivide. Back to Example 1: >> load week3-example1.mat >> x = data(:,1); y = data(:,2); r=length(x); >> A = [ones(r,1), x, x.ˆ2]; % f 1 (x) = 1, f 2 (x) = x, f 3 (x) = x 2 >> c = A \ y % LS solution >> yls = A c; LSerror = y - A c; >> figure(1); plot(x,y, o ); hold on; plot(x, yls, r ) >> figure(2); plot(1:r, LSerror);

16 Linear Least Squares Method Example 2 Cost minimization. A producer wants to model its total cost TC as a function of the number of the items produced X. TC is the sum of the electricity usage Y, and the maintenance costs Z. {(x k, y k, z k ), k = 1,..., r} is gathered over a period of r days, We want to fit this data to the model f(x, c) = Y + Z, Y = c 1 + c 2 X 2, Z = c 3 log(x) + c 4 sin(0.01πx) f(x, c) = [1, X 2, log(x), sin(0.01πx)] c LS is linear. >> load week3-example2.mat >> x = data2(:, 1); y = data2(:, 2); z = data2(:, 3); >> A = [ ones(r,1), x.ˆ2, log(x), sin(0.01*pi*x) ]; >> TC = y+z; >> c = A \ TC >> yls = A( :, [1,2] ) c( [1; 2] ); LSerrory = y - yls; >> zls = A( :, [3,4] ) c( [3; 4] ); LSerrorz = z - zls;

17 Nonlinear Least Squares Method When f is not linear, we directly minimize E : R N R r ( E(c) = yk f(x k, c) ) 2. k=1 A twice differentiable function E : R N R has a local minimum at c = (c 1, c 2,..., c N ) if - The gradient, the vector of the 1st partial derivatives at c is zero. - The Hessian, the matrix of the 2nd partial derivatives at c, has all positive eigenvalues. - Use, e.g., fminunc, lsqnonlin. Provide the gradient and the Hessian if possible, which will speed up the calculations. If E : R N R is not differentiable, use fminsearch. Now, lets see examples on how to use fminunc, lsqnonlin, fminsearch with Nonlinear LS problems.

18 Nonlinear Least Squares Method Example 3 {(x k, y k ), k = 1,..., r} in week3-example3.mat, We want to fit this data to the model Y = f(x, c), where f(x, c) = c 1 + cos(c 2 X + c 3 ) Form r ( E(c 1, c 2, c 3 ) = yk c 1 cos(c 2 x k + c 3 ) ) 2 k=1 Write a Matlab function E.m, where the inputs are c,x,y. Use a Matlab function to find the minimum of G. For example, >> c0 = [1,2,3]; c = fminsearch(@(c)e(c,x,y), c0) >> c0 = [1,0,0]; c = fminunc(@(c)e(c,x,y), c0)

19 Outline 1 Optimization with Matlab 2 Least Squares problem Linear Least Squares Method Nonlinear Least Squares Method 3 1 norm minimization

20 1 norm minimization We use 1 norm minimization if we know that The vector y - f(x,c) has only a few nonzero entries. 1 norm minimization problem Find c = (c 1, c 2,..., c N ), which minimizes E(c) = r yk f(x k, c) k=1 among all possible choices (unconstrained) of parameters c R N. In most cases, 1 norm minimization results in better outcome compared to LS.

21 1 norm minimization Example 4 {(x k, y k ), k = 1,..., r} in week3-example4.mat, We know that the data strictly obeys the model Y = f(x, c), f(x, c) = c 1 + cos(c 2 X + c 3 ) When collecting data, only a few y k s are corrupted. We want to find the unknown parameters c 1, c 2, c 3. We use fminsearch since E is not differentiable everywhere. >> E sum(abs( c(1)+cos(c(2)*x+c(3))-y )) ); >> c0 =[1,2,1] ; c = fminsearch(@(c)e(c,x,y), c0)

22 1 norm minimization Example y data Y = 4 + cos(2x 3) x

23 Recommended Reading Scientific Computing with MATLAB and Octave by Quarteroni & Saleri, Chapter 3, Sections 3.1 and 3.4.