Numerical Optimization

Transcription

1 Numerical Optimization General Setup Let (. be a unction such that bn R R where b is a vector o unnown parameters. In many cases, b will not have a closed orm solution. We will estimate b by minimizing some loss unction., Popular loss unction: A sum o squares. Let {y t, t } be a set o measurements/constraints. That is, we it (., in general a nice smooth unction, to the data by solving: 1, min b yt t b t,or min b with y b t t t t t 1

2 Finding a Minimum - Review When is twice dierentiable, a local minima is characterized by: 1. (b min =0 (.o.c. hhbt T. 0, or all small enough (s.o.c. min hhusual approach: Solve (b min =0 or b min Chec H(b min is positive deinitive. Sometimes, an analytical solution to (b min =0 is not possible or hard to ind. For these situations, a numerical solution is needed. Notation: (.: gradient (o a scalar ield. : Vector dierential operator ( del. Finding a Univariate Minimum Finding an analytic minimum o a simple univariate sometimes is easy given the usual.o.c. and s.o.c. Eample: Quadratic Function U( 5 4 U ( 10 * 4 0 U ( 10 0 * 5 Analytical solution (rom.o.c.: *= 0.4. The unction is globally conve, that is *=/5 is a global minimum. Note: To ind *, we ind the zeroes o the irst derivative unction.

3 Finding a Univariate Minimum Minimization in R: We can easily use the popular R optimizer, optim to ind the minimum o U( --in this case, not very intersting application! Eample: Code in R > <- unction( (5*^-4*+ > optim(10,, method="brent",lower=-10,upper=10$par [1] 0.4 >curve(,rom=-5,to=5 Finding a Univariate Minimum Straightorward transormations add little additional compleity: U ( e 5 4 U ( U ( * 10 * 4 0 * 5 Again, we get an analytical solution rom the.o.c. => *=/5. Since U(. is globally conve, *=/5 is a global minimum. Usual Problems: Discontinuities, Unboundedness 3

4 Finding a Univariate Minimum - Discontinuity Be careul o discontinuities. Need to restrict range or. Eample: 5 4 ( Finding a Univariate Minimum - Discontinuity This unction has a discontinuity at some point in its range. I we restrict the search to points where >-10, then the unction is deined at all points ( (10 * 4( * 10 [5 * '( ( * * * ] 0 Ater restricting the range o, we ind an analytical solution as usual i.e., by inding the zeroes o (. 4

5 Finding a Univariate Minimum - Unbounded '( Finding a Univariate Minimum No analytical Solution So ar, we have presented eamples were an analytical solution or close solution was easy to obtain rom the.o.c. In many situations, inding an analytical solution is not possible or impractical. For eample: ( ( sin( (1 / 6 4 In these situations, we rely on numerical methods to ind a solutions. Most popular numerical methods are based on iterative methods. These methods provide an approimation to the eact solution, *. We will concentrate on the details o these iterative methods. 5

6 Numerical solutions: Iterative algorithm Eample: We want to minimze ( = We use the R leible minimizer, Optim. > curve(, rom=-3, to=5 > <- unction( (^6-4*^5-*^3+*+40 > optim(0.5,, method="brent", lower=-10, upper=10$par [1] > curve(, rom=-3, to=5 We get the solution *= Optim oers many choices to do the iterative optimization. In our eample, we used the method Brent, a miture o a bisection search and a secant method. We will discuss the details o both methods in the net slides. Numerical solutions: Iterative algorithm Old method: There is a Babylonian method. Iterative algorithms provide a sequence o approimations { 0, 1,, n }such that in the limit converge to the eact solution, *. Computing +1 rom is called one iteration o the algorithm. Each iteration typically requires one evaluation o the objective unction (or and evaluated at. An algorithm needs a convergence criterion (or, more general, a stopping criterion. For eample: Stop algorithm i ( +1 -( < pre-speciied tolerance 6

7 Iterative algorithm Goal o algorithm: Produce a sequence {}={ 0, 1,, n }such that ( 0 > ( 1 > > ( n Algorithm: Setch - Start at an initial position 0, usually an initial guess. - Generate a sequence {}, which hopeully convergence to *. - {} is generated according to an iteration unction g, such that +1 = g(. Algorithm: Speed The speed o the algorithm depends on: the cost (in lops o evaluating ( (and, liely, (. the number o iterations. Iterative algorithm Characteristics o algorithms: - We need to provide a subroutine to compute and, liely, at. - The evaluation o and can be epensive. For eample, it may require a simulation or numerical integration. Limitations o algorithms - There is no algorithm that guarantees inding all solutions - Most algorithms ind at most one (local solution - Need prior inormation rom the user: an initial guess, an interval that contains a zero, etc. 7

8 Speed o algorithm Suppose * with (* = 0 -i.e., we ind the roots o. Q: How ast does go to *? Error ater iterations: - absolute error: - * - relative error: - * / * (deined i * 0 The number o correct digits is given by: -log 10 [ - * / * ] (when it is well deined i.e., * 0 and [ - * / * ] 1. Speed o algorithm Rates o Convergence Rates o convergence o a sequence with limit *. - Linear convergence: There eists a c (0, 1 such that +1 - * c - * or suiciently large - R-linear convergence: There eists c (0, 1, M > 0 such that - * Mc or suiciently large - Quadratic convergence: There eists a c > 0 such that +1 - * c - * or suiciently large - Superlinear convergence: There eists a sequence c, with c 0 s.t * c - * or suiciently large. 8

9 Speed o algorithm Interpretation Assume * 0. Let r = -log 10 [ - * / * ] - r the number o correct digits at iteration. - Linear convergence: We gain roughly log 10 c correct digits per step: r +1 r log 10 c - Quadratic convergence: For suiciently large, the number o correct digits roughly doubles in one step: r +1 log 10 (c * + r - Superlinear convergence: Number o correct digits gained per step increases with : r +1 r Speed o algorithm Eamples Let * = 1. The number o correct digits at iteration, r, can be approimated by: r = -log 10 [ - * / * ] We deine 3 sequences or with dierent types o convergence: 1. Linear convergence: +1 = Quadratic convergence: +1 = 1 + ( Superlinear convergence: +1 = 1 + (1/(+1 For each sequence we calculate or = 0, 1,..., 10. 9

10 Speed o algorithm Eamples Sequence 1: we gain roughly -log 10 (c = 0.3 correct digits per step +1-1 / - 1 = / +1 = 0.5 c (c =0.5 Sequence : r almost doubles at each step Sequence 3: r per step increases slowly but it gets up to speed later / - 1 = (+ 1 / ( Convergence Criteria An iterative algorithm needs a stopping rule. Ideally, it is stopped because the solution is suiciently accurate. Several rules to chec or accuracy: - X vector criterion: +1 - < tolerance - Function criterion: ( +1 ( < tolerance - Gradient criterion: ( +1 < tolerance I none o the convergence criteria is met, the algorithm will stop by hitting the stated maimum number o iterations. This is not a convergence! I a very large number o iterations is allowed, you may want to stop the algorithm i the solution diverges and or cycles. 10

11 Numerical Derivatives Many methods rely on the irst derivative: '(. Eample: Newton s method s requires '( and ''(. Newton s method algorithm: +1 = λ '( / ''( It is best to use the analytical epression or (. But, it may not be easy to calculate and/or epensive to evaluate. In these situations it may be appropriate to approimate ( numerically by using the dierence quotient: '( h, Then, pic a small h and compute at using: '(,,1,1 h h h Numerical Derivatives Graphical View Approimation errors will occur. For eample, using the traditional deinition, we have that the secant line diers rom the slope o the tangent line by an amount that is approimately proportional to h. 11

12 Numerical Derivatives -point Approimation We can approimate '( with another two-point approimation: [ '( h ] [ h] h h h Errors may cancel out on both sides. For small values o h this is a more accurate approimation to the tangent line than the one-sided estimation. h Q: What is an appropriate h? Floating point considerations (and cancellation errors point out to choose an h that s not too small. On the other hand, i h is too large, the irst derivative will be a secant line. A popular choice is sqrt(ε, where ε is o the order I woring with returns (in %, h= is ine. Source o errors - Computation We use a computer to eecute iterative algorithms. Recall that iterative algorithms provide a sequence o approimations that in the limit converge to the eact solution, *. Errors will occur. (1 Round-o errors: A practical problem because a computer cannot represent all R eactly. Eample: Evaluate two epression or same unction at =0.000: ( = [1-cos (]/ and g( = [sin (]/ (=0.000 = and g(=0.000 = ( Cancelation errors: a and b should be the same (or almost the same, but errors during the calculations maes them dierent. When we subtract them the dierence is no longer zero. 1

13 Source o errors Truncation and Propagation (3 Truncation errors: They occurs when the iterative method is terminated, usually ater a convergence criterion is met. (4 Propagation o errors: Once an error is generated, it will propagate. This problem can arise in the calculation o standard errors, CI, etc. Eample: When we numerically calculate a second derivative, we use the numerically calculated irst derivate. We potentially have a compounded error: an approimation based on another approimation. Source o errors Data Ill-conditioned problem This is a data problem. Any small error or change in the data produce a large error in the solution. No clear deinition on what large error means (absolute or relative, norm used, etc. Usually, the condition number o a unction relative to the data (in general, a matri X, κ(x, is used to evaluate this problem. A small κ(x, say close to 1, is good. Eample: A solution to a set o linear equations, A = b Now, we change b to (b + b => new solution is ( +, which satisies A( + = (b + b -The change in is = A -1 b - We say, the equations are well-conditioned i small b results in small. (The condition o the solution depends on A. 13

14 Source o errors - Algorithm Numerical stability It reers to the accuracy o an algorithm in the presence o small errors, say a round-o error. Again, no clear deinition o accurate or small. Eamples: A small error grows during the calculation and signiicantly aects the solution. A small change in initial values can produce a dierent solution. Line Search Line search techniques are simple optimization algorithms or onedimensional minimization problems. Considered the bacbone o non-linear optimization algorithms. Typically, these techniques search a braceted interval. Oten, unimodality is assumed. Usual steps: - Start with bracet [ L, R ] such that the minimum * lies inside. - Evaluate ( at two point inside the bracet. - Reduce the bracet in the descent direction (. - Repeat the process. Note: Line search techniques can be applied to any unction and dierentiability and continuity is not essential. 14

15 Line Search Basic Line Search (Ehaustive Search 1. Start with a [ L, R ] and divide it in T intervals.. Evaluate in each interval (T evaluations o. Choose the interval where is smaller. 3. Set [ L, R ] as the endpoints o the chosen interval => Bac to Continue until convergence. Key step: Sequential reduction in the bracets. The ewer evaluations o unctions, the better or the line search algorithm. - There are many techniques to reduce the bracets: - Dichotomous i.e., divide in parts- search - Fibonacci series - Golden section Line Search: Dichotomous Braceting Very simple idea: Divide the bracet in hal in each iteration. Start at the middle point o the interval [a,b]: (a+b/. Then evaluate (. at two points near the middle ( 1 &. Near :. Then, evaluate the unction at 1 and.. Move the interval in the direction o the lower value o the unction. a 1 b The interval size ater 1 iteration (& evaluations: b 1 a 1 = (1/ b 0 a

16 Line Search: Dichotomous Braceting - Steps a 1 Steps: 0 Assume an interval [a,b] 1 Find 1 = a + (b-a/ - = a+(b-a/ + / Compare ƒ( 1 andƒ( b ( is the resolution 3 I ƒ( 1 <ƒ( then eliminate > and set b = I ƒ( 1 >ƒ( then eliminate < 1 and set a = 1 I ƒ( 1 =ƒ( then pic another pair o points 4 Continue until interval < (tolerance Line Search: Fibonacci numbers Fibonacci numbers: 1, 1,, 3, 5, 8, 13, 1, 34,... That is, the sum o the last numbers: F n =F n-1 +F n- (F 0 =F 1 =1 The Fibonacci sequence becomes the basis or choosing sequentially N points such that the discrepancy +1 1 is minimized. L L 3 a L R b L L 1 Q: How is the interval reduced? Start at inal interval, ater K iterations and go bacwards: b K-1 a K-1 = b K a K ; b K- a K- = 3 b K a K ;... ; b K-J a K-J = F J+1 b K a K b a = b - a - - b K-1 a K-1 16

17 Line Search: Fibonacci numbers Let =b-a. Then, evaluate ( at L =a+f - (b-a / F and R =a+f -1 (b-a / F We eep the smaller unctional value and its corresponding opposite end point. L L 3 Eample: ( = /(1+, ϵ [-0.6,0.75] 1=-.6+(.75+.6*1/3= => ƒ1= =-.6+(.75+.6*/3= 0.30 =>ƒ= 0.75 a L R b New ϵ [-0.6,. 0.30] L L 1 1=-.6+(.3+.6*/5=-0.4 => ƒ1=-0.67 L 1 =-.6+(.3+.6*3/5=-0.06 =>ƒ= New ϵ [-.6,-0.06] Continue until bracet < tolerance. * = Search methods - Eamples U L L L R UR Dichotomous L L UR U Fibonacci:

18 Line Search: Golden Section b a a a - b b Discard In Golden Section, you try to have b/(a-b = a/b, b*b = a*a - ab Solving gives a = (b ± b* sqrt(5/ a/b = or (Golden Section ratio Note that 1/1.618 = Note: lim n F n- /F n =.38 and lim n F n-1 /F n =.618 Golden section method uses a constant interval reduction ratio: Line Search: Golden Section - Eample Initialize: 1=a+(b-a*0.38 =a+(b-a* = ƒ(1 = ƒ( Loop: i 1 > then a = 1; 1 = ; 1 = =a+(b-a*0.618 = ƒ( else b = ; = 1; = 1 1=a+(b-a* = ƒ(1 endi a 1 b Eample: ( = /(1+, ϵ [-0.6,0.75] 1=-.6+(.75+.6*.38= => ƒ1= =-.6+(.75+.6*.618=0.343 =>ƒ=0.1 New ϵ [-0.6,. 0.34] 1=-.6+( *.38= => ƒ1= =-.6+( *618= =>ƒ= New ϵ [-.6, ] Continue until bracet < tolerance. * =

19 Pure Line Search - Remars Since only ƒ is evaluated, a line search is also called 0 th order method. Line searches wor best or unimodal unctions. Line searches can be applied to any unction. They wor very well or discontinuous and non-dierentiable unctions. Very easy to program. Robust. They are less eicient than higher order methods i.e., methods that evaluate ƒ and ƒ. Golden section is very simple and tends to wor well (in general, ewer evaluations than most line search methods. Line Search: Bisection The bisection method involves shrining an interval [a, b] nown to contain the root (zero o the unction, using (=> a 1 st order method. Bisection: Interval is replaced by either its let or right hal at each. Method: - Start with an interval [a, b] that satisies (a (b < 0. Since is continuous, the interval contains at least one solution o ( = 0. - In each iteration, evaluate at the midpoint o the interval [a,b], '((a+b/. - Reduce interval: Depending on the sign ', we replace a or b with the midpoint value, (a + b/. (The new interval still satisies (a (b<0. Note: Ater each iteration, the interval [a, b]ishalved: a - b = (1/ a 0 b 0 19

20 Line Search: Bisection - Eample 5 4 Eample: ( 10 - We restrict the range o to >-10. -Steps: (1 Start with interval [a=-8, b=0]. The bisection o that interval is 6. At =6, (=6= is.88 > 0 (. increasing. (1a Eclude subinterval [6,0] rom search: New interval [a=-8, b=6]. Bisection at -1. ( New interval [-8,6]. At (=-1= < 0 (. decreasing (3 New interval becomes [a=-1,b=6]. Bisection at.5. - Continue until interval < (tolerance Since at each iteration, the interval is halved. Then, * (1/ a 0 b 0 R-linear convergence. Line Search : Bisection Eample Graph o ollowing unction: 5 4 ( 10 0

21 Line Search : Bisection Eample Lower Upper Midpoint Gradient Replace Upper Lower Upper Upper Lower Lower Upper Upper Lower Lower Lower Upper Upper Lower Lower We can calculate the required or convergence, say with ε=0.00: log[(b 0 a 0 /ε ] [1/log(] = log(8/.00/log( = Line Search : Bisection Eample in R bisect <- unction(n, lower, upper, tol=.000,... {.lo <- n(lower,....hi <- n(upper,... eval <- i (.lo *.hi > 0 stop("root is not braceted in the speciied interval\n" chg <- upper - lower while (abs(chg > tol {.new <- (lower + upper /.new <- n(.new,... i (abs(.new <= tol brea i (.lo *.new < 0 upper <-.new i (.hi *.new < 0 lower <-.new chg <- upper - lower eval <- eval + 1 } list( =.new, value =.new, evals=eval } # Eample => unction to minimize = (a*^-4*+/(+10 # irst derivative => n1= (*a*-4/(+10-(a*^-4*+/(+10^ n1 <- unction(, a { (*a*-4/(+10-(a*^-4*+/(+10^ } bisect(n1, -8, 0, a=5 1

22 Descent Methods Typically, the line search pays little attention to the direction o change o the unction i.e., '(. Given a starting location, 0, eamine ( and move in the downhill direction to generate a new estimate, 1 = 0 + δ Q: How to determine the step size δ? Use the irst derivative: '(. Steepest (Gradient descent Basic algorithm: 1. Start at an initial position 0. Until convergence Find minimizing step δ, which will be a unction o. +1 = + δ In general, it pays to multiply the direction δ by a constant, λ, called step-size. That is, +1 = + λ δ To determine the direction δ, note that a small change proportional to '( multiplied by (-1 decreases (. That is, +1 = - λ '(

23 Steepest descent Theorem Theorem: Given a unction : R n R, dierentiable at 0, the direction o steepest descent is the vector - ( 0. Proo: Consider the unction z(λ = ( 0 + λ u (u: unit vector u =1 By chain rule: d1 d dn z'( u1 u... ( u u Thus, z (0 = ( 0 u = ( 0 u cos(θ (dot product = ( 0 cos(θ (θ: angle between ( 0 & u Then, z (0 is minimized when θ= z (0 = - ( 0. Since u is a unit vector ( u =1 u = - ( 0 / ( 0 n u n n 0 Steepest descent Algorithm Since u is a unit vector u =- ( 0 / ( 0 The method becomes the steepest descent, due to Cauchy (1847. The problem o minimizing a unction o N variables can be reduced to a single variable minimization problem, by inding the minimum o z(λ or this choice o u: Find λ that minimizes z(λ = ( - λ ( Algorithm: 1. Start with 0. Evaluate ( 0.. Find λ 0 and set 1 = 0 - λ 0 ( 0 3. Continue until convergence. Sequence or {}: +1 = - λ ( 3

24 Steepest descent Algorithm Note: Two ways to determine λ, in the general +1 = - λ ( 1. Optimal λ. Select λ that minimizes g(λ = ( - λ (.. Non-optimal λ can be calculated using line search methods. Good property: The method o steepest descent is guaranteed to mae at least some progress toward * during each iteration. (Show that z (0 <0, which guarantees there is a λ>0, such that z(λ<z(0. It can be shown that the steepest descent directions rom +1 and are orthogonal, that is ( +1 ( =0. For some unctions, this property maes the method to zig-zag" (or ping pong rom 0 to *. Steepest descent Graphical Eample min 1 1 4

25 Steepest descent Limitations Limitations: - In general, a global minimum is not guaranteed. (This issue is also a undamental problem or the other methods. - Steepest descent can be relatively slow close to the minimum: going in the steepest downhill direction is not eicient. Technically, its asymptotic rate o convergence is inerior to many other methods. - For poorly conditioned conve problems, steepest descent increasingly 'zigzags' as the gradients point nearly orthogonally to the shortest direction to a minimum point. - Steepest descent Eample (Wiipedia Steepest descent has problems with pathological unctions such as the Rosenbroc unction. Eample: ( 1, = ( ( - 1 This unction has a narrow curved valley, which contains *. The bottom o the valley is very lat. Because o the curved lat valley the optimization zig-zags slowly with small stepsizes towards *=(1,1. 5

26 Newton-Raphson Method (Newton s method It is a method to iteratively ind roots o unctions (or a solution to a system o equations, (=0. Idea: Approimate ( near the current guess by a unction ( or which the system o equations (=0 is easy to solve. Then use the solution as the net guess +1. A good choice or ( is the linear approimation o ( at : ( ( + ( ( -. Solve the equation ( +1 =0, or the net +1. Setting (=0: 0 = ( + ( ( Or +1 = - ( -1 ( (NR iterative method Newton-Raphson Method History History: - Heron o Aleandria (10 70 AD described a method (called Babylonian method to iteratively approimate a square root. - François Viète ( developed a method to approimate roots o polynomials. - Isaac Newton ( in 1669 (published in 1711 improved upon Viète s method. A simpliied version o Newton s method was published by Joseph Raphson ( in Though, Newton (and Raphson did not see the connection between his method and calculus. - The modern treatment is due to Thomas Simpson (

27 7 ( ( X +1 X X B C A ( ( 1 1 ( '( AC AB tan( We want to ind the roots o a unction (. We start with: ( ' ( ( ' ( 0 ( ' ( ( ( ' ( 1 1 y y Newton-Raphson Method Univariate Version Alternative derivation: Note: At each step, we need to evaluate and. Iterations: Starting at 0 inscribe triangles along ( = 0. Newton-Raphson Method Graphical View

28 Newton-Raphson Method Minimization We can use NR method to minimize a unction. Recall that '(* = 0 at a minimum or maimum, thus stationary points can be ound by applying NR method to the derivative. The iteration becomes: 1 '( ''( We need ''( 0; otherwise the iterations are undeined. Usually, we add a step-size, λ, in the updating step o : +1 = λ '( / ''(. Note: NR uses inormation rom the second derivative. This inormation is ignored by the steepest descent method. But, it requires more computations. NR Method As a Quadratic Approimation When used or minimization, the NR method approimates ( by its quadratic approimation near. Epand ( locally using a nd-order Taylor series: (+δ = ( + '( δ + ½ ''( (δ + o(δ ( 0 +δ= ( 0 +'( 0 δ+½ ''( 0 (δ 0 Findtheδ which minimizes this local quadratic approimation: δ = '(/ ''( Update: +1 = λ '(/ ''(. 8

29 NR Method N-variate Version TheN-variate version o the NR method is based on the 1st-order Taylor series epansion o (= 0: Then, ( ( * ( * ( * ( 0 * 1 * ( ( 1 1 ( ( H ( t1 t t t t t * It is always convenient to add a step-size λ. Algorithm: 1. Start with 0. Evaluate ( 0 andh( 0.. Find λ 0 and set 1 = 0 - λ 0 H( 0-1 ( 0 3. Continue until convergence. NR Method Bivariate Version - Graph hhhht 1 T T 1 min1 1 -H 9

30 NR Method Properties I H( is pd, the critical point is also guaranteed to be the unique strict global minimizer o (. For quadratic unctions, NR method applied to ( converges to in one step; that is, =1 = *. I ( is not a quadratic unction, then NR method will generally not compute a minimizer o ( in one step, even i its H( is pd. But, in this case, NR method is guaranteed to mae progress. Under certain conditions, the NR method has quadratic convergence, given a suiciently close initial guess. NR Method Eample Find the roots or the ollowing unction: Set 0 =.05. Use the Newton s method o inding roots o equations to ind a The root. b The absolute relative approimate error at the end o each iteration. c The number o signiicant digits at least correct at the end o each iteration. 30

31 NR Method Eample Calculate ( Iterations: ' ' Iteration 1( 0 = ' NR Method Eample Estimate o the root and abs relative error or the irst iteration. The absolute relative approimate error ε α at the end o Iteration 1 is a % Number o signiicant digits at least correct: 0. (Suppose we use as stopping rule ε α < 0.05%. 31

32 NR Method Eample Iteration ( 1 = ' Asolute relative approimate error ε α at the end o Iteration is: 4 a Number o signiicant digits at least correct: % NR Method Eample 3 Iteration 3 ( = ' Absolute relative approimate error ε α at the end o Iteration 3 is: 4 a % 64 Number o signiicant digits at least correct: 4. ( ε α <.05% stop. 3

33 NR Method Eample (Code in R 65 > # Newton Raphson Method > > 1<-unction({ + return(^3-.165*^ ; + } > > d1<-unction({ + return(3*^-.33* ; + } > NR.method<-unction(unc,dunc,{ + i (abs(dunc(< { + return (; + }else{ + return(-unc(/dunc(; + } + } > curve(1, -.05,.15 > > iter=0; > n<-null; > n[1]=.05; > n=1; > > while(n<iter{ + n=n+1 + n<-c(n,nr.method(1,d1,n[n-1]; + i(abs(n[n]-n[n-1]< brea; + } > n [1] NR Method Convergence Under certain conditions, the NR method has quadratic convergence, given a suiciently close initial guess. 0 ( * ( d ( d ( * d ( ~ ( d * But 0 ( d ( d Subtracting (* rom above: ( 1 d ( d Mean Value theorem truncates Taylor series ( (* 1 by Newton deinition * d ( ~ ( d Then, solving or ( +1 - : d d ( [ ( ] ( ( d d 1 * 1 * * 33

34 NR Method Convergence From d d ( [ ( ] ( ( d d 1 * 1 * d 1 d Let [ ( ] ( K d d then K 1 * * Local Convergence Theorem I a d d b d d bounded away rom zero bounded => Convergence is quadratic. bounded Then, NR method converges given a suiciently close initial guess (and convergence is quadratic. K is Convergence Chec Chec ( to avoid alse convergence. Chec more than one criterion. ( Method Convergence 1 NR 1 1 1* 34

35 NR Method Convergence Local Convergence Convergence depends on initial values. Ideally, select 0 close to *. Note: Inlection point or lat regions can create problems. demo NR Method: Limitations Local results Multiple roots: Dierent initial values tae us to dierent roots. For the equation For 0 = 0.05, > n [1] For 0 = 0.15, > n [1] For 0 = -0.05, > n [1]

36 Division by zero. During the iteration we get such that ( =0. When ( 0 (lat region o, the algorithm diverges. Method Limitations (=0 or ( 0 Both problems can be solved by using dierent 0 (popular solution ( 0 10NR 1 courtesy Alessandra Nardi UCB NR Method: Limitations Division by zero Division by zero For the equation the NR method reduces to i1 3 6 i i i i For 0 = 0, or 0 = 0., the denominator will equal zero. i 36

37 NR Method: Limitations Inlection Points Divergence at inlection points Selection o 0 or an iteration value o the root that is close to the inlection point o the unction ( may start diverging away rom the root in the Newton-Raphson method. 0 Eample: Find the root o the equation: i The NR method reduces to i 1 i. 3 i 1 The root starts to diverge at Iteration 6 because the previous estimate o is close to the inlection point o =1.. Note: Ater >1, the root converges to the root o *=0.. 3 NR Method: Limitations Inlection Points Divergence near inlection point or: Iteration Number i

38 NR Method: Limitations Oscillations It is possible that ( and ( +1 have opposite values (or close enough and the algorithm becomes loc in an ininite loop. Again, dierent 0 can help to move the algorithm away rom this problem. NR Method: Limitations Oscillations Oscillations near local maimum and minimum Results obtained rom the Newton-Raphson method may oscillate about the local maimum or minimum without converging on a root but converging on the local maimum or minimum. Eventually, it may lead to division by a number close to zero and may diverge. For eample or 0 the equation has no real roots. 38

39 NR Method: Limitations Oscillations Oscillations near local maima and minima in NR method. Iteration Number i (λ ( ε α 6 5 ( Oscillations around local minima or NR Method: Limitations Oscillations Note: Let s add a step-size, λ, in the updating step o : +1 = λ '( / ''(. λ = (.8,.9, 1, 1.1, 1. Iteration i (λ ( ( i (λ ε α => The step-size improves the value o unction! 39

40 NR Method: Limitations Root Jumping Root Jumping In some cases where the unction ( is oscillating and has a number o roots, one may choose 0 close to a root. However, the guesses may jump and converge to some other root. Eample: Choose sin ( It will converge to = 0, not to Root jumping rom intended location o root or sin 0. NR Method: Advantages and Drawbacs Advantages - Converges ast (quadratic convergence, i it converges. - Requires only one guess. Drawbacs - Initial values are important. -I ( =0, algorithm ails. - Inlection points are a problem. - I '( is not continuous the method may ail to converge. - It requires the calculation o irst and second derivatives. - Not easy to compute H(. - No guarantee o global minimum. 40

41 NR Method: Numerical Derivatives NR algorithm: +1 = λ '( / ''(. It requires '( and ''(. We can use the quotient ratio as a starting point, which delivers a one-point approimation: '( h,,,1,1 We can approimate '( with a two-point approimation, where errors may cancel out: [ h ] [ h ] h h '( h h This approimation produces the secant method ormula or +1 : +1 = ( / '( = ( /{[ ( - ( -1 ] /[ - -1 ].} = ( [ - -1 ] / [ ( - ( -1 ] (secant method, with a slower convergence than NR method (1.618 relative to. h NR Method: Numerical Derivatives We also need ''(. A similar approimation or the second derivative ''( is also done. Let s suppose we use a one-point approimation: ''( ' ' ' h ',,1,,1 The approimation problems or the nd derivative become more serious: The approimation o the 1st derivative is used to approimate the nd derivative. A typical propagation o errors situation. h Eample: In the previous R code, we can use: > # num derivatives > d1<-unction(,h=0.001{ + return(((+h-(/h; + } 41

42 Multivariate Case - Eample : N-variate NR-method iteration: ma Starting with 0 =(1,1,1 ( ( ( ( ( t1 H ( 1 t1 t t Eample: ma 3 4 s.t Solution * = ( Multivariate Case Eample in R library(numderiv <- unction(z {y <- -((100-*z[1]-3*z[]-z[3]^.*z[1]^.3*z[]^.4*z[3]^.1 return(y } <- c(1,1,1 # numerical gradient & hessian # d1 <- grad(,, method="richardson" # d1 <- hessian(,, method="comple" ma_ite = 10; tol=.0001 #NR i (abs((.new - ( < tol brea <-.new } return(p[1:(i-1,] } NR_num(,tol,,ma_ite NR_num <- unction(,tol,,n { [,1] [,] [,3] [,4] i<-1;.new<- p <- matri(1,nrow=n,ncol=4 while(i<n { d1 <- grad(,, method="richardson" d1 <- hessian(,, method="comple".new<--solve(d1%*%d1 p[i,] <- rbind(.new,(.new i<-i+1 [1,] [,] [3,] [4,] [5,] [6,] [7,] [8,]

43 Multivariate Case Eample in R Note: H( -1 can create problems. I we change the calculation method to the Richardson etrapolation, with 0 =(1,1,1, we get a NaN result or.new ater 3 iterations => H( -1 is not pd! > d1 <- hessian(,, method= Richardson" But, i we use the Richardson etrapolation, with 0 =(,,, we get [,1] [,] [,3] [,4] [1,] [,] [3,] [4,] [5,] [6,] Lots o compuational trics are devoted to deal with these situations. Multivariate Case Computational Drawbacs Basic N-variate NR-method iteration: 1 1 H( ( As illustrated beore, H( -1 can be diicult to compute. In general, the inverse o H is time consuming. (In addition, in the presense o many parameters, evaluating H can be impractical or costly. In the basic algorithm, it is better not to compute H( -1. Instead, solve H( ( +1 =-( Each iteration requires: - Evaluation o ( - Computation o H( - Solution o a linear system o equations, with coeicient matri H( and RHS matri - (. 43

44 Multivariate Case H Matri In practice, H( can be diicult to calculate. Many times, H( is just not pd. There are many trics to deal with this situation. A popular tric is to add a matri, E (usually, δi, where δ is a constant, that ensures H( is pd. That is, H( ( +E. The algorithm can be structured to tae a dierent step when H( is not pd, or eample, the steepest descent. That is, H( I. Note: Beore using the Hessian to calculate standard errors, mae sure it is pd. This can be done by computing the eigenvalues and checing they are all positive. Multivariate Case H Matri NR method is computationally epensive. The structure o the NR algorithm does not help (there is no re-use o data rom one iteration to the other. To avoid computing the Hessian i.e., second derivatives-, we'll approimate. Theory-based approimations: - Gauss-Newton: - BHHH: L ( ' ( H ( E[ ] [ ] ' L L T t t H ( ] t 1 ' Note: In the case we are doing MLE, or each algorithm, -H( can serve as an estimator or the asymptotic covariance matri or the maimum lielihood estimator o. 44

45 Modiied Newton Methods The Modiied Newton method or inding an etreme point is +1 = - S y( Note that: i S = I, then we have the method o steepest descent i S = H -1 ( and = 1, then we have the pure Newton method i y( = 0.5 T Q - b T, then S = H -1 ( = Q (quadratic case Classical Modiied Newton s Method: +1 = - H -1 ( 0 y( Note that the Hessian is only evaluated at the initial point 0. Quasi-Newton Methods Central idea underlying quasi-newton methods (a variable metric method is to use an approimation o the inverse Hessian (H -1. By using approimate partial derivatives, there is a slightly slower convergence resulting rom such an approimation, but there is an improved eiciency in each iteration. Idea: Since H( consists o the partial derivatives evaluated at an element o a convergent sequence, intuitively Hessian matrices rom consecutive iterations are close" to one another. Then, it should be possible to cheaply update an approimate H( rom one iteration to the other. With an NN matri D: D = A + A u, A u : update, usually o the orm uv T. 45

46 Quasi-Newton Methods Or D +1 = A + A u, A u : update, usually o the orm uv T, where u and v are N1 given vectors in R n. (This modiication o A to obtain D is called a ran-one update, sinceuv T has ran one. In quasi-newton methods, instead o the true Hessian, an initial matri H 0 is chosen (usually, H 0 = I, which is subsequently updated by an update ormula: H +1 = H + H u, where H u is the update matri. Since in the NR method, we really care about H -1, not H. The updating is done or H -1. Let B = H -1 ; then the updating ormula or H -1 is also o the orm: B +1 =B +Bu Quasi-Newton Methods Conjugate Gradient Conjugate Method or Solving A=b - Two non-zero vectors u and v are conjugate (with respect to A i u T Av =0 (A=H symmetric and pd =><u,av>= u T Av. - Suppose we want to solve A=b. Wehaven mutually conjugate directions, P (a basis o R n. Then, *=Σ α i p i. - Thus, b = A*= Σ α i Ap i. -Foranyp K P, Or p KT b = p KT A*= Σ α i p KT Ap i = α K p KT Ap K α K = p KT b / p KT Ap K Method or solving A=b: Find a sequence o n conjugate directions, and then compute the coeicients α K. 46

47 Quasi-Newton Methods Conjugate Gradient Conjugate Gradient Methods - Conjugate gradient methods build up inormation on H. - From our standard starting point, we tae a Taylor series epansion around the point + s : Or s ( ( '[ s s ( s s KT Q K = s KT H s K ( ( ( H ( ] ( s s s ' H ( s Note: H(, scaled by s, can be approimated by the change in the gradient: q = ( -1 - (. Hessian Matri Updates Deine g = (. p = +1 - and q = g +1 - g Then, q = g +1 - g H( p (secant condition. I the Hessian is constant: H( =H => q = H p I H is constant, then the ollowing condition would hold as well H q i =p i 0 i This is called the quasi-newton condition (also, inverse secant condition. Let B = H -1, then the quasi-newton condition becomes: p i = B +1 q i 0 i. 47

48 Ran One and Ran Two Updates Substitute the updating ormula B +1 = B + B u and we get: p i = B q i + B u q i (1 (remember: p i = i+1 - i and q i =g i+1 -g i Note: There is no unique solution to inding the update matri B u A general orm is B u = a uu T + b vv T where a and b are scalars and u and v are vectors satisying condition (1. Note: a uu T & b vv T are symmetric matrices o (at most ran one. Quasi-Newton methods that tae b =0areusingran one updates. Quasi-Newton methods that tae b 0areusingran two updates. Note that b 0 provides more leibility. Update Formulas: Ran One Simple approach: Add new inormation to the current B.For eample, using a ran one update: B u = B +1 - B = uv T => p i =(B + uv T q i => p i - B q i = uv T q i => u =[1/(v T q i ] (p i - B q i Set v T =(p i - B q i => B +1 = B +[1/(v T q i ] (p i - B q i v T => B +1 = B + [1/((p i - B q i T q i ] (p i - B q i (p i - B q i T No systems o linear equations need to be solved during an iteration; only matri-vector multiplications are required, which are computationally simpler. 48

49 Ran One and Ran Two Updates Ran one updates are simple, but have limitations. It is diicult to ensure that the approimate H is pd. Ran two updates are the most widely used schemes, since it is easier to get the approimate H to be pd. Two popular ran two update ormulas: - Davidon -Fletcher-Powell (DFP ormula - Broyden-Fletcher-Goldarb-Shanno (BFGS ormula. Davidon-Fletcher-Powel (DFP Formula Earliest (and one o the most clever schemes or constructing the inverse Hessian, H -1, was originally proposed by Davidon (1959 and later developed by Fletcher and Powell (1963. Ithasthenicepropertythat,oraquadratic objective, it simultaneously generates the directions o the conjugate gradient method while constructing H -1 (or B. Setch o derivation: - Ran two update or B: B +1 = B + a uu T + b vv T (* - Recall B +1 must satisy the Inverse Secant Condition: B +1 q = p - Post-multiply (* by q : p - B q =a uu T q + b vv T q (=0! - The RHS must be a linear combination o p and B q, and it is already a linear combination o u and v. Set u= p & v=b q. -This maes a u T q = 1, & b v T q = -1 - DFP update ormula: B+1 = B + pp T ptq Bqq TB qtbq 49

50 DFP Formula - Remars It can be shown that i p is a descent direction, then each B is pd. The DFP Method beneits rom picing an arbitrary pd B 0, instead o evaluating H( 0-1, but in this case the beneit is greater because computing an inverse matri is very epensive. I you select B 0 = I, we use the steepest descent direction. Once B is computed, the DFP Method computes +1 = - λ B (, where λ >0 is chosen to mae sure ( +1 < ( (use an optimal search or line search. Broyden Fletcher Goldarb Shanno Formula Remember secant condition: q i = H +1 p i and B q i = p i 0 i Both equations have eactly the same orm, ecept that q i and p i are interchanged and H is replaced by B (B =H (or vice versa. Observation: Any update ormula or B can be transormed into a corresponding complimentary ormula or H by interchanging the roles o B and H and o q and p. The reverse is also true. BFGS ormula update o H : Tae complimentary ormula o DFP: H +1 = H + q q T q Tp H p p TH p TH p By taing the inverse, the BFGS update ormula or B +1 is obtained: B +1 = B + ( 1 + q T B q q Tp p T p p Tq p q TB + B q p T q Tp 50

51 Some Comments on Broyden Methods BFGS ormula is more complicated than DFP, but straightorward to apply. UnderBFGS,iB is psd, then B +1 is also psd. BFGS update ormula can be used eactly lie DFP ormula. Numerical eperiments have shown that BFGS ormula's perormance is superior over DFP ormula. Both DFP and BFGS updates have symmetric ran two corrections that are constructed rom the vectors p and B q. Weighted combinations o these ormulae will thereore also have the same properties. This observation leads to a whole collection o updates, now as the Broyden amily, deined by: B =(1 w B DFP + w B BFGS where w is a parameter that may tae any real value. Quasi-Newton Algorithm 1. Input 0,B 0 (say, I, termination criteria.. For any, set S = B g. 3. Compute a step size λ (e.g., by line search on ( + λ S and set +1 = + λ S. 4. Compute the update matri B u accordingtoagivenormula(say, DFP or BFGS using the values q =g +1 -g,p = +1 -,and B. 5. Set B +1 =B +B u. 6. Continue with net until termination criteria are satisied. Note: You do have to calculate the vector o irst order derivatives g or each iteration. 51

52 Some Closing Remars Both DFP and BFGS methods have theoretical properties that guarantee superlinear (ast convergence rate and global convergence under certain conditions. However, both methods could ail or general nonlinear problems. In particular: - DFP is highly sensitive to inaccuracies in line searches. - Both methods can get stuc on a saddle-point. In NR method, a saddle-point can be detected during modiications o the (true Hessian. Thereore, search around the inal point when using quasi- Newton methods. - Update o Hessian becomes "corrupted" by round-o and other inaccuracies. All ind o "trics" such as scaling and preconditioning eist to boost the perormance o the methods. Iterative approach In economics and inance, many maimization problems involve, sums o squares: arg min S ( y i i ( ; i i i where is a nown data set and β is a set o unnown parameters. The above problem can be solved by many nonlinear optimization algorithms: - Steepest descent - Newton-Raphson - Gauss-Newton 5

53 53 Gauss-Newton Method Gauss-Newton taes advantage o the quadratic nature o the problem. Algorithm: Step 1: Initialize β = β 0 Step : Update the parameter β. Determine optimal update, Δβ. i y i i ( min arg ( ( min arg i i i i y i i i i i i i y ( ( ( arg min Taylor series epansion Note: This is a quadratic unction o Δβ. Straightorward solution. where J is the Jacobian o (β. Setting the gradient equal to zero: Δβ =(J T J -1 J T ε => LS solution! Notice the setting loos lie the amiliar linear model: T K K T T T K K ( ( ( min arg ΔβεΔβεJ J T i i i Or, J Δβ = ε (A linear system o TN equations Gauss-Newton Method

54 Gauss-Newton Method From the LS solution, the updating step involves an OLS regression: β +1 = β + (J T J -1 J T ε A step-size, λ, can be easily added: β +1 = β + λ (J T J -1 J T ε Note: The Guass-Newton method can be derived rom the Newton- Raphson s method. N-R s updating step: β +1 = β - λ H -1 (β where H (J T J (β = J T ε -i.e., second derivatives are ignored Gauss-Newton Method - Application Non-Linear Least Squares (NLLS ramewor: y i = h ( i ; β + ε i - Minimization problem: arg mins( i y h( i i ; i i - Iteration: b NLLS,+1 = b NLLS, + λ (J T J -1 J T ε where J T ε = - Σ i δh( i ; β/δβ ε i (J T J -1 =- Σ i δh( i ; β/δβ δh( i ; β/δβ Note: (J T J -1 ignored the term { δ h( i ; β/δβ δβ T ε i }. Or, b NLLS,+1 = b NLLS, + λ ( 0T 0-1 0T ε = J(b NLLS, 54

55 General Purpose Optimization Routines in R The most popular optimizers are optim and nlm: - optim gives you a choice o dierent algorithms including Newton, quasi-newton, conjugate gradient, Nelder-Mead and simulated annealing. The last two do not need gradient inormation, but tend to be slower. (The option method="l-bfgs-b allows or parameter constraints. - nlm uses a Newton algorithm. This can be ast, but i ( is ar rom quadratic, it can be slow or tae you to a bad solution. (nlminb can be used in the presence o parameter constraints. Both optim and nlm have an option to calculate the Hessian, which is needed to calculate standard errors. 55