Minimization of Static! Cost Functions!

Transcription

1 Minimization of Static Cost Functions Robert Stengel Optimal Control and Estimation, MAE 546, Princeton University, 2018 J = Static cost function with constant control parameter vector, u Conditions for a minimum in J with respect to u Analytical and numerical solutions Copyright 2018 by Robert Stengel. All rights reserved. For educational use only Static Cost Function Minimum value of the cost, J, is fixed, but may be unknown Corresponding control parameter, u*, also is fixed but possibly unknown Cost function preferably has Single minimum Locally smooth, monotonic contours away from the minimum 2

2 Vector Norms for Real Variables Norm = Measure of length or magnitude of a vector, x Scalar quantity axicab or Manhattan norm L 1 norm = x 1 = Euclidean or Quadratic Norm n i=1 x i x 1 x x = 2 x n dim(x) = n 1 L 2 norm = x 2 = ( x x) 1/2 = x x 2 2 ( 2 ++ x n ) 1/2 p Norm L p norm = x p = n i=1 x i p 1/ p Infinity Norm: L norm = x = x imax 3 p Norm (L p Norm) Example L p norm x p = n i=1 x i p 15-dimensional vector 1/ p p L p Serie As p increases, the norm approaches the value of the maximum component of the vector 4

3 Apples and Oranges Problem Suppose elements of x represent quantities with different units One solution: Vector, y, normalized by ranges of elements of x y = x = y 1 y 2 y n x 1 x 2 x n = Velocity, m / s Angle, rad emperature, K ( ) ( ) d 1 x 1 x 1 x 1max ( x 1min d 2 x 2 x 2 x 2max ( x 2min d n x n x n x nmax ( x nmin ( ) What is a reasonable definition for the norm? = d d d n x 1 x 2 x n = Dx 5 Weighted Euclidean (Quadratic) Norm of x ( ) 1/2 = y y 2 2 ( 2 ++ y m ) 1/2 ( ) 1/2 dim y = Dx 2 dim x y 2 = y y = x D Dx dim D ( ) = m 1 ( ) = n 1 ( ) = m n If m = n, D is square D D is square and symmetric If D is diagonal D D is diagonal If m n, D is not square D D is square and symmetric Rank(D) m, n > m Rank(D) n, n < m 6

4 Rank of Matrix D Maximal number of linearly independent rows or columns of D Order of the largest non-zero minor determinant of D Examples D = D = a c e b d f a b c d e f ; maximal rank 2 ; maximal rank 2 7 Quadratic Forms x D Dx x Qx = Quadratic form D D Q = Defining matrix of the quadratic form dim(q) = n x n Q is symmetric x Qx is a scalar Rank(Q) m, n > m Rank(Q) n, n < m Useful identity for the trace of the quadratic form x Qx = r( x Qx) = r( xx Q) = r( Qxx ) ( 1 n) ( n n) ( n 1) = ( 1 1) = r 1 1 ( ) = r ( n n) ( ) = r n n = Scalar 8

5 Why are Quadratic Forms Useful? 2 x 2 example J ( x) x Qx = x 1 x 2 = q 11 x q 22 x ( q 12 + q 21 )x 1 x 2 q 11 q 12 q 21 q 22 x 1 x 2 = q 11 x q 22 x q 12 x 1 x 2 for symmetric matrix 2 nd -order term of aylor series expansion of any vector function f ( x) f ( x 0 + x) = f ( x 0 ) + x ( + 1 f 2 ( x) x x=x0 ( 2 x ( x +... H.O. x 2 x=x 0 ( Gradient (as row vector) well-defined everywhere J x J x 1 J x 2 ( ) ( 2q 22 x 2 + 2q 12 x 1 ) = 2q 11x 1 + 2q 12 x 2 Large values weighted more heavily than small values Some terms can count more than others Coupling between terms can be considered Asymmetric Q can always be replaced by a symmetric Q 9 Definiteness 10

6 Definiteness of a Matrix If J = x Qx > 0 for all x 0 a) he scalar, J, is definitely positive b) he matrix Q is a positive definite matrix Q = If J = x Qx 0 for all x 0 Q is a positive semi - definite matrix Q = If J = x Qx < 0 for all x 0 Q is a negative definite matrix Q = Positive-Definite Matrix Q is positive-definite if All leading principal minor determinants are positive All eigenvalues are real and positive 3 x 3 example q 11 q 12 q 13 Q = q 21 q 22 q 23 q 31 q 32 q 33 ( ) = s 3 + a 2 s 2 + a 1 s + a 0 = ( s 1 )( s 2 )( s 3 ) det si Q 1, 2, 3 > 0 q 11 > 0, q 11 q 12 q 21 q 22 > 0, q 11 q 12 q 13 q 21 q 22 q 23 > 0 q 31 q 32 q 33 What s an eigenvalue? 12

7 Characteristic Polynomial Characteristic polynomial of a matrix, F si F det( si F) = Scalar ( si F) = (s) = s n + a n1 s n a 1 s + a 0 = s n r( F)s n a 1 s + a 0 where ( s f 11 ) f f 1n f 21 s f 22 ( )... f 2n ( ) f n1 f n2... s f nn (n x n) 13 Eigenvalues Characteristic equation of the matrix (s) = s n + a n1 s n a 1 s + a 0 0 ( )( s 2 )(...)( s n ) 0 = s 1 i are solutions that set ( s) = 0 hey are called the eigenvalues of F the roots of the characteristic polynomial a n1 = r( F), a 0 = 1 n ( ) i i i=1 14

8 Examples of Positive Definite Matrices Diagonal matrix with positive elements Q = Inertia matrix of a rigid body I = (y 2 + z 2 ) xy xz ( xy (x 2 + z 2 ) yz dm = xz yz (x 2 + y 2 ) Bo dy I xx I xy I xz I xy I yy I yz I xz I yz I zz 15 Rotation (Direction Cosine) Matrix y = Hx Projections of unit vector components of one Cartesian reference frame on another Rotational orientation of one Cartesian reference frame with respect to another H = h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 = cos 11 cos 21 cos 31 cos 12 cos 22 cos 32 cos 13 cos 23 cos 33 16

9 Euler Angles Body attitude measured with respect to inertial frame hree-angle orientation expressed by sequence of three orthogonal single-angle rotations from inertial to body frame : Yaw angle : Pitch angle : Roll angle H = cos cos cos sin sin ( * cos sin + sin sin cos cos cos + sin sin sin sin cos * sin sin + cos sin cos sin cos + cos sin sin cos cos * ) det(h) = 1 17 Euclidean Norm Rotational ransformation Position of a particle in two coordinate frames with common origin y = Hx x = H 1 y Orthonormal transformation [H 1 = H ] Non-singular matrix [det(h) = 1 0] Not a positive-definite matrix y 2 = ( y y) 1/2 = y y 2 2 ( 2 + y 3 ) 1/2 ( ) 1/2 = Hx 2 = x 2 = x x 2 2 ( 2 + x 3 ) 1/2 = x H Hx as H H = H 1 H = I 3 (positive-definite) 18

10 Conditions for a Static Minimum 19 he Static Minimization Problem Find the value of a continuous control parameter, u, that minimizes a continuous scalar cost junction, J Single control parameter min J ( u ); dim J wrt u ( ) = 1, dim ( u ) = 1 u* = argmin J ( u) Argument that minimizes J u Many control parameters min J ( u); dim J wrt u ( ) = 1, dim( u) = m 1 u* = argmin J ( u) u 20

11 Necessary Condition for Static Optimality Single control dj du u =u* = 0 i.e., the slope is zero at the optimum point Example: J = ( u 4) 2 dj du = 2 u 4 ( ) = 0 when u* = 4 Optimum point is a stationary point 21 Necessary Condition for Static Optimality Multiple controls J u u=u* = J J... u 1 u 2 J u m = 0 u=u* Gradient, defined as a row vector i.e., all the slopes are concurrently zero at the optimum point Example: J = ( u 1 4) + ( u 2 8) dj = 2( u 1 4) = 0 du 1 when u 1 * = 4 dj = 2( u 2 8) = 0 du 2 when u 2 * = 8 J u u=u* = J J u 1 u 2 4 u=u*= 8 Optimum point is a stationary point =

12 But the Slope can be Zero for More than One Reason Minimum Maximum Either Neither (Inflection Point) 23 But the Gradient can be Zero for More than One Reason Minimum Maximum Either Neither (Saddle Point) 24

13 Sufficient Condition for Extremum Single control Minimum Satisfy necessary condition plus d 2 J du 2 u =u* > 0 Maximum Satisfy necessary condition plus d 2 J du 2 u =u* < 0 i.e., the curvature is positive at the optimum point Example: J = ( u 4) 2 dj du = 2 u 4 d 2 J du = 2 > 0 2 ( ) i.e., the curvature is negative at the optimum point Example: J = ( u 4) 2 dj du = 2 u 4 d 2 J du = 2 < 0 2 ( ) 25 Sufficient Condition for a Local Minimum Multiple controls Satisfy necessary condition plus Hessian matrix 2 J u 2 u=u* = 2 J u J u 1 u J u 1 u m 2 J 2 J 2 J... 2 u 2 u 1 u 2 u 2 u m J 2 J... u m u 1 u 2 u m 2 J u m 2 u=u* > 0 i.e., J must be convex for a minimum 26

14 Minimized Cost Function, J* Gradient is zero at the minimum Hessian matrix is positive-definite at the local minimum J ( u*+u) J ( u* ) + J ( u* ) + 2 J ( u* ) +... ( ) = u J J u* 2 J u* ( ) = 1 2 u 2 J u ) u=u* ( = 0 u 2 ) u * 0 u=u* ( J ( u*+u) J ( u* ) + 2 J ( u* ) +... First variation is zero at the minimum Second variationis positive at the minimum 27 Numerical Optimization 28

15 Numerical Optimization What if J is too complicated to find an analytical solution for the minimum? or J has multiple minima? Use numerical optimization to find local and/or global solutions 29 wo Approaches to Numerical Minimization Gradient-Free Search Evaluate J and search directly for min J Gradient-Based Search Evaluate J/u and search for zero J u o = J u u=u0 = Starting guess Iterate: J u n = J + ) J u u=un(1 u u=u n(1 J u u=un such that J u n < J u n(1 30

16 Gradient-Free Search [Based on J(u)] Exhaustive grid search Unstructured random search Structured random search 31 Gradient-Free Search Structured random search Nelder-Mead (Downhill Simplex) algorithm Simulated annealing Genetic algorithm Particle-swarm optimization 32

17 Downhill Simplex Search (Nelder-Mead Algorithm)* Simplex: N-dimensional figure in control space defined by N + 1 vertices (N + 1) N / 2 straight edges between vertices 33 Search Procedure for Downhill Simplex Method Select starting set of vertices, u(0) Evaluate cost at each vertex Determine vertex with largest cost (e.g., J 1 at right) Project search from this vertex through middle of opposite face (or edge for N = 2) Reflection [equal distance along direction] Expansion [longer distance along direction] Contraction [shorter distance along direction] Shrink [replace all but best point with points contracted toward best point] Evaluate cost at new vertex (e.g., J 4 ) Drop J 1 vertex, and form simplex with new vertex Repeat until cost is small enough (termination) 34

18 Simulated Annealing Algorithm Goal: Find global minimum among local minima Approach: Randomized search, with convergence that emulates physical annealing Evaluate cost, J i, with u(i) Accept if J i < J i 1 Accept with probability Pr(E) if J i > J i 1 Probability distribution of energy state, E (Boltzmann Distribution) Pr(E) e E /k k : Boltzmanns constant : emperature Algorithm s cooling schedule accepts many bad guesses at first, fewer as iteration number, i, increases MALAB 35 Application of Annealing Principle to Search If cost decreases (J 2 < J 1 ), always accept new point If cost increases (J 2 > J 1 ), accept new point with probability proportional to generalized Boltzmann distribution Pr( Accept) e ( J 2J 1 )/k Occasional diversion from convergent path intended to prevent entrapment by a local minimum As search progresses, decrease k, making probability of accepting a cost increase smaller 36

19 Broad Characteristics of Genetic Algorithms Search based on the coding of a parameter set, not the parameters themselves Search evolves from a population of points Blind search, i.e., without gradient Probabilistic transitions from one control state to another (using random number generator) Control parameters assembled as genes of a single chromosome strand (Example: four 4-bit parameters = four genes ) John Holland, Progression of a Genetic Algorithm Most fit chromosome evolves from a sequence of reproduction, crossover, and mutation Initialize algorithm with N (even) random chromosomes, c n (two 8-bit genes or parameters in example) Evaluate fitness, F n, of each chromosome Compute total fitness, F total, of chromosome population F total = N F Bigger F is better n n=1 F 1 F 2 F 3 F 4 38

20 Genetic Algorithm: Reproduction Reproduce N additional copies of the N originals with probabilistic weighting based on relative fitness, F n /F total, of originals (Survival of the fittest) Roulette wheel selection: Pr(c n ) = F n /F total Multiple copies of most-fit chromosomes No copies of least-fit chromosomes Starting Set Reproduced Set 39 Genetic Algorithm: Crossover Arrange N new chromosomes in N/2 pairs chosen at random Interchange tails that are cut at random locations Head ail Head ail 40

21 Create New Generations By Reproduction, Crossover, and Mutation Until Solution Converges Chromosomes narrow in on best values with advancing generations F max and F total increase with advancing generations MALAB: Minimizes cost rather than maximizing fitness 41 Particle Swarm Optimization Converse of the GA: Uses multiple cost evaluations to guide parameter search directly Stochastic, population-based algorithm Search for optimizing parameters modeled on social behavior of groups that possess cognitive consistency Particles = Parameter vectors Particles have position and velocity Projection of own best (Local best) Knowledge of swarm s best Neighborhood best Global best Peregrine Falcon Hunting Murmuration of Starlings in Rome 42

22 Particle Swarm Optimization Find min J(u) = J *(u*) u Jargon : argminj(u) = u* i.e., argument of J that minimizes J Recursive algorithm to find best particle or configuration of particle values u: Parameter vector Position of the particles v: Velocity of u dim(u) = dim(v) = Number of particles 43 Particle Swarm Optimization Local best: RNG, downhill simplex, or SA step for each particle Neighborhood best: argmin of closest n neighboring points Global best: argmin of all particles v k = bv k1 + c u bestlocal u k1 k1 u k = u k1 + av k1 ( ) + d( u bestneighborhood u k1 k1 ) + e( u bestglobal u k1 k1 ) u 0 : Starting value from random number generator v 0 : Zero 44 a, b, c, d :Search tuning parameters

23 Gradient Search 45 Gradient Search to Minimize a Quadratic Function Cost function, gradient, and Hessian matrix of a quadratic function Guess a starting value of u, u o Solve gradient equation J = 1 ( 2 u u* ) R( u u* ), R > 0 = 1 ( 2 u Ru u Ru*u* Ru + u* Ru* ) J u = ( u u* ) R = 0 when u = u* 2 J = R = symmetric constant 2 u J = ( u o u* ) R = ( u o u* ) 2 J u u=uo u 2 u=u o ( u o u* ) = J R 1 u u=uo J u* = u o R 1 ( u u=uo ( row) ( column) 46

24 For a quadratic cost function J u* = u o R 1 ( u u=uo = u o 2 J ( u 2 u=u o ( 1 Optimal Value Found in a Single Step J ( u u=uo Gradient establishes general search direction Hessian fine-tunes direction and tells exactly how far to go 47 Newton-Raphson Iteration Many cost functions are not quadratic However, the surface is well-approximated by a quadratic in the vicinity of the optimum, u* J ( u*+u) J ( u* ) + J ( u* ) + 2 J ( u* ) +... J J ( u* ) = u = 0 u u=u* ( ) = 1 2 J 2 u 2 J u* ( u 2 ) u=u* * u + 0 Optimal solution requires multiple steps 48

25 Newton-Raphson Iteration Newton-Raphson algorithm is an iterative search patterned after the quadratic search u k +1 = u k 2 J u 2 u=u k ( ( 1 J u u=uk ( ( 49 Difficulties with Newton- Raphson Iteration u k +1 = u k 2 J u 2 u=u k Good when close to the optimum, but Hessian matrix (i.e., the curvature) may be Difficult to estimate from local measurements of the cost May have the wrong sign (e.g., not positive-definite) May lead to large errors in incremental control variation ( ( 1 J u u=uk ( ( 50

26 Steepest-Descent Algorithm Multiplies Gradient by a Scalar Constant ( Gain ) u k +1 = u k J u u=uk Replace Hessian matrix by a scalar constant Gradient is orthogonal to equal-cost contours ) () 51 Choice of Steepest- Descent Gain If gain is too small Convergence is slow If gain is too large Convergence oscillates or may fail Solution: Make gain adaptive 52

27 Optimal Steepest- Descent Gain Find optimal gain by evaluating cost, J, for intermediate solutions with same J u Adjustment rule for Starting estimate, J 0 First estimate, J 1, using ( ) Second estimate, J 2, using 2 If J 2 > J 1 Quadratic fit through three points to find * Else, third estimate, J 3, using 4 Use optimal gain, *, on each major iteration 53 Equality Constraints 54

28 Static Cost Functions with Equality Constraints Minimize J(u ), subject to c(u ) = 0 dim(c) = [n x 1] dim(u ) = [(m + n) x 1] 55 wo Approaches to Optimization with a Constraint 1.Use constraint to reduce control dimension u = 2.Augment the cost function to recognize the constraint u 1 u 2 Example : min J u 1,u 2 subject to c( u ) = c( u 1,u 2 ) = 0 u 2 = fcn u 1 then ( ) = J ( u 1,u 2 ) = J u 1, fcn( u 1 ) J u ( ) ( ) = J u 1 Lagrange multiplier,, is an unknown constant dim () = dim( c) = n 1 J A ( u ) = J ( u ) + c( u ) Warning Lagrange multiplier,, is not the same as an eigenvalue, 56

29 Solution Example: First Approach Cost function J = u 1 2 2u 1 u 2 + 3u Constraint c = u 2 u 1 2 = 0 u 2 = u Solution Example: Reduced Control Dimension Cost function and gradient with substitution J = u 1 2 2u 1 u 2 + 3u = u 1 2 2u 1 u = 2u u 1 28 J u 1 = 4u = 0 ( ) + 3( u 1 + 2) 2 40 Optimal solution u 1 * = 2 u 2 * = 0 J* = 36 58

30 Solution: Second Approach Partition u into a state, x, and a control, u, such that dim(x) = [n x 1] = dim(c) dim(u) = [m x 1] x u = u Add constraint to the cost function, weighted by Lagrange multiplier, u J A ( ) = J ( u ) + c( u ) ( x,u) = J ( x,u) + c( x,u) J A c is required to be zero when J A is a minimum ( ) = c c u x u = 0 59 J A x = J x + c x = 0 Solution: Adjoin Constraint with Lagrange Multiplier Gradients with respect to x, u, and are zero at the optimum point J A u = J u + c u = 0 J A = c = 0 60

31 Simultaneous Solutions for State and Control First equation: find optimal Lagrange multiplier (n scalar equations) Second and third equations: specify the state and control (n + m) scalar equations * = J c x x( ) or * c * =, x( ) + ( 2n + m) values must be found: ( x, u, ) 1 - /. 1 [Row] J x ( ) [Column] J u J x J u + c * u = 0 c x( ) c( x,u) = 0 1 c u = 0 61 Solution Example: Lagrange Multiplier Cost function Constraint Partial derivatives J x J u J = u 2 2xu + 3x 2 40 c = x u 2 = 0 = 2u + 6x = 2u 2x c x = 1 c u = 1 62

32 Solution Example: Lagrange Multiplier From first equation * = 2u 6x From second equation ( 2u 2x) + ( 2u 6x) (1) = 0; x = 0 From constraint u = 2 Optimal solution x* = 0 u* = 2 J* = Next ime: Principles for Optimal Control of Dynamic Systems Reading: OCE: Sections 3.1, 3.2,

33 Supplemental Material 65 L p norm = x p = n i=1 Infinity Norm Norm As p -> Norm is the value of the maximum component x i p 1/ p n ) ***( x p() i i=1 1/) = x imax = L ) norm x 1 x x = 2 x n dim(x) = n n Also called Supremum norm Chebyshev norm Uniform norm L norm = x = max{ x 1, x 2,, x n } 66

34 Gradient-Free vs. Gradient-Based Searches J is a scalar J provides no search direction Search may provide a global minimum J/u is a vector J/u indicates feasible search direction Search defines a local minimum 67 Numerical Example Cost function and derivatives J = 1 ( 2 u u *) R( u u *), R > 0 5 J 5 u J = 1. 0 ( /* 2 0* 1) ( = * * ) u 1 u 2 u 1 u ( 3 - * -, ) 1 + ( 3 - * -, ) ( - *, * ) u 1 u 2 + ( - ; R = *, ) , First guess at optimal control (details of the actual cost function are unknown) u 1 4 = u 2 7 Derivatives evaluated at starting point J ) = + u u=u0 * ( 1, ) * ,. = , ; R = ) 1 2, +. * u* = = u 1 u * = Solution from starting point J u* = u o R = 1 3 u u=uo ( ( ( 9 / 5 2 / 5 + * - ) 2 / 5 1/ 5,

35 Conjugate Gradient Algorithm First step H u 1 (t) = u k (t) K ( 0 u t) ( Calculate ratio of integrated gradient magnitudes squared b = t f ( t o t f ( t o H ( u t) H ( 1 u t) dt 1 H ( u t) H ( 0 u t) dt 0 0 Calculate gradient of improved trajectory H u t ( ) 1 Second step ) + H u k +1 (t) = u 2 (t) = u 1 (t) K ( 1 u t) * (,+ 1 b H ( u t) -+ (. 0 /+ 69 Rosenbrock Function ypical test function for numerical optimization algorithms J(u 1,u 2 ) = ( 1 u 1 ) ( u 2 u 1 ) 2 One minimum 70

36 How Many Maxima/Minima does the Mexican Hat Have? z = sinc( R) sin R R J u u=u* = J J... u 1 u 2 J u m = 0 u=u* R = u u J u 2 u=u* = 2 J u J u 1 u J 2 J... 2 u 2 u 1 u 2 u 2 u m J 2 J... u m u 1 u 2 u m 2 J u 1 u m 2 J 2 J 2 u m u=u* > < 0 Prior example One maximum 71 Physical Annealing Produce a strong, hard object made of crystalline material High temperature allows molecules to redistribute to relieve stress, remove dislocations Gradual cooling allows large, strong crystals to form Low temperature working (e.g., squeezing, bending, drawing, shearing, and hammering) produces desired crystal structure and shape urbojet Engine urbines urbine Blade Casting Single-Crystal urbine Blade 72

37 e ( J 2 J 1 )/k Introduce random wobble to simplex search Add random components to costs evaluated at vertices Project new vertex as before based on modified costs With large, this becomes a random search Decrease random components on a cooling schedule Same annealing strategy as before If cost decreases (J 2 < J 1 ), always accept new point If cost increases (J 2 > J 1 ), accept new point probabilistically As search progresses, decrease Combination of Simulated Annealing with Downhill Simplex Method SA Mona Lisa ( ) ( ) ( ) J 1SA = J 1 + J 1 rng J 2SA = J 2 + J 2 rng J 3SA = J 3 + J 3 rng... = Genetic Coding: Replication, Recombination, and Mutation of Chromosomes 74

38 Crossover Creates New Chromosome Population Containing Old Gene Sequences Reproduced Set Crossover Set 75 Starting Set Reproduction Eliminates Least Fit Chromosomes Probabilistically Reproduced Set 76

39 Genetic Algorithm: Mutation Flip a bit, 0 -> 1 or 1 -> 0, at random every 1,000 to 5,000 bits Crossover Set Mutated Set 77 Comments on GA GA Mona Lisa Short, fit genes tend to survive crossover Random location of crossover produces large and small variations in genes interchanges genes in chromosomes Multiple copies of best genes evolve Alternative implementations Real numbers rather than binary numbers Retention of elite chromosomes Clustering in fit regions to produce elites

40 Particle Swarm Optimization Particle Swarm oolboxes Comparison of Algorithms in Caterpillar Gait-raining Example Performance: distance traveled Y. Bourquin, U. Sussex, BIRG 80

41 Generalized Direct Search Algorithm J u k+1 (t) = u k (t) K k u t ( ) ( k Choose optimal elements of K by sequential line search before recalculating the gradient K k = k k Ad Hoc Modifications to the Newton-Raphson Search u k+1 = u k 2 J ) u 2 u=u k () 1 J ), < 1 u u=uk () u k+1 = u k 2 J u J 2 u m ( ( ( ( ( ( ( ( ( ( ( ( ( 1 J ( u u=uk u k+1 = u k 2 J + K( u 2 u=u k ( 1 J (, K > 0, diagonal u u=uk u k+1 = u k 2 J + I) u 2 u=u k () 1 J ) u u=uk ( With varying, this is the Levenberg-Marquardt algorithm 82