TMA 4180 Optimeringsteori Least Squares Optimization. Harald E. Krogstad, rev. 2010

Transcription

1 TMA 48 Optimeringsteori Lest Squres Optimiztion Hrld E. Krogstd, rev. This note ws originlly prepred for erlier versions of the course. Nocedl nd Wright hs nice introduction to Lest Squre (LS) optimiztion problems in Chpter, nd the note is now therefore only smll supplement. It re ects tht LS problems re by fr the most common cse for unconstrined optimiztion. See lso N&W, p for typicl exmples of LS problems. Lest Squre problems hve often their origin in tting models to observtions. In its simplest form, we know this from the problem of tting regression line, y = x + b, through set of dt points fx i ; y i g ; i = ; ; N. When N >, it is in generl impossible to put the line through ll points, but we try to determine n optiml line, for exmple by determining the pir f ; b g which minimizes the objective function f(; b) = NX (x i + b y i ) : () i= In this simple cse, it is esy to derive the solution nlyticlly, but in generl the solution hs to be found numericlly. Since these problems re so common, there hs been lot of work involved in dpting the generl lgorithms for unconstrined optimiztion to this specil cse. It is more e ective to use specilly dpted lgorithms insted of the more generl ones. Below we rst consider the liner LS problem, which some of you would know from liner nlysis or multiple liner regression in sttistics. The nottion my di er somewht from wht you know, but the ides re the sme. The singulr vlue decomposition is n importnt tool for nlyzing the liner problem, nd the liner problem is lso importnt for the solutions of the vrious itertion steps in non-liner LS problems. In the non-liner lgorithms we once more meet bsic tools like line serch nd trust region methods, which utilize the specil form of the grdient nd Hessin the LS problems hve. Liner Lest Squre Problems. The origin Although LS problems stem from vriety of di erent situtions, we shll consider sitution where n fctors in uence the result of n experiment. For ech experiment we know the vlues of the fctors, sy f ; :::; n g, nd we re ble to mesure the result. We re interested in nding function f ( ; :::; n ) which cn be used to compute the result without hving to repet the experiment ech time. In prctice, this my be quite di cult: A lot of fctors my be involved

2 We re not quite sure whether fctor counts or not There my be fctors unknown to us We hve no ide how the function f looks It is expensive nd/or time consuming to crry out experiments We hve only limited time nd budget to gure out solution All these points re prt of rel life, where you often nd yourself in group of collegues where you re the lest incompetent member. The rst guess if one does not know nything will often be to ssume tht there is liner dependency between the fctors nd the result. In other words, we re looking for set of coe cients or weight fctors fx ; ; x n g such tht x + x + ::: + n x n : () Note tht things re turned round here: When we pply the formul fter the x-s re determined, we put in f ; ; n g nd get n estimte of, denoted ^. The expression should in generl lso include constnt term, nd this is obtined by letting =, so tht the formul reds ^ = x + x + + n x n. In order to determine the x-s, we need to crry out set of clibrtion experiments. If we do not know nything bout the x-s, we would like to hve m > n experiments: If the number of experiments is less thn n (often the sitution in prctice), there will in generl be mny di erent sets of weight fctors tting Eqn., nd if we hve exctly n experiments, we could still stisfy Eqn. exctly, but would hve no ide how well the eqution is stis ed in other cses. Let us store the vlues of the fctors in the mn mtrix A nd the results in the m dimensionl vector b, such tht A = f i ; i ; ; in g; i = ; ; m; () b = f ; ; m g : (4) Since it would be too good to be true if we relly found vector x such tht Ax = b, we re stis ed if we re ble to nd x such tht the sum of the squred devitions is minimized: min x mx ( i x + i x + ::: + in x n i ) : (5) i= In mtrix nottion this is equivlent to nding the minimum of the function f(x) = kax bk = (Ax b) (Ax b): (6)

3 . Solution of the full rnk problem By crrying out the multiplictions in Eqn. 6, we nd tht f(x) = (Ax b) (Ax b) = x A Ax x A b + b b ; (7) which we, by now, should recognize s the qudrtic test problem. If A A hppens to be positive de nite (which requires tht m n), we lredy know tht the solution, x, stis es the eqution A Ax = A b: (8) (Convince yourself tht ll mtrix multiplictions mke sense!). The system of equtions in 8 is clled the Norml Equtions. It turns out tht A A is positive de nite if nd only if A hs full column rnk. In this cse, where m > n, this mens tht rnk(a) = n, or tht ll columns of A re linerly independent. Tht is equivlent to Ax =, x =, which in turn is equivlent to x A Ax = kaxk =, x =. Even if A hs full rnk mthemticlly, it is not necessrily resonble numericlly to solve the norml equtions. The condition number of A is the rtio between its lrgest nd smllest singulr vlue (see below), nd the condition number of A A will be the squre of tht. In order to void numericl problems, the best is to use QR-fctoriztion of A. The -norm is invrint under orthogonl trnsformtions, nd this reduces the problem to solving liner system of equtions with n unknowns which hs the upper n n sub-mtrix of R s its coe cient mtrix. A short summry of this is given in N&W p. 5, nd more extensive tretment is found in the book of Golub nd vnlon [], Chpter 5. Cses with few fctors nd lrge number of experiments re not likely to experience ny problems with the Norml Equtions.. The rnk de cient problem If the rnk of A, r = rnk (A), is less thn n, the problem will still hve solutions, but they re not unique. The null spce of A, N (A), is the set of ll vectors z such tht Az =, nd when the column rnk, rnk (A), is less thn n, N (A) contins non-zero vectors (recll the de nition of rnk!). Obviously, if x is solution to 6, then x + z, z N (A) will be solution s well since A Az = nd f (x + z) = f (x ). The existence of solutions is not obvious, but follows by considering Eqn. 8 s n eqution not in R n but in the smller orthogonl complement of the null-spce of A, N (A)?. Check out ll de nitions you do not know/recll nd try to crry out the following steps:. Show tht A b nd A Ax re members of N (A)? by tking the sclr product with vector z N (A).

4 . Show tht the mtrix A A is non-singulr when restricted to N (A)? (This mounts to show tht kayk = () y = when y N (A)? ).. Show tht there is unique x N (A)? such tht A Ax = A b (This my be little tough if you re not fmilir with liner lgebr nd liner trnsformtions. It is not tht importnt, the solution exists). The rnk de cient cse will occur if we try to explin our result by including too mny fctors. In prctice, it is smrt to strt with few fctors nd then include one dditionl fctor t time. By supervising kax bk, we get feeling of when enough is enough. Even if A is rnk de cient, the function f (x) in Eqn. 7 is still convex since A A, nd the set of globl minim, tht there re indeed optiml solutions, so = fy; f(y) = ming, is therefore closed convex set (We proved bove is de nitely non-empty). Are ll these possible solutions eqully good? Mthemticlly yes, but in prctice we re using x for predicting or estimting, nd the predictor ^ for given set of fctors,, is simply ^ = x : (9) Unnecessry lrge components in x will mgnify ny mesuring errors in the fctors. Therefore, it is resonble to use s smll x s possible. If we, on the convex set, de ne the strictly convex function g(y) = kyk, the function will hve unique minimum, nd this would be the specil solution we re looking for: x = rgmin kxk : () A t Ax=A t b Cn you prove tht x hs to be equl to the solution in N (A)?? (Recll tht ny solution my be written uniquely s x = x + z, where z N (A) nd x N (A)? ). There re severl numericl methods for nding x described in N&W, s well s in []. The most robust lthough not the chepest computtionlly is bsed on the Singulr Vlue Decomposition of A... The Singulr Vlue Decomposition The Singulr Vlue Decomposition (SVD) is door-opener to lot of prcticl lgorithms, nd very much used in modern pplictions of liner lgebr. The reduced form SVD of generl m n mtrix A is A = UV = rx i u i vi; () i= where U = 4 j j j u u ::: u r j j j 5 ; V = 4 j j j v v ::: v r j j j 5 () 4

5 hve orthogonl columns, U U = I (r) ; V V = I (r) ; () nd r = rnk (A). The r r digonl mtrix contins the so-clled singulr vlues of A. These singulr vlues re the squre root of the non-negtive eigenvlues of A A, or AA (Note tht the mtrices U nd V my be extended to full m m nd n n orthogonl mtrices nd to n m n mtrix, s shown in N&W, p We do not ssume or need this below). There re few things to check here: The mtrix A represents liner trnsformtion from R n! R m nd, similrly, A liner trnsformtion from R m! R n. v ; v ; :::; v r R n nd u ; u ; :::; u r R m. U nd V consist of the eigenvectors of AA nd A A tht correspond to non-zero eigenvlues, (AA ) U = U nd (A A) V = V. The columns of V spn n orthogonl bsis for N (A)? R n, nd the vectors in U n orthogonl bsis for the rnge of A, R (A) R m. The mtrix A + = V U (4) is clled pseudo-inverse or generlized inverse of A, nd is the most fmous member mong lrge number of generlized inverses. This prticulr generlized inverse is clled the Moore- Penrose inverse of A. Exercise: Prove tht A + A cts s the identity mtrix on N (A)?, nd AA + s the identity mtrix on R (A) (Hint: Use tht ll vectors x N (A)? my be written x = V s, s R r, nd similrly z = Ut; t R r for z R (A) ). You could consult textbook (E.g. [] or your own liner lgebr book) in order to brush this up. It is not so esy to grsp, or to remember!.. The solution in terms of the SVD Given the SVD of A, the unique solution with the smllest norm of Eqn. 8 is x = A + b = V U b = rx i= u i b i v i : (5) (Prove this yourself by showing tht A A (A + b) = A b!). In Eqn. 5, we ssume tht the singulr vlues re ordered ccording to their size, tht is, r : (6) 5

6 The dt set Figure : Simulted 6 dt points with some underlying function dded to lot of noise. This is the opposite ordering from wht we hve been using for eigenvlues. Contrry to wht you lern in the liner lgebr courses, determining the rnk of big mtrix is fr from trivil. If the singulr vlues decrese grdully to, it is quite di cult to sy where to stop (perhps when i < 6?) nd de ne the rest to be. The nice thing bout Eqn. 5 is tht we cn dd one term t time nd stop when the solution strts to get unstble. Smll singulr vlues mgnify errors in b, U or V, nd this cse is clled n ill-conditioned problem. It seems to be n inherent property of rel life problems of some size tht they ll tend to be ill-conditioned. This will be discussed lter for Inverse Problems... A worked exmple The following simple nd somewht unrelistic exmple shows how the SVD works. We hve lrge nd very noisy dt set of function of one vrible (y). The vlues of y rnge between nd, nd the ide is to extrct the function by tting polynomil. Typicl dt re shown in Fig., nd if this hs been rel dt set, we would not even think of tting something else thn constnt vlue, or t most stright line. The dt re produced by dding independent normlly distributed noise with stndrd devition equl to.6 to the function vlues. Let us nevertheless tke very bold ttitude nd try to t polynomil of order 6 (!) by just running the Mtlb functions poly t nd polyvl: p = polyfit(y,b,6); Y = polyvl(p,y); plot(y,dt,.,y,y); In this cse, the fctors re the vrious powers of vrible y, = ; y; y ; ; y 6 weights x re the coe cients in the polynomil, nd the ^ = x + x y + x y + + x 6 y 6 : (7) 6

7 Number of terms = Figure : Dt nd 6th order polynomil tted by poly t in Mtlb. As expected, the result is relly not tht gret, s Fig. shows. Also Mtlb protests nd wrns us tht the mtrices involved re close to singulr. The progrm suggests removing some dt points, or do some centering nd scling of the dt. Nothing of this would ctully help, the obvious cure is clerly to reduce the order of the polynomil. But let us not give up tht fst. We stick to 6th order nd form the fctor mtrix A of dimension 6 6 (6 fctors nd 6 experiments). A = 6 4. nd the dt vector b = (b ; b ; ; b 6 ). y y y 6 y y y 6.. y 6 y6 y6 7 5 ; (8) We re then fced with the problem of nding the best lods x = (x ; x ; ; x 6 ) to the dt, tht is, solve the LS problem, Let us consider two possibilities: x = rg min xr 6 kax bk : (9) Use of the SVD Use of the norml equtions Both re strightforwrd to progrm, the SVD even s simple s [U Sigm V] = svd(a)! The rnk (A) computed by Mtlb is equl to 6, nd this seems to be in rough greement with plot of the singulr vlues shown on Fig.. It is tempting to use the solution on the SVD-form, x = V U b = rx i= u i b i v i : () 7

8 5 5 σ k k Figure : The singulr vlues of the A-mtrix. The mchine ccurcy seems to be reched fter bout singulr vlues, but the singulr vlues never become exctly. where we include one term t time. This is displyed in Fig. 4, where the result is plotted every eight term. The grphs in the middle show some similrities, nd this is even true for the grphs t the bottom, lthough using more terms in Eqn. thn the rnk is not very meningful. The SVD solution with r = rnk (A) terms, the Norml eqution solution, nd the (up to now) unknown true function re shown in Fig. 5. The nl plot, Fig. 6 shows the SVD solution nd the t of polynomil of order r. The solutions re bout eqully good, but we hve not tried to optimize this further. By running the progrm with di erent dt, the optiml SVD solution seems, on the verge, to be slightly better thn the Norml Eqution solution. The reder should be wrned tht the, fter ll, civilized behviour of these results is in prt due to Mtlb s very crefully coded numerics! Non-Liner Lest Squre Problems In the generl non-liner lest squre cse we hve vector-vlued function, h(x) R m where x R n, nd we wnt to nd the minimum of f(x) = kh(x)k = mx h i (x) = h(x) h(x): () i= The simple Exmple. in N&W on p is of this form. All the grdients of the vrious components in h mke up wht is clled the Jcobin mtrix of h, de ned by J(x) (x) j i=;:::;m; j=;:::;n 6 4 rh (x) rh (x). rh m (x) 7 5 : () Recll tht we ssume tht rh i re n-dimensionl row vectors. By tking derivtives term by 8

9 Number of terms =8 Number of terms =6 Dt nd polynomil Dt nd polynomil Number of terms =4 Number of terms = Dt nd polynomil Dt nd polynomil Number of terms =4 Number of terms =48 Dt nd polynomil Dt nd polynomil Figure 4: SVD solutions truncted t the term indicted. All polynomils hve order 6 (The dt set here di ers from the bove). 9

10 Dt nd ll solutions Figure 5: Dt nd ll polynomils: Red: SVD solution with r terms; Blue: Norml Equtions; Green: Underlying exct function..5.5 Solutions Figure 6: Red: SVD solution of order r = rnk (A). Blue: A simple polynomil t of order r. Green: Input function.

11 term, we esily obtin (do it yourself!) tht the grdient nd the Hessin of f re rf(x) = J (x)h(x); () mx r f (x) = J (x)j(x) + h i (x)r h i (x); (4) where r h i re the Hessin mtrices for h i, i = ; ; m. Note tht if h i (x) re liner functions, sy h(x) = Ax b, then f(x) = kax bk, nd i= rf(x) = A (Ax b) = A h(x); r f(x) = A A: (5) We observe tht A corresponds to the Jcobin, J, wheres the Hessin in the generl non-liner cse gets n dditionl term B = mx h i (x)r h i (x): (6) i= If the minimiztion leds to smll vlues of h i (x), or the problem is nerly liner such tht r h i (x)-s re relly smll, the rst term in r f(x) will dominte nd This is utilized in Guss-Newtons method. method could be written or, in terms of the serch direction, p k, r f(x) J(x) J(x): (7) We remember tht the itertion step in Newton s r f(x k )(x k+ x k ) = rf(x k ) ; (8) r f(x k )p k = rf(x k ) : (9) In the Guss-Newtons method, the serch direction p k for line serch, x k+ = x k + k p k, is obtined by pproximting the Hessin by the rst prt of the sum in Eqn. 4, J(x k ) J(x k )p k = J(x k ) h(x k ): () We observe from the previous section tht this is the sme s solving the liner LS problem min p kj(x k )p + h(x k )k : () Prcticl experience hs shown tht if J(x) hs full rnk nd h(x) relly gets smll close to the solution, J(x k ) J(x k ) becomes good pproximtion to the exct Hessin nd Guss-Newton converges bout s fst s full Newton. This is illustrted for Guss-Newton on the Rosenbrock Bnn Function on Fig. 7, copied from Mtlb Optimiztion Toolbox documenttion. The guide sys tht only 48 function evlutions were needed, even when the grdient ws computed numericlly. If, on the contrry, J(x) is rnk de cient, or the residuls (the components of h) re not smll, the performnce my be very poor. This is discussed in N&W, p

12 Figure 7: This gure, copied from the documenttion of Mtlb Optimiztion Toolbox, shows tht Guss-Newton is very e cient when the lst prt of the Hessin mtrix my be ignored. Wheres Guss-Newton is line serch method, it is lso resonble use the sme pproximte Hessin in trust region setting. The qudrtic pproximtion to the objective is then m (p) = f k + J k h k p + p J k J k p; () nd with sphericl domin D with rdius, we hve the sme type of model problem s before, p = rgmin sd m (p) : () If J(x k ) hs full rnk, the solution of will be identicl to the Guss-Newton solution, p GN = Jk J k J k h k ; (4) s long s kp GN k. Otherwise, the solution hs to be on the boundry of D nd obtined from J k J k + k I p = J k h k : (5) with vlue on k tht ensures kpk =. Note tht in this cse, the solution of the constrined problem is strightforwrd (in principle): J k J k, nd J k J k + I > for ll >. Moreover, kp k!!. This trust region lgorithm is clled Levenberg-Mrqurdt s method, nd is the stndrd method for non-liner LS problems nd one of the lgorithm implemented in Optimiztion Toolbox in MATLAB. If it is possible to compute the mtrix B in resonble wy, it will in generl be preferble to use it, s discussed in []. Such methods re implemented in the NAG librry of numericl lgorithms.

13 References [] Gill, P.E. nd W. Murry: Algorithms for the solution of the nonliner lest-squre problems, SIAM J. Num. Anl. Vol. 5(978) pp [] Golub, G.H. nd C.F. VnLon: Mtrix Computtions, Johns Hopkins University Press, 989. [] Luenberger, D.G.: Liner nd Nonliner Progrmming, nd ed., Addison Westley, 984.