eoscences 567: CHAPER 3 (RR/Z) CHAPER 3: IVERSE EHODS BASED O EH 3. Inroucon s caper s concerne w nverse eos base on e leng of varous vecors a arse n a ypcal proble. e wo os coon vecors concerne are e aa-error or sf vecor an e oel paraeer vecor. eos base on e frs vecor gve rse o classc leas squares soluons. eos base on e secon vecor gve rse o wa are known as nu leng soluons. Iproveens over sple leas squares an nu leng soluons nclue e use of nforaon abou nose n e aa an a pror nforaon abou e oel paraeers, an are known as wege leas squares or wege nu leng soluons, respecvely. s caper wll en w aeral on ow o anle consrans an on varances of e esae oel paraeers. 3. Daa Error an oel Paraeer Vecors e aa error an oel paraeer vecors wll play an essenal role n e evelopen of nverse eos. ey are gven by an aa error vecor e obs pre (3.) oel paraeer vecor (3.) e enson of e error vecor e s, wle e enson of e oel paraeer vecor s, respecvely. In orer o ule ese vecors, we nex conser e noon of e se, or leng, of vecors. 3.3 easures of eng e nor of a vecor s a easure of s se, or leng. ere are any possble efnons for nors. e are os falar w e Caresan ( ) nor. Soe exaples of nors follow: e (3.3) 3
eoscences 567: CHAPER 3 (RR/Z) / e (3.4) e / (3.5) an fnally, ax e (3.6) Iporan oce! Inverse eos base on fferen nors can, an ofen o, gve fferen answers! e reason s a fferen nors gve fferen weg o oulers. For exaple, e nor gves all e weg o e larges sf. ow-orer nors gve ore equal weg o errors of fferen ses. e nor gves e falar Caresan leng of a vecor. Conser e oal sf E beween observe an prece aa. I as uns of leng square an can be foun eer as e square of e nor of e, e error vecor (Equaon 3.), or by nong a s also equvalen o e o (or nner) prouc of e w self, gven by e e E e e [ e e e ] e (3.7) e Inverse eos base on e nor are also closely e o e noon a errors n e aa ave aussan sascs. ey gve conserable weg o large errors, wc woul be consere unlkely f, n fac, e errors were srbue n a aussan fason. ow a we ave a way o quanfy e sf beween prece an observe aa, we are reay o efne a proceure for esang e value of e eleens n. e proceure s o ake e paral ervave of E w respec o eac eleen n an se e resulng equaons o ero. s wll prouce a syse of equaons a can be anpulae n suc a way a, n general, leas o a soluon for e eleens of. e nex secon wll sow ow s s one for e leas squares proble of fnng a bes f srag lne o a se of aa pons. 3
eoscences 567: CHAPER 3 (RR/Z) 3.4 nng e sf eas Squares 3.4. eas Squares Proble for a Srag ne Conser e fgure below (afer Fgure 3. fro enke, page 36): (a) (b)............. obs pre e. { (a) eas squares fng of a srag lne o (, ) pars. (b) e error e for eac observaon s e fference beween e observe an prece au: e obs pre. e prece au pre for e srag lne proble s gven by pre (3.8) were e wo unknowns, an, are e nercep an slope of e lne, respecvely, an s e value along e axs were e observaon s ae. For pons we ave a syse of suc equaons a can be wren n arx for as: Or, n e by now falar arx noaon, as (3.9) 33
eoscences 567: CHAPER 3 (RR/Z) (.3) ( ) ( ) ( ) e oal sf E s gven by E obs pre [ ] e e (3.) obs [ ( )] Droppng e obs n e noaon for e observe aa, we ave (3.) [ ] E (3.) en, akng e parals of E w respec o an, respecvely, an seng e o ero yels e followng equaons: an E (3.3) E (3.4) Rewrng Equaons (3.3) an (3.4) above yels (3.5) an (3.6) Cobnng e wo equaons n arx noaon n e for A b yels (3.7) or sply 34
eoscences 567: CHAPER 3 (RR/Z) A b (3.8) ( ) ( ) ( ) oe a by e above proceure we ave reuce e proble fro one w equaons n wo unknowns ( an ) n o one w wo equaons n e sae wo unknowns n A b. e arx equaon A b can also be rewren n ers of e orgnal an wen one noces a e arx A can be facore as ( ) ( ) ( ) ( ) (3.9) Also, b above can be rewren slarly as (3.) us, subsung Equaons (3.9) an (3.) no Equaon (3.7), one arrves a e so-calle noral equaons for e leas squares proble: e leas squares soluon S s en foun as assung a [ ] exss. (3.) S [ ] (3.) In suary, we use e forwar proble (Equaon 3.9) o gve us an explc relaonsp beween e oel paraeers ( an ) an a easure of e sf o e observe aa, E. en, we ne E by akng e paral ervaves of e sf funcon w respec o e unknown oel paraeers, seng e parals o ero, an solvng for e oel paraeers. 35
eoscences 567: CHAPER 3 (RR/Z) 3.4. Dervaon of e eneral eas Squares Soluon e sar w any syse of lnear equaons wc can be expresse n e for Agan, le E e e [ pre ] [ pre ] (.3) ( ) ( ) ( ) E [ ] [ ] (3.3) E j j j k k k (3.4) As before, e proceure s o wre ou e above equaon w all s cross ers, ake parals of E w respec o eac of e eleens n, an se e corresponng equaons o ero. For exaple, followng enke, page 4, Equaons (3.6) (3.9), we oban an expresson for e paral of E w respec o q : E q k qk k q (3.5) e can splfy s expresson by recallng Equaon (.4) fro e nroucory rearks on arx anpulaons n Caper : C j oe a e frs suaon on n Equaon (3.5) looks slar n for o Equaon (.4), bu e subscrps on e frs er are backwars. If we furer noe a nercangng e subscrps s equvalen o akng e ranspose of, we see a e suaon on gves e qk enry n : us, Equaon (3.5) reuces o k qk [ ] q k [ ] a k b kj qk (.4) (3.6) E q [ k ] qk k q (3.7) ow, we can furer splfy e frs suaon by recallng Equaon (.6) fro e sae secon j j j (.6) 36
eoscences 567: CHAPER 3 (RR/Z) o see s clearly, we rearrange e orer of ers n e frs su as follows: k k [ ] qk [ ] qk k [ k ] q (3.8) wc s e q enry n. oe a as enson ( )( )( ) ( ). a s, s an -ensonal vecor. In a slar fason, e secon suaon on can be reuce o a er n [ ] q, e q enry n an ( )( ) ( ) ensonal vecor. us, for e q equaon, we ave E [ ] q [ q ] q (3.9) Droppng e coon facor of an cobnng e q equaons no arx noaon, we arrve a e leas squares soluon for s us gven by e leas squares operaor, S, s us gven by (3.3) S [ ] (3.3) S [ ] (3.3) Recallng basc calculus, we noe a S above s e soluon a nes E, e oal sf. Suarng, seng e q paral ervaves of E w respec o e eleens n o ero leas o e leas squares soluon. e ave jus erve e leas squares soluon by akng e paral ervaves of E w respec o q an en cobnng e ers for q,,...,. An alernave, bu equvalen, forulaon begns w Equaon (3.) bu s wren ou as E [ ] [ ] (3.3) [ ][ ] (3.33) en, akng e paral ervave of E w respec o urns ou o be equvalen o wa was one n Equaons (3.5) (3.3) for q, naely 37
eoscences 567: CHAPER 3 (RR/Z) E/ (3.34) wc leas o (3.3) an S [ ] (3.3) I s also peraps neresng o noe a we coul ave obane e sae soluon wou akng parals. o see s, conser e followng four seps. Sep. e begn w (.3) Sep. e en preulply bo ses by (3.3) Sep 3. Preulply bo ses by [ ] [ ] [ ] (3.35) Sep 4. s reuces o S [ ] (3.3) as before. e pon s, owever, a s way oes no sow wy S s e soluon wc nes E, e sf beween e observe an prece aa. All of s assues a [ ] exss, of course. e wll reurn o e exsence an properes of [ ] laer. ex, we wll look a wo exaples of leas squares probles o sow a srkng slary a s no obvous a frs glance. 3.4.3 wo Exaples of eas Squares Probles Exaple. Bes-F Srag-ne Proble e ave, of course, alreay erve e soluon for s proble n e las secon. Brefly, en, for e syse of equaons 38
eoscences 567: CHAPER 3 (RR/Z) 39 (.3) gven by (3.9) we ave (3.36) an (3.37) us, e leas squares soluon s gven by S (3.38) Exaple. Bes-F Parabola Proble e prece au for a parabola s gven by 3 (3.39) were an ave e sae eanngs as n e srag lne proble, an 3 s e coeffcen of e quarac er. Agan, e proble can be wren n e for: (.3) were now we ave
eoscences 567: CHAPER 3 (RR/Z) 4 3 (3.4) an 4 3 3, (3.4) As before, we for e leas squares soluon as S [ ] (3.3) Aloug e forwar probles of precng aa for e srag lne an parabolc cases look very fferen, e leas squares soluon s fore n a way a epases e funaenal slary beween e wo probles. For exaple, noce ow e srag-lne proble s bure wn e parabola proble. e upper lef an par of n Equaon (3.4) s e sae as Equaon (3.36). Also, e frs wo enres n n Equaon (3.4) are e sae as Equaon (3.37). ex we conser a four-paraeer exaple. 3.4.4 Four-Paraeer oograpy Proble Fnally, le's conser a four-paraeer proble, bu s one base on e concep of oograpy. S R 3 4 3 4 ) ( s s v v ) ( 4 3 4 3 s s v v ) ( 3 3 3 s s v v ) ( 4 4 4 s s v v (3.4)
eoscences 567: CHAPER 3 (RR/Z) 4 4 3 4 3 s s s s (3.43) or (.3) (3.44) 4 3 4 3 (3.45) So, e noral equaons are (3.) 4 3 4 3 4 3 s s s s (3.46) or 4 3 4 3 4 3 s s s s (3.47) Exaple: s s s 3 s 4, ; en 3 4 By nspecon, s s s 3 s 4 s a soluon, bu so s s s 4, s s 3, or s s 4, s s 3.
eoscences 567: CHAPER 3 (RR/Z) Soluons are nonunque! ook back a. Are all of e coluns or rows nepenen? o! a oes a ply abou (an )? Rank < 4. a oes a ply abou ( )? I oes no exs. So oes S exs? o. Oer ways of sayng s: e vecors g o no span e space of. Or, e experenal se-up s no suffcen o unquely eerne e soluon. oe a s analyss can be one wou any aa, base srcly on e experenal esgn. Anoer way o look a : Are e coluns of nepenen? o. For exaple, coeffcens,,, an wll ake e equaons a o ero. a paern oes a sugges s no resolvable? ow a we ave erve e leas squares soluon, an consere soe exaples, we nex urn our aenon o soeng calle e eernancy of e syse of equaons gven by Equaon (.3): (.3) s wll begn o per us o classfy syses of equaons base on e naure of. 3.5. Inroucon 3.5 Deernancy of eas Squares Probles (See Pages 46 5, enke) e ave seen a e leas squares soluon o s gven by S [ ] (3.3) ere s no guaranee, as we saw n Secon 3.4.4, a e soluon even exss. I fals o exs wen e arx as no aeacal nverse. e noe a s square ( ), an s a leas aeacally possble o conser nverng. (.B. e enson of s, nepenen of e nuber of observaons ae). aeacally, we can say e as an nverse, an s unque, wen as rank. e rank of a arx was consere n Secon..3. Essenally, f as rank, en as enoug nforaon n o resolve ngs (n s case, oel paraeers). s appens wen all rows (or equvalenly, snce s square, all coluns) are nepenen. Recall also a nepenen eans you canno wre any row (or colun) as a lnear cobnaon of e oer rows (coluns). wll ave rank < f e nuber of observaons s less an. enke gves e exaple (pp. 45 46) of e srag-lne f o a sngle aa pon as an llusraon. If [ ] oes no exs, an nfne nuber of esaes wll all f e aa equally well. aeacally, as rank < f, were s e eernan of. 4
eoscences 567: CHAPER 3 (RR/Z) ow, le us nrouce enke s noenclaure base on e naure of an on e precon error. In all cases, e nuber of oel paraeers s an e nuber of observaons s. 3.5. Even-Deerne Probles: If a soluon exss, s unque. e precon error [ obs pre ] s encally ero. For exaple, 5 (3.48) for wc e soluon s [, 3]. 3.5.3 Overeerne Probles: ypcally, > ore observaons an unknowns, ypcally one canno f all e aa exacly. e leas squares proble falls n s caegory. Conser e followng exaple: 5 3 (3.49) s overeerne case consss of ang one equaon o Equaon (3.48) n e prevous exaple. e leas squares soluon s [.333, 4.833]. e aa can no longer be f exacly. 3.5.4 Unereerne Probles: ypcally, > ore unknowns an observaons, as no unque soluon. A specal case of e unereerne proble occurs wen you can f e aa exacly, wc s calle e purely unereerne case. e precon error for e purely unereerne case s exacly ero (.e., e aa can be f exacly). An exaple of suc a proble s [] [ ] (3.5) Possble soluons nclue [, ], [.5, ], [5, 9], [/3, /3] an [.4,.]. e soluon w e nu leng, n e nor sense, s [.4,.]. 43
eoscences 567: CHAPER 3 (RR/Z) e followng exaple, owever, s also unereerne, bu no coce of,, 3 wll prouce ero precon error. us, s no purely unereerne. 4 3 (3.5) (You g wan o verfy e above exaples. Can you nk of oers?) Aloug I ave sae a overeerne (unereerne) probles ypcally ave > ( < ), s poran o reale a s s no always e case. Conser e followng: 3 4 3 (3.5) For s proble, s overeerne, (a s, no coce of can exacly f bo an unless appens o equal ), wle a e sae e an 3 are unereerne. s s e case even oug ere are wo equaons (.e., e las wo) n only wo unknowns (, 3 ). e wo equaons, owever, are no nepenen, snce wo es e nex o las row n equals e las row. us s proble s bo overeerne an unereerne a e sae e. For s reason, I a no very sasfe w enke s noenclaure. As we wll see laer, wen we eal w vecor spaces, e key wll be e sngle values (uc lke egenvalues) an assocae egenvecors for e arx. 3.6 nu eng Soluon e nu leng soluon arses fro e purely unereerne case ( <, an can f e aa exacly). In s secon, we wll evelop e nu leng operaor, usng agrange ulplers an borrowng on e basc eas of nng e leng of a vecor nrouce n Secon 3.4 on leas squares. 3.6. Backgroun Inforaon e begn w wo peces of nforaon:. Frs, [ ] oes no exs. erefore, we canno calculae e leas squares soluon S [ ]. 44
eoscences 567: CHAPER 3 (RR/Z). Secon, e precon error e obs pre s exacly equal o ero. o solve unereerne probles, we us a nforaon a s no alreay n. s s calle a pror nforaon. Exaples g nclue e consran a ensy be greaer an ero for rocks, or a v n, e sesc P-wave velocy a e oo falls wn e range 5 < v n < k/s, ec. Anoer a pror assupon s calle soluon splcy. One seeks soluons a are as sple as possble. By analogy o seekng a soluon w e sples sf o e aa (.e., e salles) n e leas squares proble, one can seek a soluon wc nes e oal leng of e oel paraeer vecor,. A frs glance, ere ay no see o be any reason o o s. I oes ake sense for soe cases, owever. Suppose, for exaple, a e unknown oel paraeers are e veloces of pons n a flu. A soluon a ne e leng of woul also ne e knec energy of e syse. us, woul be approprae n s case o ne. I also urns ou o be a nce propery wen one s ong nonlnear probles, an e a one s usng s acually a vecor of canges o e soluon a e prevous sep. en s nce o ave sall sep ses. e requreen of soluon splcy wll lea us, as sown laer, o e so-calle nu leng soluon. 3.6. agrange ulplers (See Page 5 an Appenx A., enke) agrange ulplers coe o n wenever one wses o solve a proble subjec o soe consrans. In e purely unereerne case, ese consrans are a e aa sf be ero. Before conserng e full purely unereerne case, conser e followng scusson of agrange ulplers, osly afer enke. agrange ulplers Unknowns an Consran Conser E(x, y), a funcon of wo varables. Suppose a we wan o ne E(x, y) subjec o soe consran of e for φ(x, y). e seps, usng agrange ulplers, are as follows (nex page): 45
eoscences 567: CHAPER 3 (RR/Z) Sep. A e nu n E, sall canges n x an y lea o no cange n E: E(x, (y consan)) E nu x E E E x y x y (3.53) Sep. e consran equaon, owever, says a x an y canno be vare nepenenly (snce e consran equaon s nepenen, or fferen, fro E). Snce φ(x, y) for all x, y, en so us φ(x, y). Bu, φ φ φ x y (3.54) x y Sep 3. For e wege su of (3.53) an (3.54) as E φ E φ E λ φ λ x λ y (3.55) x x y y were λ s a consan. oe a (3.55) ols for arbrary λ. Sep 4. If λ s cosen, owever, n suc a way a E x φ λ x (3.56) en follows a E y φ λ y (3.57) 46
eoscences 567: CHAPER 3 (RR/Z) snce a leas one of x, y (n s case, y) s arbrary (.e., y ay be cosen nonero). en λ as been cosen as ncae above, s calle e agrange ulpler. erefore, (3.55) above s equvalen o nng E λφ wou any consrans,.e., x E x φ x ( E λφ) λ (3.58) an y E y φ y ( E λφ) λ (3.59) Sep 5. Fnally, one us sll solve e consran equaon φ(x, y) (3.6) us, e soluon for (x, y) a nes E subjec o e consran a φ (x, y) s gven by (3.58), (3.59), an (3.6). a s, e proble as reuce o e followng ree equaons: E x φ λ x (3.56) an E φ λ y y (3.57) φ (x, y) (3.6) n e ree unknowns (x, y, λ). Exenng e Proble o Unknowns an Consrans e above proceure, use for a proble w wo varables an one consran, can be generale o unknowns n a vecor subjec o consrans φ (), j,...,. s leas o e followng syse of equaons,,..., : w consrans of e for E φ j λ j (3.6) j φ j () (3.6) 47
eoscences 567: CHAPER 3 (RR/Z) 3.6.3 Applcaon o e Purely Unereerne Proble e backgroun we now ave n agrange ulplers, we are reay o reconser e purely unereerne proble. Frs, we pose e followng proble: fn suc a s ne subjec o e consrans a e aa sf be ero. a s, ne obs pre obs e j j,,..., (3.63) j ψ ( ) λ (3.64) e w respec o e eleens n. e can expan e ers n Equaon (3.64) an oban ψ ( ) k λ k j j j (3.65) en, we ave bu ψ q k j k λ (3.66) j k q j q k q δ kq j an δ jq (3.67) q were δ j s e Kronecker ela, gven by δ j,, j j us ψ q λ q q q,,..., (3.68) In arx noaon over all q, Equaon (3.68) can be wren as λ (3.69) 48
eoscences 567: CHAPER 3 (RR/Z) were λ s an vecor conanng e agrange ulplers λ,,...,. oe a λ as enson ( ) x ( ), as requre o be able o subrac fro. ow, solvng explcly for yels e consrans n s case are a e aa be f exacly. a s, Subsung (3.7) no (.3) gves wc ples λ (3.7) (.3) ( λ) (3.7) λ (3.7) were as enson ( ) ( ), or sply. Solvng for λ, wen [ ] exss, yels λ [ ] (3.73) e agrange ulplers are no ens n an of eselves. Bu, upon subsuon of Equaon (3.73) no (3.7), we oban Rearrangng, we arrve a e nu leng soluon, : λ {[ ] } (3.9) [ ] (3.74) were s an arx an e nu leng operaor,, s gven by [ ] (3.75) e above proceure, en, s one a eernes e soluon wc as e nu leng ( nor [ ] / ) aongs e nfne nuber of soluons a f e aa exacly. In pracce, one oes no acually calculae e values of e agrange ulplers, bu goes recly o (3.74) above. e above ervaon sows a e leng of s ne by e nu leng operaor. I ay ake ore sense o seek a soluon a evaes as lle as possble fro soe pror esae of e soluon, <>, raer an fro ero. e ero vecor s, n fac, e pror 49
eoscences 567: CHAPER 3 (RR/Z) esae <> for e nu leng soluon gven n Equaon (3.74). If we ws o explcly nclue <>, en Equaon (3.74) becoes <> [ ] [ <>] <> [ <>] [I ]<> (3.76) e noe eaely a Equaon (3.76) reuces o Equaon (3.74) wen <>. 3.6.4 Coparson of eas Squares an nu eng Soluons In closng s secon, s nsrucve o noe e slary n for beween e nu leng an leas squares soluons: eas Squares: S [ ] (3.3) w S [ ] (3.3) nu eng: <> [ ] [ <>] (3.76) w [ ] (3.75) e nu leng soluon exss wen [ ] exss. Snce s, s s e sae as sayng wen as rank. a s, wen e rows (or coluns) are nepenen. In s case, your ably o prec or calculae eac of e observaons s nepenen. 3.6.5 Exaple of nu eng Proble Recall e four-paraeer, four-observaon oograpy proble we nrouce n Secon 3.4.4. A a e, we noe a e leas squares soluon no exs because [ ] oes no exs, snce oes no conan enoug nforaon o solve for 4 oel paraeers. In e sae way, oes no conan enoug nforaon o f an arbrary 4 observaons, an [ ] oes no exs eer for s exaple. e basc proble s a e four pas roug e srucure o no prove nepenen nforaon. However, f we elnae any one observaon (le s say e four), en we reuce e proble o one were e nu leng soluon exss. In s new case, we ave ree observaons an four unknown oel paraeers, an ence <., wc sll as enoug nforaon o eerne ree observaons unquely, s now gven by (3.77) 5
eoscences 567: CHAPER 3 (RR/Z) An s gven by (3.78) ow [ ] oes exs, an we ave.5.5.5.75.5.5 [ ] (3.79).5.5.5.5.75.5 If we assue a rue oel gven by [.,.5,.5,.5], en e aa are gven by [.5,.,.5]. e nu leng soluon s gven by [ ] [.875,.65,.65,.375] (3.8) oe a e nu leng soluon s no e "rue" soluon. s s generally e case, snce e "rue" soluon s only one of an nfne nuber of soluons a f e aa exacly, an e nu leng soluon s e one of sores leng. e leng square of e "rue" soluon s.75, wle e leng square of e nu leng soluon s.6875. oe also a e nu leng soluon vares fro e "rue" soluon by [.5,.5,.5,.5]. s s e sae recon n oel space (.e., [,,, ] ) a represens e lnear cobnaon of e orgnal coluns of n e exaple n Secon 3.4.4 a a o ero. e wll reurn o s subjec wen we ave nrouce sngular value ecoposon an e paronng of oel an aa space. 3.7 ege easures of eng 3.7. Inroucon One way o prove our esaes usng eer e leas squares soluon S [ ] (3.3) or e nu leng soluon <> [ ] [ <>] (3.76) s o use wege easures of e sf vecor 5
eoscences 567: CHAPER 3 (RR/Z) e obs pre (3.8) or e oel paraeer vecor, respecvely. e nex wo subsecons wll eal w ese wo approaces. 3.7. ege eas Squares ege easures of e sf Vecor e e saw n Secon 3.4 a e leas squares soluon S was e one a ne e oal sf beween prece an observe aa n e nor sense. a s, E n s ne. Conser a new E, efne as follows: e [ ] e E e e e e e e (3.7) e E e e e (3.8) an were e s an, as ye, unspecfe wegng arx. e can ake any for, bu one convenen coce s e [cov ] (3.83) were [cov ] s e nverse of e covarance arx for e aa. s coce for e wegng arx, aa w large varances are wege less an ones w sall varances. le s s rue n general, s easer o sow n e case were e s agonal. s appens wen [cov ] s agonal, wc ples a e errors n e aa are uncorrelae. e agonal enres n [cov ] are en gven by e recprocal of e agonal enres n [cov ]. a s, f en σ σ [cov] (3.84) O σ 5
eoscences 567: CHAPER 3 (RR/Z) σ σ [cov] (3.85) O σ s coce for e, e wege sf becoes E e e e e j je j (3.86) Bu, j δ j (3.87) σ were δ j s e Kronecker ela. us, we ave E σ e (3.88) If e varance σ s large, en e coponen of e error vecor n e recon, e, as lle nfluence on e se of E. s s no e case n e unwege leas squares proble, were an exanaon of Equaon (3.4) clearly sows a eac coponen of e error vecor conrbues equally o e oal sf. Obanng e ege eas Squares Soluon S If one uses E e e e as e wege easure of error, we wll see below a s leas o e wege leas squares soluon: w a wege leas squares operaor S gven by S [ e ] e (3.89) S [ e ] e (3.9) le s s rue n general, s easer o arrve a Equaon (3.89) n e case were e s a agonal arx an e forwar proble s gven by e leas squares proble for a bes-fng srag lne [see Equaon (3.9)]. 53
eoscences 567: CHAPER 3 (RR/Z) 54 Sep. j j j e e e e E e e (3.9) ( ) pre obs j j j (3.9) ( ) (3.93) Sep. en E (3.94) an E (3.95) s can be wren n arx for as (3.96) Sep 3. e lef-an se can be facore as O (3.97) or sply e (3.98) Slarly, e rg-an se can be facore as
eoscences 567: CHAPER 3 (RR/Z) O (3.99) or sply e (3.) Sep 4. erefore, usng Equaons (3.98) an (3.), Equaon (3.96) can be wren as e e (3.) e wege leas squares soluon, S fro Equaon (3.89) s us S [ e ] e (3.) assung a [ e ] exss, of course. 3.7.3 ege nu eng e evelopen of a wege nu leng soluon s slar o a of e wege leas squares proble. e seps are as follows. Frs, recall a e nu leng soluon nes. By analogy w wege leas squares, we can coose o ne nsea of. For exaple, f one wses o use en one us replace above w (3.3) [cov ] (3.4) <> (3.5) were <> s e expece, or a pror, esae for e paraeer values. e reason for s s a e varances us represen flucuaons abou ero. In e wege leas squares proble, s assue a e error vecor e wc s beng ne as a ean of ero. us, for e wege nu leng proble, we replace by s eparure fro e expece value <>. erefore, we nrouce a new funcon o be ne: 55
eoscences 567: CHAPER 3 (RR/Z) [ <>] [ <>] (3.6) If one en follows e proceure n Secon 3.6 w s new funcon, one evenually (as n I s lef o e suen as an exercse!! ) s le o e wege nu leng soluon gven by <> [ ] [ <>] (3.7) an e wege nu leng operaor,, s gven by [ ] (3.8) s expresson ffers fro Equaon (3.38), page 54 of enke, wc uses raer an. I beleve enke s equaon s wrong. oe a e soluon epens explcly on e expece, or a pror, esae of e oel paraeers <>. e secon er represens a eparure fro e a pror esae <>, base on e naequacy of e forwar proble <> o f e aa exacly. Oer coces for nclue:. D D, were D s a ervave arx (a easure of e flaness of ) of enson ( ) : D (3.9) O O. D D, were D s an ( ) rougness (secon ervave) arx gven by D (3.) O O O oe a for bo coces of D presene, D D s an arx of rank less an (for e frs-ervave case, s of rank, wle for e secon s of rank ). s eans a oes no ave a aeacal nverse. s can nrouce soe nonunqueness no e soluon, bu oes no preclue fnng a soluon. Fnally, noe a any coces for are possble. 56
eoscences 567: CHAPER 3 (RR/Z) 3.7.4 ege Dape eas Squares In Secons 3.7. an 3.7.3 we consere wege versons of e leas squares an nu leng soluons. Bo unwege an wege probles can be very unsable f e arces a ave o be nvere are nearly sngular. In e wege probles, ese are an e (3.) (3.) respecvely, for leas squares an nu leng probles. In s case, one can for a wege penaly, or cos funcon, gven by E ε (3.3) were E s fro Equaon (3.9) for wege leas squares an s fro Equaon (3.6) for e wege nu leng proble. One en goes roug e exercse of nng Equaon (3.3) w respec o e oel paraeers, an obans wa s known as e wege, ape leas squares soluon D. I s, n fac, a wege x of e wege leas squares an wege nu leng soluons. One fns a D s gven by eer or D <> [ e ε ] e [ <>] (3.4) D <> [ ε e ] [ <>] (3.5) were e wege, ape leas squares operaor, D, s gven by or D [ e ε ] e (3.6) D [ ε e ] (3.7) e wo fors for D can be sown o be equvalen. e ε er as e effec of apng e nsably. As we wll see laer n Caper 6 usng sngular-value ecoposon, e above proceure nes e effecs of sall sngular values n e or. In e nex secon we wll learn wo eos of nclung a pror nforaon an consrans n nverse probles. 57
eoscences 567: CHAPER 3 (RR/Z) 3.8 A Pror Inforaon an Consrans (See enke, Pages 55 57) 3.8. Inroucon Anoer coon ype of a pror nforaon akes e for of lnear equaly consrans: F (3.8) were F s a P arx, an P s e nuber of lnear consrans consere. As an exaple, conser e case for wc e ean of e oel paraeers s known. In s case w only one consran, we ave (3.9) en, Equaon (3.8) can be wren as F [ ] (3.) As anoer exaple, suppose a e j oel paraeer j s acually known n avance. a s, suppose en Equaon (3.8) akes e for j (3.) F [ ] j j colun (3.) oe a for s exaple woul be possble o reove j as an unknown, ereby reucng e syse of equaons by one. I s ofen preferable o use Equaon (3.), even n s case, raer an rewrng e forwar proble n a copuer coe. 58
eoscences 567: CHAPER 3 (RR/Z) 3.8. A Frs Approac o Inclung Consrans e wll conser wo basc approaces o nclung consrans n nverse probles. Eac as s srengs an weaknesses. e frs nclues e consran arx F n e forwar proble, an e secon uses agrange ulplers. e seps for e frs approac are as follows. Sep. Inclue F as rows n a new a operaes on e orgnal : F ( P) ( P) (3.3) Sep. e new ( P) sf vecor e becoes obs pre e pre (3.4) ( P) ( P) Perforng a leas squares nverson woul ne e new e e, base on Equaon (3.4). e fference pre (3.5) wc represens e sf o e consrans, ay be sall, bu s unlkely a woul vans, wc us f e consrans are o be sasfe. Sep 3. Inrouce a wege sf: were e s a agonal arx of e for e e e (3.6) 59
eoscences 567: CHAPER 3 (RR/Z) 6 _ (bg #) (bg #) (bg #) P e O O O O (3.7) a s, as relavely large values for e las P enres assocae w e consran equaons. Recallng e for of e wegng arx use n Equaon (3.83), one sees a Equaon (3.7) s equvalen o assgnng e consrans very sall varances. Hence, a wege leas squares approac n s case wll gve large weg o fng e consrans. e se of e bg nubers n e us be eerne eprcally. One seeks a nuber a leas o a soluon a sasfes e consrans accepably, bu oes no ake e arx n Equaon (3.) a us be nvere o oban e soluon oo poorly conone. arces w a large range of values n e en o be poorly conone. Conser e exaple of e soong consran ere, P : D (3.8) were e ensons of D ( ),, an ( ). e augene equaons are D (3.9) e's use e followng wegng arx: P P e I I θ θ θ O (3.3) were θ s a consan. s resuls n e followng, w e ensons of e ree arces n e frs se of brackes beng ( P), ( P) ( P), an ( P), respecvely:
eoscences 567: CHAPER 3 (RR/Z) 6 S I I D D I I D θ θ < > < > [ ] D θ [ ] D θ e lower arces avng ensons of ( P ) ( P). [ θ D D] [ ] (3.3) ] [ D D θ θ (3.3) e ree arces wn (3.3) ave ensons ( P), ( P), an, respecvely, wc prouce an arx wen evaluae. In s for we can see s s sply e S for e proble D θ (3.33) By varyng θ, we can rae off e sf an e sooness for e oel. 3.8.3 A Secon Approac o Inclung Consrans enever e subjec of consrans s rase, agrange ulplers coe o n! e seps for s approac are as follows. Sep. For a wege su of e sf an e consrans: φ() e e [F ] λ (3.34) wc can be expane as ) ( j j j P j j j F λ φ (3.35) were ncaes a fference fro Equaon (3.43) on page 56 n enke, an were ere are P lnear equaly consrans an were e facor of as been ae as a aer of convenence o ake e for of e fnal answer spler.
eoscences 567: CHAPER 3 (RR/Z) Sep. One en akes e parals of Equaon (3.35) w respec o all e enres n an ses e o ero. a s, wc leas o φ( ) q,,..., q (3.36) jq j j q P λ F q,,..., (3.37) q were e frs wo ers are e sae as e leas squares case n Equaon (3.5) snce ey coe recly fro e e an e las er sows wy e facor of was ae n Equaon (3.35). Sep 3. Equaon (3.37) s no e coplee escrpon of e proble. o e equaons n Equaon (3.37), P consran equaons us also be ae. In arx for, s yels F F λ (3.38) Sep 4. ( P) ( P) ( P) ( P) e above syse of equaons can be solve as λ F F (3.39) As an exaple, conser consranng a srag lne o pass roug soe pon (', '). a s, for observaons, we ave subjec o e sngle consran en Equaon (3.8) as e for, (3.4) (3.4) F [ ] (3.4) e can en wre ou Equaon (3.39) explcly, an oban e followng: 6
eoscences 567: CHAPER 3 (RR/Z) 63 λ (3.43) oe e slary beween Equaons (3.43) an (3.36), e leas squares soluon o fng a srag lne o a se of pons wou any consrans: S (3.36) If you wane o ge e sae resul for e srag lne passng roug a pon usng e frs approac w e, you woul assgn,..., (3.44) an, bg # (3.45) wc s equvalen o assgnng a sall varance (relave o e unconsrane par of e proble) o e consran equaon. e soluon obane w Equaon (3.3) soul approac e soluon obane usng Equaon (3.43). oe a s easy o consran lnes o pass roug e orgn usng Equaon (3.43). In s case, we ave (3.46) an Equaon (3.43) becoes λ (3.47) e avanage of usng e agrange ulpler approac o consrans s a e consrans wll be sasfe exacly. I ofen appens, owever, a e consrans are only approxaely known, an usng agrange ulplers o f e consrans exacly ay no be approprae. An exaple g be a gravy nverson were ep o berock a one pon s known fro rllng. Consranng e ep o be exacly e rll ep ay be sleang f e ep n e oel s an average over soe area. en e exac ep a one pon ay no be e bes esae of e ep over e area n queson. A secon savanage of e agrange ulpler approac s a as one equaon o e syse of equaons n Equaon (3.43) for eac consran. s can a up quckly, akng e nverson conserably ore ffcul copuaonally.
eoscences 567: CHAPER 3 (RR/Z) An enrely fferen class of consrans are calle lnear nequaly consrans an ake e for F (3.48) ese can be solve usng lnear prograng ecnques, bu we wll no conser e furer n s class. 3.8.4 Sesc Recever Funcon Exaple e followng s an exaple of usng soong consrans n an nverse proble. Conser a general proble n e seres analyss, w a ela funcon npu. en e oupu fro e "oel" s e reens funcon of e syse. e nverse proble s s: ven e reens funcon, fn e paraeers of e oel. npu pulse oel reens Funcon In a lle ore concree for: 3. oel space c c c c 3... F() 3. aa space a c F. If s very nosy, en S wll ave a g-frequency coponen o ry o "f e nose," bu s wll no be real. How o we preven s? So far, we ave learne wo ways: use S f we know cov, or f no, we can place a soong consran on. An exaple of s approac usng recever funcon nversons can be foun n Aon, C. J.,. E. Ranall an. Zan, On e nonunqueness of recever funcon nversons, J. eopys. Res., 95, 5,33-5,38, 99. e poran pons are as follows: 64
eoscences 567: CHAPER 3 (RR/Z) s approac s use n e real worl. e forwar proble s wren j F j j,, 3... s s nonlnear, bu afer lnearaon (scusse n Caper 4), e equaons are e sae as scusse prevously (w nor fferences). oe e correlaon beween e rougness n e oel an e rougness n e aa. e way o coose e wegng paraeer, σ, s o plo e rae-off beween sooness an wavefor f. 3.9 Varances of oel Paraeers (See Pages 58 6, enke) 3.9. Inroucon Daa errors are appe no oel paraeer errors roug any ype of nverse. e noe n Caper [Equaons (.6) (.63)] a f es v (.6) an f [cov ] s e aa covarance arx wc escrbes e aa errors, en e a poseror oel covarance arx s gven by [cov ] [cov ] (.63) e covarance arx n Equaon (.63) s calle e a poseror oel covarance arx because s calculae afer e fac. I gves wa are soees calle e foral unceranes n e oel paraeers. I s fferen fro e a pror oel covarance arx of Equaon (3.85), wc s use o consran e unereerne proble. e a poseror covarance arx n Equaon (.63) sows explcly e appng of aa errors no unceranes n e oel paraeers. Aloug e appng wll be clearer once we conser e generale nverse n Caper 7, s nsrucve a s pon o conser applyng Equaon (.63) o e leas squares an nu leng probles. 3.9. Applcaon o eas Squares e can apply Equaon (.63) o e leas squares proble an oban 65
eoscences 567: CHAPER 3 (RR/Z) [cov ] {[ ] }[cov ]{[ ] } (3.49) Furer, f [cov ] s gven by [cov ] σ I (3.5) en [cov ] [ ] [σ I]{[ ] } σ [ ] {[ ] } σ {[ ] } σ [ ] (3.5) snce e ranspose of a syerc arx reurns e orgnal arx. 3.9.3 Applcaon o e nu eng Proble Applcaon of Equaon (.63) o e nu leng proble leas o e followng for e a poseror oel covarance arx: If e aa covarance arx s agan gven by [cov ] { [ ] }[cov ]{ [ ] } (3.5) we oban [cov ] σ I (3.53) [cov ] σ [ ] (3.54) were [ ] [ ] [ ] (3.55) 3.9.4 eoercal Inerpreaon of Varance ere s anoer way o look a e varance of oel paraeer esaes for e leas squares proble a consers e precon error, or sf, o e aa. Recall a we efne e sf E as E e e [ pre ] [ pre ] [ ] [ ] (3.3) 66
eoscences 567: CHAPER 3 (RR/Z) wc explcly sows e epenence of E on e oel paraeers. a s, we ave E E() (3.56) If E() as a sarp, well-efne nu, en we can conclue a our soluon S s well consrane. Conversely, f E() as a broa, poorly efne nu, en we conclue a our soluon S s poorly consrane. Afer Fgure 3., page 59, of enke, we ave (nex page), (a) (b) E() E() E E es oel paraeer es oel paraeer (a) e bes esae es of oel paraeer occurs a e nu of E(). If e nu s relavely narrow, en rano flucuaons n E() lea o only sall errors n es. (b) If e nu s we, en large errors n can occur. One way o quanfy s qualave observaon s o reale a e w of e nu for E() s relae o e curvaure, or secon ervave, of E() a e nu. For e leas squares proble, we ave E [ ] S S (3.57) Evaluang e rg-an se, we ave for e q er E q q [ ] j (3.58) j q j ( ) j j (3.59) q q j 67
eoscences 567: CHAPER 3 (RR/Z) q jq (3.6) j q j Usng e sae seps as we n e ervaon of e leas squares soluon n Equaons (3.4) (3.9), s possble o see a Equaon (3.6) represens e q er n [ ]. Cobnng e q equaons no arx noaon yels [ ] { [ ] } (3.6) Evaluang e frs ervave on e rg-an se of Equaon (3.6), we ave for e q er jq (3.6) j { [ ] } q q q j j q ( ) (3.63) j q j q q (3.64) wc we recogne as e (q, q) enry n. erefore, we can wre e arx equaon as { [ ] } (3.65) Fro Equaons (3.5) (3.58) we can conclue a e secon ervave of E n e leas squares proble s proporonal o. a s, E S (consan) (3.66) Furerore, fro Equaon (3.5) we ave a [cov ] s proporonal o [ ]. erefore, we can assocae large values of e secon ervave of E, gven by (3.66) w () sarp curvaure for E, () narrow well for E, an (3) goo (.e., sall) oel varance. As enke pons ou, [cov ] can be nerpree as beng conrolle eer by () e varance of e aa es a easure of ow error n e aa s appe no oel paraeers or () a consan es e curvaure of e precon error a s nu. 68
eoscences 567: CHAPER 3 (RR/Z) I lke enke s suary for s caper (page 6) on s aeral very uc. Hence, I've reprouce s closng paragrap for you as follows: e eos of solvng nverse probles a ave been scusse n s caper epase e aa an oel paraeers eselves. e eo of leas squares esaes e oel paraeers w salles precon leng. e eo of nu leng esaes e sples oel paraeers. e eas of aa an oel paraeers are very concree an sragforwar, an e eos base on e are sple an easly unersoo. evereless, s vewpon ens o obscure an poran aspec of nverse probles. aely, a e naure of e proble epens ore on e relaonsp beween e aa an oel paraeers an on e aa or oel paraeers eselves. I soul, for nsance, be possble o ell a well-esgne experen fro a poorly esgne one wou knowng wa e nuercal values of e aa or oel paraeers are, or even e range n wc ey fall. Before conserng e relaonsps ple n e appng beween oel paraeers an aa n Caper 5, we exen wa we now know abou lnear nverse probles o nonlnear probles n e nex caper. 69