Linear Regression Using Combined Least Squares

Transcription

1 Abtrat Linar Rgrion Uing Combind Lat Squar R.E. Dakin Bonbah VIC, 396, Autralia Vrion : 3-No-4 Fitting a traight lin through, data i a ommon problm in timation. Uing a data plot and a rulr, th problm i old b lowl moing th rulr to a poition that minimiz (iua) th ditan btwn th data point and th rulr. h i a r apabl tool and th ditan that ar bing minimizd ar prpndiular to th ariabl loation of th rulr. hi problm, ad hr Linar Rgrion, i alo old mathmatia uing th thor of lat quar that uppo th mot probabl anwr i on that minimiz th um of th quar of th wightd ridual. Hr ridual man ma orrtion to maurmnt and wight ar numbr rflting priion of maurmnt. Uual lat quar olution onnintl aum ridual ar aoiatd with on maurmnt onl uua th - oordinat and th othr maurmnt i rror-fr. hi aumption do not aord with, data that ha aring priion. hi papr wi dmontrat how Combind Lat Squar an b ud for linar rgrion with propr rgard to, priion. Combind Lat Squar A ommon tratmnt of th lat quar thniqu of timation tart with impl linar mathmatial modl haing obration (or maurmnt) a pliit funtion of paramtr with non-linar modl dlopd a tnion. hi adjutmnt thniqu i gnra dribd a adjutmnt of indirt obration (alo ad paramtri lat quar). Ca whr th mathmatial modl ontain onl maurmnt ar uua tratd paratl and thi thniqu i oftn dribd a adjutmnt of obration onl (alo ad ondition quation). Both thniqu ar of our partiular a of a gnral thniqu, omtim ad Combind Lat Squar, th olution of whih i t out blow. hi gnral thniqu alo aum that th paramtr, if an, an b tratd a obrabl, i.., th ha an a priori oarian matri. hi onpt aow th gnral thniqu to b adaptd to quntial proing of data whr paramtr ar updatd b th addition of nw obration. In gnral, lat quar olution rquir itration, in a non-linar modl i aumd. h itrati pro i plaind blow. In addition, a propr tratmnt of oarian propagation i prntd and ofator matri gin for a th omputd and drid quantiti in th adjutmnt pro. Fina, th partiular a of th gnral lat quar thniqu ar dribd. h Combind Lat Squar Adjutmnt Modl Conidr th foowing t of non-linar quation rprnting th mathmatial modl in an adjutmnt ( ˆ, ˆ ) F l () whr l i a tor of n obration and i a tor of u paramtr; ˆl and ˆ rfrring to timat drid from th lat quar pro uh that ˆl l + and ˆ + δ () whr i a tor of ridual or ma orrtion and δ i a tor of ma orrtion. h obration l ha an a priori ofator matri Q ontaining timat of th arian and oarian of th obration. In man a th obration ar indpndnt and Q i diagonal. In thi gnral tahniqu, th paramtr ar tratd a obrabl with a fu a priori ofator matri Q. h diagonal lmnt of Q ontain timat of arian of th paramtr and th off-diagonal lmnt ontain timat of th oarian btwn paramtr. Cofator matri Q and Q ar rlatd to th oarian matri Σ and Σ b th arian fator

2 Σ Q Σ Q (3) Alo, wight matri W ar uful and ar dfind, in gnral, a th inr of ofator matri W Q (4) and oarian, ofator and wight matri ar a mmtri, hn Q Q and W W whr th uprript dnot th tranpo of th matri. Not alo, that in thi dlopmnt whr Q and W ar writtn without ubript th rfr to th obration, i.., Q Q and W W Linarizing () uing alor' thorm and ignoring nd and highr-ordr trm, gi ( ˆ F ) ( ) ( ˆ F, ˆ F, ) ( ˆ ) F l l + + ˆl l l ˆ l, l, (5) and with ˆl l and δ ˆ from () w ma writ th linarizd modl in mboli form a A + Bδ f (6) Equation (6) rprnt a tm of m quation that wi b ud to timat th u paramtr from n obration. It i aumd that thi i a rdundant tm whr n m u (7) and r m u (8) i th rdundan or dgr of frdom. In quation (6) th offiint matri A and B ar dign matri ontaining partial driati of th funtion aluatd uing th obration l and th "obrd" paramtr. A F F B ˆ ˆ m, n m, u l l, l, (9) h tor f ontain m numri trm alulatd from th funtional modl uing l and. { F ( )} f l () m,, h Lat Squar Solution of th Combind Modl h lat quar olution of (6), i.., th olution whih mak th um of th quar of th wightd ridual a minimum, i obtaind b minimizing th alar funtion ϕ ( ) ϕ W + δ W δ k A + Bδ f () whr k i a tor of m Lagrang multiplir. ϕ i a minimum whn it driati with rpt to and δ ar quatd to zro, i.. ϕ ϕ and δ W k A δ W k B h quation an b implifid b diiding both id b two, tranpoing and hanging ign to gi W + A k and W + B k () δ

3 Equation () an b ombind with (6) and arrangd in matri form a W A A B k f B W δ (3) A rdution pro applid to (3) gin b Cro (99, pp. -3) lad to a t of normal quation ( ( ) + ) δ ( ) B AQA B W B AQA f (4) Mikhail (976, p. 4) implifi (4) b introduing quialnt obration l whr l A l (5) Appling th matri rul for ofator propagation (Mikhail 976, pp ) gi th ofator matri of th quialnt obration a Q AQA (6) With th uual rlationhip btwn wight matri and ofator matri, [ (4)], w ma writ Uing (7) in (4) gi th normal quation a with th auiliari N and t a ( ) W Q AQA (7) ( ) B W B + W δ B W f (8) N B W B t B Wf (9) h tor of orrtion δ i gin b ( ) h rdution pro applid to (3) alo ild th tor of Lagrang multiplir k and th tor of ridual i obtaind from () a δ N + W t () ( ) ( δ ) ( δ ) k AQA f B W f B () W A k QA k () h Itrati Pro of Solution Rmmbring that ˆ + δ, whr i th tor of a priori timat of th paramtr, δ i a tor of orrtion and ˆ i th lat quar timat of th paramtr. At th bginning of th itrati olution, it an b aumd that ˆ qual th a priori timat and a t of orrtion δ omputd. h ar addd to to gi an updatd t. A and B ar ralulatd and a nw wight matri W omputd b ofator propagation. h orrtion ar omputd again, and th whol pro l through until th orrtion rah om prdtrmind alu, whih trminat th pro. ˆ +δ n + n n (3) 3

4 Cofator Matri Driation of th ofator matri i a lngth pro and th rult gin blow an b found in Mikhail (976, pp ) Cofator Matri for ˆ ( + ) Cofator Matri for ˆl ( ) ˆˆ Q N W (4) ˆˆ Q Q + QA W B N + W B W AQ QA W AQ (5) Cofator Matri for δ ( + ) Q N W NQ (6) δδ Cofator Matri for Coarian Matri Σ ˆˆ h timatd arian fator i whr th dgr of frdom r ar Q Q Q (7) ˆˆ ˆˆ Σ Q (8) + δ ˆ ˆ W (9) r δ r m u + u (3) m i th numbr of quation ud to timat th u paramtr from n obration. u i th numbr of wightd paramtr. [Equation (3) i gin b Krakiwk (975, p.7, qn -6) who not that it i an approimation onl and dirt th radr to Bolr (97) for a omplt and rigorou tratmnt.] Gnration of th Standard Lat Squar Ca Combind Ca with Wightd Paramtr ( A; B; W; W ) h gnral a of a non-linar impliit modl with wightd paramtr tratd a obrabl i known a th Combind Ca with Wightd Paramtr. It ha a olution gin b th foowing quation. ( ) δ N + W t (3) with ( ) N B W B t B W f W Q AQA (3) ˆ + δ (33) ( δ ) k W f B (34) W A k QA k (35) ˆˆ ˆˆ ˆ l l + (36) ( ) Q N + W (37) ( ) Q Q + QA W B N + W B W AQ QA W AQ (38) Q Q Q (39) ˆˆ + δ Wδ + δ Wδ r m u + u (4) 4

5 Σ Q Σ Q Σ Q (4) ˆ ˆ ˆ ˆ ˆˆ ˆˆ Combind Ca ( A ; B ; W ; W ) h Combind Ca i a non-linar impliit mathmatial modl with no wight on th paramtr. h t of quation for th olution i ddud from th Combind Ca with Wightd Paramtr b onidring that if thr ar no wight thn W and Q. hi impli that i a ontant tor (dnotd b ) of approimat alu of th paramtr, and partial driati with rpt to ar undfind. Subtituting th two nu matri and th ontant tor into quation (3) to (4) gi th foowing rult. δ N t (4) with (, ) ( F ) N B W B t B W f f l W Q AQA (43) ˆ δ + (44) ( δ ) k W f B (45) W A k QA k (46) ˆ l l + (47) Qδ δ Q ˆ ˆ N (48) Q Q + QA W B N B W AQ QA W AQ (49) ˆˆ Q Q Q (5) ˆˆ r m u (5) Σ Q Σ Q Σ Q (5) ˆ ˆ ˆ ˆ ˆˆ ˆˆ Paramtri Ca ( A I B W W ) ; ; ; h Paramtri Ca i a mathmatial modl with th obration l pliitl prd b om non-linar funtion of th paramtr onl. hi impli that th dign matri A i qual to th idntit matri I. Stting A I in th Combind Ca (with no wight) lad to th foowing quation. with F (, ) δ N t (53) N B WB t B Wf f l (54) ˆ + δ (55) f Bδ (56) ˆ l l + (57) Qδ δ Q ˆ ˆ N (58) Q Q BN B (59) 5

6 Q ˆˆ B N B (6) r n u (6) Σ Q Σ Q Σ Q (6) ˆ ˆ ˆ ˆ ˆˆ ˆˆ Condition Ca ( A B W W ) ; ; ; h Condition Ca i haratrizd b a non-linar modl oniting of obration onl. Stting B in th Combind Ca (with no wight) lad to th foowing quation. with ( ) ( ) k W f (63) F W Q AQA f l (64) W A k QA k (65) ˆ l l + (66) Q ˆˆ Q QA W AQ (67) Q Q Q (68) ˆˆ r m (69) Σ Q Σ Q (7) ˆˆ ˆˆ wo ampl foow howing linar rgrion uing Combind Lat Squar. Eampl : Lin of Bt Fit indpndnt data of qual priion 5 C 3 4 m + Point (mm) (mm) Figur Lin of Bt Fit through data point to 5 6

7 h lin of bt fit hown in th Figur ha th quation m + whr m i th lop of th lin m tan θ and i th intrpt of th lin on th ai. m and ar th paramtr and th data point ar aumd to aord with th mathmatial modl m + and th, oordinat pair of ah data point ar onidrd a indirt maurmnt of th paramtr m and of th mathmatial modl. h oordinat ar aumd to b indpndnt and of qual priion and W I. Auming that ridual and ar aoiatd with both th and maurmnt (th oordinat pair) th obration quation i ( ) + m + + (7) k k k k Lt + (7) whr But, in m i an approimat alu and δ m i a ma orrtion and ubtituting (7) into (7) gi ( δ )( ) + m + m + + k k k k m + m + δ m + δ m + (73) k k k k k and δ m ar both ma, thn th produt k δ m in (73) and w writ th obration quation a + m + m + δ m + (74) k k k k k R-arranging th obration quation o that unknown quantiti ar on th lft-hand-id and known quantiti ar on th right-hand-id of th qual ign gi m + δ m m (75) k k k k k For th 5 data pair ( n obration), th ( m 5 ) obration quation for th ( u ) paramtr ar m + δ m m m + δ m m m + δ m m m + δ m m m + δ m m h an b writtn in th matri form A( ) ( ) + B,, (, ) δ m n n m u ( u, ) f ( m,) (Combind Ca A; B; W I; W ) a m m m m 3 δ m m m 3 3 m 4 m m 5 m

8 h lat quar olution for th paramtr δ m and and th rlatd priion timation ar gin b quation (4) to (5). h olution i itrati, trminating whn δ m rah a uffiintl ma alu. For a firt itration with an approimat alu m.55 th numrial alu in th matri A, B and f ar A(, ) m n.55 B(, ) m u f ( m,) With W(, ) n n I and Q W th quialnt wight matri W i W ( ) ( A( ) Q( ) A ( ) ) m, m m, n n, n n, m h matri N and t ar N( ) B, (, ) W (, ) B(, ) u u u m m m m u ( u, ) ( u, m) ( m, m) ( m,) t B W f h olution ar m δ ( u, ) N( u, u) t ( u,) and A ond (and lat) itration with m gi th olution a m 5.9 δ ( u, ) N( u, u) t ( u,) and and th ridual ( n, ) Q( n, n) A( m, n) k ( n,)

9 Eampl : Lin of Bt Fit orrlatd data of aring priion 5 C 3 m + 4 Point (mm) (mm) Figur Lin of Bt Fit through data point to 5 that ha aring priion h data point in Figur ha aring priion indiatd b rror ip and th iz, hap and orintation of th rror ip ar funtion of th arian and oarian of th oordinat at ah point. A a gnral rul, a point with a ma rror ip i mor pril loatd than a point with a larg rror ip. h ofator matri Q of th maurmnt (th oordinat pair) ha th foowing form Q Whr, ar timat of th arian k k and rptil and k k i an timat of th oarian., ar timat of th tandard diation, and tandard diation i th poiti quar root of th arian. Not that orrlation ρ and ρ h atual numri alu (mm ) for ah point ar: Q

10 h mthod of olution i idntial to Eampl For a firt itration with an approimat alu m.55 th numrial alu in th matri A, B and f ar A(, ) m n.55 B(, ) m u f ( m,) With W(, ) n n I and Q W th quialnt wight matri W i W ( ) ( A( ) Q( ) A ( ) ) m, m m, n n, n n, m h matri N and t ar N( ) B, (, ) W (, ) B(, ) u u u m m m m u ( u, ) ( u, m) ( m, m) ( m,) t B W f h olution ar m δ ( u, ) N( u, u) t ( u,) and A ond itration with gi th olution a m δ ( u, ) N( u, u) t ( u,) and A third itration with gi th olution a m.455 δ ( u, ) N( u, u) t ( u,) and A fourth (and lat) itration with gi th olution a m.58 δ ( u, ) N( u, u) t ( u,) and and th ridual ( n, ) Q( n, n) A( m, n) k ( n,)

11 Rfrn Bolr, John D. 97, 'Baian Infrn in God', Ph.D. Dirtation, Dpartmnt of Godti Sin, Ohio Stat Unirit, Columbu, Ohio, USA. Cro, P.A. 99, Adand Lat Squar Applid to Poition Fiing, Working Papr No. 6, Dpartmnt of Land Information, Unirit of Eat London. Krakiwk, E.J. 975, A Snthi of Rnt Adan in th Mthod of Lat Squar, Ltur Not No. 4, 99 rprint, Dpartmnt of Suring Enginring, Unirit of Nw Brunwik, Frdriton, Canada Mikhail, E.M. 976, Obration and Lat Squar, IEPA Dun-Donn, Nw York.