Lest squres Václv Hlváč Czech echncl Unversty n Prgue hlvc@fel.cvut.cz http://cmp.felk.cvut.cz/~hlvc Courtesy: Fred Pghn nd J.P. Lews, SIGGRAPH 2007 Course;
Outlne 2 Lner regresson Geometry of lest-squres Dscusson of the Guss-Mrkov theorem
One-dmensonl regresson 3
One-dmensonl regresson 4 Fnd lne tht represent the est lner reltonshp:
One-dmensonl regresson Prolem: the dt does not go through lne e 5
One-dmensonl regresson Prolem: the dt does not go through lne e 6 Fnd the lne tht mnmzes the sum: ( ) 2
One-dmensonl regresson Prolem: the dt does not go through lne e 7 Fnd the lne tht mnmzes the sum: ( We re lookng for mnmzes ) ˆ e( ) ( 2 tht 2 )
Lest squres emple here re 3 mountns u, y, z tht from one ste hve een mesured s 2474 m, 3882 m, nd 4834 m. But from u, y looks 1422 m tller nd the z looks 2354 m tller, nd from y, z looks 950 m tller. Set up the overdetermned system. 8 A 1 0 0 0 1 0 0 0 1-1 1 0-1 0 1 0-1 1 u y z ~ 2474 3882 4834 1422 2354 950 Wnt to mnmze A- 2
Approches to solve A Norml equtons-quck nd drty QR- stndrd n lrres uses orthogonl decomposton SVD - decomposton whch lso gves ndcton how lner ndependent columns re Conjugte grdent - no decompostons, good for lrge sprse prolems 9
10 Mtr notton Usng the followng nottons nd we cn rewrte the error functon usng lner lger s: n : 1 n : 1 2 2 ) ( ) ( ) ( ) ( ) ( e e
Multdmentonl lner regresson 11 Usng model wth m prmeters +... + 1 1 m m j j j 1 2
12 Usng model wth m prmeters nd n mesurements + + j j j m m... 1 1 2 2 1, 2 1 1, ) ( ) ( ) ( A e e m j j j n m j j j Multdmentonl lner regresson
13 + + + + )... ( : )... ( :... : :.. :, 1,1 1, 1 1,1 1 1,,1 1, 1,1 1 m m n n n m m n m n n m n A mesurement n prmeter 1 Mtr representton
Mnmzng e() e() s flt t mn ) 0 e( mn 14 mn mnmzes e( ) f e() does not go down round mn e() H e ( mn ) s sem- defnte postve mn
Postve sem-defnte A s postve sem- defnte A 0, for ll 15 In 1-D In 2-D
Mnmzng e() 16 e() 1 2 H ( ˆ ) e ˆ
Mnmzng e( ) A 2 17 1 e ( ) H ( ˆ e ) 2 ˆ
Mnmzng e( ) A 2 18 A Aˆ A he norml equton ˆ mnmzes e( ) f 2A A s postve sem- defnte Alwys true
Geometrc nterpretton s vector n R n 19
Geometrc nterpretton s vector n R n he columns of A defne vector spce rnge(a) 20 1 2
Geometrc nterpretton s vector n R n he columns of A defne vector spce rnge(a) A s n rtrry vector n rnge(a) 21 1 1 1 + 2 2 A 2
Geometrc nterpretton s vector n R n he columns of A defne vector spce rnge(a) A s n rtrry vector n rnge(a) 22 A 1 1 1 + 2 2 A 2
Geometrc nterpretton s the orthogonl projecton of onto rnge(a) A ˆ 23 A ( Aˆ ) 0 A Aˆ A Aˆ 1 ˆ ˆ 1 + 2 2 1 Aˆ 2
he norml equton: A Aˆ A 24
he norml equton: Estence: A Aˆ A A Aˆ hs lwys soluton A 25
he norml equton: A Aˆ Estence: A Aˆ A hs lwys soluton Unqueness: the soluton s unque f the columns of A re lnerly ndependent A 26
he norml equton: Aˆ Estence: A Aˆ A hs lwys soluton Unqueness: the soluton s unque f the columns of A re lnerly ndependent A A 27 Aˆ 2 1
Under-constrned prolem 28 1 2
Under-constrned prolem 29 1 2
Under-constrned prolem 30 1 2
Under-constrned prolem Poorly selected dt One or more of the prmeters re redundnt 1 31 2
Under-constrned prolem Poorly selected dt One or more of the prmeters re redundnt 1 32 Add constrnts A A A wth mn 2
How good s the lest-squres? Optmlty: the Guss-Mrkov theorem Let nd e two sets of rndom vrles nd defne: If { } { } j,... e 11, m A1: A2: { },j E(e A3: vr(e A4 : cov( e re not rndom vrles, ) 0, for ll, ) σ, for ll,, e j m ) 0, for ll nd j, 33 hen ˆ 2 rg mn e s the est unsed lner estmtor
34 e no errors n
35 e e no errors n errors n
36 homogeneous errors
37 homogeneous errors non-homogeneous errors
38 no outlers
outlers 39 no outlers outlers