Least squares. Václav Hlaváč. Czech Technical University in Prague

Transcription

1 Lest squres Václv Hlváč Czech echncl Unversty n Prgue hlvc@fel.cvut.cz Courtesy: Fred Pghn nd J.P. Lews, SIGGRAPH 2007 Course;

2 Outlne 2 Lner regresson Geometry of lest-squres Dscusson of the Guss-Mrkov theorem

3 One-dmensonl regresson 3

4 One-dmensonl regresson 4 Fnd lne tht represent the est lner reltonshp:

5 One-dmensonl regresson Prolem: the dt does not go through lne e 5

6 One-dmensonl regresson Prolem: the dt does not go through lne e 6 Fnd the lne tht mnmzes the sum: ( ) 2

7 One-dmensonl regresson Prolem: the dt does not go through lne e 7 Fnd the lne tht mnmzes the sum: ( We re lookng for mnmzes ) ˆ e( ) ( 2 tht 2 )

8 Lest squres emple here re 3 mountns u, y, z tht from one ste hve een mesured s 2474 m, 3882 m, nd 4834 m. But from u, y looks 1422 m tller nd the z looks 2354 m tller, nd from y, z looks 950 m tller. Set up the overdetermned system. 8 A u y z ~ Wnt to mnmze A- 2

9 Approches to solve A Norml equtons-quck nd drty QR- stndrd n lrres uses orthogonl decomposton SVD - decomposton whch lso gves ndcton how lner ndependent columns re Conjugte grdent - no decompostons, good for lrge sprse prolems 9

10 10 Mtr notton Usng the followng nottons nd we cn rewrte the error functon usng lner lger s: n : 1 n : ) ( ) ( ) ( ) ( ) ( e e

11 Multdmentonl lner regresson 11 Usng model wth m prmeters m m j j j 1 2

12 12 Usng model wth m prmeters nd n mesurements + + j j j m m , 2 1 1, ) ( ) ( ) ( A e e m j j j n m j j j Multdmentonl lner regresson

13 )... ( : )... ( :... : :.. :, 1,1 1, 1 1,1 1 1,,1 1, 1,1 1 m m n n n m m n m n n m n A mesurement n prmeter 1 Mtr representton

14 Mnmzng e() e() s flt t mn ) 0 e( mn 14 mn mnmzes e( ) f e() does not go down round mn e() H e ( mn ) s sem- defnte postve mn

15 Postve sem-defnte A s postve sem- defnte A 0, for ll 15 In 1-D In 2-D

16 Mnmzng e() 16 e() 1 2 H ( ˆ ) e ˆ

17 Mnmzng e( ) A e ( ) H ( ˆ e ) 2 ˆ

18 Mnmzng e( ) A 2 18 A Aˆ A he norml equton ˆ mnmzes e( ) f 2A A s postve sem- defnte Alwys true

19 Geometrc nterpretton s vector n R n 19

20 Geometrc nterpretton s vector n R n he columns of A defne vector spce rnge(a)

21 Geometrc nterpretton s vector n R n he columns of A defne vector spce rnge(a) A s n rtrry vector n rnge(a) A 2

22 Geometrc nterpretton s vector n R n he columns of A defne vector spce rnge(a) A s n rtrry vector n rnge(a) 22 A A 2

23 Geometrc nterpretton s the orthogonl projecton of onto rnge(a) A ˆ 23 A ( Aˆ ) 0 A Aˆ A Aˆ 1 ˆ ˆ Aˆ 2

24 he norml equton: A Aˆ A 24

25 he norml equton: Estence: A Aˆ A A Aˆ hs lwys soluton A 25

26 he norml equton: A Aˆ Estence: A Aˆ A hs lwys soluton Unqueness: the soluton s unque f the columns of A re lnerly ndependent A 26

27 he norml equton: Aˆ Estence: A Aˆ A hs lwys soluton Unqueness: the soluton s unque f the columns of A re lnerly ndependent A A 27 Aˆ 2 1

28 Under-constrned prolem

29 Under-constrned prolem

30 Under-constrned prolem

31 Under-constrned prolem Poorly selected dt One or more of the prmeters re redundnt

32 Under-constrned prolem Poorly selected dt One or more of the prmeters re redundnt 1 32 Add constrnts A A A wth mn 2

33 How good s the lest-squres? Optmlty: the Guss-Mrkov theorem Let nd e two sets of rndom vrles nd defne: If { } { } j,... e 11, m A1: A2: { },j E(e A3: vr(e A4 : cov( e re not rndom vrles, ) 0, for ll, ) σ, for ll,, e j m ) 0, for ll nd j, 33 hen ˆ 2 rg mn e s the est unsed lner estmtor

34 34 e no errors n

35 35 e e no errors n errors n

36 36 homogeneous errors

37 37 homogeneous errors non-homogeneous errors

38 38 no outlers

39 outlers 39 no outlers outlers