Structure from Motion. Forsyth&Ponce: Chap. 12 and 13 Szeliski: Chap. 7

Transcription

1 Structure from Moton Forsyth&once: Chap. 2 and 3 Szelsk: Chap. 7

2 Introducton to Structure from Moton Forsyth&once: Chap. 2 Szelsk: Chap. 7

3 Structure from Moton Intro

4 he Reconstructon roblem p 3?? p p 2

5 he Reconstructon roblem m Images n onts Assume correspondences p m p p 2

6 Example from Uenn s group

7 Communty photo collectons Example: Query Notre Dame, ars on Flckr

8 he Reconstructon roblem m Images n onts Reconstructon = Assume correspondences Scene ponts, =,..n p m Camera proecton matrces M, =,..,m Such that: p p p 2 M,

9 Scale Ambguty p Z p 2 D

10 Scale Ambguty If we scale the entre scene by some factor k and, at the same tme, we scale the dstance between the cameras by the same scale factor, the proectons of the scene ponts n the mage reman exactly the same! It s mpossble to recover the absolute scale of the scene. p p 2 D = kd Z = kz

11 If ~ ~ M ( (,..,,.., n) m hen, gven a 4x4 matrx Q, the other reconstructon defned as: s also a soluton because: p M ) M s the true reconstructon of all the ponts and all the proecton matrces ~ Q ~ M M Q herefore: he reconstructon s defned only up to a global transformaton. We consder 3 types of transformatons: metrc, affne, proectve., QQ ~ M ~

12 If ~ (,.., n) ~, (,.., m M ) s the true reconstructon of all the ponts and all the proecton matrces In plan Englsh: If we transform ~ the scene and we transform the Q cameras n ~ the opposte way, then Mthe mages M Qdo hen, gven a 4x4 matrx Q, the other reconstructon defned as: s also a soluton because: not change! p M M herefore: he reconstructon s defned only up to a global transformaton. We consder 3 types of transformatons: metrc, affne, proectve., QQ ~ M ~

13 Q = smlarty (scale+rotaton+translaton) Metrc Affne Q = affne (last row s [0 0 0 ]) Q = general proectve (arbtrary 4x4 matrx) roectve

14

15 he Affne Case Forsyth&once: Chap. 2 Szelsk: Chap. 7

16 Orthographc roecton p v u u v I 2 x 3 X Y Z 0 I 2 x 3 0 X Y Z u = X v = Y

17 Orthographc roecton: General p v u u v I 2 x 3 R X Y Z t R 2 x 3 b X Y Z A b X Y Z A X Y Z b

18 Weak erspectve v p u u v s 0 0 s R 2 x 3 b X Y Z A b X Y Z A X Y Z b

19 General Affne v p u u v s 0 a s 2 R 2 x 3 t ' X Y Z A b X Y Z A X Y Z b

20 Affne Eppolar Geometry p p' A A' b b' p p F p ' 0

21 Affne Eppolar Geometry In pthe affne A case, b the fundamental matrx smplfes to a 4-degree of p' A' b' p freedom matrx, whch can be estmated from 4 pont correspondences: F p F p ' 0 he eppolar lnes are always // n the affne case

22 Fundamental SFM theorem: Scene structure (the coordnates of the scene ponts ) and camera geometry (proecton matrces (A,b)) can be recovered from 4 pont correspondences from 2 vews. In general, the reconstructon can be solved f: 2mn 8m 3n 2 he reconstructon s not unque, t s defned up to an arbtrary affne transformaton Q (2 degrees of freedom) A b Q A b Q Q C 0 d

23 m Images n Features Reconstructon from many mages: p = A + b p m p p 2

24 m Images n Features p = A + b Centerng: Substract the centrod p n p from all the mage ponts p,..,p n p m p p 2

25 After centerng, each pont s related to ts proecton by: p = A Stackng all 2m equatons nto a sngle matrx: p p n A D MS D M S n p m p mn A m In partcular, D s of rank 3

26 2m Fundamental Decomposton Measurements = Moton Shape 3 n D = MS n 3

27 n n n n 2m D = U W V n 3 3 o reduce to rank 3, we ust need to set all the sngular values to 0 except for the frst 3 2m D = U 3 3 W 3 V 3 3

28 3 n 2m D = U 3 3 W 3 V M ossble decomposton: U / 2 / 2 W S W V D = M S

29 Gven m mages and n features p Center: For each mage : Compute the centrod: Center the feature coordnates: Accumulate: Construct a 2m by n matrx D such that: Columns: Column contans the proecton of pont n all vews Rows: Row contans one coordnate of the proectons of all the n ponts n mage Factorze: Decompose D: Compute the SVD of D: D = UWV Create U 3 by takng the frst 3 columns of U Create V 3 by takng the frst 3 columns of V Create W 3 by takng the upper left 3x3 block of W Create the moton and shape matrces: M = U and S = W V (or M=U W /2 S=W ½ V )

30

31 he decomposton s not unque. We get the same D by usng any 3x3 C and applyng the n transformatons: 3 M 2m D = U 3 3 W 3 V 3 3 hat s because we have only an affne transformaton and we have not enforced any Eucldean constrants (lke forcng the mage axs to be perpendcular, for example) 3 MC S C S D = M S

32 Orthographc: Image axs are perpendcular and scale s p a 2 a a. a 0 a a

33 hs translates nto a set of 3m lnear equatons n L = CC Orthographc: Image axs are perpendcular and scale s : A LA,. Solve for L 2. Recover C from L by Cholesky decomposton: L = CC 3. Correct M and S: p M MC S C S Id, m a 2 a a. a 0 a a

34 Weak erspectve: Image axs are perpendcular and scales are equal (not necessarly ) p a 2 a a. a a a

35 Weak hs erspectve: translates nto Image a set axs of 2m are lnear perpendcular equatons and n L scales = CCare : equal (not necessarly ) a La 0 a La a La, Solve for L 2. Recover C from L by Cholesky p decomposton: L = CC 3. Correct M and S: M MC S C S a 2, m a a. a a a

36 Summary Affne SFM theorem: Affne reconstructon from two cameras can be recovered ff at least n = 4 ndependent features Affne reconstructon defned up to a global affne transformaton of 3D space (2 parameters) Eppolar geometry: Affne fundamental matrx between 2 cameras defned by 4 ndependent parameters. Eppolar lnes are always parallel Affne camera geometry can be recovered lnearly from the affne fundamental matrx General case: Affne reconstructon from n cameras possble f 2mn >= 8m+3n-2 Factorzaton algorthm: Rank 3 SVD of the matrx of feature coordnates yelds affne reconstructon and moton Metrc reconstructon possble under assumptons: Orthography Weak perspectve araperspectve Metrc reconstructon defned up to a global rotaton

37 General Case: roectve Reconstructon Forsyth&once: Chap. 3 Szelsk: Chap. 7

38 General roecton Model Z Y v p p K R t A X b u p u v X Y Z

39 m Images n Features Assume correspondences p m p p 2 roectve SFM heorem: Reconstructon s possble as long as 2mn >= m +3n 5

40 Metrc Affne roectve

41 p p e C C p M p ' M ' p Fp 0 '

42 p M p ' M ' p Fp 0 ' wo-mage case key result: Reconstructon from 2 mages s possble from at least p7 correspondences he proecton matrx can be computed p from the fundamental matrx F M ' Id 0 F F b e 0 C C A b F

43 Issues We know how to compute the camera geometry from 2 mages * Remanng ssues:. What about m > 2 mages? 2. rangulaton to compute pont n 3D 3. roectve reconstructon s not unque: How can we fnd the metrcally correct reconstructon? 4. How can we fnd the optmal reconstructon? *: (or 3, not shown n class)

44 Issue: Arbtrary number of mages Factorzaton (e.g., assumng affne frst) Sequental methods Herarchcal methods..

45 Sequental Methods M M2 M M m M + Start reconstructon from two (or three) mages by usng the fundamental matrx (or the trfocal tensor). Gven proecton matrx M of mage, fnd the proecton matrx of mage + such that: M = M + for the ponts n common between the two mages M + can be determned from 6 ponts RANSAC f correspondences not known

46 Herarchcal Methods F Q Example from ollefeys Compute F, from groups of 2 or 3 mages Iteratvely sttch the mages nto larger groups Sttchng = Fnd a matrx Q that algns the reconstructon from one group wth the reconstructon from another group Need some features n common between the

47 =coordnates of pont for frst group =coordnates of pont for second group Sttchng Fnd Q that transforms the pont n the second group to the ponts n the frst group M M2 M M + M m mn d (, Q ) Dstance n space Dstance n camera geometry mn d ( M, M Q ) Reproecton error mn d M Q, p d M Q, p

48 3792 ponts 270 ponts 3792 ponts Correspondences 64 mages 9.8m/pt 4.8m/pt roblem: Need enough correspondences n common between vews

49 Issues We know how to compute the camera geometry from 2 mages * Remanng ssues:. What about m > 2 mages? 2. rangulaton to compute pont n 3D 3. roectve reconstructon s not unque: How can we fnd the metrcally correct reconstructon? 4. How can we fnd the optmal reconstructon? *: (or 3, not shown n class)

50 Issue: rangulaton u p C

51 u p C. ont closest to the rays n space (lnear) 2. ont wth lowest algebrac reproecton error (lnear) 3. ont wth lowest geometrc reproecton error (non-lnear) 2 Z Y X W s C Mn 2 M p Mn m m v m m u

52 Issues We know how to compute the camera geometry from 2 mages * Remanng ssues:. What about m > 2 mages? 2. rangulaton to compute pont n 3D 3. roectve reconstructon s not unque: How can we fnd the metrcally correct reconstructon? 4. How can we fnd the optmal reconstructon? *: (or 3, not shown n class)

53 From roectve to Metrc Reconstructon Auto Calbraton Forsyth&once: Chap. 3 Szelsk: Chap. 7

54 M M Q roectve Q Metrc

55 K u s u 0 v v o 0 0 o M =K [Id 0] M m =K m [R m t m ] M =K [R t ]

56 K u s u 0 v v o 0 0 o M m =K[R m t m ] M =K[R t ]

57 M =K[R t ] M m =K[R m t m ] K K K R R K M Q Q M R K Q M t R K Q M Basc trck: Look only at the frst 3 columns Use the fact that R s a rotaton

58 K u s u 0 v v o 0 0 o M m =K[R m t m ] M =K[R t ]

59 u s u o K 0 v v o u s 2 u 2 o s v u o v o u o ω KK 2 v v 2 o v o

60 Assumpton Fxed f= Known k= Constrants Constant K 5 0 / 33 = / 33 3 Image s m= rncpal pont known Aspect rato and skew constant 0 2 3= 23 = Zero Skew = known + Zero skew 0 3 2=0 3= 23 =0 3 mk + (m-)f >= 8

61 U D U L = U 3 D 3 U 3 L 3 = U 3 D 3 ½ D 3 ½ U 3 L 3 = L 3 = Q 3 Q3

62 Gven M Solve for L such that: M LM ~ M LM ( = 2,..m) Dagonalze L: L = UDU Approxmate by rank-3 matrx: L 3 = U 3 D 3 U 3 Compute Q 3 : Q 3 = U 3 D 3 ½ Compute q 4 by settng the orgn of the frst camera to 0: M q 4 = 0 Return Q = [Q 3 q 4 ]

63

64 Issues We know how to compute the camera geometry from 2 mages * Remanng ssues:. What about m > 2 mages? 2. rangulaton to compute pont n 3D 3. roectve reconstructon s not unque: How can we fnd the metrcally correct reconstructon? 4. How can we fnd the optmal reconstructon? *: (or 3, not shown n class)

65 Non-Lnear Refnement: Bundle Adustment

66 M p p M M2 p 2

67 E m u v, m 3 2 m m M,..,M m,.., n

68 E m u v, m 3 2 m m K,..,K m,r,..,r m,t,..,t m rncpal drectons of J J,.., n he drectons of uncertanty are the egenvectors of J J he magntudes of the uncertanty are the nverse egenvalues of J J Formally: he covarance matrx of the error on all the varables s = (J J) -

69 Image 3 Image 2 Image M M 2 M n 2 2n 3 3n

70 General case: Example J = J J = Features: A,B,C,D,E Images:,2,3

71 Example: n=42,453 m=285 N, Ka Steedly, Drew Dellaert, Frank, Out-of-Core Bundle Adustment for Large-Scale 3D Reconstructon,

72 E M,..,M m,.., n

73 (From D. Morrs)

74 Example from Uenn s group

75 Communty photo collectons Example: Query Notre Dame, ars on Flckr

76 mages + features d d Mn p F p Fp p, ' ', ) ' ( Mn Fp p + rank-2 SVD reducton Eppolar geometry: Fundamental matrx estmaton (mn. 2 mages + 7 correspondences) roectve reconstructon: ] [ b A M 0 b F F b A F Metrc reconstructon: Q t R K M Q ] [ Self-calbraton (ntrnsc parameter matrx K): K K M Q Q M 3 3 ROJECIVE MERIC Lnear eght-pont + RANSAC Non-lnear refnement , 3,,, u v Mn m m m m t R K REFINEMEN Bundle adustment: COMLEE SYSEM