EECS 442 Computer vson Multple vew geometry Perspectve Structure from Moton - Perspectve structure from moton prolem - mgutes - lgerc methods - Fctorzton methods - Bundle djustment - Self-clrton Redng: [HZ] Chpters:,8,9 [FP] Chpter: 3 Some sldes of ths lectures re courtesy of prof. S. Lzenk
Structure from moton prolem X j j M m j 2j M 2 M m From the mn correspondences j, estmte: m projecton mtrces M n 3D ponts X j moton structure
Structure from moton mguty - Poston mguty: t s mpossle sed on the mges lone to estmte the solute locton nd pose of the scene w.r.t. 3D world coordnte frme
Structure from moton mguty -Scle mguty: t s mpossle sed on the mges lone to estmte the solute scle of the scene (.e. house heght)
Structure from moton mguty -he scene s determned y the mges only up smlrty trnsformton (rotton, trnslton nd sclng) H s s s R t
Structure from moton mguty he mguty ests even for clrted cmers For clrted cmers, the smlrty mguty s the only mguty [Longuet-Hggns 8]
Structure from moton mguty In the generl cse (nothng s known) the mguty s epressed y n rtrry ffne or projectve trnsformton M j X j M R H X M H j j j M X j M H - H X j
Projectve mguty
ffne mguty
Structure from moton prolem X j j M m j 2j M 2 M m Gven m mges of n fed 3D ponts j = M X j, =,, m, j =,, n
Structure from moton prolem X j j M m j 2j M 2 M m m cmers M M m M 2 3 2 22 32 3 23 33 2
he Structure-from-Moton Prolem Gven m mges of n fed ponts X j we cn wrte M j X j Prolem: estmte the m 34 mtrces M nd the n postons X j from the mn correspondences j. Wth no clrton nfo, cmers nd ponts cn only e recovered up to 44 projectve Gven two cmers, how mny ponts re needed? How mny equtons nd how mny unknown? 2m n equtons n m+3n 5 unknowns So 7 ponts! [227 = 28; 2 + 37 5 = 28]
Structure-from-Moton lgorthms lgerc pproch (y fundmentl mtr) Fctorzton method (y SVD) Bundle djustment
lgerc pproch (2-vew cse) j M 2j M j X j M 2 pply projectve trnsformton H such tht: M H I M H Cnoncl perspectve cmers 2
X H X ~ X I X ~ ] [ ~ H M lgerc pproch (Fundmentl mtr) ~ ] [I X ~ X ~ X ~ ] [I X ~ X ~ X ~ ] [ ~ ] [ 3 2 3 2 X X X X ~ ] [ ~ H M 2 ) ( ) (? ) (
] [ z y y z y z Cross product s mtr multplcton
X H X ~ ) ( X I X ~ ] [ ~ H M lgerc pproch (Fundmentl mtr) ~ ] [I X ~ X ~ X ~ ] [I X ~ X ~ X ~ ] [ ~ ] [ 3 2 3 2 X X X ~ ] [ H M 2? ] [ F ] [ s ths fmlr? F ) ( ) (
Eppolr Constrnt [lecture 6] X 2 e e 2 O O 2 F 2 s the eppolr lne ssocted wth 2 (l = F 2 ) F s the eppolr lne ssocted wth (l 2 = F ) F s sngulr (rnk two) F e 2 = nd F e = F s 33 mtr; 7 DOF
lgerc pproch (Fundmentl mtr) F F [ ] F [ ] Snce F =, =e Compute s lest sq. soluton of F = det(f)=; = = [ ] - F = [ ] F M p I M p [e ]F e 2 Perspectve cmers re known
Structure-from-Moton lgorthms lgerc pproch (y fundmentl mtr) Fctorzton method (y SVD) Bundle djustment
Projectve fctorzton D z z z 2 m 2 m z z z 2 22 m2 2 22 m2 z z z n 2n mn n 2n mn M M M 2 m cmers (3 m 4) X X 2 ponts (4 n) X n D = MS hs rnk 4 If we knew the depths z, we could fctorze D to estmte M nd S If we knew M nd S, we could solve for z Soluton: tertve pproch (lternte etween ove two steps)
Structure-from-Moton lgorthms lgerc pproch (y fundmentl mtr) Fctorzton method (y SVD) Bundle djustment
Bundle djustment Non-lner method for refnng structure nd moton Mnmzng re-projecton error E(M, X) m n j D 2, X j M j X j? M X j j 3j P M 2 X j 2j M 3 X j P 2 P 3
Bundle djustment Non-lner method for refnng structure nd moton Mnmzng re-projecton error E(M, X) m n j dvntges Hndle lrge numer of vews Hndle mssng dt D 2, X j M Lmttons Lrge mnmzton prolem (prmeters grow wth numer of vews) requres good ntl condton j Used s the fnl step of SFM
Removng the mgutes: the Strtfed reconstructon up grde reconstructon from perspectve to ffne [y mesurng the plne t nfnty] up grde reconstructon from ffne to metrc [y mesurng the solute conc] Recoverng the metrc reconstructon from the perspectve one s clled self-clrton
Self-clrton Process of determnng ntrnsc cmer prmeters drectly from un-clrted mges Suppose we hve projectve reconstructon M,X } { j GOL: fnd rectfyng (non-sngulr) homogrphy H such tht { M H,H X j} M M M X H j s metrc reconstructon m M [R If world ref. system = cmer ref. system: [I ] M If the perspectve cmer s cnoncl: M [I ] ]
Self-clrton M M H M [I M [I ] ] [ ] [I ] H H v t k t We cn set k= (ths fes the scle of the reconstructon) H v
Plnes t nfnty (lecture 5) z y In the metrc (Euclden) world coordntes 2 plnes re prllel ff ther ntersectons s lne tht elongs to he projectve trnsformton of plne t nfnty cn e epressed s H p
Self-clrton H v v v H p v p
Self-clrton GOL: fnd rectfyng homogrphy H such tht { M,X j} { M H,H X j} s metrc reconstructon H p cot sn u v o o = clrton mtr of frst cmer [p ] = plne t nfnty n the projectve reconstructon 5 unknowns 3 unknowns
Self-clrton sc equton ] [ M = perspectve reconstructon of the cmer (known) p H ] [R M p R m 2 H M M R p p p R = metrc reconstructon of the cmer = rectfyng homogrphy (unknown)
Self-clrton sc equton p R p R I R R I p p p p?
solute conc ny [From lecture 5] stsfes: s C 2 2 2 4 Projectve trnsformton of 2 3 ( ) * Dul mge of the solute conc
) ( It s not functon of R, 6 5 4 5 3 2 4 2 symmetrc (5 unknowns) [From lecture 5] Propertes of
Self-clrton sc equton p p * * p p =2 m How mny unknowns? 3 from p 5 from How mny equtons? [ nd re known] [per vew] 5 ndependent equtons [per vew] rt of self-clrton: use constrnts on () to generte enough equtons on the unknowns
Self-clrton dentcl s * * p p * * p p For m vews, 5(m-) constrnts Numer of unknowns: 8 m>=3 provdes enough constrnts o solve the self-clrton prolem wth dentcl cmers we need t lest 3 vews
Propertes of [From lecture 5] ( ) 2 4 2 3 5 2 4 5 6. 2. zero-skew 3. 2 3 squre pel 4. 4 5 zero-offset
Self-clrton other constrnts * * p p zero-offset 4 5 2 m lner constrnts zero-skew etc 2 m lner constrnts
Self-clrton - summry Condton N. Vews Constnt nternl prmeters 3 spect rto nd skew known Focl length nd offset vry spect rto nd skew known Focl length nd offset vry 4* 5* skew =, ll other prmeters vry 8* Issue: the lrger s the numer of vew, the hrder s the correspondence prolem Bundle djustment helps!
SFM prolem - summry. Estmte structure nd moton up perspectve trnsformton. lgerc 2. fctorzton method 3. undle djustment 2. Convert from perspectve to metrc (self-clrton) 3. Bundle djustment ** or **. Bundle djustment wth self-clrton constrnts
Self-clrton - summry Constrnts on cmer moton cn e ncorported Lnerly trnsltng cmer Sngle s of rotton: turntle moton
pplctons Courtesy of Oford Vsul Geometry Group
pplctons D. Nstér, PhD thess
pplctons M. Pollefeys et l 98---
pplctons M. Brown nd D. G. Lowe. Unsupervsed 3D Oject Recognton nd Reconstructon n Unordered Dtsets. (3DIM25)
Photo synth Noh Snvely, Steven M. Setz, Rchrd Szelsk, "Photo toursm: Eplorng photo collectons n 3D," CM rnsctons on Grphcs (SIGGRPH Proceedngs),26,