AUTO-CALIBRATION. FACTORIZATION. STRUCTURE FROM MOTION.

Transcription

1 AUO-CALIBRAION. FACORIZAION. SRUCURE FRO OION.

2 hank you for the sldes. hey coe ostly fro the followng sources. arc ollefeys U. of North Carolna artal Hebert CU Slvo Savarese U. of chgan Dan Huttenlocher Cornell U.

3 otvaton Avod explct calbraton procedure Coplex procedure Need for calbraton obect Need to antan calbraton

4 Factorzaton Factorse observatons n structure of the scene and oton/calbraton of the caera Use all ponts n all ages at the sae te Affne factorsaton roectve factorsaton

5 Affne caera he affne proecton equatons are y x Z Y X y x 000 y x Z Y X y x ~ ~ 4 4 y x y x Z Y X y x y x

6 Orthographc factorzaton he ortographc proecton equatons are where n,...,,,...,, All equatons can be collected for all and where [ ] n n n n,...,,,, L O L L ~ ~ y x Z Y X,, y x Note that and are resp. x3 and 3xn atrces and therefore the rank of s at ost 3 (oas anade 9)

7 Orthographc factorzaton Factorze through sngular value decoposton An affne reconstructon s obtaned as follows V U Σ V U Σ ~, ~ (oas anade 9) [ ] n n n n,...,, n L O L L Closest rank-3 approxaton yelds LE!

8 A etrc reconstructon s obtaned - ~A - - A~ A a 3 3 atrx he atrx A can be obtaned by self-calbraton or knowng the age of absolute conc and dong Cholesky factorzaton A A ( 3 ω... 3 ) - or for,..., AA AA AA 0 Frst RANSAC (or slar) to elnate outlers. Not all the nler ponts are present all the fraes! If three ponts are present n four ages and the fourth one s only present n three ages, t can be recovered. ont are recovered one by one. Start fro the begnnng fro te to te.

9 Factorzaton Results 8

10 erspectve factorzaton he caera equatons for a fxed age can be wrtten n atrx for as where λ,,...,,,..., Λ [,,..., ], [,,..., ] Λ dag ( λ, λ,..., λ )

11 erspectve factorzaton All equatons can be collected for all as where Λ Λ Λ n n...,... In these forulas are known, but Λ, and are unknown Observe that s a product of a 3x4 atrx and a 4xn atrx,.e. t s a rank 4 atrx

12 Iteratve perspectve factorzaton When Λ are unknown the followng algorth can be used:. Set λ (affne approxaton).. Factorze and obtan an estate of and. If σ 5 s suffcently sall then SO. 3. Use, and to estate Λ fro the caera equatons (lnearly) Λ 4. Goto. In general the algorth nzes the proxty easure (Λ,,)σ 5 Note that structure and oton recovered up to an arbtrary proectve transforaton

13 Noralzng depths. α and β X does not change α β λ x. -th colun and/or -th row can brng λ close to one. Noralzng the age coordnates. o centrod values 0 and average dstance \sqrt{}. he λ s vald at frst teraton f the rato of true depths n 3D s approxately constant through the sequence! Bundle adustent. he fnal step! Needs good ntalzaton and has any paraeter nvolved Levenberg-arquardt (3n ) atrx factored possble wthout sparse ethod.

14 Further Factorzaton work Factorzaton wth uncertanty (Iran & Anandan, IJCV 0) Factorzaton for dynac scenes (Costera and anade 94) (Bregler et al. 000, Brand 00)

15 Extensons araperspectve [oelan & anade, AI 97] Sequental Factorzaton [orta & anade, AI 97] Factorzaton under perspectve [Chrsty & Horaud, AI 96] [Stur & rggs, ECCV 96] Factorzaton wth Uncertanty [Anandan & Iran, IJCV 00] 9

16 Herarchcal structure and oton recovery Copute -vew Copute 3-vew Sttch 3-vew reconstructons erge and refne reconstructon F H

17 Deternng close vews If vewponts are close then ost age changes can be odelled through a planar hoography Qualtatve dstance easure s obtaned by lookng at the resdual error on the best possble planar hoography Dstance n edan D( H, )

18 Coputaton of ntal structure and oton accordng to Hartley and Zsseran ths area s stll to soe extend a black-art All features not vsble n all ages No drect ethod Buld partal reconstructons and asseble (ore vews s ore stable, but less corresp.) ) Sequental structure and oton recovery ) Herarchcal structure and oton recovery

19 Structure fro oton: Ltatons Very dffcult to relably estate etrc structure and oton unless: Large (x or y) rotaton, or Large feld of vew and depth varaton Caera calbraton portant for Eucldean reconstructons Need good feature tracker 7

20 Constrants? Scene constrants arallells, vanshng ponts, horzon,... Dstances, postons, angles,... Unknown scene no constrants Caera extrnscs constrants ose, orentaton,... Unknown caera oton no constrants Caera ntrnscs constrants Focal length, prncpal pont, aspect rato & skew erspectve caera odel too general soe constrants

21 Self-calbraton Upgrade fro proectve structure to etrc structure usng constrants on ntrnsc caera paraeters Constant ntrnscs (Faugeras et al. ECCV 9, Hartley 93, rggs 97, ollefeys et al. AI 98,...) Soe known ntrnscs, others varyng (Heyden&Astro CVR 97, ollefeys et al. ICCV 98,...) Constrants on ntrncs and restrcted oton (e.g. pure translaton, pure rotaton, planar oton) (oons et al. 94, Hartley 94, Arstrong ECCV 96,...)

22 Stratfcaton of geoetry roectve Affne etrc 5 DOF DOF plane at nfnty parallels 7 DOF absolute conc angles, rel.dst. ore general ore structure

23 A countng arguent o go fro proectve (5DOF) to etrc (7DOF) at least 8 constrants are needed nal sequence length should satsfy (# known) + ( n ) (# ) 8 n fxed Independent of algorth Assues general oton (.e. not crtcal)

24 Iages n Features Assue correspondences p p p roectve SF heore: Reconstructon s possble as long as n > +3n 5 roectve SF heore Gven ages and n features Each pont s represented by ts hoogeneous coordnates [X Y Z ] Each feature s represented by ts hoogeneous coordnates n the age plane p [u v ] Each age s represented by ts 3x4 proecton atrx [A b ] (A s a 3x3 atrx, b s a 3 vector) For each feature, we have: p ~ (~ s the sae as the trple-bar hoogeneous equalty: left hand sde proportonal to rght hand sde). ey results:. he unknowns and can be recovered only up to a 4x4 proectve transforaton Q. hat s, for any 4x4 Q, ( Q,Q - ) yelds the sae age proectons as (, ).. he unknowns and can be recovered f n > +3n 5 3. In partcular, for caeras, at least 7 ponts are needed. 3

25 α u 0 s 0 uo v 0 αv o [Id 0] [R t ] [R t ] etrc Upgrade: he proectve reconstructon gves us a set of 3x4 proecton atrces for each caera,..,. he next proble s to convert ths proectve reconstructon to a etrc reconstructon. Specfcally, we want to fnd a 4x4 atrx Q such that: Q [R t ] R and t are the rotaton/translaton between the coordnate syste of caera and an arbtrary coordnate syste. s the atrx of ntrnsc paraeters of caera, whch s defned as: α u 0 s α 0 u α α cotθ o β vo 0 snθ 0 0 α x and α y are the scales n the x and y drectons, x o and y o are the coordnates of the center, and s s the skew of the caera (s 0 f the axes are orthogonal.) uo v 0 v o 0

26 α u 0 s 0 uo v 0 αv o [R t ] [R t ] Fundaental ransforaton: Our fundaental equaton s: Q [R t ] Denotng the atrx fored by takng the frst 3 coluns of Q by Q 3, such that Q [Q 3 q 4 ], we have: Q 3 R. akng the frst 3 coluns and observng that R s a rotaton atrx: Q 3 Q 3 hs s the key observaton: By wrtng that the frst three coluns of the product of by Q s a rotaton, we are able to elnate the rotaton fro the unknowns. All that s left are the atrces of nternal paraeters for each of the ages. hs process s soetes called auto-calbraton snce t aounts to calbratng the nternal paraeters of the caeras drectly fro ages. It s portant to understand the nuber of degrees of freedo n Q 3. he total nuber of entres n Q 3 s 4x3. he atrx s defned up to scale snce all the equaltes are hoogeneous. oreover, the atrx s defned up to a rotaton snce for any arbtrary rotaton R: Q 3 RR Q 3 Q 3 Q 3 so that f Q 3 s a soluton, so s Q 3 R. hese addtonal degrees of freedo sply reflect the fact that one can choose the orentaton and scale of the global coordnate syste arbtrarly. herefore, Q 3 s characterzed by 3 8 unknowns.

27 Look only at the frst 3 coluns Q Q 3 3 Q Q Basc trck: 3 [ R t ] R R R [R t ] Use the fact that R s a rotaton [R t ] For convenence, we denote the atrx the atrx Q 3 Q 3 by L (a 4x4 atrx) and L by ω. he set of equatons to solve s: L,.., Each age generates 5 ndependent equatons (the left hand sde s a 3x3 syetrc atrx, but the equalty s up to scale). he total nuber of unknowns s 8 (Q 3 ) + 5 ( ). herefore, the nuber of equatons (5) s always lower than the nuber of unknowns (8 + 5) and we can never solve ths syste of equatons wthout soe constrants on the caeras. he key queston s what constrants can be used. A couple of constrants are nvestgated below, followed wth a general result. Case : Identcal Intrnsc araeters: Let us suppose now that we do not know the ntrnsc paraeters of the caeras, but that we do know that they are all dentcal, that s, for all caeras. For all the caeras, we have: Where ω s coputed fro the frst age: L ω,.., (wth the equalty up to scale) L L,.., hs gves us 5(-) ndependent equatons for 8 unknowns (n Q 3 ). herefore, we can solve the reconstructon proble n ths case f: 5(-) > 8 - > 3

28 α u 0 s 0 uo v 0 αv o [R t ] [R t ] Note also that we solved for 8 unknowns for Q 3, even though t s a 4x3 atrx eleents. he dfference s due to the fact that the reconstructon s defned up to a global rotaton, n other words, Q 3 R for any rotaton R s also a soluton. We can verfy the paraeter count: Q 3 Q 3 Arbtrary rotaton Arbtrary scale q 4 It s portant to note that by anpulatng the equatons so that the unknown becoes L, we have effectvely elnated the rotaton and translaton and reduced the proble to the recovery of the ntrnsc paraeters. hs step s often tered self-calbraton. 3

29 o v o u v u s α α o o v o o o v o u v v u v u s u s α α α ω Case : rncpal pont at orgn In that case, u o v o 0, whch ples that ω 3 ω 3 0. herefore, gong back to the orgnal equaton, we can wrte that: ( L ) 3 0 and ( L ) 3 0 hose are two equatons n L that are ndependent of. We have such equatons for vews for 8 unknowns n L (eanng, 8 unknowns n Q 3 ). hese equatons are ndependent of. herefore: If the prncpal pont s at the orgn, a etrc reconstructon can be obtaned fro a nu of 4 vews. > 4

30 Assupton Fxed f nown k Constrants Iage s Constant 5 0 ω /ω 33 ω /ω 33 3 rncpal pont known 0 ω 3 ω Aspect rato and skew constant 0 5 Zero Skew 0 ω ω 33 ω 3 ω known + Zero skew 0 3 ω 0 ω 3 ω k + (-)f > 8 Case 3: Zero-Skew If the skew s zero but all the other paraeters are allowed to vary, then we have the constrant: ω ω 33 ω 3 ω 3 hs provdes constrants. herefore, we ust have >8, thus 8 ages are necessary. General countng arguent: he total nuber of paraeters to be estated fro the set of equatons above s 8: L Q 3 Q 3 s a 4x4 syetrc atrx that s defned by 0 entres, 9 of whch are ndependent because of the scale factor. here s one ore constrant that det(l) 0 (t s of rank 3). hus the nuber of ndependent paraeters s If we know k nternal paraeters, then we have k constrants - If we know that f nternal paraeters are fxed (but unknown) we have f(-) constrants - herefore, we can recover a etrc reconstructon ff: k + (-)f > 8 5

31 U D U L U 3 D 3 U 3 L 3 U 3 D 3 ½ D 3 ½ U 3 L 3 L 3 Q 3 Q3 Recoverng Q: One proble n the prevous result s that, whle we used the atrx L for convenence, the actual unknown s Q 3 wth L Q 3 Q 3, whch s a non-lnear relaton. hs can be nconvenent because we have to solve a set of non-lnear equatons. In fact, t s possble n any cases to solve the proble lnearly. hs s done by usng the sae trck as before wth F and E, and wth the factorzaton ethod: Let s pretend frst that we solve the equatons n L, whch has 9 degrees of freedo (4x4 syetrc atrx 0, but t s up to scale), and then decopose L nto L Q 3 Q 3. For exaple, n case (rncpal pont at orgn), u o v o 0, whch ples that ω 3 ω 3 0. herefore: ( L ) 3 0 and ( L ) 3 0 hose are two equatons that are lnear n L and can be solved very easly!! Once we have L, we need to decopose t back nto L Q 3 Q 3. here s a practcal dffculty here: Snce Q 3 s a 4x3 atrx, for such a decoposton to exst, L ust be of rank at ost 3 whch s not enforced n the lnear soluton. We can fnd the closest atrx L 3 of rank three as follows: L s a syetrc atrx so L UDU, where D s a 4x4 dagonal atrx of egenvalues and U s a 4x4 rotaton atrx. he atrx L 3 of rank 3 that s closest to L can be fored by elnatng the sallest egenvalue of L, that s, L 3 U 3 D 3 U 3, where U 3 s the 4x3 atrx obtaned by reovng the last colun of Q and D 3 s the 3x3 upper rght block of D. Wth ths decoposton, the atrx Q 3 U 3 D 3 ½ s a soluton snce L 3 Q 3 Q 3. can be recovered by Cholesky decoposton ω. 6

32 Gven Solve for L such that: L ~ L (,..) Dagonalze L: L UDU Approxate by rank-3 atrx: L 3 U 3 D 3 U 3 Copute Q 3 : Q 3 U 3 D 3 ½ Copute q 4 by settng the orgn of the frst caera to 0: q 4 0 Return Q [Q 3 q 4 ] Gven Q 3, the last colun of Q s coputed by settng the orgn at the orgn of the frst caera, that s: q 4 0. Note that any scaled verson of q 4 s a soluton. hs s a consequence of the fact that t s not possble to recover the absolute scale of the translaton between the caeras. Reeber that we solved for 8 unknowns for Q 3, even though t s a 4x3 atrx eleents. he dfference s due to the fact that the reconstructon s defned up to a global rotaton, n other words, Q 3 R for any rotaton R s also a soluton. We can verfy the paraeter count: Q 3 Q 3 Arbtrary Arbtrary q rotaton 4 scale It s portant to note that by anpulatng the equatons so that the unknown becoes L, we have effectvely elnated the rotaton and translaton and reduced the proble to the recovery of the ntrnsc paraeters. hs step s often tered auto-calbraton. 7

33 p p p Non-Lnear Approach: Bundle Adustent (proectve case) he dscusson so far has assued a lnear reconstructon that (plctly) nzes the error λ p. hs error s not the true geoetrc error,.e., the error between a feature poston and the proecton of a reconstructed pont. Suppose that we have estated the proecton atrces (..) usng the prevous (lnear) technques. Suppose also that we a set of correspondng ponts n the ages p (..n) correspondng to n ponts n the scene. For each..n, we assue that we have an ntal reconstructon of the correspondng scene pont. Ideally, the data pont p [u v ] should be dentcal to the proecton of the reconstructed pont. Startng wth an ntal estate of the proecton atrces and the scene ponts, we want to nze the geoetrc dstance between p and : u + v 3 Where,, 3 are the rows of. Sung over all the ponts and all the ages, we have to fnd the nu of: E u + v, 3 3 Over all the and (a total of + 3n varables). hs technque s called bundle adustent, wdely used n photograetry. 0

34 E u + v, 3 3,..,,.., n nzaton algorth: Let X be the vector fored by concatenatng all the N + 3n unknowns of the proble, the atrces and the ponts. he error functon can be wrtten as: E(X) f(x) f ( X) wth [ ε ε... ε... ε ] A basc pleentaton of bundle adustent uses an teratve Gauss-Newton algorth for solvng the non-lnear least-squares: If X k s the value at teraton k, a frst order approxaton gves us: f(x k + X) f(x k ) + J k X Where J k s the Jacoban (n rows by N coluns atrx of dervatves) of f. We want to fnd X that nzes the rght-hand sde of ths frst order equaton. he soluton s found usng the standard least-squares soluton: X (J k J k ) - J k f(x k ) n wth X s added to the current estate X k to yeld the next estate X k+. Convergence ssues: he algorth assues that the startng pont X o s close to the nu. he algorth ay converge to a local nu or even dverge otherwse. ypcally, X o s obtaned fro one of the prevous lnear technques. ε u 3 v 3

35 E u + v, 3 3,..,,R,..,R,t,..,t rncpal drectons of J J,.., n Fnal Adusteent (etrc case): Once we have the ntrnsc paraeters through self-calbraton, we recover the perspectve proecton atrces by transforng the ones obtaned fro the proectve reconstructon as: Q. Slarly, the reconstructed ponts are transfored by the nverse transforaton: Q -. hs gves an ntal estate of the s. In general, reconstructon systes nclude a last step n whch all the atrces and ponts are adusted sultaneously usng bundle adustent. hs s the sae bundle adustent obectve functon as before, except that the nzaton s done explctly wth respect to the coponents, R, and t nstead of wth respect to the s snce we need to enforce a etrc reconstructon.,,, n u + v R t, 3 3 It s also portant to know the uncertanty on the resultng reconstructon: onts far away are ore uncertan, for exaple. If we assue that all the errors are Gaussan, the atrx J J s the covarance atrx representng the error dstrbuton of all the unknown paraeters. he prncpal drecton of J J are the an drectons of uncertanty, the egenvalues are the uncertantes n those drectons. he atrx can be nterpreted as characterzng the curvature of the error surface n dfferent drectons.

36 3 3 ε ε n ε ε n ε 3 Iage 3 Iage Iage ε 3n Coputatonal ssues: Bundle adustent ay nvolve hundreds of varables. As a result, the coputaton of J k J k and ts nverson ay be expensve and nuercally unstable. An portant fact to note s that the dervatves of ε wth respect to l for l / s zero. Slarly, the dervatve of ε wth respect to l s zero for any l other than l. herefore, J k J k s very sparse and ths property can be exploted to speed up the teratons. 3

37 E,..,,.., n -Gauge ssues: he reconstructon s defned up to a slarty transforaton. hat s, one can transfor the entre set of by an arbtrary transforaton and transfor the proecton atrces accordngly to get a copletely equvalent reconstructon. hs causes a nuercal proble because the nu of the surface E E(X) s not a pont but t s a valley of equvalent na correspondng to dfferent transforatons. It s therefore portant to ether fx the transforaton, or to odfy the functon so that the nzaton s nvarant to transforatons. echncally, ths the gauge ssue (choosng a partcular transforaton s choosng a gauge.) A sple way to do ths, for exaple, s to fx the proecton atrx of the frst caera and to express everythng wth respect to the frst caera. Another possblty s to fx a set of scene ponts (a bass). It s portant to note that, for nuercal reasons, the result ay be dfferent dependng on the selectng gauge. - Outlers: hs has assued so far that all the correspondences are correct. In fact, f soe correspondences are ncorrect (outlers), the entre functon E s corrupted. o avod ths, a dfferent functon s used n place of the squared dstance between features and scene proectons. hs s the obect of robust estaton, to be dscussed later. 4

38 (Fro D. orrs) Uncertanty: he uncertanty on each pont and the caeras orentaton R and poston t can be recovered by proectng on the approprate subspaces of paraeters (techncally, takng the argnal of the dstrbutons). he dstrbuton of uncertanty depends strongly on the gauge constrant: Obvously, the uncertanty s 0 around the caera that s chosen as reference. hs s the sae stuaton n oble robotcs n whch one gets a dfferent dstrbuton of uncertanty f the ap s expressed wth respect to the startng poston of the robot, or wth respect to the current poston. 5

39 6 ages + features ( ) ( ) d d n p F p Fp p, ' ', + ) ' ( n Fp p + rank- SVD reducton Eppolar geoetry: Fundaental atrx estaton (n. ages + 7 correspondences) roectve reconstructon: [ ] ] [ b A 0 b F F b A F etrc reconstructon: Q t R Q ] [ Self-calbraton (ntrnsc paraeter atrx ): Q Q 3 3 ROJECIVE ERIC Lnear eght-pont + RANSAC Non-lnear refneent 3, 3,,, + u v n t R REFINEEN Bundle adustent: COLEE SYSE

40 Structure fro oton proble X x x x Fro the xn correspondences x, estate: proecton atrces n 3D ponts X oton structure

41 he Structure-fro-oton roble Gven ages of n fxed ponts X we can wrte x X roble: estate the 3 4 atrces and the n postons X fro the n correspondences x. Wth no calbraton nfo, caeras and ponts can only be recovered up to a 4x4 proectve Gven two caeras, how any ponts are needed? How any equatons and how any unknown? n equatons n +3n 5 unknowns So 7 ponts! [xx7 8; x + 3x7 5 8]

42 Bundle adustent Non-lnear ethod for refnng structure and oton nzng re-proecton error E(, X) n ( ) x, D X X? X x x 3 X x 3 X 3

43 Bundle adustent Non-lnear ethod for refnng structure and oton nzng re-proecton error E(, X) n Advantages Handle large nuber of vews Handle ssng data ( ) x, D X Ltatons Large nzaton proble (paraeters grow wth nuber of vews) requres good ntal condton Used as the fnal step of SF

44 Reovng the abgutes: the Stratfed reconstructon up grade reconstructon fro perspectve to affne [by easurng the plane at nfnty] up grade reconstructon fro affne to etrc [by easurng the absolute conc] Recoverng the etrc reconstructon fro the perspectve one s called self-calbraton

45 Self-calbraton rocess of deternng ntrnsc caera paraeters drectly fro un-calbrated ages Suppose we have a proectve reconstructon, X } { GOAL: fnd a rectfyng hoography H such that { H, H X } X H s a etrc reconstructon L [R ] If world ref. syste caera ref. syste: [I 0] If the perspectve caera s canoncal: [I 0]

46 Self-calbraton basc equaton ] a [A p 0 H perspectve reconstructon of the caera ] [R ( ) p a A R H L [ ] [ ] a A R p 0 [ ] a p a A ( ) p a A R

47 Self-calbraton basc equaton ( ) p a A R ( ) p a A R I R R ( ) I p a A ( ) p a A ( ) p a A ( ) p A a?

48 Absolute conc Any x Ω [Fro lecture 5] satsfes: Ω s a C Π Ω 0 Ω x + x + x 4 0 x x roectve transforaton of x 0 3 Ω 0 ω ( ) ω* Dual age of the absolute conc

49 Self-calbraton basc equaton ( A ) a p ( ) A a p ( ) *( ) * A a p ω A a p ω How any unknowns? 3 fro p 5 fro ω [per vew] How any equatons? Art of self-calbraton: [A and a are known] 5 ndependent equatons [per vew] use constrants on ω () to generate enough equatons on the unknowns

50 Self-calbraton dentcal s ( ) *( ) * A a p ω A a p ω ( ) *( ) * A a p ω A a p ω For vews, 5(-) constrants Nuber of unknowns: 8 >3 provdes enough constrants o solve the self-calbraton proble wth dentcal caeras we need at least 3 vews