Lecture 4 Single View Metrology. Lecture 4 Single View Metrology = $! $ $ " World ref. system. Calibration Problem

Transcription

1 Lecture 4 Singe View Metroog Professor Siio Srese Comuttion Vision nd Geometr L Siio Srese Lecture 4-6-Jn-5 Lecture 4 Singe View Metroog Reiew cirtion nd D trnsformtions Vnishing oints nd ines Estimting geometr from singe imge Etensions Reding: [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter 8 More Singe View Geometr [Hoiem Srese] Chter Siio Srese Lecture 4-6-Jn-5 Cirtion Proem he cirtion roem ws discussed in detis during ecture. j C ui Pi M P i i i Word ref. sstem In ies M K[ R ] α K α cotθ β sinθ uo o

2 Cirtion Proem j C ui Pi M P i i i Word ref. sstem In ies M K[ R ] unknown Need t est 6 corresondences Once the cmer is cirted... Pinhoe ersectie rojection P Line of sight Once the cmer is cirted (intrinsics re known) nd the trnsformtion from the word reference sstem to the cmer reference sstem (which ccounts for the etrinsics) is so known, cn we estimte the oction of oint P in D from its osertion? he gener nswer to this question is no. his is ecuse, gien the osertion, een when the cmer intrinsics nd etrinsic re known, the on thing we cn s is tht the oint P is octed somewhere ong the ine defined C nd. his ine is ced the ine of sight. O w M K[ R ] C -Intern rmeters K re known -R, re known ut these cn on rete C to the cirtion rig Cn I estimte P from the mesurement from singe imge? No - in gener (P cn e nwhere ong the ine defined C nd ) Recoering structure from singe iew Pinhoe ersectie rojection he ctu oction of P ong this ine is unknown nd cnnot e determined from the osertion one. P Line of sight O w C unknown known Known/ Prti known/ unknown

3 Recoering structure from singe iew In the reminder of this ecture, we wi introduce toos nd techniques for erring the geometr of the cmer nd the D enironment from just one imge of such enironment. htt:// rnsformtion in D Before we go into the detis of tht, et me rec some of the concets ou he red eored when we tked out trnsformtions. here re four imortnt trnsformtions in D. A these trnsformtions wi e descried in homogenous coordintes. -Isometries -Simirities -Affinit -Projectie rnsformtion in D he first one is ced isometric trnsformtion which is in gener the conctention of rottion nd trnstion trnsformtion nd eressed the mtri H e nd Eq. 4. his trnsformtion reseres the distnce etween n ir of oints nd hs degrees of freedom ( for trnstion, for rottion). his grou ctures the motion of rigid oject. Isometries: [Euciden] R t H e [Eq. 4] - Presere distnce (res) - DOF - Regute motion of rigid oject

4 Simirities: S s s rnsformtion in D SR t H s [Eq. 5] he second grou of trnsformtions is ced simirit trnsformtion. he trnsformtion is conctention of trnstion, rottion nd sce trnstion trnsformtions. It reseres the rtio of engths etween n two ine segments efore nd fter trnsformtion. It so reseres the nge etween n intersecting ines. he grou hs 4 degrees of freedom ( trnstion, rottion, sce) - Presere - rtio of engths - nges -4 DOF Affinities: rnsformtion in D A t H [Eq. 6] s A R( θ ) R( φ ) D R( φ) D s [Eq. 7] he third trnsformtion grou is the ffine trnsformtion. his grou cn e interreted s series of trnstion, rottion nd nisotroic scing trnsformtions. he core of the ffine trnsformtion is chrcterized the mtri A which incudes rottion hi, n nisotroic scing [s ; s], n inerse rottion hi nd n ritrr rottion thet (see Eq. 7). his trnsformtion hs so mthemtic interrettion: An ositie definite mtri cn e decomosed into two orthogon mtrices nd digon mtri Singur Vue Decomosition (this is ssuming ositie scing; for negtie scing, the trnsformtion m not e unique.) hus we cn eress A UDV (UV ) (VDV ) R(thet) R(-hi) D R(hi) recing UV R(thet), V R(hi), nd where D is digon mtri with the singur ues. Note tht trnsose of rottion mtri is the inerse of the rottion mtri. hus, n ritrr mtri cn e decomosed into R(thet), R(hi), D. rnsformtion in D his trnsformtion grou reseres re ines (re ines re sti re fter trnsformtion). he rtio of engths on coiner ines is resered fter trnsformtion nd thus it foows tht the rtio of res within n ritrr shes is resered. Affinities: A t H [Eq. 6] he trnsformtion hs 6 degree of freedom ( 4 eements in the mtri A, for trnstion ). s A R( θ ) R( φ ) D R( φ) D s -Presere: - Pre ines - Rtio of res - Rtio of engths on coiner ines - others - 6 DOF [Eq. 7]

5 rnsformtion in D Fin, the fourth trnsformtion is the rojectie trnsformtion which is generiztion of the ffine trnsformtion where two ddition eements (the ector ) re non-zero. his trnsformtion hs 8 degrees of freedom nd reseres the cross rtio of 4 coiner oints. Projectie: A t H [Eq. 8] - 8 DOF - Presere: - cross rtio of 4 coiner oints - coinerit - nd few others he cross rtio An eme of cross rtio is defined in Eq.9 for the oints P,P,P,P 4. he cross-rtio of 4 coiner oints P P [Eq. 9] P P 4 P P P P P P 4 4 P P X i Yi P i Z i P P Cn ermute the oint ordering P P P P 4 4 P P Lecture 4 Singe View Metroog In the net sides we introduce numer of imortnt definitions out ines nd oints in D nd D nd introduce the concets of nishing oints nd ines. Reiew cirtion nd D trnsformtions Vnishing oints nd ines Estimting geometr from singe imge Etensions Reding: [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter 8 More Singe View Geometr [Hoiem Srese] Chter Siio Srese Lecture 4-6-Jn-5

6 Lines in D ne -/ -c/ + + c c If [, ] c [Eq. ] A ine in D cn e reresented s the D ector [ c] in homogeneous coordintes; he rtio -/ ctures the soe of the ine nd the rtio -c/ defines the oint of intersection of the ine with the is. If oint eongs to ine, thn the dot roduct etween nd is equ to zero (Eq ). his equtions so defines ine in D. Intersecting ines Lines in D ne [Eq. ] Proof ( ) ( ) [Eq. ] [Eq. ] In gener, two ines nd intersect t oint. his oint is defined s the the cross roduct etween nd [Eq. ]. his is es to erif s the side shows. Proof: Gien two intersecting ines nd, the intersection oint shoud ie on oth ines nd ine ; thus the oint is the intersection if nd on if [Eq.] nd [Eq.]. Let e. hen, the ector is erendicur to the ector nd the ector nd, thus, it stisfies the oe constrints. Since the intersection is unique (set ritrr \ nd show tht is ), is the oint of intersection of the two ines. is the intersecting oint D Points t init (ide oints), Let s intersect two re ines: ) In Eucidin coordintes this oint is t init Agree with the gener ide of two ines intersecting t init / / c c Eq.] Let now us comute the oint of intersection of two re ines. We strt osering tht oint t init in Euciden coordintes corresonds to oint in homogenous coordintes whose third coordinte is equ to zero. Let us now consider two re ines nd. When two ines re re, their soe is equ nd thus /- /. Let s now comute the oint of intersection of these two ines. Using Eq. we otin Eq. which is ect the eression of oint t init (in homogenous coordintes). his confirms the intuition tht two re ines intersect t init. he oint of intersection of two re ines returns oint t init which is so ced ide oint.

7 D Points t init (ide oints), c c [ ] c Note: the ine [ c] ss trough the ide oint So does the ine since / / [Eq. 5] One interesting roert of oint t init is tht the re ines with the sme soe -/ sses through the oint [ - ] [Eq.5]. Lines init Set of ide oints ies on ine ced the ine t init. How does it ook ike? Indeed: A ine t init cn e thought of the set of directions of ines in the ne We cn further etend this concet nd define the ines t init. Consider or more irs of re ines (right side of the side). Ech ir of re ines intersect into oint t init. Let us c these set of oints,,. he ine tht sses through these oints t init must stisf,, etc nd is sim [ c]. Since c in n ritrr ue, we cn sim write [,,]. A ine t init cn e thought of the set of directions of ines in the ne Projectie trnsformtion of oint t init H t A H? H z t A is it oint t init? no? H A t A An ffine trnsformtion of oint t init is sti oint t init [Eq. 7] [Eq. 8] Now, et s see wht hens if we generic rojectie trnsformtion H to oint t init (Eq. 7). Notice tht the st eement H ecomes non-zero which suggests tht rojectie trnsformtion in gener ms oints t init to oints tht re no onger t init. his is not true for ffine trnsformtions [Eq.8]. If we n ffine trnsformtion H A to wi sti otin oint t init.

8 Projectie trnsformtion of ine (in D) A t H H [Eq. 9] H? [Eq. ] H A? [Eq. ] A A t is it ine t init? t t t A t A no his time, et s rojectie trnsformtion H to ine. he rojectie trnsformtion of ine is H - (Eq 9). Let s derie this eqution. A oints tht ss through ine must stisf the ine eqution: ; > H H - Since H (nd H ), then H - ; Becuse, fter the trnsformtion, rojected oint must sti eong to the rojected ine ( ), it imies tht H -. Let s now the rojectie trnsformtion H to ine t init. Is the rojected ine sti t init? No. Let s now the ffine trnsformtion H to ine t init. Is the A rojected ine sti t init? Yes, s the derition net to Eq shows. Points nd nes in D Π Π [Eq. ] Π c d z + + cz + d How out ines in D? Lines he 4 degrees of freedom - hrd to reresent in D-sce Cn e defined s intersection of nes [Eq. ] So fr we he introduced the concets of ines nd oints t init in D. Let s introduce the equient concets in D (nd in the corresonding homogenous coordintes). Points in D (homogenous coordintes) re denoted s nd their corresonding rojectie trnsformtions in the D imge re denoted s. Foowing the D cse, we cn reresent ne s norm ector (,,c) nd distnce from the origin (d) which is [Eq.]. hus, oint is on ne if nd on if [Eq.] hods. In D, it is trick to reresent ine ut one cn reresent it using n intersection of nes. See [HZ] Ch... for deti. Phiosohic interude C. Cho (course ssistnt): When ou see n oject, ou re not erceiing D sce (there is no w to sense sce direct), ou re mere seeing the D ersectie rojection of the oject on our retin. hus seeing is equient to ersectie rojection. So it is essenti to mode the word s is in D sce nd ersectie rojection s we do. Points t init in D Points where re ines intersect in D In D, simir to ides oints in D, oints t init re defined s the oint of intersection of re ines in D. word oint t init Pre ines

9 Vnishing oints he rojectie rojection of oint t init into the imge ne defines nishing oint. Simir to the D cse, ing the rojectie trnsformtion M to oint t init we otin oint in the imge ne which is no onger t init. Interesting, simir to the D cse, the direction of the (re) ines in D ssocited to is gien the coordintes, nd of word oint t init Pre ines M direction of the ine in D Vnishing oints nd directions Net we derie usefu retionshi etween re ines in D, the corresonding nishing oint in the imge nd the cmer rmeters. d direction of the ine [,, c] K d K d K [Eq. 4] [Eq. 5] d C Let s define s d[,,c] the direction of set of re ines in D in the cmer reference sstem. hese ines intersect to oint t init nd the rojection of such oint in the imge returns the nishing oint. It s es to roe tht is reted to d i Eq. 4, where K is the cmer mtri. Equient, d cn e eressed s function of Eq. 5, where the diision K - gurntees tht d hs unit norm (d is direction). he roof of Eq 4 is reorted in the ottom rt of the side. X Proof: c M M X K[ I ] c K c Vnishing (horizon) ine π If we consider ne π s suerset of set of re ines, ech set of re ines intersects t oint t init. he ine tht sses through such set of oints t init is the ine t init ssocited to π. A ine t init is so defined s the ine where two re nes intersect (in gener, the intersection of two nes in D is ine). he rojectie trnsformtion of to the imge ne is no onger ine t init nd is ced the nishing ine or horizon ine horiz (see Eq. 6). he horizon ine is ine tht sses through the corresonding nishing oints in the imge. horizon Projectie trnsformtion M Imge hor H [Eq. 6] P

10 Are these two ines re or not? he concet of horizon ine ows to nswer interesting questions out imges. One is the foowing: Are the two dshed ines in the imge re or not? In order to nswer this question we cn ssume tht n orce tes us tht the horizon ine is the ornge one. hen, we cn erif whether the dshed ines meet t the horizon or not. If es, these ines re re in D. Recognition hes reconstruction Humns he ernt this his is n nice eme which shows tht recognition cn he soe reconstruction roem. If we recognize the horizon ine, we cn er roerties out the word (erif the roert tht two ines re re or not). Humns he ernt mn of these roerties from our di eerience nd we use those for soing imortnt estimtion roems. - Recognize the horizon ine - Mesure if the ines meet t the horizon - if es, these ines re // in D Vnishing oints nd nes It is es to show tht the norm n of π nd re reted Eq. 7 horiz (see sec [HZ] for detis), where K is cmer mtri. n π Agin, Eq. 7 cn e usefu for estimting roerties of the word. If we recognize the horizon nd our cmer is cirted (K is known), we cn estimte the orienttion of the ground ne. horiz C n K horiz [Eq. 7] Pnes t init Before introducing the st roert tht retes nishing oints nd ines, we define the ne t init π. Π z Π ne t init A set of or more nishing ines (ue ines in the figure) defines the ne t init π (eow ne in the figure). he ne t init is descried [ ] in homogenous coordintes Pre nes intersect t init in common ine the ine t init A set of or more ines t init defines the ne t init Π

11 Ange etween nishing oints θ d d C he st roert we introduce (which we don t roe either) is eressed Eq. 8. Suose tht two irs of re ines he directions d nd, nd re ssocited to the oints t init nd, resectie. Let d nd e the corresonding nishing oints. Eq. 8 retes the nge etween d nd d nd the nishing oints nd. We define the mtri (K K ) - s ω. One seci cse which is usefu in rctice is when the two set of re ines re orthogon to ech other. In this cse, d nd d re orthogon, which gies us [Eq. 9]. Notice tht this is scr eqution. cosθ [Eq. 8] ω ω ω ω (K K ) If θ 9 ω Scr eqution [Eq. 9] Projectie trnsformtion of ω M Ω (K K ) M. It is not function of R, [Eq. ] ω ω ω4. ω ω ω ω5 ω 4 ω5 ω6 ω ω. zero-skew 4. ω ω Ω Asoute conic M K R smmetric nd known u sce squre ie he mtri ω hs seci geometric mening in tht it is the rojectie trnsformtion of the soute conic Ω in the imge. his retionshi is eressed [Eq. ]. Wht s the soute conic Ω? It s conic tht ies in the ne t init [ ]. Note tht in gener, the rojectie trnsformtion of qudric or conic Ω is eressed s M - Ω M - (see HZ ge 7, eq..6). he mtri ω stisfies other interesting roerties: - It deends on on the intern mtri nd not on the etrinsic rmeters of the cmer R nd. - It is smmetric nd known u to sce - If w is, the cmer hs no skew. - If w w, the cmer hs squre ies. o cirte the cmer Wh is this usefu? o estimte the geometr of the D word hese roerties re usefu for two resons: o cirte the cmer: B using Eq 8 or Eq 9, we cn set u sstem of equtions tht ows us to cirte our cmer tht is, to estimte intern rmeters of the cmer o estimte the geometr of the D word: Once K is estimted or K is known, we cn use these equtions to estimte the orienttion of nes in D w.r.t. to the cmer reference sstem Let s see some emes net.

12 Lecture 4 Singe View Metroog Reiew cirtion Vnishing oints nd ine Estimting geometr from singe imge Etensions Reding: [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter 8 More Singe View Geometr [Hoiem Srese] Chter Siio Srese Lecture 4-6-Jn-5 [Eq. 8] cosθ Singe iew cirtion - eme ω ω ω Suose we cn identif two nes in n imge of the D word (e.g., the two uiding fcdes) nd suose we cn identif ir of re ines on ech of these nes. his ows to estimte two nishing oints in the imge nd. Suose we know tht these nes re erendicur in D (the two uiding fcdes re erendicur in D). hn we cn set u the sstem of equtions oe using Eq 9 nd the definition of ω. Is this sufficient to estimte the cmer rmeters? K hs in gener 5 degrees of freedom nd Eq.9 is scr eqution; so cer we don t he enough constrints. θ? 9 o ω ω (K K ) Do we he enough constrints to estimte K? K hs 5 degrees of freedom nd Eq.9 is scr eqution [Eq. 8] cosθ Singe iew cirtion - eme ω ω ω Let s now ssume we cn identif nother ir of re ines nd its corresonding nishing oint. With third nishing oint nd ssuming tht the set of irs of re ines re irwise orthogon in D (which is true in this eme), we cn use Eq. 9 to set u sstem of equtions (constrints) (Eqs. ). [Eqs. ] ω ω ω

13 ω ω ω ω Singe iew cirtion - eme 4 ω ω ω 5 ω4 ω 5 ω 6 ω known u to sce ω ω à Comute ω : Let s now so mke some ssumtions out the cmer. Let s ssume the cmer is zero-skew nd with squre ies. Using the roerties of ω: - ω is smmetric which mens we he 6 unknown - ω is known u sce which reduces to 5 unknown - ω nd ω ω which reduces to unknown We he now enough constrints to soe for the unknowns nd cn ccute ω. he ctu rmeters of K cn e comuted from ω using the Choesk fctoriztion. We don t roof this resuts; for more detis ese refer to HZ g 58. [Eqs. ] ω ω ω Once ω is ccuted, we get K: ω (K K ) K (Choesk fctoriztion; HZ g 58) hus, t the end of this rocedure we he mnged to cirte the cmer from just one singe imge Singe iew reconstruction - eme Once K is known we cn reconstruct the geometr of the scene; for instnce, we cn comute the orienttion of the nes in D using Eq. 7. In order to do so, we need to identif the corresonding ines t init nd seect the orienttion discontinuities (tht is, where nes fod). h K known n K horiz Scene ne orienttion in the cmer reference sstem Seect orienttion discontinuities Singe iew reconstruction - eme Agin, the ssortment of toos introduced in this ecture ows us to estimte roerties of the cmer from osertions nd/or estimte roerties of the word ssuming we he some knowedge out the word (e.g., where the horizon ines re; nes discontinuities; etc ). Notice tht tht ctu sce of the scene cnnot e recoered uness we ssume we he ccess to some mesurements in D (e.g., window size); this is simir to wht we did when we cirted the cmer. C Recoer the structure within the cmer reference sstem Notice: the ctu sce of the scene is NO recoered Recognition hes reconstruction Humns he ernt this

14 Lecture 4 Singe View Metroog Reiew cirtion Vnishing oints nd ines Estimting geometr from singe imge Etensions Reding: [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter Projectie Geometr nd rnsformtion in D [HZ] Chter 8 More Singe View Geometr [Hoiem Srese] Chter Siio Srese Lecture 4-6-Jn-5 Criminisi Zissermn, 99 Here we see some resuts otined using the roch roosed A. Criminisi nd Zissermn (999). his roch uses mn of the resuts introduced in this ecture. htt:// Criminisi Zissermn, 99 htt://

15 L rinit (46) Firenze, Snt Mri Noe; Msccio (4-48) L rinit (46) Firenze, Snt Mri Noe; Msccio (4-48) htt://

16 Singe iew reconstruction - drwcks Mnu seect: Vnishing oints nd ines; Pnr surfces; Occuding oundries; Etc.. Automtic Photo Po-u Hoiem et, 5 A few ers ter, D Hoiem roosed n roch where the rocess of recoering the geometr from singe imge is most utomtic. his roch eerges recognition nd segmenttion resuts; Automtic Photo Po-u Hoiem et, 5

17 Automtic Photo Po-u Hoiem et, 5 Softwre: htt:// MkeD rining Imge During the sme eriod, A. Ng nd his student A. Sen t Stnford so demonstrted tht it is ossie to recoer the geometr of the scene from singe imge using recognition resuts nd roiistic erence. Sen, Sun, Ng, 5 Prediction Deth Pne Prmeter MRF Pnr Surfce Segmenttion outue Connectiit Co-Pnrit Singe Imge Deth Reconstruction Sen, Sun, Ng, 5 A softwre: MkeD Conert our imge into d mode htt://mked.stnford.edu/ htt://mked.cs.corne.edu/

18 Coherent oject detection nd scene out estimtion from singe imge Y. Bo, M. Sun, S. Srese, CVPR, BMVC In m own grou, we he shown it is ossie to comine recognition nd reconstruction from singe imge in coherent formution. M. Sun Y. Bo Net ecture: Muti-iew geometr (eior geometr) Aendi