Lecture 11: Stereo and Surface Estimation

Transcription

1 Lecure : Sereo and Surface Emaon When camera poon have been deermned, ung rucure from moon, we would lke o compue a dene urface model of he cene. In h lecure we wll udy he o called Sereo Problem, where he goal o emae he deph of each pxel n an mage. Th requre compung a mach for each pxel n he mage even f he exure ambguou. Fgure : Lef and Mddle: Two mage ued for compung deph emae for every pxel n he lef mage. Rgh: The deph emae color coded. Recfed Camera, Dpary v. Deph Dene deph emaon requre machng of each pxel o a correpondng pxel n neghborng mage. Becaue of ambguou exure h problem dffcul o olve. Snce camera are known we can however ue he eppolar lne o lm he earch. We wll ar by aumng ha we have a par of o called recfed camera, P = K[I P 2 = K I 0] and b 0. () 0 Geomercally h mean ha boh camera have he ame orenaon and ha he econd camera poon a ranlaon n he x-drecon of he fr camera, ee Fgure 2. In general mage par aken wh regular camera do no fulfll hee aumpon, however hey can be modfed o do o. The he lne egmen onng he wo camera cener called he baelne. There are everal way of recfyng wo camera. A mple approach o fr elec he orenaon of axe of he new camera coordnae yem n one of he camera and hen roae boh he camera o h new coordnae yem. The new x-ax hould be parallel o he baelne and he new y and z-axe hould be perpendcular o. To keep he change ha occur when ranformng he mage mall, we hould elec he new coordnae yem o be a mlar o he old one a poble. We can elec he new x-ax by proecng he old x-ax ono he baelne and normalzng. When he x-ax ha been deermned we can choe he new-z ax a he proecon of he old z-ax ono he plane gong hrough he camera cener wh he new x-ax a normal. When boh he x and z axe are deermned we can fnd he y ax by ung he cro produc. Once we known he new camera orenaon we roae he camera. Snce he recfed camera and old camera are relaed hrough pure roaon he recfed mage are obaned by ranformng he mage ung a homography.

2 d x l x r x l x r x l C C 2 C C 2 Fgure 2: Sereo wh recfed camera. The camera have he ame orenaon and he mage plane normal are perpendcular o he baelne. Lef: The eppolar lne are parallel o he x-ax. Rgh: The dpary. Nex we wll how ha for h eup he eppolar lne are parallel o he x-ax of he mage. Suppoe ha γf f x 0 K = 0 f y 0. (2) 0 0 Then he proecon of a cene pon X = (X, Y,, ) n he camera P and P 2 gven by x l = K [ I 0 ] X Y γfx + fy + x 0 = fy + y 0 (3) X b x R = K I 0 Y γf(x + b) + fy + x 0 = fy + y 0 (4) 0 In regular coordnae h gve u he proecon (x L, y L ) = (x R, y R ) = ( fx+fy +x0, ( f(x+b)+fy +x0 ) fy +y0, fy +y0 (5) ). (6) Therefore y l = y R for any wo correpondng pon, whch mean ha he eppolar lne are horzonal. The dfference beween he x-coordnae d = x R x L = γfb (7) called he dpary. The machng problem can be een a he problem of agnng a dpary o each pon n he mage. The dpary nverely proporonal o he deph, ee Fgure 3. If we aume ha dpare can only be deermned up o neger value (no-ubpxel accuracy), hen can be een from Fgure 3 ha accurae (hgh reoluon) deph emaon can only be acheved when relavely mall (wh repec o he baelne b). 2 Machng Crera Gven a pxel and a canddae deph/dpary we need o have a crera for evaluang f h a good mach or no. Prevouly n he coure we have ued SIFT feaure. However, h ype of feaure have varou nvarance propere ha we do no wan here, nce he camera poon are already known. A mple commonly ued crera he normalzed cro correlaon. The cro correlaon compare a pach around he pxel o a pach around he poenal mach. If I and I 2 are he wo pache hen her correlaon NCC(I, I 2 ) = n n = (I (x ) Ī)(I 2 (x ) Ī2), (8) σ(i )σ(i 2 ) 2

3 Fgure 3: Deph a a funcon of dpary (γfb = ). Large dpare gve hgher deph reoluon. where he equence x are he pxel beng compared, Ī, Ī2, σ(i ) and σ(i 2 ) are he mean value and andard devaon of of he wo pache. The reul a number beween and where mean ha he pache mlar. The normalzed cro correlaon nvaran o ranlaon and recalng of he mage nene, whch very ueful f he wo mage are capured under dfferen lghng condon. Fgure 4 how he evaluaon of normalzed cro correlaon along an eppolar lne. Fgure 4: Evaluaon of he normalzed cro correlaon along an eppolar lne. Lef: Lef mage and he mage pon of nere. Mddle: Correpondng eppolar lne and a few local maxma of he NCC. Rgh: NCC for a range of dpare. Red rng are he ame local maxma a n he mddle mage. 3 Plane Sweep Algorhm When we ue more han wo camera mgh no be poble o recfy all he mage mulaneouly. In h cae an alernave o employ a plane weep algorhm. The bac dea ha f all he pxel have he ame deph hen all he cene pon are locaed on a plane n 3D. Snce proecon n wo mage from pon on a plane are relaed by a homography, we can hypoheze a deph, ranform he pxel of one camera no he econd and compue cro correlaon. Sweepng he plane hrough a ere of hypohee gve a co for agnng pxel o he hypohezed deph. If he camera are P = K[I 0] and P 2 = K[R ] hen he plane Π wh cene pon a he deph gven by Π = (0, 0, /, ). The homography H for he deph hen gven by he formula (See Agnmen, Exerce 6 for a dervaon.) H = K ( R + ( 0 0 )) K. (9) 3

4 (a) (b) (c) Fgure 5: Plane weep algorhm. Gven wo camera (a), hypoheze a deph for all he pxel (b), proec no he econd camera and compare pxel value (c). Sweepng he cene plane over dfferen deph gve co of agnng pxel o hee hypohezed deph. Compared o recfed camera where we can mply compare pache along he eppolar lne, ranformng he mage ung he homography more compuaonally expenve. On he oher hand h approach very convenen n ha work wh general camera confguraon. Furhermore, nce he mage re-ampled durng he ranformaon we do no need o lm ourelve o deph ha are are nvere value of neger valued dpare. Therefore up-pxel accuracy naurally acheved wh h approach. 4 Regularzaon/Energy Mnmzaon The reul of he plane weep algorhm a funcon for each pxel ha pecfe a co of agnng ha pxel a ceran deph. By elecng he malle co for each pxel we oban a dene deph emae. The reulng urface wll ofen be noy. Th becaue ndvdual pxel emae can be unrelable due o ambguou exure and elf occluon. To mprove he emaed urface and reduce he noe a common approach o eek an agnmen where neghborng pxel have mlar deph. Typcally one mnmze an energy funconal of he form N () E (, ) + E ( ). (0) The econd erm E ( ) he co of agnng he deph o pxel. Th erm ypcally referred o a he daa erm nce baed on mage daa. The fr erm E (, ) penalze dfference n deph beween pxel and a pxel ha n he neghborhood N () of pxel. Th erm uually called he moohne erm nce favor oluon wh mooh urface. Opmzng h ype of energy challengng due o he numerou local mnma of he ndvdual daa erm E ( ), ee Fgure 4. The mo ucceful mehod for dong h are baed on dcree graph mehod. If bnary, hen he energy (0) can be een a he mnmum cu on a graph where he node correpond o he pxel and he edge correpond o he neghborhood rucure. Fndng he mnmum cu n a graph can be olved effcenly (n polynomal me complexy) by compung he maxmal flow on he ame graph. Le u fr conder a wo pxel cae wh he energy funconal E(, ) = E (, ) + E (, ) + E ( ) + E ( ). () The correpondng graph ha wo node repreenng he pxel and wo pecal node; he ource and he nk. The graph ha dreced edge from o and, from and o and beween and. In addon each edge ha a (non-negave) capacy c(, ). A cu n h graph a paronng of he node no wo don e S and T where S are he node conneced o he ource and T he node conneced o he nk. The value of he cu he um of all he edge capace of he edge gong form S o T. 4

5 E (0,0) E (,0) E (0,) E (,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) c(,) Fgure 6: The four cu correpondng o he value of he energy erm n (2)-(5). The energy () can be olved a a mnmal cu problem on h graph under ceran condon. Fr of all we ee ha E(, ) 0 nce graph edge have non-negave capace. Noe ha f he energy no pove we can alway add a conan o E(, ) whou changng he mnmzer and hereby oban an equvalen pove energy. If we le S mean ha = hen from Fgure 6 we ee ha E(0, 0) = c(, ) + c(, ) (2) E(, 0) = c(, ) + c(, ) + c(, ) (3) E(0, ) = c(, ) + c(, ) + c(, ) (4) E(, ) = c(, ) + c(, ). (5) Snce he capace are all pove we ee from hee equaon ha n order o be able o repreen h energy wh a graph, hen E(0, 0) + E(, ) E(, 0) + E(0, ) (6) mu hold. Energe wh parwe moohng erm ha fulfll (6) are called ubmodular. I can be hown ha he condon (6) acually enure ha he energy can be repreened by a graph. Furhermore, eay o ee ha he um of ubmodular energe alo ubmodular. Th mean ha f he erm E of he energy (0) are all ubmodular hen we can mnmze h energy by olvng a max-flow/mn-cu problem. When no bnary (a n ereo) we ypcally employ move makng algorhm. The goal of hee algorhm o modfy a many pxel a poble mulaneouly n order o avod geng uck n poor local mnma. One of he mo popular approache ha of α-expanon. Suppoe ha we have a curren agnmen of deph. Then each pxel gven he opon o wch from he curren deph o a new deph α. Snce each pxel ha wo choce (move o α or rean he old value) h can be formulaed a a bnary problem. If h new bnary problem ubmodular can be olved effcenly ung max-flow/mn-cu algorhm. Suppoe ha n he curren agnmen = β and = γ and conder he erm E (, ) when we le he pxel wch o α. We defne a new bnary energy erm E B (x, x ) by leng x = 0 when rean curren agnmen, x = when wche o α (and x mlarly). Then Therefore, f he moohne erm of he energy (0) fulfll E B (0, 0) = E (β, γ) (7) E B (0, ) = E (β, α) (8) E B (, 0) = E (α, γ) (9) E B (, ) = E (α, α). (20) E (β, γ) + E (α, α) E (β, α) + E (α, γ) (2) we can olve he α-expanon effcenly ung max-flow/mn-cu algorhm. 5

6 In many cae we can aume ha he energy non-negave, ymmerc E, (β, γ) = E(γ, β) and ha E(α, α) = 0. In hee cae can be een ha E ha o be a merc dance nce (2) reduce o he rangle nequaly. A commonly ued moohne erm ha fulfll he merc condon he runcaed abolue dance E (β, γ) = mn( β γ, ). (22) Th erm favor agnmen where neghborng deph have mall dpary dfference. In real mage we hould however alo allow for dconnuou deph agnmen due o obec boundare (ranon beween mooh urface). Therefore he hrehold added o he energy o preven over-moohng a dconnue. 6