where λ = Z/f. where a 3 4 projection matrix represents a map from 3D to 2D. Part I: Single and Two View Geometry Internal camera parameters

Transcription

1 Imagng Geometry Multple Vew Geometry Perspectve projecton Y Rchard Hartley and Andrew Zsserman X λ y = Y f Z O X p y Z X where λ = Z/f. mage plane CVPR June 1999 Ths can be wrtten as a lnear mappng between homogeneous coordnates (the equaton s only up to a scale factor): X 1000 y = 0100 Y Z f where a 3 4 projecton matr represents a map from 3D to 2D. Image Coordnate System Part I: Sngle and Two Vew Geometry Internal camera parameters The man ponts covered n ths part are: A perspectve (central) projecton camera s represented by a 3 4 matr. The most general perspectve transformaton transformaton between two planes (a world plane and the mage plane, or two mage planes nduced by a world plane) s a plane projectve transformaton. Ths can be computed from the correspondence of four (or more) ponts. The eppolar geometry between two vews s represented by the fundamental matr. Ths can be computed from the correspondence of seven (or more) ponts. k cam = 0 k y y cam = y y 0 where the unts of k,k y are [pels/length]. y y 0 p ycam = y = 1 α 0 cam cam α y y 0 y cam = K y cam f 1 1 f f 0 cam where α = fk,α y = fk y.

2 Camera Calbraton Matr K s a 3 3 upper trangular matr, called the camera calbraton matr: α 0 K = α y y 0 1 There are four parameters: () The scalng n the mage and y drectons, α and α y. () The prncpal pont ( 0,y 0 ), whch s the pont where the optc as ntersects the mage plane. The aspect rato s α y /α. Concatenatng the three matrces, X 1000 [ ] R t Y = y = K = K [R t] X 1 Z 1 whch defnes the 3 4 projecton matr from Eucldean 3-space to an mage as = PX P = K [R t] =KR[I R t] Note, the camera centre s at (X, Y, Z) = R t. In the followng t s often only the 3 4 form of P that s mportant, rather than ts decomposton. World Coordnate System A Projectve Camera Eternal camera parameters X cam Y cam Z cam 1 = [ R t 0 1 X ] Y Z 1 X cam Ycam O cam Eucldean transformaton between world and camera coordnates R s a 3 3 rotaton matr t s a 3 1 translaton vector Z cam R, t X O Z Y The camera model for perspectve projecton s a lnear map between homogeneous pont coordnates y P (3 4) 1 Image Pont X Y Z 1 Scene Pont = P X The camera centre s the null-vector of P e.g. f P =[I 0] then the centre s X =(0, 0, 0, 1). P has 11 degrees of freedom (essental parameters). P has rank 3.

3 What does calbraton gve? Camera Calbraton (Resectonng) K provdes the transformaton between an mage pont and a ray n Eucldean 3-space. Once K s known the camera s termed calbrated. A calbrated camera s a drecton sensor, able to measure the drecton of rays lke a 2D protractor. α 0 = y = α y y X cam Y cam Z cam = Kd Problem Statement: Gven n correspondences X, where X s a scene pont and ts mage: Compute P = K [R t] such that = PX. The algorthm for camera calbraton has two parts: () Compute the matr P from a set of pont correspondences. () Decompose P nto K, R and t va the QR decomposton. Angle between rays Algorthm step 1: Compute the matr P d 1 = PX. d 1.d 2 cos θ = (d 1.d 1 ) 1 2(d 2.d 2 ) 1 2 C θ 1 2 d2 Each correspondence generates two equatons = p 11X + p 12 Y + p 13 Z + p 14 y = p 21X + p 22 Y + p 23 Z + p 24 p 31 X + p 32 Y + p 33 Z + p 34 p 31 X + p 32 Y + p 33 Z + p 34 Multplyng out gves equatons lnear n the matr elements of P d 1 d 2 cos θ = (d 1 d 1 ) 1/2 (d 2 d 2 ) = 1 (K K 1 ) 2 1/2 ( 1 (K K 1 ) 1 ) 1/2 ( 2 (K K 1 ) 2 ) 1/2 = 1 ω 2 ( 1 ω 1 ) 1/2 ( 2 ω 2 ) 1/2 where ω =(KK ) 1. (p 31 X + p 32 Y + p 33 Z + p 34 )=p 11 X + p 12 Y + p 13 Z + p 14 y (p 31 X + p 32 Y + p 33 Z + p 34 )=p 21 X + p 22 Y + p 23 Z + p 24 These equatons can be rearranged as ( ) XYZ10000 X Y Z p = XYZ1 yx yy yz y wth p =(p 11,p 12,p 13,p 14,p 21,p 22,p 23,p 24,p 31,p 32,p 33,p 34 ) a 12-vector.

4 Algorthm step 1 contnued Algorthm step 2: Decompose P nto K, R and t Solvng for P () Concatenate the equatons from (n 6) correspondences to generate 2n smultaneous equatons, whch can be wrtten: Ap = 0, where A s a 2n 12 matr. () In general ths wll not have an eact soluton, but a (lnear) soluton whch mnmses Ap, subject to p =1s obtaned from the egenvector wth least egenvalue of A A. Or equvalently from the vector correspondng to the smallest sngular value of the SVD of A. () Ths lnear soluton s then used as the startng pont for a non-lnear mnmsaton of the dfference between the measured and projected pont: mn P ((,y ) P (X, Y, Z, 1)) 2 The frst 3 3 submatr, M, ofp s the product (M = KR) of an upper trangular and rotaton matr. () Factor M nto KR usng the QR matr decomposton. Ths determnes K and R. () Then t = K 1 (p 14,p 24,p 34 ) Note, ths produces a matr wth an etra skew parameter s α s 0 K = α y y 0 1 wth s =tanθ, and θ the angle between the mage aes. Eample - Calbraton Object Determne accurate corner postons by () Etract and lnk edges usng Canny edge operator. () Ft lnes to edges usng orthogonal regresson. () Intersect lnes to obtan corners to sub-pel accuracy. The fnal error between measured and projected ponts s typcally less than 0.02 pels.

5 Weak Perspectve Plane projectve transformatons Track back, whlst zoomng to keep mage sze fed C π Y Z X π perspectve weak perspectve The magng rays become parallel, and the result s: r 11 r 12 r 13 P = K r 21 r 22 r A generalzaton s the affne camera Choose the world coordnate system such that the plane of the ponts has zero Z coordnate. Then the 3 4 matr P reduces to X 1 p 11 p 12 p 13 p 14 p 11 p 12 p 14 X Y 2 = p 21 p 22 p 23 p 24 0 = p 21 p 22 p 24 Y 3 p 31 p 32 p 33 p 34 p 1 31 p 32 p 34 1 whch s a 3 3 matr representng a general plane to plane projectve transformaton. The matr M 2 3 has rank two. The Affne Camera m 11 m 12 m 13 t 1 P = m 21 m 22 m 23 t Projecton under an affne camera s a lnear mappng on non-homogeneous coordnates composed wth a translaton: ( ) [ ] X ( ) m11 m = 12 m 13 t1 Y + y m 21 m 22 m 23 t 2 The pont (t 1,t 2 ) s the mage of the world orgn. The centre of the affne camera s at nfnty. An affne camera has 8 degrees of freedom. It models weak-perspectve and para-perspectve. Z Projectve transformatons contnued h 11 h 12 h 13 = h 21 h 22 h 23 h 31 h 32 h or = H, where H s a 3 3 non-sngular homogeneous matr. O Ths s the most general transformaton between the world and mage plane under magng by a perspectve camera. It s often only the 3 3 form of the matr that s mportant n establshng propertes of ths transformaton. A projectve transformaton s also called a homography and a collneaton. H has 8 degrees of freedom. / Y / X / X π Y X X Π

6 Four ponts defne a projectve transformaton The Cone of Rays Gven n pont correspondences (, y) (,y ) Compute H such that = H Each pont correspondence gves two constrants = 1 = h 11 + h 12 y + h 13, y = 2 3 h 31 + h 32 y + h 33 = h 21 + h 22 y + h 23 3 h 31 + h 32 y + h 33 and multplyng out generates two lnear equatons for the elements of H (h 31 + h 32 y + h 33 )=h 11 + h 12 y + h 13 y (h 31 + h 32 y + h 33 )=h 21 + h 22 y + h 23 If n 4 (no three ponts collnear), then H s determned unquely. The converse of ths s that t s possble to transform any four ponts n general poston to any other four ponts n general poston by a projectvty. An mage s the ntersecton of a plane wth the cone of rays between ponts n 3-space and the optcal centre. Any two such mages (wth the same camera centre) are related by a planar projectve transformaton. = H e.g. rotaton about the camera centre C / X Eample 1: Removng Perspectve Dstorton Eample 2: Synthetc Rotatons Gven: the coordnates of four ponts on the scene plane Fnd: a projectve rectfcaton of the plane Ths rectfcaton does not requre knowledge of any of the camera s parameters or the pose of the plane. It s not always necessary to know coordnates for four ponts. The synthetc mages are produced by projectvely warpng the orgnal mage so that four corners of an maged rectangle map to the corners of a rectangle. Both warpngs correspond to a synthetc rotaton of the camera about the (fed) camera centre.

7 Correspondence Geometry Two Vew Geometry Gven the mage of a pont n one vew, what can we say about ts poston n another? Cameras P and P such that X X? = PX = P X X? Baselne between the cameras s non-zero. / l Gven an mage pont n the frst vew, where s the correspondng pont n the second vew? e e/ eppolar lne for What s the relatve poston of the cameras? What s the 3D geometry of the scene? A pont n one mage generates a lne n the other mage. Ths lne s known as an eppolar lne, and the geometry whch gves rse to t s known as eppolar geometry. Images of Planes Eppolar Geometry Projectve transformaton between mages nduced by a plane Eppolar Plane Π X = H 1π π = H 2π π π π / = H 2π π = H 2π H 1 1π = H / C C H can be computed from the correspondence of four ponts on the plane. / O Left eppolar lne e / e / O Rght eppolar lne The eppolar lne l s the mage of the ray through. The eppole e s the pont of ntersecton of the lne jonng the camera centres the baselne wth the mage plane. The eppole s also the mage n one camera of the centre of the other camera. All eppolar lnes ntersect n the eppole.

8 Eppolar pencl Homogeneous Notaton Interlude e e / baselne As the poston of the 3D pont X vares, the eppolar planes rotate about the baselne. Ths famly of planes s known as an eppolar pencl. All eppolar lnes ntersect at the eppole. A lne l s represented by the homogeneous 3-vector l 1 l = for the lne l 1 + l 2 y + l 3 =0. Only the rato of the homogeneous lne coordnates s sgnfcant. pont on lne: l. =0or l =0or l =0 l 2 l 3 two ponts defne a lne: l = p q p q l l two lnes defne a pont: = l m m Eppolar geometry eample Matr notaton for vector product e at nfnty / e at nfnty The vector product v can be represented as a matr multplcaton v =[v] where 0 v z v y [v] = v z 0 v v y v 0 Eppolar geometry depends only on the relatve pose (poston and orentaton) and nternal parameters of the two cameras,.e. the poston of the camera centres and mage planes. It does not depend on structure (3D ponts eternal to the camera). [v] s a 3 3 skew-symmetrc matr of rank 2. v s the null-vector of [v], snce v v =[v] v = 0.

9 Algebrac representaton - the Fundamental Matr Propertes of F F =0 l = F F s a 3 3 rank 2 homogeneous matr F e = 0 It has 7 degrees of freedom Countng: = 7. Compute from 7 mage pont correspondences F s a rank 2 homogeneous matr wth 7 degrees of freedom. Pont correspondence: If and are correspondng mage ponts, then F =0. Eppolar lnes: l = F s the eppolar lne correspondng to. l = F s the eppolar lne correspondng to. Eppoles: Fe = 0 F e = 0 Computaton from camera matrces P, P : F =[P C] P P +, where P + s the pseudo-nverse of P, and C s the centre of the frst camera. Note, e = P C. Canoncal cameras, P =[I 0], P =[M m], F =[e ] M = M [e], where e = m and e = M 1 m. Fundamental matr - sketch dervaton Plane nduced homographes gven F X π Gven the fundamental matr F between two vews, the homography nduced by a world plane s O Hπ / e e / / l / O H =[e ] F + e v where v s the nhomogeneous 3-vector whch parametrzes the 3- parameter famly of planes. Step 1: Pont transfer va a plane = H π Step 2 : Construct the eppolar lne l = e =[e ] l =[e ] H π = F F =[e ] H π e.g. compute plane from 3 pont correspondences. Gven a homography Ĥ nduced by a partcular world plane, then a homography nduced by any plane may be computed as H = Ĥ + e v Ths shows that F s a 3 3 rank 2 matr.

10 Projectve ambguty of reconstructon Projectve Reconstructon from 2 vews Soluton s not unque wthout camera calbraton Soluton s unque up to a projectve mappng : P PH 1 P P H 1 X HX Then verfy =(PH 1 )(HX )=PX =(P H 1 )(HX )=P X Same problem holds however many vews we have Statement of the problem Projectve Dstorton demo Gven Correspondng ponts n two mages. < Projectve dstorton demo > Fnd Cameras P and P and 3D ponts X such that = PX ; = PX

11 Basc Theorem Detals of Projectve Reconstructon - Computaton of F. Gven suffcently many ponts to compute unque fundamental matr : 8 ponts n general poston 7 ponts not on a ruled quadrc wth camera centres Then 3D ponts may be constructed from two vews Up to a 3D projectve transformaton Ecept for ponts on the lne between the camera centres. Methods of computaton of F left untl later Several methods are avalable : () Normalzed 8-pont algorthm () Algebrac mnmzaton () Mnmzaton of eppolar dstance (v) Mnmzaton of symmetrc eppolar dstance (v) Mamum Lkelhood (Gold-standard) method. (v) Others,... Steps of projectve reconstructon Factorzaton of the fundamental matr Reconstructon takes place n the followng steps : Compute the fundamental matr F from pont correspondences Factor the fundamental matr as F =[t] M The two camera matrces are P =[I 0] and P =[M t]. Compute the ponts X by trangulaton SVD method () Defne Z = () Compute the SVD F = UDV where D = dagr, s, 0 () Factorzaton s F =(UZU )(UZDV ) Smultaneously corrects F to a sngular matr.

12 Factorzaton of the fundamental matr Trangulaton Drect formula Let e be the eppole. Specfc formula Solve e F =0 Trangulaton : Knowng P and P Knowng and Compute X such that = PX ; = P X X P =[I 0] ; P =[[e ] F e ]=[M e ] Ths soluton s dentcal to the SVD soluton. / / d / d / O e e / O / Non-unqueness of factorzaton Trangulaton n presence of nose Factorzaton of the fundamental matr s not unque. General formula : for varyng v and λ P =[I 0] ; P =[M + e v λe ] Dfference factorzatons gve confguratons varyng by a projectve transformaton. 4-parameter famly of solutons wth P =[I 0]. In the presence of nose, back-projected lnes do not ntersect. / O O / Rays do not ntersect n space l = F / / / l = F e e / mage 1 mage 2 Measured ponts do not le on correspondng eppolar lnes

13 Whch 3D pont to select? Lnear trangulaton methods Md-pont of common perpendcular to the rays? Not a good choce n projectve envronment. Concepts of md-pont and perpendcular are meanngless under projectve dstorton. Weghted pont on common perpendcular, weghted by dstance from camera centres? Dstance s also undefned concept. Some algebrac dstance? Wrte down projecton equatons and solve? Lnear least squares soluton. Mnmzes nothng meanngful. Drect analogue of the lnear method of camera resectonng. Gven equatons = PX = P X p are the rows of P. Wrte as lnear equatons n X p 3 p 1 yp 3 p 2 p 3 p 1 X =0 yp 3 p 2 Solve for X. Generalzes to pont match n several mages. Mnmzes no meanngful quantty not optmal. Problem statement Mnmzng geometrc error Assume camera matrces are gven wthout error, up to projectve dstorton. Hence F s known. A par of matched ponts n an mage are gven. Possble errors n the poston of matched ponts. Fnd 3D pont that mnmzes sutable error metrc. Method must be nvarant under 3D projectve transformaton. Pont X n space maps to projected ponts ˆ and ˆ n the two mages. Measured ponts are and. Fnd X that mnmzes dfference between projected and measured ponts.

14 Geometrc error... Mnmzaton method / X / d / d / O e e / O / Our strategy for mnmzng cost functon s as follows () Parametrze the pencl of eppolar lnes n the frst mage by a parameter t. Eppolar lne s l(t). () Usng the fundamental matr F, compute the correspondng eppolar lne l (t) n the second mage. () Epress the dstance functon d(, l(t)) 2 +d(, l (t)) 2 eplctly as a functon of t. (v) Fnd the value of t that mnmzes ths functon. Cost functon C(X) =d(, ˆ) 2 + d(, ˆ ) 2 Dfferent formulaton of the problem Mnmzaton method... Mnmzaton problem may be formulated dfferently: Mnmze d(, l) 2 + d(, l ) 2 l and l range over all choces of correspondng eppolar lnes. ˆ s the closest pont on the lne l to. Same for ˆ. Fnd the mnmum of a functon of a sngle varable, t. Problem n elementary calculus. Dervatve of cost reduces to a 6-th degree polynomal n t. Fnd roots of dervatve eplctly and compute cost functon. Provdes global mnmum cost (guaranteed best soluton). Detals : See Hartley-Sturm Trangulaton. l = F / d / d / / l = F θ(t) e θ / (t) mage 1 mage 2 e / /

15 Multple local mnma Cost functon may have local mnma. Shows that gradent-descent mnmzaton may fal. 1.2 Ths page left empty Left : Eample of a cost functon wth three mnma. Rght : Cost functon for a perfect pont match wth two mnma. Uncertanty of reconstructon Ths page left empty Uncertanty of reconstructon. The shape of the uncertanty regon depends on the angle between the rays.

16 Sngle pont equaton - Fundamental matr Computaton of the Fundamental Matr Gves an equaton : (, y,,y, y y, y,,y,1) f 11 f 12 f 13 f 21 f 22 f 23 f 31 f 32 f 33 =0 where f =(f 11,f 12,f 13,f 21,f 22,f 23,f 31,f 32,f 33 ) holds the entres of the Fundamental matr Basc equatons Gven a correspondence The basc ncdence relaton s F =0 May be wrtten f 11 + yf 12 + f 13 + y f 21 + y yf 22 + y f 23 + f 31 + yf 32 + f 33 =0. Total set of equatons 1 1 1y 1 1 y 1 1 y 1y 1 y 1 1 y 1 1 Af = n n ny n n y n n y ny n y n n y n 1 f 11 f 12 f 13 f 21 f 22 f 23 f 31 f 32 f 33 = 0

17 Solvng the Equatons Computng F from 7 ponts Soluton s determned up to scale only. Need 8 equatons 8 ponts 8 ponts unque soluton > 8 ponts least-squares soluton. F has 9 entres but s defned only up to scale. Sngularty condton det F = 0gves a further constrant. F has 3 rows = det F =0s a cubc constrant. F has only 7 degrees of freedom. It s possble to solve for F from just 7 pont correspondences. Least-squares soluton () Form equatons Af = 0. () Take SVD : A = UDV. () Soluton s last column of V (corresp : smallest sngular value) (v) Mnmzes Af subject to f =1. The sngularty constrant 7-pont algorthm Fundamental matr has rank 2 : det(f) =0. Computaton of F from 7 pont correspondences () Form the 7 9 set of equatons Af =0. () System has a 2-dmensonal soluton set. () General soluton (use SVD) has form f = λf 0 + µf 1 Left : Uncorrected F eppolar lnes are not concdent. Rght : Eppolar lnes from corrected F. (v) In matr terms F = λf 0 + µf 1 (v) Condton det F =0gves cubc equaton n λ and µ. (v) Ether one or three real solutons for rato λ : µ.

18 Correctng F usng the Sngular Value Decomposton The normalzed 8-pont algorthm If F s computed lnearly from 8 or more correspondences, sngularty condton does not hold. SVD Method () SVD : F = UDV () U and V are orthogonal, D = dag(r, s, t). () r s t. (v) Set F = U dag(r, s, 0) V. (v) Resultng F s sngular. (v) Mnmzes the Frobenus norm of F F (v) F s the closest sngular matr to F. Raw 8-pont algorthm performs badly n presence of nose. Normalzaton of data 8-pont algorthm s senstve to orgn of coordnates and scale. Data must be translated and scaled to canoncal coordnate frame. Normalzng transformaton s appled to both mages. Translate so centrod s at orgn Scale so that RMS dstance of ponts from orgn s 2. Average pont s (1, 1, 1). Complete 8-pont algorthm Normalzed 8-pont algorthm () Normalzaton: Transform the mage coordnates : 8 pont algorthm has two steps : () Lnear soluton. Solve Af =0to fnd F. () Constrant enforcement. Replace F by F. Warnng Ths algorthm s unstable and should never be used wth unnormalzed data (see net slde). ˆ = T ˆ = T () Soluton: Compute F from the matches ˆ ˆ ˆ Fˆ =0 () Sngularty constrant : Fnd closest sngular F to F. (v) Denormalzaton: F = T F T.

19 Statue mages Comparson of Normalzed and Unnormalzed Algorthms Lfa House mages Grenoble Museum

20 Oford Basement Testng methodology () Pont matches found n mage pars and outlers dscarded. () Fundamental matr was found from varyng number (n) of ponts. () F was tested aganst other matched ponts not used to compute t. (v) Dstance of a pont from predcted eppolar lne was the metrc. (v) 100 trals for each value of n. (v) Average error s plotted aganst n. Calbraton object Comparson of normalzed and unnormalzed 8-pont algorthms Average Error House Average Error Statue N N Dstance of ponts from eppolar lnes : Top : Unnormalzed 8-pont algorthm. Bottom : Normalzed 8-pont algorthm.

21 Comparson of normalzed and unnormalzed 8-pont algorthms. Illustraton of Effect of Normalzaton Average Error Museum Average Error N N Dstance of ponts from eppolar lnes : Top : Unnormalzed 8-pont algorthm. Bottom : Normalzed 8-pont algorthm Calbraton Smlar problem : Computaton of a 2D projectve transformaton gven pont matches n two mages. () Homography s computed from 5 nosy pont matches. () Homography s appled to a further (6th) pont () Nose level appromately equal to wth of lnes n the crosses (net page) (v) Repeated 100 tmes. (v) Spread of the transformed 6th pont s shown n relaton to the 5 data ponts. (v) 95% ellpses are plotted. Comparson of normalzed and unnormalzed 8-pont algorthms. Normalzaton and 2D homography computaton Average Error Corrdor Unnormalzed data N Dstance of ponts from eppolar lnes : Top : Unnormalzed 8-pont algorthm. Bottom : Normalzed 8-pont algorthm. Normalzed data

22 Normalzaton and 2D homography computaton Algebrac Mnmzaton Algorthm Unnormalzed data Normalzed data Condton number The algebrac mnmzaton algorthm Bad condton number the reason for poor performance. Condton number = κ 1 /κ 8, rato of sngular values of A. Condton number House N no normalzaton wth normalzaton Enforcng the sngularty constrant SVD method mnmzes F F. smple and rapd. Not optmal Treats all entres of F equally. However, some entres of F are more tghtly constraned by the data. Bad condtonng acts as a nose amplfer. Normalzaton mproves the condton number by a factor of Reference : Hartley In defence of the 8-pont algorthm.

23 The Algebrac Method Mnmze Af subject to f =1AND det F =0. det F =0s a cubc constrant. Requres an teratve soluton. However, smple teratve method works. Soluton assumng known eppole - contnued Wrte Af = AEm. Mnmze AEm subject to Em =1. Ths s a lnear least-squares estmaton problem. Non-teratve algorthm nvolvng SVD s possble Reference : Hartley, Mnmzng Algebrac Error, Royal Socety Proceedngs, Soluton assumng known eppole Iteratve Algebrac Estmaton We may wrte F = M[e], where e s eppole. F of ths form s sngular. Assume e s known, fnd M. Wrte F = M[e] as f = Em f 11 0 e 3 e 2 f 12 e 3 0 e 1 f 13 e 2 e 1 0 f 21 0 e 3 e 2 f 22 = e 3 0 e 1 f 23 e 2 e 1 0 f 31 0 e 3 e 2 f 32 e 3 0 e 1 e 2 e 1 0 f 33 m 11 m 12 m 13 m 21 m 22 m 23 m 31 m 32 m 33 Fnd the fundamental matr F that mnmzes the algebrac error Af subject to f =1and det F =0. Concept : Vary eppole e to mnmze the algebrac error Af = AEm. Remark : Each choce of eppole e defnes a mnmmum error vector AEm as above. Use Levenberg-Marquardt method to mnmze ths error. Smple 3 9 mnmzaton problem. 3 nputs the coordnates of the eppole 9 outputs the algebrac error vector Af = AEm. Each step requres estmaton of m usng SVD method. Trcks can be used to avod SVD (see Hartley-Royal-Socety).

24 The Gold Standard (ML) Method Assumes a Gaussan dstrbuted nose. Measured correspondences. Mnmzaton of Geometrc Error Estmated correspondences ˆ and ˆ d(, ˆ ) 2 + d(, ˆ ) 2 subject to ˆ Fˆ =0eactly for some F. Smultaneous estmaton of F and ˆ and ˆ. / d d / / / l = F l = F Mnmzaton of Geometrc Error The Gold Standard (ML) Method Algebrac error vector Af has no clear geometrc meanng. Should be mnmzng geometrc quanttes. Errors derve from ncorrect measurements of match ponts. / d / / l = F l = F We should be measurng dstances from eppolar lnes. d / Mnmzng the Gold-Standard error functon. Intal 3D reconstructon : P =[I 0] P =[M t] X =(X, Y, 1, T ) Compute ˆ = PX =(X, Y, 1) and ˆ = P X. Iterate over P and X =(X, Y, 1, T ) to mnmze cost functon : d(, ˆ ) 2 + d(, ˆ ) 2 Total of 3n +12parameters. 12 parameters for the camera matr P 3 parameters for each pont X. Once P =[M t] s found, compute F =[t] M.

25 Sparse Levenberg-Marquardt Parametrzaton of rank-2 matrces Reference : Hartley- Azores () Coordnates of X do not affect ˆ j or ˆ j (for j). () Sparse LM takes advantage of sparseness () Lnear tme n n (number of ponts). (v) Reference : Hartley- Azores Both eppoles as parameters. The resultng form of F s a b αa+ βb F = c d αc+ βd α a + β cα b + β dα αa + α βb + β αc + β βd Parametrzaton of rank-2 matrces Eppolar dstance Estmaton of F may be done by parameter mnmzaton. Parametrze F such that det F =0. Varous parametrzatons have been used. Overparametrzaton. Wrte F =[t] M. 3 parameters for t and 9 for M. Eppolar parametrzaton. abαa+ βb F = cdαc+ βd efαe+ βf To acheve a mnmum set of parameters, set one of the elements, for nstance f to 1. / d / / l = F l = F Pont correspondence : Pont eppolar lne F. Eppolar dstance s dstance of pont to eppolar lne F. Wrte F =(λ, µ, ν) and =(,y, 1). Dstance s d(, F) = F(λ 2 + µ 2 ) 1/2 d /

26 Eppolar dstance - contnued Luong / Zhang s other error functon Eppolar dstance may be wrtten as d( F, F) = ((F) 2 1 +(F)2 2 )1/2 Total cost functon : ( F ) 2 ( F ) 2 (F ) 2 1 +(F ) 2 2 +(F )2 1 +(F F )2 2 Represents a frst-order appromaton to geometrc error. d(, F ) 2 = (F ) 2 1 +(F ) 2 2 Total cost : sum over all. Mnmze ths cost functon over parametrzaton of F. Symmetrc eppolar dstance Eppolar dstance functon s not symmetrc. Prefer sum of dstances n both mages. Symmetrc cost functon s d(, F) 2 + d(, F ) 2 Sum over all ponts : Cost = ( ) ( F ) (F ) 2 1 +(F ) (F )2 1 +(F )2 2 Ths page left empty Problem Ponts near the eppole have a dsproportonate nfluence. Small devaton n pont makes bg dfference to eppolar lne.

27 Epermental procedure More Algorthm Comparson () Fnd matched ponts n mage par. () Select n matched ponts at random () Compute the fundamental matr (v) Compute eppolar dstance for all other ponts. (v) Repeat 100 tmes for each n and collect statstcs. The error s defned as 1 N (d( N, F )+d(, F )).e Average symmetrc eppolar dstance. Epermental Evaluaton of the Algorthms Results Three of the algorthms compared () The normalzed 8-pont algorthm () Mnmzaton of algebrac error whlst mposng the sngularty constrant () The Gold Standard geometrc algorthm Error houses Error statue Number of ponts Number of Ponts Normalzed 8-pont, geometrc and algebrac error.

28 Results - contnued Error Corrdor Error calbraton Covarance computaton Number of Ponts Number of Ponts Normalzed 8-pont, geometrc and algebrac error. Results - contnued Covarance of P Error museum To compute the covarance matr of the entres of P : () Defne Q =(, ) vector of meaurments n both mages. () Compute the dervatve matrces A =[ ˆQ / P ] and B =[ ˆQ / X ] () Compute n steps Number of Ponts Normalzed 8-pont, geometrc and algebrac error. U = A Σ 1 Q A V = B Σ 1 Q B W = A Σ 1 Q B Σ P =(U W V 1 W ) + (pseudo-nverse)

29 Covarance of F Specal cases of F-computaton To compute the covarance of F : () Gven P =[M m], then F =[m] M. () Compute J = F/ P. () Σ F = JΣ P J Usng specal motons can smplfy the computaton of the fundamental matr. Covarance of eppolar lne correspondng to () Eppolar lne s l = F. () Gven and Σ F compute J = l/ F () Σ l = JΣ F J The envelope of eppolar lnes Pure translaton May compute envelope of eppolar lnes. C = l l k 2 Σ l C s a hyperbola that contans the eppolar lne wth a gven probablty α. k 2 chosen such that F2 1 (k 2 )=α, F 2 (k 2 ) represents the cumulatve χ 2 2 dstrbuton, wth probablty α the lnes le wthn ths regon. Can assume P =[I 0] and P =[I t]. F =[t]. F s skew-symmetrc has 2 dof. Beng skew-symmetrc, automatcally has rank 2. e mage < eppolar lne demonstraton here > For a pure translaton the eppole can be estmated from the mage moton of two ponts.

30 Cameras wth the same prncpal plane Ponts on a ruled quadrc Prncpal plane of the camera s the thrd row of P. Cameras have the same thrd row. Affne cameras - last row s (0, 0, 0, 1). Smple correspondences est : (,y, 0)F(, y, 0) =0 for any (,y, 0) and (, y, 0). F has the followng form : a F = b cde () If all the ponts and the two camera centres le on a ruled quadrc, then there are three possble fundamental matrces. () ponts le n a plane. The correspondences lead to a 3-parameter famly of possble fundamental matrces F (note, one of the parameters accounts for scalng the matr so there s only a two-parameter famly of homogeneous matrces). () Two cameras at the same pont : The fundamental matr does not est. There s no such thng as an eppolar plane, and eppolar lnes are not defned. Correspondences gve at least a 2-parameter famly of F. Degeneraces Correspondences are degenerate f they satsfy more than one F. F 1 =0 and F 2 =0 (1 n). Ths page left empty

31 Robust lne estmaton Automatc Estmaton of Eppolar Geometry Ft a lne to 2D data contanng outlers c b a d There are two problems: () a lne ft to the data mn l d2 ; and, () a classfcaton of the data nto nlers (vald ponts) and outlers. Problem Statement RANdom SAmple Consensus (RANSAC) Gven Image par [Fschler and Bolles, 1981] Objectve Robust ft of a model to a data set S whch contans outlers. Algorthm Fnd The fundamental matr F and correspondences. Compute mage ponts Compute correspondences Compute eppolar geometry () Randomly select a sample of s data ponts from S and nstantate the model from ths subset. () Determne the set of data ponts S whch are wthn a dstance threshold t of the model. The set S s the consensus set of the sample and defnes the nlers of S. () If the sze of S (the number of nlers) s greater than some threshold T, re-estmate the model usng all the ponts n S and termnate. (v) If the sze of S s less than T, select a new subset and repeat the above. (v) After N trals the largest consensus set S s selected, and the model s re-estmated usng all the ponts n the subset S.

32 Robust ML estmaton Correlaton matchng A B C D A B C D An mproved ft by A better mnmal set Robust MLE: nstead of mn l d2 mn γ (d ) wth γ(e) = l { e 2 e 2 <t 2 nler t 2 e 2 t 2 outler Match each corner to most smlar lookng corner n the other mage Many wrong matches (10-50%), but enough to compute the fundamental matr. Feature etracton: Corner detecton Correspondences consstent wth eppolar geometry Interest ponts [Harrs] 100s of ponts per mage Use RANSAC robust estmaton algorthm Obtan correspondences and F Guded matchng by eppolar lne Typcally: fnal number of matches s about , wth dstance error of 0.2 pels.

33 Automatc Estmaton of F and correspondences Adaptve RANSAC Algorthm based on RANSAC [Torr] () Interest ponts: Compute nterest ponts n each mage. () Putatve correspondences: use cross-correlaton and promty. () RANSAC robust estmaton: Repeat (a) Select random sample of 7 correspondences (b) Compute F (c) Measure support (number of nlers) Choose the F wth the largest number of nlers. (v) MLE: re-estmate F from nler correspondences. (v) Guded matchng: generate addtonal matches. N =, sample count= 0. Whle N>sample count Repeat Choose a sample and count the number of nlers. Set ɛ =1 (number of nlers)/(total number of ponts) Set N from ɛ wth p =0.99. Increment the sample count by one. Termnate. e.g. for a sample sze of 4 Number of 1-ɛ Adaptve nlers N 6 2% % % % % % 43 How many samples? For probablty p of no outlers: N =log(1 p)/ log(1 (1 ɛ) s ) N, number of samples s, sze of sample set ɛ, proporton of outlers e.g. for p =0.95 Sample sze Proporton of outlers ɛ s 5% 10% 20% 25% 30% 40% 50% Ths page left empty

34 Two Vew Reconstructon Ambguty Part 2 : Three-vew and Multple-vew Geometry Computng a Metrc Reconstructon Gven: mage pont correspondences, compute a reconstructon: {P, P, X } wth = PX = P X Ambguty = PX = PH(H) 1 X = P X = P X = P H(H) 1 X = P X { P, P, X } s an equvalent Projectve Reconstructon. Reconstructon from two vews Metrc Reconstructon Gven only mage ponts and ther correspondence, what can be determned? projectve metrc Correct: angles, length ratos.

35 Algebrac Representaton of Metrc Reconstructon Drect Metrc Reconstructon Compute H {P 1, P 2,..., P m, X } Projectve Reconstructon H {P 1 M, P2 M,..., Pm M, XM } Metrc Reconstructon Use 5 or more 3D ponts wth known Eucldean coordnates to determne H Remanng ambguty s rotaton (3), translaton (3) and scale (1). Only 8 parameters requred to rectfy entre sequence (15 7 =8). How? Calbraton ponts: poston of 5 scene ponts. Scene geometry: e.g. parallel lnes/planes, orthogonal lnes/planes, length ratos. Auto-calbraton: e.g. camera aspect rato constant for sequence. Projectve Reconstructon Stratfed Reconstructon Gven a projectve reconstructon {P j, X }, compute a metrc reconstructon va an ntermedate affne reconstructon. () affne reconstructon: Determne the vector p whch defnes π.anaffne reconstructon s obtaned as {P j H P, H 1 P X } wth [ ] I 0 H P = p 1 () Metrc reconstructon: s obtaned as {P j A H A, H 1 A X A } wth [ ] K 0 H A = 0 1

36 Stratfed Reconstructon Stratfed reconstructon... Start wth a projectve reconstructon. Fnd transformaton to upgrade to affne reconstructon. Equvalent to fndng the plane at nfnty. Fnd transformaton to upgrade to metrc (Eucldean) reconstructon. Equvalent to fndng the absolute conc Equvalent to camera calbraton If camera calbraton s known then metrc reconstructon s possble. Metrc reconstructon mples knowledge of angles camera calbraton. () Apply the transformatons one after the other : Projectve transformaton reduce to affne ambguty [ ] I v 1 Affne transformaton reduce to metrc ambguty [ ] K 1 Metrc ambguty of scene remans Anatomy of a 3D projectve transformaton Reducton to affne General 3D projectve transformaton represented by a 4 4 matr. [ ] [ ][ ][ ] srk t sr t K I H = v = v 1 = metrc affne projectve Affne reducton usng scene constrants - parallel lnes

37 Reducton to affne Metrc Reconstructon Other scene constrants are possble : Ratos of dstances of ponts on lne (e.g. equally spaced ponts). Ratos of dstances on parallel lnes. Ponts le n front of the vewng camera. Constrans the poston of the plane at nfnty. Lnear-programmng problem can be used to set bounds on the plane at nfnty. Gves so-called quas-affne reconstructon. Reference : Hartley-Azores. Reducton to affne... Metrc Reconstructon... Common calbraton of cameras. Wth 3 or more vews, one can fnd (n prncple) the poston of the plane at nfnty. [ ] I Iteraton over the entres of projectve transform : v. 1 Not always relable. Generally reducton to affne s dffcult. Assume plane at nfnty s known. Wsh to make the step to metrc reconstructon. [ ] I Apply a transformaton of the form v 1 Lnear soluton ests n many cases.

38 The Absolute Conc Eample of calbraton Absolute conc s an magnary conc lyng on the plane at nfnty. Defned by Ω : X 2 + Y 2 + Z 2 =0 ; T =0 Contans only magnary ponts. Determnes the Eucldean geometry of the space. Represented by matr Ω=dag(1, 1, 1, 0). Image of the absolute conc (IAC) under camera P = K[R t] s gven by ω =(KK ) 1. Basc fact : Images taken wth a non-translatng camera: ω s unchanged under camera moton. Usng the nfnte homography Mosaced mage showng projectve transformatons () When a camera moves, the mage of a plane undergoes a projectve transformaton. () If we have affne reconstructon, we can compute the transformaton H of the plane at nfnty between two mages. () Absolute conc les on the plane at nfnty, but s unchanged by ths mage transformaton : (v) Transformaton rule for dual conc ω = ω 1. ω = H j ω H j (v) Lnear equatons on the entres of ω. (v) Gven three mages, solve for the entres of ω. (v) Compute K by Cholesk factorzaton of ω = KK.

39 Computaton of K Changng nternal camera parameters Calbraton matr of camera s found as follows : Compute the homographes (2D projectve transformatons) between mages. Form equatons ω = H j ω H j Solve for the entres of ω Cholesk factorzaton of ω = KK gves K. The prevous calbraton procedure (affne-to-metrc) may be generalzed to case of changng nternal parameters. See paper tomorrow gven by Agapto. Affne to metrc upgrade Prncpal s the same for non-statonary cameras once prncpal plane s known. H j s the nfnte homography (.e. va the plane at nfnty) between mages and j. May be computed drectly from affnely-correct camera matrces. Gven camera matrces Ths page left empty P =[M t ] ; P =[M t ] Infnte homography s gven by H = M M 1 Algorthm proceeds as for fed cameras.

40 Geometry of three vews Pont-lne-lne ncdence. L The Trfocal Tensor C l / l // / / C / C Correspondence l l The Trfocal Tensor Geometry of three vews... () Defned for three vews. () Plays a smlar rôle to Fundmental matr for two vews. () Unlke fundamental matr, trfocal tensor also relates lnes n three vews. (v) Med combnatons of lnes and ponts are also related. Let l (1) and l (2) be two lnes that meet n. General lne l back-projects to a plane l P. Four plane are l (1) P, l (2) P, l P and l P The four planes meet n a pont. L l // / / C C l / / C

41 The trfocal relatonshp Tensor Notaton... Four planes meet n a pont means determnant s zero. l (1) P l det (2) P l P =0 l P Ths s a lnear relatonshp n the lne coordnates. Also (less obvously) lnear n the entres of the pont = l (1) l (2). Ths s the trfocal tensor relatonshp. Lne coordnates Lne s represented by a vector l =(l 1,l 2,l 3 ) In new coordnate system ê j, lne has coordnate vector ˆl, ˆl = l H Lne coordnates transform accordng to H. Preserves ncdence relatonshp. Pont les on lne f : ˆl ˆ =(l H)(H 1 )=l =0 Termnology : l j transforms covarantly. Use lower ndces for covarant quanttes. Tensor Notaton Summaton notaton Pont coordnates. Consder bass set (e 1, e 2, e 3 ). Pont s represented by a vector =( 1, 2, 3 ). New bass : ê j = H j e. Wth respect to new bass represented by ˆ =(ˆ 1, ˆ 2, ˆ 3 ) where ˆ = H 1 Repeated nde n upper and lower postons mples summaton. Eample Incdence relaton s wrtten l =0. Transformaton of covarant and contravarant ndces Contravarant transformaton If bass s transformed accordng to H, then pont coordnates transform accordng to H 1. Termnology : transforms contravarantly. Use upper ndces for contravarant quanttes. Covarant transformaton ˆ j =(H 1 ) j ˆl j = H jl

42 More transformaton eamples Basc Trfocal constrant Camera mappng has one covarant and one contravarant nde : Pj. Transformaton rule ˆP = G 1 PF s ˆP j =(G 1 ) j s Pr s F r Basc relaton s a pont-lne-lne ncdence. L Trfocal tensor T jk Transformaton rule : has one covarant and two contravarant ndces. C l / l // / / C ˆT jk = F r (G 1 ) j s (H 1 ) k t Tr st / C Pont n mage 1 lnes l j and l k n mages 2 and 3. The ɛ tensor Basc Trfocal constrant... Tensor ɛ rst : L Defned for r, s, t =1,...,3 ɛ rst =0 unless r, s and t are dstnct =+1 f rst s an even permutaton of 123 = 1 f rst s an odd permutaton of 123 C l / / C l // / / C Related to the cross-product : c = a b c = ɛ jk a j b k. Let l r (1) and l s (2) be two lnes that meet n pont. The four lnes l r (1), l s (2), l j and l k back-project to planes n space. The four planes meet n a pont.

43 Dervaton of the basc three-vew relatonshp Lne Transfer Lne l back-projects to plane l P where P s -th row. Four planes are concdent f (l r (1) P r ) (l s (2) P s ) (l jp j ) (l kp k )=0 where 4-way wedge means determnant. Thus l r (1) l s (2) l jl k P r P s P j P k =0 Multply by constant ɛ rs ɛ rs gves ɛ rs l (1) r l (2) s l jl k ɛ rs P r P s P j P k =0 Intersecton (cross-product) of l (1) r and l (2) s s the pont : l (1) r l (2) s ɛ rs = Basc relaton s Interpretaton : l j l k T jk =0 Back projected ray from meets ntersecton of back-projected planes from l and l. Lne n space projects to lnes l and l and to a lne passng through. C L l / C / l // C / / Defnton of the trfocal tensor Lne transfer Basc relatonshp s Defne Pont-lne-lne relaton s T jk l j l k ɛ rs P r P s P j P k =0 ɛ rs P r P s P j P k = T jk l j l k T jk =0 s covarant n one nde (), contravarant n the other two. Denote l = l j l k T jk See that l =0when les on the projecton of the ntersecton of the planes. Thus l represents the transferred lne correspondng to l j We wrte l l j l k T jk Cross-product of the two sdes are equal : l r ɛ rs l j l k T jk =0 s Derved from basc relaton l j l k T jk by replacng by l r ɛ rs. and l k.

44 Pont transfer va a plane Contracton of trfocal tensor wth a lne C 2 C 1 mage 2 / l // C 3 Wrte H k = l j T jk Then k = H k. H k represents the homography from mage 1 to mage 3 va the plane of the lne l j. mage 1 X π / mage 3 C 2 mage 2 Lne l back-projects to a plane π. Ray from meets π n a pont X. Ths pont projects to pont n the thrd mage. For fed l, mappng s a homography. C 1 mage 1 / l // X mage 3 π / C 3 Pont-transfer and the trfocal tensor Three-pont correspondence If l s any lne through, then trfocal condton holds. l k ( l j T jk )=0 l j T jk must represent the pont k. k l j T jk Alternatvely (cross-product of the two sdes) l j ɛ krs r T jk =0 s Derved from basc relaton l j l k T jk by replacng l k by r ɛ krs. Gven a trple correspondence Choose any lnes l and l passng through and Trfocal condton holds C1 l j l k T jk =0 mage 2 / l C 2 // l // mage 1 L mage 3 π / / X C 3

45 Geometry of the three-pont correspondence Summary of ncdence relatons C 2 () Pont n frst vew, lnes n second and thrd mage 2 l j l k T jk =0 C1 / / l // l // X mage 1 L mage 3 π / C 3 () Pont n frst vew, pont n second and lne n thrd j l k ɛ jpr T pk =0 r () Pont n frst vew, lne n second and pont n thrd 4 choces of lnes 4 equatons. May also be wrtten as r ɛ rs t ɛ tuj T jk =0 Gves 9 equatons, only 4 lnearly ndependent. (v) Pont n three vews l j k ɛ kqs T jq j k ɛ jpr ɛ kqs T pq =0 s = 0 rs Summary of transfer formulas Degeneraces of lne transfer () Pont transfer from frst to thrd vew va a plane n the second. L k = l j T jk π π / () Pont transfer from frst to second vew va a plane n the thrd. l l / j = l k T jk () Lne transfer from thrd to frst vew va a plane n the second; or, from second to frst vew va a plane n the thrd. l = l j l k T jk e e / C 1 C 2 eppolar plane Degeneracy of lne transfer for correspondng eppolar lnes. When the lne les n an eppolar plane, ts poston can not be nferred from two vews. Hence t can not be transferred to a thrd vew.

46 Degeneracy of pont transfer Fndng eppolar lnes C 2 B 12 e21 l / B 23 To fnd the eppolar lne correspondng to a pont : C1 e 12 mage 2 mage 1 mage 3 X π / Transferrng ponts from mages 1 and 2 to mage 3 : Only ponts that can not be transferred are those ponts on the baselne between centres of cameras 1 and 2. For transfer wth fundamental matr, ponts n the trfocal plane can not be transferred. // C 3 Transfer to thrd mage va plane back-projected from l j k = T jk Eppolar lne satsfes l k k =0for each such k. For all l j l j l k T jk =0 Eppolar lne correspondng to found by solvng l j k ( T jk )=0 j l Result : Eppole s the common perpendcular to the null-space of all T jk. Contracton on a pont In k l jt jk wrte T jk = G jk Represents a mappng from lne l k to the pont j : k G jk l j =( T jk )l j As l j vares k traces out the projecton of the ray through. Eppolar lne n thrd mage. Eppole s the ntersecton of these lnes for varyng. Ths page left empty // C / l C / / C /

47 Where does ths formula come from? Etracton of camera matrces from trfocal tensor Formula for trfocal tensor T jk Notaton : P means omt row. = ɛ rs P r P s P j P k P =2det P j P k Eample, when = =det a j 1 aj 2 aj 3 aj 4 b k 1 b k 2 b k 3 b k 4 T jk = a j 1 bk 4 a j 4 bk 1 Formula for Trfocal tensor Etracton of the camera matrces. Trfocal tensor s ndependent of projectve transformaton. May assume that frst camera s [I 0] Other cameras are [A a 4 ] and [B b 4 ] Formula = a j bk 4 a j 4 bk T jk Note : a 4 and b 4 represent the eppoles : Centre of the frst camera s (0, 0, 0, 1). Eppole s mage of camera centre. 0 0 a 4 =[A a 4 ] 0 1 Basc formula T jk = a j bk 4 a j 4 bk Entres of T jk are quadratc n the entres of camera matrces. But f eppoles a j 4 and bk 4 are known, entres are lnear. Strategy : Estmate the eppoles. Solve lnearly for the remanng entres of A and B. 27 equatons n 18 unknowns. Eact formulae are possble, but not requred for practcal computaton.

48 Matr formulas nvolvng trfocal tensor Gven the trfocal tensor wrtten n matr notaton as [T 1, T 2, T 3 ]. () Retreve the eppoles e 21, e 31 Let u and v be the left and rght null vectors respectvely of T,.e. T u = 0, T v = 0. Then the eppoles are obtaned as the null-vectors to the followng 3 3 matrces Ths page left empty [u 1, u 2, u 3 ]e 21 = 0 [v 1, v 2, v 3 ]e 31 = 0 () Retreve the fundamental matrces F 12, F 13 F 12 =[e 21 ] [T 1, T 2, T 3 ]e 31 F 13 =[e 31 ] [T 1, T 2, T 3 ]e 21 () Retreve the camera matrces P, P (wth P =[I 0]) Normalze the eppoles to unt norm. Then P =[(I e 21 e 21 )[T 1, T 2, T 3 ]e 31 e 21 ] P =[ [T 1, T 2, T 3 ]e 21 e 31 ] Ths page left empty Ths page left empty

49 Summary of relatons Other pont or lne correspondences also yeld constrants on T. Computaton of the trfocal tensor Correspondence Relaton number of equatons three ponts j k ɛ jqu ɛ krv T qr =0 uv 4 two ponts, one lne j l rɛ jqu T qr =0 u 2 one pont, two lnes l q l rt =0 1 three lnes l p l q l rɛ T qr =0 w 2 Trlnear Relatons Lnear equatons for the trfocal tensor Solvng the equatons Gven a 3-pont correspondence The trfocal tensor relatonshp s j k ɛ jqu ɛ krv T qr =0 uv Relatonshp s lnear n the entres of T. each correspondence gves 9 equatons, 4 lnearly ndependent. T has 27 entres defned up to scale. 7 pont correspondences gve 28 equatons. Lnear or least-squares soluton for the entres of T. Gven 26 equatons we can solve for the 27 entres of T. Need 7 pont correspondences or 13 lne correspondences or some mture. Total set of equatons has the form Et = 0 Wth 26 equatons fnd an eact soluton. Wth more equatons, least-squares soluton.

50 Solvng the equatons... What are the constrants Soluton : Take the SVD : E = UDV. Soluton s the last column of V correspondng to smallest sngular value). Mnmzes Et subject to t =1. Normalzaton of data s essental. Some of the constrants are easy to fnd. () Each T jk must have rank 2. () Ther null spaces must le n a plane. () Ths gves 4 constrants n all. (v) 4 other constrants are not so easly formulated. Constrants Constrants through parametrzaton. T has 27 entres, defned only up to scale. Geometry only has 18 degrees of freedom. 3 camera matrces account for 3 11=33dof. Invarant under 3D projectve transformaton (15 dof). Total of 18 dof. T must satsfy several constrants to be a geometrcally vald trfocal tensor. To get good results, one must take account of these constrants (cf Fundamental matr case). Defne T n terms of a set of parameters. Only vald T s may be generated from parameters. Recall formula for T : T jk = a j bk 4 a j 4 bk Only vald trfocal tensors are generated by ths formula. Parameters are the entres a j and bk. Over-parametrzed : 24 parameters n all.

51 Algebrac Estmaton of T Mnmzaton knowng the eppoles... Smlar to the algebrac method of estmaton of F. Mnmze the algebrac error Et subject to () t =1 () t s the vector of entres of T. () T s of the form T jk = a j bk 4 a j 4 bk. Dffculty s ths constrant s a quadratc constrant n terms of the parameters. Mnmzaton problem Mnmze Et subject to t =1. becomes Mnmze EGp subject to Gp =1. Eactly the same problem as wth the fundamental matr. Lnear soluton usng the SVD. Reference : Hartley Royal Socety paper. Mnmzaton knowng the eppoles Algebrac estmaton of T Camera matrces [I 0], [A a 4 ] and [B b 4 ]. T jk = a j bk 4 a j 4 bk As wth fundamental matr, a 4 and b 4 are the eppoles of the frst mage. If a 4 and b 4 are known, then T s lnear n terms of the other parameters. We may wrte t = Gp Complete algebrac estmaton algorthm s () Fnd a soluton for T usng the normalzed lnear (7-pont) method () Estmated T wll not satsfy constrants. () Compute the two eppoles a 4 and b 4. (a) Fnd the left (respectvely rght) null spaces of each T jk. (b) Eppole s the common perpendcular to the null spaces. (v) Reestmate T by algebrac method assumng values for the eppoles. p s the matr of 18 remanng entres of camera matrces A and B. t s the 27-vector of entres of T. G s a matr.

52 Iteratve Algebrac Method Fnd the trfocal tensor T that mnmzes Et subject to t =1and T jk = a j bk 4 a j 4 bk. Concept : Vary eppoles a 4 and b 4 to mnmze the algebrac error Et = EGp. Remark : Each choce of eppoles a 4 and b 4 defnes a mnmmum error vector EGp as above. Use Levenberg-Marquardt method to mnmze ths error. Smple 6 27 mnmzaton problem. 6 nputs the entres of the two eppoles 27 outputs the algebrac error vector Et = EGp. Each step requres estmaton of p usng algebrac method. Ths page left empty Ths page left empty Ths page left empty

53 Reconstructon from three vews Automatc Estmaton of a Projectve Reconstructon for a Sequence Gven: mage pont correspondences 1 2 3, compute a projectve reconstructon: {P 1, P 2, P 3, X } wth j = P j X What s new? X verfy correspondences O O 1 3-vew tensor: the trfocal tensor Compute from 6 mage pont correspondences. Automatc algorthm smlar to F. [Torr & Zsserman] 2 O Outlne Automatc Estmaton of the trfocal tensor and correspondences () Parwse matches: Compute pont matches between vew pars usng robust F estmaton. () Putatve correspondences: over three vews from two vew matches. () RANSAC robust estmaton: Repeat (a) Select random sample of 6 correspondences frst frame of vdeo () Projectve reconstructon: 2-vews, 3-vews, N-vews () Obtanng correspondences over N-vews (b) Compute T (1 or 3 solutons) (c) Measure support (number of nlers) Choose the T wth the largest number of nlers. (v) MLE: re-estmate T from nler correspondences. (v) Guded matchng: generate addtonal matches.

54 Projectve Reconstructon for a Sequence Interest ponts computed for each frame () Compute all 2-vew reconstructons for consecutve frames. () Compute all 3-vew reconstructons for consecutve frames. () Etend to sequence by herarchcal mergng: (v) Bundle-adjustment: mnmze reprojecton error mn P j X ponts j frames d ( j ) 2, Pj X (v) Automatc algorthm [Ftzgbbon & Zsserman ] About 500 ponts per frame frst frame of vdeo Cameras and Scene Geometry for an Image Sequence Pont trackng: Correlaton matchng Gven vdeo frst frame of vdeo Pont correspondence (trackng). Projectve Reconstructon % wrong matches frst frame of vdeo

55 Pont trackng: Eppolar-geometry guded matchng Reconstructon from Pont Tracks Compute 3D ponts and cameras from pont tracks frst frame of vdeo Compute F so that matches consstent wth eppolar geometry. Many fewer false matches, but stll a loose constrant. a frame of the vdeo Herarchcal mergng of sub-sequences. Bundle adjustment. Pont trackng: Trfocal tensor guded matchng Applcaton I: Graphcal Models Compute VRML pecewse planar model frst frame of vdeo Compute trfocal tensor so that matches consstent wth 3-vews. Tghter constrant, so even fewer false matches. Three vews s the last sgnfcant mprovement.

56 Eample II: Etended Sequence Applcaton II: Augmented Realty 140 frames of a 340 frame sequence Usng computed cameras and scene geometry, nsert vrtual objects nto the sequence. a frame of the vdeo a frame of the vdeo 330 frames Metrc Reconstructon 3D Inserton 140 frames of a 340 frame sequence a frame of the vdeo a frame of the vdeo