14 The Trfocal Tensor The trfocal tensor plays an analogous role n three vews to that played by the fundamental matrx n two. It encapsulates all the (projectve) geometrc relatons between three vews that are ndependent of scene structure. We begn ths chapter wth a smple ntroducton to the man geometrc and algebrac propertes of the trfocal tensor. A formal development of the trfocal tensor and ts propertes nvolves the use of tensor notaton. To start, however, t s convenent to use standard vector and matrx notaton, thus obtanng some geometrc nsght nto the trfocal tensor wthout the addtonal burden of dealng wth a (possbly) unfamlar notaton. The use of tensor notaton wll therefore be deferred untl secton 14.2. The three prncpal geometrc propertes of the tensor are ntroduced n secton 14.1. These are the homography between two of the vews nduced by a plane back-projected from a lne n the other vew; the relatons between mage correspondences for ponts and lnes whch arse from ncdence relatons n 3-space; and the retreval of the fundamental and camera matrces from the tensor. The tensor may be used to transfer ponts from a correspondence n two vews to the correspondng pont n a thrd vew. The tensor also apples to lnes, and the mage of a lne n one vew may be computed from ts correspondng mages n two other vews. Transfer s descrbed n secton 14.3. The tensor only depends on the moton between vews and the nternal parameters of the cameras and s defned unquely by the camera matrces of the vews. However, t can be computed from mage correspondences alone wthout requrng knowledge of the moton or calbraton. Ths computaton s descrbed n chapter 15. 14.1 The geometrc bass for the trfocal tensor There are several ways that the trfocal tensor may be approached, but n ths secton the startng pont s taken to be the ncdence relatonshp of three correspondng lnes. 355
356 14 The Trfocal Tensor L l l l Fg. 14.1. A lne L n 3-space s maged as the correspondng trplet l l l n three vews ndcated by ther centres,,,, and mage planes. onversely, correspondng lnes back-projected from the frst, second and thrd mages all ntersect n a sngle 3D lne n space. Incdence relatons for lnes. Suppose a lne n 3-space s maged n three vews, as n fgure 14.1, what constrants are there on the correspondng mage lnes? The planes back-projected from the lnes n each vew must all meet n a sngle lne n space, the 3D lne that projects to the matched lnes n the three mages. Snce n general three arbtrary planes n space do not meet n a sngle lne, ths geometrc ncdence condton provdes a genune constrant on sets of correspondng lnes. We wll now translate ths geometrc constrant nto an algebrac constrant on the three lnes. We denote a set of correspondng lnes as l l l. Let the camera matrces for the three vews be P =[I 0], as usual, and P =[A a 4 ], P =[B b 4 ], where A and B are 3 3 matrces, and the vectors a and b are the -th columns of the respectve camera matrces for = 1,...,4. a 4 and b 4 are the eppoles n vews two and three respectvely, arsng from the frst camera. These eppoles wll be denoted by e and e throughout ths chapter, wth e = P, e = P,where s the frst camera centre. (For the most part we wll not be concerned wth the eppoles between the second and thrd vews). A and B are the nfnte homographes from the frst to the second and thrd cameras respectvely. As has been seen n chapter 8, any set of three cameras s equvalent to a set wth P =[I 0] under projectve transformatons of space. In ths chapter we wll be concerned wth propertes (such as mage coordnates and 3D ncdence relatons) that are nvarant under 3D projectve transforms, so we are free to choose the cameras n ths form. Now, each mage lne back-projects to a plane, as shown n fgure 14.1. From result
14.1 The geometrc bass for the trfocal tensor 357 7.1(p186) these three planes are ( ) ( l A π = P l = π = P l l = 0 a 4 l ) π = P l = ( B l b 4 l ). Snce the three mage lnes are derved from a sngle lne n space, t follows that these three planes are not ndependent but must meet n ths common lne n 3-space. Ths ntersecton constrant can be expressed algebracally by the requrement that the 4 3 matrx M =[π, π, π ] has rank 2. Ths may be seen as follows. Ponts on the lne of ntersecton may be represented as X = αx 1 + βx 2, wth X 1 and X 2 lnearly ndependent. Such ponts le on all three planes and so π X = π X = π X =0. It follows that M X = 0. onsequently M has a 2-dmensonal null-space snce M X 1 = 0 and M X 2 = 0. Ths ntersecton constrant nduces a relaton amongst the mage lnes l, l, l. Snce the rank of M s 2, there s a lnear dependence between ts columns m. Denotng [ ] l A l B l M =[m 1, m 2, m 3 ]= 0 a 4 l b 4 l the lnear relaton may be wrtten m 1 = αm 2 + βm 3. Then notng that the bottom left hand element of M s zero, t follows that α = k(b 4 l )andβ = k(a 4 l )for some scalar k. Applyng ths to the top 3-vectors of each column shows that (up to a homogeneous scale factor) l =(b 4 l )A l (a 4 l )B l =(l b 4 )A l (l a 4 )B l. The -th coordnate l of l may therefore be wrtten as l = l (b 4 a )l l (a 4 b )l = l (a b 4 )l l (a 4 b )l and ntroducng the notaton the ncdence relaton can be wrtten T = a b 4 a 4 b (14.1) l = l T l. (14.2) Defnton 14.1. The set of three matrces {T 1, T 2, T 3 } consttute the trfocal tensor n matrx notaton. We ntroduce a further notaton 1. Denotng the ensemble of the three matrces T by [T 1, T 2, T 3 ], or more brefly [T ], ths last relaton may be wrtten as l = l [T 1, T 2, T 3 ]l (14.3) where l [T 1, T 2, T 3 ]l s understood to represent the vector (l T 1 l, l T 2 l, l T 3 l ). 1 Ths notaton s somewhat cumbersome, and ts meanng s not qute self-evdent. It s for ths reason that tensor notaton s ntroduced n secton 14.2.
358 14 The Trfocal Tensor Of course there s no ntrnsc dfference between the three vews, and so by analogy wth (14.3) there wll exst smlar relatons l = l [T ]l and l = l [T ]l. The three tensors [T ], [T ] and [T ] exst, but are dstnct. In fact, although all three tensors may be computed from any one of them, there s no very smple relatonshp between them. Thus, n fact there are three trfocal tensors exstng for a gven trple of vews. Usually one wll be content to consder only one of them. However, a method of computng the other trfocal tensors [T ] and [T ] gven [T ] s outlned n exercse (v) on page 377. Note that (14.3) s a relatonshp between mage coordnates only, not nvolvng 3D coordnates. Hence (as remarked prevously), although t was derved under the assumpton of a canoncal camera set (that s P =[I 0]), the value of the matrx elements [T ] s ndependent of the form of the cameras. The partcular smple formula (14.1) for the trfocal tensor gven the camera matrces holds only n the case where P =[I 0], but a general formula (16.12 p404) for the trfocal tensor correspondng to any three cameras wll be derved later. Degrees of freedom. The trfocal tensor conssts of three 3 3 matrces, and thus has 27 elements. There are therefore 26 ndependent ratos apart from the (common) overall scalng of the matrces. However, the tensor has only 18 ndependent degrees of freedom. In other words once 18 parameters are specfed, all 27 elements of the tensor are determned up to a common scale. The number of degrees of freedom may be computed as follows. Each of 3 camera matrces has 11 degrees of freedom, whch makes 33 n total. However, 15 degrees of freedom must be subtracted to account for the projectve world frame, thus leavng 18 degrees of freedom. The tensor therefore satsfes 26 18 = 8 ndependent algebrac constrants. We return to ths pont n chapter 15. 14.1.1 Homographes nduced by a plane A fundamental geometrc property encoded n the trfocal tensor s the homography between the frst vew and the thrd nduced by a lne n the second mage. Ths s llustrated n fgure 14.2 and fgure 14.3. A lne n the second vew defnes (by back-projecton) a plane n 3-space, and ths plane nduces a homography between the frst and thrd vews. We now derve the algebrac representaton of ths geometry n terms of the trfocal tensor. The homography map between the frst and thrd mages, defned by the plane π n fgure 14.2 and fgure 14.3, may be wrtten as x = Hx and (1.6 p15) l = H l respectvely. Notce that the three lnes l, l and l n fgure 14.3 are a correspondng lne trple, the projectons of the 3D lne L. Therefore, they satsfy the lne ncdence relatonshp l = l T l of (14.2). omparson of ths formula and l = H l shows that H =[h 1, h 2, h 3 ] wth h = T l.
14.1 The geometrc bass for the trfocal tensor 359 mage 2 l x x mage 1 X π mage 3 Fg. 14.2. Pont transfer. A lne l n the second vew back-projects to a plane π n 3- space. A pont x n the frst mage defnes a ray n 3-space whch ntersects π n the pont X. Ths pont X s then maged as the pont x n the thrd vew. Thus, any lne l nduces a homography between the frst and thrd vews, defned by ts back-projected plane π. mage 2 l l l mage 1 L π mage 3 Fg. 14.3. Lne transfer. The acton on lnes of the homography defned by fgure 14.2 may smlarly be vsualzed geometrcally. A lne, l, n the frst mage defnes a plane n 3-space, whch ntersects π n the lne L. Ths lne L s then maged as the lne l n the thrd vew. Thus, H defned by the above formula represents the (pont) homography H 13 between vews one and three specfed by the lne l n vew two. The second and thrd vews play smlar roles, and the homography between the frst and second vews defned by a lne n the thrd can be derved n a smlar manner. These deas are formalzed n the followng result. Result 14.2. The homography from the frst to the thrd mage nduced by a lne l n the second mage (see fgure 14.2) s gven by x = H 13 (l ) x, where H 13 (l )=[T 1, T 2, T 3 ]l. Smlarly, a lne l n the thrd mage defnes a homography x = H 12 (l ) x from the frst to the second vews, gven by H 12 (l )=[T 1, T 2, T 3 ]l.
360 14 The Trfocal Tensor Once ths mappng s understood the algebrac propertes of the tensor are s- traghtforward and can easly be generated. In the followng secton we deduce a number of ncdence relatons between ponts and lnes based on (14.3) and result 14.2. 14.1.2 Pont and lne ncdence relatons It s easy to deduce varous lnear relatonshps between lnes and ponts n three mages nvolvng the trfocal tensor. We have seen one such relatonshp already, namely (14.3). Ths relaton holds only up to scale snce t nvolves homogeneous quanttes. We may elmnate the scale factor by takng the vector cross product of both sdes, whch must be zero. Ths leads to the formula (l [T 1, T 2, T 3 ]l )[l] = 0, (14.4) where we have used the matrx [l] to denote the cross product (see (A3.4 p554)), or more brefly (l [T ]l )[l] = 0. Note the symmetry between l and l swappng the roles of these two lnes s accounted for by transposng each T, resultng n a relaton (l [T ]l )[l] = 0. onsder agan fgure 14.3. Now, a pont x on the lne l must satsfy x l = x l = 0 (usng upper ndces for the pont coordnates, foreshadowng the use of tensor notaton). Snce l = l T l, ths may be wrtten as l ( x T )l = 0 (14.5) (note that ( x T ) s smply a 3 3 matrx). Ths s an ncdence relaton n the frst mage: the relatonshp wll hold for a pont lne lne correspondence that s whenever some 3D lne L maps to l and l n the second and thrd mages, and to a lne passng through x n the frst mage. An mportant equvalent defnton of a pont lne lne correspondence for whch (14.5) holds results from an ncdence relaton n 3-space thereexstsa3dpontx mappng to x n the frst mage, and to ponts on the lnes l and l n the second and thrd mages as shown n fgure 14.4(a). From result 14.2 we may obtan relatons nvolvng ponts x and x n the second and thrd mages. onsder a pont lne pont correspondence as n fgure 14.4(b) so that x = H 13 (l ) x =[T 1 l, T 2 l, T 3 l ] x =( x T )l whch s vald for any lne l passng through x n the second mage. The homogeneous scale factor may be elmnated by (post-)multplyng the transpose of both sdes by [x ] to gve x [x ] = l ( x T )[x ] = 0, (14.6)
14.1 The geometrc bass for the trfocal tensor 361 L X l x x x l (a) pont lne lne L X x x x l (b) pont lne pont X x x x (c) pont pont pont Fg. 14.4. Incdence relatons. (a) onsder a 3-vew pont correspondence x x x. If l and l are any two lnes through x and x respectvely, then x l l forms a pont lne lne correspondence, correspondng to a 3D lne L. onsequently, (14.5) holds for any choce of lnes l through x and l through x. (b) The space pont X s ncdent wth the space lne L. Ths defnes an ncdence relaton x l x between ther mages. (c) The correspondence x x x arsng from the mage of a space pont X.
362 14 The Trfocal Tensor () Lne lne lne correspondence l [T 1, T 2, T 3 ]l = l () Pont lne lne correspondence or ( l [T 1, T 2, T 3 ]l ) [l] = 0 l ( x T )l = 0 for a correspondence x l l () Pont lne pont correspondence l ( x T )[x ] = 0 for a correspondence x l x (v) Pont pont lne correspondence [x ] ( x T )l = 0 for a correspondence x x l (v) Pont pont pont correspondence [x ] ( x T )[x ] = 0 3 3 Table 14.1. Summary of trfocal tensor ncdence relatons usng matrx notaton. A smlar analyss may be undertaken wth the roles of the second and thrd mages swapped. Fnally, for a 3-pont correspondence as shown n fgure 14.4(c), there s a relaton [x ] ( x T )[x ] = 0 3 3. (14.7) Proof The lne l n (14.6) passes through x and so may be wrtten as l = x y =[x ] y for some pont y on l. onsequently, from (14.6) l ( x T )[x ] = y [x ] ( x T )[x ] = 0. However, the relaton (14.6) s true for all lnes l through x and so s ndependent of y. The relaton (14.7) then follows. The varous relatonshps between lnes and ponts n three vews are summarzed n table 14.1, and ther propertes are nvestgated further n secton 14.2.1, once tensor notaton has been ntroduced. Note that there are no relatons lsted for pont lne lne correspondence n whch the pont s n the second or thrd vew. Such smple relatons do not exst n terms of the trfocal tensor n whch the frst vew s the specal vew. It s also worth notng that satsfyng an mage ncdence relaton does not guarantee ncdence n 3-space, as llustrated n fgure 14.5. We now begn to extract the two-vew geometry, the eppoles and fundamental matrx, from the trfocal tensor.
14.1 The geometrc bass for the trfocal tensor 363 L X x x l Fg. 14.5. Non-ncdent confguraton. The maged ponts and lnes of ths confguraton satsfy the pont lne pont ncdence relaton of table 14.1. However, the space pont X and lne L are not ncdent. ompare wth fgure 14.4. 14.1.3 Eppolar lnes A specal case of a pont lne lne correspondence occurs when the plane π backprojected from l s an eppolar plane wth respect to the frst two cameras, and hence passes through the camera centre of the frst camera. Suppose X s a pont on the plane π ; then the ray defned by X and les n ths plane, and l s the eppolar lne correspondng to the pont x, the mage of X. Ths s shown n fgure 14.6. The plane π back-projected from a lne l n the thrd mage wll ntersect the plane π n a lne L. Further, snce the ray correspondng to x les entrely n the plane π t must ntersect the lne L. Ths gves a 3-way ntersecton between the ray and planes back-projected from pont x and lnes l and l, and so they consttute a pont lne lne correspondence, satsfyng l ( x T )l = 0. The mportant pont now s that ths s true for any lne l, and t follows that l ( x T )=0. The same argument holds wth the roles of l and l reversed. To summarze: Result 14.3. If x s a pont and l and l are the correspondng eppolar lnes n the second and thrd mages, then l ( x T )=0 and ( x T )l = 0. onsequently, the eppolar lnes l and l correspondng to x may be computed as the left and rght null-vectors of the matrx x T. As the pont x vares, the correspondng eppolar lnes vary, but all eppolar lnes n one mage pass through the eppole. Thus, one may compute ths eppole by computng the ntersecton of the eppolar lnes for varyng values of x. Three convenent choces of x are the ponts represented by homogeneous coordnates (1, 0, 0),(0, 1, 0) and (0, 0, 1), wth x T equal to T 1, T 2 and T 3 respectvely for these three choces of x. From ths we deduce the followng mportant result:
364 14 The Trfocal Tensor π l x X l e Fg. 14.6. If the plane π defned by l s an eppolar plane for the frst two vews, then any lne l n the thrd vew gves a pont lne lne ncdence. Result 14.4. The eppole e n the second mage s the common ntersecton of the eppolar lnes represented by the left null-vectors of the matrces T, =1,...,3. Smlarly the eppole e s the common ntersecton of lnes represented by the rght null-vectors of the T. Note that the eppoles nvolved here are the eppoles n the second and thrd mages correspondng to the frst mage centre. The usefulness of ths result may not be apparent at present. However, t wll be seen below that t s an mportant step n computng the camera matrces from the trfocal tensor, and n chapter 15 n the accurate computaton of the trfocal tensor. Algebrac propertes of the T matrces. Ths secton has establshed a number of algebrac propertes of the T matrces. We summarze these here: Each matrx T has rank 2. Ths s evdent from (14.1) snce T = a e e b s the sum of two outer products. The rght null-vector of T s l = e b, and s the eppolar lne n the thrd vew for the pont x =(1, 0, 0),(0, 1, 0) or (0, 0, 1),as =1, 2 or 3 respectvely. The eppole e s the common ntersecton of the eppolar lnes l for =1, 2, 3. The left null-vector of T s l = e a, and s the eppolar lne n the second vew for the pont x =(1, 0, 0),(0, 1, 0) or (0, 0, 1),as =1, 2 or 3 respectvely. The eppole e s the common ntersecton of the eppolar lnes l for =1, 2, 3. The sum of the matrces M(x) =( x T ) also has rank 2. The rght null-vector of M(x) s the eppolar lne l of x n the thrd vew, and ts left null-vector s the eppolar lne l of x n the second vew. Its worth emphaszng agan that although a partcular canoncal form of the camera matrces P, P and P s used n the dervaton, the eppolar propertes of the T matrces are ndependent of ths choce.
14.1 The geometrc bass for the trfocal tensor 365 14.1.4 Extractng the fundamental matrces It s smple to compute the fundamental matrces F 21 and F 31 between the frst 1 and the other vews from the trfocal tensor. It was seen n secton 8.2.1(p222) that the eppolar lne correspondng to some pont can be derved by transferrng the pont to the other vew va a homography and jonng the transferred pont to the eppole. onsder a pont x n the frst vew. Accordng to fgure 14.2 and result 14.2, a lne l n the thrd vew nduces a homography from the frst to the second vew gven by x =([T 1, T 2, T 3 ]l ) x. The eppolar lne correspondng to x s then found by jonng x to the eppole e. Ths gves l =[e ] ([T 1, T 2, T 3 ]l ) x, from whch t follows that F 21 =[e ] [T 1, T 2, T 3 ]l. Ths formula holds for any vector l, but t s mportant to choose l to avod the degenerate condton where l les n the null-space of any of the T. A good choce s e snce as has been seen e s perpendcular to the rght null-space of each T. Ths gves the formula F 21 =[e ] [T 1, T 2, T 3 ]e. (14.8) A smlar formula holds for F 31 =[e ] [T 1, T 2, T 3 ]e. 14.1.5 Retrevng the camera matrces It was remarked that the trfocal tensor, snce t expresses a relatonshp between mage enttes only, s ndependent of 3D projectve transformatons. onversely, ths mples that the camera matrces may be computed from the trfocal tensor only up to a projectve ambguty. It wll now be shown how ths may be done. Just as n the case of reconstructon from two vews, because of the projectve ambguty, the frst camera may be chosen as P =[I 0]. Now, snce F 21 s known (from (14.8)), we can make use of result 8.9(p235) to derve the form of the second camera as P =[[T 1, T 2, T 3 ]e e ] and the camera par {P, P } then has the fundamental matrx F 21. It mght be thought that the thrd camera could be chosen n a smlar manner as P = [[T 1, T 2, T 3 ]e e ], but ths s ncorrect. Ths s because the two camera pars {P, P } and {P, P } do not necessarly defne the same projectve world frame; although each par s correct by tself, the trple {P, P, P } s nconsstent. The thrd camera cannot be chosen ndependently of the projectve frame of the frst two. To see ths, suppose the camera par {P, P } s chosen and ponts X reconstructed from ther mage correspondences x x. Then the coordnates of X are specfed n the projectve world frame defned by by {P, P }, and a consstent camera 1 The fundamental matrx F 21 satsfes x F 21 x = 0 for correspondng ponts x x. The subscrpt notaton refers to fgure 14.8.
366 14 The Trfocal Tensor Gven the trfocal tensor wrtten n matrx notaton as [T 1, T 2, T 3 ]. () Retreve the eppoles e, e Let u and v be the left and rght null-vectors respectvely of T,.e. u T = 0, T v = 0. Then the eppoles are obtaned as the null-vectors to the followng 3 3 matrces: e [u 1, u 2, u 3 ]=0 and e [v 1, v 2, v 3 ]=0. () Retreve the fundamental matrces F 21, F 31 F 21 =[e ] [T 1, T 2, T 3 ]e and F 31 =[e ] [T 1, T 2, T 3 ]e. () Retreve the camera matrces P, P (wth P =[I 0]) Normalze the eppoles to unt norm. Then P =[[T 1, T 2, T 3 ]e e ]andp =[(e e I)[T 1, T 2, T 3 ]e e ]. Algorthm 14.1. Summary of F and P retreval from the trfocal tensor. Note, F 21 and F 31 are determned unquely. However, P and P are determned only up to a common projectve transformaton of 3-space. P may be computed from the correspondences X x. learly, P depends on the frame defned by {P, P }. However, t s not necessary to explctly reconstruct 3D structure, a consstent camera trplet can be recovered from the trfocal tensor drectly. The par of camera matrces P =[I 0]andP =[[T 1, T 2, T 3 ]e e ] are not the only ones compatble wth the gven fundamental matrx F 21. Accordng to (8.10 p238), the most general form for P s P =[[T 1, T 2, T 3 ]e + e v λe ] for some vector v and scalar λ. A smlar choce holds for P. To fnd a trple of camera matrces compatble wth the trfocal tensor, we need to fnd the correct values of P and P from these famles so as to be compatble wth the form (14.1) of the trfocal tensor. Because of the projectve ambguty, we are free to choose P =[[T 1, T 2, T 3 ]e e ], thus a = T e. Ths choce fxes the projectve world frame so that P s now defned unquely (up to scale). Then substtutng nto (14.1) (observng that a 4 = e and b 4 = e ) T = T e e e b from whch t follows that e b = T (e e I). Snce the scale may be chosen such that e = e e = 1, we may multply on the left by e and transpose to get b =(e e I)T e so P = [(e e I)[T 1, T 2, T 3 ]e e ]. A summary of the steps nvolved n extractng the camera matrces from the trfocal tensor s gven n algorthm 14.1.
14.2 The trfocal tensor and tensor notaton 367 We have seen that the trfocal tensor may be computed from the three camera matrces, and that conversely the three camera matrces may be computed, up to projectve equvalence, from the trfocal tensor. Thus, the trfocal tensor completely captures the three cameras up to projectve equvalence. 14.2 The trfocal tensor and tensor notaton The style of notaton that has been used up to now for the trfocal tensor s derved from the standard matrx vector notaton. Snce a matrx has two ndces only, t s possble to dstngush between the two ndces usng the devces of matrx transposton and rght or left multplcaton, and n dealng wth matrces and vectors, one can do wthout wrtng the ndces explctly. Because the trfocal tensor has three ndces, nstead of the two ndces that a matrx has, t becomes ncreasngly cumbersome to persevere wth ths style of matrx notaton, and we now turn to usng standard tensor notaton when dealng wth the trfocal tensor. For those unfamlar wth tensor notaton a gentle ntroducton s gven n appendx 1(p545). Ths appendx should be read before proceedng wth ths chapter. Image ponts and lnes are represented by homogeneous column and row 3-vectors, respectvely,.e. x =(x 1,x 2,x 3 ) and l =(l 1,l 2,l 3 ). The j-th entry of a matrx A s denoted by a j, ndex beng the contravarant (row) ndex and j beng the covarant (column) ndex. We observe the conventon that ndces repeated n the contravarant and covarant postons mply summaton over the range (1,...,3) of the ndex. For example, the equaton x = Ax s equvalent to x = j a j xj, whch may be wrtten x = a j xj. We begn wth the defnton of the trfocal tensor gven n (14.1 p357). Usng tensor notaton, ths becomes T jk = a j bk 4 a j 4 bk. (14.9) The postons of the ndces n T jk (two contravarant and one covarant) are dctated by the postons of the ndces on the rght sde of the equaton. Thus, the trfocal tensor s a mxed contravarant covarant tensor. In tensor notaton, the basc ncdence relaton (14.3 p357) becomes l = l jl kt jk. (14.10) Note that when multplyng tensors the order of the entres does not matter, n contrast wth standard matrx notaton. For nstance the rght sde of the above expresson s l jl kt jk = l jl kt jk = l jt jk l k = l jt jk l k. j,k j,k The homography maps of fgure 14.2 and fgure 14.3 may be deduced from the
368 14 The Trfocal Tensor ncdence relaton (14.10). In the case of the plane defned by back-projectng the lne l, l = l jl kt jk = l k(l jt jk )=l k h k where h k = l jt jk and h k are the elements of the homography matrx H. Ths homography maps ponts between the frst and thrd vew as x k = h k x. Note that the homography s obtaned from the tensor by contracton wth a lne (.e. a summaton over one contravarant (upper) ndex of the tensor, and the covarant (lower) ndex of the lne),.e. l extracts a 3 3 matrx from the tensor thnk of the trfocal tensor as an operator whch takes a lne and produces a homography matrx. Table 14.2 summarzes the defnton and transfer propertes of the trfocal tensor. A par of partcularly mportant tensors are ɛ jk and ts contravarant counterpart ɛ jk, defned n secton A1.1(p546). Ths tensor s used to represent the vector product. For nstance, the lne jonng two ponts x and y j s equal to the cross product x y j ɛ jk = l k, and the skew-symmetrc matrx [x] s wrtten as x ɛ rs n tensor notaton. It s now relatvely straghtforward to wrte down the basc ncdence results nvolvng the trfocal tensor gven n table 14.1. The results are summarzed n table 14.3. In ths table, a notaton such as 0 r represents an array of zeros. The form of the relatons n table 14.3 s more easly understood f one observes that three ndces, j and k n T jk correspond to enttes n the frst, second and thrd vews respectvely. Thus for nstance a partal expresson such as l j T jk cannot occur, because the ndex j belongs to the second vew, and hence does not belong on the lne l n the thrd vew. Repeated ndces (ndcatng summaton) must occur once as a contravarant (upper) ndex and once as a covarant (lower) ndex. Thus, we cannot wrte x j T jk, snce the ndex j occurs twce n the upper poston. Thnk of the ɛ tensor as beng used to rase or lower ndces, for nstance by replacng l j by x ɛ jk. However, ths may not be done arbtrarly, as ponted out n exercse (x) on page 378. 14.2.1 The trlneartes The ncdence relatons n table 14.3 are trlnear relatons or trlneartes n the coordnates of the mage elements (ponts and lnes). Tr- snce every monomal n the relaton nvolves a coordnate from each of the three mage elements nvolved; and lnear because the relatons are lnear n each of the algebrac enttes (.e. the three arguments of the tensor). For example n the pont pont pont relaton, x (x j ɛ jpr )(x k ɛ kqs )T pq =0 rs, suppose both x 1 and x 2 satsfy the relaton, then so does x = αx 1 + βx 2,.e. the relaton s lnear n ts frst argument. Smlarly,
14.2 The trfocal tensor and tensor notaton 369 Defnton. The trfocal tensor T s a valency 3 tensor T jk wth two contravarant and one covarant ndces. It s represented by a homogeneous 3 3 3 array (.e. 27 elements). It has 18 degrees of freedom. omputaton from camera matrces. If the canoncal 3 4 camera matrces are P =[I 0], P =[a j], P =[b j] then T jk = a j bk 4 a j 4 bk. See (16.12 p404) for computaton from three general camera matrces. Lne transfer from correspondng lnes n the second and thrd vews to the frst. Transfer by a homography. l = l jl kt jk () Pont transfer from frst to thrd vew va a plane n the second The contracton l j T jk s a homography mappng between the frst and thrd vews nduced by a plane defned by the back-projecton of the lne l n the second vew. x k = h k x where h k = l j T jk () Pont transfer from frst to second vew va a plane n the thrd The contracton l k T jk s a homography mappng between the frst and second vews nduced by a plane defned by the back-projecton of the lne l n the thrd vew. x j = h j x where h j k = l k T jk Table 14.2. Defnton and transfer propertes of the trfocal tensor. the relaton s lnear n the second and thrd argument. Ths mult-lnearty s a standard property of tensors, and follows drectly from the form x l j l k T jk = 0 whch s a contracton of the tensor over all three of ts ndces (arguments). We wll now descrbe the pont pont pont trlneartes n more detal. There are nne of these trlneartes arsng from the three choces of r and s. Geometrcally these trlneartes arse from specal choces of the lnes n the second and thrd mage for the pont lne lne relaton (see fgure 14.4(a)). hoosng r = 1, 2or3 corresponds to a lne parallel to the mage x-axs, parallel to the mage y-axs, or through the mage coordnate orgn (the pont (0, 0, 1) ), respectvely. For example, choosng r = 1 and expandng x j ɛ jpr results n l p = x j ɛ jp1 =(0, x 3,x 2 ) whch s a horzontal lne n the second vew through x (snce ponts of the form
370 14 The Trfocal Tensor () Lne lne lne correspondence () Pont lne lne correspondence (l r ɛ rs )l jl kt jk =0 s x l jl kt jk =0 () Pont lne pont correspondence (v) Pont pont lne correspondence x l j(x k ɛ kqs )T jq =0 s x (x j ɛ jpr )l k T pk =0 r (v) Pont pont pont correspondence x (x j ɛ jpr )(x k ɛ kqs )T pq =0 rs Table 14.3. Summary of trfocal tensor ncdence relatons the trlneartes. y =(x 1 + λ, x 2,x 3 ) satsfy y l =0foranyλ). Smlarly, choosng s = 2 n the thrd vew results n the vertcal lne through x l q = x k ɛ kq2 =(x 3, 0, x 1 ) and the trlnear pont relaton expands to 0 = x x j x k ɛ jp1 ɛ kq2 T pq = x [ x 3 (x 3 T 21 x 1 T 23 )+x 2 (x 3 T 31 x 1 T 33 )]. Of these nne trlneartes, four are lnearly ndependent. Ths means that from a bass of four trlneartes all nne can be generated by lnear combnatons. The four degrees of freedom may be traced back to those of the pont-lne-lne relaton = 0 and are counted as follows. There s a one-parameter famly of lnes through x n the thrd vew. If m and n are two members of ths famly, then any other lne through x can be obtaned from a lnear combnaton of these: x l j l k T jk l = αm + βn. The ncdence relaton s lnear n l, so that gven l jm kt jk x = 0 l jn kt jk x = 0 then the ncdence relaton for any other lne l can be generated by a lnear combnaton of these two. onsequently, there are only two lnearly ndependent ncdence relatons for l. Smlarly there s a one-parameter famly of lnes through x,and the ncdence relaton s also lnear n lnes l through x. Thus, there are a total of
14.3 Transfer 371 mage 1 mage 2 F32x F31 x x x mage 3 x F 31 eppolar lne from mage 1 F 32 eppolar lne from mage 2 e31 e32 mage 3 a b Fg. 14.7. Eppolar transfer. (a) The mage of X n the frst two vews s the correspondence x x. The mage of X n the thrd vew may be computed by ntersectng the eppolar lnes F 31 x and F 32 x. (b) The confguraton of the eppoles and transferred pont x as seen n the thrd mage. Pont x s computed as the ntersecton of eppolar lnes passng through the two eppoles e 31 and e 32. However, f x les on the lne through the two eppoles, then ts poston cannot be determned. Ponts close to the lne through the eppoles wll be estmated wth poor precson. four lnearly ndependent ncdence relatons between a pont n the frst vew and lnes n the second and thrd. The man vrtue of the trlneartes s that they are lnear, otherwse ther propertes are often subsumed by transfer, as descrbed n the followng secton. 14.3 Transfer Gven three vews of a scene and a par of matched ponts n two vews one may wsh to determne the poston of the pont n the thrd vew. Gven suffcent nformaton about the placement of the cameras, t s usually possble to determne the locaton of the pont n the thrd vew wthout reference to mage content. Ths s the pont transfer problem. A smlar transfer problem arses for lnes. In prncple the problem can generally be solved gven the cameras for the three vews. Rays back-projected from correspondng ponts n the frst and second vew ntersect and thus determne the 3D pont. The poston of the correspondng pont n the thrd vew s computed by projectng ths 3D pont onto the mage. Smlarly lnes back-projected from the frst and second mage ntersect n the 3D lne, and the projecton of ths lne n 3-space to the thrd mage determnes ts mage poston. 14.3.1 Pont transfer usng fundamental matrces The transfer problem may be solved usng knowledge of the fundamental matrces only. Thus, suppose we know the three fundamental matrces F 21, F 31 and F 32 relatng the three vews, and let ponts x and x n the frst two vews be a matched par. We wsh to fnd the correspondng pont x n the thrd mage.
372 14 The Trfocal Tensor 2 e 21 e 23 1 e 13 e 12 mage 2 e 32 3 e31 mage 1 mage 3 Fg. 14.8. The trfocal plane s defned by the three camera centres. The notaton for the eppoles s e j = P j. Eppolar transfer fals for any pont X on the trfocal plane. If the three camera centres are collnear then there s a one-parameter famly of planes contanng the three centres. The requred pont x matches pont x n the frst mage, and consequently must le on the eppolar lne correspondng to x. Snce we know F 31, ths eppolar lne may be computed, and s equal to F 31 x. By a smlar argument, x must le on the eppolar lne F 32 x. Takng the ntersecton of the eppolar lnes gves x =(F 31 x) (F 32 x ). See fgure 14.7a. Note that the fundamental matrx F 21 s not used n ths expresson. The queston naturally arses whether we can gan anythng by knowledge of F 21, and the answer s yes. In the presence of nose, the ponts x x wll not form an exact matched par, meanng that they wll not satsfy the equaton x F 21 x = 0 exactly. Gven F 21 one may use optmal trangulaton as n algorthm 11.1(p304) to correct x and x, resultng n a par ˆx ˆx that satsfes ths relaton. The transferred pont may then be computed as x =(F 31ˆx) (F 32ˆx ). Ths method of pont transfer usng the fundamental matrces wll be called eppolar transfer. Though at one tme used for pont transfer, eppolar transfer has a serous defcency that rules t out as a practcal method. Ths defcency s due to the degeneracy that can be seen from fgure 14.7(b): eppolar transfer fals when the two eppolar lnes n the thrd mage are concdent (and becomes ncreasngly llcondtoned as the lnes become less transverse ). The degeneracy condton that x, e 31 and e 32 are collnear n the thrd mage means that the camera centres and and the 3D pont X le n a plane through the centre of the thrd camera; thus X les on the trfocal plane defned by the three camera centres, see fgure 14.8. Eppolar transfer wll fal for ponts X lyng on the trfocal plane and wll be naccurate for ponts lyng near that plane. Note, n the specal case that the three camera centres are collnear the trfocal plane s not unquely defned, and eppolar transfer fals for all ponts. In ths case e 31 = e 32.
14.3 Transfer 373 14.3.2 Pont transfer usng the trfocal tensor The degeneracy of eppolar transfer s avoded by use of the trfocal tensor. onsder a correspondence x x. If a lne l passng through the pont x s chosen n the second vew, then the correspondng pont x may be computed by transferrng the pont x from the frst to the thrd vew usng x k = x l j T jk, from table 14.2. It s clear from fgure 14.4(p361)(b) that ths transfer s not degenerate for general ponts X lyng on the trfocal plane. However, note from result 14.3(p363) and fgure 14.6 that f l s the eppolar lne correspondng to x, then x l j T jk =0 k, so the pont x s undefned. onsequently, the choce of lne l s mportant. To avod choosng only an eppolar lne, one possblty s to use two or three dfferent lnes passng through x, namely l jp = x r ɛ rjp for the three choces of p =1,...,3. For each such lne, one computes the value of x and retans the one that has the largest norm (.e. s furthest from beng zero). An alternatve method entrely for fndng x s as the least-squares soluton of the system of lnear equatons x (x j ɛ jpr )(x k ɛ kqs )T pq =0 rs, but ths method s probably an overkll. The method we recommend s the followng. Before attemptng to compute the pont x transferred from a par of ponts x x, frst correct the par of ponts usng the fundamental matrx F 21, as descrbed above n the case of eppolar transfer. If ˆx and ˆx are an exact match, then the transferred pont x k =ˆx l j T jk does not depend on the lne l chosen passng through ˆx (as long as t s not the eppolar lne). Ths may be verfed geometrcally by referrng to fgure 14.2(p359). A good choce s always gven by the lne perpendcular to F 21ˆx. To summarze, a measured correspondence x x s transferred by the followng steps: () ompute F 21 from the trfocal tensor (by the method gven n algorthm 14.1), and correct x x to the exact correspondence ˆx ˆx usng algorthm 11.1- (p304). () ompute the lne l through ˆx and perpendcular to l e = F 21ˆx. If l e =(l 1,l 2,l 3 ) and ˆx =(ˆx 1, ˆx 2, 1), then l =(l 2, l 1, ˆx 1 l 2 +ˆx 2 l 1 ). () The transferred pont s x k =ˆx l j T jk. Degenerate confguratons. onsder transfer to the thrd vew va a plane, as shown n fgure 14.9. The 3D pont X s only undefned f t les on the baselne jonng the frst and second camera centres. Ths s because rays through x and x are collnear for such 3D ponts and so ther ntersecton s not defned. In such a case, the ponts x and x correspond wth the eppoles n the two mages. However, there s no problem transferrng a pont lyng on the baselne between vews two and three, or anywhere else on the trfocal plane. Ths s the key dfference between eppolar transfer and transfer usng the trfocal tensor. The former s undefned for any pont on the trfocal plane.
374 14 The Trfocal Tensor 2 B 12 e21 l B 23 1 x e 12 mage 2 x 3 mage 1 X π mage 3 Fg. 14.9. Degeneracy for pont transfer usng the trfocal tensor. The 3D pont X s defned by the ntersecton of the ray through x wth the plane π. A pont X on the baselne B 12 between the frst and second vews cannot be defned n ths manner. So a 3D pont on the lne B 12 cannot be transferred to the thrd vew va a homography defned by a lne n the second vew. Note that a pont on the lne B 12 projects to e 12 n the frst mage and e 21 n the second mage. Apart from the lne B 12 any pont can be transferred. In partcular there s not a degeneracy problem for ponts on the baselne B 23, between vews two and three, or for any other pont on the trfocal plane. 14.3.3 Lne transfer usng the trfocal tensor Usng the trfocal tensor, t s possble to transfer lnes from a par of mages to a thrd accordng to the lne-transfer equaton l = l j l k T jk of table 14.2. Ths gves an explct formula for the lne n the frst vew, gven lnes n the other two vews. Note however that f the lnes l and l are known n the frst and second vews then l may be computed by solvng the set of lnear equatons (l r ɛ rs )l j l k T jk =0 s,thereby transferrng t nto the thrd mage. Smlarly one may transfer lnes nto the second mage. Lne transfer s not possble usng only the fundamental matrces. Degeneraces. onsder the geometry of fgure 11.7(p308). The lne L n 3- space s defned by the ntersecton of the planes through l and l, namely π and π respectvely. Ths lne s clearly undefned when the planes π and π are concdent,.e. n the case of eppolar planes. onsequently, lnes cannot be transferred between the frst and thrd mage f both l and l are correspondng eppolar lnes for the frst and second vews. Algebracally, the lne-transfer equaton gves l = l j l k T jk = 0, and the equaton matrx (l r ɛ rs )l j T jk used to solve for l becomes zero. It s qute common for lnes to be near eppolar, and ther transfer s then naccurate, so ths condton should always be checked for. There s an equvalent degeneracy for lne transfer between vews one and two defned by a lne n vew three. Agan, t occurs f the lnes n vews one and three are correspondng eppolar lnes for these two vews. In general the eppolar geometres between vews one and two, and one and three wll dffer, for nstance the eppole e 12 arsng n the frst vew from vew two wll not concde wth the eppole e 13 arsng n the frst vew from vew three. Thus an
14.4 Relatonshp between fundamental matrces and the trfocal tensor 375 eppolar lne n the frst vew for vews one and two wll not concde wth an eppolar lne for vews one and three. onsequently, when lne transfer nto the thrd vew s degenerate, lne transfer nto the second vew wll not n general be degenerate. However, for lnes n the trfocal plane transfer s degenerate (.e. undefned) always. 14.4 Relatonshp between fundamental matrces and the trfocal tensor Snce the projectve structure of the three cameras may be computed explctly from the trfocal tensor, t follows that all three fundamental matrces for the three vew pars are determned by the trfocal tensor. In fact smple formulae exst for the two fundamental matrces F 21 and F 31 as gven n algorthm 14.1. The queston arses whether t s possble to determne the trfocal tensor gven the three fundamental matrces. The answer s that ths s possble n most cases, as wll be demonstrated. A frst observaton s that the trfocal tensor s determned by the three camera matrces. In fact, t s suffcent to determne the camera matrces, up to projectve transformaton. The frst two camera matrces P and P may be determned from the fundamental matrx F 12 by two-vew technques (chapter 8). It remans to determne the thrd camera matrx P n the same projectve frame. In prncple, ths may be done as follows. () Select a set of matchng ponts x x n the frst two mages, satsfyng x F 21 x = 0, and use trangulaton to determne the correspondng 3D ponts X. () Use eppolar transfer to determne the correspondng ponts x n the thrd mage, usng the fundamental matrces F 31 and F 32. () Solve for the camera matrx P from the set of 3D 2D correspondences X x. The second step n ths algorthm wll fal n the case where the pont X les n the trfocal plane. Such a pont X s easly detected and dscarded, snce t projects nto the frst mage as a pont x lyng on the lne jonng the two eppoles e 12 and e 13. Snce there are nfntely many possble matched ponts, we can compute suffcently many such ponts to compute P. The only stuaton n whch ths method wll fal s when all space ponts X le n a trfocal plane. Ths can occur only n the degenerate stuaton n whch the three camera centres are collnear, n whch case the trfocal plane s not unquely determned. Thus, we see that unless the three camera centres are collnear, the three camera matrces may be determned from the fundamental matrces. On the other hand, f the three cameras are collnear, then there s no way to determne the relatve spacngs of the cameras along the lne of ther centres. Ths s because the length of the baselne cannot be determned from the fundamental matrces, and the three baselnes (dstances between the camera centres) may be arbtrarly chosen
376 14 The Trfocal Tensor and reman consstent wth the fundamental matrces. Thus we have demonstrated the followng fact: Result 14.5. The trfocal tensor relatng three cameras s determned from the three fundamental matrces alone f and only f the three camera centres are not collnear. 14.5 losure The development of three-vew geometry proceeds n an analogous manner to that of two-vew geometry covered n part II of ths book. The trfocal tensor may be computed from mage correspondences over three vews, and a projectve reconstructon of the cameras and 3D scene then follows. Ths computaton s descrbed n chapter 15. The projectve ambguty may be reduced to affne or metrc by supplyng addtonal nformaton on the scene or cameras n the same manner as that of chapter 9. A smlar development to that of chapter 12 may be gven for the relatons between homographes nduced by scene planes and the trfocal tensor. 14.5.1 The lterature Wth hndsght, the dscovery of the trfocal tensor may be traced to [Spetsaks-91] and [Weng-88], where t was used for scene reconstructon from lnes n the case of calbrated cameras. It was later shown n [Hartley-94c] to be equally applcable to projectve scene reconstructon n the uncalbrated case. At ths stage matrx notaton was used, but [Vévlle-93] used tensor notaton for ths problem. Meanwhle n ndependent work, Shashua ntroduced trlnearty condtons relatng the coordnates of correspondng ponts n three vews wth uncalbrated cameras [Shashua-94, Shashua-95a]. [Hartley-95a, Hartley-97c] then showed that Shashua s relaton for ponts and scene reconstructon from lnes both arse from a common tensor, and the trfocal tensor was explctly dentfed. In subsequent work propertes of the tensor have been nvestgated, e.g. [Shashua-95b]. In partcular [Trggs-95] descrbed the mxed covarant contravarant behavour of the ndces, and [Zsserman-96] descrbed the geometry of the homographes encoded by the tensor. Faugeras and Mourran [Faugeras-95c] gave enlghtenng new dervatons of the trfocal tensor equatons and consdered the trfocal tensor n the context of general lnear constrants nvolvng multple vews. Ths approach wll be dscussed n chapter 16. Further geometrc propertes of the tensor were gven n Faugeras & Papadopoulo [Faugeras-97]. Eppolar pont transfer was descrbed by [Barrett-92, Faugeras-94], and ts defcences ponted out by [Zsserman-94], amongst others. The trfocal tensor has been used for varous applcatons ncludng establshng correspondences n mage sequences [Beardsley-96], ndependent moton detecton [Torr-95b], and camera self-calbraton [Armstrong-96a].
14.5 losure 377 14.5.2 Notes and exercses () The trfocal tensor s nvarant to 3D projectve transforms. Verfy explctly that f H 4 4 s a transform preservng the frst camera matrx P =[I 0], then the tensor defned by (14.1 p357) s unchanged. () In ths chapter the startng pont for the trfocal tensor dervaton was the ncdence property of three correspondng lnes. Show that alternatvely the startng pont may be the homography nduced by a plane. Here s a sketch dervaton: choose the camera matrces to be a canoncal set P =[I 0], P =[A a 4 ], P =[B b 4 ] and start from the homography H 13 between the frst and thrd vews nduced by a plane π. From result 12.1(p312) ths homography may be wrtten as H 13 = B b 4 v,whereπ =(v, 1). In ths case the plane s defned by a lne l n the second vew as π = P l. Show that result 14.2(p360) follows. () Homographes nvolvng the frst vew are smply expressed n terms of the trfocal tensor T jk as gvenby result 14.2(p360). Investgate whether a smple formula exsts for the homography H 23 from the second to the thrd vew, nduced by a lne l n the frst mage. (v) The contracton x T jk s a 3 3 matrx. Show that ths may be nterpreted as a correlaton (see defnton 1.28(p39)) mappng between the second and thrd vews nduced by the lne whch s the back-projecton of the pont x n the frst vew. (v) In the case where two of the three cameras have the same camera centre, the trfocal tensor may be related to smpler enttes. There are two cases. (a) If the second and thrd camera have the same centre, then T jk = F r H k sɛ rjs,wheref r s the fundamental matrx for the frst two vews, and H s the homography from the second to the thrd vew nduced by the fact that they have the same centre. (b) If the frst and the second vews have the same centre, then T jk = H j e k,wherehs the homography from the frst to the second vew and e s the eppole n the thrd mage. Prove these relatonshps usng the approach of chapter 16. (v) onsder the case of a small baselne between the cameras and derve a dfferental form of the trfocal tensor, see [Åström-98, Trggs-99b]. (v) There are actually three dfferent trfocal tensors relatng three vews, dependng on whch of the three cameras corresponds to the covarant ndex. Gven one such tensor [T ], verfy that the tensor [T ] may be computed n several steps, as follows: (a) Extract the three camera matrces P =[I 0], P and P from the trfocal tensor.
378 14 The Trfocal Tensor (b) Fnd a 3D projectve transformaton H such that P H =[I 0], and apply t to each of P and P as well. (c) ompute the tensor [T ] by applyng (14.1 p357). (v) Investgate the form and propertes (e.g. rank of the matrces T ) of the trfocal tensor for the specal motons (pure translaton, planar moton) descrbed n secton 8.3(p228) for the fundamental matrx. (x) omparson of the ncdence relatonshps of table 14.3 ndcates that one may replace a lne l j by the expresson ɛ jrsx r, and proceed smlarly wth l k. Also, one gets a three-vew lne equaton by replacng x by ɛ rs l. an both of these operatons be carred out at once to obtan an equaton )( (ɛ ru l ɛ jsv x j)( ɛ ktw x k) Tr st =0 u vw? Why, or why not? (x) Affne trfocal tensor. If the three cameras P, P and P are all affne (defnton 5.3(p153)), then the correspondng tensor T A s the affne trfocal tensor. Ths affne specalzaton of the tensor has 12 degrees of freedom and 16 non-zero entres. It was frst defned n [Torr-95a]. (x) Relatons amongst fundamental matrces and eppoles. The three fundamental matrces F 12, F 23, F 31 are not ndependent, but satsfy three (quadratc) relatons: F 12 e 13 = e 23 e 21 F 23 e 21 = e 31 e 32 F 31 e 32 = e 12 e 13. These may be deduced from the trfocal plane geometry of fgure 14.8, e.g. the frst relaton states that the eppolar lne n vew 2 of e 13 n vew 1 (whch s the mage of the baselne between centres 1 and 3) s the ntersecton of the trfocal plane wth vew 2, gven by e 23 e 21. The trfocal tensor has 18 degrees of freedom and ths may be accounted for as 21 for the 3 7 degrees of freedom of the fundamental matrces less 3 for the quadratc relatons. Fundamental matrces computed from the trfocal tensor wll automatcally satsfy these relatons.