Tensor Embedding of the Fundamental Matrix SHAI AVIDAN.

Transcription

1 ??,??, 1{14 (??) c?? Kluwer Academc Publshers, Boston. Manufactured n The Netherlands. Tensor Embeddng of the Fundamental Matrx SHAI AVIDAN avdan@cs.huj.ac.l Insttute of Computer Scence, The Hebrew Unversty, Jerusalem, Israel. AMNON SHASHUA shashua@cs.huj.ac.l Insttute of Computer Scence, The Hebrew Unversty, Jerusalem, Israel. Receved??; Revsed?? Edtors:?? Abstract. We revst the blnear matchng constrant between two perspectve vews of a 3D scene. Our objectve s to represent the constrant n the same manner and form as the trlnear constrant among three vews. The motvaton s to establsh a common termnology that brdges between the fundamental matrx F (assocated wth the blnear constrant) and the trfocal tensor T jk (assocated wth the trlneartes). By achevng ths goal we can unfy both the propertes and the technques ntroduced n the past for workng wth multple vews for geometrc applcatons. Dong that we ntroduce a tensor F jk, we call the bfocal tensor, that represents the blnear constrant. The bfocal and trfocal tensors share the same form and share the same contracton propertes. By close nspecton of the contractons of the bfocal tensor nto matrces we show that one can represent the famly of rank-2 homography matrces by [] F where s a free vector. We then dscuss four applcatons of the new representaton: () Quas-metrc vewng of projectve data, () trangulaton, () vew synthess, and (v) recovery of camera ego-moton from a stream of vews. 1. Introducton The geometry of multple vews s governed by certan mult-lnear constrants, blnear for pars of vews and trlnear for trplets of vews all other mult-lnear constrants (four vews and beyond) are spanned by the blnear and trlnear constrants. The tradtonal representaton of the coecents of the blnear constrant sbya33 matrx, F, that satses p 0> Fp = 0 for all matchng mage ponts p; p 0 (represented n the 2D projectve space) across two vews. On the other hand, the three-vew relatons are represented by a set of 4 trlnear constrants, each of the form p s j r k T jk = 0 where s and r are lnes concdent wth the matchng ponts p 0 and p 00, respectvely. In other words, the blnear constrant represents a \pont+pont" relaton, whereas each of the trlnear constrants represents a \pont+lne+lne" relaton (further detals can be found n the Appendx). Because of the derence n form between the fundamental matrx and the trfocal tensor, the analyss tools are derent and the propertes dscovered for one do not easly carry over to the other. For example, the trfocal tensor contracts

2 2?? (reduces) to matrx forms that carry geometrc nformaton: one type of contracton produces subgroups of 2D homography matrces and another type of contracton produces a subgroup of 2D correlaton matrces. There s no such equvalence known for the fundamental matrx, for nstance. In ths paper we revst the blnear constrant and represent t usng a tensor p s j r k F jk = 0 where s; r are two concdent lnes wth the matchng pont p 0. We call the tensor F jk the \bfocal" tensor and show that not only t shares the same form as the trfocal tensor but t also shares the same propertes. We can therefore consder contractons of the bfocal tensor just as was done wth the trfocal counterpart. Through the nspecton of tensor contractons we derve the representaton of the subgroup of rank-2 homography matrces n the smple form of [] F where s a free vector. We ntroduce the group of \prmtve homographes" and dscuss 4 applcatons of the new representaton: () Quas-metrc vewng of projectve data, () trangulaton, () vew synthess, and (v) recovery of camera ego-moton from a stream of vews. Ths work n ts ntal form was presented at the meetng found n [1]. 2. Notatons A pont x n the 3D projectve space P 3 s projected onto the pont p n the 2D projectve space P 2 bya34 camera projecton matrx A =[A; v 0 ] that satses p = Ax, where = represents equalty up to scale. The left 3 3 mnor of A, denoted by A, stands for a 2D projectve transformaton of some arbtrary plane (the reference plane) and the fourth column of A, denoted by v 0, stands for the eppole (the projecton of the center of camera 1 on the mage plane of camera 2). In a calbrated settng the 2D projectve transformaton s the rotatonal component of camera moton (the reference plane s at nnty) and the eppole s the translatonal component of camera moton. Snce only relatve camera postonng can be recovered from mage measurements, the camera matrx of the rst camera poston n a sequence of postons can be represented by [I; 0]. We wll occasonally use tensoral notatons, whch are brey descrbed next. We use the covarant-contravarant summaton conventon: a pont s an object whose coordnates are speced wth superscrpts,.e., p =(p 1 ;p 2 ; :::). These are called contravarant vectors. An element n the dual space (representng hyper-planes e.g., lnes n P 2 ), s called a covarant vector and s represented by subscrpts,.e., s j =(s 1 ;s 2 ; ::::). Indces repeated n covarant and contravarant forms are summed over,.e., p s = p 1 s 1 + p 2 s 2 + ::: + p n s n. Ths s known as a contracton. For example, f p s a pont ncdent to a lne s n P 2, then p s =0. Vectors are also called 1-valence tensors. 2-valence tensors (matrces) have two ndces and the transformaton they represent depends on the covarant-contravarant postonng of the ndces. For example, a j s a mappng from ponts to ponts, and hyper-planes to hyperplanes, because a j p = q j and a j s j = r (n matrx form: Ap = q and A > s = r); a j maps ponts to hyper-planes; and a j maps hyper-planes to ponts. When vewed as a matrx the row and column postons are determned accordngly: n a j and a j the ndex runs over the columns and j runs over the rows, thus b k j aj = ck s BA = C n matrx form. An outer-product of two 1-valence tensors (vectors), a b j, s a 2-valence tensor c j whose ; j entres are a b j note that n matrx form C = ba >. An n-valence tensor descrbed as an outer-product of n vectors s a rank-1 tensor. Any n-valence tensor can be descrbed as a sum of rank-1 n-valence tensors. The rank of an n-valence tensor s the smallest number of rank-1 n-valence tensors wth sum equal to the tensor. For example, a rank-1 trvalent tensor s a b j c k where a,b j and c k are three vectors. The rank of a trvalent tensor jk s the smallest r such that, jk = rx s=1 a s b js c ks : (1) We wll make extensve use of the \cross-product tensor" dened next. The cross product (vector product) operaton c = ab s dened for vectors n P 2. The vector c s the lne jonng the ponts a; b, or the pont ofntersecton of the lnes a; b. The product operaton can also be represented as the product c = [a] b where [a] x s called the

3 ?? 3 \skew-symmetrc matrx of a" and has the form: [a] = 0 0,a 3 a 2 a 3 0,a 1 A,a 2 a 1 0 In tensor form we have jk a b j = c k representng the cross products of two ponts (contravarant vectors) resultng n the lne (covarant vector) c k. Smlarly, jk a b j = c k represents the pont ntersecton of the to lnes a and b j. The tensor s dened such that jk a produces the matrx [a] (.e., contans 0;,1; 1 n ts entres such that ts operaton on a sngle vector produces the skewsymmetrc matrx of that vector). 3. Tensor Embeddng of the Fundamental Matrx Our goal s to derve a trvalent tensor representaton (.e., a 333 tensor) of the 33 fundamental matrx and to llumnate the advantages of dong so. In partcular, once we have the trvalent tensor representaton n our hand we wsh to nvestgate ts contracton propertes (as was done for the trfocal tensor n [22]) and recast them back n matrx form. We start wth dervng the fundamental matrx from basc prncples. Let A be a 2D homography (collneaton) from mage 1 to mage 2 due to some plane,.e., f p s a pont n mage 1, then Ap s a pont concdent wth the eppolar lne p 0 v 0 n mage 2, where the exact locaton of Ap on the eppolar lne s determned by the poston of the plane. Thus, (v 0 p 0 ) > Ap = 0, or n tensor notaton, 0 = lj p 0j v 0 p a l = p 0j ( lj v 0 a l ) {z } F j where lj p 0j v 0 s the cross-product p 0 v 0. The matrx F j = lj v 0 a l s the fundamental matrx that satses the blnear constrant p p 0j F j =0 (cf. [14, 6]). In matrx form, snce lj v 0 s the skew-symmetrc matrx [v 0 ], then F =[v 0 ] A. Next, we begn wth the blnear constrant p p 0j F j = 0 and consder replacng the pont p 0 wth a cross product of any two ncdent lnes s; r,.e., p 0l = ljk s j r k. The reason for dong so p wll be apparent later on. We have therefore a \pont+lne+lne" relatonshp p s j r k F jk =0as follows: p p 0l F l = p ( ljk s j r k ) {z } p 0l F l = p s j r k ( ljk F l ) {z } F jk = 0 and the tensor F jk = ljk F l s a trvalent form of the fundamental matrx. Ths form s equvalent to consderng the trfocal tensor of vews 1,2,3 where vews 2,3 are dentcal. Thus we obtan a relatonshp between three vews, but only two of the vews are dstnct. We can represent the \bfocal" tensor F jk drectly as a functon of v 0 and A as follows: F jk = v 0j a k, v 0k a j : The mportance of the trvalent tensor embeddng of the fundamental matrx (whch we wll denote by bfocal tensor from now on) s that we have arrved to an equvalent representaton wth 3-vew geometry: both the trfocal and bfocal tensors are and operate on a conguraton of a pont+lne+lne. In the case of three vews, the lnes are n two dstnct vews (the lne s concdes wth p 0 and the lne r concdes wth p 00 ) and there are 4 such relatonshps (due to the fact that there are two choces for each lne). In the case of two vews the two lnes are n the same vew and therefore there s only one conguraton of pont+lne+lne. The advantage of ths equvalence n form between the trfocal and bfocal tensors appears when one consders contractons nto bvalent forms (matrces). The propertes of contractons of the trfocal tensor are well understood (see [22, 18] and n the appendx here) and provde the buldng blocks for makng use of the trfocal tensor n applcatons. We can apply now an dentcal analyss on the bfocal tensor whch we wll do next Bfocal Tensor Contractons Gven an arbtrary vector, the trfocal tensor reduces to a matrx of three types: T jk ; j T jk

4 4?? and k T jk. Note that when =(1; 0; 0); (0; 1; 0) or (0; 0; 1) we obtan \slces" of the tensor. The rst type produces a rank-2 correlaton matrx,.e., a mappng from all 2D lnes to collnear ponts (where the orentaton of collnearty s determned by ) by slcng the tensor n that way we obtan the three matrces of \lne geometry" ntroduced n the calbrated context by [23, 24, 28]. The second and thrd types produce homography matrces (collneatons). The second type s a homography matrx from vew 1 to 3 due to a plane determned by the lne j n vew 2 and the center of projecton of camera 2. Lkewse, the thrd type s a homography from vew 1 to 2 va a plane determned by the lne k n vew 3 and the center of projecton of camera 3. These homography matrces were ntroduced n [22] and are descrbed n more detal n the appendx here. We wsh to consder the same types of contractons on the bfocal tensor F jk by equvalence of form, we should obtan collneatons and correlatons as well. Consder the contracton k F jk for some arbtrary vector. By substtuton n the denton of F jk we obtan k F jk =( ljk k ) {z } [] F l whch n matrx form becomes [] F. Our queston therefore s about the geometrc nterpretaton of ths matrx (for an arbtrary ). Gven the form-equvalence of the two tensors the answer s mmedate: [] F s a homography matrx from vew 1 to vew 2 va a plane concdent wth the center of projecton O 0 of camera 2 and the lne n vew 2. The famly of such matrces over all choces of corresponds to the famly of homography matrces whose planes are concdent wth O 0. The famly s spanned by three matrces (snce s spanned by three vectors), and for example, the three slces usng =(1; 0; 0); (0; 1; 0) or (0; 0; 1) wll provde the bass for ths subgroup of homography matrces. More formally, consder the plane dened by the pont O 0 and the lne n vew 2. Consder a pont p n vew 1 and the ray from the center of projecton O of the rst camera and the pont p. The ray ntersects at P whch projects to (a) Fg. 1. The matrx [] F s a homography matrx due to a plane concdent wth the center of projecton O 0 and the lne n vew 2. The lne s s the eppolar lne and the pont P s at the ntersecton of the optc ray from the rst vew and the plane. The pont p 0 s the projecton of P onto vew 2. Therefore the pont+lne+lne conguraton of p; ; ssatses the bfocal tensor relaton p s j kf jk = 0, where kf jk s the matrx [] F. a pont p 0 n vew 2 whch s concdent wth the lne (by constructon). Let s j be the eppolar lne of p n vew 2, thus p s j k F jk = 0 because they provde a pont+lne+lne conguraton, and ths holds for all ponts p (see Fg. 1). Thus, the matrx k F jk maps vew 1 onto ponts along the correspondng eppolar lnes and s therefore a homography matrx, and snce the projected ponts are collnear the rank of the matrx s 2. We have the followng result: Theorem 1. The matrx [] F s a homography matrx of rank2from vew 1 to vew 2 due to the plane concdent wth the center of projecton of camera 2 and the lne n vew 2. Note that the theorem generalzes the observaton due to [16] that [v 0 ] x F s a homography matrx. We see that ths s true for any choce of skew-symmetrc matrx [].

5 ?? 5 The same result apples for the contracton j F jk (wth a change of sgn). Note that wth the trfocal tensor there s a derence between the contractons j T jk and k T jk n whch the former produces a homography matrx from vew 1 to 3 and the latter produces a homography matrx from vew 1 to 2. In the case of the bfocal tensor vews 2 and 3 concde thus the two types of contractons are equvalent. The remanng contracton type s F jk. In the case of the trfocal tensor the contracton T jk produces a correlaton matrx whch maps the space of lnes from vew 2 to a set of collnear ponts (on the eppolar lne of the pont n vew 1) n vew 3. The transpose of that matrx s the same type of mappng, but from vew 3 to vew 2 (see Appendx). We should obtan somethng smlar for the bfocal tensor and snce vew 2 and 3 concde the matrx F jk should map the space of lnes n vew 2 onto collnear ponts n vew 2 that dene the eppolar lne, F, of the pont. Indeed, by substtuton we obtan: F jk = ljk F l = [F] (2) Thus, [F] s for all lnes s n vew 2 s the pontof ntersecton of the eppolar lne F and the lne s. In other words, the matrx [F] s the correlaton matrx we descrbed above. Note that the reason we have obtaned a trval mappng s due to fact that ths type of contracton s assocated wth reconstructon for lnes. The three matrces T jk for =(1; 0; 0); (0; 1; 0) and (0; 0; 1) are known to arse from consderatons of matchng lnes across three vews (cf. [23, 28, 10]). However, the relatve camera postons cannot be recovered from matchng lnes across two vews only (only from matchng ponts), whch swhy the correspondng correlaton matrces F jk of the bfocal tensor become trval. To summarze, the embeddng of the fundamental matrx n trvalent tensor format (the bfocal tensor) provdes a uned termnology of a \pont+lne+lne" that apples for both the bfocal and trfocal relatonshps across multple vews. In partcular, as s the case wth the trfocal tensor, contractons of the bfocal tensor nto reduced forms (matrces) have a geometrc sgnfcance. The contractons propertes of the bfocal tensor are lsted n Table 1. We see a clear analogy to the type of resultng matrces (homography and correlatons) one obtans from the same contractons appled to the trfocal tensor. Furthermore, the homography contracton provdes the bass for all rank-2 homography matrces whose planes are concdent wth the center of projecton of camera 2. All lnear combnatons of the rank-2 homography matrces are of the form [] F for some vector. 4. The Prmtve Homography Matrces We have seen that the famly of matrces [] F parameterzed by the choce of the vector spans the famly of homography matrces from vew 1 to vew 2 due to the planes concdent wth the center of projecton O 0 of camera 2. The vector determnes the orentaton of the plane and s the lne of ntersecton of the plane and vew 2. Snce s spanned by three vectors, say (1; 0; 0); (0; 1; 0) and (0; 0; 1), the bfocal tensor contractons provde three dstnct homography matrces that span the subgroup of homography matrces (those whose planes are concdent wth O 0 ). Snce the entre group of all homography matrces les n a 4 dmensonal subspace [20],.e., spanned by 4 homography matrces whose planes do not all concde wth a sngle pont, we must produce an addtonal homography matrx n order to complete the bass of the subgroup dened by [] F to a full bass for the entre group. The elements (matrces) of the full bass wll be called \prmtve homographes". The addtonal homography matrx we seek must therefore be assocated wth a plane concdent wth the center of projecton O of camera 1 (and s therefore of rank 1). We have the followng Lemma whch s adapted from [17]: Lemma 1. Gven the fundamental matrx F and the eppole v 0 dened byf > v 0 =0, then the famly of matrces v 0 > are homography matrces from vew 1 to vew 2 due to planes concdent wth the center of projecton O of camera 1 and the vector s the ntersecton lne of the plane and vew 1. Proof: Let A 1 ;A 2 be any two homography matrces. Thus, A 1 p; A 2 p and v 0 are collnear for all ponts p n vew 1. Let q 6= v, where v s

6 6?? Table 1. The three types of contractons of the bfocal tensor (embeddng of the fundamental matrx F as a trvalent tensor F jk ), ther matrx form, and the property they produce. Note that the rst two contractons produce a homog- raphy matrx of a plane whose orentaton s determned by the vector of contracton. Contracton Matrx Form Result 5.1. Quas-Metrc Reference Plane Let p ;p 0, =1; :::; N, be matchng ponts n vew 1 and 2 respectvely. Gven the fundamental matrx F and the eppole v 0 n vew 2, then the 3D projectve representaton of the object space ponts P can be descrbed relatve to a reference plane : k F jk [] F Homography Matrx. j F jk [] F Same as above. F jk [F] Trval Correlaton Mappng. the eppole n vew 1 (Fv = 0), be some pont n vew 1 and let be a scalar dened such that A 1 q,a 2 q = v 0. Let H = A 1,A 2 be a homography matrx (because all homography matrces are closed under lnear combnatons). Clearly, snce Hq = v 0, then Hp = v 0 for all p (because Hv = v 0 as well). Thus H = v 0 > for some vector. Therefore, as long as > v 6= 0, where v s the eppole n vew 1 (.e., Fv = 0), then the homography matrx v 0 > does not concde wth O 0 (only wth O) and thus can be used to complete the full bass for the group of homography matrces. Wthout loss of generalty assume that (1; 0; 0) s not concdent wth v, thus we have a bass of 4 homography matrces H 1 ; :::; H 4, denoted as \prmtve homographes", dened below: H = [e ] F; =1; 2; 3 (3) H 4 = v 0 e > 1 (4) p 0 = A p + v 0 =[A ;v 0 ]P where A s the homography matrx mappng vew 1onto vew 2 due to the plane. The scalar represents the relatve devaton of the pont P from the plane and s called the \relatve ane structure"[21]. The choce of the plane determnes the projectve representaton of object space. For purposes of vsualzaton, t s useful to choose such that t s stuated \n-between" the space ponts makng t possble to treat as smple depth varable. In other words, let A P = j jh j,we seek to solve for the scalar j, j = 1; :::; 4, that mnmze: 4X j=1 ( j H j )p = p 0 =1; :::; N whch provdes an over-determned lnear set of equatons. We wll refer to as the \quas-metrc" plane. The choce of the quas-metrc plane provdes a better chance that the projectve vewng of the object (treatng the coordnates x ;y ; as Eucldean coordnates by the vewng program) wll have less projectve dstortons than other choces Trangulaton from 3 Vews where e are the dentty vectors: e 1 = (1; 0; 0);e 2 =(0; 1; 0) and e 3 =(0; 0; 1). 5. Applcatons Usng Prmtve Homography Matrces The prmtve homography matrces are a useful tool for representng geometrc data. We wll consder two examples here, the rst on obtanng a \quas-metrc" representaton of 3D space from a par of uncalbrated cameras, and the second on \trangulaton" from 3 vews. Hartley and Sturm [11] consdered the problem, they called "trangulaton", of modfyng the locatons of nput matchng ponts ^p; ^p 0 that are gven wth nose to new locatons p; p 0 that satsfy p 0> Fp = 0 such that (p, ^p) 2 +(p 0, ^p 0 ) 2 s mnmzed. The trangulaton problem n 3 vews can be stated n a smlar manner: gven p n vew 1, the matchng process produces an error n the matches n vew 2 and 3. The nput matches are ^p 0 and ^p 00 and we wsh to nd new matches p 0 ;p 00 wth p such that the trplet p; p 0 ;p 00 satsfy the trlnear equatons whle (p 0, ^p 0 ) 2 +(p 00, ^p 00 ) 2 s mnmzed. Note that we do not add an error term to p and

7 ?? 7 rather take p as a reference. The reason for that s twofold: rst due to the asymmetry of the trfocal tensor wth respect to vew orderng as t s dened wth respect to a reference vew (unlke the fundamental matrx whch remans xed under vew orderng). Secondly, n most matchng approaches that use a correlaton prncple, lke the popular Lucas-Kanade [15] method wth the coarse-to-ne mplementaton by Sarno Corp. [4], there s also an ntrnsc asymmetry that assumes one of the vews as a reference. Taken together, we can wthout loss of generalty assume that the eect of error n the matchng process s represented n the dsplacementof^p 0 and ^p 00 from ther true locatons p 0 ;p 00. The trangulaton process usng the trfocal tensor can proceed as follows. We rst note that the followng relatonshp exsts: p 0 = Ap + v 0 (5) p 00 = Bp + v 00 (6) where A; B are two homography matrces from vew 1 to 2 and from vew 1 to 3 va some reference plane (any plane). Gven the trfocal tensor T jk one can recover the eppoles v 0 ;v 00 (and fundamental matrces) [10, 22] and proceed to recover a par of homography matrces A; B as descrbed below. One can solve for A be ether choosng some lnear combnaton of the prmtve homographes or solvng for the quas-metrc plane as descrbed n the prevous secton. Thus we can assume that A s known. The correspondng homography B cannot be chosen arbtrarly because t must be assocated wth the same plane that was assocated wth the homography A. Let Hl, l =1; :::; 4, be the prmtve homographes from vew 1 to 3. Let P the sought after matrx B be represented by B = l lh l.we seek a soluton of the scalars l. We have the followng relatonshp: T jk = v 0j b k, v 00k a j = v 0j ( 4 l=1 l H l ) k, v 00k a j (7) where the left-hand sde s known (the trfocal tensor) and the rght-hand sde contans 5 unknowns whch together form an over-determned lnear system. The scalar xes the scale because v 0 ;v 00 ;Aare all determned up to scale. Taken together, from the trfocal tensor and wth the use of the prmtve homographes we can extract a set of compatble homographes (assocated wth the same plane) and the eppoles v 0 ;v 00. We are now left wth mnmzng the followng expresson: mnf(^x 0, a> 1 p + v0 1 a > 3 p + ) 2 +(^y 0, a> 2 p + v0 2 v0 3 a > 3 p + ) 2 + v0 3 (^x 00, b> 1 p + v00 1 b > 3 p + v00 3 ) 2 +(^y 00, b> 2 p + v00 2 b > 3 p + v00 3 ) 2 g whch s mnmzed wth respect to. Ths yelds a 4th order polynomal n whch thus has a closed-form soluton. The geometrc nterpretaton of ths mnmzaton process s that the soluton determnes the ponts p 0 ;p 00 on ther correspondng eppolar lnes such that the dstance (p 0, ^p 0 ) 2 +(p 00, ^p 00 ) 2 s mnmzed. Note that unlke the case of two vews, one cannot place p 0 and p 00 anywhere on ther eppolar lnes because they are coupled together by a 1-parameter degree of freedom. In partcular, the projectons of ^p 0 and ^p 00 on ther eppolar lnes may not be an admssble soluton Experments We have tested the deas put forward n the prevous secton on several real mage trplets. We took a sequence of three mages (Fg. 2). In both trplets we automatcally extracted feature ponts, and used them to compute the trfocal tensor. In addton by usng optc ow methods we have generated a dense correspondence eld between the source mages and used the tensor to reproject the rst mage onto the thrd mage. Fg. 5a dsplays the reprojected mages and as can be seen the qualty s farly good (evdence of a good tensor and good correspondence eld). We then \corrupted" the correspondence eld by applyng the optc ow algorthm on blurred copes of the orgnal mages wth a 77 kernel. Reprojecton of the rst mage usng the orgnal tensor and the corrupted ow eld s dsplayed n Fg. 5b. The deteroraton s solely due to the corrupted matches, because the tensor has remaned unchanged. We next used the \trangulaton" dea derved above to \correct" for the pont matches. Fg. 5c dsplays the reprojected thrd mage usng the orgnal tensor and the corrected ow eld. The

8 8?? (a-1) (b-1) (a-2) (b-2) (a-3) (b-3) Fg. 2. The \lab" and the \outdoor" sequence used for testng. Each sequence conssts of three mages. qualty has mproved consderably and matches the qualty of the reprojecton usng the orgnal ow eld.

9 ?? 9 6. Other Applcatons of the Bfocal Tensor Representaton In the prevous sectons we presented applcatons of the prmtve homographes whch n turn are due to the dscovery of [] F representng the famly of rank-2 homographes whch n turn are due to the bfocal tensor representaton. However, one could possbly re-derve the result [] F from purely matrx consderatons wthout relyng on the bfocal tensor. Nevertheless, there are applcatons that crtcally rely on the tensor embeddng of the fundamental matrx n the form of the bfocal tensor and n ths secton we brey dscuss two of them Vew-Synthess Fund. Matrx F Tensor <1,2,2> Pre Processng mage 2 mage 1 Dense Correspondence Tensor <1,2,2> Renderng R,t Tensor <1,2,ψ> Image ψ mage 2 mage 1 Dense Correspondence Fg. 3. Vew synthess s dvded nto two parts. In the pre-processng stage, done only once, we compute the dense correspondence and the bfocal tensor. The renderng stage, done for every novel mage, transforms the \seed" bfocal tensor to a general threevew trfocal tensor, usng user-speced parameters R; t and renders the novel vew usng the transformed tensor, the model mages and the dense correspondence Ego-moton Recovery The noton of mage-based renderng s ganng momentum both n the computer graphcs and computer vson communtes. Usng the trfocal tensor for mage-based renderng was proposed by [2]. In a nutshell, the method lnks together two real vews of a 3D scene wth a thrd vrtual vew of the scene. The tensor s then used to reproject a pont appearng n the rst two vews drectly onto the vrtual vew, wthout ever recoverng 3D structure. Movng the vrtual camera n space s done by modfyng the tensor to reect the change n the relatve poston of the vrtual vew. To bootstrap the seed tensor one would need three real vews of the object, but only two of them wll be later used for the generaton of the vrtual vew. However, usng the tensor-embedded fundamental matrx, one can use only two real mages to generate the bfocal \seed tensor". One starts wth the bfocal tensor whch s then transformed usng the user speced moton of the vrtual camera to the approprate trfocal tensor (of the orgnal two model vews and the vrtual vew to be syntheszed). From there on the trfocal tensors transform as the vrtual camera changes postons (see Fg. 3). Thus, for ths applcaton to work t s necessary to have a unform termnology for handlng 2 and 3 vews. When consderng the problem of recoverng the camera ego-moton (projecton matrces) from a stream of vews, one faces the problem of mantanng a consstency of parwse fundamental ma- (a) Fg. 4. One can compute two tensors T 123;T 234 from the four mages of the 3D scene. However, each tensor can gve rse to a derent reconstructon of the 3D structure due to nose or errors n measurements, and therefor the camera trajectory between mages 2 and 3, as captured by the fundamental matrx F 23, snconsstent between the two tensors. Fgure taken from [3]

10 10?? (a-1) (a-2) (b-1) (b-2) (c-1) (c-2) Fg. 5. The rst row (b-1,b-2) shows the reprojected thrd mage usng the orgnal dense pont matches and the tensor recovered from the pont matches. The second row (b-1,b-2) shows the reprojected thrd mage, usng the \corrupted" pont matches (by blurrng the mages pror to the computaton of ow) and the orgnal tensor. Note that the reprojected mage s corrupted due to the corrupted optc ow. The bottom row (c-1,c-2) shows the reprojected thrd mage after correctng the corrupted ow usng the orgnal tensor. trces. The consstency requrement arses from the smple fact that from an algebrac standpont a camera trajectory must be concatenated from

11 ?? 11 pars or trplets of mages. Therefore, a sequence of ndependently computed fundamental matrces or trfocal tensors, maybe optmally consstent wth the mage data, but not necessarly consstent wth a unque camera trajectory (see Fgure 4). The consstency problem can be approached by ntroducng the followng equaton whch relates the trfocal tensor between vews 1,2,3 and the bfocal tensor between vews 1,2 and the elements of the fundamental matrx between vews 2,3: T jk where T jk A, whose elements are a j = c k l F jl, v 000k a j (8) s the tensor of vews 1,2,3, the matrx, s a homography from vews 1 to 2 va some arbtrary plane, F jl s the bfocal tensor of vews 1,2, and C =[C; v 000 ] s the camera moton from vew 2 to 3 where c k l s a homography matrx from vew 2 to 3 va the (same) plane. As a result, gven the fundamental matrx between vews 1,2 and (at least) 6 matchng ponts between vews 1,2,3 one can solve for the fundamental matrx between vews 2 and 3 (.e., [v 000 ] C) whch s consstent wth the trfocal relatonshp among vews 1,2,3. Also, as a byproduct, the projecton matrx [C; v 000 ] s consstent wth the same projectve representaton due to the fact that the homographes A; C are of the same reference plane. The detals and demonstraton of ths dea can be found n [3]. 7. Summary We have ntroduced a new representaton of the blnear matchng constrant between a par of vews n terms of a tensor whch we termed the \bfocal" tensor. The motvaton for the new representaton s to establsh a uned termnology between the elements of 2-vew and 3-vew constrants. The uned termnology s acheved by representng the 2-vew constrant n a way analogously (and dentcal n form) to the trfocal tensor relatonshp. As a result, we were able to transfer the propertes known today about the trfocal tensor (especally the contracton nto homography matrces) to the realm of the 2-vew case. The byproduct of the new representaton s twofold. Frst, we have derved the famly of rank-2 homography matrces represented by[] F and ntroduced the \prmtve homographes" and ther applcatons. Second, we mentoned two other applcatons for whch the uned termnology s necessary. Taken together, t s useful to have a common language for analyzng the geometrc constrants arsng from multple-vew geometry both at the theoretcal level for purposes of obtanng a clean representaton and for applcatons where the common language s sometmes necessary (as was shown n Secton 6). Acknowledgements Our thanks to Danna Segal for wrtng the \trangulaton" code. We thank Rchard Hartley for comments on the prevous verson of ths work (n [1]) whch led to a smpler representaton of F jk. A.S. also thanks Nr Avraham for notcng the need for the scale factor n eqn. 7. Appendx A.0.1. Trlneartes and the Trfocal Tensor Three vews, p =[I;0]x;p 0 = Ax and p 00 = Bx, are known to produce four trlnear forms whose coecents are arranged n a tensor representng a blnear functon of the camera matrces A; B: T jk = v 0j b k, v 00k a j (A1) where A =[a j ;v0j ](a j s the 3 3 left mnor and v 0 s the fourth column of A) and B =[b k ;v00k ]. The tensor acts on a trplet of matchng ponts n the followng way: p s j r k T jk =0 (A2) where s are any two lnes j (s1 and j s2)ntersect- ng at p 0, and r are any two lnes ntersectng k p00. j Snce the free ndces are ; each n the range 1,2, we have 4 trlnear equatons (unque up to lnear combnatons). If we choose the standard form where s (and r ) represent vertcal and horzon-

12 12?? tal scan lnes,.e., s j =,1 0 x 0 0,1 y 0 then the four trlnear forms, referred to as trlneartes [17], have the followng explct form: x 00 T 13 p, x 00 x 0 T 33 p + x 0 T 31 p,t 11 p =0; y 00 T 13 p, y 00 x 0 T 33 p + x 0 T 32 p,t 12 p =0; x 00 T 23 p, x 00 y 0 T 33 p + y 0 T 31 p,t 21 p =0; y 00 T 23 p, y 00 y 0 T 33 p + y 0 T 32 p,t 22 p =0: These constrants were rst derved n [17]; the tensoral dervaton leadng to eqns. A1 and A2 was rst derved n [19]. The tensor s often referred to as \trlnear" or \trfocal", and we adopt here the term trfocal tensor. The trfocal tensor has been well known n dsguse n the context of Eucldean lne correspondences and was not dented at the tme as a tensor but as a collecton of three matrces (a partcular contracton of the tensor, correlaton contractons, as explaned next) [23, 24, 28]. The lnk between the trlneartes and the matrces of lne geometry was dented later by Hartley [9, 10]. Addtonal work n ths area can be found n [22, 7, 27, 12, 20, 3, 2, 25, 8, 13, 5, 26]. The tensor has certan contracton propertes and can be slced n three prncpled ways nto matrces wth dstnct geometrc propertes. These propertes s what makes the tensor dstnct from smply beng a collecton of three matrces and wll be brey dscussed next further detals can be found n [22, 18]. A.0.2. Contracton Propertes and Tensor Slces Consder the matrx arsng from the contracton, k T jk (A3) whch sa33 matrx, we denote by E, obtaned by the lnear combnaton E = 1 T j1 + 2 T j2 + 3 T j3 (whch s what s meant by a contracton), and k s an arbtrary covarant vector. The matrx E has a general meanng ntroduced n [22]: Proposton 1. (Homography Contractons) The contracton k T jk for some arbtrary k s a homography matrx from mage one onto mage two determned by the plane contanng the thrd camera center C 00 and the lne k n the thrd mage plane. Generally, the rank of E s 3. Lkewse, the contracton j T jk s a homography matrx from mage one onto mage three. For proof see [22]. Clearly, snce s spanned by three vectors, we can generate up to at most three dstnct homography matrces by contractons of the tensor. We dene the Standard Homography Slcng as the homography contractons assocated by selectng be (1; 0; 0) or (0; 1; 0) or (0; 0; 1), thus the three standard homography slces between mage one and two are T j1 ; T j2 and T j3, and we denote them by E 1 ;E 2 ;E 3 respectvely, and lkewse the three standard homography slces between mage one and three are T 1k ; T 2k and T 3k, and we denote them by W 1 ;W 2 ;W 3 respectvely. Smlarly, consder the contracton T jk (A4) whchsa33 matrx, we denote by T, and where s an arbtrary contravarantvector. The matrx T has a general meanng s well, as detaled below [18]: Proposton 2. The contracton T jk for some arbtrary sarank2correlaton matrx from mage two onto mage three, that maps the dual mage plane (the space of lnes n mage two) onto a set of collnear ponts n mage three that form the eppolar lne correspondng to the pont n mage one. The null space of the correlaton matrx s the eppolar lne of n mage two. Smlarly, the transpose of T s a correlaton from mage three onto mage two wth the null space beng the eppolar lne n mage three correspondng to the pont n mage one. For proof see [18]. We dene the Standard Correlaton Slcng as the correlaton contractons assocated wth selectng be (1; 0; 0) or (0; 1; 0) or (0; 0; 1), thus the three standard correlaton slces are T jk 1 ; T jk 2 and T jk 3, and we denote them by T 1 ;T 2 ;T 3, respectvely. The three standard correlatons date back to the work on structure from moton of lnes across three vews [23, 28] where these matrces were rst ntroduced.

13 ?? 13 References 1. S. Avdan and A. Shashua. Unfyng two-vew and three-vew geometry.inarpa, Image Understandng Workshop, S. Avdan and A. Shashua. Novel vew synthess by cascadng trlnear tensors. IEEE Transactons on Vsualzaton and Computer Graphcs, 4(3), Short verson can be found n CVPR' S. Avdan and A. Shashua. Threadng fundamental matrces. In Proceedngs of the European Conference on Computer Vson, Freburg, Germany, June Sprnger, LNCS J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hngoran. Herarchcal model-based moton estmaton. In Proceedngs of the European Conference on Computer Vson, Santa Margherta Lgure, Italy, June A. Crmns, I. Red, and A. Zsserman. Dualty, rgdty and planar parallax. In Proceedngs of the European Conference on Computer Vson, Freburg, Germany, Sprnger, LNCS O.D. Faugeras. Stratcaton of three-dmensonal vson: projectve, ane and metrc representatons. Journal of the Optcal Socety of Amerca, 12(3):465{ 484, O.D. Faugeras and B. Mourran. On the geometry and algebra of the pont and lne correspondencesbetween N mages. In Proceedngs of the Internatonal Conference on Computer Vson, Cambrdge, MA, June O.D. Faugeras and T. Papadopoulo. A nonlnear method for estmatng the projectve geometry of three vews. In Proceedngs of the Internatonal Conference on Computer Vson, Bombay, Inda, January R. Hartley. Lnes and ponts n three vews a un- ed approach. In Proceedngs of the DARPA Image Understandng Workshop, Monterey, CA, November R.I. Hartley. Lnes and ponts n three vews and the trfocal tensor. Internatonal Journal of Computer Vson, 22(2):125{140, R.I. Hartley and P. Sturm. Trangulaton. In Proceedngs of the DARPA Image Understandng Workshop, pages 972{966, Monterey, CA, Nov A. Heyden. Reconstructon from mage sequences by means of relatve depths. In Proceedngs of the Internatonal Conference on Computer Vson, pages 1058{1063, Cambrdge, MA, June M. Iran, P. Anandan, and D. Wenshall. From reference frames to reference planes: Multvew parallax geometry and applcatons. In Proceedngs of the European Conference on Computer Vson, Freburg, Germany, Sprnger, LNCS H.C. Longuet-Hggns. A computer algorthm for reconstructng a scene from two projectons. Nature, 293:133{135, B.D. Lucas and T. Kanade. An teratve mage regstraton technque wth an applcaton to stereo vson. In Proceedngs IJCAI, pages 674{679, Vancouver, Canada, Q.T. Luong and T. Vevlle. Canonc representatons for the geometres of multple projectve vews. In Proceedngs of the European Conference on Computer Vson, pages 589{599, Stockholm, Sweden, May Sprnger Verlag, LNCS A. Shashua. Algebrac functons for recognton. IEEE Transactons on Pattern Analyss and Machne Intellgence, 17(8):779{789, A. Shashua. Trlnear tensor: The fundamental construct of multple-vew geometry and ts applcatons. In G. Sommer and J.J. Koendernk, edtors, Algebrac Frames For The Percepton Acton Cycle, number 1315 n Lecture Notes n Computer Scence. Sprnger, Proceedngs of the workshop held n Kel, Germany, Sep A. Shashua and P. Anandan. The generalzed trlnear constrants and the uncertanty tensor. In Proceedngs of the DARPA Image Understandng Workshop, Palm Sprngs, CA, February A. Shashua and S. Avdan. The rank4 constrant n multple vew geometry. InProceedngs of the European Conference on Computer Vson, Cambrdge, UK, Aprl A. Shashua and N. Navab. Relatve ane structure: Canoncal model for 3D from 2D geometry and applcatons. IEEE Transactons on Pattern Analyss and Machne Intellgence, 18(9):873{883, A. Shashua and M. Werman. Trlnearty of three perspectve vews and ts assocated tensor. In Proceedngs of the Internatonal Conference on Computer Vson, June M.E. Spetsaks and J. Alomonos. Structure from moton usng lne correspondences. Internatonal Journal of Computer Vson, 4(3):171{183, M.E. Spetsaks and J. Alomonos. A uned theory of structure from moton. In Proceedngs of the DARPA Image Understandng Workshop, G. Sten and A. Shashua. Model based brghtness constrants: On drect estmaton of structure and moton. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, Puerto Rco, June G. Sten and A. Shashua. On degeneracy of lnear reconstructon from three vews: Lnear lne complex and applcatons. In Proceedngs of the European Conference on Computer Vson, Freburg, Germany, Sprnger, LNCS B. Trggs. Matchng constrants and the jont mage. In Proceedngs of the Internatonal Conference on Computer Vson, pages 338{343, Cambrdge, MA, June J. Weng, T.S. Huang, and N. Ahuja. Moton and structure from lne correspondences: Closed form soluton, unqueness and optmzaton. IEEE Transactons on Pattern Analyss and Machne Intellgence, 14(3), S ha Avdan receved the B.Sc. degree n Mathematcs and Computer Scence from Bar-Ilan Unversty, Ramat-Gan, Israel, n Currently he s a

14 14?? PhD canddate at the Hebrew Unversty, where hs research nterests are n Computer Vson and Computer Graphcs. In addton he has been workng for the past 10 years n the ndustry n the elds of CAD, GIS and Photogrammetry. A mnon Shashua, Senor Lecturer at the Insttute of Computer Scence, The Hebrew Unversty of Jerusalem, receved the B.Sc. degree n Mathematcs and Computer Scence from Tel-Avv Unversty, Tel-Avv, Israel, n 1986; the M.Sc. degree n Mathematcs and Computer Scence from the Wezmann Insttute of Scence, Rehovot, Israel, n 1989; and the Ph.D. degree n Computatonal Neuroscence, workng at the Artcal Intellgence Laboratory, from the Massachusetts Insttute of Technology, n Hs research nterests are n Computer Vson and computatonal modelng of human vson. Hs prevous work ncludes early vsual processng of Salency and Groupng mechansms, Vsual Recognton, Image Synthess for Anmaton and Graphcs, and Theory of Computer Vson n the areas of three-dmensonal processng from a collecton of two-dmensonal vews.