Generi Deouple Imge-Bse Visul Servoing for Cmers Oeying the Unifie Projetion Moel Omr Thri, Youef Mezour, Frnçois Chumette n Peter Corke Astrt In this pper generi eouple imge-se ontrol sheme for lirte mers oeying the unifie projetion moel is propose. The propose eouple sheme is se on the surfe of ojet projetions onto the unit sphere. Suh fetures re invrint to rottionl motions. This llows the ontrol of trnsltionl motion inepenently from the rottionl motion. Finlly, the propose results re vlite with experiments using lssil perspetive mer s well s fisheye mer mounte on 6 ofs root pltform. I. INTRODUCTION In imge se visul servoing (IBVS), the ontrol of the mer position is performe y neling the feture errors in the imge [8]. This yiels some egree of roustness to isturnes s well s to lirtion errors. On the other hn, if the initil error etween the initil n the esire positions is lrge, IBVS my proue errti ehvior suh s onvergene to lol minim n n inpproprite mer trjetory ue to oupling etween the ontrolle ofs []. Usully, IBVS is onsiere s servoing pproh suitle only for smll isplements so si ie onsists of smpling the initil errors in orer to ensure tht the error t eh itertion remins smll in orer to overome the IBVS prolems lrey mentione. Tht is, using pth plnning step jointly with the servoing one[2][4]. The min use of troule for IBVS is the strong nonlinerities in the reltion from the imge spe to the workspe whih re generlly oserve in the intertion mtrix. In priniple, n exponentil eouple erese woul e otine simultneously on the visul fetures n on the mer veloity (perfet ehvior) if the intertion mtrix ws onstnt, whih is unfortuntely not the se. To overome the non-linerity prolem, the pproximtion n e improve y inorporting seon orer terms (se on the Hessin [8], for instne). Another pproh onsists of seleting fetures with goo eoupling n linerizing properties. In ft, the hoie of feture iretly influenes the lose-loop ynmis in tsk-spe. In [3] fetures inluing the istne etween two points in the imge plne n the orienttion of the line onneting those two points ws propose. In [8] the reltive re of two projete surfes hs een propose s feture. In [3], O. Thri is with ISR, University of Coimr, Polo II PT-3030-290 Coimr, Portugl omrthri@isr.u.pt Y. Mezour is with LASMEA, Université Blise Psl, 6377 AUBIERE, Frne. youef.mezour@lsme.univ-plermont.fr F. Chumette is with INRIA Rennes Bretgne Atlntique, Frne. frnois.humette@iris.fr P. Corke is with CSIRO ICT Centre, PO Box 883 Kenmore 4069, Austrli Peter.Corke@siro.u vnishing point n the horizon line hve een selete. This hoie ensures goo eoupling etween trnsltionl n rottionl ofs. In [9], vnishing points hve lso een use for eite ojet ( 3D retngle), one gin to otin some eoupling properties. For the sme ojet, six visul fetures hve een esigne in [2] to ontrol the six ofs of root rm, following prtitione pproh. In [7], the oorintes of points re expresse in ylinril oorinte system inste of the lssil Crtesin one, so s to improve the root trjetory. In [6], the three oorintes of the entroi of n ojet in virtul imge otine through spheril projetion hve een selete to ontrol three ofs of n uner-tute system. In [0], Mhony et l el with the seletion of the optiml feture to ontrol the mer motion with respet to the epth xis. Ttsmon et l in [7] propose eouple visul servoing from spheres using spheril projetion moel. Despite of the lrge quntity of results otine in the lst few yers, the hoie of the set of visul fetures to e use in the ontrol sheme is still n open question in terms of stility nlysis n vliity for ifferent kins of sensor n environment. The results presente in this pper elong to series of reserh out the use of invrints for eouple imge se visul ontrol. More preisely, the invrine property of some omintions of imge moments ompute from imge regions or set of points re use to eouple the egrees of freeom from eh-other. For instne, in [4], [5], moments llow the use of intuitive geometril fetures, suh s the enter of grvity or the orienttion of n ojet. By seleting n equte omintion of moments, it is then possile to etermine prtitione systems with goo eoupling n linerizing properties [5]. For instne, using suh fetures, the intertion mtrix lok orresponing to the trnsltionl veloity n e onstnt lok igonl. However, these works only onerne plnr ojets n onventionl perspetive mers. More reently, new eouple imge-se ontrol sheme from the projetion onto unit sphere hs een propose in [6]. The propose metho is se on polynomils invrint to rottionl motion ompute from set of imge points. In this pper, we propose more generi n effiient eouple sheme vli when the ojet is efine y set of points s well s y imge regions (or lose ontour). The propose fetures lso reue the sensitivity of intertion mtrix entries to ojet epth istriution. In the next setion we rell the unifie mer moel. In Setion III, theoretil etils out feture seletion n otining the intertion mtries re etile. A new vetor
Cmer z y x X mu p Xs K πp πmu A. Invrints to rottionl motion from projetion onto sphere The eouple ontrol we propose is simply se on the invrine property of the projetion shpe of n ojet onto sphere uner rottionl motion. In this wy, the following invrint polynomil to rottions hs een propose in [6] to ontrol the trnsltionl ofs: Fig.. left) z y x Convex mirror/lens Fp Fm zm ym Unifie imge formtion (on the right), xis onvention (on the of six fetures to ontrol the six mer egrees of freeom is propose. Finlly, in Setion IV, experimentl results otine using onventionl mer n fisheye mer mounte on 6 ofs root re presente to vlite our pproh. Cm xm zs ys II. CAMERA MODEL Centrl imging systems n e moele using two onseutive projetions: spheril then perspetive. This geometri formultion lle the unifie moel ws propose y Geyer n Dniiliis in [5]. Consier virtul unitry sphere entere on C m n the perspetive mer entere on C p. The frmes tthe to the sphere n the perspetive mer re relte y simple trnsltion of ξ long the Z-xis. Let X e 3D point with oorintes X = [X Y Z] in F m. The worl point X is projete on to the imge plne t point with homogeneous oorintes p = Km, where K is 3 3 upper tringulr mtrix ontining the onventionl mer intrinsi prmeters ouple with mirror intrinsi prmeters n m = [ x y ] [ = X Z+ξ X Cp xs Y Z+ξ X The mtrix K n the prmeter ξ n e otine fter lirtion using, for exmple, the methos propose in []. In the sequel, the imging system is ssume to e lirte. In this se, the inverse projetion onto the unit sphere n e otine y: X s = λ [ x y ξ λ where λ = ξ+ +( ξ 2 )(x 2 +y 2 ) +x 2 +y 2 Note tht the onventionl perspetive mer is nothing ut prtiulr se of this moel (when ξ = 0). The projetion onto the unit sphere from the imge plne is possile for ll sensors oeying the unifie moel. III. THEORETICAL BACKGROUND In this Setion, the theoretil kgroun of the min ie of this work will first e etile. Then, six new fetures will e propose to ontrol the six ofs of the root-mounte mer. ] ξ ] () (2) I = m 200m 020 m 200m 002 +m 2 0 +m 2 0 m 020m 002 +m 2 0 (3) where: N m i,j,k = x i s h ys j h zs k h (4) h= (x s, y s, z s ) eing the oorintes of 3D point. In our pplition, these oorintes re nothing ut the oorintes of point projete onto the unit sphere. This invrine to rottions is vli whtever the ojet shpe n orienttion. In this pper, the surfe of the ojet projetion onto sphere will e use inste of the polynomil mentione ove to ontrol the trnsltionl ofs. In ft, the surfe of the ojet projetion onto sphere is nothing ut the moment of orer 0 tht n e ompute using the generl formul: m si,j,k = region xi s yj s zk ss (5) The surfe is generi esriptor tht n e ompute from n imge region efine y lose n omplex ontour or simply y polygonl urve. Furthermore, s it will e shown, fter n equte trnsformtion, new feture n e otine from the projetion surfe suh tht the orresponing intertion mtrix is lmost onstnt with the epth istriution. In the reminer of this pper, the surfe of the tringles uilt y the omintion of three nonolliner points (from set of N points) will e onsiere. For plnr ojets n tringle is of ourse plnr ojet, it hs een shown tht the intertion mtrix relte to the moment n e otine y [4]: L msi,j,k = ( m svx m svy m svz m swx m swy m swz ) (6) where : m svx = A(βm i+2,j,k (i + )m i,j,k )+ B(βm i+,j+,k im i,j+,k ) + C(βm i+,j,k+ im i,j,k+ ) m svy = A(βm i+,j+,k jm i+,j,k )+ B(βm i,j+2,k (j + )m i,j,k ) + C(βm i,j+,k+ jm i,j,k+ ) m svz = A(βm i+,j,k+ km i+,j,k )+ B(βm i,j+,k+ km i,j+,k ) + C(βm i,j,k+2 (k + )m i,j,k ) m swx = jm i,j,k+ km i,j+,k m swy = km i+,j,k im i,j,k+ m swz = im i,j+,k jm i+,j,k where β = i + j + k + 3 n (A, B, C) re the prmeters efining the ojet plne in the mer frme: r = Ax s + By s + C (7) From (6) we n show tht m swx = m swy = m swz = 0 when i = j = k = 0, thus the feture = m 000 is invrint to rottionl motions.
B. Vritions of the intertion mtrix relte to the surfe with respet to the mer position As hs een mentione ove, the prolems oserve using IBVS re in generl ue to the strong vritions of the intertion mtrix with respet to mer position. Therefore, one of the min gols of this work is to erese these vritions. Note firstly tht esigning eouple or prtitione system is step towr this gol, sine it introues terms equl to 0 in the intertion mtrix. In the following, trnsformtion is propose to erese the vrition of the intertion mtrix with respet to the ojet epth. ) Vrition with respet to trnsltionl motion: In [0], using squre ojet, it ws shown tht for goo z-xis ehvior in IBVS, one shoul hoose imge fetures tht sle s s z (z is the ojet epth). In [5], the sme ie is extene to ny ojet shpe using i-imensionl moments. Using the onventionl perspetive projetion moel, the selete feture is s = m00 in the se where the ojet is efine y n imge region. m 00 is the i-imensionl moment of orer 0 (tht is the ojet surfe in the imge). In the se where the ojet is efine y set of isrete points, the selete optiml feture is s = (µ20+µ 02), where µ ij re the entrl moments ompute from set of isrete points (see [5], for more theoretil etils). In ft, the selete fetures llows us to otin n intertion mtrix tht hnges slowly with respet to epth (n is even onstnt if the ojet is prllel to the imge plne). We now show tht, the ehvior of the surfe of n ojet projetion onto the unit sphere is similr ( z n 2 z). Let L = [L x, L y, L z, 0, 0, 0] n L = [L x, L y, L z, 0, 0, 0] e the intertion mtries relte to the projetion surfe n respetively. From (6), it n e otine tht: L x = A(3m 200 m 000 ) + 3Bm 0 + 3Cm 0 L y = 3Am 0 + B(3m 020 m 000 ) + 3Cm 0 L z = 3Am 0 + 3Bm 0 + C(3m 002 m 000 ) L x = Lx 2, L y = Ly 2, L z = Lz 2 It n e shown tht the hoie of s = is etter thn the hoie s =. To illustrte this, Figure 3 gives the vrition of the intertion mtrix entries with respet to trnsltionl motion pplie to the following tringle in the unit sphere frme: X = 0.5 0.5 0.3 0.2598 0.2598 0. 0.5 0.5 0.5 (8) (9) The tringle shpe is given on Figure 2.. The results presente in Figure 3 orrespon to onfigurtions where A = B = 0 n m 0 = m 00 = 0. Plugging ll into (8) we n otin L x = L x = L y = L y = 0. Inee, from Figures 3.() n 3.(), it n e seen tht L x = L x = L y = L y = 0 whtever the ojet epth is. In prtie, the fetures n epen minly on the trnsltionl motion long z-xis. From Figures 3.() n Fig. 2. Tringle shpes 3.(), it n lso e seen tht L z = C(3m 002 )/(2 3 2 ) is lmost onstnt n lrgely invrint to the ojet epth. On the other hn L z = C(3m 002 ) ereses to 0 when the ojet epth inreses. The vrition of intertion mtrix elements for trnsltionl motion with respet to x-xis n y-xis motion re given in Figures 3.() to 3.(f). Firstly, it n e seen tht x-xis trnsltionl motion influenes minly the entries orresponing to the x-xis n z-xis. In the sme wy, y-xis trnsltionl motion influenes minly the entries orresponing to the y-xis n z-xis. Furthermore, vrition of the intertion mtrix entries for x-xis n y-xis trnsltionl motion re more uniform for thn for. () () (e) Fig. 3. Results otine for s = ( () vrition with respet to epth, () vrition with respet to x-xis trnsltion (e) vrition with respet to y-xis trnsltion); Results otine for s = (() vrition with respet to epth, () vrition with respet to x-xis trnsltion (f) vrition with respet to y-xis trnsltion) () () (f)
2) Vritions with respet to the mer frme orienttion: Despite the ft tht the surfe of the projetion of trget onto sphere is invrint to rottions, its relte intertion mtrix epens nturlly on the mer frme orienttion. In orer to explin tht, let us onsier two frmes F n F 2 relte to the unit sphere with ifferent orienttions ( R 2 is the rottion mtrix etween the two frmes) ut with the sme enter. In this se, the vlue of the projetion surfe onto the sphere is the sme for the two frmes, sine it is invrint to rottionl motions. Now, let us onsier tht trnsltionl veloity v is pplie to the frme F. This is equivlent to pplying trnsltionl veloity to the frme F 2 ut tking into ount the hnge of frme (v 2 = R 2 v ). Sine the intertion mtrix links the fetures vrition with the veloities (i.e. ṡ = L s v), the intertion mtrix for the frme F 2 is nothing ut the intertion mtrix ompute for the frme F y the rottion mtrix R 2. This result shows tht rottionl motions o not hnge the rnk of the intertion mtrix of the fetures use to ontrol the trnsltionl ofs. C. Fetures seletion As in [6], we oul onsier the enter of grvity of the ojet s projetion onto the unit sphere to ontrol the rottionl egrees of freeom: x sg = ( x sg, y sg, z sg ) = ( m00 m 000, m 00 m 000, m 00 ) m 000 In ft, only two oorintes of x sg re useful for the ontrol sine x sg elongs to the unit sphere mking one oorinte epenent. Tht is why in orer to ontrol rottion roun the optil xis, the men orienttion of ll segments in the imge is use s feture. Eh segment is uilt using two ifferent points in geometrilly orret imge. In the se where the ojets re efine y ontours rther thn simple tringles, the ojet orienttion in the imge n e use s in [5] for instne. Finlly, s mentione previously, the invrints to 3D rottion will e onsiere to ontrol the trnsltion. For the reson mentione ove it is s = tht will e use to ontrol the trnsltionl motions inste of s =. In prtie, three ifferent trgets (i.e. three ifferent tringles or three ifferent ontours) suh tht their enters re nonolliner might e enough to ontrol the three trnsltionl ofs. In the next setion, experimentl results re presente to vlite these theoretil results. IV. EXPERIMENTAL RESULTS In this setion, simultions results re firstly presente using four non oplnr points. Therey, series of experiments using two kins of mer (onventionl n fisheye) will e shown. A. Simultion results using 3D ojets In these simultions, the set of points is ompose of 4 non oplnr points. The esire position orrespons to the 3D points oorintes efine in the mer frme s follow: 0 0.2 0 0.2 X = 0.2 0 0.2 0 (0) 0.9..2 In the first simultion, only the rottionl motion given y () hs een onsiere. The orresponing results re given on Figure 4. From Figure 4., it n e seen tht nie erese of the fetures errors is otine. Furthermore, sine the onsiere trnsltionl motion is null, the trnsltionl veloity ompute using the invrints to rottions re null lso (see Fig. 4.). If the point Crtesin oorintes were use to ontrol the mer position, s in lssil IBVS, n unesire n strong trnsltionl motion with respet to the optil xis woul hve een otine [6], []. Finlly, Figure 4. shows goo ehvior of the rottionl veloities espite the lrge rottionl isplement to perform etween the esire n the initil mer positions. θu = [ 7.90 23.70 58.0 ] o () In the seon simultion, generi motion omining the rottionl motion given y () n the following trnsltionl motion hs een onsiere: t = [ 0. 0.3 ] (2) The otine results re given on Figs. 4., 4.e n 4.f. Despite the lrge motion, it n e seen tht stisftory ehvior is otine for the feture errors (see Fig 4.). Furthermore, similr stisftory ehviors re simultneously otine for the veloities (see Figs 4.e n 4.f). From these plots, it n e lso seen tht the ehvior of the rottionl motion is still lmost ientil to the ehvior otine when only rottionl motion ws onsiere (ompre 4.e n 4.), thnks to the effiient eoupling otine using the invrint to rottions. B. Experimentl vlitions results using onventionl n fisheye mers For ll these experiments, only pproximtions of the point epths for the esire position re use. More preisely, the intertion mtries re ompute using the urrent vlues of the points in the imge n onstnt pproximte esire point epths. ) Results using onventionl perspetive mer: In first experiment, only rottionl motion roun the mer optil xis (80g) hs een onsiere etween the initil n the esire mer positions. The esire imge n the urrent one re given respetively on Figures 5. n 6.. Four omintions of tringles otine from the four point trget re use to ontrol the trnsltionl motion. The otine results re given on Figures 6., 6. n 6.. From 6., it n e seen tht nie erese of the fetures errors is otine. Furthermore, from Fig. 6., sine the onsiere trnsltionl motion is null, the trnsltionl veloity ompute using the invrints to rottions re lmost
null. The oserve smll trnsltionl veloities re ue to the wek lirtion of the mer. Finlly, Fig. 6. shows goo ehvior of the rottionl motions. In seon experiment using onventionl mer, omplex motion is onsiere etween the initil n the esire mer positions. The sme esire mer position s for the first experiment is use. The imge orresponing to the initil position of the mer is given in Figure 7.. From Figures 7., it n e notie tht the feture errors ehvior is very stisftory, espite the errors in mer lirtion n points epth (the point epths re not ompute t eh itertion). The sme stisftory ehvior is otine for trnsltionl n rottionl veloities (see Figures 7. n 7.). Inee, nie ereses of the feture errors s well s for the veloities re otine. 2) Results using fisheye mer: As for the onventionl mer, only rottionl motion roun the mer optil xis (80g) hs een onsiere first etween the initil n the esire mer positions. The esire imge n the urrent one re given respetively in Figures 5. n 8.. The otine results re given in Figures 8., 8. n 8.. From these figures, it n e notie tht the trnsltionl veloity ompute using the invrints to rottions re lmost null, s for the onventionl mer. The otine results for the rottionl veloities s well s for the feture errors is lso similr to those otine using the onventionl mer. In the lst experiment, generi motion omining rottionl motion n trnsltionl one is onsiere etween the initil n the esire positions. The imge orresponing to the esire position is given in Figure 5.. The imge orresponing to the initil position is given in Figure 9.. From Figures 9., 9. n 9., it n e seen tht stisftory ehvior is otine using the propose fetures. As with the onventionl mer, nie erese of the fetures errors s well s of the veloities is lso otine using fisheye mer. V. CONCLUSIONS AND FUTURE WORKS In this pper, generi eouple imge-se ontrol using the projetion onto the unit sphere ws propose. More preisely, the surfe of the projetions of ojets onto the sphere were use to inepenently ontrol the trnsltionl motion from the rottionl motion. Firstly, the propose eouple ontrol is vli for ll mers oeying the unifie mer moel. Further, it is lso vli for ojets efine y lose ontours (3 ontours t lest) s well s y set of points. The propose fetures llows lso to erese signifitively the vritions of the intertion mtrix with respet to the mer positions. Finlly, the ontroller hs een experimentlly vlite n results presente using two kins of mer: onventionl n fisheye. Both plnr n non plnr trget hve een use for vlitions results. Future works will e evote to exten these results to the pose estimtion prolem. REFERENCES [] F. Chumette. Potentil prolems of stility n onvergene in imgese n position-se visul servoing. In Springer-Verlg, Fig. 4. Left: Results for lrge pure rottionl motion () feture errors, ) trnsltionl veloities(m/s), ) rottionl veloities (egree/s)), right: results for lrge generl motion () feture errors, e) trnsltionl veloities (m/s), f) rottionl veloities (egree/s)) Fig. 5. Desire imges: ) onventionl mer, ) fisheye mer eitor, The Confluene of Vision n Control, volume 237 of LNCIS, pges 66 78, 998. [2] P. I. Corke n S. A. Huthinson. A new prtitione pproh to imge-se visul servo ontrol. IEEE Trnstion on Rootis n Automtion, 7(4):507 55, August 200. [3] J. Feem n O. Mithell. Vision-guie servoing with feturese trjetory genertion. 5(5):69 700, Otoer 989. [4] D. Fiorvnti, B. Allott, n A. Rini. Imge se visul servoing for root positioning tsks. Meni, 43(3):29 305, June 2008. [5] C. Geyer n K. Dniiliis. Mirrors in motion: Epipolr geometry n motion estimtion. Interntionl Journl on Computer Vision, 45(3):766 773, 2003. [6] T. Hmel n R. Mhony. Visul servoing of n uner-tute ynmi rigi oy system: n imge-se pproh. IEEE Trnstion on Rootis n Automtion, 8(2):87 98, April 2002. [7] M. Iwtsuki n N. Okiym. A new formultion of visul servoing se on ylinril oorintes system with shiftle origin. In IEEE/RSJ Interntionl Conferene on Intelligent Roots n Systems, e f
Fig. 6. Results for pure rottion motion (80g) using onventionl mer: ) initil imge, ) feture errors, )trnsltionl veloities(mm/s), ) rottionl veloities (r/s) Fig. 8. Results for pure rottion motion (80g) using fisheye mer: ) initil imge, ) feture errors, )trnsltionl veloities(mm/s), ) rottionl veloities (r/s) Fig. 7. Results for omplex motion using onventionl mer: ) initil imge, ) feture errors, )trnsltionl veloities(mm/s), ) rottionl veloities (r/s) Fig. 9. Results for omplex motion using fisheye mer:) initil imge, ) feture errors, ) trnsltionl veloities(mm/s), ) rottionl veloities (r/s) Lusnne, Switzerln, Otoer 2002. [8] J. T. Lpreste n Y. Mezour. A hessin pproh to visul servoing. In Interntionl Conferene on Intelligent Roots n Systems, pges 998 003, Seni, Jpn, Septemer 28 Otoer 2 2004. [9] J.-S. Lee, I. Suh, B.-J. You, n S.-R. Oh. A novel visul servoing pproh involving isturne oserver. In IEEE Interntionl Conferene on Rootis n Automtion, ICRA 99, pges 269 274, Detroit, Mihign, My 999. [0] R. Mhony, P. Corke, n F. Chumette. Choie of imge fetures for epth-xis ontrol in imge-se visul servo ontrol. In IEEE/RSJ Int. Conf. on Intelligent Roots n Systems, IROS 02, volume, pges 390 395, Lusnne, Switzerln, Otoer 2002. [] C. Mei n P. Rives. Single view point omniiretionl mer lirtion from plnr gris. In IEEE Interntionl Conferene on Rootis n Automtion, Rom, Itly, April 2007. [2] Y. Mezour n F. Chumette. Pth plnning for roust imge-se ontrol. IEEE Trnstion on Rootis n Automtion, 8(4):534 549, August 2002. [3] P. Rives n J. Azinheir. Liner strutures following y n irship using vnishing points n horizon line in visul servoing sheme. In IEEE Interntionl Conferene on Rootis n Automtion, ICRA 04, pges 255 260, New Orlens, Louisin, April 2004. [4] O. Thri. Utilistion es moments en sservissement visuel et en lul e pose. PhD thesis, University of Rennes, 2004. [5] O. Thri n F. Chumette. Point-se n region-se imge moments for visul servoing of plnr ojets. IEEE Trnstion on Rootis, 2(6):6 27, Deemer 2005. [6] O. Thri, F. Chumette, n Y. Mezour. New eouple visul servoing sheme se on invrints from projetion onto sphere. In In IEEE Interntionl Conferene on Rootis n Automtion, ICRA 08, Psen, Cliforni, USA, My 9-23 2008. [7] R. Ttsmon Fomen n F. Chumette. Visul servoing from spheres using spheril projetion moel. In IEEE Interntionl Conferene on Rootis n Automtion, ICRA 07, Rom, Itli, April 2007. [8] L. Weiss, A. C. Snerson, n C. P. Neumn. Dynmi sensor-se ontrol of roots with visul feek. IEEE Journl on Rootis n Automtion, RA-3(5), Otoer 987.