REMOTE POINT-OF-GAZE ESTIMATION WITH SINGLE-POINT PERSONAL CALIBRATION BASED ON THE PUPIL BOUNDARY AND CORNEAL REFLECTIONS

Transcription

1 REMOTE POINT-OF-GAZE ESTIMATION WITH SINGLE-POINT PERSONAL CALIBRATION BASED ON THE PUPIL BOUNDARY AND CORNEAL REFLECTIONS Elias D. Guestin, and Mshe Eizenman,,3 Depatment f Electical and Cmpute Engineeing, Institute f Bimateials and Bimedical Engineeing, 3 Depatment f Ophthalmlgy and Visin Sciences, Univesity f Tnt, Tnt, Ontai, Canada. elias.guestin@utnt.ca ABSTRACT This pape descibes a new methd f emte, nn-cntact pint-f-gaze (PG) estimatin that tleates head mvements and equies a simple pesnal calibatin pcedue in which the subect has t fixate a single calibatin pint. This methd uses the pupil bunday and at least tw cneal eflectins (vitual images f light suces) that ae extacted fm images captued by at least tw vide cameas. Expeimental esults btained with a pttype system exhibited RMS PG estimatin es f appx f visual angle. Such accuacy is cmpaable t that f the best cmmecially available systems, which use multiple-pint pesnal calibatin pcedues, and significantly bette than that f any the systems that use a single-pint pesnal calibatin pcedue and have been peviusly descibed in the liteatue. Index Tems Pint-f-gaze, pupil bunday, cneal eflectins, single-pint pesnal calibatin, emte gaze estimatin. INTRODUCTION The pint-f-gaze (PG) is the pint within the visual field that is imaged n the highest acuity egin f the etina nwn as the fvea. Systems that estimate the PG ae used in a lage vaiety f applicatins []-[4] such as studies f md disdes, analysis f eading and diving behavi, pilt taining, egnmics, maeting and advetising eseach, human-cmpute intefaces and assistive devices f mt-disabled pesns. Accuate PG estimatin equies the deteminatin f a set f subect-specific paametes thugh a pesnal calibatin pcedue in which subects ae equied t fixate ne me specific pints at specific times. Minimizing pesnal calibatin equiements is citical in applicatins equiing minimal subect cpeatin such as thse with infants and mentally challenged individuals. By using at least tw vide cameas and at least tw nea-infaed light suces (nea-infaed is invisible t the human ), it is IEEE CCECE pssible t estimate the PG in the pesence f head mvements and withut any cntact with the use (emte peatin) afte cmpleting a pesnal calibatin pcedue in which the subect is equied t fixate nly a single pint [5]-[8]. These single-pint calibatin gaze estimatin methdlgies ae based n the cente f the pupil and the cneal eflectins (Fig. - Inset) extacted fm the images captued by the cameas. The cneal eflectins (fist Puine images) ae vitual images f the light suces that ae ceated by the fnt suface f the cnea, which acts as a cnvex mi. Gaze estimatin methdlgies that ae based n the pupil cente inheently assume that the cente f the pupil extacted fm the images captued by the cameas cespnds t the image f the pupil cente. Hweve, in geneal, due t the efactin at the cnea (pimaily its fnt suface) fllwed by the pespective pectin at the camea, the cente f the pupil image is nt cincident with the image f the pupil cente [9]-[0]. The discepancy between these tw pints inceases with the pupil diamete and with the angle between the ptic axis f the and the ptic axis f the camea [7], []. The methdlgy pesented in this pape attempts t vecme this pblem by using the pupil bunday instead f the pupil cente. The next sectin pesents the mathematical mdel used t estimate the PG fm the pupil bunday and multiple cneal eflectins extacted fm images captued by at least tw synchnized cameas. Sectin 3 descibes a pttype system, Sectin 4 shws expeimental esults, and Sectin 5 pesents the cnclusins f this w.. MATHEMATICAL MODEL Unde the assumptins that the light suces ae mdeled as pint suces, the vide cameas ae mdeled as pinhle cameas and the cneal suface is mdeled as a spheical sectin, Fig. pesents a ay-tacing diagam, whee all pints ae epesented as 3-D clumn vects (bld fnt) in a ight-handed Catesian Wld Cdinate System (WCS). The fist stage in the estimatin f the PG cnsists f detemining the 3-D psitin f the cente f cuvatue f the cnea, c, fm the cneal eflectins extacted fm the

2 Fig. : Ray-tacing diagam (nt t scale in de t be able t shw all the elements f inteest), shwing schematic epesentatins f an, a camea and a light suce. Inset: image indicating the pupil and tw cneal eflectins. images captued by the vide cameas. Twads this end, cnside a ay that iginates fm light suce i, l i, and eflects at a pint q i n the cneal suface such that the eflected ay passes thugh the ndal pint (a..a. camea cente, cente f pectin) f camea,, and intesects the camea image plane at a pint u i (the image f the cneal eflectin pduced by light suce i in the image plane f camea ). The cnditin that the ay cming fm the pint f eflectin q i and passing thugh the ndal pint f camea,, intesects the camea image plane at pint u i, can be expessed in paametic fm as qi q, i ( ui ) f sme q,i. () The law f eflectin states tw cnditins: (i) the incident ay, the eflected ay and the nmal at the pint f eflectin ae cplana; and (ii) the angles f incidence and eflectin ae equal (the angles f incidence and eflectin ae measued with espect t the nmal at the pint f eflectin). Since any line ging thugh the cente f cuvatue f the cnea, c, is nmal t the spheical cneal suface, vect (q i c) is nmal t the cneal suface at the pint f eflectin q i. It then fllws that the line defined by c and q i must bisect the angle defined by l i, q i, and. Nting that given tw vects a and b such that a = b and a b, vect a + b bisects the angle fmed by vects a and b, the peceding cnditin can be expessed as c q i R li qi l q i i i li qi l q i qi qi qi q i whee R is the adius f cuvatue f the cnea (subectspecific and unnwn a pii). Ntice that the case a = b wuld happen nly if the line cnnecting the light suce l i and the ndal pint f the camea wee tangent t the cneal suface at q i, which des nt happen in pactice. Substituting () int (), c is expessed as a functin f q,i and R: c i ( q,i, R). Ideally, all the c i ( q,i, R) shuld be equal t each the. Hweve, in geneal, due t nise in the cdinates f the cneal eflectins extacted fm the images captued by the cameas, and because eal cneas ae nt exactly spheical, the cnditin that all the c i ( q,i, R) ae equal t each the des nt hld. T be able t slve the pblem in such situatin, it is estated as finding the values f R and q,i that minimize the sum f the squaed distances between all pssible pais f c i ( q,i, R). This minimizatin pblem can be expessed in a genealized fm as min wi (,, ),, a w a ib c b ia a q ia R c a ib q i R ia, a, ib, b, b ( q, ib b, R) whee the w i ae weight facts. The use f binay weight, () (3) IEEE CCECE

3 facts t select a subset f all available cneal eflectin images is discussed in Sectin 3. Having btained the ptimum values f R and q,i, c is finally btained as the mean f all the c i ( q,i, R): c mean ( wi ci ( q, i, R)). i, It shuld be nted that althugh this methd f the calculatin f the 3-D cdinates f the cente f cuvatue f the cnea is me cmplex than thse descibed in [5]- [8], it is me bust t nise in the cdinates f the cneal eflectins extacted fm the images captued by the cameas and t cneal aspheicity (deviatin f the shape f eal cneas fm the ideal spheical shape assumed in the mathematical mdel) []. Having detemined the 3-D cdinates f the cente f cuvatue f the cnea, c, and the adius f cuvatue f the cnea, R, the next stage in the estimatin f the PG cnsists f detemining the ientatin f the ptic axis f the fm the bunday f the pupil extacted fm the images captued by the vide cameas. The ientatin f the ptic axis f the has degees f feedm and can thus be descibed by a unit vect,, that is a functin f paametes such as its pan and tilt angles with espect t the WCS. T pvide a fmal expessin f, suppse that the WCS has its X-axis hizntal, its psitive Y-axis vetical pinting up and its psitive Z-axis pinting twads the subect. Then, let be the pan angle (angle between the pectin f the ptic axis f the n the XZ-plane and the Z-axis f the WCS) and be the tilt angle (angle between the ptic axis f the and the XZ-plane f the WCS), with > 0 f tatins t the ight, and > 0 f tatins upwads. With these definitins, is expessed as sin cs (, ) sin. cs cs Assuming that (i) the ptic axis f the cntains bth the cente f cuvatue f the cnea, c, and the pupil cente, p, (ii) the ptic axis f the is pependicula t the plane f the pupil, and (iii) the distance between c and the plane f the pupil is epesented by K (subect-specific and unnwn a pii), it fllws that p is given by p p,, c K(, ), (6) ( and the pupil bunday pints, p b,, have t satisfy ( p b, c) (, ) K. (7) (4) (5) Let p b, be a pint in the pupil bunday such that a ay f light iginating fm it tavels thugh the aqueus hum and cnea (effective index f efactin n =.3375 []) and efacts at pint b, n the fnt cneal suface as it tavels int the ai (index f efactin n = ), such that the efacted ay ges thugh the ndal pint f camea,, and intesects the camea image plane at pint v b, (the - th bunday pint f the pupil image captued by camea ). Given c, R, and v b,, the pint f efactin b, is btained as the intesectin f the line defined by and v b, and the spheical suface with cente c and adius R: b, b, b,, b, b, ( b, ) R, c whee b, v v b, b, is the unit vect in the diectin f the efacted ay passing thugh the ndal pint f the camea. Having btained b,, the unit vect nmal t the cneal suface at the pint f efactin can be expessed as b, b, b, c b, c. c R (8) (9) (0) It then fllws [9] that the unit vect in the diectin f the incident ay iginating fm the pupil bunday pint is given by n n n n ( b, b, b, () b, b, b, b, ) p Having b, and b,, p b, is given by b, b, pb, b, f sme pb,. () Substituting () int (7), slving f pb, and substituting bac int (), it fllws that p b, K ( c) (, ) (,,. b, b, b, b, (, ) (3) Assuming that the pupil is cicula with adius pupil (unnwn and time-vaying), all the p b, als have t satisfy p (. (4) b,,, p(,, pupil IEEE CCECE

4 The pblem f ecnstucting the ptic axis f the cnsists then f finding,, K and pupil such that (4) is satisfied f all and all. Since, in pactice, the cdinates f v b,, and theefe the cdinates f b, given by (8)-(9) and the cmpnents f b, given by (9)- (), ae cupted by nise, the cnditin expessed by (4) f all and all is tuned int the fllwing minimizatin pblem: min,, K, pupil p (,, p(,,, b, pupil (5) whee the ptimizatin ve pupil can be avided by simply substituting pupil pb, (,, p(,, N bunday pints (ttal f all cameas) (6) Nte that, theetically, thee pupil bunday pints ae enugh t ecnstuct the pupil (thee nn-cllinea pints define a cicle). Hweve, als due nise in the cdinates f the v b, s, many me pupil bunday pints need t be used in pactice t btain a stable slutin. The methdlgy descibed s fa can be used t ecnstuct the ptic axis f the in 3-D space withut a pii nwledge f the values f subect-specific paametes (R and. In the wds, the ptic axis f the can be ecnstucted withut the need f a pesnal calibatin pcedue, theeby allwing f the calibatinfee measuement f elative mvements. The PG, g, is defined as the intesectin f the visual axis, athe than the ptic axis, with the scene. The visual axis is the line defined by the ndal pint f the and the cente f the fvea, and exhibits a subect-specific deviatin fm the ptic axis f as much as 5º [3]. Since the ndal pint is within mm f the cente f cuvatue f the cnea [3], c, it can be assumed t be cincident with c (Fig. ). The last stage in the estimatin f the PG, then, equies the ecnstuctin f the visual axis in 3-D space. T accmplish this, the elatin between the visual and ptic axes has t be mdeled. A geneal appach t mdel such elatin elies n the definitin f an Eye Cdinate System (ECS) that is igidly attached t the. This ECS is a ighthanded 3-D Catesian cdinate system whse axes ae epesented by x, y, z. The igin f the ECS is defined at the cente f the ball and the z -axis is cincident with the ptic axis f the, pinting fwad. The x -axis and y -axis ae defined such that, when the is in the pimay psitin, the x -axis is hizntal and the y -axis pints up. The pimay psitin can be defined as the psitin f the s in the sull when the head is in a natual eect psitin and the visual axis is hizntal and pependicula t the line cnnecting the centes f tatin f bth s [4]. Let the diectin f the visual axis with espect t the ECS be descibed by the subect-specific unit vect ECS. Then, the unit vect in the diectin f the visual axis with espect t the WCS is given by R (, (7),, ) ECS whee R (,, ) is the tatin matix f the ECS with espect t the WCS, and ae the pan and tilt angles defined peviusly, and is the tsin angle abut the ptic axis f the (the cmplete analytical expessin can be fund in []). F the emainde f this pape it will be assumed that = 0 (efe t [], [5] f a discussin abut the tsin f the ). With this assumptin, it can be shwn that the tatin matix R is given by cs sin sin sin cs R 0 cs sin (8) sin cs sin cs cs Since the visual axis is assumed t g thugh c, the entie visual axis in 3-D space is given by g c,, (9) g g and the PG is given by (9) with a value f g such that the visual axis intesects the scene. In the paticula case f a plana scene at Z = 0, that value f g is given by c Z g. Z (0) T be able t calculate the PG with the abve methdlgy, the subect-specific deviatin f the visual axis fm the ptic axis f the mdeled by the unit vect ECS needs t be nwn a pii. The unit vect ECS can be detemined thugh a single-pint pesnal calibatin pcedue, which, cnceptually, can be descibed as fllws. The cente f cuvatue f the cnea, c, and the ientatin f the ptic axis f the with espect t the WCS descibed by its pan and tilt angles, and, ae detemined while the subect fixates the calibatin pint g. Having c and g, the unit vect in the diectin f the visual axis with espect t the WCS,, is fund as g c. g c Finally, ECS can be eadily btained fm (7)-(8) as ECS () T R R. () IEEE CCECE

5 T estimate the PG with the peceding mathematical mdel, it is necessay t nw the wld cdinates f the psitins f the light suces (l i ), the ndal pints f the cameas ( ), and the pupil bunday pints (v b, ) and cneal eflectins (u i ) in the images. Since the cdinates f the pupil bunday pints and cneal eflectins that ae estimated in each image ae initially measued in pixels in an image cdinate system, they have t be tansfmed int the WCS [], [6]. This tansfmatin equies all intinsic and extinsic camea paametes (nte that the psitin f is ne f the extinsic camea paametes). In summay, t be able t estimate the PG, the psitin f the light suces and the intinsic and extinsic camea paametes must be measued / estimated accuately. An example f hw t d s is pvided biefly in the next sectin. 3. PROTOTYPE SYSTEM A pttype system t estimate the PG n a cmpute sceen was implemented. This system uses tw synchnized mnchme CCD cameas (Scpin SCOR-4SOM, Pint Gey Reseach, Richmnd, BC, Canada) with 35 mm lenses (apetue set t f/5.6) and fu nea-infaed light suces (each light suce has 0 infaed LEDs with nminal wavelength f 850 nm) attached t a 9 LCD mnit (5:4 aspect ati) by a custm aluminum fame (Fig. ). The cameas ae iented such that thei ptic axes intesect at a distance f appximately 65 cm fm the sceen (typical viewing distance). Bth cameas peate at a fame ate f 0 Hz with a eslutin f 39 pixels (maximum hizntal eslutin) by 900 pixels (highest vetical eslutin that allws f 0 Hz peatin). In these cnditins, the ange f lateal head mvements is abut cm when the PG is estimated f a specific, abut 7 cm when the PG is estimated f eithe and abut 5 cm when the PG is estimated f bth s simultaneusly (bincula mde). The ange f vetical head mvements is abut 7 cm and the ange f bacwads/fwad head mvements is abut 0 cm. The use f me than the minimum f tw light suces can help t impve the system bustness by inceasing the lielihd that at least tw cneal eflectins ae available f each camea egadless f the head psitin and the PG n the sceen, and in the pesence f lid and lash intefeences. Meve, the use f me than tw light suces can help t educe the effect f cneal aspheicity. Typically, the adius f cuvatue f the fnt cneal suface inceases twads the bunday with the sclea and nly the cental pat is appximately spheical [3]. By using the tw light suces that pduce cneal eflectins that ae clsest t the pupil cente in the images, it is then pssible t educe the PG estimatin bias due t cneal aspheicity []. F the esults shwn in this pape, the light suces ae selected in such a way that the same pai f light Fig. : System setup with tw cameas and fu light suces. suces is used with bth cameas. Fmally, the selectin f the light suces f each gaze estimate is caied ut as fllws. Let DPCCR i be the distance between the cente f the pupil and the cneal eflectin cespnding t light suce i in the image fm camea. Then, afte finding M i = max(dpccr i, DPCCR i ) f each f the light suces, the tw light suces cespnding t the tw smallest M i ae selected. Afte selecting the tw light suces and thei cespnding cneal eflectin images, thee ae tw ptins. One f them is t use bth cneal eflectin images fm bth cameas, f a ttal f 4 cneal eflectin images (4CR vaiant). The the ptin is t discad the cneal eflectin image with the lagest DPCCR i ut f thse 4 cneal eflectin images (3CR vaiant). While the 4CR vaiant exhibits a lwe sensitivity t nise in the cdinates f the cneal eflectins extacted fm the images, the 3CR vaiant exhibits a lwe sensitivity t cneal aspheicity []. Nte that this selectin f cneal eflectin images cespnds t setting w i = in (3)-(4) f the selected nes and w i = 0 f the est. As indicated in the pevius sectin, t be able t estimate the PG, the psitins f the light suces and the intinsic and extinsic camea paametes must be nwn accuately. The psitins f the light suces, l i, with espect t the WCS (attached t the LCD mnit, with the XY-plane cincident with the plane f the sceen and the igin at the cente f the sceen) ae measued diectly using ules and calipes. The intinsic camea paametes and the psitin and ientatin f the cameas with espect t the WCS (extinsic camea paametes) ae detemined thugh the camea calibatin pcedue utlined in [7], [], which is based n [6]. 4. EXPERIMENTAL RESULTS Pf-f-cncept expeiments wee caied ut with thee adults withut glasses cntact lenses in bincula mde. The head f each subect was placed at 65 cm (nminal distance), 60 cm (minimum distance) and 70 cm (maximum distance) fm the sceen. F each head psitin, each subect cmpleted a numbe f tials ( f Subects and, and 7 f Subect 3). In each tial, the subect was ased t sequentially fixate 5 taget pints (aanged in a 5-by-5 ectangula gid) n the cmpute IEEE CCECE

6 Subect Subect Subect 3 Fig. 3: Mean pint-f-gaze estimates (+ : taget fixatin pint, : left, : ight ). All cdinates ae in mm. sceen. F each taget pint, 50 estimates (.5 s) f the image cdinates f the centes f the cneal eflectins and the paametes f the ellipse (cente, ma and min axes, ientatin f the ma axis) fitted t the pupil-iis bunday pints extacted fm each image wee ecded (8 pints n the fitted ellipse wee subsequently sampled at egula angula intevals). The subect-specific deviatin f the visual axis fm the ptic axis, descibed by the unit vect ECS, was detemined when the head was at the nminal distance fm the sceen and the subect fixated a taget pint at the cente f the sceen. The PG was estimated using the 3CR vaiant descibed in the pevius sectin. The RMS PG estimatin e f the thee subects is pvided in Table. Fig. 3 shws the mean PG estimates btained in each tial f bth s. Table : Expeimental RMS pint-f-gaze estimatin e. Subect Eye RMS E 3 L 4.8 mm (0.40º) R 4.84 mm (0.40º) L 7.6 mm (0.58º) R 6.6 mm (0.55º) L 6.79 mm (0.56º) R 5.99 mm (0.49º) 5. CONCLUSIONS The expeimental esults shw that using the ppsed methdlgy it is pssible t estimate the PG with an accuacy f º f visual angle afte cmpleting a single-pint pesnal calibatin pcedue. Such accuacy is cmpaable t that f the best cmmecially available systems, which use multiple-pint pesnal calibatin pcedues. Futheme, t the best f u nwledge, the PG estimatin accuacy btained with u pttype system is significantly bette than that f any the systems that use a single-pint pesnal calibatin pcedue and have been peviusly descibed in the liteatue (e.g., [5], [7], [8]). REFERENCES [] A. T. Duchwsi, A beadth-fist suvey f -tacing applicatins, Behav. Res. Meth., Instum., Cmput., vl. 34, n. 4, pp , Nv. 00. [] M. Eizenman, L. H. Yu, L. Gupp, E. Eizenman, M. Ellenbgen, M. Gema, and R. D. Levitan, A natualistic visual scanning appach t assess selective attentin in ma depessive disde, Psychiat. Res., vl. 8, n., pp. 7-8, May 003. [3] J. L. Hablu, Y. I. Ny, P. L. Tbvich, and M. Eizenman An nad assessment f cgnitive distactin: impacts n dives visual behaviu and baing pefmance, Accid. Anal. Pev., vl. 39, n., pp , Ma [4] P. Maaanta and K.-J. Räihä, Twenty yeas f typing: System and design issues, in Pc. ETRA 00, Ma. 00, pp. 5-. [5] S.-W. Shih and J. Liu, A nvel appach t 3-D gaze tacing using stee cameas, IEEE Tans. Syst., Man, Cyben. B, vl. 34, n., pp , Feb [6] E. D. Guestin and M. Eizenman, Geneal they f emte gaze estimatin using the pupil cente and cneal eflectins, IEEE Tans. Bimed. Eng., vl. 53, n. 6, pp. 4-33, Jun [7] E. D. Guestin and M. Eizenman, Remte pint-f-gaze estimatin equiing a single-pint calibatin f applicatins with infants, in Pc. ETRA 008, Ma. 008, pp [8] T. Nagamatsu, J. Kamahaa, T. I, and N. Tanaa, One-pint calibatin gaze tacing based n ball inematics using stee cameas, in Pc. ETRA 008, Ma. 008, pp [9] T. Ohn, N. Muawa, and A. Yshiawa, FeeGaze: A gaze tacing system f eveyday gaze inteactin, in Pc. ETRA 00, Ma. 00, pp [0] A. Villanueva and R. Cabeza, Mdels f gaze tacing systems, EURASIP J. Image Vide Pcess., vl. 007, aticle ID 3570, 007. [] E. D. Guestin, Remte, nn-cntact gaze estimatin with minimal subect cpeatin, PhD Thesis, Univ. Tnt, Tnt, ON, Canada, 00. (Available at [] M. C. Cbett, E. S. Rsen, and D. P. S. O Bat, Cneal Tpgaphy: Pinciples and Applicatins. Lndn, UK: BMJ Bs, 999, p. 6. [3] L. R. Yung and D. Sheena, Methds and designs - Suvey f mvement ecding methds, Behav. Res. Meth. Instum., vl. 7, n. 5, pp , 975. [4] R. H. S. Capente, Mvements f the Eyes. Lndn, UK: Pin, 977. [5] E. D. Guestin and M. Eizenman, Listing s and Dndes laws and the estimatin f the pint-f-gaze, in Pc. ETRA 00, Ma. 00, pp [6] J.-Y. Buguet. (0, Jan.) Camea Calibatin Tlbx f MATLAB. Califnia Institute f Technlgy, Depatment f Electical Engineeing, Pasadena, CA. [Online]. Available: IEEE CCECE