Modeling Pointing Tasks in Mouse-Based Human-Comuter Interactions Stanislav Aranovskiy, Rosane Ushirobira, Denis Efimov, Géry Casiez To cite this version: Stanislav Aranovskiy, Rosane Ushirobira, Denis Efimov, Géry Casiez. Modeling Pointing Tasks in Mouse-Based Human-Comuter Interactions. CDC 216-55th IEEE Conference on Decision and Control, Dec 216, Las Vegas, United States. <hal-138318> HAL Id: hal-138318 htts://hal.inria.fr/hal-138318 Submitted on 12 Oct 216 HAL is a multi-discilinary oen access archive for the deosit and dissemination of scientific research documents, whether they are ublished or not. The documents may come from teaching and research institutions in France or abroad, or from ublic or rivate research centers. L archive ouverte luridiscilinaire HAL, est destinée au déôt et à la diffusion de documents scientifiques de niveau recherche, ubliés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires ublics ou rivés.
Modeling Pointing Tasks in Mouse-Based Human-Comuter Interactions Stanislav Aranovskiy 1, Rosane Ushirobira 1, Denis Efimov 1,2 and Géry Casiez 2,3 Abstract Pointing is a basic gesture erformed by any user during human-comuter interaction. It consists in covering a distance to select a target via the cursor in a grahical user interface e.g. a comuter mouse movement to select a menu element. In this work, a dynamic model is roosed to describe the cursor motion during the ointing task. The model design is based on exerimental data for ointing with a mouse. The obtained model has switched dynamics, which corresonds well to the state of the art acceted in the humancomuter interaction community. The conditions of the model stability are established. The resented model can be further used for the imrovement of user erformance during ointing tasks. I. INTRODUCTION Pointing is a fundamental task in any grahical user interface requiring users to cover a distance to select a target with a given width. In the case of direct interactions e.g. on smarthones and tablets, users move their hand and fingers while touching the screen where the target is located. In the case of indirect interactions, users remotely control a ointer dislayed on the screen using a comuter mouse or a touchad. Indirect interactions allow to adjust the relationshi transfer function between the hysical dislacement of the finger or mouse and the ointer on screen. The transfer function defines a multilier Control-Dislay gain or CD gain alied on the inut device velocity to determine the ointer velocity [1]. The CD gain can be constant or dynamically adjusted over time deending on the inut device velocity dynamic transfer functions. All modern oerating systems use dynamic ointing transfer functions PTFs having different shaes and whose erformances vary accordingly to the target characteristics [2]. The otimization of these functions remains an oen question due to the lack of models describing the dynamics of human ointing. Such ointing models would also be relevant in the context of endoint rediction [3], where the system must know in advance where the user will oint at, in order to modify the visual feedback. Dynamically exanding targets in a toolbar is a well-known examle of such techniques [4]. Also, some tools and methods to imrove D/W ratio including target rediction can be found in [5]. *This work was suorted by ANR TurboTouch, ANR-14- CE24-9- 1. 1 Inria, Non-A team, 4 avenue Halley, 5965 Villeneuve d Ascq, France. stanislav.aranovskiy@inria.fr 2 CRIStAL UMR CNRS 9189, Villeneuve d Ascq, France. 3 Inria, Mjolnir team, 4 avenue Halley, 5965 Villeneuve d Ascq, France. Modeling ointing motions Poular models of ointing motion used for interface design and for ergonomics imrovement are usually static. They do not describe the exact ointing motion, but consider only averaged static relationshis. The most famous and ubiquitous model, Fitts law, was roosed in the seminal aer [6] and is given by MT = a + blog 2 1 + D W, 1 where MT is the movement time, a and b are ositive constants, W is the target width and D is the distance to the target center, and the logarithmic term characterizes difficulty of the task. Since the relation 1 was verified in various exerimental studies with different environmental conditions [7], many researchers were motivated to find the reasons behind Fitts law. To this end, the Iterative Corrections model has been roosed in [8]. This model considers the ointing motion as governed entirely by a visual feedback, while the Imulse Variability model roosed by [9] attributes the motion almost comletely to an initial muscle imulse. However, it was shown that neither of these models takes into account all the diverse effects observed in exeriments, so the hybrid Otimized Initial Imulse model has been develoed by Meyer et al. in [1]. This latter is acceted by the Human-Comuter Interaction HCI community as the most comlete and successful exlanation for Fitts law [5]. According to Meyer s model, a ointing motion can be divided into two stages: a raid and large movement bringing the ointer reasonably close to the target without visual tracking, followed by a slower corrective movement hitting the target under visual feedback control. It is worth stressing that the models in [8] [1] almost do not contain differential equations for motion descrition, but mainly determines ossible ointing scenarios verbally. The authors in [11] have observed in exeriments that for the raid motion, called ballistic, the movement time MT deends weakly on the target width W and is mainly roortional to D. They have also noticed that for ointing tasks with low difficulty, i.e. with relatively large values of W, the target may be reached by the ballistic motion only. Nowadays, Fitts law and the Otimized Initial Imulse model remain the most widely acceted static models of ointing motions. Recent advances in this direction are mainly focused on their extension, e.g. a model for analyzing raid oint-and-click motions taking into account human effects has been recently roosed in [12].
On the other hand, dynamic models are not as well studied and are not so numerous as static ones. The Vector Integration To Endoint VITE model [13] was widely alied in robotics to design human-like ointing movements [14]. This model describes a motion governed by an agonist-antagonist air of muscles, e.g. wrist rotation. For one muscle, the VITE model is given by Ṗt = Gt[V t] +, 2 V t = γ V t + T Pt, 3 12 1 8 6 4 2 1 2 3 a Non-adated. 12 1 12 1 8 6 4 2.5 1 1.5 2 b Partially adated. where T is the target osition, Pt is the ointer osition, V t is the ointer velocity, γ > is a constant, Gt is the GO signal, which gates the movement execution and is often assumed to be a ositive constant or a ste function, and [ ] + denotes a ositive rojection. By considering two equivalent muscles, the rojection in 2 can be omitted, then the VITE model yields a linear time-invariant secondorder system with the stable equilibrium oint P = T, V =. Moreover, since for linear systems the transient time is logarithmically related with the traveled distance, then VITE model asymtotic behavior reroduces the Fitts law 1. A slightly modified version of the VITE model with Gt 1 was used in [15] to study sensorimotor integration in absence of visual feedback. Very recently, the VITE model has been extended in [16] to handle ointing transfer functions. In this work, the GO signal G in 2 has been relaced with a motion acceleration function G V and closed-loo stability has been roven assuming no feedback delay with a non-decreasing G V function. Nevertheless, the VITE model 2 is caable only to describe the corrective stage of a ointing motion and it is not able to model the ballistic art with movement time roortional to T. Hence, the roblem of designing a dynamic model that is able to reresent both ballistic and corrective stages remains oen. Problem statement The aim of this work is to identify a dynamic model of ointing motion based on exerimental data with comuter mouse from [2] and using the methods of control systems theory. Motivated by the Otimized Initial Imulse model [1], the obtained model should have hybrid or switching nature to take into account ballistic and visual tracking hases. Additionally, it should consider the PTF imlemented in the oerating system as in [16] the case of a static PTF is selected. Such a model can be further used for designing PTFs that imrove user erformance in ointing tasks. The outline of this aer is as follows. In Section II we describe the exerimental data used in our studies. The roosed model and investigation of its roerties are given in Section III. In Section IV we resent the results of exerimental validation, and the aer is wraed u with conclusions and discussion of future directions in Section V. 8 6 4 2.5 1 1.5 c Adated. Fig. 1: Pointing trajectories for articiants differently adated to a constant-gain PTF, cursor osition in ixels II. EXPERIMENTAL DATA DESCRIPTION The model design is based on the data from the exeriments described in [2], where the authors comared the erformance of default transfer functions used by modern oerating systems Windows, OS X and Xorg in a one dimensional ointing task. They also added a constant CD gain function, as a baseline for comarison. Particiants used a 4 counts-er-inch USB corded Logitech mouse to select targets dislayed on a 23 dislay. Targets were rendered as solid vertical bars 1163 ixels 3 mm aart with 4 target widths corresonding to 9 ixels 2.32 mm, 6 ixels 1.55 mm, 3 ixels.77 mm and 1 ixel.26 mm. All the details about the exerimental setu and rocedure are available in [2]. Below, we kee only the data for the constant gain PTF. Such a choice eliminates all the uncertainties related to PTFs with variable gains and allows us to focus on modeling of human movements. However, it should be highlighted that such a constant PTF is not of everyday use for most articiants [2]. This fact is esecially imortant during the ballistic movement hase, which is not visually tracked: some articiants erformed usual wrist-elbow gestures execting to find the cursor reasonably close to the target osition, but they did not. Tyically, a constant PTF rovides less motion acceleration than a PTF with variable gain, thus the cursor is found farther from the target than exected. If a articiant did not adat his/her gestures to the constant PTF, then several ballistic motions had been erformed before the corrective hase began. In Fig. 1 trajectories of adated trained, artially adated, and non-adated articiants are given illustrating multile ballistic hases. Further we will use only data associated with the adated articiants.
T m T m T Ballistics Tracking Sensorimotor feedback A bal A tr V er P er A m 1 V m Visual ercetion 1 P m G V c Mouse 1 P Cursor c Fig. 2: The structure of the roosed ointing movement model. A. Model overview III. MODEL The structure of the roosed model is given in Fig. 2. The dynamics of the ointing device, in our case a mouse, is described by the double integrator: Ṗ m t = V m t, V m t = A m t, where P m t is the mouse osition, V m t is the mouse velocity, and A m t is the mouse acceleration. The mouse velocity is measured in counts er second and it is maed to the cursor velocity V c t measured in ixels er second, with the PTF G given by: V c = GV m. The model under consideration has a constant gain PTF, thus GV m := g V m, 4 where g > is a constant reresenting both velocity amlification and counts-to-ixels scaling. The cursor osition P c t satisfies the following differential equation Ṗ c t = V c t, and it is visually erceived yielding the observed cursor osition P er t and the observed cursor velocity V er t. The ercetion is modeled with the first-order LTI stable filter V er t = 1 τ er P c t P er t, Ṗ er t = V er t, which can also be written as 1 P er t = τ er + 1 P ct, V er t = τ er + 1 P ct, where := d dt and τ er > is the ercetion time constant. The erceived osition P er t is used as a switching signal for the mouse acceleration A m t. First, when the ointing motion has just begun, the movement is governed by the ballistic dynamics. Next, when the articiant visually marks the cursor, which means the cursor has reached some 5 redefined osition P sw, the motion switches to the visual tracking. This commutation is not instantaneous, and during the commutation eriod δ sw > zero acceleration is alied. Let t sw > be the first instant of time when P er t = P sw. Then A bal t for t < t sw, A m t = for t sw t t sw + δ sw, A tr t for t sw + δ sw < t, where A bal t and A tr t are the accelerations rovided by the ballistic dynamics and the tracking dynamics corresondingly. The reference signal T is the desired cursor osition measured in ixels and is assumed to be a constant or a ste function. When the value of T is set, the articiant estimates the desired mouse dislacement T m = T m T, which is measured in mm, according to his/her exerience with the used PTF. We assume that this aiming function can be aroximated well as a constant gain, T m T := g m T, 6 where g m >. The value T m initiates the ballistic movement that is suosed to bring the cursor sufficiently close to the desired osition T. However, if the user is not well-exerienced with the given PTF, then the resulting ballistic movement may have oor accuracy, see Fig. 1a. Now we are going to rovide the models of the accelerations A tr and A bal, which corresond to the tracking and ballistic hases, resectively. Remark 1: It is worth noting that for the considered scenario there is no stability issue related to switching, since for adated users each hase is activated just once. B. Tracking dynamics The tracking dynamics is an extension of the VITE model 2, 3 that takes into account the visual ercetion 5 with A tr t := γ k v V er t + T P er t, 7 where k v > is a model arameter. Considering the model in the corrective stage, i.e. for t > t sw + δ sw, and combining together the mouse and cursor dynamics, the ercetion and the tracking law 7, we result with the following closed-loo system: Ṗ er t = 1 τ er P c t P er t, Ṗ c t = V c t, V c t = g γ k v P c t P er t + T P er t. τ er Proosition 1: Consider the system 8 with γ >, g >, and k v > τ er >, then the system is exonentially stable with the equilibrium oint P er = P c = T and V c =. The roof of Proosition 1 is omitted due to the lack of sace. 8
1 8 6 4 2 Measurements Aroximation.2.4.6 a Position in ixels versus.5.5 1 x 14 3 25 2 15 1 Aroximation 1.2.4.6 c Acceleration in ixels er square seconds versus time in seconds. 5 Measurements Aroximation.2.4.6 b Velocity in ixels er second Fig. 3: Measured mouse osition and velocity for an exerienced articiant and their aroximations. C. Ballistics The ballistic movement is erformed without visual feedback, i.e. users do not observe the cursor osition during this hase. However, due to sensorimotor integration and roriocetion, users can estimate their wrist velocity and dislacement yielding the mouse velocity and osition feedback, see Fig. 2. Analysis of the exerimental data described in Section II shows that a tyical ballistic movement can be aroximated well with the curve resented in Fig. 3, where both measurements and aroximations are given. The aroximating curve is defined by its acceleration rofile, Fig. 3c, that consists of two constant levels with a linear transient between them. It is worth noting that this ballistic movement is not finished, i.e. the mouse velocity did not reach zero and the articiant s wrist did not sto. The reason is that, as it was found from the data, exerienced articiants usually notice the cursor and switch to visual tracking before finishing the ballistic hase, cf. Fig. 1a and Fig. 1c. The ballistic movement model is roosed in the form P m t = V m t, V m t = A bal t = f bal P m t,v m t, where P m t := P m t T m, and the function f bal, is to be defined. To calculate the function f bal, we define the acceleration rofile, Fig. 3c, for the initial conditions P m = T m and 9 V m = as a 1 for t < t 1, A bal t = a 1 + lt t 1 for t 1 t t 2, a 2 for t > t 2, where a 1, a 2 and t 2 > t 1 > are rofile arameters, a 1 a 2 <, and l := a 2 a 1 t 2 t 1 is the sloe of the linear art, a 1 l <. Define also ηt := P m t k bal V m t, 1 where k bal > is a ballistic model arameter. With some tiresome but straightforward comutations, it can be shown that the choice k bal = 2a3 1 6a2 1 lt 1 + 3a 1 l 2 t 2 1 6l2 T m 3a 1 la 1 2lt 1 ensures that for t [t 1, t 2 ] the following holds: where 11 ηt = A bal t 2 A 2 bal t + 1A bal t +, 12 := a 1a 1 2lt 1 2l 2, 1 := k bal 2l, 2 := 1 6l 2, and >, 2 <. Obviously, there is a value of the gain g m, defined in 6, such that k bal comuted in 11 is ositive. To construct the function f bal P m,v m we want to inverse 12 on the interval [a l, a r ], where a l := mina 1,a 2 < and a r := maxa 1,a 2 >. Define such an inversion as A bal = φη, and denote η l := a l 2 a 2 l + 1a l +, η r := a r 2 a 2 r + 1 a r +. To roceed we need the following assumtion. Assumtion 1: The arameters t 1, a 1, a 2 and l are such that the inequality t 1 > a2 1 a2 2. 2a 1 l holds. Now we can formulate the following roosition, whose roof is omitted due to the lack of sace. Proosition 2: Choose < k bal < k, where k := a2 1 a2 2 2t 1a 1 l, 2a 2 l where k > due to Assumtion 1. Then φη for η [η l, η r ] is a uniquely defined real-valued monotonic function with φη l = a l, φ = and φη r = a r. Remark 2: Actually, Assumtion 1 is not restrictive. It imlies that the velocity does not cross zero during the linear art of the acceleration rofile. This assumtion is reasonable for any ractical ballistic movement, see Fig. 3. Particularly, it is satisfied for all the exerimental data we analyzed. Remark 3: The inequality k bal < k also restricts admissible values of T m, see 11. In other words, there exists the uer bound ḡ m, such that k bal < k is satisfied only for
2 1 fbalη 1 2 1 5 η 5 1 15 Fig. 4: An examle of the function f bal η reconstructed from the exerimental data. g m < ḡ m, where g m is defined in 6. Practically it means that given the arameters a 1, t 1, and l, the ballistic movement cannot achieve arbitrary big values T m. Now we are in the osition to construct the desired function f bal P m,v m. Using definition 1 and choosing k bal, η l, η r and φη according to Proosition 2, we define a 2 for η < η l, f bal η := φη for η l η η r, 13 a 1 for η > η r. It is worth noting that f bal η is continuous non-decreasing function, which is strictly monotonic on [η l, η r ], and f bal =. An examle of the function f bal η reconstructed from our exerimental data is given in Fig. 4. To summarize, the ballistic movement model is given by 9, 1 and 13. Stability of this model is considered in the following roosition. Proosition 3: Consider the system 9, 1 and 13 with < k bal < k. All the trajectories of the system are bounded, and the equilibrium oint P m = V m = is globally asymtotically stable. The roof of Proosition 3 is omitted due to the lack of sace. IV. EXPERIMENTAL VALIDATION In this section the results of exerimental verification of the roosed model are resented. From the data set described in Section II we have chosen three trajectories corresonding to an adated trained articiant with a constant-gain PTF. The mouse had 4 counts er inch, and the dislay admitted a resolution 98.5 ixels er inch. The gain of the used PTF equals g = 1.5 98.5 4, and it rovides both counts-to-ixels scaling and 1.5 times velocity amlification. The samling frequency is 125Hz. For each of these trajectories we have identified arameters of the model; see Table I for the list. Comarison of the measured data with the model outut is resented in Figs. 5, 6 and 7. Here the first trajectory corresonds to a tyical ointing movement, the second trajectory resents a movement with an overshoot due to a long ballistic motion, 12 1 8 6 4 TABLE I: List of the model arameters. Ballistics Switching Tracking 2.5 1 1.5 a Position in ixels versus a 1, a 2, t 1, t 2, g m, k bal P sw, δ sw τ er, k v, γ 25 2 15 1 5.5 1 1.5 b Velocity in ixels er second Fig. 5: Comarison of the measurements with the model oututs, trajectory #1. and the third trajectory has a relatively short ballistic hase. As it can be seen from the figures, the roosed model can handle all these tyes of ointing motions and fits well the measurements. V. CONCLUSIONS The roblem of modeling a ointing movement with a comuter mouse has been considered in this aer. Following the Otimized Initial Imulse model [1], we have divided the ointing motion into three hases: the ballistics hase, the commutation hase with zero acceleration and the tracking. Therefore, the roosed model is a switching model. The ballistic hase is not visually guided by the user, but it has a sensorimotor feedback and it is modeled by a nonlinear system of Lurie form. When the redefined cursor osition is reached, and the cursor is observed by a user, the model switches after the commutation transients to the tracking hase. In this hase, the user visually erceives the cursor osition the ercetion is modeled with a linear filter and the tracking dynamics is given by an extended VITE model [13]. It is shown that both ballistic and tracking dynamics are globally asymtotically stable under some established mild conditions. The roblem of instability caused by commutation has not been analyzed since for adated users, which 12 1 8 6 4 2.5 1 1.5 a Position in ixels versus 3 25 2 15 1 5 5.5 1 1.5 b Velocity in ixels er second Fig. 6: Comarison of the measurements with the model oututs, trajectory #2.
12 1 8 6 4 2.5 1 a Position in ixels versus 25 2 15 1 5.5 1 b Velocity in ixels er second Fig. 7: Comarison of the measurements with the model oututs, trajectory #3. are the only considered in this note, each hase is activated just once. The model is validated in a series of exeriments. Validation results show that the model fits well different tyes of ointing movements. Our further researches aim in the following directions. First, the stability analysis and exerimental validation have been erformed for constant-gain PTFs only. A natural extension is to consider more realistic PTFs that deend both on cursor velocity and osition. Second, a model extension for non-adated users can be obtained, which requires the analysis of stability in the resence of switching since in such a case the ballistic and the commutation motions can be activated several times before the tracking hase is reached. Third, the effect of delays on the closed-loo stability and the erformance of ointing task can be analyzed. Fourth, the roosed model can be alied to identify users in terms of their exerience and to tune automatically the interface to match the exact user. A long-term goal in this direction is to roose an otimal in a certain sense PTF design given the model, and to extend the aroach for different ointing devices. REFERENCES [1] G. Casiez, D. Vogel, R. Balakrishnan, and A. Cockburn, The imact of control-dislay gain on user erformance in ointing tasks, HumanComuter Interaction, vol. 23, no. 3,. 215 25, 28. [2] G. Casiez and N. Roussel, No more bricolage! Methods and Tools to Characterize, Relicate and Comare Pointing Transfer Functions, Proceedings of the 24th annual ACM symosium on User interface software and technology - UIST 11,. 63, 211. [3] E. Lank, Y.-C. N. Cheng, and J. Ruiz, Endoint rediction using motion kinematics, Proceedings of the SIGCHI conference on Human factors in comuting systems CHI 7,. 637 646, 27. [4] M. McGuffin and R. Balakrishnan, Acquisition of exanding targets, in Proceedings of the SIGCHI Conference on Human Factors in Comuting Systems, ser. CHI 2. New York, NY, USA: ACM, 22,. 57 64. [5] R. Balakrishnan, Beating Fitts law: Virtual enhancements for ointing facilitation, International Journal of Human Comuter Studies, vol. 61, no. 6,. 857 874, 24. [6] P. M. Fitts, The information caacity of the human motor system in controlling the amlitude of movement. Journal of Exerimental Psychology, vol. 47, no. 6,. 381 391, 1954. [7] I. S. MacKenzie, Fitts law as a research and design tool in humancomuter interaction, Human-Comuter Interaction, vol. 7, no. 1,. 48, 1992. [8] E. R. Crossman and P. J. Goodeve, Feedback control of handmovement and Fitts Law. The Quarterly journal of exerimental sychology. A, Human exerimental sychology, vol. 35, no. Pt 2,. 251 278, 1983. [9] R. a. Schmidt, H. Zelaznik, B. Hawkins, J. S. Frank, and J. T. Quinn, Motor-outut variability: a theory for the accuracy of raid motor acts. Psychological review, vol. 47, no. 5,. 415 451, 1979. [1] D. E. Meyer, R. a. Abrams, S. Kornblum, C. E. Wright, and J. E. Smith, Otimality in human motor erformance: ideal control of raid aimed movements. Psychological review, vol. 95, no. 3,. 34 37, 1988. [11] K. C. Gan and E. R. Hoffmann, Geometrical conditions for ballistic and visually controlled movements. Ergonomics, vol. 31, no. 5,. 829 39, 1988. [12] A. Almanji, A. R. Payne, R. Amor, and T. C. Davies, A Nonlinear Model for Mouse Pointing Task Movement Time Analysis Based on Both System and Human Effects, IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 23, no. 6,. 13 111, 215. [13] D. Bullock and S. Grossberg, Neural dynamics of lanned arm movements: emergent invariants and seed-accuracy roerties during trajectory formation. Psychological review, vol. 95, no. 1,. 49 9, 1988. [14] M. Hersch and A. G. Billard, A biologically-insired controller for reaching movements, Proceedings of the First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 26, BioRob 26, vol. 26,. 167 172, 26. [15] D. M. Wolert, Z. Ghahramani, and M. I. Jordan, An internal model for sensorimotor integration, Science, vol. 269, no. 5232,. 188, 1995. [16] P. Varnell and F. Zhang, Characteristics of Human Pointing Motions with Acceleration, in 54th Conference on Decision and Control, Osaka, Jaan, 215,. 5364 5369.