Development of a Research System for the Analysis of Eyetracker Data. Ann Stainforth

Transcription

1 Development of a Research System for the Analysis of Eyetracker Data Ann Stainforth Bachelor of Science in Mathematics and Computer Science with Honours The University of Bath May 2007

2 This dissertation may be made available for consultation within the University Library and may be photocopied or lent to other libraries for the purposes of consultation. Signed:

3 Development of Research System for the Analysis of Eyetracker Data Submitted by: Ann Stainforth COPYRIGHT Attention is drawn to the fact that copyright of this dissertation rests with its author. The Intellectual Property Rights of the products produced as part of the project belong to the University of Bath (see This copy of the dissertation has been supplied on condition that anyone who consults it is understood to recognise that its copyright rests with its author and that no quotation from the dissertation and no information derived from it may be published without the prior written consent of the author. Declaration This dissertation is submitted to the University of Bath in accordance with the requirements of the degree of Bachelor of Science in the Department of Computer Science. No portion of the work in this dissertation has been submitted in support of an application for any other degree or qualification of this or any other university or institution of learning. Except where specifically acknowledged, it is the work of the author. Signed:

4 Abstract Eyetrackers are used to gather data on where a person is looking. These data are typically summarised with descriptive statistics and graphical traces of the eye movements over an image. A novel technique for the analysis of eyetracker data was developed to augment these descriptive summaries, in the form of cluster analysis. A Java Swing program was then designed and implemented as a stand-alone tool to combine both approaches. Further work is discussed to extend clustering techniques and to integrate them with signal processing techniques.

5 Contents 1 Introduction Areas of application Literature Review Review of Eyetracking Literature The Human Visual System Eyetracking as an analytical methodology Technology involved in eyetracker systems Format of eyetracker data Previous eyetracker research Analytical Techniques Used for Eyetracker Data Common analysis techniques Further analysis techniques for eyetracker data Signal processing analysis methods Final word of Literature Review Requirements Example Eyetracker User: Dr Ian Walker Overall Requirements Specific Requirements External interface requirements Functional requirements Design 22 ii

6 CONTENTS iii 4.1 Choice of Analysis Methods Possible signal processing implementation Decision on analysis method System design overview Interaction design and interface Interaction approach 1: Data before analysis Interaction approach 2: Analysis before data Interaction approach decision Interface Design GUI classes overview Algorithm design Constructing the System Implementation Language and Programming Choice of implementation language Integrated development environments Prototyping First Swing Application User Interface The EyeMatic System Basic Fixations Analysis Cluster Analysis Interface Testing and Evaluation System Testing Example of unit testing Results of unit testing End-user Evaluation Outline of end-user evaluation User Evaluation Results

7 CONTENTS iv 7 Conclusion Meeting Requirements Critical Evaluation Future Work Improving existing functionality Meeting un-met requirements Research investigations using this tool Patterns of eye movement Research with patterns of eye movement Personal Reflections Final statement A Initial Meeting With Ian Walker 55 A.1 Minutes of meeting B Sample Eyetracker Data 58 C Testing 59 C.1 Main Method Output C.2 Test Scenarios C.2.1 EyeMatic Testing: Scenario C.2.2 EyeMatic Testing: Scenario C.2.3 EyeMatic Testing: Scenario D Code 62 D.1 File: EyeMatic.java D.2 File: Panels.java D.3 File: myfixations.java D.4 File: myclusters.java

8 List of Figures 2.1 Visual angle (Watts) Diagram of the human fovea (Watts) The Visual Field (Watts) The ASL 504 eyetracking camera The ASL 504 eyetracker in use Example of eyetracker data An example of fixations and saccades over text (Wikipedia) Sinusoid sound wave Approximating a square wave Class structure design for eyetracker data analysis system Interaction design, Approach 1: Data before analysis Interaction design, Approach 2: Analysis before data Interface design: Analysis settings Interface design: Results view Organisation of main GUI elements Prototype swingapplication.java Protytpe interface.java Prototyping the menu bar Basic Fixations Analysis Cluster Analysis EyeMatic screen shot: Initial view v

9 LIST OF FIGURES vi 5.7 EyeMatic screen shot: File menu EyeMatic screen shot: File menu EyeMatic screen shot: Basic Fixations Analysis tab EyeMatic screen shot: Cluster Analysis tab EyeMatic screen shot: Drop down menu EyeMatic screen shot: Fixations Analysis results EyeMatic screen shot: Cluster Analysis results

10 Acknowledgements Add any acknowledgements here. Dr Leon Watts (supervisor) for his relentless support, inspiration and patience Dr Ian Walker for giving this project a purpose Mark Campbell for understanding my programming Fonsaca Malyan for endless support and proof reading Wikipedia for knowing a little about a lot vii

11 Chapter 1 Introduction When people look around themselves they gather visual information about the world, to help them understand their current situation and to work out what they might do to maintain or change it. Only a fraction of a person s environment can be inspected at any time, so examining how people direct their gaze can help to shed light on what matters to them and how they systematically look around their visual world. Eyetracking machines record the movements of a subjects eyes around a screen. Some of these movements are from consciously directed gaze, at areas that take the subject s interest, but the majority of eye movements are automatic, and completely unconscious. Eyetrackers record all these movements as coordinate data, in terms of the relative position of gaze on the screen, and do not discern between the voluntary and involuntary movements. If a scientist is to draw meaning from these raw data, it is therefore necessary to perform some degree of analysis on the eyetracker s output. The purpose of this project will be to investigate and develop computer support for the analysis of eyetracker data. This computer support will be in the form of a tool to help futher the task of researchers in inferring details of user attention and intention from patterns and cues in the eyetracker output. 1.1 Areas of application of eyetracker technology It is interesting to identify in what fields eyetracker technology is used, in order to better view the context that the project will be a part of. Where do Amazon.co.uk users look first when they arrive at the website? (etre: Five days / five heatmaps, n.d.) What size of newspaper advert gives you the best value for your money? (Duchowski, 2003). Does talking on a hands-free phone change how a lorry driver concentrates on the road? (Tijerina, s. Johnston, Parmer and Winterbottom, 2000). All these questions are investigated using eyetracker technology. 1

12 CHAPTER 1. INTRODUCTION 2 Eye tracking devices are, historically, a well-used tool for a number of research fields. Not just for academics interested in how the eyes behave in different situations, but also to psychologists who analyse those movements and use them to infer meaning about people s attention and interest when faced with different situations. A second area is the development of interactive systems, where a user navigates and communicates their intention purely through eye movements. Now the technology is also contributing in the commercial arena with widespread use in marketing and advertising. It has lead to the development of a new kind of marketresearch that tells a company detailed information about the way a user looks at their product (Interactive Minds- Remote Eye Tracking, n.d.), down to their precise eye movements around it. This has the advantage over standard market research, since even subconscious points of gaze and interest are tracked and, unlike questioning the participant, it is completely non-subjective. The computer support tool that this project intends to produce will be aimed at academic researchers, not to the commercial arena, and it will aim to provide at least one new kind of eyetracker data analysis. The project will proceed by first studying the subject of eyetrackers in depth, in a literature review, then a detailed and complete set of requirements will be drawn up. A thorough design will be produced, describing a system to meet these requirements, and that system will be built and tested. Finally, both the system and the project itself will be appraised in a detailed conclusion.

13 Chapter 2 Literature Review Eyetrackers are used to gather data on where a person is looking. In our Introduction it has been noted that they are used in both academic and commercial research and that the aim of this project is to produce a novel research tool for the analysis of eyetracker data. In order to design this tool it is important to understand two key foundation areas; the first being the comprehensive detail of the eyetracking process itself, and the second, the task of finding a novel and useful analysis technique for research in the field. 2.1 A Review of the Literature Surrounding Eyetracking as a Method for Examining Human Attention This first section identifies what eyetracking is and how it relates to the human visual system and then continues by describing the different areas where eyetracking technology is used. This is followed by a discussion of the potential of Fourier Analysis and other signal processing techniques to be used in the analysis of eyetracker data The Human Visual System Eyetracking is based on a sound understanding of the HVS (human visual system) and so it is something that is important to address before eyetracking can be discussed in any detail. The things a person can perceive are based on 3 main factors; the size as seen by the person, the visual distinctiveness of the object compared to the rest of the scene and whether it is moving. Familiarity of the object, expertise of the viewer and the environment in which it is presented can also effect the perception of an object but these are more complicated factors and beyond the scope of this work. The eye is made up of a sphere with an aperture at the front (the pupil) to allow light 3

14 CHAPTER 2. LITERATURE REVIEW 4 in onto the layer of photosensitive cells on the inside (the retina). Sensory signals from these cells are conveyed down nerves that collect at the back of the eye and transmit the impulses towards the brain (via the optic nerve). A transparent lens is positioned over the pupil and its thickness is controlled by the muscles that hold it in place. Adjustments in this thickness allow images to be focussed on when they are different distances away, by focussing light to a sharp point on the retina. The size of a visual object is normally expressed in terms of visual angle, or the angle subtended by the light projected onto the retina from the object in question. This is a convenient way of referring to all matters of extent in visual sensation since it dispenses with the need to qualify statements about absolute distance of objects from the viewer. Figure 2.1: Visual angle (Watts) Other muscles allow the eyes to work together (stereoscopically), by turning them slightly inward and outward (vergence) so that they simultaneously resolve images of the same object. These images present slightly different perspectives on the object that can be used to provide extra information on the object, in particular its distance and three-dimensional form. The retina is made up of two very different kinds of light sensitive cell. Rod cells are in the majority and can function in very low light conditions. In addition, they work together in order that movement can be detected. Cone cells require more light to function than rod cells and are concentrated in a central area of the retina known as the macular. There is a particularly high concentration in pit that equate to the central 2 degrees of the retina. This very small central region of the retina in known as the fovea and is arguably the most important component of visual-sensory system for interpreting the content of our visual world. Cone cells can differentiate light depending on its spectral frequency, meaning in effect that they can function as colour detectors. They are also capable of resolving much finer detail than rod cells. Foveal vision thus has very high acuity compared to vision arising from stimulation of other parts of the retina. All of the world around cannot be seen us at any one time, human vision is fundamentally restricted by eye s physiology. Light must enter the eye through the pupil and fall on part of the light-sensitive retina, but this does not cover the entire inner surface of the eye. Additionally, the physical location of the eye in the human head is such that in vertical plane our eyebrows and cheek bones obscure what we can seen. In the horizontal plane, the nose intrudes to the binocular centre of the field. In consequence, human vision extends for about 155 degrees in the horizontal plane of each eye. This is known as the visual field and represents the extent of visual sensation that

15 CHAPTER 2. LITERATURE REVIEW 5 Figure 2.2: Diagram of the human fovea (Watts) can contribute to monocular vision. The overlapping, or binocular, visual field extends for about 120 degrees of visual angle. Figure 2.3: The Visual Field (Watts) Seeing things in the world thus require the viewer to maximise the potential of their visual apparatus by directing the eyes and focusing light onto the fovea. Having only 2 degrees of clear vision is in principle a severe limitation. In order to build up sufficient visual information about the world, the human visual system must constantly work to gather light from different regions of the world, maximising the value of low-level features within the visual field to direct its foveal resource. In order to collect the appropriate visual information about the world using only this small window of clear vision, eye movements a constantly and involuntarily scanning the scene in front of them in a continuous series of very quick movements called saccades, Saccades: Rapid eye movements that reposition the fovea to a new location in the visual

16 CHAPTER 2. LITERATURE REVIEW 6 environment. Ie. When the eye is moves between points. Fixations: Eye movements which stabilizes the retina over a stationary object of interest. Ie. When the eye is rests on a point Eyetracking as an analytical methodology Eyetracking: The process of recording eye movements, in our case, the eye movements of humans. Further, It is assumed that these movements provide evidence of voluntary, overt visual attention. This assumption does not preclude the plausible involuntary utility of these movements, or conversely, the covert non-use of these eye-movements. (Duchowski, 2003; p.50) The book Eye Tracking Methodology : Theory and Practice (Duchowski, 2003) is a complete, concise and up-to-date text, on the field of eyetracking. Eyetrackers must be capable of gathering data that describe fixation and saccade movements. A spatio-temporal record of these data as a subject s eyes move around a scene can produce the necessary trail for analysis. The motivation for tracking someone s eyes is usually to deduce a subject s path of attention within their field of vision. It is generally assumed that from these data, some kind of insight can be obtained into the subject s interest and thought processes. Hence, questions arise of how we distinguish involuntary eye movements from those indicating conscious interest and whether there are any patterns to such movements. The discrimination of voluntary and involuntary eye movements is conceptually important and frames the approach to be taken in this project Technology involved in eyetracker systems There are two main types of eyetracker system- those that measure the position of the head, and the eye relative to that, and ones that measure the position of the eye relative to space (point of regard systems). Early technology relied on invasive eyetracking techniques, including, Electro-oculography (EOG): Electrodes are placed around the eyes to measure voltage and hence pick up eye movements relative to the head. Scleral Contact Lens: The subject wears an oversized contact lens containing a wire coil and its movements are tracked inside a magnetic field frame.

17 CHAPTER 2. LITERATURE REVIEW 7 These methods are still used today and can produce very accurate eye movement data. They do however, require expensive and technical equipment with lengthy calibrations and are very uncomfortable for the subject to wear, restricting normal head- and hence eyemovements. They must also be used in conjunction with a head tracker to produce the point-of-regard data. Duchowski (2003), describes how point of regard systems are most commonly used for tracking eye movements when vision is directed at graphics and on a screen. They are usually implemented as a video-based eyetracker that follows the pupil and corneal reflection. Corneal reflection eyetracker systems work by shining an infra-red light at the subject s eye and detecting it s reflection in the cornea, as well as using normal video-based techniques to identify the position of the pupil. Using these two pieces of information, the movement of the eye can be followed and it can be distinguished from any movement of the head. The subject sits at a monitor with a video camera positioned to view the eye. The camera is connected to specialist image-processing hardware and after some calibrations, the subject s point of gaze can be calculated in real time, as he is presented with different images on the screen. Both the operator and the subject s computers must be installed with eyetracker software, for monitoring the process and recording eye movements respectively. This project will make use of a point of regard system called ASL 504 with a video-based corneal reflection eyetracker Format of eyetracker data The software that is used with the ASL 504 systems is called GazeTracker and incorporates all the analysis methods listed above. It outputs the spatio-temporal eye movement data that is recorded in the following format: The data is in a tab separated text file and contains 9 columns of information, as shown in Figure 2.6. The system uses an initial calibration procedure to relate measurements of visual angle into x,y pixel locations on the computer screen. For each test recorded by the eyetracking device, the output data consists of a complete point-by-point eye position breakdown, a summary of the total results and then a summary for each of the user-defined look zones. These look zones are specified by the user before analysis takes place. Regions of the computer screen are defined as look zones by dragging out a selection area with a mouse. The system records these as pixel offsets from the starting location and considers them as points-of-interest during its calculations. The eyetracker continually records the subject s position of focus on the screen as an x,y pixel location. Every time the position of the eye is picked up by the video camera it is numbered and listed in the results. The x,y coordinates are noted, along with the time, the pupil size (in pixels), and the relevant look zone, if the point falls inside one. From this information it is possible to calculate fixations and saccades during the time period, but not to identify smooth pursuit movements. This is clearly a limitation to

18 CHAPTER 2. LITERATURE REVIEW 8 Figure 2.4: The ASL 504 eyetracking camera the technology, hence any experiment that is designed or used should not involve such movements Investigation of research previously done with eyetracker data All modern resources on the subject of eyetracker data, investigated in preparation for this task, refer to a landmark paper written in 1998 by Keith Rayner Eye Movements in Reading and Information Processing: 20 Years of Research (Rayner, 1998). The paper identifies the mid-1970s onwards as a third era of eye movement research. Research was focussed on basic eye movements and initial scene perception in the first and second era s respectively, but the development of computer-based eyetracker technology introduced this third era as a time where far larger amounts of data could be collected much more easily. Experimentation of all kinds was far more easily accessible to researchers and a vast new wave of work has been published. The paper brings together this whole body of eyetracker research into a single document review. Eyetracker studies of reading The majority of research done using eye tracking technology has been into the movement of eyes during reading. Reading involves a distinct and structured pattern of eye movement so it can easily be analysed and investigated and with a wide range of different questions. When reading, the eyes make jump-like saccade movements along the line of text, fixating on most words and occasionally jumping back when a target has been overshot or a word

19 CHAPTER 2. LITERATURE REVIEW 9 Figure 2.5: The ASL 504 eyetracker in use Figure 2.6: Example of eyetracker data is re-read (see Figure 2.7). The eyes never move smoothly over text like this like one would expect, in fact examples of smooth pursuits tend only to occur when the eyes are following a moving object. The details of these eye movements during normal reading were investigated in the 80s, including general length of fixations, saccade sizes and frequency of the jump-back saccades (regressions). Research was done into the details of these regressions and on when words are and aren t fixated on. They showed that regressions make up 10-15% of saccades, some are only a few letters long and occur when processing of a word isn t immediate. Others are more than ten letters and happen when the sentence isn t understood; these are less frequent in expert readers (Rayner cites Murray & Kennedy 1988). It appears that words which aren t fixated on are more likely to be short words that could be visualised peripherally from surrounding fixations, and more likely to be function rather than content words which are easier to recognise from their context. Further studies have shown the extent of variability between readers reading the same passage and even for a single reader on different passes. They show that eye movements

20 CHAPTER 2. LITERATURE REVIEW 10 Figure 2.7: An example of fixations and saccades over text (Wikipedia) can be used to infer the online processing of the words being read, that complexity and style of text effect reading speed (Rayner cites Jacobson & Dodwell 1979) and that reading aloud slows you down, so your saccades jump around until your speech catches up (Rayner cites Levy-Schoen 1981). Eyetracker studies of perceptual span An interesting technique was developed to investigate the concept of the perceptual span, ie. the effective visual field, or number of letters that can be picked up in each fixation. A moving window blanks out text a certain distance away from the point of regard, moving interactively on a monitor as the subjects eyes move. This allows the size of perceptual span to be considered, from investigating when the subject does or doesn t notice the effect. The converse device of a foveal mask blanks out the text at the point of focus, then the subject is questioned on what peripheral information they can remember. Resulting studies concluded things like how the perceptual span in asymmetric in English readers covering up to 4 letters to the left of the fixation and up to 15 to the right, how this varies with different difficulties of text, different ages of subject and whether or not the span extends to the lines above and below. Eyetracker studies of visual search Moving on from the area of reading, eyetracker studies are also employed in investigations of eye movements in visual search situations- a user is set to locate a target item in a scene of other items. An example of one such study has shown that search task difficulty affects saccade and fixations lengths; the harder the target is to distinguish from other objects, the longer the fixation durations and the shorter the saccade lengths (Rayner cites Zelinsky and Sheinberg 1997). Other work explores the direction and accuracy of initial saccades,

21 CHAPTER 2. LITERATURE REVIEW 11 and whether general search paths are pre-programmed and hence repeatable. In scene perception analysis, eyetracker studies have led to the theory that subjects regard the entire scene in low detail in a brief initial instance. Further eye movements are considered to do the job of filling in the gaps or examining the details (Rayner cites GR Loftus and Mackworth 1978). Hence, these later movements are considered to be irrelevant by some, while others debate the validity of the study (Rayner cites Rayner and Pollatsek 1992). Findings have also come to light that when perceiving a scene, the eyes quickly move to an object that is out of place, or at least physically distinct in the scene (Rayner cites Freidman 1979), but these have not been replicated and the conclusion can only be drawn that more important and interesting objects are fixated generally more and longer. Fixation time on objects that do not belong in the scene, however, is shown to be longer and this shows that the movements are related to some real-time cognitive processing of what is being viewed. (Rayner cites Antes and Penland 1981) Other, minor eyetracker studies involve tracking how eyes navigate sheet music, deal with optical illusions and read foreign languages. Summary of eyetracker studies The application of eyetracking in research is clearly a vast field that is expanding all the time. The main issues to draw from the field, for purposes of this project, are threefold, 1. reading tasks are a special case 2. perceptual span is an important analytic concept related to difficulty of interpretation 3. initial whole-scene inspection suggests differences in inspection strategy as a function of time spent looking at a visual scene 2.2 Analytical Techniques Used for Eyetracker Data Common techniques for analysing eyetracker data The kinds of analysis techniques that are commonly used to analyse eyetracker data often take the form of statistical summaries and simple descriptive information. The most widely used examples of such techniques are look zones, gaze trails and heat maps Look zones Look zones are regions of interest on the screen or image that are defined by the user before analysis takes place. Usually the method of defining these zones is for the user to identify areas of the image in which he expects more interest or eye movement. Once the area has

22 CHAPTER 2. LITERATURE REVIEW 12 been decided it is commonly described to the analysis system by selecting the look zone area on the screen. A number of these look zones can be defined for each image and when the analysis process runs, extra information is collected about movements in these areas of the scene. The kind of data that is collected about the look zones are the number of fixations in the defined area, the frequency of fixations and the total duration of fixation in the zone. Gaze trails Gaze trails involve the analysis system plotting all the recorded eye-positions from the scan of a particular scene and superimposing them over the original image. This allows the user to see graphically how eye movement progressed around the scene during scene processing. It also allows the user to identify correlation between the pattern of gaze and the features of the original image. Heat maps Heat maps are a very effective visual output created from analysis of eyetracker data. They are constructed through the process of identifying hotspot areas of the image, areas where the coordinated were fixated frequently or for a long duration. These hotspot areas areas are displayed overlayed onto the original image, where hot areas are identified with the colours of a standard thermal imaging display. The process creates an immediately interpretable image, helping the analyst to quickly identify the points of interest on the original image. Although providing interesting information about the exact movements of the subject s gaze around the screen, these analysis methods are lacking in finding a way that summarises the nature of movement around an image, in terms of a pattern or movement signature that could be used to compare different scene scanning styles from different images Further analysis techniques for eyetracker data Applications of more complex analysis of eyetracker data than the kind described above have been emerging in eyetracker research. A study by Bednarik and Tukiainen considers eye movement patterns and expertise. The work examines the ratio of fixations between different look zones over time, converting them into binomial form of increase or decrease in ratio over time (Bednarik and Tukiainen 2006). The method of Fourier analysis is by Harris, Wallman and Scudder (1990) in their study, Fourier Analysis of Saccades in Monkeys and Humans. Here, the Fourier analysis technique is used to transform eye movement data from two kind of subject (monkeys and

23 CHAPTER 2. LITERATURE REVIEW 13 humans) and the graphical results of the method (power plots) are compared in an attempt to identify similar patterns in visual responses. The conclusion here is that further analysis on eyetracker data is being done in research situations and that there is an interest in signal processing methods (eg. Fourier analysis) for drawing higher level information than the standard descriptive statistics Signal processing analysis methods There are many kinds of dynamic situation that can be represented as a waveform. Network traffic, cosmic radiation, use of colour in an image. Usually complex and hard to understand. Signal processing describes any technique that will decompose a complex signal or waveform and reduce it to isolate a pattern or meaningful signal. An example of signal processing is in receiving a noisy radio signal and processing it so that original signal is clear again. The technique in this case is to identify the meaningful data in the complex, noisy signal. Signal processing searches for a discernable pattern or trend in the data, and in this case finds the pattern of the original radio signal. The value of signal processing in this project is in finding a pattern from the eyetracker data. The analysis techniques that are provided by current eyetracker software take the form of simple descriptive statistics. An area of analysis that is much less well represented by common eyetracker software is in finding a pattern or signature to the process of visually processing an image. Signal processing techniques can be used in identifying such a pattern, to break down the complex and multivariate eye-movement data into far more simple components. Introducing a technique to eyetracker analysis like this would complement the existing functionality of eyetracker software. It could allow the analyst to investigate wider questions such as what kind of visual or mental tasks produce similar results from the signal processing or whether any kind of signature pattern can be identified for different visual tasks. Fourier analysis, components analysis and cluster analysis are examples of a signal processing technique that perhaps could be used in the eyetracker system to be developed. Using the Fourier transform for data analysis Fourier Analysis: The mathematical process of using the Fourier transform to break down a complex waveform into a series of component sinusoid curves with specific frequencies, amplitudes and phases. The Fourier transform to the frequency domain w is given by the function, for every real number w. X(w) = 1 2π x(t)e iwt dt (2.1)

24 CHAPTER 2. LITERATURE REVIEW 14 The book An Introduction to the Psychology of Hearing (Moore, 2003), describes the use of Fourier analysis, in the context of processing sound, to examine characteristics of the human auditory system. The waveforms in question are sound waves, which, in their simplest form, look like repeating sinusoid curves. Figure 2.8: Sinusoid sound wave Frequency is the number of times per second the wave repeats Amplitude is the degree of movement the sound vibration is causing Phase describes how far through the repeat you are at a given time Fourier analysis transforms this data from a function of time into a function of frequency. The frequency of the component results are plotted on a graph called a power spectrum and the component with the lowest frequency (tallest sinusoid curve) is called the fundamental component. Example: A square sound wave can be built up by successively adding together a series of smaller and smaller sinusoid waves. The Fourier transform gives a description of these component waves. frequencies and amplitudes give the relevant power plot. Their descending Components analysis Components analysis methods are statistical techniques to simplify a multidimensional data sets into lower dimensions for analysis. The dimensionality is most commonly reduced to 2D to facilitate visualisation of the results graphically. Principal Components Analysis (PCA) starts with its data in terms of a number of coeffi-

25 CHAPTER 2. LITERATURE REVIEW 15 Figure 2.9: Approximating a square wave cients, or as signal described at discrete intervals and first completely deconstructs it. The method then attempts to build new coefficients (components), in terms of the old ones, in such a way that the maximum amount of the data s variance is described in the the first component, the principal component. The following components are ordered in terms of the amount of variance where the principal component and just one or two others should encapsulate the majority of the variance of the data. The lower components are discarded and the principal components serve to maintain the characteristics of the data set, but at a lower dimension. In the terms of signal processing the 2d dimensional graph of the first two components should represent significant features or patterns of the original signal. Cluster analysis Cluster analysis is another method for identifying patterns in data. The method works by identifying clusters of points related by some difference factor. Often this difference factor is distance and cluster analysis can be used to identify groups of points that are physically close to one another. The process works by considering each datum and considering its similarity to other data

26 CHAPTER 2. LITERATURE REVIEW 16 in the set. A specific algorithm is used to determine how close or dense a number of points must be to be described as a cluster, as well as taking into consideration how many points must be in a cluster for it to be of interest. The results from cluster analysis can be in the form of a list where the data points in each cluster are identified, the time the clusters were identified and the density of that particular cluster, often described by some measure of clustering. This is another, higher level example of analysis of data that could be used in processing gaze points from an eyetracker machine. The results of the cluster analysis can be seen as a description of the pattern of movement around the screen, making up a kind of signature from each analysis. These results could even be transformed into a kind of visual signature graph that immediately represents the important information from the results in a graphical form. 2.3 Final word of Literature Review The key issues to be drawn from this review are that the data produced from an eyetracker are currently analysed in a number of ways, many descriptive statistics are standard, but the scientific interest is in techniques which process the data at a higher level. The method for associating collections of fixations as part of the strategy of scanning the scene have been considered and cluster analysis and Fourier analysis techniques appear to be the two most promising approaches. The next chapter reports the requirements analysis, combining the issues identified here with advice from a domain expert

27 Chapter 3 Software Requirements For Eyetracker Data Program This chapter will discuss how an example target user outlines his needs for a new analysis system for eyetracker data. These will then be used to draw up a key set of specific production requirements, forming the basis of this development project. Finally, the requirements will be briefly discussed to assess their suitability and important dependencies. 3.1 Example Eyetracker User: Dr Ian Walker An example user for the eyetracker data analysis tool being developed was identified in the person of Dr Ian Walker, a psychologist at the University of Bath Psychology Department. Dr Walker fits the criteria from our Introduction of an academic who uses eyetracker data in his research. He uses eyetracker data in the field of driving safety research, to investigate drivers attention in traffic situations. The reasoning behind the work is that a driver s gaze fixations when scanning a traffic scene correlate with his attention. Fixations that last a long time, or are revisited can be considered as points-of-interest in the scenes. In highly dynamic traffic situations if these points of interest are on non-essential components of the scene then they can be considered as distractions to the driver. Hence, eyetracker investigations are made with the purpose of analysing how drivers scan scenes and how this might lead to accidents. An example of the kind of research question he addresses is given by the paper Drivers gaze fixations during judgements about a bicyclists intentions (Walker and Brosnan, 2007). The paper investigates the question of where a driver is looking as he approaches the situation of a cyclist manoeuvring at a T junction. The experiment involved taking photographs of the situation and tracking the eye movements of subjects presented with the images. Analysis of the eyetracker data leads to the conclusions that because drivers can see a cyclist s face they automatically make eye contact eye movements, which can lead to 17

28 CHAPTER 3. REQUIREMENTS 18 distraction that wouldn t happen when presented with a car. Dr Walker will be broadly cast in the role of our software engineering client for the purpose of this project. His input will be valuable both in constructing a set of requirements for our product, and in evaluating the final piece. His needs are obviously focussed towards traffic situations analysis as a particular field of study, however, it is intended that the tool we develop should be applicable to any field where diagnostic- eyetracker technology is a potential resource. 3.2 Overall Requirements of Example User Ian Walker An initial meeting with the example client Dr Ian Walker was arranged to identify what his particular requirements and priorities are for a system to analyse eyetracker data. The overall requirements that Dr Walker communicated were as follows. To produce an eyetracker data tool that provides an additional analysis method to his existing system. This analysis method should be novel and have the potential to further research in the area. Fourier analysis is widely discussed as a promising and preferred method for doing this. The tool should deal with eyetracker data in the format that it is output from the existing system. Ie. a tab separated file of eyetracker data in 9 columns, where each row of data describes a gaze point (a detected eye position) on the screen and each column is a field of that data. The program results should allow a researcher to carry out investigations with eyetracker data. The results should also be represented graphically where appropriate, especially to help interpret results from any complex analysis process like Fourier analysis. It is necessary that the system be usable by Dr Walker or a similar research scientist, who have experience with specialist computer programs, but are not necessarily programmers themselves. This means that the interface should be a GUI and should use language and layout that will suit the intended user. It is re-iterated that time should not be wasted trying to duplicate the wealth of sophisticated functions, charts and graphics provided by the existing system. Instead, effort should be concentrated on investigating the novel analysis technique and creating a tool that will perform it. Full minutes of the meeting can be found in Appendix A. 3.3 Specific Requirements Specific requirements were then drawn-up using a combination Dr Walker s needs and of prior knowledge and experience in what a good system should provide.

29 CHAPTER 3. REQUIREMENTS 19 Here, three levels of priority are indicated by the use of words low, medium and high, low signifies low priority requirements medium signifies medium priority requirements high signifies high priority requirements The basic fixations analysis that is referred-to describes a process that produces first level descriptive eyetracker results, of the kind that is trivially provided by other eyetracker software. This data will include information about how many fixations took place amongst the data in input gaze data, how long each of those fixations lasted for and where they were located. The further analysis mentioned describes some second-level eye data analysis, eg. signal processing or cluster analysis, where the results serve to describe a pattern of movement, rather than summarising x-y movement directly External interface requirements User Interface: 1. Interface is navigable by specialist, but nontechnical, user (high) 2. Interface allows user to run basic fixations analysis on the input data (high) 3. Interface allows user to run further analyses on the input data (medium) 4. Interface is designed in such a way that reasonable adjustments are made for accessibility by persons with a disability (low) 5. Interface appears aesthetically appealing and consistent (low) 6. Interface is implemented as a Java GUI using Swing components (high) 7. Interface runs fully on any Windows machine with suitable JRE (high) Functional requirements Read in: 1. Locate input file 1.1 System provides the facility to locate and input an eyetracker data file (high) 1.2 Interface provides the facility to locate and input an eyetracker data file (medium) 2. Read in eyetracker data 2.1 System reads in eyetracker data from viewing a single scene, in the format of a tab separated text file with 9 columns (high)

30 CHAPTER 3. REQUIREMENTS System reads in a single scene of eyetracker data and its summary from a tab separated text file with 9 columns (low) 2.3 System reads in eyetracker data from multiple scenes stored in a single 9 column tab separated text file (low) 3. Parse eyetracker data 3.1 System is able to parse eyetracker data into suitable data structures (high) 3.2 System is able to parse scene summary data into suitable data structures (low) 4. Display input data 4.1 System displays data from input file of eyetracker data (high) 4.2 System allows analyst to select subsets of input data to display (low) Analysis: 5. System performs basic fixations analysis on the eyetracker data (high) 6. System performs further analyses on the eyetracker data (high) 7. System allows the analyst to compare results of operations (medium) 8. System allows the analyst to compare results of operations visually (low) 9. System allows the analyst to make changes to parameters of a previous analysis and re-run the process (high) Output: 10. Produce meaningful output 10.1 System displays output from basic fixations analysis (high) 10.2 System displays output from further analysis (high) 11. Produce graphical output 11.1 System produces graphical output for basic fixations analysis (low) 11.2 System produces graphical output for further analysis (medium) 12. Allow saving of output 12.1 System allows the user to save results of analysis algorithms to file (medium) 12.2 System allows the user to save graphical represesntations of analysis results as image files (low)

31 CHAPTER 3. REQUIREMENTS Produce timely results 13.1 Time taken to produce results is reasonable for the size of input file (high) 13.2 Time taken to produce basic fixations analysis results does not exceed 5 seconds (high) 13.3 Time taken to produce further analysis results does not exceed 5 minutes (high)

32 Chapter 4 Design This chapter describes how the design of the system was approached, starting with the choice of the novel analysis method the tool should provide and then going on to cover the underlying system design and aesthetic design choices. 4.1 Choice of Analysis Methods for Eyetracker Data In the analysis system being produced, it will be a trivial task to provide various examples of first-level statistics from the eyetracker data. This will be called basic fixations analysis and will include, the number of gaze-points reported in the data the number of fixations that occurred the position and duration of any fixations that occurred the percentage of total time spent fixated The focus of the system, however, is to be on the further, novel, analysis technique it can provide. In this section, the possibilites and final choice of analysis method are discussed Possible signal processing implementation Out of the possible signal processing methods, list..., they were weighed for various reasons. weighing up. All in all, Fourier analysis proved to be the most applicable and most interesting technique. Not only do we have examples of its use with eyetracker research, but it is also a technique that the example user showed an interest in(/or say something about requirements). The following implementations were then considered as ways of providing Fourier Analysis functionality. 22

33 CHAPTER 4. DESIGN 23 Matlab and the FFT function The mathematical programming language Matlab implements the Fourier transform in the form of the fft and fft2 functions. FFT: Discrete Fourier transform of vector x. ie. X(k) = N j=1 Where, w N = e ( 2πi)/N is the Nth root of unity x(j)w (j 1)(k 1) N (4.1) Usage: Y = fft(x) Returns the discrete Fourier transform (DFT) of vector X, computed with a fast Fourier transform (FFT) algorithm. FFT2: Two dimensional discrete Fourier transform of matrix x. Usage: Y = fft2(x) [cite a reference of matlab documentation] To implement this as an analysis method in the eyetracker research system, the whole program could be written in Matlab. Alternatively, the main system could be written in another programming language and the Matlab function either integrated into the program, or studied and rewritten in native code. A further possibility is that the system simply triggers a call to open the Matlab environment and automatically calculate the fft on some eyetracker data. Java and the FFT package An open source java package jnt.fft also provides the fast fourier transform functionality. Amongst the classes are, Package jnt.fft: RealFloatFFT RealFloatFFT_Radix2 Abstract Class representing FFT s of real, single precision data. Computes FFT s of real, single precision data where n is an integral power of 2. The eyetracker system could possibly be written in another language and still integrate this package, but it is more likely that this would be used as part of a fully Java-based system Decision on analysis method The Matlab implementation of the fft function was very usable as part of the Matlab environment, but programming the whole system in the mathematical language with no GUI options was not desirable, neither was completely translating the complicated algorithm, or dealing with the cross-language problems of integration. Due to the complexities involved

34 CHAPTER 4. DESIGN 24 in the Fourier transform process, there was difficulty interpretting the results from the Java fft functions. For both Fourier analysis options, a significant further programming hurdle of graphing the output would have been required to view the results data in its optimum form, ie. power plots. Hence the primary method of the program was chosen to be cluster analysis, with coding requirements that could definitely be met and no limitating language of implementation. A secondary task was conceived, in the form of providing a Fourier Analysis method in the program, only if time and programming ability allowed. 4.2 System design overview The design of the system will be split into a number of broad classes as outlined in figure 4.1. Figure 4.1: Class structure design for eyetracker data analysis system Create & Show GUI The driving entity in the data analysis system is the Create & Show GUI object. The other four objects that make up the system are all refered to from this object. It s purpose

35 CHAPTER 4. DESIGN 25 is to first build the GUI, providing the main interface to the eyetracker system, and then to provide functionality that triggers communication with the other classes and triggers changes in the display of the GUI itself. The build panels() method causes the GUI to be built, starting with a top-level container (main panel), and including key components such as the results panel and the two buttons (cluster run btn and fixations run btn). The load eyedata() method refers to the read in object and produces a result called eye data, to hold the information about eye movements. Other methods include the the run fixations() and run clusters() methods, which refer to the Fixations analysis and Cluster analysis objects respectively, and new file() method, which empties the eye data variable and also alters the appearance of many of the GUIs panels. Cluster settings anto be passed to the Fixations analysis and Cluster analysis objects when they are refered to. Read in The Read in entity is responsible for reading-in external files of eyetracker data. It has read line() and read lines() methods defined for this purpose and also a build input data() method that converts data read in from the file into a suitable data structure or format for use by the system s other objects. Th variables input file and input data contain the external file s filepath and the data structure which is created using build input data(); Fixations Analysis Fixations analysis contains the method to find fixations() from some eye data. The fixations that are found are stored in the variable found fixations. Fixation data and fixation drift are parameters of the fixations analysis process. The Cluster Analysis object is similar, but also has a variable to store a cluster graph for a graphical representation of the output. Save Results The Save Results entity keeps variables containing the fixations rslts and the clusters rslts as well as the name of the input file it is going to write the results data back in to. It offers three saving options, save clusters(), save fixations(), or save both().

36 CHAPTER 4. DESIGN Interaction design and interface In order to design the interface for the eyetracker data analysis tool it was useful to consider the flow of action that a user would want to take through the tool s processes. Two scenarios were identified as possible approaches the user might make to the flow of data; One where the user would select their input file before addressing the question of analysis parameters, and the other where the analysis technique is chosen before the input file is Interaction approach 1: Data before analysis The first approach to the situation is that the user would load in a data file and then choose which analysis they would like to perform on it. The steps a user would take would go as follows, 1. Read input data file 2. View preview data of input file 3. Choose the required analysis technique (Fixations or Cluster analysis) 4. Choose the analysis parameters 5. Run the chosen analysis 6. View the analysis results 7. Save results to output file 8. Return to step 1a. Read new input file A sketch interface for this approach can be seen in Figure 4.2. The design in the Figure introduces the choice of analysis methods as tabs on a tabbed pane and the panes are not enabled until the user has loaded some data at the top of the screen. This forces the user to follow the data first approach Interaction approach 2: Analysis before data The second approach is that the user would decide which analysis technique they would like to use, and specify input data file as part of the settings of the method. ie. 1. Choose the required analysis technique (Fixations or Cluster analysis) 2. Choose the analysis parameters and input file 3. Run the chosen analysis 4. View the analysis results A sketch interface for this approach is Figure 4.3. On the sketch the two different analyses are represented by buttons. The button for your analysis of choice is pressed and this

37 CHAPTER 4. DESIGN 27 causes a settings box to appear as a new window. The parameters are chosen and the analysis is run Interaction approach decision Both of these two interaction approaches are good, however by comparing the two diagrams it is noticeable that Approach 1 has more flexibility. Hence, Approach 1 was chosen to to be the interaction model for this eyetracker system Interface Design The sketch of interaction approach 1 was used as a starting point for the system s main interface design. As listed in the system s functional requirements, graphical output should be provided by the eyetracker tool, and this is a medium level priority (Requirement ). This need for graphical results on the screen lead to the idea of dividing into two, with the right hand side dedicated to displaying graphics and charts. The system must also provide a facility to browse for input data files and so a filechooser was also included in the design. This was positioned at the top of the screen because it will be the first thing that a user will need to see and interact with. Other design choices include the tabbed panes, which will help help make effective use of space on the screen, and use of menu bars which will make the application appear consistant with a Windows application. These are both examples of where the design should help meet the requirement (Interface appears aesthetically appealing consistant) Two diagrams were produced as a result of this design, to summarise how the final view of the eyetracker interfaces is intended to look. The first, Figure 4.4 shows how the program will look when a file has been uploaded and analysis panels are enabled for the entry of parameters. The top panels shows a preview summary of the data that is loaded in, in order to give the user an idea of what they are working with and what stage in the processing they are at. The second diagram, Figure 4.5, shows how they program will look when displaying the results of an analysis. The name of the analysis that was carried out and the file it performed on will be listed on the left hand panel, as well as the numeric results from the process. On the right hand panel graphical output will be displayed. 4.4 GUI classes overview This section about the GUI design for the user interface will go into detail about exactly what widgets will be put together to make the interface described in the last section. The

38 CHAPTER 4. DESIGN 28 organisation chart, Figure 4.6 show hows the relationship of the key components of the interface will be, described in the language of Java Swing components. The top level container will be a JFrame and this will contain the JMenuBar and the main panel, called the NorthSouth panel. This panel will contain a North Panel, for the input data/loaded data bar, and a SouthPane which is a JSplitPane in order to divide the screen neatly in two. The facility of the split pane will also allow the user to resize the ratio of graph-panel to results-panel if they so require. The northpane will begin in the state of an input panel, containing components like a load button, a filechooser and a text box to enter the input file. Once the data has been loaded this will be replaced with a loaded panel containing labels of information about the loaded file and a New file button that will allow the GUI to be reset. The JSplitPane on the main part of the interface will contain a JTabbedPane for the analysis tabs and a scrollable panel for the graphics to be displayed in. Each tab in the tab-pane will contain a panel describing one of the analysis methods and allowing the user to select the parameters for its settings. For example, the basicpanel will contain a label describing the analysis technique, as well as a number of drop-down boxes and radio buttons for the user to set the analysis parameters. Most importantly, the panel will contain a runfixations button that will trigger the Create and Show GUI class to call Fixations Analysis on the loaded data and then display the results. 4.5 Algorithm design

39 CHAPTER 4. DESIGN 29 Figure 4.2: Interaction design, Approach 1: Data before analysis

40 CHAPTER 4. DESIGN 30 Figure 4.3: Interaction design, Approach 2: Analysis before data Figure 4.4: Interface design: Analysis settings

41 CHAPTER 4. DESIGN 31 Figure 4.5: Interface design: Results view

42 CHAPTER 4. DESIGN 32 Figure 4.6: Organisation of main GUI elements

43 Chapter 5 Constructing the System The eyetracker data analysis system was given the prototype name EyeMatic, to convey the idea that it deals with EYE movement data and that it is a theoretical research tool and not yet a proven final product. Details of how the EyeMatic system was implemented are discussed in this chapter, starting with technology choices, then moving on to describe the prototyping that took place. Finally, there is a walk through of the system that was produced, including screen shots and snippets of code for explanation. 5.1 Implementation Language and Programming The language to use for the bulk of the programming was a key decision in the design of the eyetracker analysis system Choice of implementation language The requirements for the system prescribe a user-friendly GUI interface. Various high level programming languages with GUI toolkits were considered for programming this and the underlying system, including C++ for its speed, and Python for its ease of coding, so quick protoype development. Eventually, Java was chosen to implement the system, with its Swing toolkit for the GUI. This was due to a combination of prior experience with the language, breadth of its functionality, and lightweight nature of the toolkit elements, giving uniform behaviour on all platforms and a pluggable look-and-feel dependant on the native environment. 33

44 CHAPTER 5. CONSTRUCTING THE SYSTEM Integrated development environments The benefits of using an integrated development environment (IDE) for the engineering of code were considered against the initial overheads that using new software can entail. Good looking GUI interfaces are quick and easy to construct with an IDE once the user has spent some time learning the process, and advanced source code editors are also provided. However, persistant problems with downloading and installation of the chosen Java IDE (NetBeans), coupled with the restriction of only developing on machines with the environment installed, lead to the decision of hand coding the Java. This choice also has the benefit of producing generally more pure and readable source code. The aims and requirements of the project were also taken into consideration, in that their emphasis is not on the development of a very complex or detailed GUI, for which an IDE would be necessary. Instead, the focus is on the provision of a good-looking and functional interface that needs to be suitable only for specialist use, not widespread commercial sale. It was decided that, for this, a simple hand-coded GUI is quite suitable. 5.2 Prototyping Prototyping was done before programming could begin on the system. This was important to assess initial programming experience and capabilities as well as the capabilities of the Java language with Swing First Swing Application The program swingapplication.java (Figure 5.1) was written to demonstrate the simple use of Java Swing. The short program displays as a simple GUI with a button and a label. When the button is clicked, the first line of an eyetracker data file is read-in and the label alters to display the contents of that line. The key value of this program was to prototype a Java GUI application, demonstrating the build, run and display of its frames and panels. This capability is an essential start-point in meeting requirement (implementing the program s interface as a Java GUI). The prototype demonstrates adding swing components (a button and a label) to a top-level container, and to handling a button click with an ActionListener and produce a change. Further, the read-in method prototypes the handling of a line of eyetracker data in the format of 9 column, tab separated data (requirement ) User Interface Interface.java (Figure 5.2) was written to demonstrate the layout and use of some more complex swing components. The program builds, runs and displays the backbone of the

45 CHAPTER 5. CONSTRUCTING THE SYSTEM 35 Figure 5.1: Prototype swingapplication.java GUI outlined in section (Section 4.4) consisting of a top-level frame containing 3 main areas; a file input panel, a graphical display panel, and an analysis methods panel. Figure 5.2: Protytpe interface.java The features prototyped in this section include building the split pane, tabbed panes, filechooser and menubar (Figure 5.3) widgets into an interface, as well as general layout managing. Such features help to meet the requirements that the interface must be navigable by a non-technical user (requirement ) and that it should appear aesthetically appealing and consistent (requirement ).

46 CHAPTER 5. CONSTRUCTING THE SYSTEM The EyeMatic System Figure 5.3: Prototyping the menu bar shows how the final EyeMatic system was produced using Java and Swing components. The system itself can be run from the executable file EyeMatic.jar or by running the Java EyeMatic.class file. 5.4 Basic Fixations Analysis Fixations analysis is a simple analysis process that goes through every gaze point from the input data and determines the number and location of any fixations made. FIXATION: An instance when the eye rests on a point for a certain amount of time as it gathers visual information. The settings for basic fixation analysis: DRIFT: This is the number of pixels that the measured gaze point can move by and still count as the same fixation. This means the program allows for eyetracker error, and genuine fixation eye-drift. This figure can be set by the user. DURATION: This is the length of time that the eye must be measured resting on a point before it counts as a fixation and is measured in seconds. 5.5 Cluster Analysis Cluster analysis uses a simple clustering algorithm to identify clusters in the eyetracker s gaze data as possible points of interest. For each cluster it determines a centre point, identifies a rectangle around the cluster and then the points-per-pixel density of this rectangle. The time the cluster was first entered is also reported, as well as a list and total number of the gaze-points involved.

47 CHAPTER 5. CONSTRUCTING THE SYSTEM 37 Figure 5.4: Basic Fixations Analysis CLUSTER: A region or chain of gaze points where no member of the cluster is more than a given cut-off distance from another member of the cluster. The settings for cluster analysis: CUT-OFF DISTANCE: The distance in pixels that identifies how close together points must be to count as a cluster. MIN CLUSTER SIZE: Once all the gaze points have been separated into clusters of the above cut-off distance, this setting allows the user to state how large a cluster must be to be of interest to them, in terms of the number of gaze points in the cluster. Figure 5.5: Cluster Analysis

48 CHAPTER 5. CONSTRUCTING THE SYSTEM Interface A series of screen shots showing the interface and final implementation of EyeMatic. Figure 5.6: EyeMatic screen shot: Initial view

49 CHAPTER 5. CONSTRUCTING THE SYSTEM 39 Figure 5.7: EyeMatic screen shot: File menu Figure 5.8: EyeMatic screen shot: File menu Figure 5.9: EyeMatic screen shot: Basic Fixations Analysis tab

50 CHAPTER 5. CONSTRUCTING THE SYSTEM 40 Figure 5.10: EyeMatic screen shot: Cluster Analysis tab Figure 5.11: EyeMatic screen shot: Drop down menu

51 CHAPTER 5. CONSTRUCTING THE SYSTEM 41 Figure 5.12: EyeMatic screen shot: Fixations Analysis results Figure 5.13: EyeMatic screen shot: Cluster Analysis results