Assessing the Performance of a BCI: A Task-Oriented Approach

Transcription

1 Assessing the Performane of a BCI: A Task-Oriented Approah B. Dal Seno, L. Mainardi 2, M. Matteui Department of Eletronis and Information, IIT-Unit, Politenio di Milano, Italy 2 Department of Bioengineering, IIT-Unit, Politenio di Milano, Italy bernardo.dalseno@polimi.it Abstrat An aurate way to measure the performane of a brain-omputer interfae (BCI) is important to ompare different analysis methods and different protools. The deision of whih BCI and what parameters to use should take into onsideration the expeted performane. Information transfer rate has been proposed as a benhmark, but existing information-based metris measure the hannel apaity of the BCI lassifier, whih may be muh higher than what a BCI ahieves in pratie. Therefore, we introdue a novel task-oriented approah to the measuring of BCI performanes, whih takes into aount how all the omponents of the BCI and the user interat. We apply it to the ase of a P3 speller and show how the information transfer rate may be misleading. Moreover, we determine when the introdution of an automati error-orretion method is advantageous for a given user. This shows that our approah an be used to ompare BCI variants. Introdution A brain-omputer interfae (BCI) [] is an interfae that does not entail musle movements, but it bypasses any musle or nerve mediation and onnets a omputer diretly with the brain by piking up the brain ativity signals. This simple definition hides two relevant issues: how fast it is possible to ommuniate with suh an interfae, and how often errors are made. These issues are obviously related sine the ommuniation speed highly depends on the error rate of the BCI. An aurate way to measure a BCI performane is important to ompare different analysis methods and different protools. If many BCIs or many variants are available to a user, the deision of whih one to use should take into onsideration the expeted performane for that user (other aspets, whih are important, are not the fous of this work). To this aim, the measurement of the performane should be tailored to eah speifi BCI. For example, there are ongoing studies to introdue automati orretion systems in BCIs [2] based on error potentials (ErrPs), whih are speifi variations in the EEG indued by the subjetive reognition of a ommitted mistake [3]. Suh systems may have false positives, and hene introdue new errors; so, the evaluation of the opportunity of their introdution in a BCI requires the estimation of the potential improvement in the performane of the partiular BCI. Intuitively, if the reliability of the ErrP detetor is poor, the introdution of ErrP-based orretions will ause more damages than benefits. In the literature, the performane of a BCI has been quantified by using different metris, suh as lassifiation auray, information transfer rate, letters or words per minute, kappa statisti, and others. Among them, the information transfer rate (sometimes simply alled bit rate) has been proposed as a benhmark for the evaluation of BCI performanes beause it does not depend on any partiular protool, it takes into aount both the number of hoies and time needed, and it ould be applied also to ontinuous ranges of hoies [4]. A formula for the information transfer rate is derived in [5] to ompute the (mean) number of bits transferred per trial: B = log 2 N + p log 2 p + ( p) log 2 p N, ()

2 where N is the number of possible hoies per trial, and p is the auray of the BCI, i.e., the probability that the BCI selets what the user intends. Equation () divided by the trial duration gives the mean number of bits transferred per time unit. This formula is derived from Shannon s theory [6], and it represents a measure of the mutual information between the user s hoie and the BCI seletion, under the assumption that all hoies onvey the same amount of information (i.e., they are hosen by the user with equal probability), p is the same for all the possible hoies, and that all the wrong hoies have the same probability in ase of error. In other words, a BCI system is seen as a noisy hannel, in whih the seletion of a wrong option is the noise. Aording to Shannon s noisy hannel oding theorem [6], it is possible to ahieve an arbitrarily small error probability in a ommuniation on a noisy hannel as long as the information transfer rate does not go beyond a ertain limit (the hannel apaity). The hannel apaity is given by the mutual information, and this seems to justify the use of mutual information in Equation (). The only problem is that Shannon proved his famous theorem by transferring information embedded in ever inreasing bloks of bits, and in teleommuniation pratie only very omplex error orretion shemes have permitted to get near Shannon s limit. For a BCI, where a human subjet sits at one end of the noisy hannel, it is not possible to implement suh omplex error orretion shemes, and thus the limit given by the mutual information, as in Equation (), represents a theoretial figure, unreahable by any real BCI whose error rate is signifiantly different from zero. To better understand how far from pratie an be the performane measure of Equation (), let us fous on a simple example with a standard P3 speller [7], where the P3 is used to selet letters in matrix of 36 symbols. Let us suppose that the speller speed is 4 letters per minute, and that a user ahieves a performane of 45% in auray, whih is low, but still far better than random-level auray (2.7%). By substituting p = 45% and N = 36 in Equation (), we get B =.36 bits, and the information transfer rate for this user would be 4 B = 5.4 bits/min; this is not very fast, but, still, ommuniation should be possible. Now let us look at the pratial use of suh a BCI: the most natural way would be to move on to the next letter when the speller gets one right, and to hit bakspae every time the speller is wrong. What is the real transfer rate for this speller? Sine every letter is more likely to be wrong than not, and this happens to bakspae as well, the expeted time to spell a letter orretly is infinite, and thus the answer is, on average, exatly bits/min (see also the derivation of Equation (7) in the next setion). While it ould be still possible to raise p above.5 by inreasing the number of stimulations per letter and render the P3 speller of the example usable, the hannel apaity measured by Equation () promises a performane far beyond what is attainable one the details of the BCI are taken into aount. For this reason, we propose that the measure of the performane of a BCI, e.g., the information transfer rate, takes into aount not only the behavior of the lassifier ontained in the BCI, but how all the omponents of the BCI and the user interat to perform the task the BCI is designed for. There exists a generalization of formula () that makes use of the onfusion matrix and allows eah letter to have a different probability of ourrene and a different auray [8], but suh formula has the same shortoming of (), i.e., it treats the BCI as a ommuniation hannel with no referene to the way the hannel is atually used, and it has a huge number of parameters. In order to keep the exposition simpler, we have hosen to limit the disussion to the simplified formula. All our onsiderations an be easily extended to the general formula as well. In the next setion we show how the idea of a task-oriented approah to performane measurement an be applied to a P3 speller, and we ompare it to Equation (). In Setion 3 we show how our approah an be used to evaluate the opportunity of introduing ErrP detetion in the P3 speller. Some onluding remarks follow in Setion 4. 2 Task-Related Performane Measurement We give an example of a task-related performane measurement by deriving the performane of a P3 speller, based on the omputation of the expeted time t L required to spell a letter orretly. As explained above, we use the assumptions of Equation () to keep the exposition simpler; 2

3 moreover, we assume that the auray of the speller p is onstant and the system has no memory, i.e., eah trial is not influened by the result of the previous one. If is the time duration of every single trial, the expeted time to orretly spell a letter is t L = p + ( p) ( + t B + t () L ) = + ( p) (t B + t () L ), (2) where the term p is the ontribution of the ase where the letter is orretly spelled at the first attempt, while the seond term represents the ase where the letter is wrong, so a bakspae must be entered (whih takes t B time) and the letter respelled (t () L time). As the system is stationary, we obviously have t () L = t L. In addition, as the bakspae should be treated as any other symbol, t B = t L (this an be derived formally). We an rewrite Equation (2) in an iterative formulation whih leads to t L = + 2 ( p) t L, (3) t L = 2p. (4) This relationship is valid only when 2p >, i.e., p >.5; when p.5, it should be apparent from the expanding of (2) or (3) that the expeted time to orretly spell a letter goes to infinite. Using Equation (4), we an ompute the information transfer rate for our P3 speller. This an be obtained as the ratio between the information ontained in an ever growing number of symbols spelled and the time taken to spell them: I R = E [ lim K b K K i= n i ], (5) where b is the information ontent (in bits) of one spelled symbol, and n i is the number of trials needed to spell orretly the i-th symbol. Sine K i= lim n i = E[ n] = t L, (6) K K and it holds b = log 2 (N ) bits, Equation (5) an be rewritten as I R = b = (2p ) log 2(N ). (7) t L Only N symbols an appear in real words (the bakspae annot), and we are measuring the information ontained in the spelled text, therefore the assumption of equal probability leads to the value of log 2 (N ) bits. This expression represents the expeted performane of our speller, and we an ompare it with the theoretial limit derived from Equation (): I T = B = log p 2 N + p log 2 p + ( p) log 2 N. (8) Figure ompares the two measures of information transfer rates and shows how the effetive performanes an be far from the theory. In fat, while the (8) measures the apaity of a hannel, i.e., the maximum performane obtainable by a noisy hannel, the (7) measures the expeted performane of the same hannel when information is onveyed in a speifi way; in our ase, this is the natural way of using a P3 speller. As expeted, the latter urve lies always below the theoretial limit, and it is equal to zero when the auray is too low. For high auray values, the two urves almost oinide. It is worth noting that the graph may evidene regions in where the hannel annot work (when p.5, in our ase) and also areas where differenes are very far from the theoretial limit. 3

4 6 5 Pratial Theoretial 4 Bit rate Auray Figure : Comparison between theoretial and pratial information transfer rate for a P3 speller with 36 symbols. 3 Performane Measurement with Error Detetion We now make use of the previous result to measure the improvement of the performane gained when an error-orretion apability (based on ErrP detetion) is added to our P3 speller. With the new feature added, the speller works as follows: it selets a letter by means of P3 detetion and displays it on the sreen; if an error is reognized, the latest letter is aneled, while if no error is deteted, the latest letter is kept. To derive the performane of this new system, we need to define the performane of the errororretion system. We haraterized it with two parameters:. the reall for errors (r E, the fration of times that an atual error is reognized by the error lassifier), and 2. the reall for orret trials (r C, the fration of times that a orretly spelled letter is reognized by the error lassifier). We assume that r E and r C are onstant and do not depend on the atual letter. For eah trial, four possible ases an happen, whih are listed in Table. Eah ase ours with a ertain probability, and the expeted time to orretly spell a letter obviously varies ase by ase. Both the probability and the expeted time are reported in the table. The expeted time to spell a letter orretly an be omputed as in Equation (2) by summing the time required by eah ase weighted by their respetive probabilities: t L = p + p 2 ( + t B + t L ) + p 3 ( + t L ) + p 4 ( + t L ) = p r C + ( p) ( r E ) ( + t B + t L ) + p ( r C ) ( + t L ) + ( p) r E ( + t L ), (9) where is the onstant duration of a trial, as before. Reasoning as in the previous setion, we obtain t L = t B = p r C + ( p) r E + p, () Event Probability Expeted time The P3 speller selets the orret letter, and p = p r C the ErrP lassifier orretly reognizes it. The P3 speller selets the wrong letter, and the p 2 = ( p) ( r E ) + t B + t L ErrP lassifier does not reognizes the error. The P3 speller selets the orret letter, and p 3 = p ( r C ) + t L the ErrP lassifier wrongly detets an error. The P3 speller selets a wrong letter, and the ErrP lassifier reognizes the error. p 4 = ( p) r E + t L Table : Probabilities and expeted times for the four possible outomes of a trial 4

5 p =. p =.2 p =.9 p =.4 p =.3 p =.2p =. p =.9.8 p =.3.8 p =.8.8 p =.8.6 r C.4 p =.4 p =.5 p =.6 p =.7.6 r C.4 p =.7 p =.6.6 r C.4 p =.5 p =.7 p =.6 p =.8.2 p =.9.2 p = r E r E r E (a) (b) () Figure 2: (a) Condition for the usability of a P3 speller with ErrP detetion. (b) Comparison between two P3 speller with and without ErrP detetion. () When ErrP detetion improves the performane of a P3 speller. where the result is valid only if the denominator is positive, i.e., when r C > p p ( r E ). () Figure 2.a shows the boundaries defined by Inequality () for different values of p; the inequality is satisfied for the points lying above the lines, and only in these ases the time for spelling a letter is finite (i.e., the P3 speller an be useful). It an be notied that the onstraint beomes tighter as p diminishes, with reall of errors beoming more and more important. If we now ompare Equation (4) with (), we an evaluate when the error-detetion system gives any improvement to the P3 speller. In order to have an improvement, the expeted time, t L, should be lower when error detetion is used; this leads to the inequality r C > p p r E +. (2) Figure 2.b shows the boundaries defined by Inequality (2) for different values of p (for p <.5 the omparison has no sense); points above the lines represent values of r C and r E for whih ErrP detetion is advantageous. In this ase, as p grows the area defined by Inequality (2) shrinks; this happens, beause as p grows the performane of the P3 speller gets better and better, and it beomes harder and harder for the ErrP lassifier to improve the speller performane. Figure 2. summarizes the first two graphs in Figure 2, and shows the values of r C and r E for whih ErrP detetion is advantageous for the whole range of p. As before, the part of the plane above the lines is the useful part; values below the lines are either useless or ounterprodutive. Figure 2. an be used as a guide to deide to bias the ErrP lassifier either toward orret or erroneous epohs, depending on the value of p. Some pratial examples may help to better understand the above ideas. Let say that for a partiular user the P3 speller reahes 9% auray without error orretion, and the error detetion reahes r C = r E = 83%. This situation orresponds to the ross in Figure 2., and the ross lies below the line p = 9%. So, for this partiular user the automati error orretion system is ounterprodutive. Another user s performane may be expressed by p = 7%, r E = 65%, r C = 75% (the asterisk in Figure 2.); the asterisk lies above the line p = 7%, and therefore the automati error orretion system should help this user. 4 Conlusions We have shown that the measure of the information transfer rate of a BCI, intended as the hannel apaity of the BCI lassifier, an be highly misleading. We have proposed a task-oriented approah 5

6 that takes into aount how all the omponents of the BCI and the user interat, and we have applied it to a P3-based speller. We have ompared the measure obtained with our approah to a formula for the information transfer rate that relies on a few simplifying assumptions; it is been possible to use a more general formula with less assumptions and reah the same onlusions, beause the main shortoming lies in the measuring of the hannel apaity of the BCI lassifier and not in the simplifying assumptions used. The results for the P3 speller have been extended to derive a formula for a speller with an ErrP-based orretion system added, and the formula has been used to assess the utility of the orretion system. The use of a task-oriented model permits to make firm observations about the usefulness of ErrPs in a P3 speller, and to identify operating regions/settings in whih a real improvement an be obtained. Using a model stritly related to the task performed by the BCI under study is fundamental to understand and quantify the real impat of variations of a BCI protool, like the introdution of ErrP-based orretions. While we have applied the proposed approah to two speifi ases, it is possible to use the same approah to study other kinds of BCIs and the impat of the modifiation of other parameters, and we think this should lead to a better omparison between different protools. Aknowledgments This work has been partially supported by the Italian Institute of Tehnology (IIT), and by the grant Brain-Computer Interfaes in Everyday Appliations from Politenio di Milano and Regione Lombardia. Referenes [] J. R. Wolpaw, N. Birbaumer, D. J. MFarland, G. Pfurtsheller, and T. M. Vaughan. Brainomputer interfaes for ommuniation and ontrol. Clinial Neurophysiology, 3:767 79, 22. [2] P. W. Ferrez and J. d. R. Millán. Error-related EEG potentials generated during simulated brain-omputer interation. IEEE Transations on Biomedial Engineering, 55(3): , 28. [3] M. Falkenstein, J. Hohnsbein, J. Hoormann, and L. Blanke. Effets of rossmodal divided attention on late ERP omponents. II. Error proessing in hoie reation tasks. Eletroenephalography and Clinial Neurophysiology, 78(6): , 99. [4] A. Shlögl, C. Keinrath, R. Sherer, and G. Pfurtsheller. Information transfer of an EEG-based brain omputer interfae. In Proeedings of the st International IEEE EMBS Conferene on Neural Engineering, pages , 23. [5] J. R. Wolpaw, N. Birbaumer, W. J. Heetderks, D. J. MFarland, G. S. P. Hunter Pekham, E. Donhin, L. A. Quatrano, C. J. Robinson, and T. M. Vaughan. Brain-omputer interfae tehnology: A review of the first international meeting. IEEE Transations on Rehabilitation Engineering, 8(2):64 73, 2. [6] C. E. Shannon and W. Weaver. The Mathematial Theory of Communiation. The University of Illinois Press, 949. [7] L. A. Farwell and E. Donhin. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Eletroenephalography and Clinial Neurophysiology, 6(7):5 523, 988. [8] A. Shlögl, J. Kronegg, J. E. Huggins, and S. G. Mason. Towards Brain-Computer Interfaing, hapter Evaluation Criteria for BCI Researh, pages MIT Press, 27. 6