Combining evidence in personal identity verification systems

Transcription

1 Pattern Recognition Letters ELSEVIER Pattern Recognition Letters 18 (1997) Combining evidence in personal identity verification systems J. Kittler *, J. Matas 1, K. Jonsson, M.U. Ramos S~nchez University of Surrey, Guildford GU2 5XH, United Kingdom Abstract A methodology for fusing multiple instances of biometric data to improve the performance of a personal identity verification system is developed. The fusion problem is formulated in the framework of the Bayesian estimation theory. The effect of different fusion strategies on the error probability is analysed theoretically. The proposed methodology is then demonstrated on the problem of personal identity verification using multiple facial images. Experimental studies on the M2VTS database confirm the predicted improvements in performance. A reduction in error rates of up to 40% is achieved. The performance gains are initially monotonic but they tend to saturate after integrating the first few observations. It is also shown that the fusion based on rank order statistic, i.e., the median, is robust to outliers Elsevier Science B.V. Keywords: Evidence combination; Face identification; Multiple observation fusion; Error sensitivity I. Introduction The ultimate goal of designing a personal identity verification system is to achieve the best possible performance. Traditionally, the decision making process would draw on a single modality of biometric information. However, recent studies show that the fusion of multiple sources of evidence is likely to lead to tangible benefits in terms of improved performance and robustness of the verification system. Several different sources of information can be brought to bear on the verification task. In the first instance one can employ multiple experts to provide opinions about the same biometric data. Such approaches have been shown theoretically to improve recognition rates both for discriminant function (Tumer and Ghosh, 1996) and Bayesian (Kittler et * Corresponding author. j.kittler@ee.surrey.ac.uk. J g.matas@ee.surrey.ac.uk. al., 1997) decision rules. Experimental evidence of performance gains that can be achieved by expert fusion has been reported in (Genoud et al., 1996) in the context of voice based speaker verification. Three verification methods are used to compare voice references extracted from the speech signal represented by Linear Prediction Coefficients. The global decision is taken by applying a Furui threshold to the individual methods and then combining the individual results according to a majority rule. The fusion of multimodal biometric data is exemplified by (Brunelli and Falavigna, 1995; Brunelli et al., 1995; Bigiin et al., 1997; Jourlin et al., 1997). For instance the person identification system developed by Brunelli (Brunelli and Falavigna, 1995; Brunelli et al., 1995) is based on acoustic and visual features. The system is organised as a set of non-homogeneous classifiers whose outputs are integrated after a normalisation step. In particular, two classifiers based on acoustic features and three based on visual ones provide data for an integration module /97/$ Elsevier Science B.V. All rights reserved. PII SO (97)

2 846 J. Kittler et al. / Pattern Recognition Letters 18 (1997) whose performance is shown to be superior to that of the acoustic and visual subsystems. Another system based on geometrical facial features and achieving similar results can be found in (Brunelli et al., 1995). Although the face recognition performance using geometrical features was slightly worse than that obtained using a template matching (Brunelli and Poggio, 1993), face representation is more compact and less computationally demanding, thus providing an example of integration where simple and fast recognition modules are combined. In this paper we investigate the benefits of fusing multiple measurements on a single biometric modality using a single expert. In particular we use a recently developed person identification system based on frontal face image verification (Matas et al., 1997). The system is applied to a number of images acquired during a dialogue with the prospective user and the final decision is made by fusing the soft decision outputs for the individual frames. The fusion problem is formulated for different combination strategies in the framework of Bayesian estimation theory. We analyse the effect of fusing multiple observations on error probability theoretically and confirm the predicted improvements experimentally. We show that the performance gains achieved by means of fusion are initially monotonic but soon reach saturation (after three or four frames in our experiments). It is also shown that fusion rules based on rank order statistics are robust to outliers. In summary, the contribution of the paper is twofold. First of all, we develop a methodology for fusing multiple instances of biometric data to ameliorate the verification system performance. We demonstrate the proposed methodology on the problem of personal identity verification using facial images and confirm the theoretical prediction of performance gains experimentally on the M2VTS database (Pigeon and Vandendrope, 1997). 2 The paper is organised as follows. In Section 2 we formulate the evidence combination problem and introduce the necessary notation. In this section we also present various information fusion strategies. Section 3 investigates the effect of a typical integra- 2 This database is in the public domain. tion scheme, fusion by averaging, on the verification errors. Section 4 describes the face verification system used in the experimental studies of the advocated fusion strategies. In Section 5 we present the results of experiments which show up to 40% improvement in performance. Finally, Section 6 summarises the main results of the paper and offers concluding remarks. 2. Combination strategies Consider a personal identity verification problem where user Z is claiming access to some services or a facility. The claimed identity can be either accepted or rejected. We do not consider the semi-automatic scenario where ask f o r assistance decision may be taken in borderline cases. In other words the user is to be assigned to one of the two possible classes {o~,,w 2} where w, means accept and oj 2 signifies an impostor. The user is represented by a measurement on a biometric modality, such as a face image. We assume that the verification system will engage the user in a dialogue over a short period of time and that during this period it will be possible to acquire several instances of the client description for decision making. Let us assume that we have R instances of the biometric measurement vector x i, i = R. In the measurement space each class w k is modelled by the probability density function p ( x i] w k) and its apriori probability of occurrence is denoted P ( w k ). We shall consider the models to be mutually exclusive. For a single instance xi the decision to accept or reject would be based on the aposteriori class probabilities P ( w j l x i ), j = 1,2, However we wish to take advantage of the additional measurement information and base the decision on all the instances available. In order to investigate the benefit of the additional information, let us examine possible fusion models that could be adopted. Note that the successive measurements x i are acquired within a short interval of time and therefore under the same measurement conditions. We can therefore consider x~s as repeated (multiple) measurements which will differ from each other only in terms of their noise components. In other words we can view each observation

3 J. Kittler et al. / Pattern Recognition Letters 18 (1997) as a corrupted version of some nominal value x. Thus the aposteriori class probabilities based on these observations can be considered as noisy estimates of the nominal value P(coflx), i.e., P(coflxi) = P(cojlx) + e(co, lxi), (1) where 6(co, Ix i) denotes the estimation error associated with the ith instance. For simplicity we shall assume that irrespective of class the estimation errors are identically distributed with a zero mean and variance Oe 2. This formulation of the verification problem immediately suggests that better estimates of the aposteriori class probabilities P(cojlx), j = 1,2 can be obtained by a straightforward application of the estimation theory. Accordingly, the estimates can be combined linearly or using rank order statistics to obtain an improved estimate /~(coilx). This then leads to the following combination strategies: Averaging 1 R /3(co, Ix) = ~- EP(co, lxi). (2) i=l Max rule fi( co, Ix) = max 5,P( coklxi). (3) Min rule fi( coflx ) = min~=, P( cokix i)" (4) Median rule fi( co, Ix ) = med~=, P( coklx i ). (5) In the following section we shall investigate the effect of such improved estimates on the system verification error. 3. Effect of fusion Since the estimation errors are assumed to be unbiased, the combined estimate fi(co, lx) will also be unbiased, i.e., E( fi( cojlx ) } = P( cojlx ) (6) and its variance will be reduced depending on the combination strategy used. For the averaging combi- nation, assuming the errors are uncorrelated, the variance reduces to ~ 2 = o.ez/r. Let us, for the moment, confine our discussion to the averaging combiner and explore how this reduced variance impacts on the error probability. To this end we have to establish what the probability is for the recognition system to make an error which will exceed the Bayes error defined by the nominal posteriors P(coflx). Suppose that P(collX)> P(co2[x). The Bayes error will be exceeded when the aposteriori class probability estimate for class co2 becomes maximum. Let us derive the probability of the event fi(co21x) - / ; ( c o, I x ) > 0 (7) occurring. Note that the left-hand side of Eq. (7) can be expressed as P(co2lx) - P(tOllX) + e(co2tx) - e(collx) > 0, (8) where e(cojlx) is the average error after fusion. For the averaging combiner the error after fusion is given as 1 n ~(cojix) = -'e i ~ 1 ~ ( cojlx i ). (9) Eq. (8) defines a constraint for the two estimation errors e(coflx), j = 1,2 as o ( co21x)- e ( coil X ) > P ( col IX ) -- P(co21x). (10) Note that the errors on the left-hand side of Eq. (10) satisfy s ( c o 2 1 x ) = - e ( c o l l x ). Let us assume that the distribution of the random variable e(coilx) is Gaussian with standard deviation 6". This in practice will approximate the true distribution of estimation errors very coarsely, as both ends of the [0,1 ] interval from which the aposteriori class probabilities can assume values, will clip the errors. Nevertheless, the analysis under even such a simplistic assumption will give an indication of the benefits of fusion of multiple observations. Since the error distribution is Gaussian, the distribution of the difference of the two dependent random variables will also be Gaussian with a twice as large standard deviation. The probability of the constraint in Eq. (10) being satisfied is given by the area

4 848 J. Kittler et al. / Pattern Recognition Letters 18 (1997) under the Gaussian tail with a cut off point at P ( w t l x ) - P(wzlX). More specifically, this probability, which we shall denote Q21(A P12(x)), is given by o:,(ae,:(x)) AP'2(x) ] AP,2(x ) > 0, (11) 2 V~- 6- ] where A P I 2 ( X ) = P ( t o l l x ) - P ( w 2 l x ), and erf(ap12(x)/2v~6") is the error function defined as erf(~p~2(x)2v~-~ ) ~[AP'2(x)a'Yn-(I/2Xv2/4d'2)d T. (12) Thus the client will be misclassified as an impostor with probability Q21(APx2(x)). By analogy, an impostor will be accepted as a client with probability Q12(A P21(x)). Let us denote the additional (over and above the Bayes error) probability by Q(x), i.e., Q(x) ={ Q21(AP12(X)) i f P ( t o, l x ) > P ( w 2 I x ), O,2(AP2,(x)) i f P ( t o l l x ) ~ e ( w 2 l x ). (13) The additional error probability Q(x) will be significant only in the regions where the two aposteriori class probabilities are comparable as a large positive argument of Qij(APji(x)) will drive the additional probability to zero. The average additional misclassification error will then be ~= fq(x)p(x)dx, (14) where p(x) is the mixture density. Recalling Eq. (11), each probability Q i j ( A e j i ( x ) ) in Eq. (13) depends heavily on the variance of the error of the aposteriori class probability estimate. With the multiplicity of observations increasing, the estimate variance will be reduced commensurately. However, the probability of the additional error goes down much more dramatically. In comparison with a single in- stance R = 1, the probability of the pointwise error, for APji(x) > 0 is reduced by a factor 1 - erf( A pji( x ) /2x/2O-e) l _ erf( A ei,( x ) ( ) ) (15) Note that these improvements are achieved only near the decision boundaries, as far from the boundaries the probability of a pattern x being misclassified is negligible. Thus these impressive improvements will be diluted by the averaging process in Eq. (14) where over large regions the local probability of additional error will be effectively zero because of the large difference between the two class aposteriori probabilities. However, whatever the final improvement is, any enhancement in the system performance, especially when one operates close to the 100% mark will be welcome and may make all the difference between an exercise of theoretical interest and a commercially viable solution. Similarly, one can derive the variance reduction gains for rank order statistics combination rules, such as max, min and median rules. The gains can be shown to depend on the number of instances R, the shape of the estimation error distribution and also on the rank order. Although the gains are, on the whole, less dramatic as compared with the averaging combiner, they are more robust to outliers. 4. Optimised robust correlation The face verification system we used to evaluate the different combination strategies is based on optimised robust correlation (Matas et al., 1997). The objective of this method is to find the global extremum in a multi-dimensional search space that corresponds to the best match between a pair of images. This search space is defined by the set of all valid geometric and photometric transformations. In our implementation of the optimised robust correlation the geometric transformations are translation, scaling and rotation. Below we present the score function used to evaluate a point in the multi-dimensional search space, the search method employed to find the global optimum of the score function and, finally, a sampling technique used to speed up the correlation.

5 J. Kittler et al. / Pattern Recognition Letters 18 (1997) I I I I I Fig. 1. Sample images from the M2VTS database illustrating changes in the appearance of a client Score function Given a transformation t, a match score s is computed as a weighted sum of two scores, an area score Sarea and a grey level score Sgrey: s(t) = a. Sarea(t ) q- (1 -- or) " Sgrey(t), where a denotes a constant in the interval [0,1]. The area score is included to encourage large overlaps and is defined as Iar n Stl- Iatl S.... (l)- iarl+larnatl, where S r and S t denote the sampling sets corresponding to the reference and test images, I r and I t, respectively. The grey level score, which measures the similarity between the intensity distributions, is defined as Sgrey(t ) = E f k ( f i ( l r ( P r ) ), I t ( f p ( t ' P r ) ) ) Pr~SrnSt f~nax IS r n Stl where fk denotes a robust kernel, f~ a linear intensity transformation, fp an affine projection function and f~nax the maximum response of the robust kernel. Refer to (Matas et al., 1997) for a more detailed discussion of the score function. of the search, the transformation between reference and test image is perturbed by adding a random vector drawn from an exponential distribution. The new transformation is accepted only if the score was increased. This optimisation method is similar to simulated annealing (Kirkpatrick et al., 1983) at zero temperature Random sampling A sampling technique commonly used in Monte Carlo integration is based on Sobol sequences (Press et al., 1992). These quasi-random sequences allow faster convergence compared to uniformly distributed random numbers since the fractional error of the approximation decreases as ln(n)d/n instead of 1 / v ~ (Press et al., 1992), where N is the number of samples and d the dimensionality of the approximated function. Combining the optimised robust correlation method with random sampling yields several benefits: the execution time can be significantly reduced since only a fraction of the pixels is needed, the quasi-random nature of the process prevents aliasing and it is possible to continuously increase the sampling density and at the same time maintain approximately uniform density throughout the image Optimisation method 5. Experimental results The search technique we employ is based on random exponential perturbations. In each iteration The experiments summarised below were all performed on images from the M2VTS multi-modal,, Fig. 2. An example of a client test: Person JR shot 1 against shot 3.

6 850 J. Kittler et al. / Pattern Recognition Letters 18 (1997) &) O.lO 0.08 " i i ' ~ T ( ' - ' - " ~... T O,OB i.i i,. ~ i i L. "~ ~"i i ~... i i ~ k- ' t..-+. " i t ~ * i : ~,,e~ t!...~ i...i..._.~...i , i! ' i i - - ' ~ = i i! ~-~-.o s o.08 I--...r-...l--.-.%~... N - ~ ~ : ~ ~ 0.02 O'OCo.o , False Acceptance?. ~ I.... i z : FaIN Acceptance ) o _ ~o.o4... i... ~ ~ ~ L [ i i i i.i. lil I,i 0"0~ , False Acceptance Fig. 3. (a) Comparison between different combination strategies (2 models per shot) and verification performance as a function of number of test frames using (b) average and (c) median for combination (1 model per lip state and shot). database (Pigeon and Vandendrope, 1997). This publicly available database contains facial images and recordings of speech from 37 persons. For each person, 5 "shots" 3 acquired over a period of several weeks are available. A single shot is made up of 3 sequences: (1) a frontal-view sequence in which the person is counting from 0 to 9, (2) a rotation sequence in which the person is moving his or her head and (3) a rotation sequence identical to the previous one except that, if glasses are present, they are removed. Some sample images from the M2VTS database are shown in Fig. 1. An example of a client test output from the experiments reported below is shown in Fig. 2. The interleaved image in Fig. 2(c) was obtained by transforming the reference image and then selecting rows interchangeably from the transformed image and the test image. The response image in Fig. 2(d) was computed by applying the robust kernel to each pixel 3 A take is called a shot in the M2VTS terminology. in the overlapping region between the transformed reference image and the test image. To illustrate the benefits of the different combination strategies an experiment was performed using images from one of the two rotation sequences of the first four shots. The test set contained both frontalview and non-frontal-view images whereas the reference set contained frontal-view images only. For each test image, the maximum score obtained over all reference images were taken. This effectively corresponds to a nearest-neighbour classification. The total score for each sequence of test images was then computed by using the four different combination rules described in Section 2. The verification performance was estimated using the l e a v e - o n e - o u t methodology in which training and testing sets are disjoint. The receiver operating characteristics (ROC) are shown in Fig. 3(a). The equal error rates (EERs) for the product, median, average and maximum rules are 5.5, 5.1, 5.5 and 4.6%, respectively. As can be seen in Fig. 3(a), the combination rules based on rank order statistics, the median and maximum rules,

7 J. Kittler et al. / Pattern Recognition Letters 18 (1997) Table l Equal error rate as a function of number of test frames using average (A) and median (M) for combination No. frames EER, A EER, M reduction in error rates of up to 40% was achieved. We showed that the performance gains are initially monotonic but they tend to saturate after integrating the first few observations. It was also shown that the fusion based on a rank order statistics, i.e., the median, is robust to outliers. perform better which is a direct consequence of their robustness to outliers (such as non-frontal views). In order to investigate the relationships between verification performance and number of test frames used an experiment was performed using images from the frontal-view sequences of the first four shots. A chromaticity-based lip tracker described in (Ramos et al., 1997) was used to select three "open-mouth" and three "shut-mouth" images from each shot. 4 The ROC curves obtained by varying the number of test frames from one to six and using average and median for combination are shown in Fig. 3(b) and (c), respectively. The corresponding EERs are listed in Table 1. Note the reduction in error rate for the first three or four frames followed by a saturation in recognition performance. The average outperforms the median which is mainly due to the absence of outliers in the dataset and the statistical superiority of the average combination rule in these situations. 6. Conclusions A methodology for fusing multiple instances of biometric data to improve the performance of a personal identity verification system has been developed. The fusion problem was formulated in the framework of the Bayesian estimation theory. We analysed the effect of different fusion strategies on the error probability theoretically. We then demonstrated the proposed methodology on the problem of personal identity verification using facial images and confirmed the predicted improvements in performance experimentally on the M2VTS database. A 4 The use of a lip tracker to determine the state of the mouth facilitates model selection with a resulting reduction in population entropy (Kittler et al., 1997). Acknowledgements This work was carried out as part of the European Commission ACTS Project M2VTS and was partially supported by ESPRIT Projects SAM (contract ECIS003) and RETINA. References Bigiin, E.S., Bigiin, J., Duc, B., Fischer, S., Expert conciliation for multi modal person authentication systems by bayesian statistics. In: Bigun, J., Chollet, G., Borgefors, G. (Eds.), Audio- and Video-based Biometric Person Authentication. Lecture Notes in Computer Science, vol Springer, Berlin, pp Brunelli, R., Falavigna, D., Person identification using multiple cues. IEEE Trans. Pattern Anal. Machine Intell. 17 (10), Brunelli, R., Falavigna, D., Poggio, T., Stringa, L., Automatic person recognition by using acoustic and geometric features. Machine Vision and Applications 8, Brunelli, R., Poggio, T., Face recognition: Features versus templates. IEEE Trans. Pattern Anal. Machine Intell. 15 (10), Genoud, D., Gravier, G., Bimbot, F., Chollet, G., Combining methods to improve the phone based speaker verification decision. In: Chollet, G., BigUn, J., Borgefors, G. (Eds.) ICSLP'96, vol. 3, pp Jourlin, P., Luettin, J., Genoud, D., Wassner, H., Acousticlabial speaker verification. In: Chollet, G., Bigiin, J., Borgefors, G. (Eds.), Lecture Notes in Computer Science, vol Springer, Berlin, pp Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P., Optimization by simulated annealing. Science 220 (4598), Kittler, J., Li, Y.P., Matas, J., Ramos Sfinchez, M.U., Lip-shape dependent face verification. In: Bigun, J., Chollet, G., Borgefors, G. (Eds.), Audio- and Video-based Biometric Person Authentication. Lecture Notes in Computer Science, vol Springer, Berlin, pp Matas, J., Jonsson, K., Kittler, J., Fast face localisation and verification. In: Clark, A. (Ed.), British Machine Vision Conference, BMVA Press. Pigeon, S., Vandendrope, L., The m2vts multimodal face database (release 1.00). In: Bigun, J., Chollet, G., Borgefors,

8 852 J. Kittler et al. / Pattern Recognition Letters 18 (1997) G. (Eds.), Audio- and Video-based Biometric Person Authentication, Lecture Notes in Computer Science, vol Springer, Berlin, pp Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T., Numerical Recipes in C. Cambridge University Press. Ramos, M.U., S~nchez, Matas, J., Kittler, J., Statistical chromaticity-based lip tracking with b-splines. In: Proc. Internat. Conf. on Acoustics, Speech and Signal Processing, Mu- nich, Germany, April 21-24, vol. 4. IEEE Computer Society P r e s s, p p S e e a l s o : /lip_tracking/index.html. Tumer, K., Ghosh, J., Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recognition 29 (2),