Metrics Performance Evaluation: Alication to Face Recognition Naser Zaeri, Abeer AlSadeq, and Abdallah Cherri Electrical Engineering Det., Kuwait University, P.O. Box 5969, Safat 6, Kuwait {zaery, abeer, cherri}@eng.univ.edu.w Abstract Countless number of alications varying from music, document classification, image and video retrieval, require measuring similarity between the query and the corresonding class. o achieve this, features, that belong to these objects are extracted and modified to roduce an N- dimensional feature vector. A database containing these feature vectors is constructed, allowing query vectors to be alied and the distance between these vectors and those stored in the database to be calculated. As such, the careful choice of suitable roximity measures is a crucial success factor in attern classification. he evaluation resented in this aer aims at showing the best distance measure that can be used in visual retrieval and more secifically in the field of face recognition. here exist a number of commonly used distance or similarity measures, where we have tested and imlemented eight of these metrics. hese eight metrics are famous in the field of attern recognition and are recommended by the Moving Picture Exerts Grou (MPEG. More than tests on different databases were erformed to consolidate our conclusion. he evaluation shows that the and Minowsi distance measures are the best. On the other hand, the distance measure gives the worst results. Index erms Distance measure, Pattern classification, Face recognition. I. INRODUCION Pattern recognition is the art of matching a feature vector of an object to a database of vectors. Interest in the area of attern recognition has been renewed recently due to emerging many interesting and imortant alications. hese alications include data mining (identifying a attern, e.g., correlation, or an outlier in millions of multidimensional atterns, document classification (efficiently searching text documents, financial forecasting, organization and retrieval of multimedia databases, and biometrics (ersonal identification based on various hysical attributes such as face and fingerrints []. Human ability to recognize a face is tremendous. Human brain can identify a familiar object at a glance, desite internal changes or external environmental factors. On the other hand, the Comuter based face recognition is a very challenging dilemma, the recognition system has to be flexible with the external changes lie; environmental light, erson's orientations, distance from camera, and internal deformations (facial exression, aging and maeu. One of the imortant fields of attern recognition is the face recognition. Face recognition finds many imortant alications in many life sectors and in articular in commercial and law enforcement alications. In general, the rocess of recognizing a face asses through many stes such as segmentation of faces from cluttered scenes, extraction of face features, identification, and matching. Facial features extraction to find the most aroriate reresentation of face images is one of the most imortant comonents of a face recognition system for identification uroses. he Princial Comonent Analysis (eigenface technique is one of the well nown aroaches to achieve a good reresentation of a face image [, 3]. In addition, the matching rocedure and technique has a great influence on the final result of the recognition system. Various similarity measures were roosed for the matching stage [3, 6]. In the different attern recognition and classification areas, each one of these similarity (or dissimilarit measures has its advantages, disadvantages, and alications. In this aer, we study the most famous and imortant metrics and distance measures that are used in attern recognition, where we focus on the face recognition field. We study eight of these metrics that are roosed and recommended by the MPEG [4]. We aly these eight metrics on different databases. he databases are constructed and taen from the Olivetti Research Ltd. (ORL database. More details are given in the following sections. his wor is organized as follows. A brief descrition of Princial Comonent Analysis (PCA is given in Section. he study of the different metrics, their formulation, and characteristics is resented and discussed in Section 3. In Section 4, results of testing and imlementing the study on the ORL database are resented. Conclusions are given in Section 5. II. PRINCIPAL COMPONEN ANALYSIS he PCA technique, roosed by ur and Pentland [], extracts the relevant information in a face image, encodes it
as efficiently as ossible, catures the variation in a collection of face images, and comares one face encoding with a database of models encoded similarly. he images of faces, being similar in overall configuration, will not be randomly distributed in the huge image sace and, consequently, they can be described by a relatively low dimensional subsace. his rocess is achieved by finding the rincial comonents of the distribution of faces, or the eigenvectors of the covariance matrix of the set of face images. he eigenvectors are ordered, each one accounting for a different amount of the variation among the face images. hese eigenvectors can be thought of as a set of features that together characterize the variation between face images. Each image contributes more or less to each eigenvector, so that the eigenvector is dislayed as a sort of ghostly face which is called an eigenface. Examle of some of the images used from the ORL database and some of the resultant eigenfaces using the PCA is shown in Figs. and 3, resectively. Fig.. Examle of some of the images used from the ORL. Fig.. Some of the eigenvectors using conventional PCA. In brief, if the training set of face images is Γ, Γ, Γ3,..., ΓM., then, the average face of the set is defined by M Ψ = Γ ( M n= Each face differs from the average by the vector Φ i = Γi Ψ. his set of vectors is subject to rincial comonent analysis. hese comonents are the eigenfaces of the covariance matrix C = A A ( where the matrix A = [ Φ Φ... Φ M ]. In ractice, a M eigenvectors smaller than M is sufficient for identification, since accurate reconstruction of the image is not a requirement. In this framewor, identification becomes a attern recognition tas. he eigenfaces san an M - n dimensional subsace of the original image sace N. he M significant eigenvectors are chosen as those with the largest associated eigenvalues. Now, a face image ( Γ is transformed into its eigenface comonents by the following equation, ω u ( Γ Ψ (3 = th for =,., M., where u is the eigenvector of the covariance matrix. he weights form a vector Ω = [ ω, ω,..., ωm '] that describes the contribution of each eigenface in reresenting the inut face image, treating the eigenfaces as a basis set for face images. o determine which face class rovides the best descrition of an inut face image, we find the face class that minimizes a certain distance ε = Dis Ω, Ω } (4 { th where Ω is a vector describing the of a face class. hese classes are calculated by averaging the results of the eigenvector reresentation over a small number of face descrition vectors (as few as one of each individual. A face is classified as belonging to class if the corresonding ε is the minimum among all other ε s. III. DISANCE MEASURES he main tas of good distance measures is to reorganise descritor sace in a way that media objects with the highest similarity are nearest to the query object. If distance is defined minimal, the query object is always in the origin of distance sace and similar candidates should form clusters around the origin that are as large as ossible. he distance between two oints is either greater than zero or equal to zero. It is greater than zero if the oints are distinct; and it is equal to zero only when the two oints are the same; in other words, when they are not distinct. We will mae use of this roerty of distance to build our recognition system where the degree of similarity and dissimilarity between the tested image and the database is measured. A. Distance he distance between two vectors [5] taes the form ( xi yi D = (5.. where x i and y i ; reresent the coordinates of the two vectors X and Y in the lane. he distance, socalled -norm distance, has been alied in many life and scientific fields due to its simlicity to be alied, and tends
to obtain a convenient and flexible closed form solution in many comlicated situations, while roviding better accuracy under certain conditions. B. Minowsi Geometry Minowsi distance [5, 7] is a very oular method to measure image similarity. It is based on an assumtion: the similar objects should be close to a query object in all dimensions. It consists of: the features of a scaled image that are of the same imortance as the features of a croed image. Suose two images X and Y are reresented by two dimensional vector, X=(x, x x and Y= (y, y y, resectively, the weighted Minowsi metric is defined as D (/ = ( wi xi yi (6.. Where and reresent the Minowsi factor and number of dimensions (attributes resectively, need not be an integer, but it cannot be less than. w i is the weighting to identify imortant features. If w i is normalized then the above method will be more alicable and gives great results simultaneously. he Minowsi distance then is defined as D (/ = ( xi yi (7.. Different values of the arameter give us different distance measurements. If the is set to be, we will get the distance. he various forms of the Minowsi distance do not account for different metrics of the individual coordinates. If the coordinates san different ranges, the coordinate with the largest range will dominate the results. herefore scale the data before calculating the distances is needed. C. Distance he distance metric [] is used for similarity/dissimilarity comarison. It examines the sum of series of a fraction differences between coordinates of a air of objects according to the following equation xi yi D( x, = (8 xi yi + he distance is suitable for variables taing nonnegative values and is sensitive to small changes. It comletely ignores comarison when the two coordinates of concerned data are equal to zero. Due to robustness of technique, it is a highly recommended method in medical researches, esecially in the cases of missing data. D. Chebyshev Distance In Chebyshev distance [5, 9] we tae the maximum difference between the comared coordinates of two objects. Its mathematical reresentation is D { x y, x y x y } = max,..., n n (9 he geometric analysis of Chebyshev distance is given for the functioning of discrete detectors with alications for two-dimensional event recognition roblems. A big disadvantage of the Chebyshev method is that if one element in the vectors has a wider range than the other elements then that large range may dilute the distances of the small-range elements. E. Coefficient Similarity measures [7, 9] the extent of relationshi between the two variables; where simle linear regression rovides the rediction equation that quantifies such relationshi at the best way. is measured by the correlation coefficient ( x ( x x( y r = ( x ( y he correlation coefficient r has a value between - and +, and equals zero if the two variables are not associated. It is ositive if the two continuous X and Y variables have direct relation (i.e. they increase or decrease together. If r is negative, the relationshi is indirect (i.e. if X values increase, Y values decrease. he larger the value of r, the stronger is the association. he maximum value of occurs in case of erfect correlation. F. Divergence Coefficient Clar ( Divergence coefficient Clar method [5] can be reresented as ( xi yi D = ( yi xi + Due to the existence of square root formula, this method suffers from extreme sensitivity to negative values at the denominator. Numbers should be chosen carefully then. G. Distance he figurative meaning of Distance [] can be shown through the following equation ^ D = (( xi xi+ ( yi yi+ ( From this equation, we can see that the distance deends on the two consecutive oints in the feature vector. H. D Similarity his similarity [8] does not deend on the coordinates of the oints but it deends on the angle measured between them. It is nown that two x and y arrays can meet in a
t single oint to form an arbitrary angle α measured in degrees. Fig. 3. Angle measurement between two vectors. Suose two images X and Y are reresented by two dimensional vector, X = x, x,..., x and ( Y = ( y, y,..., y, resectively. he two vectors X and Y are similar when α =, otherwise dissimilar. similarity or in other word inner roduct distance can be figured as: sim x y i i = cosα = = x y (3 n n xi yi i n x y he advantage of this method is that we can normalize the zero mean and the unit variance of the outut by dividing each of its comonents by any related length, (e.g.:. he major advantage of this method over the revious metric distances is that it measures similarity by the angle between the comared object and does not deend on the variance of their coordinates. his characteristic is very imortant esecially in strings objects. IV. EXPERIMENS Although we can aly the eight similarity/ dissimilarity measuring method illustrated before in a huge attern recognition rocess, we are here interesting in alying this whole rocess to face recognition secifically, where our aim is to find the best metric that suits the face recognition. Fig. 4. Examles from the ORL database used in the evaluation of the different metrics. We have alied our exeriments on the ORL database. It consists of 4 different individuals with each individual reresented by different images. hese images were taen randomly under different external environmental factors lie the intensity of light, the bac ground effect and with different ose orientations. Further, some internal factors were also considered: different facial features, facial hair, with or without glasses. We have erformed exeriments. In each one of these exeriments, we tae a art of the image for each one of the 4 images under test. his art is the same for all the images in the same exeriments. In each one of these exeriments, this art differs in location and size from other arts. As exlained in Section II, each image is reresented by a number of weights forming a vector Ω = [ ω, ω,..., ωm '] that describes the contribution of each eigenface in reresenting the inut query image. 9 8 7 6 5 4 EXP: Distance Performance n= n= n=3 n=4 n=5 Minowsi Fig. 5. Recognition rate for different values of cumulative matching score for the various distance measurements for exeriment no.. All the images, or the image arts are reresented with vectors of size. o determine which face art class rovides the best descrition of an inut face art image, we find the face art class that minimizes a certain distance, as exlained in equation (4. Figures (5-7 show the results and the erformance of each of the distance measures that were exlained in the revious section. he figures shown are randomly selected for the 5 different exeriments out of the exeriments that were erformed. Note that the figures show the results as a function of the cumulative match score n. Cumulative match score (CMS is an evaluation methodology roosed by the develoers of FERE In this case, identification is regarded as correct if the true object is in the to Ran n matches. As the charts illustrates measure tends to get the best detection results, however Minowsi with = gives very close results to the case. hat roves that both give best results. As the ower of Minowsi increases, the erformance degrades. he low erformance of Chebyshev may be exlained by the dilution of the small values of similarity and the domination of the largest value. he lowest erformance was for distance. his is somewhat exected, since this distance measure suffers
from an extreme sensitivity of negative values. Figure 8 summarizes the erformance of distances after erforming the exeriments. 9 8 7 6 5 4 EXP6: Distance Performance Minowsi V. CONCLUSION he evaluation resented in this aer aims at testing the recommended distance measures and finding better ones for the basic visual descritions, in articular for face recognition. Eight different distance measures were imlemented, where different databases were used all taen and derived from the main ORL database. he choice of the right distance function for similarity measurement deends on the attern classification roblem under study. he results show that the best erformance was for the distance and Minowsi, resectively. On the other hand, the gave the worst results. n= n= n=3 n=4 n=5 Fig. 6. Recognition rate for different values of cumulative matching score for the various distance measurements for exeriment no. 6. 9 8 7 6 5 4 EXP9: Distance Performance n= n= n=3 n=4 n=5 Minowsi Fig. 7. Recognition rate for different values of cumulative matching score for the various distance measurements for exeriment no. 9. CMS 9 8 7 6 5 4 Distance Performance n= n= n=3 n=4 n=5 Distance Miniowisi CDD Fig. 8. Average Cumulative matching score for the various distance measurements for the exeriments erformed. REFERENCES [] R. Chellaa, C. L. Wilson, and N. Sirohey Human and machine recognition of faces, a survey, Proc. IEEE 83, 75-74 (995. [] ur and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience, March 99. [3] B. Moghaddam and A. Pentland, Probabilistic Matching for Face Recognition, IEEE, 998. [4] Farzin Mohtarian and Miroslaw Bober, Curvature Scale Sace: heory, Alications, and MPEG-7 Standardization, Kluwer Academic Publishers, Netherlands, 3. [5] Marvin Jay Greenberg. " and Non- Geometries, Develoment and History". hird Edition.W.H Freeman and Comany, 999. [6] H. Kong, X. Li, J.-G. Wang, E.K. eoh, C. Kambhamettu, Discriminant Low-dimensional Subsace Analysis for Face Recognition with Small Number of raining Samles, British Machine Vision Conference (BMVC, Oxford, UK, Set. 5-9, 5. [7] Eidenberger, H., and Breiteneder, C. Visual similarity measurement with the Feature Contrast Model. In Proceedings SPIE Storage and Retrieval for Media Databases Conference (Santa Clara CA, January 3, SPIE Vol. 5, 64-76. [8] Gower, J.G. Multivariate analysis and multidimensional geometry. he Statistician, 7 (967,3-5. [9] Santini, S., and Jain, R. Similarity measures. IEEE ransactions on Pattern Analysis and Machine Intelligence, /9, 87-883, 999. [] Cohen, J. A rofile similarity coefficient invariant over variable reflection. Psychological Bulletin, 7 (969, 8-84. [], P. E. he roblem is eistemology, not statistics: Relace significance tests by confidence intervals and quantify accuracy of risy numerical redictions. In Harlow, L.L., Mulai, S.A., and Steiger, J.H. (Eds.. What if there were no significance tests? Erlbaum, Mahwah NJ, 393-45. [] Lance, G.N., and Williams, W.. Mixed data classificatory rograms. Agglomerative Systems Australian Com. Journal, 9 (967, 373-38.