VUS and HUM Represented with Mann-Whitney Statistic

Transcription

1 Communications for Statistical Applications and Methods 05, Vol., No. 3, 3 3 DOI: Print ISSN / Online ISSN VUS and HUM Represented with Mann-Whitney Statistic Chong Sun Hong,a, Min Ho Cho a a Department of Statistics, Sungkyunkwan University, Korea Abstract The area under the ROC curve AUC, the volume under the ROC surface VUS and the hypervolume under the ROC manifold HUM are defined and interpreted with probability that measures the discriminant power of classification models. AUC, VUS and HUM are expressed with the summation and integration notations for discrete and continuous random variables, respectively. AUC for discrete two random samples is represented as the nonparametric Mann-Whitney statistic. In this work, we define conditional Mann-Whitney statistics to compare more than two discrete random samples as well as propose that VUS and HUM are represented as functions of the conditional Mann-Whitney statistics. Three and four discrete random samples with some tie values are generated. Values of VUS and HUM are obtained using the proposed statistic. The values of VUS and HUM are identical with those obtained by definition; therefore, both VUS and HUM could be represented with conditional Mann-Whitney statistics proposed in this paper. Keywords: AUC, classification, HUM, manifold, nonparametric, ROC, surface, VUS. Introduction The receiver operating characteristic ROC curve is a technique to visualize and organize the classification model or classifiers based on performance. ROC curve has been widely used in signal detection theory to depict the tradeoff between Sensitivity and -Specificity of classifiers. ROC analysis has been extended to visualize and analyze the behavior of diagnostic systems Egan, 975; Swets, 988; Swets et al., 000. The medical decision-making community has extensive literature on the use of ROC graphs for diagnostic testing. Research regarding the property of ROC curve and information about application of ROC analysis can be found in many papers, such as Provost and Engelmann et al. 003, Fawcett 003, Hong 009, Hong and Choi 009, Hong et al. 00, Provost and Fawcett 00, Sobehart and Keenan 00 and Zou et al The area under the ROC curve AUC is used as an objective statistic to measure the power of discriminant through ROC curve Bradley, 997; Hanley and McNeil, 98. Most circumstances in real world consist of multiple categories rather than only two that require methods to measure the power of discriminant of multiple category classification model are required. For example, the triple categories Nondefault, Warning, Default are more realistic rather than two categories Nondefault, Default in order to classify the credit assessment models. Whereas both ROC curve and AUC are for two dimensions, ROC surface and the volume under the ROC surface VUS are extended to three dimensions Dreiseitl et al., 000; Fawcett, 003; Heckerling, 00; Hong et al., 03; Mossmann, 999; Nakas and Yiannoutsos, 004; Nakas et al., 00; Patel and Markey, 005; Scurfield, 996; Wandishin and Mullen, 009, Zou et al., 007. By extending the ROC surface Corresponding author: Department of Statistics, Sungkyunkwan University, Seoul 0-745, Korea. cshong@skku.edu Published 3 May 05 / journal homepage: c 05 The Korean Statistical Society, and Korean International Statistical Society. All rights reserved.

2 4 Chong Sun Hong, Min Ho Cho and VUS for three dimensions, Li and Fine 008 and Hong and Jung 04 defined and suggested both the ROC manifold and the hypervolume under the ROC manifold HUM statistic to discriminate more than three categories. The properties of discrete form of both VUS and HUM are discussed and explained. We will also express VUS and HUM as functions of Mann-Whitney statistics in this paper because AUC can be represented with the well-known nonparametric Mann-Whitney statistic. First, the conditional Mann- Whitney statistic is suggested and defined. Then VUS and HUM for discrete random variables might be represented with conditional Mann-Whitney statistics. Hence values of the VUS and HUM could be obtained using the conditional Mann-Whitney statistics proposed in this paper. In Section of this paper, we explain the definitions of AUC, VUS and HUM. Especially, these are explored for discrete variables and the HUM is concerned only for four categories of the classification model. Section 3 suggests and defines the conditional Mann-Whitney statistics for multiple random samples. Then VUS and HUM are proposed to represent with conditional Mann-Whitney statistics. Some illustrative examples consisting of three and four random variables with some tie values are generated. Then values of VUS and HUM are obtained and compared with values calculated by definitions in Section 4. Section 5 provides the conclusion and future works.. ROC Manifold and HUM Suppose k random variables X, X,..., X k and their cumulative distribution functions, F, F,..., F k satisfying F x F x F k x for all x. Nakas and Yiannoutsos 004 and many others defined AUC, VUS and HUM for ROC curve, surface and manifold, respectively, such as AUC PX X, VUS PX X X 3, HUM k PX X X k for k 4. The AUC, VUS and HUM could be expressed by the following integral and probability notations for continuous and discrete random variables, respectively Hong and Jung, 04. When random variables are continuous, AUC, VUS and HUM 4 are represented as AUC VUS HUM 4 ROCu du 0 FF3 u FF u 3 0 F F u du, ROC s u, u 3 du du 3 F F u [ F 4 F 3 u 3 ] du du 3. 0 F F u F 3 F u du, When random variables are discrete, values of AUC, VUS and HUM 4 are expressed such as AUC PX < X PX X,. VUS PX < X < X 3 PX X < X 3 PX < X X 3 PX X X 3,.

3 VUS and HUM Represented with Mann-Whitney Statistic 5 HUM 4 PX < X < X 3 < X 4 PX X < X 3 < X 4 PX < X X 3 < X 4 PX < X < X 3 X 4 PX X < X 3 X 4 PX X X 3 < X 4 PX < X X 3 X 4 3 PX X X 3 X 4..3 AUC in. can be represented with Mann-Whitney statistic such as AUC [U X<X ] n n U XX, where U X <X n n j IX i < X j, U X X n n j IX i X j Rosset, 004. In this work, we would like to express VUS and HUM in. and.3, respectively, as functions of Mann- Whitney statistics. 3. VUS and HUM Represented with Conditional Mann-Whitney Statistics 3.. VUS with conditional Mann-Whitney statistics Suppose three random samples {X i }, {X j }, {X 3k } of sizes n, n, n 3, respectively. And with their cumulative distribution functions, F, F, F 3, assume that F x F x 0 and F x F 3 x 0 for all x. By using the property of the conditional probability, the VUS under the assumption F x F x F 3 x can be defined as follows. VUS PX < X < X 3 PX X < X 3 PX < X X 3 PX X X 3 PX < X 3 X < X PX < X PX < X 3 X X PX X PX X 3 X < X PX < X PX X 3 X X PX X. 3. Consider a paired sample {X i, X j } satisfying X i < X j for each pair. The {X j } in {X i, X j X i < X j } is denoted as {X j ; X i < X j }. In order to compare {X 3k } with {X j ; X i < X j }, we will define the conditional Mann-Whitney statistic. Definition. Definition of the conditional Mann-Whitney statistic n n n 3 U X <X 3 X <X I X j < X 3k X i < X j. The conditional probability PX < X 3 X < X in 3. could be defined with the conditional Mann-Whitney statistic which is calculated from two random samples {X 3k } and {X j ; X i < X j }. Now we propose alternative statistic to obtain the VUS for ROC surface. Theorem. The VUS could be obtained by using the conditional Mann-Whitney statistic, such as VUS MW [U X<X3 X<X n n n U XX3 X<X U X<X3 XX ] U XX3 XX, 3

4 6 Chong Sun Hong, Min Ho Cho where U X <X 3 X <X U X <X 3 X X n n n 3 I n n n 3 X j < X 3k X i < X j, UX X 3 X <X I X j X 3k X i < X j, n n n 3 I n n n 3 X j < X 3k X i X j, UX X 3 X X I X j X 3k X i X j. Proof: Since n n n 3 I n n X i < X j < X I X j < X 3k X i < X n n j 3k n n n 3 n n n 3 j I j I X i < X j X i < X j n n n n n 3 I X j < X 3k X i < X j, n n n 3 the VUS is obtained that n n n 3 n n n 3 n n n 3 n n n 3 n n n 3 n n n 3 n n n 3 I X i < X j < X 3k I X i < X j X 3k n n n 3 I X i X j < X 3k n n n 3 I X j < X 3k X i < X j I X j < X 3k X i X j I X i X j X 3k n n n n n 3 I X j X 3k X i < X j n 3 I X j X 3k X i X j [U X<X3 X<X U XX3 X<X U X<X3 XX U XX3 XX ]. 3.. HUM 4 with conditional Mann-Whitney statistics Now we suppose four random samples {X i }, {X j }, {X 3k }, {X 4l } of sizes n, n, n 3, n 4, respectively. For cumulative distribution functions, F, F, F 3, F 4, assume that F x F x F 3 x F 4 x for all x. By using the property of the conditional probability, the HUM 4 under the assumption F x F x F 3 x F 4 x can be represented as follows. HUM 4 PX < X < X 3 < X 4 PX X < X 3 < X 4 PX < X X 3 < X 4 PX < X < X 3 X 4 PX X < X 3 X 4 PX X X 3 < X 4 PX < X X 3 X 4 3 PX X X 3 X 4

5 VUS and HUM Represented with Mann-Whitney Statistic 7 PX 3 < X 4 X < X < X 3 PX < X < X 3 PX 3 < X 4 X X < X 3 PX X < X 3 PX 3 < X 4 X < X X 3 PX < X X 3 PX 3 X 4 X < X < X 3 PX < X < X 3 PX 3 X 4 X X < X 3 PX X < X 3 PX 3 < X 4 X X X 3 PX X X 3 PX 3 X 4 X < X X 3 PX < X X 3 3 PX 3 X 4 X X X 3 PX X X The probability PX < X < X 3, PX X < X 3, PX < X X 3 and PX X X 3 in 3. can be generalized by using the conditional probabilities in 3.. Therefore, the conditional probability PX 3 < X 4 X < X < X 3 could be represented with the conditional Mann-Whitney statistic which is calculated from two random samples {X 4l } and {X 3k ; X i < X j < X 3k }. Then the n n4 l IX 3k < conditional Mann-Whitney statistic, U X3 <X 4 X <X <X 3 could be defined as n X 4l X i < X j < X 3k. Other conditional probabilities are also represented with conditional Mann- Whitney statistics. Therefore, HUM for ROC manifold for four random variables could be represented with the following conditional Mann-Whitney statistics. Theorem. The HUM 4 could be obtained by using the conditional Mann-Whitney statistic such as [ HUM 4 MW U X3 <X n n n 3 n 4 X <X <X 3 4 U X 3 X 4 X <X <X 3 U X 3 <X 4 X X <X 3 U X 3 X 4 X X <X 3 U X 3 <X 4 X <X X 3 U X 3 X 4 X <X X 3 U X 3 <X 4 X X X 3 ] U 3 X 3 X 4 X X X 3, 3.3 where U X3 <X 4 X <X <X 3 U X3 X 4 X <X <X 3 U X3 <X 4 X X <X 3 U X3 X 4 X X <X 3 U X3 <X 4 X <X X 3 U X3 X 4 X <X X 3 U X3 <X 4 X X X 3 n n n 3 n 4 I X 3k < X 4l X i < X j < X 3k, l n n n 3 n 4 I X 3k X 4l X i < X j < X 3k, l n n n 3 n 4 I X 3k < X 4l X i < X j X 3k, l n n n 3 n 4 I X 3k X 4l X i X j < X 3k, l n n n 3 n 4 I X 3k X 4l X i X j < X 3k, l n n n 3 n 4 I X 3k X 4l X i < X j X 3k, l n n n 3 n 4 I X 3k < X 4l X i X j X 3k, l

6 8 Chong Sun Hong, Min Ho Cho U X3 X 4 X X X 3 n n n 3 n 4 I X 3k X 4l X i X j X 3k. l Proof: The HUM 4 can be defined by using indicator functions. n n n 3 n 4 n n n 3 n 4 I n n n 3 n 4 X i < X j < X 3k < X 4l I X i X j < X 3k < X 4l l n n n 3 n n n 3 n n n 3 I X i < X j X 3k < X 4l I X,i X, j < X 3,k X 4,l I X i < X j X 3k X 4l 3 l n n n 3 n n n 3 n n n 3 I X i < X j < X 3k X 4l I X i X j X 3k < X 4l I X i X j X 3k X 4l. Since n n n 3 n 4 I X i < X j < X 3k < X 4l n n n 3 n 4 l n n n4 l I X 3k < X 4l X i < X j < X n n 3k n n n 4 I I X i < X j < X 3k X i < X j < X 3k n n n 3 n n n 3 n 4 I X 3k < X 4l X i < X j < X 3k, n n n 3 n 4 l the HUM 4 could be obtained that n n n 3 n 4 n n n 3 n 4 I n n n 3 n 4 X 3k < X 4l X i < X j < X 3k I X 3k X 4l X i < X j < X 3k l n n n 3 n n n 3 n n n 3 4. Some Illustrative Examples 4.. Example of VUS I X 3k < X 4l X i X j < X 3k I X 3k < X 4l X i < X j X 3k I X 3k < X 4l X i X j X 3k 3 l n n n 3 n n n 3 n n n 3 I X 3k X 4l X i X j < X 3k I X 3k X 4l X i < X j X 3k I X 3k X 4l X i X j X 3k. Table shows three samples {X i }, {X j }, {X 3k } of sizes n 5, n 6, n 3 5, respectively. The ROC surface could consist of three positive rates such as F x, F y F x, F 3 y, for all x, y x < y

7 VUS and HUM Represented with Mann-Whitney Statistic 9 Table : Three random samples X n 5 X n 6 X n 3 5 Table : Subsamples and the conditional Mann-Whitney statistics {X, X X < X } X 3 > X {X, X X X } X 3 > X {X, X X < X } X 3 X {X, X X X } X 3 X, 7 7, 7, 7 7, 7,, 7, 7,, , 39 39, , , 39 7, 39 3, 39 3, 39, 48, 48 7, 48 7, 48 3, 48 3, 48 39, 48 39, 48 44, 48 44, 48, , , 57 7, 57 3, 57 3, 57 39, 57 39, 57 44, 57 44, , 7, 7 7, 7 7, 7 3, 7 3, 7 39, 7 39, 7 44, 7 44, U X <X 3 X <X 7 U X <X 3 X X 9 U X X 3 X <X 8 U X X 3 X X on unit dice Hong et al., 03. There are two subsamples {X, X X < X } and {X, X X X } collected in Table. Then {X 3 } is compared with {X } in these subsamples {X ; X < X } and {X ; X X }, so that conditional Mann-Whitney statistics are obtained in Table. Then with conditional Mann-Whitney statistics in Table, the VUS is obtained that VUS MW [U X<X3 X<X n n n U X<X3 XX U XX3 X<X ] U XX3 XX Example of HUM 4 There are four illustrative random samples {X i }, {X j }, {X 3k }, {X 4l } of sizes n 4, n 4, n 3 6, n 4 5, respectively, in Table 3. Four subsamples {X, X, X 3 X < X < X 3 }, {X, X, X 3 X X < X 3 }, {X, X, X 3 X < X X 3 } and {X, X, X 3 X X X 3 } are collected in Table 4. Then {X 4 } is compared of {X 3 } in these four subsamples, so that conditional Mann-Whitney statistics are obtained in Table 4.

8 30 Chong Sun Hong, Min Ho Cho Table 3: Four random samples X n 4 X n 4 X n 3 6 X n 4 5 Table 4: Subsamples and the conditional Mann-Whitney statistics {X, X, X 3 X < X < X 3 } X 4 {X, X, X 3 X X < X 3 } X 4 {X, X, X 3 X < X X 3 } X 4 {X, X, X 3 X X X 3 } X 4,, 9 7,, 9,, , 45, 45 7,, 45 7, 45, , 45, 45 45,, 54 45, 45, 54 3, 45, 45 7,, 54, 45, 54 7, 45, 54 3, 45, ,, 7 45, 45, 7 7,, 7, 45, 7 7, 45, 7 3, 45, 7, 6, 7 7, 6, 7 3, 6, 7 45, 6, 7,, 83 45, 45, 83 7,, 83, 45, 83 7, 45, 83 3, 45, 83, 6, 83 7, 6, 83 3, 6, 83 45, 6, 83, 77, 83 7, 77, 83 3, 77, 83 45, 77, ,, 90 45, 45, 90 7,, 90, 45, 90 7, 45, 90 3, 45, 90, 6, 90 7, 6, 90 3, 6, 90 45, 6, 90, 77, 90 7, 77, 90 3, 77, 90 45, 77, U X3 <X 4 X <X <X 3 30 U X3 <X 4 X X <X 3 U X3 <X 4 X <X X 3 U X3 <X 4 X X X 3 4 U X3 X 4 X <X <X 3 U X3 X 4 X X <X 3 0 U X3 X 4 X <X X 3 3 U X3 X 4 X X X 3

9 VUS and HUM Represented with Mann-Whitney Statistic 3 Then, the HUM 4 is obtained that [ HUM 4 MW U X3 <X n n n 3 n 4 X <X <X 3 4 U X 3 X 4 X <X <X 3 U X 3 <X 4 X X <X 3 U X 3 X 4 X X <X 3 U X 3 <X 4 X <X X 3 U X 3 X 4 X <X X 3 U X 3 <X 4 X X X 3 ] U 3 X 3 X 4 X X X 3 5. Conclusion [ ] The Mann-Whitney test statistic is used to compare two location parameters of discrete random samples. The conditional Mann-Whitney statistics are proposed in order to compare more than equal to three random samples. Whereas the Mann-Whitney statistic, U X <X, may be defined as n n j IX i < X j from two random samples {X i } and {X j } of sizes n and n, respectively, the conditional Mann-Whitney statistic, U X <X 3 X <X, could be defined as n n IX j < X 3k X i < X j from two random samples {X 3k } and {X j ; X i < X j }, where the subsample {X j ; X i < X j } is collected satisfying the state {X i < X j } from two random samples {X i } and {X j }. It is known that the Mann-Whitney statistic, U X <X, can be used to define the conditional probability PX < X. Moreover it is found that the conditional Mann-Whitney statistic, U X <X 3 X <X, is used to derive the probability PX < X 3 X < X. With the similar argument that AUC has a linear relation with the Mann-Whitney statistics, U X <X and U X X, it might be concluded that VUS could be represented with conditional Mann-Whitney statistics, U X <X 3 X <X, U X X 3 X <X, U X <X 3 X X and U X X 3 X X. In addition, the HUM with more than three random variables is proposed to define with the conditional Mann-Whitney statistics in this work. The Mann-Whitney statistic for two random samples is also known to have a linear relation with Wilcoxon rank sum statistic; therefore, it might be derived that the conditional Mann-Whitney statistic for more than two random samples has a relationship with some modified Wilcoxon rank sum statistic in a further study. References Bradley, A. P The use of the area under the ROC curve in the evaluation of machine learning algorithms, Pattern Recognition, 30, Dreiseitl, S., Ohno-Machado, L. and Binder, M Comparing three-class diagnostic tests by three-way ROC analysis, Medical Decision Making, 0, Egan, J. P Signal Detection Theory and ROC Analysis, Academic Press, New York. Engelmann, B., Hayden, E. and Tasche, D Testing rating accuracy, Risk, Fawcett, T ROC graphs: notes and practical considerations for data mining researchers, Available from: Hanley, J. A. and McNeil, B. J. 98. The meaning and use of the area under a receiver operating characteristic ROC curve, Radiology, 43, Heckerling, P. S. 00. Parametric three-way receiver operating characteristic surface analysis using mathematica, Medical Decision Making,, Hong, C. S Optimal threshold from ROC and CAP curves, Communications in Statistics -

10 3 Chong Sun Hong, Min Ho Cho Simulation and Computation, 38, Hong, C. S. and Choi, J. S Optimal threshold from ROC and CAP curves, The Korean Journal of Applied Statistics,, 9 9. Hong, C. S., Joo, J. S. and Choi, J. S. 00. Optimal thresholds from mixture distributions, The Korean Journal of Applied Statistics, 3, 3 8. Hong, C. S. and Jung, D. G. 04. Standard criterion of hypervolume under the ROC manifold, Journal of the Korean Data & Information Science Society, 5, Hong, C. S., Jung, E. S. and Jung, D. G. 03. Standard criterion of VUS for ROC surface, The Korean Journal of Applied Statistics, 6, Li, J. and Fine, J. P ROC analysis with multiple classes and multiple tests: Methodology and its application in microarray studies, Biostatistics, 9, Mossman, D Three-way ROCs, Medical Decision Making, 9, Nakas, C. T., Alonzo, T. A. and Yiannoutsos, C. T. 00. Accuracy and cut-off point selection in three class classification problems using a generalization of the Youden index, Statistics in Medicine, 9, Nakas, C. T. and Yiannoutsos, C. T Ordered multiple-class ROC analysis with continuous measurements, Statistics in Medicine, 3, Patel, A. C. and Markey, M. K Comparison of three-class classification performance metrics: A case study in breast cancer CAD, Proceedings of SPIE, 5749, Provost, F. and Fawcett, T. 00. Robust classification for imprecise environments, Machine Learning, 4, Rosset, S Model Selection via the AUC, In Proceedings of the st International Conference on Machine Learning ICML 04, Banff, Canada, 89. Scureld, B. K Multiple-event forced-choice tasks in the theory of signal detectability, Journal of Mathematical Psychology, 40, Sobehart, J. and Keenan, S. 00. Measuring default accurately, Risk - Credit Risk Special Report, 4, Swets, J. A Measuring the accuracy of diagnostic systems, Science, 40, Swets, J. A., Dawes, R. M. and Monahan, J Better decisions through science, Scientific American, 83, Wandishin, M. S. and Mullen, S. J Multiclass ROC analysis, Weather and Forecasting, 4, Zou, K. H., O Malley, A. J. and Mauri, L Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models, Circulation, 5, Received December 6, 04; Revised February 4, 05; Accepted February 5, 05