Proximity Measures for Data Described By Histograms Misure di prossimità per dati descritti da istogrammi Antonio Irpino 1, Yves Lechevallier 2

Transcription

1 Proximity Measures for Data Described By Histograms Misure di prossimità per dati descritti da istogrammi Antonio Irpino 1, Yves Lechevallier 2 1 Dipartimento di Studi Europei e Mediterranei, Seconda Università di Napoli antonio.irpino@unina2.it 2 INRIA, Institut National de Recherche en Informatique et en Automatique Riassunto: Il presente lavoro ha come obiettivo quello di confrontare due o più distribuzioni empiriche (di frequenza o di probabilità) descritte in termini di istogrammi ed estratte da campioni di differenti dimensioni. Al fine di considerare la differente dimensione campionaria nel calcolo delle misure di prossimità tra distribuzioni, proponiamo di confrontare gli intervalli di confidenza associati alle stime. In particolare, proponiamo l utilizzo di distanze basate sulla metrica di Hausdorff ed una metrica basata sulla distanza di Wasserstein, confrontandone le capacità interpretative e le proprietà analitiche. Keywords: empiric distribution, distance between two intervals, confidence intervals 1. Introduction The grouping of observations is a convenient and economical method of representing the information contained in a sample. It is also known that a sample can provide estimates of the population parameters that cannot be simply referred to their point estimate. It is more convenient to provide interval estimates. Naturally, interval estimates are generated by probabilistic inferential procedures. In this paper, we propose to compare two or more experimental results using the interval estimates instead of their point estimates. In defining the proximity between two experiments, we introduce the effect of the estimation procedures that is generally expressed by the variability of the estimators. The effect of this is shown in the definition of the width of the interval estimates. In particular, we focus on estimating the empirical frequency functions from different sized samples, but the proposed techniques can be generalized to all cases of parameter estimation from different samples. The paper is organized as follows. In Section (2), we introduce the histogram data as particular modal data, we recall the main estimation procedures and we introduce new data that describes the fact that an histogram has been estimated from a sample. In Section (3) we outline the proposed distances and we show some analytical results related to the adoption of these distances for comparing estimated histograms. Section (4) concludes the paper with comments and perspectives regarding future research. 2. Data described by histograms Data described by histograms are becoming more and more frequent in practise. For example, we have a sample size n = 10, 000 whose values observed in the interval [0, 1)can be measured as nearly one tenth. Here, it is clear there is no need to know all the 10, 000 values, but is sufficient to indicate the counts 1, 2,... associated with 155

2 intervals I i = [(i 1)/10,i/10),i.e., only we know the histogram of the sample. If we have a set of sensors that count the number of impacts observed in their observation area, we do not have the precise location of the impact, but only the knowledge that this sensor took this impact in its area. We can take as an example the analysis of acoustic emission sources described in Hamdan (2005). In this case, our data are uncertain discretized data described in this thesis. Each experiment s, of a sample size equal to, will be represented by its histogram defined by all the counts done by all the sensors. Let a set I = {I 1,...,I h,...,i H } of semi-open disjoint intervals. With every semi-open interval I h of I is associated a random variable, where the value h related to the experiment s is equal to: where h = I xl (I h ) (1) I xl (I h ) = l=1 { 1 if xl I h 0 otherwise Let Y be a continuous variable, with each sample s of size extracted by this random variable associated with a histogram of the sample ((I 1,1 ),..., (I h,h ),..., (I H,H )) where the H couples are built on a set of H semi-open disjoined intervals I = {I 1,...,I h,...,i H }, said elementary, satisfying the following properties: 1. H I h [, + [; 2. I h I h = if h h ; the weights (1,...,H ) satisfy the following properties: (2) h 0 and h = (3) In this case, to effectively compare two histograms it is essential to take into account the size of each of the samples associated with these histograms. The empiric frequency q sh = h / is calculated from a sample of size, where h is the number of realizations observed in the I h interval. A CI(q sh ) confidence interval with the confidence level α (generally 0.05, 0.01 or 0.001) of the probability of the interval I h is calculated on the sample s by: CI(q sh ) q sh z α/2 q sh (1 q sh ) ;q sh + z α/2 q sh (1 q sh ). (4) So it is possible to describe a histogram as a vector of pairs ((I 1,CI(q s1 )),..., (I h,ci(q sh )),..., (I H,CI(q sh ))). Each pair consists of an elementary I h interval and a CI(q sh ) confidence interval. 156

3 In order to show some analytical properties of the proposed distances, we chose to use a midpoint-radius notation to describe this interval, where the midpoint of the interval is q sh and the radius is the half of its length l sh = z α/2 q sh (1 q sh ). (5) The confidence interval can be as described in the following way: CI(q sh ) = {x [0, 1]/x = q sh + (2t h 1)l sh where t h [0, 1]}. (6) Thus our histogram is not only described by a list of frequencies, but also by a list of confidence intervals that measure the quality of the estimation by the frequencies. In this paper, we refer only to the interval of weights as a generalization of the so-called compositional data. Indeed, in the following, for the computation of the distance we consider elementary intervals as categories of a variable. Figure 1: Histogram with the confidence interval associated with each frequency 3. Metrics for interval data extended to confidence intervals In this section we present the main distances used for comparing interval data. We use these distances to compare the confidence intervals Hausdorff distance The most common distance used for the comparison of two sets is the Hausdorff (1) distance. Considering two sets A and B of points of R n, and a distance d(x,y) where x A and y B, the Hausdorff distance is defined as follows: ( ) d H (A,B) = max sup inf d (x,y),sup inf d (x,y) (7) y B x A x A If d(x,y) is the L 1 City block distance, then Chavent et al. (2002) proved that d H (A,B) = max ( a u, b v ) = a+b + b a. (8) y B u+v 2 2 v u 2 2 (1) The name derives from Felix Hausdorff who is well-known for the separability theorem on topological spaces at the end of the 19 th century. 157

4 Given two samples s and r described by a histogram partitioned into H bins, the Hausdorff distance between two sets of H α confidence intervals is: d H (r,s) = H d Haus (CI(q rh ),CI(q sh )) = = H max ( (q sh l sh ) (q rh l rh ), (q sh + l sh ) (q rh + l rh ) ) = = H ( q sh q rh + l sh l rh ) 3.2. Wasserstein metric If F and G are the distribution functions of two random variables f and g respectively, with first moments µ f and µ g, and σ f and σ g their standard deviations, the Wasserstein L2 metric is defined as (Gibbs and Su, 2002) 1 1/2 ( d W (F,G) := F 1 (t) G 1 (t) ) 2 dt (10) 0 where F 1 and G 1 are the quantile functions of the two distributions. Irpino and Romano (2007) proved that the distance can be decomposed as: d 2 W = (µ f µ g ) 2 + (σ }{{} f σ g ) 2 + 2σ }{{} f σ g (1 ρ QQ (F,G)) }{{} Location Size Shape where ρ QQ (F,G) is the correlation of the quantiles of the two distributions as represented in a classical QQ plot. It is worth noting that 0 < ρ QQ 1 differs from the classical range of variation of the Bravais-Pearson s ρ. This decomposition allows us to take into consideration three aspects in the comparison of distribution function. The first aspect is related to the location: two distributions can differ in position and this aspect is explained by the distance between the mean values of the two distributions. The second aspect is related to the different variability of the compared distribution. This aspect is related to the different standard deviations of the distributions and to the different shapes of the density functions. While the former sub-aspect is explained by the distance between the standard deviations, the latter sub-aspect is explained by the value of ρ QQ. Indeed, ρ QQ is equal to one only if the two (standardized) distributions are of the same shape. If the two samples s and r are described by two histograms with the same support it is possible to define an extension of the Euclidean distance, taking into account the various confidence intervals associated with frequencies using the Wasserstein metric. In order to use the Wasserstein distance, it is important to recall what a confidence interval is. In the literature, there are two main approaches to understand confidence intervals (cfr. Piccolo, 2000). Given the sample s of size, and a point estimates of the proportion q s, and q L s = q s z s(1 q s) α/2 P θ z α/2 q and R s = q s z s(1 q s) α/2 q s (1 q s ) q s θ + z α/2 q s (1 q s ) Piccolo (2000) wrote: (9) (11) P (L s θ R s ) = 1 α (12) The first term refers to the classic frequentist approach, while the second term considers the bounds as random variables. 158

5 The classic (frequentist) probability approach to confidence intervals The classic (frequentist) probability approach says that confidence intervals are regions of the parameters space induced by sample space, and contain the true parameters of the population with 1 α confidence. In this sense, if all the values of the confidence interval have an equal probability of being the right parameter of the population, we may compute the Wasserstein distance between two confidence intervals induced by two samples r and s of size n r and in the following manner: where d 2 W (s,r) = H (q sh q rh ) (l sh l rh ) 2 (13) (q sh q rh ) 2 is the Euclidean norm between the frequencies and 1 3 (l sh l rh ) 2 is a measure integrating the different length related to the different variability of the estimators for the two samples. It is easy to demonstrate that lim (l sh l rh ) 2 = n r,, lim n r,, z α/2 q sh (1 q sh ) z α/2 q rh (1 q rh ) n r (14) So, if the frequencies are estimated from large samples, this distance is identical to the Euclidean distance between the distributions. If the two histograms are equal, then the distance proposed is equal to: d 2 W (s,r) = 1 3 z α/2 2 qh (1 q h ) So the distance depends only on the sample size of s and r. ( 1 ns 1 2 = 0. nr ) 2 (15) The pivot method for confidence interval estimates One of the most useful methods for constructing CIs is that of pivotal quantities. This method constructs the CIs for an auxiliary quantity called a pivot, and then transforms the interval into a CI for the parameter θ h. Recalling that a pivot is a function of θ h and of the sample whose distribution does not depends on θ h, we may consider the building of a CI as: P(g Lh g(θ h,s) g Uh ) = P(g 1 (g Lh,s) θ h g 1 (g Uh,s)) = 1 α. In the case of proportion estimation, it is known that the following function is used in estimating the CI (Φ(t h ) is the inverse function of the standard normal distribution): g 1 q sh (1 q sh ) (t h ) = q sh + Φ(t h ). Then, the CI is the region identified by: [L h = g 1 (α/2),r h = g 1 (1 α/2)]. 159

6 Then, given two samples s and r, we can use the Wasserstein distance to compare the two samples by means of the pivotal functions for the definition of (1 α) CIs as follows: d 2 W(g r,g s ) := 1 α/2 α/2 ( g 1 r (t h ) g 1 s (t h ) ) 2 dth (16) but, furthermore, we can compare the function without considering α as follows: d 2 W (g r,g s ) := 1 0 ( ) 2 (gr 1 (t h ) gs 1 (t h )) 2 dt = (q rh q sh ) 2 q + rh (1 q rh ) q n r sh (1 q sh ). (17) Considering two samples s and r described by two histograms partitioned into H bins, the Minkowski version of the Wasserstein distance is equal to: d 2 W(r,s) = (q rh q sh ) 2 + q rh (1 q rh ) n r q sh (1 q sh ) 2 (18) This is a special case deriving from equation (11), that is applicable to all interval estimates derived using several kinds of procedures: classic frequentist inference, bootstrap, jackknife or bayesian Comparing estimated histogram data The proposed distances consider data as described by vectors of proportions independently from the nature of their support. We propose to consider new distances between two histograms using their empirical distribution functions instead of only their frequency functions. In this case, the upper and lower limits contains all the possible distribution functions. Also, the upper and lower limits are distribution functions themselves. Each histogram can be associated with a distribution function. Considering the two histograms estimated from the two samples s and r, the aim is to compare the 1 α areas associated with each distribution function (see Fig. 2). In order to preserve Figure 2: Empiric histograms and estimated distribution functions (with associated upper and lower bounds) 160

7 the order and the quantitative nature of the support, we propose four extensions of the proposed distances to compare the two estimated histogram data. For the sake of brevity, we outline the distances and their asymptotic properties without analytical developments. L1-Hausdorff distance R 2 [0, 1.5 D ]. Given the two samples r and s described by the two histograms estimating the true (but unknown) frequency functions f and g, with their associated distribution functions F and G, where D is the length of the domain of the support and q c are the estimates of cumulate proportions, we have: d L1 H(r,s) = d H [CI(qr(x)),CI(q c s(x))]dx c (19) D Limits property: lim d L 1 H(r,s) = d L1 (F,G). (20) n r, Kolmogorov-Hausdorff distance R 2 [0, 1] d K H (r,s) = sup (d H [CI(q c r(x)),ci(q c s(x))]) (21) Limits property: lim d K H(r,s) = d K (F,G). (22) n r, L2-Wasserstein distance R 2 [0, 1.25 D ] d 2 L 2 W(r,s) = d 2 W [CI(qr(x)),CI(q c s(x))]dx c (23) D Limits property: lim d L 2 W(r,s) = d L2 (F,G). (24) n r, Wasserstein-Wasserstein distance quantile using the r sample. [0, 1] 2 R +. Let x r(t) be the estimate of the t-th d 2 W W(r,s) = 1 d 2 W [ CI(xr(t) )),CI(x s(t) ) ] dt (25) Limits property: 0 lim d W W(r,s) = d W (F,G). (26) n r, 161

8 4. Conclusions The treatment of histogram data knows a new life considering the overwhelming growth of data stored in large databases. Ieveral fields of research and application, the representation of summaries of data is becoming the starting input for further analysis. On the other side, the knowledge of the techniques of summarizing data have to be taken into consideration when we use these kind of aggregated data to be further analyzed. We have presented new metrics for these kind of data and we showed its properties. The definition of such metrics allows to define new techniques, or to extend classic techniques based on the comparison of complex data. The paper offers some new developments for the comparison of data summarized by histograms. The main novelty is the possibility of taking into account the summarization process. The proposed distances have been also used for comparing modeled data, where a comparison is performed among the (interval) estimates of the parameters of the model. In this paper we propose to build the distances in an additive way without considering the covariance of estimators in the computation. The next step, very hard from the computational point of view, is to integrate such information in the distance formula. References Chavent M., Lechevallier Y. (2002) Dynamical clustering of interval data, optimization of an adequacy criterion based on Hausdorff distance, Classification, Clustering and Data Analysis, Jaguga K. et al. (Eds.), Springer, Gibbs A.L., Su F.E. (2002) On choosing and bounding probability metrics, International Statistical Review, 7, Hamdan H. (2005) Développement de Méthodes de Classification et de Discrimination pour le Contrôle par Émission Acoustique d Appareils à Pression, Thèse à Université de Technologie de Compiègne. Irpino A., Lechevallier Y., Verde R. (2006) Dynamic clustering of histograms using Wasserstein metric, in: COMPSTAT 2006, Rizzi A. & Vichi M. (Eds.), Physica- Verlag, Berlin, Irpino A., Romano E. (2007) Optimal histogram representation of large data sets: Fisher vs piecewise linear approximations, RNTI E-9, Irpino A., Verde R. (2006) A new Wasserstein based distance for the hierarchical clustering of histogram symbolic data, in: Data Science and Classification, IFCS 2006, Batanjeli V., Bock H.H., Ferligoj A. & Ziberna A. (Eds.), Springer, Berlin, Piccolo D. (2000) Statistica, Il Mulino, Bologna, Italy. Verde R., Irpino A. (2007) Dynamic clustering of histogram data: using the right metric, in: Selected Contributions in Data Analysis and Classification, Brito P., Bertrand P., Cucumel G. & de Carvalho F. (Eds.), Springer, Berlin,