Rank Robustness of Composite Indicators. James E. Foster Department of Economics Vanderbilt University Nashville, USA

Transcription

1 Rank Robustness of Composite Indicatos 1 Rank Robustness of Composite Indicatos James E. Foste Depatment of Economics Vandebilt Univesity Nashville, USA Mak McGillivay Austalian Agency fo Intenational Development (AusAID) Canbea, Austalia Suman Seth Depatment of Economics Vandebilt Univesity Nashville, USA Abstact Many common multidimensional indices take the fom of a composite indicato o a weighted aveage of seveal dimension-specific achievements. Rankings aising fom such an index ae dependent upon an initial weighting vecto, and any given judgment could, in pinciple, be evesed if an altenative weighting vecto wee employed. This pape examines a vaiable-weight obustness citeion fo composite indicatos that views a compaison as obust if the anking is not evesed at any weight vecto within a given set. We chaacteize the esulting obustness elations fo vaious sets of weighting vectos and illustate how they modeate the complete odeing geneated by the composite indicato. We popose a measue by which the obustness of a given compaison may be gauged and illustate its usefulness using data fom the Human Development Index. In paticula, we show how some county ankings ae fully obust to changes in weights while othes ae quite fagile. We investigate the pevalence of the diffeent levels of obustness in theoy and pactice and offe insight as to why cetain datasets tend to have moe obust compaisons. JEL Classification: I31, O12, O15, C02 Key Wods: Composite indicato, multidimensional index, Human Development Index, weighting vecto, obustness, positive association, ank coelation, Kendall s tau

2 Rank Robustness of Composite Indicatos 2 Rank Robustness of Multidimensional Composite Indices Contents I. Intoduction II. Notation III. Robust Compaisons A. Fully Robust Compaisons B. Robust Compaisons C. Gaphical Depiction of the Robustness Relations IV. Measuing Robustness V. The Pevalence of Robust Compaisons A. Some Examples of Pevalence Functions B. Fixed Robustness and Tansfomations C. Inceased Robustness and Positive Association VI. Concluding Remaks Refeences Figues Tables

3 Rank Robustness of Composite Indicatos 3 I. Intoduction It has fo many decades been ecognised that well-being, development, human development o the quality of life, as it is vaiously known, is multi-dimensional. This ecognition has spawned numeous composite indices, the constuction of which involves combining in some way two o moe indicatos, intended to captue achievement in vaious dimensions, into a single metic. Those focusing on intecounty ankings ae the subject of intense inteest. Remakable attention is given to county ankings by the media, intenational agencies, civil society, the eseach community, business goups and in paticula national govenments. A ise in paticula county s anking, o it being anked highe than counties to which it identifies, is often geeted not supisingly with geat pide by its govenment. The esponse to a fall in its anking, o it being outanked by these othe counties, is somewhat diffeent. The inteest in national govenments and othes these composite indices is, howeve, blind to long held concens within the eseach community egading the obustness of the ankings yielded by these measues. These concens aise fom vaious methodological issues in the constuction of these indices. One of the most cental methodological issues is weighting.. It is quite common to fom these indices by taking the simple aveage of dimension-specific achievements. 1 The simple aveage is undestood to be assigning equal weights and, in tun, impotance to each dimension.thee is geneal ageement that the equal weights should not in pinciple be assigned to each dimension. In a pefect wold the weights would be povided by obust empiical estimates of the paametes of a meta poduction function. Yet thee is no accepted infomation on these functions, including on thei functional foms, one is necessaily left to speculate as to coect weights to assign to each component of the multidimensional measue in question. In this cicumstance, equal weights seem to be 1 The vey well known and widely used Human Development Index and its foeunne, the Physical Quality of Life Index, ae the most well-known examples of such indices. The othe easonably wellknown examples of well-being o elated indices poposed ove the last 20 o so yeas include the Child Well-being Index, Envionmental Pefomance Index, Envionmental Sustainability Index, Gende Empowement Measue, Gende-elated Development Index, Genuine Pogess Measue, Global Peace Index, Index of Economic Feedom, the Global Peace Index, The simple aveage is also applied to geneate the Human Povety Index the Alkie and Foste class of povety indices that use a union appoach (UNDP, 2007). Composite indices used in othe aeas of concen ae commonly based on simple aveages. Well known examples in the intenational development and vulneability fields include the Commitment to Development Index and the Economic Vulneability Index.

4 Rank Robustness of Composite Indicatos 4 the only possible natual focal point. 2 Such a weighting scheme is at least easy to undestand and tanspaent and the simple mean it yields has vaious desiable popeties that makes the task of statistical infeence extemely simplified. Those esponsible fo the design of these indices ae typicaly of the view that thee is insufficient guidance and have assigned weight abitaily, although othes have poposed diffeential weights based on eithe nomative o statistical appoaches. The nomative appoaches of weighting do not povide any sufficient guidance as to whose noms would be implemented: the nom of the social planne o the nom of the individuals in the society. The statistical appoaches, on the othe hand, ae completely data diven that makes compaison acoss counties o egions and acoss time extemely difficult. Diffeent weights often do not comply with each othe and geneate diffeent conclusions while anking a pai of counties o egions. Moe pecisely, one county may dominate the othe fo one set of weights, but at the same time a diffeent set of weights may esult in the fome to be stictly dominated by the late. With the same agument, the anking that is geneated by the equal weights do not always yield the same anking fo anothe set of weights. This type of anking is said to be ambiguous. The ambiguity natually leads one to question the obustness of multidimensional composite indices to the weighting schemes that is embodied. The focus of this pape is not to popose anothe new method fo geneating weights. Instead, we accept the pioity of equal weights, but popose obustness check of judgements. It commences with the povision of an analytical famewok that allows one to assess the ank obustness of a multidimensional index that contains two o moe component vaiables. Based on a dominance appoach, this famewok is outlined in the context of any multidimensional index subject to ambiguity egading the coect numeical values of its weights. We apply patial odeing 3 as a tool to explain obustness of anking. The pape then applies this famewok to the well known and vey widely used measue used to assess well-being at the level of nations, the Human 2 The poponents of the Envionmental Sustainability Index agued fo equal weights on the pemise that no objective mechanism exists to detemine the elative impotance of the diffeent aspects of envionmental sustainability. Similaly Maye and Jenckks (1989) wee of the view that we have no eliable basis fo doing [othewise]. 3 This is analogous to Foste & Shoocks (1988).

5 Rank Robustness of Composite Indicatos 5 Development Index (HDI). 4 Fist appeaing in the UNDP Human Development Repot 1990, the HDI values have since been published annually fo moe than 170 counties. Thee is intense intenational inteest in HDI county ankings. 5 The pape shows that the HDI ankings ae quite obust with espective to altenative weighting schemes. It also discusses the easons fo this obustness. The pape concludes with some eflections on the futue design of multidimensional indices and consideation of possible extensions of its ank obustness assessment method, not only to othe multidimensional measues but to issues including measuement eo. It fom the outset be emphasised that the fundamental pupose of the pape is not to discedit the epot o discouage the use of multidimensional index ankings. Rathe, it is to facilitate moe incisive intepetation of the ankings that is poduced. The sections of the pape ae oganized as follows. Section II povides a bief desciption of the mathematical notations that ae used thoughout the pape. In section III, we explain the possible ambiguity in anking that can aise using an illustative example using the Human Development Index. In addition, we define and chaacteize the patial odeing that allows us to calculate a measue of obustness in the next section. Section IV is devoted towads the deivation of a measue of ank obustness. Section V povides a detailed analysis of ambiguous ankings and explains how the numbe of ambiguous anking depends on the association among dimensions at the aggegated level. Conclusions and diection fo futue eseach ae povded Section VI. II. Notation This section povides the mathematical notation and definitions used in the pape. The numbe of dimensions to be summaized in a composite index is denoted by an intege D 2. An achievement vecto x = (x 1,,x D ) R D indicates the scoe achieved in each of the D dimensions, while X R D gives the set of all of achievement 4 Fo the puposes of this pape tems such as well-being, human development, quality of life and the standad of living ae teated as synonymous. This is lagely consistent with the elevant liteatue, although as Alkie (2000) notes some studies imply diffeences in the usage and meaning of such tems. 5 Fo example, accoding to an aticle in the New Yok Times - Fo the sixth yea in a ow, Noway was anked fist on the United Nations' human development index as the county poviding its citizens with the best chance of living a long and pospeous life. - THE WEEK; The Othe Winnes and Loses Nov. 5-11, NYT, Novembe 12, 2006

6 Rank Robustness of Composite Indicatos 6 vectos that ae possible. A dataset is a finite set ˆX of ˆn elements dawn fom X. Fo any a,b R D, let a = Σ D d =1 a d denote the sum of a s components and let a b = Σ D d =1 a d b d epesent the inne poduct of a and b. The expession a b indicates that a d b d fo d = 1,, D; this is the vecto dominance elation. If a b with a b, then this situation is denoted by a > b; while a >> b indicates that a d > b d fo d = 1,, D. The least uppe bound of a and b, denoted by a b, is the vecto having max {a i, b i } as its i th coodinate; the geatest lowe bound of a and b, denoted by a b, is the vecto having min {a i, b i } as its i th coodinate. *** Figue 1 Hee *** The unit simplex is defined as S = {s R D : s 0 and s = 1} and is depicted in Figue 1 above fo D = 3. It contains all possible ways of weighting the vaious achievements. The vetices of the simplex ae given by v d = e d fo d = 1,, D, whee e d is the usual basis element that places full weight on the single achievement d. The centoid o cental point of the simplex v 0 = (1/D,,1/D) places equal weight on all achievements. Clealy, v 0 is the simple aveage of the vetices of S, o v 0 = Σ D d =1 v d / D. Fo example, when D = 3, the vetices of the simplex S ae v 1 = (1,0,0), v 2 = (0,1,0), and v 3 = (0,0,1), while the centoid is v 0 = (⅓,⅓,⅓). In what follows, we constuct smalle vesions of S by popotionally contacting the vetices of S towad a given point w 0 of S. Fo any (0,1] and d = 1,2,, D, define v d = (1 )w 0 + v d, and let S be the egula simplex geneated by the vetices v 1,v 2,...,v D ; equivalently, let S be the convex hull of {v 1,v 2,...,v D }. Fo example, if = 0.25 and w 0 = v 0 then v 1 = (0.5, 0.25, 0.25), v 2 = (0.25, 0.5, 0.25), and v 3 = (0.25, 0.25, 0.5) as illustated in Panel I of Figue 2. The esulting simplex S is outlined by an equilateal tiangle whose vetices ae one-fouth of the way to the vetices of S fom the centoid v 0. If w 0 = (0.6, 0.2, 0.2), then v 1 = (0.7, 0.15, 0.15), v 2 = (0.45, 0.4, 0.15), and v 3 = (0.45, 0.15, 0.4), and S is the egula simplex depicted in

7 Rank Robustness of Composite Indicatos 7 Panel II. Note that as dops to 0, the simplex S shinks to the point w 0, while if = 1, we have S = S. *** Figue 2 Hee *** Lemma 1 Let v 0 is the centoid of the unit simplex S. Fo any w 0 S such that w 0 v 0, S (w 0 ) and S (v 0 ) ae equal in volume fo all (0,1]. Poof. The d th vetex of the simplex S (w 0 ) and S (v 0 ) ae given by the vectos (1 )w 0 + v d and v d (w 0 ) = v d (v 0 ) = (1 )v 0 + v d, espectively. The diffeence between the d th vetex of these two simplexes ae given by the vecto δ d = v d (w 0 ) v d (v 0 ) = (1 )(w 0 v 0 ). Theefoe, δ d = (1 )(w 0 v 0 ) fo all d and S (w 0 ) is obtained fom S (v 0 ) by shifting all vetices of the late by δ d. Hence, S (w 0 ) and S (v 0 ) ae equal in volume fo all (0,1]. 6 III. Robust Compaisons While thee ae many conceivable ways of aggegating achievements, we focus hee on a common fom of multidimensional index based on a weighted aveage of individual levels. A composite indicato C: X S R applies the enties of a weighting vecto w S to the espective enties in the achievement vecto x X and sums to obtain the geneal fom C(x;w) = w x. To implement this appoach in pactice one must select a specific weighting vecto. In what follows, it is assumed that an initial weighting vecto w 0 S has aleady been chosen; this fixes the specific composite indicato C 0 : X R defined by C 0 (x) = C(x;w 0 ) fo all x X. The associated stict odeing of achievement vectos will be denoted by C 0, so that x C 0 y if and only if C 0 (x) > C 0 (y). 6 1 Note that if = 1, then δ = 0 and S 1 (w 0 ) = S 1 (v 0 ). d

8 Rank Robustness of Composite Indicatos 8 One well-known example of a composite indicato is given by the Human Development Index (HDI) of the United Nations Development Pogamme, which povides infomation on D = 3 achievements, namely, health, education and income. 7 The weighting used in the HDI is v 0, o equal weighting, and the esulting composite index has the fom C 0 (x) = v 0 x = (⅓)x 1 + (⅓)x 2 + (⅓)x 3. Table 1 povides data fo the ten counties with the highest HDI levels in 2004 as given in the 2006 Human Development Repot. The counties ae listed in ode of HDI fom highest (Noway), to second highest (Iceland), and so foth, until the tenth highest county (Nethelands). The associated infomation on county compaisons is also epesented in matix fom in Table 2. Wheneve a column county has a highe HDI value than a ow county, this is indicated with C 0 in the associated cell. Note that evey cell below the diagonal is filled, eflecting the fact that the HDI, like any composite index, geneates a complete odeing once a specific weighting vecto has been chosen. *** Table 1 Hee *** *** Table 2 Hee *** A geat deal of attention is paid to the coss-county HDI compaisons when the annual Human Development Repot is eleased. Howeve, it should be bone in mind that these compaisons ae entiely contingent on the chosen weighting vecto v 0 and eveal little about the obustness of judgments as the weights ae vaied. Indeed, look at the twin examples of Austalia vesus Sweden and Ieland vesus Canada. The HDI of Austalia exceeds that of Sweden by about 0.006, and Ieland is highe than Canada by the same magin. It is an easy matte to show that Austalia has a highe level of the composite indicato than Sweden fo all weighting vectos w S. On the othe hand, the pai-wise anking of Ieland and Canada can be easily evesed: fo example, the 7 The health index is based on life expectancy; the education index is based on enolment and liteacy ates; and the income index based on pe capita Goss Domestic Poduct. Fo detailed deivation of the dimension specific indices, see the technical note on p. 394 of the 2006 Human Development Repot (UNDP, 2006).

9 Rank Robustness of Composite Indicatos 9 weighting vecto w' obtained fom v 0 by a 0.05 shift of weight fom each of the education and income dimensions to the health dimension is one such example. In symbols, let C be the composite indicato ove the D = 3 dimensions of health, education and income, and let x, y, x', and y' denote the espective achievement vectos fo Austalia, Sweden, Ieland and Canada. Then fo Austalia and Sweden we have C(x;v 0 ) > C(y;v 0 ) and this anking is neve evesed at any w S; while fo Ieland and Canada, C(x';v 0 ) > C(y';v 0 ) and this anking is evesed fo the altenative weighting vecto w' given above. In sum, C 0 compaisons that appea to be identical in Tables 1 and 2 can be diffeentially obust, and this in tun may have some beaing on ou intepetation of such compaisons. To diffeentiate among C 0 compaisons (such as those found in Table 2) we fomulate a binay elation ove achievement vectos in X that will indicate when an initial compaison is obust. Ou constuction begins with a set W S of easonable weighting vectos containing the initial vecto w 0. We say that x obustly dominates y given W, witten x C W y, if and only if C(x,w 0 ) > C(y,w 0 ) and C(x,w) C(y,w) fo all w W. In othe wods, the composite indicato is highe fo x than y when the weighting vecto is w 0, and this anking is neve evesed fo any othe weighting vecto in W. The elation C W is clealy tansitive, so that if x C W y and y C W z, then x C W z. Howeve, it will often be the case that x C W y does not hold despite x C 0 y being tue, since C W equies thee to be no evesal fo any vecto in W. As W expands, the likelihood of a evesal ises, and fewe obust compaisons can be made; in othe wods, if W W' then x C W' y implies x C W y, but not vice vesa. This pape exploes the obustness elation C W fo vaious specifications of the set W, povides chaacteizations of the elations, and offes insight on thei applicability. A. Fully Robust Compaisons If W = S, the associated elation C W applies when an initial judgment x C 0 y is neve evesed at any configuation of weights. In this case we say that the compaison is fully obust and shall denote the elation C W by C 1. Requiing unanimity ove all of S is quite demanding and consequently C 1 is the least applicable among all obustness

10 Rank Robustness of Composite Indicatos 10 elations; howeve, when it applies, the associated anking of achievement vectos is maximally obust. The examples of Austalia vs. Sweden and Ieland vs. Canada fom Table 1 suggest a simple chaacteization of C 1. Notice that Austalia is highe than Sweden in each of the thee dimensions, and hence C 1 holds in this case. In contast, Ieland is highe than Canada in two dimensions and lowe in one, which is why C 1 does not apply (i.e., the anking is evesed when the weight is high enough on the evesed dimension). We have the following esult. Theoem 1 Let x C 0 y fo x, y X. Then x C 1 y if and only if x y. Poof Suppose that x C 0 y is tue. If x y holds, then clealy C(x;w) = w x w y = C(y;w) fo all w S, and thus x C 1 y. Convesely, if x C 1 y holds, then setting w = v d in C(x;w) C(y;w) yields x d y d fo all d, and hence x y. In ode to check whethe a given anking x C 0 y is fully obust, one need only veify that the achievement levels in x ae at least as high as the espective levels in y. 8 The following coollay povides two altenative sets of sufficient conditions fo C 1. Coollay Let x, y X. Then x C 1 y holds if (i) x >> y o if (ii) w 0 >> 0 and x > y. Poof Both sets of conditions entail x y, hence by Theoem 1 we need only veify that both imply x C o y. If x >> y, then fo any w S we have w x > w y, hence x C o y. If w 0 >> 0 and x > y, it follows immediately that w 0 x > w 0 y, and hence x C o y. 8 Note that x C 0 y pecludes the possibility that x = y, and hence x C 1 y actually entails x > y.

11 Rank Robustness of Composite Indicatos 11 One inteesting implication of the Theoem 1 is that the elation C 1 is meaningful when vaiables ae odinal and no basis of compaison between them has been fixed. 9 Suppose that each vaiable x d in x is independently alteed by its own monotonically inceasing tansfomation f d (x d ) and let y = (f 1 (x 1 ),, f D (x D )) be the esulting tansfomed achievement vecto. 10 We can show that x C 1 x' implies y C 1 y' fo the espective tansfomed vectos y and y'. Indeed, if x C 1 x', we know that x C 0 x' holds by definition, while x x' is tue by Theoem 1. Tansfoming the vaiables yields y y' and thus (1) w y w y' fo all w S since w 0. By x C 0 x' it follows that w 0 x > w 0 x' is tue; thus, fo some d we have w 0 0 d x d > w d x d, and hence x d > x' d with w 0 d > 0. Though the tansfomation we obtain y d > y' d and this, when combined with w 0 d > 0 and (1), yields w 0 y > w 0 y', o y C 0 y'. By (1), then, we have y C 1 y'. In othe wods, if C 1 holds fo any given cadinalization of the odinal vaiables, it holds fo all cadinalizations. Note that while C 0 on its own is not meaningful in this context (since tansfoming vaiables can lead to y' C 0 y even though initially we had x C 0 x'), the fully obust elation C 1 has the popety that x C 1 x' if and only if y C 1 y', and hence is an appopiate technology fo odinal vaiables. Now etuning to the case of the HDI, we might ask how many of the 45 C 0 compaisons given in Table 2 ae fully obust. The answe, povided in Table 3, is that just fou compaisons exhibit vecto dominance of achievement vectos and hence C 1. This is pehaps not unexpected due to the naow diffeences in HDI values among the highest anked counties. The pictue fo the entie dataset eveals geate applicability fo the elation C 1 : fo the 177 counties, thee ae a total of 10,875 fully obust compaisons out of a possible 15,576 compaisons, implying that just unde 69.8% of all compaisons ae fully obust. Why ae thee so many fully obust HDI compaisons? We etun to this point below in Section V. 9 Robets () meaningful. 10 The esulting function f:x R D defined by f(x) = (f 1 (x 1 ),, f D (x D )) is called a monotonically inceasing tansfomation below.

12 Rank Robustness of Composite Indicatos 12 *** Table 3 Hee *** B. Robust Compaisons Wheeas the pevious section took W to be the entie simplex S, we now conside using the smalle simplex S of weighting vectos. This will lead to a less demanding obustness elation than C 1, but one that is moe geneally applicable. Recall that the simplex S is the convex hull of vetices { v 1,...,v D } located a fixed popotion of the way fom w 0 to the vetices of S. When w 0 = v 0, the esulting simplex S is a smalle vesion of S with v 0 at its cente; fo geneal w 0, the set S is a scaled down vesion of S that peseves the elative position of w 0. In eithe case, the the size of S is always the same fo fixed (0,1]. Substituting W = S in the definition of C W yields the th ode obustness elation, denoted hee by C and defined as follows: x C y if and only if x C 0 y and C(x;w) C(y;w) fo all w S. This elation etains all the popeties of the geneal obustness elation, and since the sets S ae nested fo a fixed w 0, we know that that x C y implies x C ' y wheneve > '. Now suppose that x C 0 y holds fo the pai x and y of achievement vectos. What additional conditions on x and y ae needed to ensue that x C y? One easy-toveify set of necessay conditions is fo C(x;w) C(y;w) at each vetex w = v d of S. Indeed, define x = ( x 1,..., x D ) whee x d = C(x;v d ) = v d x and let y be the analogous vecto deived fom y. Then the necessay condition can be stated as x y. The next theoem shows that, in fact, this is also sufficient. Theoem 2 Let x C 0 y fo x, y X. Then x C y if and only if x y. Poof We need only veify that x C 0 y and x y imply x C y. Pick any w S, and note that since S is the convex hull of its vetices, w can be expessed as a convex combination of v 1,...,v D, say w = α 1 v + +α D v whee α 1 + +α D = 1 and α d 0 fo 1 d = 1,,D. But then C(x;w) = w x = α 1 v x+ +α D v x = α 1 x + +α D x, and 1 D D 1 D

13 Rank Robustness of Composite Indicatos 13 similaly C(y;w) = α 1 y 1 + +α D y D ; theefoe x y implies C(x;w) C(y;w). Since w was an abitay element of S, it follows that x C y. Theoem 2 shows that to evaluate whethe a given compaison x C 0 y exhibits th ode obustness, one need only compae the associated vectos x and y. If each component of x is at least as lage as the espective component of y, then the compaison is obust accoding to C ; if x has at least one component lowe than the espective enty of y, then, the oiginal anking is not obust. Note that when = 1, we have x = x, and Theoem 2 educes to Theoem *** Table 4 Hee *** *** Table 5 Hee *** Table 4 illustates this appoach fo = 0.25 fo the ten highest HDI counties, with the final thee columns listing the enties of the associated vecto x. The table eveals a host of compaisons that can be made using C fo this value of. In paticula, the anking between Iceland and USA, which was detemined not to be fully obust, is th ode obust as is appaent by the dominance of the last thee enties fo Iceland ove the espective enties fo USA. On the othe hand, the compaison between Ieland and Canada, which Table 1 showed was not fully obust, is also not obust in the pesent case, since the fouth column enty fo Canada is highe than the thid column enty fo Ieland. Table 5 lists all compaisons among the top ten counties that can be made using the obustness elation C fo = 0.25; fully 46.7% of the possible compaisons can now be made. Fo the entie list of 177 counties, 91.8% of the compaisons exhibit th ode obustness fo this level of. 11 Note that since d v = (1-)w 0 + v d, it follows that x d = v d x = (1-)C 0 (x) + x d ; in othe wods, the vecto x is a convex combination of the vecto (C 0 (x),, C 0 (x)) of aveage achievements and x itself.

14 Rank Robustness of Composite Indicatos 14 C. Gaphical Depiction of the Robustness Relations Figue 3 uses data fom Table 1 to povide gaphical epesentations of the conditions associated with the vaious obustness elations. In each panel, the twodimensional simplex S is depicted in the hoizontal plane at the base of the gaph, as ae its thee vetices v d and the initial weighting vecto v 0 at its cente. The smalle simplex S and its vetices v d ae also epesented within S (fo the case = 0.25). Now suppose that a given county with achievement vecto x has been selected. Fo any weighting vecto w in the simplex, the level of the composite indicato C(x;w) is gaphed as the height above the vecto w. Thus, the heights at v 1, v 2, v 3, and v 0 ae, espectively, x 1, x 2, x 3, and the HDI. The lineaity of C in w ensues that these points and the emaining C(x;w) values fom a tilted achievement simplex with vetices as high as the dimensional achievements and a cente as high as the county s HDI level. *** Figue 3 Hee *** Compaisons fo thee pais of counties - Austalia and Sweden, Iceland and USA, and Ieland and Canada - ae depicted. In Panel 1 the achievement simplex of Austalia is completely above the achievement simplex fo Sweden, eflecting the vecto dominance of the espective achievement vectos in Table 1. Austalia has the highe HDI and it is clea fom the gaph that thee is no weighting vecto fo which it has a lowe composite indicato level than Sweden. This is an example whee the elation C 1, o complete obustness, holds. The second panel depicts a athe diffeent scenaio fo Iceland and USA: the achievement simplexes intesect and C 1 cannot hold. Iceland pefoms bette than the USA in tems of both health and education, but USA s achievement is highe fo income. Moe weight on health and education makes Iceland s level of the composite indicato highe than that of the USA, wheeas moe weight on income makes USA s level highe. Dominance does hold if we estict consideation to the smalle simplex S. When the intesection is pojected down to S, the esulting (dashed) line does not coss S and hence all weights in S follow the oiginal HDI anking in selecting Iceland

15 Rank Robustness of Composite Indicatos 15 above USA. Indeed, Table 4 confims that Iceland has highe levels of the composite indicato than USA at each of the vetices of S. Consequently, while C 1 does not hold, C cetainly does fo = The final panel depicts the case of Ieland and Canada, which has the same HDI diffeence as Austalia and Sweden and intesecting achievement simplexes like Iceland and USA, but has diffeent obustness chaacteistics than each. While Ieland s education and income vaiables ae highe than Canada s, the health index has the opposite oientation, and C 1 cannot hold. If we poject the intesection of the espective achievement simplexes on S, we obtain a dashed line that cuts S, implying that C does not hold. The Ieland-Canada compaison is not obust even fo = This is also evident fom Table 4 since Ieland has highe levels of the composite indicato at two of the vetices of S (namely, v 2 and v 3 ) and a lowe level at the emaining one ( v 1 ). IV. Measuing Robustness Up to now, ou method of evaluating the obustness of a compaison x C 0 y has been to fix a set S of easonable weighting vectos and then to confim that the initial anking is not evesed at any membe of S, in which case the associated th ode obustness elation C applies. Theoem 2 povides simple conditions fo checking whethe x C y holds. The pesent section augments this appoach by fomulating a obustness measue that associates with any anking x C 0 y a numbe * between 0 and 1 to indicate its level of obustness. We constuct * using two statistics - one that might be expected to move in line with obustness and anothe that is likely to wok against it. The fist of these is Δ 0 = C(x;w 0 ) C(y;w 0 ) > 0, o the diffeence between the composite value of x and the composite value of y at the initial weighting vecto w 0. Intuitively, Δ 0 is an indicato of the stength of the dominance of x ove y at the initial weighting vecto. The second is Δ m = max w S [C(y;w) C(x;w), 0], o the maximal contay diffeence between the composite values of y and x. Note that when the oiginal compaison is fully obust,

16 Rank Robustness of Composite Indicatos 16 then C(y;w) C(x;w) 0 fo all w S and thee is no contay diffeence. Consequently Δ m = 0. On the othe hand, when the compaison is not fully obust, then C(y;w) C(x;w) > 0 fo some w S, and hence Δ m = max w S [C(y;w) C(x;w)] > 0. The quantity Δ m is the wost-case estimate of how fa the oiginal diffeence could be evesed at some othe weighting vecto. The measue of obustness we popose is given by * = Δ 0 /(Δ 0 +Δ m ). Notice that when the initial compaison x C 0 y is fully obust, then Δ m = 0 and hence * = 1. Altenatively, suppose that the initial compaison is not fully obust so that Δ m > 0. Then it is clea that * is stictly inceasing in the magnitude of the initial compaison Δ 0, and stictly deceasing in the magnitude of the contay wost-case evaluation Δ m. In addition, if Δ 0 tends to 0 while Δ m emains fixed, the measue of obustness * will also tend to 0. These chaacteistics accod well with an intuitive undestanding of how Δ 0 and Δ m affects obustness. Pactical applications of * may be hampeed by the fact that Δ m equies a maximization poblem to be solved, namely, max w S [C(y;w) C(x;w)]. Howeve, by the lineaity of C(y;w) C(x;w) = (y x) w in w, the poblem has a solution at some vetex v d of S. At the vetex w = v d the diffeence C(y;w) C(x;w) becomes y d x d, and so Δ m can be calculated as Δ m = max d (y d x d ), o the maximum coodinate diffeence between y and x. The measue * can be eadily deived using this equivalent definition fo Δ m. Fo example, ecall the case of Ieland and Canada, whose espective achievement vectos x' and y' ae found in Table 1. The initial HDI diffeence Δ 0 = C(x';v 0 ) - C(y';v 0 ) is = The maximal coodinate diffeence y' d x' d is in dimension d = 1 so that Δ m = = Aggegation gives us an * of In contast, the compaison of Austalia with Sweden yields the same HDI diffeence Δ 0 = 0.006, but Δ m = 0 since all dimensional diffeences suppot the initial anking of Austalia ove Sweden; hence * = 1. Finally, the example of Iceland and USA poduces Δ 0 = and Δ m = and hence a obustness level of about * = 0.28.

17 Rank Robustness of Composite Indicatos 17 Now what is the elationship between the obustness measue * and the elations C developed in the pevious section? The following theoem povides the answe. Theoem 3 Suppose that x C 0 y fo x, y X and let * be the obustness level associated with this compaison. Then the th ode obustness elation x C y holds fo 0 < * and does not fo * < 1. Poof Let x C 0 y and suppose that 0 < *. By definition of *, we have Δ 0 /(Δ 0 +Δ m ) and hence Δ m (1-)Δ 0. Pick any d = 1,,D. Then using the definitions of Δ 0 and Δ m we see that (y d x d ) (1-)(w 0 x - w 0 y) and hence v d y + (1-)w 0 y v d x + (1-)w 0 x. Consequently, v d y v d x, and since this is tue fo all d, it follows that x y and hence x C y by Theoem 2. Altenatively, suppose that x C 0 y and yet * < 1. Then (1-)Δ 0 < Δ m so that (1-)(w 0 x - w 0 y) < (y d x d ) fo some d, and hence v d y > v d x o y d > x d fo this same d. It follows, then, that x y cannot hold, and neithe can x C y by Theoem 2. Theoem 3 shows that the measue of obustness * is closely elated to the obustness elations C. Indeed, * is the lagest fo which x C y holds o, equivalently, fo which S has no weighting vecto that eveses the initial anking. Panel 1 of Figue 4 depicts the dashed line of intesection whee Iceland and USA have the same value of the composite indicato. Also depicted ae the simplexes coesponding to thee values of. The smallest simplex ( = ) contains only weighting vectos that yield a stictly highe value fo Iceland. The lagest simplex ( = ) is cut by the dashed line and hence it contains a egion whee USA has highe values. The middle simplex ( = *) contains vectos fo which Iceland has the highe value, and a single vecto (the vetex that just touches the dashed line) fo which the values ae the same. A smalle would leave oom fo the simplex to expand without evesing the initial compaison; a lage would lead to a evesal. Consequently, =

18 Rank Robustness of Composite Indicatos 18 * is the obustness level of the compaison. Panel 2 illustates the analogous constuction fo a case wee initial weights ae not equal. Note that the * value found hee is lage, showing that * may well depend upon the initial weighting vecto w 0. *** Figue 4 Hee *** *** Table 6 Hee *** To summaize, we have defined a measue of obustness *, shown how it can be calculated in pactice, and povided an altenative intepetation in tems of the C elations and thei associated simplexes. Table 6 illustates this methodology fo the ten-county HDI example via a obustness pofile that lists the obustness value * (in pecentage tems) fo each of the 45 possible compaisons. This includes the infomation given in Table 3 (which highlights the fou enties with * = 100%) and Table 5 (which depicts the 21 compaisons with * 25%), and can easily identify the C compaisons fo any given. Note that the aveage value of * in Table 3 is only 35%, which eflects the fact that the HDI levels ae quite simila fo the high HDI counties and so the Δ 0 values ae smalle. The next section will apply the methods to datasets associated with the HDI and othe well-known composite indicatos to evaluate the applicability of the C elations in pactice. V. The Pevalence of Robust Compaisons The focus now shifts fom individual compaisons to the entie collection of compaisons associated with a given dataset ˆX and an initial weighting vecto w 0. The fist question is how to judge the oveall obustness of the dataset. One option would be to use an aggegate measue (such as the mean) that is stictly inceasing in each compaison s obustness level. Howeve, athe than settling on a specific measue we use a pevalence function based on the entie cumulative distibution of obustness levels, and employ a citeion analogous to fist ode stochastic dominance to indicate

19 Rank Robustness of Composite Indicatos 19 geate obustness. We then apply the methodology to seveal datasets and investigate how vaious changes affect the pevalence function. We begin with an initial weighting vecto w 0 and a dataset ˆX containing ˆn obsevations. Without loss of geneality, we enumeate the elements of ˆX as x 1, x 2, ˆn, x whee C 0 (x 1 ) C 0 (x 2 ˆn ) C 0 ( x ). The analysis can be simplified by assuming that no two obsevations in ˆX have the same composite value, so that C 0 (x 1 ) > C 0 (x 2 ˆn ) > > C 0 ( x ). 12 Thee ae ˆk = ˆn ( ˆn -1)/2 odeed pais of obsevations x i and x j with i < j, and each compaison x i C 0 x j has an associated obustness level ij *. Let P = [ ij * ] epesent the obustness pofile of ˆX (given w 0 ), which lists the level of obustness ij * fo evey odeed pai in a manne simila to Table 6. The mean obustness level in pofile P is given by = Σ i Σ j>i ij * / ˆk ; it is the aveage level of obustness of the ˆk many compaisons. Of couse, a highe mean level does not necessaily ensue that the pevalence of C is highe fo any given. An altenate appoach is to summaize obustness levels in a way that eflects the entie distibution, and not just the mean. Fo any given dataset ˆX and initial weighting vecto w 0, define the pevalence function p:[0,1] [0,1] to be the function which associates with each [0,1] the shae p() [0,1] of the ˆk compaisons whose obustness levels ae at least. In othe wods, p() is the popotion of compaisons fo which the C elation applies. 13 Suppose that p and q ae the pevalence functions fo ˆX (given w 0 ) and Y ˆ (given u 0 ), espectively. We say that ˆX has geate obustness than Ŷ if p() q() fo all [0,1], with p() > q() fo some [0,1]. In wods, no matte the level of obustness, the shae of all compaisons that exhibit th ode obustness is no lowe fo ˆX than Ŷ, and fo some it is highe. The two ae said to have the same obustness if thei pevalence functions ae the same. 12 This is tue fo each of the examples pesented below. 13 At = 0 the complete elation C 0 is used and hence p(0) = 1.

20 Rank Robustness of Composite Indicatos 20 A. Some Examples of Pevalence Functions Figue 5 depicts the pevalence functions obtained fom seveal datasets associated with well-known composite indicatos. The fist two ae fom the 1998 and 2004 Human Development Index 14, which uses equal weights acoss thee dimensions (health, education and income) to ank 177 counties. Next is the 2008 Index of Economic Feedom (IEF) ceated by the Wall Steet Jounal and the Heitage Foundation to measue the degee of economic feedom acoss counties 15. Thee ae 10 dimensions, spanning the spectum fom business feedom to labo feedom, all weighted equally in w 0. The IEF dataset coves 157 counties fo the yea *** Figue 5 Hee *** The final indicato is the 2008 Envionmental Pefomance Index (EPI), developed by Yale Univesity, Columbia Univesity, the Wold Economic Foum and the Joint Reseach Cente of the Euopean Commission to complement the envionmental tagets of the UN Millennium Development Goals. The EPI is a composite indicato whose vaiables may be viewed at vaious levels of aggegation 16. Fo puposes of illustation, we conside thee vesions hee. EPI 10 has ten vaiables and the initial weights ae not all equal. EPI 6 combines vaiables (and initial weights) to obtain a six vaiable vesion with unequal weights. EPI 2 aggegates futhe to obtain two vaiables with equal weight on each. As the initial weighting vectos ae consistent, each vesion of the EPI poduces identical values at its espective initial 14 Note that the Human Development Indices fo the yeas 1998 and 2004 ae obtained fom the Human Development Repot 2000 and 2006, espectively. 15 The ten dimensions of economic feedoms ae Business Feedom, Tade Feedom, Fiscal Feedom, Govenment Size, Monetay Feedom, Investment Feedom, Financial Feedom, Popety ights, Feedom fom Couption, Labo Feedom. Fo each dimension, the scoe is nomalized between zeo and hunded. The final scoe is obtained by simple aveage of these scoes. 16 EPI is composed of twenty five-dimensions of pefomance on envionment. Howeve, at the objective level all dimensions ae summaized in two categoies with equal weights: envionmental health and ecosystem vitality. At the policy level, all twenty-five dimensions ae summaized into six dimensions at. At the policy level, the weight vecto used is (0.5, 0.025, 0.075, 0.075, 0.075, 0.25). Futhe, these six dimensions ae sub-categoized into ten dimensions with the weight vecto (0.25, 0.125, 0.125, 0.025, 0.075, 0.075, 0.025, 0.025, 0.025, 0.25). To have detailed infomation on indicatos, see

21 Rank Robustness of Composite Indicatos 21 weighting vecto; but due to the diffeent numbes of vaiables, each has distinct obustness chaacteistics. 17 The EPI dataset coves 149 counties duing the yea Seveal initial obsevations can be made fom the pevalence functions given in Figue 5. Each gaph is downwad sloping, eflecting the fact that as ises, the set S expands, and hence the numbe of compaisons that can be made by C is lowe (o no highe). As falls to 0, each function ises to 100% compaability fo C ; in the othe diection, the value of p() at = 1 is the pecentage of the compaisons that ae fully obust. Thee is a wide vaiation in p(1) acoss datasets. It is clealy highest fo the HDI examples, with p(1) being about 69.8% in 2004 and 73.2% in 1998; it is 47.4% fo the two vaiable EPI; and it is much lowe fo the emaining indicatos 18 (4.2% and 1.5% in the case of EPI 6 and EPI 10, espectively, and 6.5% fo the EFI). Fo between 0 and 1, the HDI compaisons ae also moe obust than the compaisons of the EPI and the EFI, and of the two HDI datasets, 1998 exhibits geate obustness than Fo the EPI examples, a highe level of aggegation and hence a lowe numbe of vaiables, leads to geate obustness. Howeve, EPI 2 is still less obust than eithe of the HDIs, which have thee vaiables. The shapes of the p() functions ae diffeent, with some being essentially linea, and othes exhibiting ponounced cuves. Dawing on these examples, we now examine the pevalence of obustness fom a moe theoetical pespective. What tansfomations allow the esulting datasets to be compaed in tems of obustness? B. Fixed Robustness and Tansfomations We begin with tansfomations of the data that leave p() unchanged and hence yield pais of datasets with the same obustness popeties. A monotonically inceasing tansfomation of X is a function f:x R D that can be witten as f(x) = (f 1 (x 1 ),, f D (x D )) whee each function f d (x d ) is monotonically inceasing; a common-slope affine tansfomation of X has the additional popety that each function f d (x d ) can be witten 17 Fo consistent initial weighting vectos, the weight on an aggegated vaiable is sum of the weights on the vaiables that wee aggegated. See the next subsection. 18 Fo EPI 6 and EPI 10, we also calculate the pevalence function using initial equal weights. We find the dominance elation to hold between EPI 6 and EPI 10 by EPI 6 is moe obust that EPI 10 fo all. The elationship is explained in Figue 6.

22 Rank Robustness of Composite Indicatos 22 as f d (x d ) = αx d + β d fo some α > 0 and β d in R. We say that Y ˆ is obtained fom ˆX by a common-slope affine tansfomation (espectively, by a monotonically inceasing tansfomation) if Y ˆ = {f(x): x ˆX } fo some tansfomation f having the appopiate popety. Applying a monotonically inceasing tansfomation to a dataset peseves the odeings of achievements within each dimension, but can disupt the weighted aveages acoss dimensions. In paticula, it is possible that C(x';w 0 ) > C(x;w 0 ) and C(y';w 0 ) < C(y;w 0 ) whee y' and y ae tansfomations of x' and x, espectively, which implies that the obustness pofiles of Y ˆ and ˆX can be athe diffeent fo the same w 0. On the othe hand, if we estict consideation to common-slope affine tansfomations, we see that C(y;w) = w y = αw x + w β whee β = (β 1,, β D ), and hence C(x';w) C(x;w) if and only if C(y';w) C(y;w), whee y' and y ae the espective tansfomations of x' and x. In this case Y ˆ and ˆX have the same obustness pofile and hence the same pevalence function p() given w 0. So, fo example, if evey dimension is scaled up o down in the same popotion, this will leave p() unchanged, as will simply adding a diffeent constant to each dimension. On the othe hand, multiplying each dimension by a diffeent positive constant altes the implicit weighting acoss dimensions, potentially changing the ankings of tansfomed obsevations. Using an abitay monotonic inceasing tansfomation likewise can alte ankings and lead to diffeent pevalence functions fo the tansfomed dataset. Note, though, that fully obust compaisons ae peseved unde a monotonic tansfomation (as noted in the discussion following Theoem 1), and hence the pevalence p(1) of full obustness does not change. These esults ae summaized in the following theoem. 19 Theoem 4 Suppose that the initial weighting vecto is fixed. If Ŷ is obtained fom ˆX by a monotonically inceasing tansfomation, then Ŷ and ˆX shae the same 19 The esult on monotonic tansfomations would be tue even if the initial weighting vectos ae diffeent just so both ae stictly positive. The ole played by common-slope affine tansfomations is simila to assumptions used in social choice theoy: see Blackoby, Donaldson, and Weymak (1984).

23 Rank Robustness of Composite Indicatos 23 pevalence value p(1). If Ŷ is obtained fom ˆX by a common-slope affine tansfomation, then they shae the same pevalence function p(). 20 In the example of the HDI, the nomalized income, education and health vaiables used to constuct index values ae actually monotonic tansfomations of undelying vaiables involving a nonlinea function in the case of income, and affine tansfomations with diffeent slopes acoss the thee vaiables. Consequently, the specific shapes of the tansfomations can influence HDI compaisons as well as thei measued obustness levels. Howeve, as indicated in Theoem 4, these tansfomations do not influence fully obust compaisons and p(1). If one esticted consideation to C 1 compaisons, thee would be no need to select the ight tansfomations o even to tansfom vaiables at all: one could use the oiginal income, education and health vaiables diectly. A second fom of tansfomation eplaces each vaiable in the achievement vecto with one o moe copies of that vaiable. A eplicating tansfomation of X is a function f:x R D' fo some D' > D such that f(x) = (f 1 (x 1 ),, f D (x D )), whee each f d (x d ) is the k d -fold eplication (x d,x d,,x d ) fo some intege k d 1. We say that that Y ˆ is obtained fom ˆX by a eplicating tansfomation if Y ˆ = {f(x): x ˆX } fo some tansfomation f of this type. Tansfomed achievement vectos have highe dimension D' and, consequently, the associated weighting vectos must be adjusted to account fo this. Now, which initial weighting vecto u 0 fo Y ˆ would coespond to the oiginal w 0 fo ˆX? One option is to divide the weight wd 0 equally among the associated dimensions in u 0 ; howeve, it tuns out that any allocation of the weight w 0 d acoss its associated dimensions will do. We say u 0 is consistent with w 0 if, fo each d = 1,, ˆn the weight w 0 d on x d is equal to the sum of the k d enties in u 0 associated with f d (x d ) = (x d,x d,,x d ). So fo example, if D = 2 and f eplicates each enty two times, then w 0 = (½, ½) is consistent with u 0 = (½, 0, ¼, ¼). We have the following esult. 20 The fist pat of Theoem 4 will geneate same pevalence function p(1) even if use the dominance citeion poposed by Chechye, Ooghe, and Puyenboeck (2005)

24 Rank Robustness of Composite Indicatos Theoem 5 If Ŷ is obtained fom ˆX by a eplicating tansfomation, and u 0 consistent with w 0, then ˆ Y and ˆX have the same pevalence function p(). 24 is Poof Suppose that y is a eplicated achievement vecto associated with x, so that y = f(x) fo a eplicating tansfomation f. Given the initial weighting vecto w 0 and a consistent weighting vecto u 0, it is clea that C(y;u 0 ) = u 0 f(x) = w 0 x = C(x;w 0 ). Now let (0,1] and select any d = 1,,D along with an index value d' of one of its copies. Let v d denote the dimension d vetex of the simplex S in R D and let v d denote the dimension d' vetex of the simplex S in R D'. It is clea that C(x; v d) = v d x = (1- )C(x;w 0 ) + x d = (1-)C(y;u 0 ) + y d' = v d y = C(y; v d ). Hence, whee y' and y ae the espective tansfomations of x' and x, we have (i) C(x';w 0 ) C(x;w 0 ) if and only if C(y';u 0 ) C(y;u 0 ), and (ii) C(x'; v d) C(x; v d) if and only if C(y'; v d ) C(y; v d ). Since (ii) holds fo each d and evey associated d', it follows fom Theoem 2 that x'c x if and only if y'c y, and p() is the same fo both. In othe wods, appending copies of one o moe existing vaiables leaves the compaisons and the obustness popeties of a dataset unaffected, as long as the effective weight on each vaiable is unchanged. As an example, conside what would happen if the education vaiable in an HDI dataset wee eplicated to obtain a fou vaiable dataset. Using equal weights of ¼ fo the fou dimensional dataset would likely alte ankings since this would, in effect, incease the aggegate weight on education. Howeve, if the total weight on the two education vaiables is maintained at ⅓, say whee each vaiable eceives a weight of 1 6, then all compaisons and obustness levels would be the same as befoe. One implication of this is that the numbe of vaiables pe se does not have an independent impact on a dataset s obustness. In contast, the empiical evidence povided by Figue 5 does seem to suggest that a geate numbe of vaiables is associated with lowe obustness. The evidence is paticulaly stiking fo the thee EPI examples, whee the aggegation of vaiables, and hence the decease in the numbe of

25 Rank Robustness of Composite Indicatos 25 vaiables, clealy leads to inceased obustness even though they use the same undelying data. Is this due to the deceased numbe of vaiables? Let us examine how EPI 6 is constucted fom EPI 10. The fist and fifth vaiables in EPI 6 ae each obtained by combining thee distinct vaiables in EPI 10 (namely, vaiables 1-3 and vaiables 7-9), while the emaining vaiables ae unchanged. Weights fom the initial weighting vecto u 0 fo EPI 10 ae used to constuct each new vaiable in EPI 6 as a weighted aveage of the souce vaiables fom EPI 10, and the weight on the new vaiable is the sum of the coesponding weights in u 0. The new w 0 is thus consistent with u 0. Now conside a ten vaiable eplication of EPI 6 that epeats vaiable 1 thee times and vaiable 5 thee times and let the initial weighting vecto be u 0. By Theoem 5, this intemediate dataset has pecisely the same obustness pofile and pevalence function as EPI 6. It is not the numbe of vaiables that is diving the obseved decease in obustness. Instead, its souce is found in the tansfomation fom the intemediate dataset to EPI 10, by which the pefectly coelated tiplets ae conveted to vaiables that ae less positively associated. The fall in obustness is due to disageements among the new vaiables, athe than the highe numbe of vaiables pe se. Association among vaiables is likely the key dive of obustness, and this is exploed futhe in the next section. C. Inceased Robustness and Positive Association What factos geneally lead to geate obustness? At an intuitive level, the possibility of fully obust compaisons is elated to the degee of coelation o positive association among the dimensional vaiables. Fo example, if two of the achievements ae pefectly negatively coelated, so that when one ises, the second falls, then it is impossible fo vecto dominance and hence C 1 to hold. On the othe hand, if thee is complete positive association 21 between all vaiables, so that when any vaiable ises, all ise, then evey achievement vecto is compaable by vecto dominance, and C 1 is univesally applicable. In Figue 5 we saw that both HDI datasets have high levels of 21 Note that if thee ae moe than two dimensions then not all of them can eve be pefectly negatively associated. In othe wods, thee does not exist any concept called pefect negative association such as pefect positive association.