Review of ranking methods in the data envelopment analysis context

Transcription

1 European Journal of Operational Research 140 (2002) Review of ranking methods in the data envelopment analysis context Nicole Adler a, *,1, Lea Friedman b,2, Zilla Sinuany-Stern b,c,2 a School of Business Administration, Hebrew University of Jerusalem, Mount Scopus, Jerusalem 91905, Israel b Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, P.O. Box 653, Beer-Sheva 84105, Israel c The College of Judea and Samaria, Ariel, Israel Abstract Within data envelopment analysis (DEA) is a sub-group of papers in which many researchers have sought to improve the differential capabilities of DEA and to fully rank both efficient, as well as inefficient, decision-making units. The ranking methods have been divided in this paper into six, somewhat overlapping, areas. The first area involves the evaluation of a cross-efficiency matrix, in which the units are self and peer evaluated. The second idea, generally known as the super-efficiency method, ranks through the exclusion of the unit being scored from the dual linear program and an analysis of the change in the Pareto Frontier. The third grouping is based on benchmarking, in which a unit is highly ranked if it is chosen as a useful target for many other units. The fourth group utilizes multivariate statistical techniques, which are generally applied after the DEA dichotomic classification. The fifth research area ranks inefficient units through proportional measures of inefficiency. The last approach requires the collection of additional, preferential information from relevant decision-makers and combines multiple-criteria decision methodologies with the DEA approach. However, whilst each technique is useful in a specialist area, no one methodology can be prescribed here as the complete solution to the question of ranking. Ó 2002 Published by Elsevier Science B.V. Keywords: Data envelopment analysis; Ranking techniques; Super-efficiency; Cross-efficiency; Benchmarking; Multivariate statistics; MCDM 1. Introduction DEA was initiated in 1978 when Charnes, Cooper and Rhodes (CCR) demonstrated how to * Corresponding author. Tel.: address: msnic@mscc.huji.ac.il (N. Adler). 1 I would like to thank the Kreitman Foundation Post- Doctoral Fellowship at Ben-Gurion University of the Negev and the partial support of the Recanati Foundation. 2 This paper was partially supported by the Paul Ivanier Center for Robotics and Production Management, Ben-Gurion University of the Negev. change a fractional linear measure of efficiency into a linear programming (LP) format. As a result, decision-making units (DMUs) could be assessed on the basis of multiple inputs and outputs, even if the production function was unknown. Indeed, this paper will only consider the multiple inputs and multiple outputs case. This non-parametric approach solves an LP formulation per DMU and the weights assigned to each linear aggregation are the results of the corresponding LP. The weights are chosen so as to show the specific DMU in as positive a light as possible, /02/$ - see front matter Ó 2002 Published by Elsevier Science B.V. PII: S (02)

2 250 N. Adler et al. / European Journal of Operational Research 140 (2002) under the restriction that no other DMU, given the same weights, is more than 100% efficient. Consequently, a Pareto frontier is attained, marked by specific DMUs on the boundary envelope of input output variable space. The frontier is considered a sign of relative efficiency, which has been achieved by at least one DMU. Charnes et al. (1978) described DEA as a mathematical programming model applied to observational data [which] provides a new way of obtaining empirical estimates of extremal relations such as the production functions and/or efficient production possibility surfaces that are a cornerstone of modern economics. Various theoretical extensions have been developed, based on the original CCR model: Banker et al. (1984) developed a variable returnsto-scale variation; the multiplicative model was developed by Charnes et al. (1982) in which the data are transformed using a logarithmic structure; Charnes et al. (1985b) developed the additive variation, in which the objective function contains slack variables alone. Seiford and Thrall (1990) provide a useful discussion and comparison of all the basic models available to date in DEA. Many additional theoretical papers in the field have adapted the models to deal with problems that have occurred in practice. One adaptation has been in the field of ranking DMUs. The basic DEA results group the DMUs into two sets, those that are efficient and define the Pareto frontier and those that are inefficient. In order to rank all the DMUs, another approach or modification was required. Often decision-makers are interested in a complete ranking, beyond the dichotomized classification, in order to refine the evaluation of the units. One problem that has been discussed frequently in the literature has been the lack of discrimination in DEA applications, in particular when there are insufficient DMUs or the number of inputs and outputs is too high relative to the number of units. This is an additional reason for the growing interest in complete ranking techniques. Furthermore, fully ranking units is an established approach in the social sciences, in general (see Young and Hamer, 1987), and in multiple-criteria decision making (MCDM), in particular. By now many papers on DEA and ranking (over 50) have been published over the last decade within the DEA context. Since most decision-makers are interested in a complete ranking, the use of such techniques aids in marketing the DEA approach. It should be noted that the methods discussed here could be considered postanalyses since they do not replace the standard DEA models but rather provide added value. This review describes the ranking methods developed in the literature and since many papers have been published in this field, we have grouped them into six basic areas. These methods are classified by several criteria and are not entirely mutually exclusive. After specifying the DEA method mathematically in Section 2, Section 3 discusses the cross-efficiency technique, which was first suggested by Sexton et al. (1986), whereby the DMUs are both self and peer evaluated. In Section 4 we will review the super-efficiency technique, first published in Andersen and Petersen s paper of 1993, in which DMUs are ranked through the exclusion of the unit being scored from the DEA dual LP. Section 5 discusses the evaluation of DMUs through benchmarking, an approach originating in Torgersen et al. (1996), in which an efficient unit is highly ranked if it appears frequently in the reference sets of inefficient DMUs. Section 6 will review the papers that apply multivariate statistical tools, such as canonical correlation analysis and discriminant analysis, in order to reach a complete DMU rank, first suggested by Friedman and Sinuany-Stern (1997). Section 7 discusses the ranking of inefficient DMUs, not always considered in the previous methodologies. Section 8 discusses the cross-pollination between multi-criteria decision-making (MCDM) methodologies and DEA in terms of ranking. Finally, Section 9 presents the results of the various methodologies on an example discussed in Sexton et al. (1986) and Section 10 provides conclusions and a summary of the review, including a table of the DEA software packages available today and the ranking approaches they offer automatically. 2. The data envelopment analysis model DEA is a mathematical model that measures the relative efficiency of decision-making units

3 N. Adler et al. / European Journal of Operational Research 140 (2002) with multiple inputs and outputs but with no obvious production function to aggregate the data in its entirety. Relative efficiency is defined as the ratio of total weighted output to total weighted input. By comparing n units with s outputs denoted by y rk ; r ¼ 1;...; s; and m inputs denoted by x ik, i ¼ 1;...; m; the efficiency measure for DMU k is h k ¼ Max u r;v i P s u ry rk P m v ix ik ; where the weights, u r and v i, are non-negative. A second set of constraints requires that the same weights, when applied to all DMUs, do not provide any unit with efficiency greater than one. This condition appears in the following set of constraints: P s u ry rj P m v ix ij 6 1 for j ¼ 1;...; n: The efficiency ratio ranges from zero to one, with DMU k being considered relatively efficient if it receives a score of one. Thus, each unit will choose weights so as to maximize self-efficiency, given the constraints. The result of the DEA is the determination of the hyperplanes that define an envelope surface or Pareto frontier. DMUs that lie on the surface determine the envelope and are deemed efficient, whilst those that do not are deemed inefficient. The formulation described above can be translated into a linear program, which can be solved relatively easily and a complete DEA solves n linear programs, one for each DMU. h k ¼ Max X m X m v i x ij Xs X s v i x ik ¼ 1; u r y rk u r P 0 for r ¼ 1;...; s; v i P 0 for i ¼ 1;...; m: u r y rj P 0 for j ¼ 1;...; n; ð1þ Model (1), often referred to as the CCR model (Charnes et al., 1978), assumes that the production function exhibits constant returns-to-scale. The BCC (Banker et al., 1984) model adds an additional constant variable, c k, in order to permit variable returns-to-scale: h k ¼ Max X m X m v i x ij Xs Xs v i x ik ¼ 1; u r y rk þ c k u r P 0 for r ¼ 1;...; s; v i P 0 for i ¼ 1;...; m: u r y rj c k P 0 for j ¼ 1;...; n; ð2þ It should be noted that the results of the CCR input-minimized or output-maximized formulations are the same, which is not the case in the BCC model. Thus, in the output-oriented BCC model, the formulation maximizes the outputs given the inputs and vice versa The dual program of the CCR model If a DMU proves to be inefficient, a combination of other efficient units can produce either greater output for the same composite of inputs, use fewer inputs to produce the same composite of outputs or some combination of the two. A hypothetical decision making unit, k 0, can be composed as an aggregate of the efficient units, referred to as the efficient reference set for inefficient unit k. The solution to the dual problem of the linear program directly computes the multipliers required to compile k 0 : Min f k Xn X n L kj x ij þ f k x ik P 0 for i ¼ 1;...; m; L kj y rj P y rk for r ¼ 1;...; s; L kj P 0 for j ¼ 1;...; n: ð3þ

4 252 N. Adler et al. / European Journal of Operational Research 140 (2002) In the case of an efficient DMU, all dual variables will equal zero except for L kk and f k, which reflect the unit k s efficiency, both of which will equal one. If DMU k is inefficient, f k will equal the ratio solution of the primal problem. The remaining variables, L kj, if positive, represent the multiples by which unit k s inputs and outputs should be multiplied in order to compute the composite efficient DMU k The slack-adjusted CCR model In the slack-adjusted DEA models, see for example model (4), a weakly efficient DMU will now be evaluated as inefficient, due to the presence of input and output oriented slacks s i and r r, respectively.! X m Min f k e s i þ Xs r r Xn X n L kj x ij þ f k x ik s i ¼ 0 for i ¼ 1;...; m; L kj y rj r r ¼ y rk for r ¼ 1;...; s; L kj ; s i ; r r P 0 for j ¼ 1;...; n; ð4þ whereby e is a positive non-archimedean infinitesimal The additive model An alternative formulation proposed by Charnes et al. (1985b) utilizes slacks alone in the objective function. This model is used in both the benchmarking approach and measure of inefficiency dominance developed in Sections 5 and 7, respectively. Min Xm Xn X n s i Xs r r L kj x ij s i ¼ x ik for i ¼ 1;...; m; L kj y rj r r ¼ y rk for r ¼ 1;...; s; L kj ; s i ; r r P 0 for j ¼ 1;...; n: ð5þ In order to avoid large variability in the weights for all DEA models, Thompson et al. (1986, 1990, 1992) suggest the use of assurance regions (ARs) i.e. bounds on the weights. This, in turn, increases the differentiability among the unit scores by reducing the number of efficient DMUs. In the extreme case, the weights will be reduced to a single set of common weights and the units will be fully ranked. However, the AR literature is not discussed in this review since the concept does not strive, nor does it generally succeed, in reaching a complete ranking of DMUs. Another attempt to improve the discriminating power of DEA can be found in Adler and Golany (2001), where principal component analysis was utilized to reduce the number of inputs/outputs, thus reducing the problem of dimensionality. However, this technique cannot ensure a complete ranking but rather a reduction in the set of efficient units. 3. Cross-efficiency ranking methods The cross-evaluation matrix was first developed by Sexton et al. (1986), inaugurating the subject of ranking in DEA. Indeed, as Doyle and Green (1994) argued, decision-makers do not always have a reasonable mechanism from which to choose assurance regions, thus they recommend the crossevaluation matrix for ranking units. The crossefficiency method simply calculates the efficiency score of each DMU n times, using the optimal weights evaluated by the n LPs. The results of all the DEA cross-efficiency scores can be summarized in a cross-efficiency matrix as shown in Eq. (6): P s h kj ¼ P u rky rj m v ; k ¼ 1;...; n; j ¼ 1;...; n: ikx ij ð6þ Thus, h kj represents the score given to unit j in the DEA run of unit k i.e. unit j is evaluated by the weights of unit k. Note that all the elements in the matrix are between zero and one, 0 6 h kj 6 1, and the elements in the diagonal, h kk, represent the standard DEA efficiency score, h kk ¼ 1 for efficient units and h kk < 1 for inefficient units. Further-

5 N. Adler et al. / European Journal of Operational Research 140 (2002) more, if the weights of the LP are not unique, a goal-programming technique can be applied to choose between the optimal solutions. According to Sexton et al. (1986), the secondary goals could be, for example, either aggressive or benevolent. In the aggressive context, DMU k chooses amongst the optimal solutions such that it maximizes selfefficiency and at a secondary level minimizes the other DMUs cross-efficiency levels. The benevolent secondary objective would be to maximize all DMUs cross-efficiency rankings. See also Oral et al. (1991) for methods of evaluating the goal programs. The cross-efficiency ranking method in the DEA context utilizes the results of the cross-efficiency matrix h kj in order to rank scale the units. Let us define h k ¼ P n h kj=n as the average crossefficiency score given to unit k. Averaging, however, is not the only possibility, as the median, minimum or variance of scores could also be applied (see Green et al. (1996) for further detailed suggestions to help in avoiding such problems as the lack of Independence of Irrelevant Alternatives). It could be argued that h k,oran equivalent, is more representative than h kk, the standard DEA efficiency score, since all the elements of the cross-efficiency matrix are considered, including the diagonal. Furthermore, all the units are evaluated with the same sets of weight vectors. Thus the h k score better represents the unit evaluation since it measures the overall ratios over all the runs of all the units. The maximum value of h k is 1, which occurs if unit k is efficient in all the runs i.e. all the units evaluate unit k as efficient. In order to rank the units, we can simply assign the unit with the highest score a rank of one and the unit with the lowest score a rank of n. While the DEA scores, h kk, are non-comparable, since each uses different weights, the h k score is comparable because it utilizes the weights of all the units equally. However, this is also the drawback of the technique, since the evaluation subsequently loses its connection to the multiplier weights. Furthermore, Doyle and Green (1994) developed the idea of a maverick index within the confines of cross-efficiency. The index measures the deviation between h kk, the self-appraised score and the unit s peer scores, as shown in Eq. (7). M k ¼ h kk e k ; where e k ¼ 1 X h kj : ð7þ e k ðn 1Þ The higher the value of M k, the more the DMU can be considered a maverick. Doyle and Green (1994) argued that this score can go handin-hand with the benchmarking process (Section 5), whereby those DMUs considered efficient under self-appraisal but fail to appear in the reference sets of inefficient DMUs will generally achieve a high M k score. Those achieving a low score are generally all-round performers and are frequently both self- and peer efficient. 4. Super-efficiency ranking techniques Andersen and Petersen (1993) developed a new procedure for ranking efficient units. The methodology enables an extreme efficient unit k to achieve an efficiency score greater than one by removing the kth constraint in the primal formulation, as shown in model (8). h k ¼ Max X m X m v i x ij Xs Xs v i x ik ¼ 1; u r y rk u r Pe for r ¼ 1;...;s; v i Pe for i ¼ 1;...;m: j6¼k u r y rj P0 for j ¼ 1;...;n; j 6¼ k; ð8þ The dual formulation of the super-efficient model, as seen in model (9), computes the distance between the Pareto frontier, evaluated without unit k, and the unit itself i.e. for J ¼ fj ¼ 1;...; n; j 6¼ kg.

6 254 N. Adler et al. / European Journal of Operational Research 140 (2002) Min f k X L kj x ij 6 f k x ik for i ¼ 1;...; m; j2j X L kj y rj P y rk for r ¼ 1;...; s; j2j L kj P 0 for j ¼ 1;...; n: ð9þ However, there are three problematic areas with this methodology. First, Andersen and Petersen refer to the DEA objective function value as a rank score for all units, despite the fact that each unit is evaluated according to different weights. This value in fact explains the proportion of the maximum efficiency score that each unit k attained with its chosen weights in relation to a virtual unit closest to it on the frontier. Furthermore, if we assume that the weights reflect prices, then each unit has different prices for the same set of inputs and outputs within the same organization. Second, the super-efficient methodology can give specialized DMUs an excessively high ranking. To avoid this problem, Sueyoshi (1999) introduced specific bounds on the weights in a super-efficient ranking model as described in Eqs. (10). v i P 1=ðm þ sþ max ðx ij Þ j ð10þ u r P 1=ðm þ sþ max ðy rj Þ: j Furthermore, in order to limit the super-efficient scores to a scale with a maximum of 2, Sueyoshi developed an Adjusted Index Number (AIN) formulation, as shown in Eq. (11). AIN k ¼ 1 þ d k min j2e d j max j2e d j where E is the set of efficient units: min j2e d j ; ð11þ The third problem lies with an infeasibility issue, which if it occurs, means that the super-efficient technique cannot provide a complete ranking of all DMUs. Thrall (1996) used the model to identify extreme efficient DMUs and noted that the superefficiency CCR model may be infeasible. Zhu (1996a), Dula and Hickman (1997) and Seiford and Zhu (1999) prove under which conditions various super-efficient DEA models are infeasible. Mehrabian et al. (1999) suggested a modification to the dual formulation in order to ensure feasibility, as specified in model (12). Min f k X L kj x ij þ x ik f k P 0 for i ¼ 1;...; m; j2j X L kj y rj P y rk for r ¼ 1;...; s; j2j L kj P 0 for j ¼ 1;...; n: ð12þ Despite these drawbacks, possibly because of the simplicity of the concept, many published papers have used this approach. For example, Hashimoto (1997) developed a DEA super-efficient model with assurance regions in order to rank the DMUs completely. Using model (13), Hashimoto avoided the need for compiling additional preference information in providing a complete ranking of n candidates. h k ¼ Max X s X s u r y rk u r y rj 6 1 for j ¼ 1;...; n; j 6¼ k; u r u rþ1 P e for r ¼ 1;...; s 1; u s P e; u r 2u rþ1 þ u rþ2 P 0 for r ¼ 1;...; s 2; ð13þ where u r is the sequence of weights given to the rth place vote (whereby each voter selects and ranks the top t candidates). The use of assurance regions avoids the specialization pitfall of the standard super-efficiency model. 5. Benchmark ranking method Torgersen et al. (1996) achieved a complete ranking of efficient DMUs by measuring their importance as a benchmark for inefficient DMUs.

7 N. Adler et al. / European Journal of Operational Research 140 (2002) The benchmarking measure is evaluated in a twostage procedure, whereby the additive model (see Section 2.3) is first used to evaluate the value of the slacks. The set of efficient units, V, is identified as those units whose slack values equal zero. In the second stage, model (14) is applied to all decisionmaking units. 1 E k ¼ Max f k X L kj x ij s ik ¼ x ik for i ¼ 1;...; m; j2v X L kj y rj f k y rk r rk ¼ 0 for r ¼ 1;...; s; j2v X L kj ¼ 1: j2v ð14þ In order to rank the efficient DMUs and evaluate which are of particular importance to the industry, the benchmarking measure aggregates the individual reference weights as shown in Eq. (15). P n q r k L jk yrj P y rj yr P y r 8k ¼ 1;...; V ; r ¼ 1;...; s; where ð15þ y P rj ¼ y rj þ r rj : E j For each efficient DMU k, the benchmark q r k measures the fraction of total aggregated potential increase in output r, over which k acts as a referent. The efficient units together define the entire potential within each output variable. An average value of q k can then be calculated in order to rank all efficient DMUs completely. Torgersen et al. (1996) apply their technique to a set of unemployment offices in Norway and show that in certain cases different conclusions were reached to that of Andersen and Petersen s super-efficiency method (see Section 4), due to the outlier problem of the latter technique. This is somewhat similar to the results reported in Sinuany-Stern et al. (1994), in which an efficient DMU is highly ranked if it is chosen as a useful target by many other inefficient units. The technique developed in this paper is applied to all DMUs in two stages. In the first stage, the efficient units are ranked by simply counting the number of times they appear in the reference sets of inefficient units, an idea first developed in Charnes et al. (1985a). The inefficient units are then ranked, in the second stage, by counting the number of DMUs that needs to be removed from the analysis before they are considered efficient. However, a complete ranking cannot be assured since many DMUs may receive the same ranked score. 6. Ranking with multivariate statistics in the DEA context An alternative approach suggested in the literature involves the use of statistical techniques in alliance with DEA to achieve a complete ranking. One of the major aims of the methodologies described in this section is to close the gap between DEA and the classical statistical approaches. It should be noted that DEA is a methodology directed towards frontiers rather than central tendencies. Instead of trying to fit regression planes through the center of the data, DEA floats a piecewise linear surface (the efficient frontier) that rests on top of the observations. DEA optimizes and thereby focuses on each unit separately, while regression is a parametric approach that fits a single function to the data collected on the basis of average behavior that requires the functional form to be pre-specified. On the other hand, in DEA the values of the weights differ from unit to unit and whilst this flexibility in the choice of weights characterizes the DEA model, different weights cannot generally be used for ranking because these scores are obtained from different comparison (peer) groups for different units. In this section we present three ranking procedures based on multivariate statistical analyses using common weights. Non-parametric statistical tests, such as Mann Whitney, are utilized throughout to verify the compatibility between the rank and the DEA dichotomic classification. Empirically, the literature shows a high statistical

8 256 N. Adler et al. / European Journal of Operational Research 140 (2002) significance between the statistical analyses and the DEA results Canonical correlation analysis for ranking Canonical correlation is an extension of regression analysis. While the regression model explains a single output using multiple inputs, canonical correlation analyzes multiple inputs and multiple outputs. CCA searches for a single vector weight for the inputs and outputs, common to all the units. CCA constructs a composite input variable Z j, as a linear combination of the m inputs and a composite output variable W j, as a linear combination of the s outputs: Z j ¼ V 1 x 1j þ V 2 x 2j þþv m x mj ; W j ¼ U 1 y 1j þ U 2 y 2j þþu s y sj ; where CCA determines the two vectors of coefficients, V 0 ¼ðV 1 ; V 2 ;...; V m Þ and U 0 ¼ðU 1 ; U 2 ;...; U s Þ, so as to maximize r zw, the coefficient of correlation between the composite input, Z, and the composite output, W. Formally, this is specified in model (16). Max V 0 S xx V ¼ 1; U 0 S yy U ¼ 1; V 0 S xy U r zw ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðv 0 S xx V ÞðU 0 S yy UÞ ð16þ where S xx ; S yy and S xy are the matrices of the sums of squares and sums of products of the variables, respectively. Note that the weights V 0 and U 0 are determined up to a proportional constant. There is a closed form solution for the weights, however, this ranking is only feasible in the DEA context if all the weights are non-negative. Friedman and Sinuany-Stern (1997) use the CCA method methodology by defining a scaling ratio score, T, as a ratio of linear combinations of the inputs and outputs. Then they utilize the common weights for the linear combinations that are drawn from the largest eigenvalue of the CCA method, as shown in Eq. (17). T j ¼ W P s j ¼ P U ry rj m Z j V ; j ¼ 1;...; n: ð17þ ix ij Note that while the DEA efficiency ratio h j is bounded above by 1, the scaling ratio T j of the CCA/DEA is unbounded. Thus, it is the rank order of the scaling ratios that is important rather than their absolute value. Although the CCA is independent of the DEA results, the same formulation has been applied to the two types of variables; weighted inputs and weighted outputs. Moreover, the scaling ratio score T is also similar to the DEA score, namely it represents the ratio described in DEA. The use of a single set of weights is possibly the most obvious methodology for ranking and in this case the CCA has been built in the light of the DEA context. Hence, empirically the results are statistically significantly similar Linear discriminant analysis for ranking Sinuany-Stern et al. (1994) used linear discriminant analysis (DDEA) in order to find a score function which ranks DMUs, given the DEA division into efficient and inefficient sets. Using traditional discriminant analysis of two groups in the DEA context, they constructed a one-dimensional linear function, as specified in Eq. (18). D j ¼ Xs u r y rj þ Xm v i ð x ij Þ; j ¼ 1;...; n: ð18þ If D j > D c, then unit j is classified as efficient, otherwise it is classified as inefficient, where D c is a critical value based on the midpoint of the means of the discriminant function value of the two groups (see Morrison, 1976). The value of D j is used for ranking, where the unit with the largest value is the most efficient etc. The common weights u r and v i have closed form solutions, however, the DDEA ranking is feasible only if all the weights are non-negative. Furthermore, as with the CCA/DEA formulation, the results of the DDEA will only be statistically similar to the standard DEA result, which means that some efficient units may be ranked lower than inefficient units and vice versa.

9 N. Adler et al. / European Journal of Operational Research 140 (2002) Discriminant analysis of ratios for ranking In addition, Sinuany-Stern and Friedman (1998) developed another technique in which discriminant analysis of ratios was applied to DEA (DR/DEA) in order to avoid the infeasibility problems of the previous two approaches. Instead of considering a linear combination of the inputs and outputs in one equation (as in the traditional discriminant analysis of the two groups), they constructed a ratio function between a linear combination of the inputs and a linear combination of the outputs. In some ways this ratio function is similar to the DEA efficiency ratio, however, whilst DEA provides weights for the inputs and outputs, which vary from unit to unit, DR/DEA provides common weights for all units. In principle, DR/DEA determines the weights such that the ratio score function discriminates optimally between two groups of observations (DMUs) on a one-dimensional scale (in our case, efficient and inefficient units predetermined by DEA). The ratio, T j, and the arithmetic means of the ratio scores of the efficient and inefficient groups are P s T j ¼ P u ry rj m v ; j ¼ 1;...; n; ix ij T 1 ¼ Xn 1 T j n 1 and T 2 ¼ Xn j¼n 1 þ1 T j n 2 ; where n 1 and n 2 are the number of efficient and inefficient units in the DEA model, respectively. The weighted mean of the entire n units ðn ¼ n 1 þ n 2 Þ will be denoted by T ¼ n 1T 1 þ n 2 T 2 : n Our problem is to find the common weights v i and u r such that the ratio of the between-group variance of T ; ðss B ðt ÞÞ and the within group variance of T ; ðss W ðt ÞÞ will be maximized, as shown in model (19). SS B ðt Þ max k ¼ max u r;v i u r;v i SS W ðt Þ ; ð19þ SS B ðt Þ¼n 1 ðt 1 T Þ 2 þ n 2 ðt 2 T Þ 2 ¼ n 1n 2 n 1 þ n 2 ðt 1 T 2 Þ 2 ; SS W ðt Þ¼ Xn 1 ðt j T 1 Þ 2 þ Xn j¼n 1 þ1 ðt j T 2 Þ 2 : DR/DEA constructs the efficiency score for each unit j as T j, the ratio between the composite output and composite input. Thus it rank scales the DMUs so that the unit with the highest score receives rank 1 and the unit with the lowest score ranks n. If any weight is negative then non-negativity constraints ought to be added to the optimization problem. To solve this problem, they used a non-linear search optimization algorithm, however, there is no guarantee that the solution found is globally optimal. 7. The ranking of inefficient decision-making units The majority of techniques so far discussed have not attempted to rank the inefficient decision-making units beyond the efficiency scores attained from the standard DEA models. However, as argued in Cooper and Tone (1997), the original efficiency value will generally be determined from different facets, which means that these values are being derived from comparisons involving performances of different sets of DMUs. The benchmarking concept only attempts to rank the efficient DMUs identified in the standard DEA models. The super-efficiency method ranks inefficient units in the same manner as the standard DEA model. It should be noted that both the cross-efficiency method and the statistical techniques do attempt to address this problem. One concept, derived in Bardhan et al. (1996), attempts to rank inefficient units using a Measure of Inefficiency Dominance (MID). The measure is based on slack-adjusted DEA models from which an overall measure of inefficiency can be computed as shown in Eq. (20). P m ðs i =x ikþþ P s ðr r =y rkþ 6 1: ð20þ m þ s The MID index ranks the inefficient DMUs according to their average proportional inefficiency in all inputs and outputs. However, just as the

10 258 N. Adler et al. / European Journal of Operational Research 140 (2002) benchmarking approach (see Section 5) only ranks the efficient units, the MID index only ranks the inefficient units. 8. DEA and multi-criteria decision-making methods The MCDM literature was entirely separate from DEA research until 1988, when Golany combined interactive, multiple-objective linear programming and DEA. Whilst the MCDM literature does not consider a complete ranking as their ultimate aim, they do discuss the use of preference information to further refine the discriminatory power of the DEA models. In this manner, the decision-makers could specify which inputs and outputs should lend greater importance to the model solution. However, this could also be considered the weakness of these methods, since additional knowledge on the part of the decision-makers is required. Golany (1988), Kornbluth (1991), Thanassoulis and Dyson (1992), Golany and Roll (1994), Zhu (1996b) and Halme et al. (1999) each incorporated preferential information into the DEA models through, for example, a selection of preferred input/output targets or hypothetical DMUs. A separate set of papers reflected preferential information through limitations on the values of the weights (assurance regions or cone-ratio models), which can almost guarantee a complete DMU ranking. Such papers include Thompson et al. (1986), Dyson and Thanassoulis (1988), Charnes et al. (1990, 1989), Cook and Kress (1990a,b), Thompson et al. (1990), Wong and Beasley (1990), Cook and Johnston (1992) and Green and Doyle (1995). Cook and Kress (1990a,b, 1991, 1994) and Cook et al. (1993, 1996) argued that by imposing ratio constraints on the multipliers and replacing the infinitesimal with a lower bound thus acting as a discrimination factor, the modified DEA model can almost ensure a single efficient DMU. For example, when considering aggregation of votes whereby y rj is the number of rth placed votes received by candidate j, one can define a discrimination intensity function dðr; eþ and solve model (21). Max e X s u r y rj 6 1; j ¼ 1;...; n; u r u rþ1 dðr; eþ P 0; u r dðr; eþ P 0; u r ; e P 0; ð21þ where dðr; eþ ensures that first place votes are valued at least as highly as second place votes and so on. The ease with which this formulation can be solved depends on the form of dðr; eþ. Model (21) is linear if the difference between the ranks is linear, but this need not always be the case. However, as pointed out in Green et al. (1996), the form of dðr; eþ affects the ranking results and no longer allows DMUs to choose their weights freely. Furthermore, which mathematical function is appropriate is unclear yet important to the analysis. Two papers have discussed the use of fuzzy logic in conjunction with DEA, both to incorporate preference information from experts and to fully rank the efficient DMUs. Both Karsak (1998) and Hougaard (1999) separately developed an essentially two-stage methodology in which DEA identifies the efficiency of the DMUs concerned and the second stage utilizes fuzzy logic to rank the efficient units based on expert knowledge and the more objective DEA findings. Karsak discusses a robot selection procedure and argues that the model can be applied to other engineering problems such as selection of sites or flexible manufacturing systems. Some have gone as far as to argue that DEA should be considered another methodology within MCDM, for example Troutt (1995), Li and Reeves (1999) and Sinuany-Stern et al. (2000). Troutt (1995) developed a maximin efficiency ratio model in which a set of common weights is evaluated to distinguish between efficient DMUs as shown in model (22). Maximize u r;v i Minimize k P s u ry rk P m v ix ik

11 N. Adler et al. / European Journal of Operational Research 140 (2002) P s u ry rk P m v ix ik 6 1 8k; X s u r ¼ 1; u r ; v i P 0 8r; i: ð22þ Li and Reeves (1999) suggested utilizing multiple objectives, such as minimax and minisum efficiency in addition to the standard DEA objective function in order to increase discrimination between DMUs, as shown in the multiple-objective linear program (MOLP) of model (23). Min d k ; Min M; Min X n Xm X m d j ; v i x ij þ Xs v i x ik ¼ 1; u r y rj þ d j ¼ 0 for j ¼ 1;...; n; M d j P 0; u r ; v i ; d j 6 0 8r; i; j: ð23þ The first objective is equivalent to the standard CCR model. The second objective function requires the MOLP to minimize the maximum deviation (slack variable) and the third objective is to minimize the sum of deviations. The aim was to increase discrimination, which the second and third objectives provide without promising complete ranking, similarly to the assurance regions and cone-ratio approaches. On the other hand, this approach does not require additional preferential information as do other approaches. Joro et al. (1998) provide a review of the literature on DEA and MOLP. Sinuany-Stern et al. (2000) suggested an analytical hierarchical process (AHP/DEA) ranking model based on a two-stage process. In the first stage, a DEA model is run for every pair of DMUs, two units at a time, ignoring all others. Based on the results of the first stage, a pairwise comparison matrix is created from which a single level AHP can be applied thus providing a full scale ranking of all DMUs. Since the basic data are pre-specified (inputs and outputs of organizational units) the axiomatic utility theory problems of AHP do not exist. Indeed, in the original AHP, the pairwise comparison matrix data are based on the subjective decision-makers preferences whilst AHP/DEA builds an objective matrix. This nonsubjective approach is easier from the decisionmakers viewpoint since there is no subjective evaluation of many pairs of alternatives, another drawback of AHP. In the AHP/DEA model, the multiple criteria are taken into account through DEA while the ranking is performed by AHP, thus the model does not suffer from the limitations of either model. However, AHP can result in a matrix of many 1 s, which would result in a failure to fully rank the DMUs. However, it should be noted that certain researchers have argued that MCDM and DEA are two entirely separate approaches, which do not overlap. MCDM is generally applied to ex ante problem areas where data are not readily available, especially if referring to a discussion of future technologies, which do not yet exist. DEA, on the other hand, provides an ex post analysis of the past from which to learn. A discussion of this topic can be found in Belton and Stewart (1999). 9. Illustration For the benefit of the reader a simple example has been analyzed using the majority of techniques presented in this review. The data has been drawn from the nursing home example developed in Sexton et al. (1986), in which six DMUs are compared over four variables: staff hours per day (StHr), supplies per day (Supp), total Medicare plus Medicaid reimbursed patient days (MCPD) and total private patient days (PPPD). The raw data are presented in Table 1 and the results of the CCR, BCC and additive models are presented in Table 2.

12 260 N. Adler et al. / European Journal of Operational Research 140 (2002) Table 1 Raw data of nursing home example Inputs Outputs StHr Supp MCPD PPPD A B C D E F As can be seen in Table 2, whilst both the CCR and additive models define DMUs E and F as inefficient, the BCC model, which allows for variable returns-to-scale, only defines F as inefficient. Table 3 presents the rankings of the DMUs according to the cross-efficiency and super-efficiency models. As can be seen in Table 3, the results of the cross-efficiency models according to the aggressive and benevolent objective functions are different one from the other, although A is clearly the most efficient whilst C and F are in the worst relative positions. In the aggressive CE model, no DMU achieves an average efficiency score of one, whilst in the benevolent CE model, both units A and D do achieve an average efficient score of one. The super-efficient model ranks A as the most efficient and E and F as inefficient (since the latter matches the CCR model). Table 4 presents the results of the benchmarking and MID rankings, with the former ranking only the efficient units (defined according to the variable Table 2 DMU scores for standard three DEA models CCR BCC Additive constant returns-to-scale A 1.00 A 1.00 A 0.00 B 1.00 B 1.00 B 0.00 C 1.00 C 1.00 C 0.00 D 1.00 D 1.00 D 0.00 E 0.98 E 1.00 E F 0.87 F 0.90 F Table 3 Results of cross-efficiency and super-efficiency constant returns-to-scale ranks Cross-efficiency aggressive Cross-efficiency benevolent Super-efficiency A A 1 A B D 1 B D E C E B D C C E F F F Table 4 Benchmarking and MID ranks Benchmarking Rank A 1 E 0.99 B 2 F 0.93 E 3 C 4.5 D 4.5 MID (based on slacks from additive constant returns-to-scale model)

13 N. Adler et al. / European Journal of Operational Research 140 (2002) returns-to-scale additive model in this example) and the latter ranking the inefficient units alone. Once again, unit A achieves the top ranking in the benchmarking approach, whilst units C and D achieve the lowest ranking, amongst efficient DMUs. In the CE models, D is always ranked higher than C, and the opposite is true in the super-efficient model. Clearly, the logic behind the reason for ranking the DMUs will decide the ultimate ranking procedure chosen and consequently the results. The MID model ranks unit E higher than F, as occurred in the three basic DEA models, but gives them a higher relative efficient score. Table 5 presents the results of the statistical models developed in Section 6. All three statistical model results rank the DMUs differently, although notably F is always ranked last. Furthermore, A loses its first ranking in the DDEA model alone, possibly due to the common weights approach, since this can also be seen in the maximin efficiency ratio model with common weights presented in Table 6. This table presents the results of the MCDM models of Troutt (1995) and Li and Reeves (1999). According to the maximin common weights Table 5 Ranking according to statistical-based models DMU Rank CCA DDEA DR/DEA A B C D E F model, DMU D is ranked first, whereas the MOLP objective functions define both units A and D as relatively efficient, in a similar manner to that of the benevolent CE model. The advantage of MOLP is the evaluation of a usable set of weights per DMU within which discussion as to improvements can be developed, which is somewhat missing from the cross-efficiency approach. 10. Conclusions The field of data envelopment analysis has grown at an exponential rate (see Seiford, 1996) since the seminal papers of Farrell (1957) and Charnes et al. (1978). The original idea of evaluating after-school programs with multiple inputs and outputs has led to an enormous body of academic literature. Within this field is a sub-group of papers in which many researchers have sought to improve the differential capabilities of DEA and to fully rank both efficient, as well as inefficient, decision-making units. In this review, the DEA ranking concepts have been divided into six general areas. The first group of papers is based on a cross-efficiency matrix. By evaluating DMUs through both self- and peer pressure, one can attain a more balanced view of the decision-making units. The second group of papers is based on the super-efficiency approach, in which the efficient units can receive a score greater than one, through the unit s exclusion from the column being scored in the linear program. This proved popular and many papers sprouted from this idea, broadening its use from mere ranking to outlier detection, sensitivity analyses and scale classification. However, each unit is Table 6 Ranking according to MCDM models Maximin efficiency ratio MOLP minimax MOLP minisum D A 1 A 1 C D 1 D 1 E E E A B F B C B F F C 0.830

14 262 N. Adler et al. / European Journal of Operational Research 140 (2002) evaluated by its own weights as opposed to the cross-efficiency concept in which all units are compared using the same sets of weights. The third grouping is based on benchmarking, in which a DMU is highly ranked if it is chosen as a useful target for many other DMUs. This is of substantial use when looking to benchmark industries. The fourth group of papers developed a connection between multivariate statistical techniques and DEA. Canonical correlation analysis and discriminant analysis were each used to compute common weights, from which the set of DMUs can be ranked. In practice, non-parametric statistical tests showed a strong correlation between the final ranking and the original DEA dichotomous classification. The fifth section discussed the ranking of inefficient units. One approach, entitled a measure of inefficiency dominance, ranks the inefficient units according to their average proportional inefficiency in all inputs and outputs. In the last set of papers, which crosses multi-criteria decision-making models with DEA, some concepts used additional, preferential information in order to aid the ranking process. The additional information can be incorporated into or alongside the standard DEA results through the use of fuzzy logic, assurance regions or discrimination intensity functions. Other concepts combined the two approaches without the need for additional information such as AHP, the maximin efficiency ratio model and a multi-objective linear program. It should be noted that many papers have been written in an empirical context, utilizing the concepts discussed here. However, referring to all the published applications in this area would result in an overly long review hence they were considered beyond the scope of this review. Our aim was to review the methodologies rather than the subsequent applications, though it can be noted that certain techniques have been heavily used in spe- Table 7 Criteria for ranking the main methodologies Ranking only efficient units Ranking by new concept (not h k ) Cross-efficiency models Sexton et al. (1986) a One set of common weights Occasional problems with infeasibility Super-efficiency models Andersen and Petersen (1993) True for Hashimoto (1997) efficient Mehrabian et al. (1999) units only Sueyoshi (1999) Benchmarking Torgersen et al. (1996) Statistics-based models Friedman and Sinuany-Stern (1997) Sinuany-Stern et al. (1994) Sinuany-Stern and Friedman (1998) Friedman and Sinuany-Stern (1998) Ranking of inefficient units Bardhan et al. (1996) MCDA/DEA Troutt (1995) Li and Reeves (1999) Karsak (1998) Hougaard (1999) Sinuany-Stern et al. (2000) a The cross-efficiency models use n sets of common weights applied to all DMUs.

15 N. Adler et al. / European Journal of Operational Research 140 (2002) Table 8 DEA software packages and their ranking capabilities Super-efficiency Cross-efficiency Benchmark (simple count) Statistics DEAP a DEA-Solver-Pro b EMS c Frontier d IDEAS e On Front f Warwick g a b See c EMS is free to the academic community and available from Holger Scheel at d See e See f See g Contact e.thanassoulis@aston.ac.uk. cific areas. Cross-efficiency has been applied in many areas of manufacturing, including engineering design, flexible manufacturing systems, industrial robotics and business process re-engineering etc. It has also been used heavily in project and R&D portfolio selection. Super-efficiency has been applied in a wide range of papers from financial institutions and industry to public regulation, education and healthcare. Benchmarking has been used extensively in the field of utilities, industry and agricultural productivity. The statistical techniques have been applied to universities and industry and MCDA/DEA to agriculture and oil. Clearly, these methodologies have a wideranging applicability in many areas of both the public and private sectors. To aid in choosing any of these concepts we have summarized the six major ranking methods according to various criteria in Table 7. In addition, the commercial DEA software manufacturers were contacted and asked to specify the ranking techniques available in their programs. Seven software programs were known to the authors and the results are as shown in Table 8. Evidently, the super-efficient and benchmarking approaches are the most widely spread. Finally, many mathematical and statistical techniques have been presented here, all with the basic aim of increasing the discriminating power of data envelopment analysis and ranking the decision-making units. However, whilst each technique may be useful in a specific area, no one methodology can be prescribed here as the panacea of all ills. It remains to be seen whether the ultimate DEA model can be developed to solve all problems and which will consequently be easy to solve by practitioners in the field and academics alike. The trend, however, would appear to be in this direction, as evidenced by Cooper et al. (1999) in which many ideas have been incorporated into one model. The model includes the capability of dealing with imprecise data as well as assurance region and cone-ratio concepts. Another potentially fruitful concept may be to invoke several ranking procedures and then to compute an average or median rank based on all the models employed, as suggested in Friedman and Sinuany-Stern (1998). This concept, used frequently by economists, statisticians and forecasters, may help to achieve a complete ranking whilst avoiding some of the pitfalls of each individual approach. References Adler, N., Golany, B., Evaluation of deregulated airline networks using data envelopment analysis combined with principal component analysis with an application to Western Europe. European Journal of Operational Research 132 (2), Andersen, P., Petersen, N.C., A procedure for ranking efficient units in data envelopment analysis. Management Science 39 (10),