ARGUS - A NEW MULTIPLE CRITERIA METHOD BASED ON THE GENERAL IDEA OF OUTRANKING 1. Introduction W.S.M. DE KEYSER and P.H.M. PEETERS Vrije Universiteit Brussel Centrum voor Statistiek en Operationeel Onderzoek Pleinlaan 2 B-1050 Brussel Belgium The field of Multiple Criteria Decision Aid (MCDA) attracts the last decades more and more researchers and practitioners. A lot of MCDA methods have been developed to help practitioners with their specific problems and needs. Although several multiple criteria methods do exist, we worked out another one. The motivation for this research and the need for another multiple criteria method is discussed in section 3. Before giving the reason of existence for ARGUS, we first want to elucidate in section 2 some concepts and ideas that are used in the development of our multiple criteria method. The modeling of the preference structure for the criteria taking their measurement scale into account is described in section 4. In section 5 the way of combining the preference structure and the weights is discussed while in section 6 the aspect of discordance is treated. In section 7, the ARGUS outranking method is described and a numerical example is given in section 8. Some conclusions are made in section 9. 2. Concepts and ideas The definition of scale types -nominal, ordinal, interval and ratio scale- can be found in Stevens (1946) or, together with a fifth scale -absolute scale-, in Roberts (1979). In this text, we will distinguish four scale types: nominal, ordinal, interval and ratio scale. The following concepts and ideas will be implemented in the ARGUS multiple criteria method: 2.1. INTENSITY OF PREFERENCE IS MEASURED ON AN ORDINAL SCALE The modeling of the preference structure of a decision maker can, grosso mode, be 263 M. Paruccini (ed.), Applying Multiple Criteria Aid for Decision to Environmental Management, 263-278. 1994 ECSC, EEC, EAEC, Brussels and Luxembourg, Printed in the Netherlands.
264 done in two ways. The first way is to ask the decision maker to give the intensity of his preference between two given alternatives on a certain criterion. This intensity will basically depend on the difference between the values of the two alternatives on the criterion. In e.g. the PROMETHEE methods (Brans et al.,1985 and 1986) the intensity of the preference is modeled through a preference function which returns for all differences, between values of two alternatives on a certain criterion, a value between 0 and 1 as intensity of the preference. The intensity of preference way is criticised (Roy and Bouyssou, 1993) because the intensity of preference can not be limited (e.g. by the value 1 in the PROMETHEE methods): the larger the difference, the larger the intensity of preference. The second way is to use credibility. For each couple of alternatives (a,b) one determines on each criterion j the concordance index cj(a,b) (being a value between 0 and 1) which reflects in which way one can affirme that "alternative a is at least as good as alternative b". Besides the concondance indexes, discordance indexes are determined and all are aggregated into a degree of credibility. In the credibility way there is no problem with stating that the concordance index can not be larger that 1. E.g. ELECTRE III is using this way (Skalka et al., 1992). We believe that the intensity of preference way is an interesting way of modeling the preference structure of the decision maker and that the intensity of preference can be modeled without the above mentioned criticism by means of an ordinal variable. Consider the following formulation of a multiple criteria problem: MAX{ f 1(x), f 2 (x),..., f k (x) x A}, (1) where A is a finite set of possible alternatives. Consider the evaluation of two alternatives a and b on the criterion fi(x): fi(a) and fi(b) and suppose fi(a) > fi(b). Based on fi(a) and fi(b), the decision maker can express his intensity of preference, for alternative a compared to alternative b, by selecting one of the following qualitative values: indifferent (= no difference) small preference moderate preference strong preference very strong preference These are values of a variable measured on an ordinal scale: ordening in magnitude is possible but there are no equal intervals (Allen and Yen, 1979). To illustrate this (Roberts, 1979, p 65), one can assign a number 1 to indifferent, 2 to small preference, 3 to moderate preference, 4 to strong preference and 5 to very strong preference. One could also assign any numbers that preserve the order (Roberts, 1979; Torgerson, 1958; Stevens, 1946) e.g. 10, 20, 30, 40, 50 or 1, 2, 3, 5, 8 or 2, 4, 8, 16, 32. Because the representation on the intensity of preference by numbers is not unique
265 and does not add anything but possible confusion with an interval or ratio scale, we prefer to use in this paper the qualitative values for representing the intensity of preference on an ordinal scale. This does not change in any way the character of the scale of the variable. The modeling of the intensity of preference by means of an ordinal variable can be illustrated by figure 1 for evaluations on an ordinal scale and by figure 2 for evaluations on a ratio scale. Figure 1: intensity of preference for evaluations on an ordinal scale Figure 2: intensity of preference for evaluations on a ratio scale We also like to remark that the use of an ordinal scale to model preference has been suggested before (Roberts, 1979).
266 2.2. WEIGHTS ARE MEASURED ON AN ORDINAL SCALE The weight of a criterion in a multiple criteria method can indicate the importance that the decision maker attaches to that criterion and can be used as a scaling constant. Since we use an ordinal variable to represent the intensity of the preference of the decision maker between two alternatives, and since an ordinal variable has no unit of measurement, the scaling aspect of weights can be omitted here. Most multiple criteria methods require numerical values (on a ratio scale) for the weights. This popular habit of assigning numerical values as weights to the criteria can only be justified under the assumption that the importance the decision maker attaches to a criterion can be measured on a ratio scale; so the decision maker must have some kind of a unit of measurement and must be able to measure correctly the importance of each criterion according to that unit. Our experiences with users and students learned us that expressing weights on an ordinal scale is easier than expressing the weights on a ratio scale. This is one of the reasons why we will use an ordinal scale to measure the weights. The following ordinal scale reflecting the importance of a criterion, will be used in this paper: not important little important moderately important very important extremely important. The decision maker must indicate for each criterion, by selecting a value from the above ordinal scale, how important that criterion is for him. In other words, the decision maker must rank and classify the criteria. Other multiple criteria methods such as e.g. the regime method (Janssen et al., 1990) are also considering the weights as values on an ordinal scale. 2.3. THE GENERAL IDEA OF OUTRANKING The general idea of outranking is due to B. Roy and can be formulates as (Vincke, 1992 p 58): 'an outranking relation is a binary relation S defined in A such that asb if, given what is known about the decision maker's preferences and given the quality of the valuations of the actions and the nature of the problem, there are enough arguments to decide that a is at least as good as b, while there is no essential reason to refute that statement'. A more precise mathematical definition is given in (Roy, 1985). The multiple criteria method presented in this paper will be based on the general idea of outranking. 2.4. OTHER CONCEPTS The following concept and ideas must in our opinion be taking in consideration when
constructing a multiple criteria method based on the general idea of outranking: 267 1. Discordance is considered as a realistic concept in multiple criteria decision aid but not all outranking methods (e.g. the PROMETHEE methods) are taking it into account. ARGUS will consider discordance. 2. Several outranking methods such as e.g. ELECTRE III, PROMETHEE II and MELCHIOR (Leclercq, 1984), end with a complete pre-order on the alternatives and present in this way one 'best' alternative to the decision maker. From the point of view that an outranking method builds an outranked relation which is an enrichment of the 'poor' dominance relationship, is it not necessarily true that the enriched relationship is so 'rich' that it can always present one solution as being better than all the other. ARGUS will not necessarily end with one 'best' alternative, but with a set of good alternatives. 3. Outranking methods such as e.g. PROMTHEE II, which are exploring the constructed outranking relation by comparing an alternative with all other alternatives have the drawback that the presence or abscence of even an alternative which is dominated by several other alternatives, can put the end result -the pre-order- up side down (De Keyser and Peeters, 1993). No all to all comparison during the exploration of the graph will be used in ARGUS. 3. Motivation Multiple criteria problems, especially in the social sciences, can have several criteria with evaluations on an ordinal scale. E.g. multiple criteria problems in the field of urban planning, which we are handling at the moment in cooperation with the Belgium firm Urban Geographic Services, have such criteria like e.g. 'level of available social facilities', 'ground destination' and 'landscape'. Also multiple criteria problems in e.g. the field of ecology and environmental management can have criteria with evaluations on an ordinal scale. Most decision makers, who are using outranking methods such as ELECTRE III and PROMETHEE, are, when handling such a criterion, assigning numbers to the different qualitative values -evaluations- on the criterion. This is a valid way of recoding qualitative values into quantitative values as long as the order is respected (Roberts, 1979). Since the outranking methods are using differences between those numbers to obtain a value that is an element of the interval [0,1], the difference between two values on an ordinal scale gives rise to a value on a ratio scale. It is clear that this way of working is not correct since differences between values on an ordinal scale have no meaning and since it is not possible to obtain a value on ratio scale starting from two values on an ordinal scale without adding, in an uncontrollable way, information. This pitfall of using the properties of the coding of the evaluations on an ordinal
268 scale of the criterion instead of the properties of the evaluations on an ordinal scale of the criterion itself will be avoided in ARGUS by handling all evaluations on the criteria as evaluations on an ordinal scale and by the fact that two values on an ordinal scale will give rise to an ordinal value as intensity of preference. It is clear that ARGUS sometimes will loose some information (on those criteria where the evaluations are on interval or ratio scale) but it will not add uncontrollable information as e.g. ELECTRE III and PROMETHEE do when they are handling criteria with evaluations on an ordinal scale in the above described way. Multiple criteria problems, with a finit set of alternatives, where the decision maker can not express or has difficulties in expressing his preference structure (intensity of preference or credibility) or the weights (importance of the criteria) on a ratio scale can be handled by ARGUS. Outranking -according to (Vincke, 1992, p 58)- methods such as QUALIFLEX (Paelinck, 1978) and ORESTE (Roubens, 1981) are implementing some of the above mentioned concepts and ideas, but we could not find an existing multiple criteria method based on the general idea of outranking which had implemented all of them. 4. Ordinal preference structure The way of obtaining the required information of the decision maker so that his preference structure can be modeled, depends on the scale of measurement of the criterion. 4.1. PREFERENCE MODELING OF A CRITERION ON AN ORDINAL SCALE Consider e.g. the following possible values of an ordinal criterion: very poor, poor, average, good, very good. To model the preference structure of the decision maker on this criterion, the decision maker must indicate his preference if fi(a) = poor and fi(b) = very poor fi(a) = average and fi(b) = very poor fi(a) = good and fi(b) = very poor fi(a) = very good and fi(b) = very poor fi(a) = average and fi(b) = poor fi(a) = good and fi(b) = poor fi(a) = very good and fi(b) = poor fi(a) = good and fi(b) = average fi(a) = very good and fi(b) = average fi(a) = very good and fi(b) = good In fact the decision maker must fill in the lower triangle of the following Preference Matrix:
269 Table 1: Preference Matrix for a criterion with evaluations on an ordinal scale f i(b) f i(a) very poor poor average good very good very poor poor average good very good indifferent indifferent indifferent indifferent indifferent The number of rows and columns of the preference matrix depends on the number of different values the ordinal criterion can have. Remarks: 1. The following can be said about the values filled in in the lower triangle of a preference matrix: - The value of a cell will be the same or a higher value as the cell above it (= same column). - The value in a cell will be the same or higher value as the cell to its right (= same row). 2. A preference matrix can also be used for a criterion with evaluations on a nominal scale. Objectively the evaluations of the criterion are on a nominal scale, but subjectively -for the decision maker-, the evaluations of the criterion are on an ordinal scale. 4.2. PREFERENCE MODELING OF A CRITERION ON AN INTERVAL OR RATIO SCALE From interval scale on, the difference: d = fi(a) - fi(b) (2) has a meaning. The preference of the decision maker on a criterion with an interval scale will depend on d while his preference on a criterion with a ratio scale will depend either on d only or on d, fi(a) and fi(b). If his preference depends on d only, this means that only the absolute difference d determines his preference. If the decision maker takes besides d, also fi(a) and fi(b) into account, then his preference can be different for the same absolute difference d but different values of fi(a) and fi(b). One possible way of taking this into account is by using e.g. the relative difference: = (fi(a) - fi(b)) / fi(b) where fi(a) fi(b) (3) Other more complex and more general ways are possible. In the method that is described further in this paper, no other way for taking the order of magnitude into account than the one of the relative differences is used, even though other ways can easily be incorporated.
270 The preference structure of the decision maker for a criterion on a ratio scale can now be modeled by determining for which (absolute or relative) difference the decision maker is indifferent, for which (absolute or relative) difference he has a small preference, for which (absolute or relative) difference he has a moderate preference, for which (absolute or relative) difference he has a strong preference and for which (absolute or relative) difference he has a very strong preference. In other words, one of the two columns of the following table must be filled in: Table 2: Preference Matrix for a criterion with evaluations on a ratio scale f i(a) f i(b) d = f i(a) - f i(b) = (f i(a)-f i(b)) / f i(b) indifferent 0 d < d 1 0 % < 1% small preference d 1 d < d 2 1% < 2% moderate preference d 2 d < d 3 2% < 3% strong preference d 3 d < d 4 3% < 4% very strong preference d 4 d 4% 5. Combined preference with 'weights' When the preference structure of the decision maker for each criterion is known as well as the importance of each criterion, the comparison of two alternatives a and b on the criteria fi(x) (i = 1,..., k) leads to a two-dimensional table (see table 3). In a cell of the table stands the number of criteria of a certain importance for which a certain preference between alternative a and alternative b occurs. In order to get one overall appreciation of the comparison between alternative a and alternative b, the decision maker must rank all cells of the above table where fi(a) > fi(b). Since there is already a certain ranking present in the table, the following ranking in eight classes is proposed to the decision maker: 1. very strong - extremely imp. 2. very strong - very imp. /strong - extremely imp. 3. very strong - moderately imp. /strong - very imp. /moderate - extremely imp. 4. very strong - little imp. /strong - moderately imp. /moderate - very imp. /small - extremely imp. 5. very strong - not imp. /strong - little imp. /moderate - moderately imp. /small - very imp. 6. strong - not imp. /moderate - little imp. /small - moderately imp. 7. moderate - not imp. /small - little imp. 8. small - not imp. The decision maker can alter this ranking (by moving a cell from one class to another, by considering more classes, by considering less classes) until it matches his personal conception.
Table 3: Preference - importance table 271 criteria preference not important little important moderate. important very important extremely important very strong f 11 f 12 f 13 f 14 f 15 f i(a) > f i(b) strong f 21 f 22 f 23 f 24 f 25 moderate f 31 f 32 f 33 f 34 f 35 small f 41 f 42 f 43 f 44 f 45 f i(a) = f i(b) no f 51 f 52 f 53 f 54 f 55 small f 61 f 62 f 63 f 64 f 65 f i(a) < f i(b) moderate f 71 f 72 f 73 f 74 f 75 Note that 9 i=1 strong f 81 f 82 f 83 f 84 f 85 very strong f 91 f 92 f 93 f 94 f 95 5 j=1 f ij = k Through this ranking, a one dimensional ordinal variable, which will be called the combined preference with 'weights' (CPW) is created. In fact there is a combined preference with 'weights' variable where fi(a) > fi(b) and there is a combined preference with 'weight' variable where fi(a) < fi(b). Based on those two variables a statement indicating the 'outranking' or indifference, or incomparability between two alternatives (('outranking', Indifference, incomparable) - relation), must be made. By using the above ranking, the two combined preferences with 'weights' variables look like: Table 4: Combined preferences with 'weights' variables CPW-(f i(a) > f i(b)) CPW-(f i(a) < f i(b)) 1 g 1 = f 15 h 1 = f 95 2 g 2 = f 14 + f 25 h 2 = f 85 + f 94 3 g 3 = f 13 + f 24 + f 35 h 3 = f 75 + f 84 + f 93 4 g 4 = f 12 + f 23 + f 34 + f 45 h 4 = f 65 + f 74 + f 83 + f 92 5 g 5 = f 11 + f 22 + f 33 + f 44 h 5 = f 64 + f 73 + f 82 + f 91 6 g 6 = f 21 + f 32 + f 43 h 6 = f 63 + f 72 + f 81 7 g 7 = f 31 + f 42 h 7 = f 62 + f 71 8 g 8 = f 41 h 8 = f 61
272 Since there are no units of measurement, the above table can only lead to: i i If i : g j = h j i =1,..., 8 then: a I b; j=1 i j=1 i if i : g _ j h j i= 1,..., 8 then: a S b; j=1 i j=1 i if i : g _ j h j i= 1,..., 8 then: b S a; j=1 j=1 in all other cases: a R b 6. Discordance According to the basic idea of outranking, an alternative a outranks alternative b means that alternative a is better on most of the criteria than alternative b. If alternative b is however much better than alternative a on one (or more) criteria than there can be discordance between alternative a and alternative b and alternative a will not outrank alternative b. The discordance concept that will be used in ARGUS is simular to that of e.g. the ELECTRE I and the MELCHIOR method. More precisely, the decision maker must indicate for each criterion when there is discordance between two alternatives on that particular criterion. For a criterion with an ordinal scale he can indicate in the upper triangle of the preference matrix when discordance occurs. For a criterion with an interval or ratio scale, he must indicate from which (absolute or relative) difference, between the evaluation on two alternatives on that criterion, on there is discordance. 7. ARGUS, a new multiple criteria method based on the general idea of outranking ARGUS stands for Achieving Respect for Grades by Using ordinal Scales only. 7.1. BASIC POINTS OF VIEW: - The decision maker's preference structure is modeled by using an ordinal measurement for the intensity of preference. - The decision maker's preference structure of the criteria is modeled by using an ordinal measurement. - Values on an ordinal scale cannot be transformed into values on an interval or ratio scale. Therefore combining the decision maker's preference structure of alternatives on each of the criteria and his preference structure of the criteria must result in an ordinal variable.
7.2. OUTLINES OF THE ARGUS MULTIPLE CRITERIA METHOD: 273 The following five steps in solving the multiple criteria problem MAX{ f 1(x), f 2 (x),..., f k (x) x A} 1 must be made: a. Data collection: - Determine the set of plausible alternatives A. - Determine the criteria f1(x),..., fk(x). - Determine the evaluations of each alternative on each criterion. Note that the evaluations can be quantitative as well as qualitative. b. Preference modeling of alternatives on a criterion: Determine for each criterion the scale of measurement: - If the scale is ordinal: - order the possible values from worst to best - complete the preference-matrix - indicate in the preference matrix the discordances - if the scale is interval: - indicate if the criterion must be MIN or MAX - construct the ordinal preference structure based on the absolute differences - indicate for which absolute difference there is discordance - if the scale is ratio: - indicate if the criterion must be MIN or MAX - construct the ordinal preference structure based on the (absolute or relative) differences - indicate for which (absolute or relative) difference there is discordance c. Preference modeling of the criteria: The decision maker ranks and classifies the criteria. d. Combined preference structure: The decision maker must construct, according to his personal conception, the combined preference with 'weights'. e. Processing: - Delete dominated alternatives (not compulsory) - Make the two by two comparisons of the alternatives on the criteria. - Determine through the two combined preference with 'weights'-variables whether there is a S-, I- or R-relationship between each couple of alternatives. - Draw a graph with the S-relationships. - Determine the kernel(s) of the graph. This way ARGUS ends with the kernel (or with the union of the kernels if the
274 kernel is not unique) of good alternatives based on the obtained relations between the alternatives. 8. Numerical example Consider the following fictive problem of choosing a repository site based on the multibarrier system to store LLW (Low-Level radioactive Waste). a. data collection There are five possible alternative: Site 1, Site 2, Site 3, Site 4, Site 5, Site 6. Four criteria are taken in consideration: Cost in 10 6 $ (ratio scale), Radioactive waste that might escape (ordinal scale with as values: sufficient for healt hazard, insufficient for healt hazard, neglectable, none), Resistance Population (ordinal scale with as values: very high, high, moderate, low, none) and Storage Capacity in 10 6 m 3 (ratio scale); Table 5: Evaluations of each alternative on each criterion Criteria alternative Cost Radioactive waste that might escape Resistance Population Storage Capacity Site 1 9.7 insufficient none 0.30 Site 2 17.5 none low 0.37 Site 3 13.6 insufficient very 0.33 Site 4 12.0 neglectable very 0.35 Site 5 10.0 insufficient low 0.31 Site 6 14.2 none moderate 0.34 b. Preference modeling of alternatives on a criterion Table 6: Criterion 1 (C1): Cost in 10 6 $ (ratio scale) to be MIN Preference (b above a) d = f i(a) - f i(b) indifferent 0 d < 1.6 small 1.6 d < 3.2 moderate 3.2 d < 4.8 strong 4.8 d < 6.5 very strong 6.5 d < Discordance d < -8.1
275 Table 7: Criterion 2 (C2): Radioactive waste that might escape (ordinal scale) sufficient insufficient neglectable none sufficient indifferent insufficient moderate indifferent neglectable strong strong indifferent none very strong strong moderate indifferent Table 8: Criterion 3 (C3): Resistance Population (ordinal scale) very high high moderate low none very high indifferent high indifferent indifferent moderate strong strong indifferent low very strong strong strong indifferent none very strong strong strong indifferent indifferent Table 9: Criterion 4 (C4): Storage Capacity in 10 6 m 3 (ratio scale) Preference (a above b) d = f i(a) - f i(b) indifferent 0.0 d < 0.0 small 0.0 d < 0.2 moderate 0.2 d < 0.5 strong 0.5 d < 0.1 very strong 0.1 d < discordance d < -0.12 c. Preference modeling of the criteria The following weights were attached to the four criteria:
276 Table 10: Weights Weight not important little important moderately important very important extremely important C4 C1, C3 C2 d. Combined preference structure The following eight classes were constructed: 1. very strong - extremely imp. 2. very strong - very imp. /strong - extremely imp. 3. very strong - moderately imp. /strong - very imp. /moderate - extremely imp. 4. very strong - little imp. /strong - moderately imp. /moderate - very imp. /small - extremely imp. 5. very strong - not imp. /strong - little imp. /moderate - moderately imp. /small - very imp. 6. strong - not imp. /moderate - little imp. /small - moderately imp. 7. moderate - not imp. /small - little imp. 8. small - not imp. e. Processing - Site 4 is dominating Site 3, so Site 3 is deleted from the set of alternatives. - Two by two comparison. Table 11 gives an example of a two by two comparison. Table 12 gives an overview of the obtained relationships. Table 11: The two by two comparison between Site 1 and Site 6 CPW-(f i(site 1) > f i(site 6)) CPW-(f i(site 1) < f i(site 6)) 1 0 0 2 0 1 3 1 0 4 1 0 5 0 1 6 0 0 7 0 0 8 0 0
277 Table 12: R-,I-,S-relations incomparable Site 1 R Site 6; Site 5 R Site 6 Indifference Site 4 I Site 5 'outranking' Site 2 S Site 1; Site 2 S Site 4; Site 2 S Site 5; Site 2 S Site 6; Site 4 S Site 1; Site 5 S Site 1; Site 6 S Site 4 - The graph: - The kernel of the graph = {Site 2} Figure 3: Graph 9. Conclusion ARGUS is a new multiple criteria method based on the general idea of outranking. It avoids the pitfall of treating a criterion with evaluations on an ordinal scale as a criterion with evaluations on an interval or ratio scale by forcing the decision maker to indicate the scale of measurement for each criterion. ARGUS offers the decision maker the opportunity of taking, for criteria with evaluations on a ratio scale, the order of magnitude of the evaluations into account when modeling his preference structure. The proposed method considers discordance, assumes that the importance of the criteria must be measured on an ordinal scale and does not always end with only one good alternative but with a set of good alternatives. ARGUS uses, as distinct from e.g. the ELECTRE and PROMETHEE methods, only ordinal allowed operations in handling multiple criteria problems. This makes ARGUS especially suited for handling multiple criteria problems where several
278 criteria have evaluations on an ordinal scale. We are convinced that in the near future ARGUS will prove its usefulness and its applicability, not only in the field of urban planning but also in other fields such as environmental management, facility location,... 10. Bibliography Allen, M. and Yen, W. (1979), Introduction to Measurement Theory, Brooks/Cole Publishing Company Brans, J.P., Mareschal, B. and Vincke, Ph. (1986), 'How to select and how to rank projects: the PROMETHEE method', European Journal of Operational Research, 24, 647-656 Brans, J.P. and Vincke, Ph. (1985), 'A preference ranking organization method', Management Science, 31/6, 647-656 De Keyser, W. and Peeters, P. (1993), 'Some critical remarks on the PROMETHEE methods and on the software PROMCALC', Internal Report CSOOTW/260, Vrije Universiteit Brussel Janssen, R., Nijkamp, P. and Rietvelt, P. (1990), 'Qualitative multicriteria methods in the Netherlands', in Readings in MCDA, Bana e Costa, C. (ed.), Springer Verlag Leclercq, J.P. (1984), 'Propositions d'extension de la notion de dominance en présence de relations d'ordre sur les pseudo-critères: la methode MELCHIOR', Belgisch Tijdschrift voor Statistiek, Informatica en Operationeel Onderzoek, 24/1, 32-46 Paelinck, J.H. (1978), 'QUALIFLEX: A flexible multiple-criteria method', Economic letters, 1, 193-197 Roberts, F.S. (1979), Measurement Theory with Applications to Decisionmaking, Utility, and the Social Sciences, Addison-Wesley Roubens, M. (1981) 'Preference relations on actions and criteria in multiple criteria decision making' European Journal of Operational Research, 10, 51-55 Roy, B. (1968), 'Classement et choix en présence de points de vue multiples (la méthode Electre)', Revue Française d'informatique et de Recherche Opérationelle, 8, 57-75 Roy, B. (1985), Méthodologie Multicritère d'aide à la Décision, Economica, Paris Roy, B. and Bouyssou, D. (1993), Aide Multicritère à la Décision: Méthodes et Cas, Economica, Paris Skalka, J.M., Bouyssou, D. and Vallée, D. (1992), 'ELECTRE III et IV Aspects méthodologiques et guide d'utilisation',lamsade Internal Report n 25 (4 th edition) Stevens, S. (1946),'On the theory of scales of measurement', Science, 103 Torgerson, W.S. (1958), Theory and Methods of Scaling, John Wiley & Sons Vincke, Ph. (1992), Multicriteria Decision - Aid, John Wiley & Sons