Applications of Mixed Pairwise Comparisons

Transcription

1 414 Int'l Conf. Artificial Intelligence ICAI'15 Applications of Mixed Pairwise Comparisons Abeer Mirdad and Ryszard Janicki Department of Computing and Software, McMaster University Hamilton, Ontario, Canada, L8S 4K1 {mirdadar, Abstract Mixed pairwise comparisons is a systematic composition of quantitative and qualitative pairwise comparisons proposed in [9], [11] based on the use of non-linear scale proposed in [10]. We will show how this method can improve accuracy of weights assignment for attributes used for assessment of Healthcare in Canada, Quality In Use of Software, Smart Energy Grids and Medical Devices Managements Systems. Keywords-weights assignment, pairwise comparisons, consistency, quality assessment, subjective judgments I. INTRODUCTION Pairwise comparisons are a very popular method of assigning weights when classification is mainly based on subjective judgments [19]. The pairwise comparisons method is based on the observation that it is much easier to rank the importance of two objects than it is to rank the importance of several objects. This very old idea goes back to Ramon Llull in the end of XIII century. Its modern quantitative version is due to papers by Marquis de Condorcet (1785), Fechner (1860), Thorstone (1927) and Saaty (1977). The last two papers provided mathematical foundations that are used today (see [10], [13], [19] for detailed references). Quantitative pairwise comparisons use numbers to describe the preference relationship between objects, while qualitative pairwise comparisons use abstract relations (as slightly in favour etc.) for description of this relationship. The model used in this paper originates from [8], [12]. In reality, most initial subjective judgments are always qualitative and transition from qualitative to quantitative is the weakest point of this technique [10], which is usually based on experience and domain knowledge of judges. A systematic composition of qualitative and quantitative pairwise comparisons was proposed in [9], [11]. This approach is based on the results of [10] where both approaches were compared and their mutual relationship was analyzed. In this paper we apply the technique of [9], [11] for finding weights assignments used in quality of Healthcare in Canada, in use quality of software, attributes of smart energy grid and quality system in medical devices. II. QUANTITATIVE PAIRWISE COMPARISONS Let E 1,...,E n be a finite set of entities (alternatives, attributes, etc.) to be judged and/or ranked. The relationship between entities E i and E j is represented by a positive number a ij. We assume a ij > 0 and a 1 ij, for i, j = 1,...,n (which implies a ii = 1 for all i). If a ij > 1 then E i is more preferred than E j and a ij is a measure of this relationship (the bigger a ij the bigger the difference), if 1 then E i and E j are indifferent. Intuitively a ij is interpreted as E j is a ij times preferred (more important, etc.) than E j. The matrix of such relative comparison coefficients: A =[a ij ] n n is called a pairwise comparison matrix. Since the entities E 1,...,E n are not random, on contrary, they are usually carefully chosen and interrelated, the values of a ij are not random, they should be somehow consistent. A pairwise comparison matrix A =[a ij ] n n is consistent [18] if and only if a ij a jk = a ik (1) for i, j,k = 1,...,n. It is known ([18]) that a pairwise comparison matrix A is consistent if and only if there exists positive numbers w 1,...,w n such that w i w j,i, j = 1,...,n. The values w i are unique up to a multiplicative constant. They are often called weights and interpreted as a measure of importance. The summation of w i often scaled to w w n = 1 or (100%). For a consistent pairwise comparison matrix A, the values w i create a ranking (i.e. a weak order): E i < E j w i < w j and E i E j w i = w j. In practice, the values of a ij rarely consistent, so some measurements of inconsistency and the ways to lower it, are needed. In this paper we will use distance based consistency index [14] which is defined as follows: cm A = max (i, j,k) ( min ( 1 a ij a ik a, 1 a )) ika kj kj a ij In this case the most inconsistent triad, a ij,a ik,a kj, is localized, which helps a lot in the process of inconsistency reduction. Other indexes do not have this useful property [2], [18]. Acceptable levels of inconsistency depend on particular interpretation of E i, however for the index cm A given by formula 1, the value 0.3 has some justification [2], [14]. Lowering a distance based consistency index cm A is rather straightforward. Since the biggest troublemakers are localized, we can improve consistency step by step, by small changes of values of the triple that results in the maximal inconsistency index. It was proved in [13] that this process converges. (2)

2 Int'l Conf. Artificial Intelligence ICAI' TABLE I: Relationship between additive, multiplicative and relational scales. Formulas 1 b ji and = a ij a ij +1 are used to calculate the relationships between Columns I and III and Columns II and V. If no other data is available, default values are recommended. quantitative scales [2], [10], [14] qualitative (relational) scale [8], [10] additive scale multiplicative scales relation definition of intensity of [10] from [10] a ij defaults from symbols or importance range of range of a ij derived from [2], [14] for R ij (E i vs E j ) range defaults range defaults Column II defaults E i E j indifferent/equal/unknown E i E j slightly in favour/weak importance E i E j in favour/moderate importance E i > E j strongly better/demonstrated imp E i E j extremely better/absolute imp. Column I Col. II Col. III Col. IV Column V Col. VI Col. VII Column VIII Since usually we are not interested in lowering inconsistency index to zero, but rather to same acceptable level, we need a method for deriving a suitable value w i from an inconsistent, but with acceptable level of inconsistency, matrix A. In this paper we will use a method proposed in [1] and calculate weights as the geometric means of columns (or equivalently, rows) of the matrix A, i.e. for i = 1,...,n, n w i = n a ij (3) j=1 When applying pairwise comparisons to various problems (see for example [11]) we have noticed that experts often felt much more comfortable and more confident when they were asked to divide 100 quality points between entities E i and E j than to provide multiplicative relationship, i.e. ratio a ij. Dividing of 100 between E i and E j means that we are replacing the multiplicative relationship a ij a ji = 1, with the additive relationship +b ji = 1. Since the analysis of inconsistency requires a ij a ji = 1 relationship, we have to translate into appropriate a ij. The following simple transformation was proposed and analyzed in [9], for all i, j = 1,...,n: = (4) 1 b ji b ji We may now analyze and reduce inconsistency by using the formula from equation 2. III. QUALITATIVE PAIRWISE COMPARISONS In qualitative model [8], [12], numerical values a ij or are replaced by the binary relations,,,>, and their inverses,,<,. The relations are interpreted as: a b: a and b are indifferent, a b: slightly in favor of b, a b: in favor of b, a < b: b is strongly better, a b: b is extremely better. The number of relations has been limited to five because of the known restrictions of human mind when it comes to subjective judgments [3], [15]. The relations,,, >,,,, <, are disjoint and cover the all cases. The relation is symmetric and includes identity. The set of relations RS = {,,,<, } is called a ranking system if the relations,, <, < and < are partial orders, i.e. irreflexive and transitive relations [5]. Quite often it is also required the relation to be an equivalence relation (c.f. [8], [12]). The consistency of RS is defined by a set of axioms it must satisfy. Intuitively consistency means that the relationships E i vs E j and E j vs E k influence the relationship E i vs E k (as a ij a jk = a ik for quantitative case). The number of axioms is substantial [12] as all combinations of all relational compositions must be taken care of, however the idea on which all those axioms are constructed is very simple, namely: composition of relations should be relatively continuous and must not change preferences in a drastic way. For example the Axiom 2.1 of [12] looks as follows: (a b b c) (a b b c) = (a c a c a c). There are two algorithms that start with arbitrary ranking data and provide consistent ranking systems RS. IV. MIXED MODEL In reality each subjective judgment starts with qualitative assessment and quantitative estimation of the assessment is often the most influential part of the entire process. Linear scales (wrt. a ij ) transforming qualitative assessments into numerical values have been proposed and discussed in [2], [14], [18], [19]. They all have only intuitive, heuristic and experimental justifications. The scale from [2], [14] is presented in Column VI of It is a scaled down initial ten points scale of [18]. According to [3], [15], the length of the scale should be between four [3] and seven alternatives [15]. In [10] a new non-linear (wrt. a ij ) five point scale has been proposed. It was derived from comparing quantitative consistency (i.e. a ij a jk = a ik ) with qualitative consistency given by axioms in [12], and the assumption that: transformation in either way should preserve appropriate consistencies.

3 416 Int'l Conf. Artificial Intelligence ICAI'15 The linear (wrt. ) and non-linear (wrt. a ij ) scales from [10] are presented in Columns I V of Composing the results of [2], [14], [8], [10], [12] resulted in Mixed Model, first suggested in [11] and formally proposed in [9], which is explained by the following procedure. Procedure 1 (Mixed Model). 1) Experts provide qualitative judgments using relations from Columns VII and VIII of 2) Experts transform their qualitative judgments into quantitative judgments either using additive scale (Columns I and II), or multiplicative scale (Columns III, IV and V) from Table I, or both scales. It is recommended to use the additive scale for the relations and. Sometimes the multiplicative scale (from Columns III and IV) works better for the relation. 3) If the additive scale has been used, the values are transformed into a ij using the equation (4). 4) A standard procedure for distance based inconsistency reduction is used (equations (2)). 5) The outcome, which is a pairwise comparison matrix with acceptable inconsistency, is transformed back into qualitative matrix using ranges from Column III of Table I, i.e. a ij s are transformed into appropriate R ik from Column VII. 6) In some cases the outcome is also transformed into additive pairwise comparison matrix [ ] n n] by using a ij a ij +1. the formula = 7) All final qualitative and quantitative tables are sent back to experts for final adjustments and potential changes. 8) The whole process is repeated as many times as necessary. 9) After the results are accepted, the weights are calculated as the geometric means of columns of the final matrix A =[a ij ] n n, i.e. w i = n n j=1 a ij, for i = 1,...,n. 10) If ranking is required, it is derived from the weights in a standard manner. A version of the above procedure will be used in all applications that follow. V. WEIGHTS ASSIGNMENT PROCESS Procedure 1 is fairly general, it many different concrete instantiations. The following version of Procedure 1 was used in all four applications discussed in this paper. Procedure 2. 1) A small group of experts were given a number of attributes and they were asked to provide qualitative values using relations from 2) Then the same experts were asked to provide quantitative evaluation using the additive scale and ranges that provided in Column I of In all cases the experts decided to use used default values from Column II. 3) Next the pairwise comparisons matrix [a ij ] n n was produced by using the formula 1. The distance based consistency index cm A (equations (2)) is calculated, and, if necessary, reduced to the level smaller than 0.3. JConcluder software [23] was used to reduce inconsistency to an acceptable level. 4) After few days the same experts were asked to transform their qualitative judgments into quantitative form, but now by using multiplicative scale and ranges from Columns III and IV of from Again in all cases the experts decided to use used default values, in this case it was Column IV. JConcluder [23] was again used to reduce inconsistency to an acceptable level. We observed that it is easier for experts to differentiate between different, say, indifferences, when were used; than when a ij were used. 5) The matrices from steps (3) and (4) were then translated back into qualitative form using Column II of Table I. Even though the quantitative matrices were slightly different, their qualitative representations were identical (which can be interpreted as yet another validation of the distance based consistency). 6) For all four applications the final result was different than the initial one. The final results were send back to the experts for final analysis and acceptance. VI. QUALITY OF HEALTHCARE IN CANADA The quality of Healthcare in Canada has been analyzed and assessed in six key domains [21], [22]: the effectiveness of the healthcare sector in improving health outcomes; access to healthcare services; the capacity of systems to deliver appropriate services; the safety of care delivered; the degree to which healthcare in Canada is patient-centred; and equity in healthcare outcomes and delivery. However the importance of particular domains/attributes is far from obvious and open for a discussion. Using pairwise comparisons based techniques can provide ranking and weights assignments that are more trustworthy than these derived from informal or other semi-formal derivations. Procedure 2 has been used by a group of experts and the results of using it are presented in Tables II VIII. Table VIII contains the final qualitative results and its bottom row contains final calculated weights. The weights in Table VIII are just averages of the appropriate weights from Tables V and VII. Note that the difference of appropriate weights from Table V and Table VII is rather small, which could also be seen as validation of our method. The experts have approved our final results, both qualitative and quantitative. The difference between initial and final judgments were not severe, but the initial judgments were not consistent, while the final judgments were consistent (i.e. with acceptable level of inconsistency), hence they were much more trustworthy. Our finding could help to develop initiatives to address specific quality problems in Canada s Healthcare, and ultimately lead to better outcomes for patients. The weights obtained by our analysis (see Table VIII) have the following values (scaled to 100%):

4 Int'l Conf. Artificial Intelligence ICAI' TABLE II: Healthcare in Canada. Initial qualitative judgments. E 6 Effectiveness E 1 > Access E 2 < Capacity E 3 < < < Safety E 4 > > Patient-Centredness E 5 > > > Equity E 6 < < TABLE III: Healthcare in Canada. Initial quantitative judgments derived from Table II by using defaults values from Column II of Name E 1 E 2 E 3 E 4 E 5 E 6 Effectiveness E Access E Capacity E Safety E Patient-Centredness E Equity E TABLE IV: Healthcare in Canada. The values of a ij obtained from Table III. 1 Name E 1 E 2 E 3 E 4 E 5 E 6 Effectiveness E Access E Capacity E Safety E Patient-Centredness E Equity E inconsistency coefficient cm A = 0.57 > 0.3. TABLE V: Healthcare in Canada. Consistent (i.e. with acceptable inconsistency) matrix derived from Table IV, using distance-based consistency. The weights were calculated using the geometric means. E 6 Effectiveness E Access E Capacity E Safety E Patient-Centredness E Equity E w 6 values 29% 12% 6% 23% 24% 6% inconsistency coefficient cm A = 0.15 < 0.3. TABLE VI: Healthcare in Canada. Initial quantitative judgments derived from Table II by using defaults values from Column IV of E 6 Effectiveness E Access E Capacity E Safety E Patient-Centredness E Equity E inconsistency coefficient cm A = 0.62 > 0.3. Effectiveness = 30%, Access = 12%, Capacity = 5%, Safety = 24%, Patient-Centredness = 24%, Equity = 5%. TABLE VII: Healthcare in Canada. Consistent pairwise comparisons matrix derived from Table VI. This table is almost identical as Table V. The different cells are shaded. E 6 Effectiveness E Access E Capacity E Safety E Patient-Centredness E Equity E w 6 values 30% 12% 5% 24% 24% 5% inconsistency coefficient cm A = 0.15 < 0.3. TABLE VIII: Healthcare in Canada. Final qualitative judgments of key attributes derived from Tables V and VII by using the intervals from Column III of Corrected cells are shaded. Also final weights. E 6 Effectiveness E 1 > > Access E 2 Capacity E 3 < < < Safety E 4 > > Patient-Centr. E 5 > > Equity E 6 < < < Final weights 29.5% 12% 5.5% 23.5% 24% 5.5% VII. SOFTWARE QUALITY IN USE ISO (c.f. [7], [16]) has recently developed a new more comprehensive definition of quality in use, which has usability, flexibility and safety as subcharacteristics that can be quantified from the perspectives of different stakeholders, including users, managers and maintainers. Quality in use depends not only on the software or computer system, but also on the particular context in which the product is being used and it can be assessed by observing representative users carrying out representative tasks in a realistic context of use [17]. The standard ISO/IEC [7] proposes the following attributes (called characteristics ) for the assessment of software product quality in use: effectiveness, efficiency, satisfaction, freedom from risk and context coverage with which users can achieve goals in a specified context of use. To provide trustworthy assignment importance indicators, Procedure 2 has been used by a group of software experts. The results are presented in Tables IX XV. Notes that in this case the weights from Table XII and Table XIV are identical. The difference of appropriate weights from Table XII and Table XIV is rather small and the experts have approved our final results, both qualitative and quantitative. As in the previous case, the difference between initial and final judgments were not severe, but the initial judgments were not consistent, while the final judgments were consistent (i.e. with acceptable level of inconsistency), hence they were much more trustworthy. The weights obtained by our analysis (see Table XV) have the following values (scaled to 100%): Effectiveness =21%, Efficiency = 12%, Satisfaction = 26%, Freedom From Risk = 25%, Context Coverage = 16%.

5 418 Int'l Conf. Artificial Intelligence ICAI'15 TABLE IX: Software Quality In Use. Initial qualitative judgments Effectiveness E 1 Efficiency E 2 Satisfaction E 3 Freedom From Risk E 4 Context Coverage E 5 TABLE X: Software Quality In Use. Initial quantitative judgments derived from Table IX by using defaults values from Column II of Name E 1 E 2 E 3 E 4 E 5 Effectiveness E Efficiency E Satisfaction E Freedom From Risk E Context Coverage E TABLE XI: Software Quality In Use. The values of a ij obtained from Table X. 1 Name E 1 E 2 E 3 E 4 E 5 Effectiveness E Efficiency E Satisfaction E Freedom From Risk E Context Coverage E inconsistency coefficient cm A = 0.81 > 0.3. TABLE XII: Software Quality In Use. Consistent pairwise comparisons matrix derived from Table XI. Effectiveness E Efficiency E Satisfaction E Freedom From Risk E Context Coverage E values 21% 12% 26% 25% 16% TABLE XIII: Software Quality In Use. Initial quantitative judgments derived from Table IX by using defaults values from Column IV of Effectiveness E Efficiency E Satisfaction E Freedom From Risk E Context Coverage E inconsistency coefficient cm A = 0.85 > 0.3. VIII. SMART GRID A smart grid is a modernized electrical grid that uses analog or digital information and communications technology to gather and act on information - such as information about the behaviours of suppliers and consumers - in an automated TABLE XIV: Software Quality In Use. Consistent pairwise comparisons matrix derived from Table XIII. This table is almost identical as Table XII. The different cells are shaded. Effectiveness E Efficiency E Satisfaction E Freedom From Risk E Context Coverage E values 21% 12% 26% 25% 16% TABLE XV: Software Quality In Use. Final qualitative judgments of key attributes derived from Tables XII and XIV by using the intervals from Column III of Corrected cells are shaded. Also final weights. Effectiveness E 1 Efficiency E 2 Satisfaction E 3 Freedom From Risk E 4 Context Coverage E 5 Final weights 21% 12% 26% 25% 16% fashion to improve the efficiency, reliability, economics, and sustainability of the production and distribution of electricity [20]. The recommended assessment attributes are usually: reliability, flexibility, efficiency, sustainability and marketenabling. The results of using Procedure 2 for these attributes are presented in Tables XVI XXII. In this case the weights from Tables XIX and XXI are slightly different and the final weights presented in Table XXII are just averages of these from Tables XIX and XXI. However, qualitative tables Tables XIX and XXI are identical and equal to Table XXII. As in the previous two cases, the experts approved final results of our analysis. The scaled weights have the following values: Reliability = 18%, Flexibility = 8%, Efficiency = 31.5%, Sustainability = 8%, Market-enabling = 31.5%. TABLE XVI: Smart Grid Initial qualitative judgments. Reliability E 1 Flexibility E 2 < < Efficiency E 3 > > Sustainability E 4 < < Market-enabling E 5 > > TABLE XVII: Smart Grid. Initial quantitative judgments derived from Table XVI by using default values from Column II of Name E 1 E 2 E 3 E 4 E 5 Reliability E Flexibility E Efficiency E Sustainability E Market-enabling E

6 Int'l Conf. Artificial Intelligence ICAI' TABLE XVIII: Smart Grid. The values of a ij obtained from Table XVII. 1 Name E 1 E 2 E 3 E 4 E 5 Reliability E Flexibility E Efficiency E Sustainability E Market-enabling E inconsistency coefficient cm A = 0.75 > 0.3. TABLE XIX: Smart Grid. Consistent pairwise comparisons matrix derived from Table XVIII. Reliability E Flexibility E Effecincy E Sustainability E Market-enabling E values 19% 8% 32% 8% 33% TABLE XX: Smart Grid. Initial quantitative judgments derived from Table XVI by using the default values from Column IV of Reliability E Flexibility E Efficiency E Sustainability E Market-enabling E inconsistency coefficient cm A = 0.79 > 0.3. TABLE XXI: Smart Grid. Consistent pairwise comparisons matrix derived from Table XX. This table is almost identical as Table XIX. The different cells are shaded. Reliability E Flexibility E Efficiency E Sustainability E Market-enabling E values 18% 8% 31% 8% 30% TABLE XXII: Smart Grid. Final qualitative judgments of key attributes derived from Tables XIX and XXI by using the intervals from Column III of Corrected cells are shaded. Also final weights in bottom row. Reliability E 1 Flexibility E 2 < < Efficiency E 3 > > Sustainability E 4 < < Market-enabling E 5 > > Final weights 18% 8% 31.5% 8% 31.5% IX. MEDICAL DEVICES The ISO [4], [6] standard proposes the following attributes (called characteristics ) for the assessment of Medical Devices Quality Managements Systems: Developing an effective design Conducting risk analyses Following adequate and appropriate standard operating procedures and protocols for testing Using validated methods and procedures Monitoring and auditing Ensuring adequate staff training Adopting and implementing appropriate corrective and preventive action plans We used Procedure 2 to provide appropriate weights for each of the above attributes. The results are presented in Tables XXIII XXIX. While the Tables XXVI and XXVIII are slightly different, the weights they generate are identical (as in the case of Software Quality In Use). Similarly as in the previous three cases, the qualitative tables Tables XXVI and XXVIII are identical and equal to Table XXIX, and the experts approved final results of our analysis. The final results are consistent, so they are more trustworthy than the initial ones, that were based only on the experts opinions. The scaled weights have the following values in this case: Effective Design = 7%, Conducting Risk Analyses = 5%, Following Standard = 12%, Using Validated Methods = 29%, Monitoring and Auditing= 15%, Staff Training = 22%, Adopting and Implementing = 10%. X. FINAL COMMENTS A novel weights assignment techniques, proposed in [9], [11] and called Mixed Pairwise Comparisons, has been applied to improve accuracy of weights assignment for attributes used for assessment of Healthcare in Canada, Quality In Use of Software, Smart Energy Grids and Quality System in Medical TABLE XXIII: Medical Devices. Initial qualitative judgments. E 6 E 7 Effective Design E 1 < < Conducting Risk Analyses E 2 < Following Standard E 3 Using Validated Methods E 4 > > Monitoring and Auditing E 5 Staff Training E 6 > > Adopting and Implem. E 7 < TABLE XXIV: Medical Devices. Initial quantitative judgments derived from Table XXIII by using the default values from Column II of Name E 1 E 2 E 3 E 4 E 5 E 6 E 7 Effective Design E Conducting Risk Analyses E Following Standard E Using Validated Methods E Monitoring and Auditing E Staff Training E Adopting and Implem. E

7 420 Int'l Conf. Artificial Intelligence ICAI'15 TABLE XXV: Medical Devices. The values of a ij obtained from Table XXIII. 1 Name E 1 E 2 E 3 E 4 E 5 E 6 E 7 Effective Design E Conducting Risk Analyses E Following Standard E Using Validated Methods E Monitoring and Auditing E Staff Training E Adopting and Implem. E inconsistency coefficient cm A = 0.81 > 0.3. TABLE XXVI: Medical Devices. Consistent pairwise comparisons matrix derived from Table XXV. E 6 E 7 Effective Design E Conducting Risk Analyses E Following Standard E Using Validated Methods E Monitoring and Auditing E Staff Training E Adopting and Implem. E w 6 w 7 values 7% 5% 12% 29% 15% 22% 10% inconsistency coefficient cm A = 0.19 < 0.3. TABLE XXVII: Medical Devices. Initial quantitative judgments derived from Table XXIII by using the default values from Column IV of E 6 E 7 Effective Design E Conducting Risk Analyses E Following Standard E Using Validated Methods E Monitoring and Auditing E Staff Training E Adopting and Implem. E inconsistency coefficient cm A = 0.99 > 0.3. TABLE XXVIII: Medical Devices. Consistent pairwise comparisons matrix derived from Table XXVII. This table is almost identical as Table XXV. The different cells are shaded. E 6 E 7 Effective Design E Conducting Risk Analyses E Following Standard E Using Validated Methods E Monitoring and Auditing E Staff Training E Adopting and Implem. E w 6 w 7 values 7% 5% 12% 29% 15% 22% 10% inconsistency coefficient cm A = 0.19 < 0.3. TABLE XXIX: Medical Devices. Final qualitative judgments of key attributes derived from Tables XXVI and XXVIII by using the intervals from Column III of Corrected cells are shaded. Final weights are also presented. E 6 E 7 Effective Design E 1 < < Conducting Risk Analyses E 2 < < Following Standard E 3 Using Validated Methods E 4 > > > Monitoring and Auditing E 5 Staff Training E 6 > > Adopting and Implem. E 7 < Final weights 7% 5% 12% 29% 15% 22% 10% Devices Managements System. ACKNOWLEDGMENT Abeer Mirdad acknowledges a support by the Ministry of Higher Education in Saudi Arabia through The Saudi Arabian Cultural Bureau in Canada, and Ryszard Janicki acknowledges a partial support by NSERC grant of Canada. REFERENCES [1] J. Barzilai. Deriving weights from pairwise comparison matrices. Journal of the operational research society, 48(12): , [2] S. Bozóki, T. Rapcsák. On Saaty s and Koczkodaj s inconsistencies of pairwise comparison matrices. Journal of Global Optimization, 42(2): , [3] N. Cowan. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioural and Brain Sci., 24:87 185, [4] C.T. DeMarco, Medical Device Design and Regulation, ASQ Quality Press, 2011 [5] S. French. Decision theory. Halsted Press, [6] ISO 13485:2003 Standard, Medical devices. Quality management systems. Requirements for regulatory purposes, catalogue detail?csnumber=36786 [7] ISO/IEC Standard, System and Software Quality Requirements and Evaluation, detail.htm?csnumber= [8] R. Janicki. Pairwise comparisons based non-numerical ranking. Fundamenta Informaticae, 94(2): , [9] R. Janicki. On Qualitative and Quantitative Pairwise Comparisons, Proc. of ICAT 2015, to appear. [10] R. Janicki, Y. Zhai. Remarks on pairwise comparison numerical and non-numerical rankings. Lecture Notes in Computer Science 6954, pp Springer, [11] R. Janicki, M. H. Soudkhah, On Classification with Pairwise Comparisons, Support Vector Machines and Feature Domain Overlapping, The Computer Journal (2014), doi: /comjnl/bxu085 [12] R. Janicki, Y. Zhai. On a pairwise comparison-based consistent nonnumerical ranking. Logic Journal of IGPL, 20(4): , [13] W. W. Koczkodaj, S.J. Szarek. On distance-based inconsistency reduction algorithms for pairwise comparisons. Logic Journal of IGPL, 18(6): , [14] W.W. Koczkodaj. A new definition of consistency of pairwise comparisons. Mathematical and Computer Modelling, 18(7):79 84, [15] G. A. Miller. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review, 63(2):81, [16] B. Nigel. Extending quality in use to provide a framework for usability measurement Lecture Notes in Computer Science 5619, pp , Springer, 2009 [17] J. Oh, D. Park, B. Lee, J. Lee, E. Hong, C. Wu. Certification of software packages using hierarchical classification Lecture Notes in Computer Science 3024, pp , Springer, 2004 [18] T. L. Saaty. A scaling method for priorities in hierarchical structures. Journal of mathematical psychology, 15(3): , [19] T. L. Saaty. Theory and applications of the analytic network process: decision making with benefits, opportunities, costs, and risks. RWS publications, [20] Smart Grids and Renewables: A Guide for Effective Deployment, International Renewable Energy Agency (IRENA), [21] K. Sutherland, S. Leatherman, S. Law, J. Verma, S. Petersen. Chartbook: shining a light on the quality of healthcare in Canada: what can be learned to catalyze improvements in healthcare quality? Healthc Pap. 12(1):10-24, [22] J.I. Williams, W. Young, A Summary of Studies on the Quality of Health Care Administrative Databases in Canada. In Patterns of Health Care in Ontario: the ICES Practice Atlas, 2 nd ed., pp , Canadian Medical Association, [23] D. Xu. Concluder, directory,april2013.