NON-NUMERICAL RANKING BASED ON PAIRWISE COMPARISONS

NON-NUMERICAL RANKING BASED ON PAIRWISE COMPARISONS By Yun Zhai, M.Sc. A Thesis Submitted to the School of Graduate Studies in partial fulfilment of the requirements for the degree of Ph.D. Department of Computing and Software McMaster University c Copyright by Yun Zhai, Dec. 2010

ii PH.D. (2010) (Computing and Software) McMaster University Hamilton, Ontario TITLE: NON-NUMERICAL RANKING BASED ON PAIRWISE COMPARISONS AUTHOR: Yun Zhai, M.Sc.(McMaster University, Canada) SUPERVISOR: Dr. Ryszard Janicki NUMBER OF PAGES: xxv, 204

Abstract The method of Pairwise Comparisons was first described by Marquis de Condorcet in 1785 [1]. At present, this method is identified with Saaty s Analytic Hierarchy Process (AHP, 1980) [25]. AHP is a formal method to derive ranking orders from pairwise comparisons. It is used around the world in a wide variety of decision making, in fields such as education, industry, and government. On the one hand, AHP has many respected practical applications. On the other hand, however, it is still considered by many researchers as a flawed procedure that produces arbitrary rankings [6]. A non-numerical partial orders based method was proposed by Janicki and Koczkodaj [14] and developed by Janicki [9, 10, 11, 12, 13]. This model used the concepts of partial orders and rough sets, and emphasized the importance of indifference and weak ordering. However, the consistency rules of the model are incomplete. When the inconsistent pairs are found, the non-numerical ranking method manually changes the relationship among those pairs to satisfy the consistency rules. We extend the consistency rules to make them complete and compact. A consistency-driven algorithm by automatically enforcing consistency is presented. Property-driven algorithms by classical partial order approximations and refined partial order approximations are discussed. We present an algorithm using refined partial i

0. Abstract ii order approximations. A method of automatically converting AHP data to nonnumerical pairwise comparison ranking system is discussed, which ensures that the generated non-numerical pairwise comparison ranking system is consistent. We implement various ranking algorithms, including the AHP method, consistency-driven method, property-driven method and property/consistency-driven method. We test the experiments referenced in some non-numerical ranking papers, and give examples to compare how well the various non-numerical ranking methods solve the rank reversal problem.

Acknowledgment I would first like to express my sincere gratitude to my supervisor, Dr. Ryszard Janicki for his supervision, advice, and guidance from the very early stages of this research, as well as letting me benefit from his extraordinary experience throughout this work. His true scientific intuition has made him as a constant source of ideas and scientific passion, which provided exceptional inspiration and enriched my growth. I am indebted to him more than he can image. Many thanks go also to the members of my Supervisor Committee: Dr. Ridha Khedri and Dr. Michael Soltys, for their valuable advice in technical discussions and comments. Thanks to all my other professors for their help during my undergraduate and graduate studies at McMaster. I really appreciate the time and trouble taken by Dr. Jeffery Zucker in improving the presentation of my thesis with his skillful editing. My special thanks go to my English tutor Pamela Mclntyre and Dr. Janicki s PhD student Nadezda Zubkova for their constructive comments on this thesis. My parents deserve special mention for their unconditional love and support. Words fail to express my appreciation to my husband Lixin with his dedication, love and thoughtful support. Finally, I would like to thank everybody who was involved with the successful iii

0. Acknowledgment iv realization of this thesis, as well as express my apologies for not naming each one individually.

Contents Abstract i Acknowledgment iii List of Figures x 1 Introduction 1 1.1 Motivation................................. 1 1.2 Overview.................................. 4 1.3 Major contributions............................ 5 1.4 Structure of the thesis.......................... 5 2 Review of the Analytical Hierarchy Process 7 2.1 Review of the Analytical Hierarchy Process............... 7 2.1.1 The eigenvector method..................... 8 3 Consistency comparisons 12 3.1 Inconsistency indexes........................... 12 3.1.1 Saaty s inconsistency index.................... 12 3.1.2 Koczkodaj s inconsistency index................. 14 v

CONTENTS vi 3.1.3 A comparison of Saaty s and Koczkodaj s inconsistency indices 16 4 Non-numerical ranking: Introduction 19 4.1 Total, weak and partial orders...................... 19 4.2 Weak order approximations....................... 21 4.3 Non-numerical ranking model...................... 22 4.4 Approximations of arbitrary binary relations by partial orders.... 26 5 Consistency-driven non-numerical ranking 29 5.1 Pre-ranking and ranking systems.................... 29 5.2 Automatic enforcement of consistency.................. 31 5.2.1 Compact set of consistency rules................ 37 5.3 Automatic consistency-driven algorithm................ 40 5.4 Rank reversal............................... 41 5.5 Example 1................................. 44 5.5.1 Solution by the AHP method.................. 44 5.5.2 Solution by non-numerical consistency-driven ranking method 46 5.6 Example 2................................. 48 5.6.1 AHP solution........................... 51 5.6.2 Consistency-driven non-numerical ranking solution...... 52 6 Property-driven non-numerical ranking 55 6.1 Mathematical preliminary........................ 55 6.2 Property-driven classical partial order approximations of arbitrary relational structures............................. 62 6.2.1 Property-driven classical partial order approximation algorithm 63

CONTENTS vii 6.2.2 An example............................ 64 6.3 Property-driven refined partial order approximations of arbitrary relational structures.............................. 66 6.3.1 Property-driven refined partial order approximation algorithm 68 6.3.2 An example............................ 68 7 Property/consistency-driven non-numerical ranking 75 7.1 Property/consistency-driven algorithm................. 75 7.2 An example................................ 78 7.2.1 Solution by refined partial order approximation of P by (P ) + 78 7.2.2 Solution by refined partial order approximation of P by (P + ) 80 7.2.3 Solution by refined partial order approximation of P by (P ) 83 8 Converting AHP to CRS 85 8.1 Definition of the new numerical pairwise comparison data...... 85 8.2 An example................................ 87 8.2.1 Automatically generated non-numerical data.......... 89 9 Testing and Experimenting 98 9.1 Test 1: Pre-ranking with acyclicity and transitivity properties.... 99 9.1.1 Solution by consistency-driven algorithm............ 101 9.1.2 Solution by property-driven algorithm............. 111 9.2 Test 2 : Pre-ranking with cycles..................... 117 9.2.1 Solution by consistency-driven algorithm............ 117 9.2.2 Solution by property-driven algorithm............. 119 9.2.3 Solution by property/consistency-driven algorithm....... 131

CONTENTS viii 9.3 Test 3: Pre-ranking with non-transitivity................ 137 9.3.1 Solution by consistency-driven algorithm............ 137 9.3.2 Solution by property-driven algorithm............. 139 9.3.3 Solution by property/consistency-driven algorithm....... 150 9.4 Test 4: Pre-ranking with cycles and non-transitivity.......... 156 9.4.1 Solution by consistency-driven method............. 156 9.4.2 Solution by property-driven method............... 159 9.4.3 Solution by property/consistency-driven method........ 169 9.5 Test 5: Pre-ranking with cycles, non-transitivity, and data errors... 175 9.5.1 Solution by consistency-driven algorithm............ 176 9.5.2 Solution by property-driven algorithm............. 177 9.5.3 Solution by property/consistency-driven algorithm....... 188 10 Comparison and analysis of various algorithms 194 10.1 Comparisons of various non-numerical ranking algorithms....... 194 10.1.1 Case 1: PRS with acyclicity and transitivity.......... 194 10.1.2 Case 2: PRS with cycles..................... 195 10.1.3 Case 3: PRS with acyclicity but non-transitivity........ 196 10.1.4 Case 4: PRS with cycles and non-transitivity......... 196 10.1.5 Case 5: PRS with cycles, non-transitivity and data errors... 198 10.2 Conclusions................................ 198 11 Conclusion and future work 200 Appendix A: Automatic non-numerical ranking system manual 205 A.1 Installation................................ 205

CONTENTS ix A.1.1 System requirements....................... 205 A.1.2 Automatic non-numerical ranking system installation..... 206 A.2 Automatic ranking system functions................... 206 A.2.1 Loading file............................ 206 A.2.2 AHP ranking........................... 208 A.2.3 Consistency-driven non-numerical ranking........... 210 A.2.4 Property-driven classical partial order approximations..... 211 A.2.5 Property-driven refined partial order approximations..... 214 A.2.6 Property/consistency-driven refined partial order approximations214 A.2.7 Variously driven algorithms................... 214 A.2.8 Consistency-driven ranking by combining and...... 220 A.2.9 Property/consistency-driven ranking by the combination of and............................... 223 A.2.10 Our algorithms ranking by combining and........ 223 A.2.11 Output............................... 223 Appendix B: Testing and Experimenting Data 230

List of Figures 4.1 Various types of partial orders...................... 20 4.2 Computing global scores to construct weak order extension...... 22 4.3 An example of arbitrary relation R and its partial order approximations 28 5.1 Automatic consistency-driven algorithm................ 42 5.2 Consistency-driven ranking system state chart............. 43 6.1 The structure P and its PRS...................... 69 6.2 The structures P, P + and (P + ).................... 69 6.3 The structures P, P and (P ) +..................... 71 6.4 The structures P, P and (P ).................... 71 6.5 The structures P, P and (P )..................... 72 7.1 Property/consistency-driven algorithm................. 77 7.2 The structures P, (P ) + and consistent (P ) +............. 80 7.3 The structures P, (P + ) and consistent (P + )............. 81 7.4 Example of a given PRS, and the RS and CRS obtained from it... 84 A.1 An example of type A input file..................... 207 A.2 An example of type B input file..................... 207 x

LIST OF FIGURES xi A.3 An example of AHP ranking....................... 208 A.4 An example of AHP rank reversal information............. 209 A.5 An example of consistency-driven ranking............... 210 A.6 An example of consistency-driven with rank reversal information... 211 A.7 An example of property-driven classical partial order approximations 212 A.8 An example of property-driven classical partial order approximations with rank reversal information...................... 213 A.9 An example of property-driven refined partial order approximations. 215 A.10 An example of property-driven refined partial order approximations with rank reversal information...................... 216 A.11 An example of property/consistency-driven algorithm......... 217 A.12 An example of property/consistency-driven algorithm with rank reversal information.............................. 218 A.13 An example of variously driven algorithm comparisons........ 219 A.14 An example of our algorithms with rank reversal information..... 220 A.15 An example of consistency-driven ranking by the combination of and...................................... 221 A.16 An example of consistency-driven ranking by the combination of and...................................... 222 A.17 An example of property/consistency-driven ranking by combining and................................... 224 A.18 An example of property/consistency-driven with rank reversal information ranking by the combination of and............ 225 A.19 An example of various driven ranking by combining and..... 226

LIST OF FIGURES xii A.20 An example of variously driven ranking with rank reversal, by combining and................................ 227 A.21 An example of saved output....................... 228 A.22 An example of saved output - (continued)............... 229

List of Tables 2.1 The fundamental scale for pairwise comparisons............ 8 3.1 Comparative values [25]......................... 13 3.2 Sándor s statistical analysis for CR 10%, GD 1 and GD 2 [2] 17 5.1 Complete set of consistency rules.................... 37 5.2 Compact set of consistency rules..................... 39 5.3 Pairwise comparison rank reversal definition.............. 44 5.4 AHP pairwise comparison data for the first three objects....... 45 5.5 AHP pairwise comparison data for the four objects.......... 46 5.6 Pre-rankings, consistency rule violations and revised consistent ranking 47 5.7 Global scores for the first three objects................. 48 5.8 Original PRS s for the four objects................... 49 5.9 Numbers of consistency rule violations for the original PRS s P C1 and P C3..................................... 49 5.10 The explanations for consistency rule violations for the original PRS s P C1 and P C3................................ 50 5.11 Revised CRS s for criteria C 1 and C 3.................. 50 5.12 Global scores for the four objects.................... 51 xiii

LIST OF TABLES xiv 5.13 Numbers of consistency rule violations for original PRS and revised CRS. 54 6.1 P for first three objects based on criteria C 1, C 2, C 3 and C 4.... 64 6.2 Global scores for the first three objects by classical p.o. approximations 65 6.3 P for the four objects based on criteria C 1, C 2, C 3 and C 4...... 66 6.4 Global scores for all four objects by classical p.o. approximations.. 67 6.5 P + and (P + ).............................. 70 6.6 P and (P ) +............................... 71 6.7 P and (P ).............................. 72 6.8 P and (P )............................... 73 6.9 Global scores and ranking orders for the refined p.o. approximations. 74 7.1 RS for (P ) + with the numbers of consistency rule violations..... 78 7.2 Explanations for the violations of the consistency rules........ 79 7.3 RS and CRS for (P ) +.......................... 79 7.4 RS for (P + ) with the numbers of consistency rule violations..... 81 7.5 Explanations for the consistency rule violations............ 82 7.6 RS and CRS for (P + ).......................... 83 7.7 RS for (P ) and the numbers of consistency rule violations..... 84 8.1 Heuristic mapping from new AHP data to non-numerical data.... 87 8.2 A proof of consistency of converted data................ 88 8.3 New AHP data for criterion C 1..................... 89 8.4 Converted CRS for criterion C 1..................... 90 8.5 New AHP data for criterion C 2..................... 90 8.6 Converted CRS for criterion C 2..................... 90 8.7 New AHP data for criterion C 3..................... 91

LIST OF TABLES xv 8.8 Converted CRS for criterion C 3..................... 91 8.9 New AHP data for criterion C 4..................... 92 8.10 Converted CRS for criterion C 4..................... 92 8.11 Global scores for the first three objects................. 92 8.12 New AHP data for criterion C 1..................... 93 8.13 Converted CRS for criterion C 1..................... 93 8.14 New AHP data for criterion C 2..................... 94 8.15 Converted CRS for criterion C 2..................... 94 8.16 New AHP data for criterion C 3..................... 95 8.17 Converted CRS for criterion C 3..................... 95 8.18 New AHP data for criterion C 4..................... 96 8.19 Converted CRS for criterion C 4..................... 96 8.20 Global scores for the four objects.................... 97 9.1 The weights of the set of stones...................... 99 9.2 Test 1: The first PRS with acyclicity and transitivity properties...... 100 9.3 Test 1: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies...................... 100 9.4 Test 1: The 2nd (revised) PRS. Light grey cells were revised........ 101 9.5 Test 1: Numbers of consistency rule violations for the 2nd PRS. Light grey cells have the most inconsistencies...................... 102 9.6 Test 1: The 3rd (revised) PRS. Light grey cells were revised........ 103 9.7 Test 1: Numbers of consistency rule violations for the third PRS. Light grey cells have the most inconsistencies...................... 103 9.8 Test 1: The 4th (revised) PRS. Light grey cells were revised........ 104

LIST OF TABLES xvi 9.9 Test 1: Numbers of consistency rule violations for the 4th PRS. Light grey cells have the most inconsistencies...................... 105 9.10 Test 1: The 5th (revised) PRS. Light grey cells were revised........ 106 9.11 Test 1: Numbers of consistency rule violations for the 5th PRS. Light grey cells have the most inconsistencies...................... 106 9.12 Test 1: The 6th (revised) PRS. Light grey cells were revised........ 107 9.13 Test 1: Numbers of consistency rule violations for the 6th PRS. Light grey cells have the most inconsistencies...................... 108 9.14 Test 1: The 7th (revised) PRS. Light grey cells were revised........ 109 9.15 Test 1: Numbers of consistency rule violations for the 7th PRS. Light grey cells have the most inconsistencies...................... 109 9.16 Test 1: The 8th (revised) PRS. It is consistent. Light grey cells were revised. 110 9.17 Test 1: Numbers of consistency rule violations for the 8th PRS...... 111 9.18 Test 1: Global scores produced by the consistency-driven algorithm.... 111 9.19 Test 1: P (X,, ) for the first combined PRS.............. 112 9.20 Test 1: Property-driven classical partial order approximation of P by (P ) + 113 9.21 Test 1: Property-driven classical partial order approximation of P by (P + ) 114 9.22 Test 1: Property-driven classical partial order approximations of P by (P ) and (P )............................ 115 9.23 Test 1: Global scores produced by the property-driven classical partial order approximations............................... 115 9.24 Test 1: Global scores produced by the property-driven refined p.o. approx. 116 9.25 Test 2: The first PRS with cycles. Light grey cells indicate a cycle..... 117 9.26 Test 2: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies..................... 118

LIST OF TABLES xvii 9.27 Test 2: The 10th (revised) PRS. It is a consistent ranking......... 119 9.28 Test 2: Global scores produced by the consistency-driven algorithm.... 119 9.29 Test 2: Combined system P = (X,, ) for the first PRS. Light grey cells indicate a cycle............................... 120 9.30 Test 2: Property-driven classical partial order approximation of P by P. Light grey cells indicate the revisions by removing all cycles from P.... 121 9.31 Test 2: Property-driven classical partial order approximation of P by (P ) + 122 9.32 Test 2: Property-driven classical partial order approximation of P by (P + ) 123 9.33 Test 2: Property-driven classical partial order approximation of P by (P ) 124 9.34 Test 2: Property-driven classical partial order approximation of P by (P )................................... 125 9.35 Test 2: Global scores produced by the classical partial order approximation algorithm.................................. 125 9.36 Test 2: Ranking orders produced by the property-driven p.o. approximations126 9.37 Test 2: Property-driven refined partial order approximation of P by (P ) + 127 9.38 Test 2: Property-driven refined partial order approximation of P by (P + ) 128 9.39 Test 2: Property-driven refined partial order approximation of P by (P ) 129 9.40 Test 2: Property-driven refined partial order approximation of P by (P ) 130 9.41 Test 2: Global scores produced by the property-driven refined partial order approximation algorithm.......................... 130 9.42 Test 2: Ranking orders produced by the property-driven refined partial order approximation algorithm....................... 131 9.43 Test 2: Property/consistency-driven refined partial order approximation of P by (P ) +................................. 132

LIST OF TABLES xviii 9.44 Test 2: Property/consistency-driven refined partial order approximation of P by (P + )................................. 133 9.45 Test 2: Property/consistency-driven refined partial order approximation of P by (P )................................. 134 9.46 Test 2: Property/consistency-driven refined partial order approximation of P by (P )................................. 135 9.47 Test 2: Global scores produced by the property/consistency-driven refined partial order approximation algorithm................... 135 9.48 Test 2: Ranking orders produced by the property/consistency-driven refined partial order approximation algorithm................... 136 9.49 Test 3: The original PRS with non-transitivity. Light grey cells show how the transitivity property is not satisfied................... 137 9.50 Test 3: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies...................... 138 9.51 Test 3: After 8 iterative revisions. This is a consistent ranking....... 138 9.52 Test 3: Global scores produced by the consistency-driven algorithm.... 139 9.53 Test 3: Combined PRS P for the first PRS without transitivity property. Light grey cells show how the transitivity property is not satisfied..... 140 9.54 Test 3: Property-driven classical partial order approximation of P by (P ) + 141 9.55 Test 3: Property-driven classical partial order approximation of P by (P + ) 142 9.56 Test 3: Property-driven classical partial order approximation of P by (P ) 143 9.57 Test 3: Property-driven classical partial order approximation of P by (P ) 144 9.58 Test 3: Global scores produced by the property-driven classical partial order approximation algorithm.......................... 144

LIST OF TABLES xix 9.59 Test 3: Ranking order produced by the property-driven classical partial order approximation algorithm....................... 145 9.60 Test 3: Property-driven refined partial order approximation of P by (P ) + 146 9.61 Test 3: Property-driven refined partial order approximation of P by (P + ) 147 9.62 Test 3: Property-driven refined partial order approximation of P by (P ) 148 9.63 Test 3: Property-driven refined partial order approximation of P by (P ) 149 9.64 Test 3: Global scores produced by the property-driven refined partial order approximation algorithm.......................... 149 9.65 Test 3: Ranking orders produced by the property-driven refined partial order approximation algorithm....................... 150 9.66 Test 3: Property/consistency-driven refined partial order approximation of P by (P ) +................................. 151 9.67 Test 3: Property/consistency-driven refine approximation of P by (P + ). 152 9.68 Test 3: Property/consistency-driven refined partial order approximation of P by (P )................................. 153 9.69 Test 3: Property/consistency-driven refined partial order approximation of P by (P )................................. 154 9.70 Test 3: Global scores produced by the property/consistency-driven refined partial order approximation algorithm................... 154 9.71 Test 3: Ranking orders produced by the property/consistency-driven refined partial order approximation algorithm................... 155 9.72 Test 4: A PRS with cycles and without transitivity. Dark grey cells indicate a cycle, and light grey cells indicate violations of the transitive property.. 156 9.73 Test 4: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies...................... 157

LIST OF TABLES xx 9.74 Test 4: After 7 iterative revisions. This is a CRS.............. 158 9.75 Test 4: Global scores produced by the consistency-driven algorithm.... 158 9.76 Test 4: Combined PRS P (X,, ). Dark grey cells indicate a cycle and light grey cells indicate two violations of transitivity............ 159 9.77 Test 4: Property-driven classical partial order approximation of P by (P ) + 160 9.78 Test 4: Property-driven classical partial order approximation of P by (P + )................................... 161 9.79 Test 4: Property-driven classical partial order approximation of P by (P ) 162 9.80 Test 4: Property-driven classical partial order approximation of P by (P ) 163 9.81 Test 4: Global scores produced by the property-driven classical partial order approximations............................... 163 9.82 Test 4: Ranking orders produced by the property-driven classical partial order approximations............................ 164 9.83 Test 4: Property-driven refined partial order approximation of P by (P ) + 165 9.84 Test 4: Property-driven refined partial order approximation of P by (P + ) 166 9.85 Test 4: Property-driven refined partial order approximation of P by (P ) 167 9.86 Test 4: Property-driven refined partial order approximation of P by (P ) 168 9.87 Test 4: Global scores produced by the property-driven refined partial order approximations............................... 168 9.88 Test 4: Ranking orders produced by the property-driven refined partial order approximations............................ 169 9.89 Test 4: Property/consistency-driven refined partial order approximation of P by (P ) +. It is a CRS........................... 170 9.90 Test 4: Property/consistency-driven refined partial order approximation of P by (P + ). It is a CRS........................... 171

LIST OF TABLES xxi 9.91 Test 4: Property/consistency-driven refined partial order approximation of P by (P ). It is a CRS........................... 172 9.92 Test 4: Property/consistency-driven refined partial order approximation of P by (P ). It is a CRS........................... 173 9.93 Test 4: Global scores produced by the property/consistency-driven refined partial order approximations........................ 173 9.94 Test 4: Ranking orders produced by the property/consistency-driven refined partial order approximations........................ 174 9.95 Test 5: The first PRS with cycles, without transitivity and with data errors. Dark grey cells indicate a cycle, light grey cells indicate a failure transitivity, and the bold fonts indicate data errors................... 175 9.96 Test 5: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies...................... 176 9.97 Test 5: After 9 iterative revisions. It is a CRS............... 177 9.98 Test 5: Global scores produced by the consistency-driven algorithm.... 177 9.99 Test 5: Combined PRS P. Dark grey cells indicate a cycle, light grey cells indicate failure of transitivity, and bold fonts indicate data errors...... 178 9.100Test 5: Property-driven classical partial order approximation of P by (P ) + 179 9.101Test 5: Property-driven classical partial order approximation of P by (P + ) 180 9.102Test 5: Property-driven classical partial order approximation of P by (P ) 181 9.103Test 5: Property-driven classical partial order approximation of P by (P ) 182 9.104Test 5: Global scores produced by the property-driven classical partial order approximations............................... 182 9.105Test 5: Ranking orders produced by the property-driven classical partial order approximations............................ 183

LIST OF TABLES xxii 9.106Test 5: Property-driven refined partial order approximation of P by (P ) + 184 9.107Test 5: Property-driven refined partial order approximation of P by (P + ) 185 9.108Test 5: Property-driven refined partial order approximation of P by (P ) 186 9.109Test 5: Property-driven refined partial order approximation of P by (P ) 187 9.110Test 5: Global scores produced by the property-driven refined partial order approximations............................... 187 9.111Test 5: Ranking orders produced by the property-driven refined partial order approximations............................ 188 9.112Test 5: Property/consistency-driven refined partial order approximation of P by (P ) +. This is a CRS......................... 189 9.113Test 5: Property/consistency-driven refined partial order approximation of P by (P + ). This is a CRS......................... 190 9.114Test 5: Property/consistency-driven refined partial order approximation of P by (P ). This is a CRS......................... 191 9.115Test 5: Property/consistency-driven refined partial order approximation of P by (P )................................. 192 9.116Test 5: Global scores produced by the property/consistency-driven refined partial order approximations........................ 192 9.117Test 5: Ranking orders produced by the property/consistency-driven refined partial order approximations........................ 193 10.1 Case 1: PRS with acyclicity and transitivity.............. 195 10.2 Case 2: PRS with cycles. Gray rows indicate the algorithm can produce rank reversals................................ 196 10.3 Case 3: PRS with acyclicity but non-transitivity. Gray rows indicate the algorithm can produce rank reversals................. 197

LIST OF TABLES xxiii 10.4 Case 4: PRS with cycles and non-transitivity. Gray rows indicate the algorithm can produce rank reversals.................. 197 10.5 Case 5: PRS with cycles, non-transitivity and data errors. Light grey cells indicate the algorithm can produce rank reversals......... 198 B.1 Data 1 and Data 2............................ 230 B.2 Data 3 and Data 4............................ 231 B.3 Data 6 and Data 7............................ 231 B.4 Data 8 and Data 9............................ 231 B.5 Data 11 and Data 12........................... 232 B.6 Data 13 and Data 14........................... 232 B.7 Data 15 and Data 16........................... 232 B.8 Data 17 and Data 18........................... 233 B.9 Data 19 and Data 20........................... 233 B.10 Data 21.................................. 234 B.11 Data 22.................................. 234 B.12 Data 23.................................. 235 B.13 Data 24.................................. 235 B.14 Data 27.................................. 236 B.15 Data 28.................................. 236 B.16 Data 29.................................. 237 B.17 Data 30.................................. 237 B.18 Data 31.................................. 238 B.19 Data 32.................................. 239 B.20 Data 33.................................. 239 B.21 Data 34.................................. 240

LIST OF TABLES xxiv B.22 Data 35.................................. 240 B.23 Data 36.................................. 241 B.24 Data 37.................................. 241 B.25 Data 38.................................. 242 B.26 Data 39.................................. 242 B.27 Data 40-1................................. 243 B.28 Data 40-(2)................................ 244 B.29 Data 41-(1)................................ 245 B.30 Data 41-(2)................................ 246 B.31 Data 42-(1)................................ 247 B.32 Data 42-(2)................................ 248 B.33 Data 44-(1)................................ 249 B.34 Data 44-(2)................................ 250 B.35 Data 45-(1)................................ 251 B.36 Data 45-(2)................................ 252 B.37 Data 46-(1)................................ 253 B.38 Data 46-(2)................................ 254 B.39 Data 47-(1)................................ 255 B.40 Data 47-(2)................................ 256 B.41 Data 47-(3)................................ 257 B.42 Data 47-(4)................................ 258 B.43 Data 47-(5)................................ 258 B.44 Data 47-(6)................................ 258 B.45 Data 48-(1)................................ 259 B.46 Data 48-(2)................................ 260

LIST OF TABLES xxv B.47 Data 48-(3)................................ 261 B.48 Data 48-(4)................................ 262 B.49 Data 48-(5)................................ 262 B.50 Data 48-(6)................................ 262 B.51 Data 49-(1)................................ 263 B.52 Data 49-(2)................................ 264 B.53 Data 49-(3)................................ 265 B.54 Data 49-(4)................................ 266 B.55 Data 49-(5)................................ 266 B.56 Data 49-(6)................................ 267 B.57 Data 50-(1)................................ 268 B.58 Data 50-(2)................................ 269 B.59 Data 50-(3)................................ 270 B.60 Data 50-(4)................................ 271 B.61 Data 50-(5)................................ 271 B.62 Data 50-(6)................................ 272

Chapter 1 Introduction This chapter provides a brief introduction to the background, aims and outline of this thesis. 1.1 Motivation The method of Pairwise Comparisons was first described by Marquis de Condorcet in 1785 [1]. At present, this method is identified with Saaty s Analytic Hierarchy Process (AHP, 1980) [25]. AHP is a formal method to derive ranking orders from pairwise comparisons. The value of paired comparisons can be obtained from actual measurements such as length, weight etc., or from one s subjective opinion such as preference. AHP is used around the world in a wide variety of decision making, in fields such as education, industry, and government. On the one hand, AHP has many respected practical applications. However, it is still considered by many researchers to be a flawed procedure that produces arbitrary rankings [6]. Consider the example of rank reversal provided by Dyer [6]: 1

1. Introduction 2 Alternatives C 1 C 2 C 3 C 4 A 1 1 9 1 3 A 2 9 1 9 1 A 3 8 1 4 5 A 4 4 1 8 5 where A 1, A 2, A 3 and A 4 represent four objects, and C 1, C 2, C 3 and C 4 denote four measured criteria. Assuming that the four criteria C 1, C 2, C 3 and C 4 are equally important, the rankings determined by AHP for the first three alternatives are given by C 1 C 2 C 3 C 4 Score Rank A 1 1/18 9/11 1/14 3/9 0.320 3 A 2 9/18 1/11 9/14 1/9 0.336 2 A 3 8/18 1/11 4/14 5/9 0.344 1 Accordingly for the first three objects, the ranking order is A 3 > A 2 > A 1. However the rankings determined by AHP for four objects are given by C 1 C 2 C 3 C 4 Score Rank A 1 1/22 9/12 1/22 3/14 0.264 1 A 2 9/22 1/12 9/22 1/14 0.243 4 A 3 8/22 1/12 4/22 5/14 0.246 2 A 4 4/22 1/12 8/22 5/14 0.246 2 Hence for four objects, the ranking order is A 1 > A 3 A 4 > A 2, so that the alternatives A 1 and A 3 have reversed rankings compared with the ranking order for the first three objects.

1. Introduction 3 For the AHP method, even though the input numbers are given roughly, the results are treated very precisely. If for example, after AHP processing, the object A obtained weight 20.01 and B obtained weight 19.99, it is usually stated that A > B. However A B seems to be more appropriate. The problem of producing arbitrary rankings is primarily caused by the following reasons [9]: Numeric values are used to calculate and compare the final ranking. The final outcome is expected to be totally ordered. In the late 1990 s, Janicki and Koczkodaj, proposed and discussed pairwise comparisons based non-numerical solutions for ranking [14, 15]. The solutions did not use numbers at all. A non-numerical model of ranking based on pairwise comparisons was proposed, using the concept of partial orders and emphasizing the importance of indifference and weak ordering. The model exploits known, but not often used results from partial order theory [8]. This model was revised and enhanced recently by Janicki [9, 10, 12]. However, the model is still underdeveloped. For example, the set of consistency rules is incomplete, and an automatic enforcement of consistency rules is nonexistent. The model proposed various types of approximations of arbitrary relational structures by partial orders, but how it reflects the final outcome is unknown. The objectives of this thesis are the following: Extend consistency rules and develop an approach to automatically enforce consistency as defined in [9]. Analyze in detail how the approximations proposed in [10] and [12] are related to the consistency ideas of [9].

1. Introduction 4 Provide a method to calculate the proper transitive closure of an arbitrary relational structure P for the refined partial order approximations of P. Discuss how various types of approximations of arbitrary relational structures by classical partial orders and refined partial orders proposed in [11] reflect the final outcome. Find a heuristic method for automatic generation of non-numerical pairwise comparisons consistent ranking system. 1.2 Overview In this thesis we present consistency-driven, property-driven and property/consistency-driven algorithms based on non-numerical pairwise comparisons. In this section, we briefly describe these algorithms and state their major contributions to give the reader an overall perspective. Consistency-driven algorithm This applies complete consistency rules to automatically enforce consistency of non-numerical pairwise comparisons pre-ranking. Property-driven algorithm This includes classical partial order approximations, and also refined partial order approximations of an arbitrary relational structure P. The properties considered here are natural properties such as transitivity closure (denoted by P + ), acyclic property (denoted by P ) and the subset closure property (denoted by P and introduced in [12]). Four partial order approximations of P, namely (P ) +, (P + ), (P ) and (P ), are discussed.

1. Introduction 5 Property/consistency-driven algorithm This is a combination of the propertydriven and consistency-driven algorithms. First the property-driven algorithm is applied to calculate refined partial order approximations of P, then the consistent-driven algorithm is used to enforce the generated pairwise comparisons ranking system to be consistent. 1.3 Major contributions Our major achievement is to extend the consistency rules for non-numerical case so as to make them complete and compact, and present consistency-driven and property/consistency-driven algorithms to automatically enforce consistency based on the pairwise comparisons. We also demonstrate a heuristic approach to automatically convert AHP numerical data to non-numerical pairwise comparisons consistent ranking system (CRS). 1.4 Structure of the thesis The rest of this thesis is organized as follows. Chapter 2 presents a brief review of the Analytical Hierarchy Process. The consistency comparisons between the classical AHP and Koczkodaj s version are given in Chapter 3. Chapter 4 provides a brief introduction to non-numerical ranking. The consistency-driven non-numerical ranking method is presented in Chapter 5. Chapter 6 is devoted to the property-driven nonnumerical ranking method. The property/consistency-driven non-numerical ranking method is presented in Chapter 7. Chapter 8 gives a heuristic method to automatically convert AHP data to non-numerical CRS. Testing and experimenting are

1. Introduction 6 presented in Chapter 9. Comparisons and an analysis of our various non-numerical ranking algorithms are presented in Chapter 10. The thesis ends with a short conclusion in Chapter 11. Appendix A gives the user documentation for the Automatic Ranking System. Appendix B provides all the testing and experimenting data that we did.

Chapter 2 Review of the Analytical Hierarchy Process 2.1 Review of the Analytical Hierarchy Process The Analytical Hierarchy Process (AHP), developed by Saaty in 1980 [25], is widely applied to make decisions in areas such as economics and planning. In short, AHP is a method to derive ratio scales from paired comparisons. It allows some inconsistency in judgment. The ratio scales are derived from the principal eigenvector, and the consistency index is derived from the principal eigenvalue. In order to help decision makers to assess pairwise comparisons, Saaty [28] created a nine 1 point intensity scale of importance between two elements, presented in Table 2.1. 1 Even though nine is within the range 7±2 [22], the number is too big according to latest studies of human abilities [5]. This may cause problems when this technique is used. 7

2. Review of the Analytical Hierarchy Process 8 Intensity of Importance Definition 1 Equal importance 3 Moderate importance 5 Strong importance 7 Very strong or demonstrated importance 9 Extreme importance 2, 4, 6, 8 For compromise between the above values Reciprocals of above If activity i has one of the above nonzero numbers assigned to it when compared with activity j, then j has the reciprocal value when compared with i Table 2.1: The fundamental scale for pairwise comparisons 2.1.1 The eigenvector method Saaty originally introduced a method of scaling ratios using the principle eigenvector of a positive pairwise comparisons matrix. The method presumes that matrix A is: w 1 w 1 w 1 w 1 w 2 w n a 11 a 12 a 1n w 2 w 2 w 2 A = w 1 w 2 w n = a 21 a 22 a 2n...... w n w n w n a n1 a n2 a nn w 1 w 2 w n where w 1, w 2,, w n are the weights obtained by the comparisons. This is a reciprocal matrix, which has all positive elements and has the reciprocal properties: a ij = 1 a ji

2. Review of the Analytical Hierarchy Process 9 and a ij = a ik a jk To explicate how the eigenvalue method works, we use the following 3 by 3 reciprocal matrix as an example: 1 1 5 3 A = 3 1 7 1 1 1 5 7 Step 1: Normalize each matrix element First, we sum each column of the reciprocal matrix, illustrated as follows: ( sum : 1 + 3 + 1 5 = 21 5 1 3 + 1 + 1 7 = 31 21 5 + 7 + 1 = 13 ) Then divide each element of the matrix with the sum of its column to get normalized relative weights, shown as follows: 5 21 A = 15 21 1 21 7 31 21 31 3 31 5 13 7 13 1 13 The sum of each column is 1, as shown below: ( 5 21 + 15 21 + 1 21 = 1 7 31 + 21 31 + 3 31 = 1 5 13 + 7 13 + 1 ) 13 = 1 Step 2: Normalize principal eigenvector The normalized principal eigenvector w can be obtained by averaging across the rows of the normalized matrix of A, as

2. Review of the Analytical Hierarchy Process 10 shown: 5 21 + 7 31 + 5 13 0.2828 w = 1 15 3 21 + 21 31 + 7 13 = 0.6434 1 21 + 3 31 + 1 0.0738 13 The normalized principal eigenvector is also called a priority vector. Since it is normalized, the sum of all its elements is 1. The priority vector shows the relative weights among the things that we are comparing. Step 3: Check consistency The consistency of the matrix can also be checked. Its computation and significance will be explained in Chapter 4. For the given pairwise comparisons matrix A: 1 1 5 3 A = 3 1 7 1 1 1 5 7 To calculate the eigenvalue of A, The determinant of (A λ I) is set to zero: 1 1 λ 5 3 det(a λ I) = 3 1 λ 7 1 1 = 0. 1 λ 5 7 The largest eigenvalue of A, denoted as λ max, is 3.0649, and we have 1 2.0649 5 3 w 1 3 2.0649 7 1 1 w 2 = 0. 2.0649 w 3 5 7

2. Review of the Analytical Hierarchy Process 11 The principal eigenvector w is the eigenvector that corresponds to λ max : w = w 1 w 2 w 3 0.3928 = 0.9140. 0.1013 As the sum of principal eigenvector w is not equal to 1, we must compute the normalized principal eigenvector: 0.2790 w = 0.6491. 0.0719 The normalized principal eigenvector w is very close to our approximation w 0.2828 w = 0.6434 0.0738 Thus, we can say that the pairwise comparison data are consistent for this example. But the question remains: how close an approximation is adequate to enable us to say whether the pairwise comparison data are consistent or not. In the next chapter, we will show two different formal methods to compute the so-called inconsistency index.

Chapter 3 Consistency comparisons A pairwise comparison matrix A is consistent if it satisfies the transitivity property a ik a kj = a ij for any indicies (i, j, k), i, j, k = 1, 2,, n. Otherwise, A is inconsistent. In real-life decision problems, pairwise comparison matrices are rarely consistent. Decision makers are naturally interested in the level of consistency of their judgments, because inconsistent judgments are sure to lead to senseless decisions [2]. 3.1 Inconsistency indexes 3.1.1 Saaty s inconsistency index Saaty proposed a method for calculating inconsistency in 1980 [25]. Let λ max be the largest eigenvalue of A. He showed that λ max n and equals n if and only if A is consistent. The inconsistency index(ci n ) is defined as below: CI n = λ max n n 1 12

3. Consistency comparisons 13 n 2 3 4 5 6 7 8 9 10 11 12 13 14 15 RI n 0 0.52 0.89 1.12 1.26 1.36 1.41 1.46 1.49 1.51 1.48 1.56 1.57 1.59 Table 3.1: Comparative values [25] which gives the average inconsistency. Notice that λ max n, hence CI n is always non-negative. The inconsistency index (CI n ) on its own has no real meaning, unless it is compared with some benchmark to determine the ratio of the deviation from consistency. Let RI n be the benchmark obtained as the inconsistency indices from a set of 1 random pairwise comparison matrices of size n n, with a ji defined as (see Table 3.1). The inconsistency ratio (CR n ) indicating inconsistency is defined as a ij CR n = CI n RI n If the pairwise comparison matrix is consistent, then λ max = n, and so CI n = 0 and CR n = 0 as well. Saaty s inconsistency ratio is an index for departure from randomness. Saaty concluded that an inconsistency ratio of about 10% or less could be considered acceptable. Saaty made some improvement on his inconsistency ratio in 1990 [26]. More recently, Saaty [27] has proposed a threshold of 5% for 3 3, and 8% for 4 4 matrices.

3. Consistency comparisons 14 3.1.2 Koczkodaj s inconsistency index Though Saaty s consistency definition is widely accepted, its definition has some drawbacks. According to Koczkodaj [18], the major drawback of Saaty s inconsistency definition seems to be the 10% threshold. In 1993, Koczkodaj [18] proposed calculating inconsistency based on a measure of deviation from the nearest consistent reciprocal matrix, as follows (in the case n = 3). If a basic reciprocal matrix 1 a b 1 1 c a 1 1 1 b c is reduced to a vector of three coordinates [a, b, c], then in the consistent cases, b = ac holds. It is always possible to produce three consistent basic reciprocal matrices (represented by three vectors) by computing one coordinate from the combination of the remaining two coordinates. These three vectors are: [ b c, b, c ], [ ] [ a, ac, c and a, b, The inconsistency index of a general 3 3 pairwise comparison matrix, defined by Koczkodaj, denoted by CM(a, b, c), is the relative distance to the nearest consistent 3 3 pairwise comparison matrix represented by one of these three vectors: b ] a { 1 CM(a, b, c) = min a a b, c 1 b ac, b 1 c b } c a The inconsistency index of an n n (n 2) reciprocal matrix A is then defined as: { { CM(A) = max min 1 b, 1 ac } } for each triad (a, b, c) in A ac b

3. Consistency comparisons 15 i.e. the maximum of CM for all possible triads. The number of all possible triads of a n n comparisons matrix is equal to n(n 1)(n 2)/3! Koczkodaj [19] proposed that the threshold CM should be 1/3 in the case of 4 4 pairwise comparison matrices with a ratio scale of 1 5, 1 4, 1 3, 1, 1, 2, 3, 4, 5. 2 Note that the threshold CM 1/3 is given for 4 4 pairwise comparison matrices. How to determine the threshold values for higher dimensions? Koczkodaj proposed two rules: the one grade off and two grade off rules, as follows. For the 3 3 positive reciprocal matrices, in the consistent cases, a = b c, 1 a = c b, b = ac, 1 b = 1 ac, c = b a, 1 c = a b. In the inconsistent cases, the grade difference [19] denoted by GD(a, b, c), is used to decide the approximation of an element by the other two elements: { GD(a, b, c) = min max{ a b/c, 1/a c/b }, max{ b ac, 1/b 1/ac } {, max c b/a, 1/c a/b } } The one grade off and two grades off rules are then defined as respectively. GD(a, b, c) 1 and GD(a, b, c) 2, For matrices bigger than 3, the one grade off and two grades off rules are defined by: { } GD(A) = max GD(a, b, c) for each triad (a, b, c) in A 1 or 2

3. Consistency comparisons 16 respectively[19]. In Figure 2 of [2], it is shown that the threshold CM 1 3 corresponds to GD 2 3, which is close to the one grade off rule. 3.1.3 A comparison of Saaty s and Koczkodaj s inconsistency indices Saaty s 10% rule allows much higher CM inconsistency when using the ratio scale An example is the following [2]: 1 9, 1 8, 1 7,, 1, 1, 2,, 8, 9. 2 A = 1 1 2 6 8 8 1 7 9 1 1 1 2 2 7 1 1 1 1 6 9 2 where CR = CR(4, 9) = 9.47% and CM = 0.8125. A statistical analysis of inconsistencies among GD 1, GD 2 and CR 10% is performed by Sándor [2], shown in Table 3.2. The elements a ij (i < j) were randomly chosen from the scale 1 9, 1 8, 1 7,, 1, 1, 2,, 8, 9 2 with a ji defined as 1 a ij. In Table 3.2, n varies from 3 to 10, and the sample size is 10 7 for all n.

3. Consistency comparisons 17 Number of Ratio of Number of Ratio of Number of Ratio of n Sample Size Matrices with Matrices with Matrices with Matrices with atrices with Matrices with CR 10% CR 10% GD 1 GD 1 GD 2 GD 2 3 10 7 2.08 10 6 20.8% 1.42 10 6 14.2% 2.08 10 6 26.8% 4 10 7 3.15 10 5 3.15% 2.76 10 4 0.276% 1.7 10 5 1.7% 5 10 7 2.39 10 4 0.239% 61 0.00061% 2404 0.024% 6 10 7 770 0.0077% 0 0% 13 0.00013% 7 10 7 9 0.00009% 0 0% 0 0% 8 10 7 0 0% 0 0% 0 0% 9 10 7 0 0% 0 0% 0 0% 10 10 7 0 0% 0 0% 0 0% Table 3.2: Sándor s statistical analysis for CR 10%, GD 1 and GD 2 [2] From Table 3.2, we see that with n increasing, the number of matrices satisfying CR n 10%, GD 1 and GD 2 decreases dramatically. In 1956, George Miller [22] proposed that the amount of information that people can process and remember is generally limited to about seven items. Likewise in Table 3.2, seven seems to be the magic number such that for a matrix size greater than or equal to it, the ratio of consistent matrices approximates to 0%. However, in 2001 N. Cowan [5] suggested that this magic number is actually 4, which also seems to be supported by Table 3.2. When n = 8, 9, 10, no matrix in the samples of ten million with acceptable inconsistency was found by Sándor. Notice also that Koczkodaj s Inconsistency Index is more restrictive than Saaty s. Returning to Table 3.1, we see that for a given RI n, the maximum matrix size n is 15. If the matrix size is greater than 15, one cannot decide, using the AHP method,

3. Consistency comparisons 18 whether the ranking orders are acceptable or not.

Chapter 4 Non-numerical ranking: Introduction In this chapter, we recall some well-known concepts and results that create the foundations of our approach and will be used in the following chapters. Almost all the concepts and results in this chapter, together with proofs, can be found in [8, 9, 10, 12]. 4.1 Total, weak and partial orders Let X be a finite set. A relation X X is a partial order (p.o.) if it is irreflexive and transitive, i.e., if a b (b a) and a b c a c, for all a, b, c X. A pair (X, ) is called a partially ordered set or poset. Let (X, ) be a poset. Two elements of a and b of X, are said to be incomparable, written a b, if they satisfy a b (a b) (b a). 19

4. Non-numerical ranking: Introduction 20 Note that for any a X, a a. Two elements a and b of X are indifferent with respect to, written a b [9, 10, 12], if they satisfy ( { a b x } { a x = x b x} ) ( { x } { x a = x } ) x b Note that for a poset (X, ), and a, b X: a b a b a b (Proofs in [9, 10, 12]) { x } { x a = x } x b The poset (X, ) is said to be total or linear, if is the identity relation, i.e., for all a, b X a b b a a = b weak or stratified, if a b c a c, i.e. is an equivalence relation. (see Figure 4.1). (a) Total order (b) Weak order (c) Neither total nor weak Figure 4.1: Various types of partial orders Evidently, every total order is weak. Weak orders are often defined in an alternative way:

4. Non-numerical ranking: Introduction 21 a poset (X, ) is a weak order iff there exists a total order (Y, ) and a mapping φ : X Y such that x, y X x y φ(x) φ(y). The preferable outcome of any ranking is a total order. Note that for any total order, both and are just the identity relation. If is a weak order, then a b a b, hence indifferent is equivalent to incomparable, and the relation can be reinterpreted as a total order on the equivalence classes of (or ). For example, in Figure 4.1(b), we can interpret by: {a} {b, c} {d} 4.2 Weak order approximations Let (X, ) be a poset. The relation may or may not be a weak order. The goal is to look for the best weak order extension of. A weak (or total) order w X X is a proper weak (or total) order extension of if and only if: (x y x w y) and (x w y x y). If X is finite, then every partial order has a weak order extension. If is weak, then its only proper weak order extension is w =. If is not weak, there are usually many such extensions. For non-numerical ranking purposes, the best

4. Non-numerical ranking: Introduction 22 such extension seems to be the one found according to the concept of a global score function, defined by: g (x) = { z z x } { z x z } (where X denotes the cardinal number of a finite set X). Given the global score function g (x), the relation g w X X is defined as a g w b g (a) < g (b). This is a proper weak extension of the partial order. An example of constructing this is shown in Figure 4.2. Weak order approximations are discussed in detail in [8], and it was argued in [12] that the global score weak approximation is the most suitable for our purposes. (a) Partial order with global scores (b) Weak order extension of (a) Figure 4.2: Computing global scores to construct weak order extension 4.3 Non-numerical ranking model In this section, we present Janicki s definition [9, 10, 12] of various types of ranking system satisfying various conditions.

4. Non-numerical ranking: Introduction 23 A pairwise comparisons ranking system [12] PCRS is a relational structure (X, R 0, R 1,..., R k ), where X is the set of objects to be ranked, k 1, and the R i s are binary relations satisfying R 0 R 1... R k = X X. The relation R 0, interpreted as indifference, is symmetric and reflexive, the relations R 1,..., R k, interpreted as preferences, are asymmetric and irreflexive. The value of k should be between 1 and probably 5 due to the limits of the human minds (see [5, 22]). In the rest of this thesis we will follow [9] where k = 4. However adaptations of the results for k = 9 are straightforward. The following model was proposed in [9] and forms the foundation of our approach. X is a finite set of objects to be ranked, and the relations,,, <, and are pairwise disjoint. The interpretation of these relations is the following: a b : a and b are indifferent, a b : slightly in favour of b, a b : in favour of b, a < b : b is strongly better, a b : b is extremely better. The relations,, <, are defined as follows: = < = < < = <

4. Non-numerical ranking: Introduction 24 = The relations,, <, are interpreted as cumulative preferences: a b : b is at least slightly preferable to a a b : b is at least moderately preferable to a a <b : b is at least strongly preferable to a a b : b is at least very strongly preferable to a A relation structure Rank = ( X,,,, <, ) is called a ranking if the following consistency rules are satisfied: 1.,, <, are partial orders 2. =, i.e. 1 = X X 3. ( a b b c ) ( a c a c c a ) 4.1. ( a b b c ) ( a b b c ) ( a c a c ) 4.2. ( a b b c ) ( a b b c ) ( a c a < c ) 4.3. ( a b b < c ) ( a < b b c ) ( a < c a c ) 5. ( a b b c ) ( a b b c ) ( a c ) 6.1. ( a b b c ) ( a c a c ) 6.2. ( a b b c ) ( a b b c ) ( a c a < c ) 6.3. ( a < b b c ) ( a b b < c ) ( a < c a c )

4. Non-numerical ranking: Introduction 25 6.3. ( a < b b c ) ( a b b < c ) ( a < c a c ) 7.1. ( a b b c ) ( a b b c ) ( a c ) 7.2. ( a b b c ) ( a b b c ) ( a c ) 7.3. ( a < b b c ) ( a b b < c ) ( a c ) 7.4. ( a b b c ) ( a b b c ) ( a c ) Unfortunately the rules above are not complete. The complete set will be given and analyzed in the next chapter. A ranking ( X,,,, <, ) is totally (weakly) ordered if the relation is a total (weak) order. Given a ranking Rank = ( X,,,, <, ), let ˆ w be a proper weak extension of and let w = w \ The weakly ordered ranking derived from Rank is then Rank w = ( X,, w,, <, ) It is a weak order extension of Rank. A relational structure ( X,,,, <, ) is called a pairwise comparisons preranking, if the following are satisfied: 1. the relations,,, <, are pairwise disjoint, and their union equals X X 2. is interpreted as equi-preference. 3.,, <, are interpreted as increasing degrees of preference.

4. Non-numerical ranking: Introduction 26 The pre-ranking is usually not a ranking. The goal is to find such a weakly ordered ranking that is the best approximation of a given pre-ranking. After finding the pairs that violate more of the consistency rules than any other pairs, a non-numerical ranking algorithm uses a moderator, which changes the relationship between those pairs, so as to satisfy these rules. The changes are usually minor, such as from to, etc. The resulting ranking Rank is usually not weak. Rank w can then be computed by applying the global score function, illustrated in Figure 4.2. 4.4 Approximations of arbitrary binary relations by partial orders This section is based on [11, 13]. Let X be a set, R be arbitrary relation on X. The relation R may or may not be a partial order. The goal is to find a partial order, which is the best approximation of R. If R is a partial order, then equals R. Let X be a finite set, for every relation R X X, the transitive closure of R is defined as: Let reflexive closure R be R + = i=1 R i R = R id { where id is the identity relation, id = (x, x) } x X. Let subset closure R be defined as: ar b ( br ar ) ( R a R b )

4. Non-numerical ranking: Introduction 27 where Ra = { x } { xra and ar = x } arx. A partial order X X is a partial order approximation of a relation R X X if it satisfies the following conditions: 1. a b = ar + b a R b. 2. a b = ar cyc b (or equivalently a b = br + a) 3. ar b ar b = a b a b 4. a R b = a b The relations (R ) +, (R + ), (R ), (R ) and R are partial order approximations of R that can be derived from R by using operations,, + and, interpreted as follows: (R ) + : remove all cycles from R, then compute the transitive closure of it (R + ) : compute the transitive closure of R, then remove all cycles from it (R ) : compute the subset closure of R, then remove all cycles from it (R ) : remove all cycles from R, then compute the subset closure of it R : compute the subset closure of R An example of arbitrary relation R and its partial order approximations (R + ), (R ) +, (R ), (R ) and R is illustrated in figure 4.3. The relations R, (R ) and (R ) are lower partial order approximations of R (since they are subsets of R), and the relations (R ) + and (R + ) are upper partial order approximations of R (since they are supersets of R). If R is transitive, i.e. R = R +, then (R ) = (R ) = (R ) + = (R + ). If R is a partial order, then R = (R ) = (R ) = (R ) + = (R + ).

4. Non-numerical ranking: Introduction 28 (a) Arbitrary relation R (b) (R + ) (c) (R ) + (d) (R ) (e) (R ) (f) R Figure 4.3: An example of arbitrary relation R and its partial order approximations

Chapter 5 Consistency-driven non-numerical ranking In this chapter, we present the concept of a consistency-driven ranking system. We first present the model of automatic enforcement of consistency, followed by the automatic consistency-driven algorithm and an example. 5.1 Pre-ranking and ranking systems In this section, we define various types of ranking system, corresponding closely (but not identical) to Janicki s ranking systems defined in Chapter 4. A pre-ranking system (PRS) is a relational structure P = (X,,,, <, ), where is a reflexive, symmetric relation on X, and,,, <, are pairwise disjoint binary relations on X such that their union is X X. 29

5. Consistency-driven non-numerical ranking 30 The relations,,, <, are arbitrary, but can be thought of as partial orders, representing preferences of increasing strength. We interpret these (as in Chapter 4) by: a b : a and b are equi-preferable a b : b is slightly preferable to a a b : b is moderately preferable to a a < b : b is strongly preferable to a a b : b is very strongly preferable to a. Definition 5.1.1. Given a PRS P = (X,,,, <, ), the combined PRS for P is the PRS P = (X,, ), where is the union of,, < and. Definition 5.1.2. A ranking system (RS) is a PRS P = (X,,,, <, ), where the cumulative preferences,, <, defined in Chapter 4 (Section 4.2) satisfy: 1.,, <, are partial orders 2. 1 = X X (Note that these are the first two consistency rules in Janicki s definition of a Ranking.)

5. Consistency-driven non-numerical ranking 31 Recall the definitions of the proper weak order extension and the global score function in Sec. 4.2. Definition 5.1.3. The weakly ordered ranking system (WRS) constructed from an PS P is a proper weak order extension of P (see Definition 5.1.1 above) P w = (X, G w, G w), where a G w b G (a) < G (b) a G w b G (a) = G (b) and G (x) = { z z x } { z x z }. If there are more than one criterion to compare, and the criteria are equally important, we define the global score function as followings: G (x) = n G ci (x) i=1 5.2 Automatic enforcement of consistency Definition 5.2.1. A consistent ranking system (CRS) is an RS P = (X,,,, <, ), which satisfies the following consistency rules [16, 29]: 1. (a b b c) (a c a c c a)

5. Consistency-driven non-numerical ranking 32 2.1 (a b b c) (a b b c) (a c a c) 2.2 (a b b c) (a b b c) (a c a < c) 2.3 (a b b < c) (a < b b c) (a < c a c) 3. (a b b c) (a b b c) (a c) 4.1 (a b b c) (a c a c a < c) 4.2 (a b b c) (a b b c) (a c a < c) 4.3 (a < b b c) (a b b < c) (a < c a c) 5.1 (a b b c) (a b b c) (a c) 5.2 (a b b c) (a b b c) (a c) 5.3 (a < b b c) (a b b < c) (a c) 5.4 (a b b c) (a b b c) (a c) 6.1. (a b b < c) (a < b b c) (a c) 6.2. (a < b b < c) (a c) 7.1. (a b b c) (a < c) (a c) 8.1. (a b b c) (a b b c) (a c) (a c) 8.2. (a b b c) (a b b c) (a c) (a > c) 8.3. (a b b > c) (a > b b c) (a > c) (a c) 8.4. (a b b c) (a b b c) (a c)

5. Consistency-driven non-numerical ranking 33 9.1. (a b b c) (a c) (a c) (a > c) 9.2. (a b b c) (a b b c) (a c) (a > c) 9.3. (a b b > c) (a > b b c) (a > c) (a c) 9.4. (a b b c) (a b b c) (a c) 10.1. (a b b c) (a > c) (a c) 10.2. (a b b > c) (a c) 10.3. (a b b c) (a c) 11.1. (a > b b > c) (a c) 11.2. (a > b b c) (a b b > c) (a c) 12.1. (a b b c) (a c) 13.1. (a b b c) (a b b c) (a c a c a c) 13.2. (a b b c) (a b b c) (a c a c) 13.3. (a b b < c) (a < b b c) (a c a < c) 13.4. (a b b c) (a b b c) (a < c a c) 14.1. (a b b c) (a b b c) (a c a c) 14.2. (a b b c) (a b b c) (a c a c a c) 14.3. (a b b < c) (a < b b c) (a c a c) 14.4. (a b b c) (a b b c) (a c a < c a c)

5. Consistency-driven non-numerical ranking 34 15.1. (a > b b c) (a b b > c) (a c a > c) 15.2. (a > b b c) (a b b > c) (a c a c) 15.3. (a > b b < c) (a < b b > c) (a c a c a c a c a c) 15.4. (a > b b c) (a b b > c) (a c a c a < c a c) 16.1. (a b b c) (a b b c) (a > c a c) 16.2. (a b b c) (a b b c) (a c a > c a c) 16.3. (a b b < c) (a < b b c) (a c a c a > c a c) 16.4. (a b b c) (a b b c) (a c a c a c a c a c a > c a < c) These consistency rules are named by grouping. Consistency rules 1-3 follow Janicki s consistency rules 3-5 naming style, but start from 1. For consistency rule 4-16.4, the rules having one precedence same and the other precedence coming from all the enumerations of the preference relations are grouped together, and they have the same starting number (before. ) and different ending number (after. ). Note that these consistency rules extend Janicki s consistency rules 3-7 (in his definition of Ranking) presented in Chapter 4. As opposed to the latter, these rules cover all cases. The set is not minimal: some of the rules can be removed, but at the expense of readability. These consistency rules formalize the idea that composition may change precedences only by one step, or not at all. Consider for example the rule: 2.1. (a b b c) (a b b c) (a c a c).

5. Consistency-driven non-numerical ranking 35 It says that if a and b are equi-preferable and c is slightly preferable to b, then c is at most moderately preferable to a, and similarly for the symmetric case. The complete set of consistency rules is shown in Table 5.1, where the light gray rows indicate that the rules can be removed since they are duplicated. Rule (a, b) (b, c) (a, b) (b, c) (a, c) 1 2.1 2.2 < 2.3 < < < 3 4.1 < 4.2 < 4.3 < < < Comments Same as 5.1 5.1 5.2 5.3 < < 5.4 < < < Same as 2.3 < < < Same as 4.3 6.1 < < 6.2 < < < < Same as 5.3 < Same as 2.2 < Same as 4.2 7.1 < < < Same as 6.1 Same as 5.2 8.1 8.2 > 8.3 > > > Continued on next page

5. Consistency-driven non-numerical ranking 36 Table 5.1 -- continued from previous page Rule (a, b) (b, c) (a, b) (b, c) (a, c) 8.4 Comments Same as 8.1 9.1 > 9.2 > 9.3 > > > 9.4 > Same as 8.2 > Same as 7.2 10.1 > 10.2 > > 10.3 > > > Same as 8.3 > > > Same as 9.3 > > Same as 10.2 11.1 > > 11.2 > > Same as 8.4 Same as 9.4 Same as 10.3 > > Same as 11.2 12.1 13.1 13.2 13.3 < < < 13.4 < 14.1 14.2 14.3 < < 14.4 < 15.1 > > > 15.2 > > Continued on next page

5. Consistency-driven non-numerical ranking 37 Table 5.1 -- continued from previous page Rule (a, b) (b, c) (a, b) (b, c) (a, c) 15.3 > < < > 15.4 > > < 16.1 > 16.2 > 16.3 < < > 16.4 < > Comments Table 5.1: Complete set of consistency rules 5.2.1 Compact set of consistency rules Note that some consistency rules can be inferred from the others in the complete set shown in Table 5.1. By removing these rules, a compact set of consistency rules is obtained, as shown in Table 5.2. The dark grey rows indicate that those rules can be inferred from the others, and can therefore be removed. The light grey rows indicate (as before) that the rules are duplicated, and so can also be deleted. Rule (a, b) (b, c) (a, b) (b, c) (a, c) 1 2.1 2.2 < 2.3 < < < 3 4.1 < 4.2 < 4.3 < < < Comments Same as 5.1 Continued on next page

5. Consistency-driven non-numerical ranking 38 Table 5.2 -- continued from previous page Rule (a, b) (b, c) (a, b) (b, c) (a, c) 5.1 Comments Inferrable from 5.1 < < Inferrable from 5.1 Inferrable from 5.1 < < < Same as 2.3 < < < Same as 4.3 6.1 < < < < Inferrable from 6.1 < < Same as 5.3 < Same as 2.2 < Same as 4.2 7.1 < < < Same as 6.1 Same as 5.2 8.1 8.2 > 8.3 > > > 8.4 Same as 8.1 9.1 > 9.2 > 9.3 > > > 9.4 > Same as 8.2 > Same as 7.2 10.1 > 10.2 > > Inferrable from 10.2 > > > Same as 8.3 > > > Same as 9.3 > > Same as 10.2 Continued on next page

5. Consistency-driven non-numerical ranking 39 Table 5.2 -- continued from previous page Rule (a, b) (b, c) (a, b) (b, c) (a, c) 11.1 > > 11.2 > > Comments Same as 8.4 Same as 9.4 Same as 10.3 > > Same as 11.2 Inferrable from 11.2 13.1 13.2 13.3 < < < 13.4 < 14.1 14.2 14.3 < < 14.4 < 15.1 > > > 15.2 > > 15.3 > < < > 15.4 > > < 16.1 > 16.2 > 16.3 < < > 16.4 < > Table 5.2: Compact set of consistency rules

5. Consistency-driven non-numerical ranking 40 5.3 Automatic consistency-driven algorithm In this section, we present the automatic consistency-driven algorithm, followed by a discussion of the concept of rank reversal. The consistency-driven algorithm takes a PRS, P = (X,,, <, ) as input, and produces a CRS, then calculates the global scores to obtain a WRS, as output, as defined in Chapter 4 (Sec. 4.2) as output. Consistency-Driven Algorithm. 1. Find all pairs that violate consistency rules; if none, proceed to 5. 2. Pick the pairs which violate the biggest number of rules. 3. Revise the relation between the pairs violating a rule, by appropriately lowering preferences, for example from < to, etc. (in general from R i to R i 1, in the notation of Chapter 4). 4. Proceed to 1. 5. Calculate the global scores to obtain the WRS. 6. End. Intuitively, after finding the pairs that violate the most consistency rules, we reduce them and check the revised PRS again until there are no more pairs violating the ranking consistency rules. Proposition 5.3.1. 1. The consistency-driven algorithm always converges.

5. Consistency-driven non-numerical ranking 41 2. Its time complexity is O(n 3 ). 3. In the worst case the outcome is = X X. Proof (1) and (3). From step 3 of the algorithm, we always increase the disorder. In the worst case, we may get = X X, but the procedure always stops. (2) The principle analysis of triples (step 1 in the algorithm) is O(n 3 ). Because of 3, each triple can violate each rule only once, but the number of rules is finite. Calculating global scores is O(n 2 ), so the algorithm is O(n 3 ). The automatic consistency-driven algorithm is shown in Figure 5.1. A PRS is generally not consistent, therefore our goal is to update the pre-ranking system and make it consistent, as shown in Figure 5.2. 5.4 Rank reversal There are two schools of thought about rank reversal. One maintains that new alternatives that introduce no additional attributes should not cause rank reversal under any circumstances. The other maintains that there are situations in which rank reversal is unreasonable, as well as situations where it is to be expected. Either mode is selected according to the problem at hand. Definition 5.4.1. [21] Rank reversal occurs whenever the relative ranking of two alternatives is reversed in the final revised ranking when a new alternative is added. To clarify this:

5. Consistency-driven non-numerical ranking 42 Figure 5.1: Automatic consistency-driven algorithm

5. Consistency-driven non-numerical ranking 43 Figure 5.2: Consistency-driven ranking system state chart Let us suppose that a and b are two alternatives belonging to X and that a is preferred to b in the weakly ordered ranking system. Let c be a new alternative (c / A where A = {a, b}) and A c = A {c}. A rank reversal occurs if b is preferred to a when a pairwise comparison method is applied to A c. Since the pairwise comparisons are transformed to a weakly ordered ranking, there can be some inconsistency in the comparison of two alternatives with the weakly ordered ranking. We therefore give the following definition: Definition 5.4.2. A pairwise comparison rank reversal occurs when there are two alternatives a and b in X such that either: a b in the combined PRS, but b G w a in the WRS; or a b in the combined PRS, but a G w b or b G w a in the WRS.

5. Consistency-driven non-numerical ranking 44 This definition is summarized in table 5.3. WRS PRS a G w b b G w a a G w b a b Rank reversal Rank reversal a b b a Rank reversal Rank reversal Table 5.3: Pairwise comparison rank reversal definition 5.5 Example 1 We present an example due to Dyer [7, 6], based on the following: C 1 C 2 C 3 C 4 A 1 9 1 3 B 9 1 9 1 C 8 1 4 5 D 4 1 8 5 In the following section, we will show how the rank reversal is produced by using the AHP method and how it can be avoided by applying the consistency-driven nonnumerical ranking algorithm. 5.5.1 Solution by the AHP method Assume the four criteria C 1, C 2, C 3 and C 4 are judged to be equally important. 5.5.1.1 AHP method for the first three objects Rankings determined by using the AHP method for the first three alternatives are given by Table 5.4:

5. Consistency-driven non-numerical ranking 45 A C1 A B C weight A 1 1/9 1/8 1/18 B 9 1 9/8 1/2 C 8 8/9 1 4/9 (a) AHP data for criterion C 1 A C3 A B C weight A 1 1/9 1/4 1/14 B 9 1 9/4 9/14 C 4 4/9 1 2/7 (c) AHP data for criterion C 3 A C2 A B C weight A 1 9 9 9/11 B 1/9 1 1 1/11 C 1/9 1 1 1/11 (b) AHP data for criterion C 2 A C4 A B C weight A 1 1/9 1/4 1/3 B 9 1 9/4 1/9 C 4 4/9 1 5/9 (d) AHP data for criterion C 4 Table 5.4: AHP pairwise comparison data for the first three objects The scores are: w A = 1 4 ( 1 18 + 9 11 + 1 14 + 1 3 ) = 0.32 w B = 1 4 (1 2 + 1 11 + 9 14 + 1 9 ) = 0.336 w C = 1 4 (4 9 + 1 11 + 2 7 + 5 9 ) = 0.344 Hence, the ranking order is C > B > A. 5.5.1.2 AHP method for the four objects Rankings determined by using AHP method for the four alternatives are presented in Table 5.5. The scores are: w A = 1 4 ( 1 22 + 9 12 + 1 22 + 3 14 ) = 0.264 w B = 1 4 ( 9 22 + 1 12 + 9 22 + 1 14 ) = 0.243 w C = 1 4 ( 8 22 + 1 12 + 4 22 + 5 14 ) = 0.246

5. Consistency-driven non-numerical ranking 46 A C1 A B C D weight A 1 1/9 1/8 1/4 1/22 B 9 1 9/8 9/4 9/22 C 8 8/9 1 2 8/22 D 4 4/9 1/2 1 4/22 (a) AHP data for criterion C 1 A C3 A B C D weight A 1 1/9 1/4 1/8 1/22 B 9 1 9/4 9/8 9/22 C 4 4/9 1 1/2 4/22 D 8 8/9 2 1 8/12 (c) AHP data for criterion C 3 A C2 A B C D weight A 1 9 9 9 9/12 B 1/9 1 1 1 1/12 C 1/9 1 1 1 1/12 D 1/9 1 1 1 1/12 (b) AHP data for criterion C 2 A C4 A B C D weight A 1 3 3/5 3/5 3/14 B 1/3 1 1/5 1/5 1/14 C 5/3 5 1 1 5/14 D 5/3 5 1 1 5/14 (d) AHP data for criterion C 4 Table 5.5: AHP pairwise comparison data for the four objects w D = 1 4 ( 4 22 + 1 12 + 8 22 + 5 14 ) = 0.246 Therefore, the ranking order for the four objects is A > C D > B, where A and C have rank reversal. 5.5.2 Solution by non-numerical consistency-driven ranking method In this section, we will show how applying the non-numerical consistency-driven algorithm eliminates rank reversal. 5.5.2.1 The first three object case The pre-rankings for criteria C 1, C 2, C 3 and C 4, denoted by P C1, P C2, P C3 and P C4, are presented in Table 5.6 for the first three objects. Among these P C1, P C2 and P C4 are

5. Consistency-driven non-numerical ranking 47 P C1 A B C A B C (a) The first PRS for criterion C 1. This is a CRS. P C2 A B C A B C (b) The first PRS for criterion C 2. This is a CRS. P C4 A B C A B C (c) The first PRS for criterion C 4. This is a CRS. P C3 A B C A B C (d) The first PRS for criterion C 3. It is inconsistent. Violation A B C A 0 1 0 B 1 0 0 C 0 0 0 (e) Numbers of consistency rule violations for the first PRS P C3. The light grey cells show the most inconsistencies. P C 3 A B C A < B > C (f) After one iteration of revision. This is a CRS for criterion C 3. The light grey cells were revised. Table 5.6: Pre-rankings, consistency rule violations and revised consistent ranking consistent and P C3 is inconsistent. The numbers of consistency rule violations for P C3 are presented in Table 5.6(e). Applying the automatic consistency-driven algorithm, after one iteration we obtain the revised CRS P C 3 displayed in Table 5.6(f). In Table 5.6(d), we have A B, which however violates the following consistency rule: (A C) (C B) (A B) (A < B) (by consistency rule 4.2), and similarly for the symmetric pair B A. From the global scores for the first three objects presented in Table 5.7, the ranking order by applying the consistency-driven

5. Consistency-driven non-numerical ranking 48 algorithm is C G w B G w A. G Object G C1 G C2 G C3 G C4 ( 4 ) i=1 G C i A -2 2-2 1-1 B 1-1 2-2 0 C 1-1 0 1 1 Table 5.7: Global scores for the first three objects 5.5.2.2 The four object case The PRS s for criteria C 1, C 2, C 3 and C 4, denoted by P C1, P C2, P C3 and P C4, are presented in Table 5.8 for the four objects. Among them P C2 and P C4 are consistent, and P C1 and P C3 are inconsistent. The numbers of consistency rule violations for P C1 and P C3 are shown in Table 5.9, and the reasons for the violations are shown in Table 5.10. Applying the consistency-driven algorithm, we get CRS s P C 1 and P C 3, shown in Table 5.11. (Recall the definition of CRS in Sec. 5.2.) The global scores for the four objects are shown in Table 5.12. The ranking order found by applying the consistency-driven algorithm for the four objects is C G w D G w B G w A, which is consistent with the ranking order for the first three objects. 5.6 Example 2 Note that if a pre-ranking system contains cycles, the ranking order found by applying the consistency-driven algorithm might not be acceptable. The following example illustrates this.

5. Consistency-driven non-numerical ranking 49 P C1 A B C D A B C D P C2 A B C D A B C D (a) Original PRS for criterion C 1 (b) Original PRS for criterion C 2 P C3 A B C D A B C D P C4 A B C D A B C D (c) Original PRS for criterion C 3 (d) Original PRS for criterion C 4 Table 5.8: Original PRS s for the four objects Violation A B C D A 0 1 1 0 B 1 0 0 0 C 1 0 0 0 D 0 0 0 0 (a) Numbers of consistency rule violations Violation A B C D A 0 1 0 1 B 1 0 0 0 C 0 0 0 0 D 1 0 0 0 (b) Numbers of consistency rule violations for the original PRC P C1. The light grey for the original PRS P C3. The light grey cells show the most inconsistencies. cells show the most inconsistencies. Table 5.9: Numbers of consistency rule violations for the original PRS s P C1 and P C3

5. Consistency-driven non-numerical ranking 50 Violation Consistency Rule Rule Label A B (A D) (D B) (A B) (A < B) Rule 4.2 A C (A D) (D C) (A C) (A < C) Rule 4.2 B A (B D) (D A) (B A) (B > A) Rule 9.2 C A (C D) (D A) (C A) (C > A) Rule 9.2 (a) The explanations for consistency rule violations for the original PRS P C1 Violation Consistency Rule Rule Label A B (A C) (C B) (A B) (A < B) Rule 4.2 A D (A C) (C D) (A D) (A < D) Rule 4.2 B A (B C) (C A) (B A) (B > A) Rule 9.2 D A (D C) (C A) (D A) (D > A) Rule 9.2 (b) The explanations for consistency rule violations for the original PRS P C3 Table 5.10: The explanations for consistency rule violations for the original PRS s P C1 and P C3 P C 1 A B C D A < < B > C > D (a) After one iteration of revision. This is a CRS for criterion C 1. The light grey cells were revised. P C 3 A B C D A < < B > C D > (b) After one iteration of revision. This is a CRS for criterion C 3. The light grey cells were revised. Table 5.11: Revised CRS s for criteria C 1 and C 3

5. Consistency-driven non-numerical ranking 51 G Object C 1 C 2 C 3 C 4 ( 4 ) i=1 G C i A -3 3-3 1-2 B 2-1 2-3 0 C 2-1 -1 1 1 D -1-1 2 1 1 Table 5.12: Global scores for the four objects are: Suppose we have 6 objects A, B, C, D, E and F, and the relations among them A B C D E F A 5.6.1 AHP solution Let the AHP pairwise comparison data be: 1 9 9 9 9 1/9 1/9 1 9 9 9 9 A = 1/9 1/9 1 9 9 9 1/9 1/9 1/9 1 9 9 1/9 1/9 1/9 1/9 1 9 9 1/9 1/9 1/9 1/9 1 Using the AHP method, we obtain the following weights: A = 0.331, B = 0.229, C = 0.146, D = 0.091, E = 0.05, F = 0.152, hence the ranking order is: A > B > F > C > D > E.

5. Consistency-driven non-numerical ranking 52 5.6.1.1 Checking the consistency To find λ max, we solve det[a λi] = 0 That is, det 1 λ 9 9 9 9 1/9 1/9 1 λ 9 9 9 9 1/9 1/9 1 λ 9 9 9 1/9 1/9 1/9 1 λ 9 9 = 0 1/9 1/9 1/9 1/9 1 λ 9 9 1/9 1/9 1/9 1/9 1 λ The maximum solution is The inconsistency index(ci n ) is λ max = 19.491 CI n = λ max n n 1 = (19.491 6) (6 1) = 2.698 The inconsistency ratio(cr n ) is CR n = CI n = 2.698 RI n 1.26 = 2.141 > 0.1 Accordingly the ranking order A > B > F > C > D > E is inconsistent and not acceptable. 5.6.2 Consistency-driven non-numerical ranking solution Let the original PRS be:

5. Consistency-driven non-numerical ranking 53 P A B C D E F A B C D E F Applying the consistency-driven ranking algorithm, we first get the numbers of consistency rule violations for the original PRS shown in Table 5.13(a). After 8 iterations, we obtain the revised CRS shown in Table 5.13(b).

5. Consistency-driven non-numerical ranking 54 Violation A B C D E F A 0 4 3 2 1 4 B 4 0 4 3 2 1 C 3 4 0 4 3 2 D 2 3 4 0 4 3 E 1 2 3 4 0 4 F 4 1 2 3 4 0 (a) Numbers of consistency rule violations for the first PRS. The light grey cells show the most inconsistencies. P A B C D E F A B C D E F (b) After 8 iterations of revisions. This is a CRS produced by the consistency driven algorithm. Table 5.13: Numbers of consistency rule violations for original PRS and revised CRS. The global scores are A = 1, B = 1, C = 2, D = 2, E = 1 and F = 1, hence the ranking order found by applying the consistency-driven algorithm is D G w A G w B G w E G w F G w C, which is not acceptable. In the next chapter, we will introduce the property-driven algorithm to solve this kind of problem.

Chapter 6 Property-driven non-numerical ranking In this chapter, we first present the property-driven classical partial order approximation algorithm, followed by the property-driven refined partial order approximation algorithm, with an example solved by both algorithms. 6.1 Mathematical preliminary In Chapter 4 (Sec. 4.4), the operations +, and were defined for single relations, so for each relation R we could calculate (R + ), (R ) +, (R ), and (R ), and then interpret them as partial order approximation of R. In this section, we extend those three operations to relational systems like pre-ranking systems. Recall from Chapter 5 (Sec. 5.1) the definition of a PRS (pre-ranking system) P = (X,,,, <, ) There maybe two problems with P as given, i.e. two things could be wrong with 55

6. Property-driven non-numerical ranking 56 the original data represented by P. 1. Non-transitivity: There are a, b, c X, such that e.g. a b < c but a c. 2. Cyclicity: There are a, b, c X, such that e.g. a b < c a. Formally transitivity and acyclicity can be defined for P as follows. Denote R 0 =, R 1 =, R 2 =, R 3 = <, and R 4 =. We will say that P = (X, R 0, R 1, R 2, R 3, R 4 ) is transitive if for all a, b, c X, ar i b br j c implies ar k c, where k 0, acyclic if for all a, b X, (a, a) / ( 4 i=0 R i) + \ R0 Note that a b a, etc., is allowed so we have to substract R 0 from in the definition of acyclicity. ( 4 i=0 R i We will now define formally P +, P and P. We will start with P and P as they are simpler cases. First, let us denote i.e. = 4 i=1 R i. = < ) +

6. Property-driven non-numerical ranking 57 Definition 6.1.1. The acyclic refinement of P is the relational structure P = (X,,,, <, ), where : = = < = < = = X X \ ( ( ) ) 1 where is as defined in Chapter 4.4, and = <. E.g. if P has a cycle a b c d < a, then the operation of P revises all the relations between a, b, c and d to a b c d. Directly from the above definition we get: Corollary 6.1.1. P is acyclic. The definition of P is the simplest one, it is just component-wise refinement. Definition 6.1.2. The subset closure of P is the relational system P = (X,,,, <, ), where,, <, are as defined in Chapter 4.4, and = X X \ ( ( ) 1 ) and = <.

6. Property-driven non-numerical ranking 58 It should be noted that all three definitions, P, P and especially P +, are somehow heuristic, since intuition and particular interpretation of the relations,,, <, play very important, if not the most decisive, role. We could not find anything fundamental in the literature on how acyclicity or transitivity should be defined for relational system with more than one relation. Probably it will always depend on particular interpretations and derived properties of particular relations. We do not claim universality of the definition provided in this chapter. They were based on the assumption that our ultimate goal is a reasonable non-numerical ranking systems, and they were not the only ones we have considered. For example transitivity could alternatively be defined as: for all a, b, c X, ar i b br j c implies ar k c, where k i and k j. Perhaps in more general settings we need more than one definition of transitivity. We think the definitions given in this chapter worked the best for our test sample that is discussed in Chapter 9, do not contradict intuition and could relatively easy be merged with other algorithms and results on non-numerical ranking. This comment is specially important for the concept of P +, which turned out to be the most complex, controversial and problematic, and which will be discussed below. To the below definition more readable, we will adopt the following convention, namely the expression like (a b b c a c) a < c should be understood as an equivalence of the following programming statement. if (a b b c a c) then begin < < {(a, c)}; \{(a, c)} end.

6. Property-driven non-numerical ranking 59 Definition 6.1.3. The weak transitive closure of P is the relational system P + = (X,,,, <, ), where,, <, result from executing for each distinct a, b, c X the set of transformations defined below, until all premises are false: 1.1 (a b b c a c) a < c 1.2 ( (a b b c) (a b b c) ) (a c) a < c 1.3 ( (a b b < c) (a < b b c) ) (a c) a c 1.4 ( (a b b c) (a b b c) ) (a c) a c 2.1 (a b b c a c) a c 2.2 ( (a b b < c) (a < b b c) ) (a c) a c 2.3 ( (a b b c) (a b b c) ) (a c) a c 3.1 (a < b b < c a c) a c 3.2 ( (a < b b c) (a b b < c) ) (a c) a c 4.1 (a b b c a c) a c 5.1 (a b b c a c) a > c 5.2 ( (a b b c) (a b b c) ) (a c) a > c 5.3 ( (a b b > c) (a > b b c) ) (a c) a c 5.4 ( (a b b c) (a b b c) ) (a c) a c

6. Property-driven non-numerical ranking 60 6.1 (a b b c a c) a c 6.2 ( (a b b > c) (a > b b c) ) (a c) a c 6.3 ( (a b b c) (a b b c) ) (a c) a c 7.1 (a > b b > c a c) a c 7.2 ( (a > b b c) (a b b > c) ) (a c) a c 8.1 (a b b c a c) a c Consider for example the weak transitive closure rule 1.1: (a b b c a c) a < c. This says that if b is slightly preferable to a, and c is slightly in preferable to b, but a and c are equi-preferable, then we revise the relation between a and c to: c is moderately preferable to a. To make the above definition well-defined, we need to show that the algorithm used converges in finite number of steps and that P + is transitive. Proposition 6.1.1. 1. The algorithm from Definition 6.1.3 converges and its time complexity is O(n 3 ), where n = X. 2. P + is transitive. Proof (1) Since each premise has a factor a c, then each triple {a, b, c} needs to be visited only once, as after changing a c, the criterion ar i b br j c a c from the definition of transitivity is satisfied. (2) The algorithm replaces all questionable a c by either a < c or a c, so the criterion ar i b br j c a c holds.

6. Property-driven non-numerical ranking 61 We called P + the weak transitive closure of P, since it does not satisfy all the properties that we usually required from the name closure. It is usually required that a closure be the smallest entity that has a given property (c.f. [3]). Definition 6.1.3 does not give the the smallest relational structure when it is interpreted as a non-numerical ranking. Take for example the rule 1.2 ((a b b c) (a b b c)) (a c) a < c It could be replaced by ((a b b c) (a b b c)) (a c) a c Proposition 6.1.1 will still be valid, from the first sight this solution looks like the more intuitive one, and it would be smaller (as a c will be replaced by a weaker preference). However the replacement of a c by a < c was chosen. The main reason is that we cannot assume or expect P + to be consistent. The relevant consistency rule for (a b b c) (a b b c) is (see Chapter 5) 4.2 (a b b c) (a b b c) (a c a < c). Our consistency-driven algorithm always degrades the preference when it is inconsistent, and never upgrades it again. If a c, the rule 1.2 will update the pair (a, b) to be a < c. Whenever it is inconsistent, later on, according to our consistency enforcing algorithm, we can always degrade it to be a c. However, if we define our rule to be a c, then whenever it is inconsistent, we could not upgrade it to be a < c any more, so we might loose the preference information (since it might really be a < c). This might at the end lead to a b c

6. Property-driven non-numerical ranking 62 as the final outcome of our consistency-driven algorithm. Using the rule 1.2 will in such case, after enforcing consistency result in a b, b c, a c, or a < c. The tests discussed in Chapter 9 indicated that replacing a c by a < c gives better final consistent outcome than replacing a c by weaker preference a c. Note that if (a b b c), but a c or a c, we always consider it to be transitive (even though it is inconsistent). The similar arguments can be applied to the remaining rules in Definition 6.1.3. 6.2 Property-driven classical partial order approximations of arbitrary relational structures The PRS P = (X,,,, <, ) can be converted to the combined PRS P = (X,, ) by substituting for,, <, and (recalling Definition 5.1.1). The following properties of four of the partial order approximations of a given PRS P have been proved in [12]: The structures (P ) +, (P + ), (P ) and (P ) are ranking systems, approximating P. (P ) (P ) (P ) + (P + ). If P is transitive, i.e. P = P +, then (P ) = (P ) = (P ) + = (P + ). If P is acyclic, i.e. P = P, then P = (P ) = (P ) and (P ) + = (P + ) = P +.

6. Property-driven non-numerical ranking 63 6.2.1 Property-driven classical partial order approximation algorithm The property-driven classical partial order approximation algorithm takes a PRS P as the input, converts it to the combined PRS P and computes one of the partial order approximations (P ) +, (P + ), (P ) and (P ), then calculates the global scores to produce the weakly ordered ranking system (WRS) written (in all 4 cases) as (X, G w, G w). (Recall the definition of weakly ordered ranking in Chapter 5 (Sec. 5.3)). Property-driven classical partial order approximation algorithm. 1. Convert the PRS P = (X,,,, <, ) to P (X,, ) by substituting for,, < and. 2. Compute one of the partial order approximations (P ) +, (P + ), (P ) and (P ). 3. Calculate the global scores for the partial order approximations to obtain the WRS s (X, G w, G w). Proposition 6.2.1. The time complexity of the property-driven classical partial order approximation algorithm is O(n 3 ). Proof Computing P + requires O(n 3 ), computing P requires O(n 2 ), and computing P requires O(n 3 ) (see [4]), and O(n 3 ) + O(n 2 ) = O(n 3 ).

6. Property-driven non-numerical ranking 64 6.2.2 An example We will continue with the example provided by Dyer [6, 7], presented in Chapter 5. 6.2.2.1 Classical partial order approximations for the first three objects The combined PRS s P = (X,, ) for the first three objects based on criteria C 1, C 2, C 3 and C 4, denoted by P C1, P C2, P C3, and P C4, are presented in Table 6.1. P A B C A B C P A B C A B C (a) P for C 1 (b) P for C 2 P A B C A B C P A B C A B C (c) P for C 3 (d) P for C 4 Table 6.1: P for first three objects based on criteria C 1, C 2, C 3 and C 4 The global scores for the first three objects by the classical partial order approximation algorithm are shown in Table 6.2, hence the ranking order for the first three objects is C G w B G w A, the same as that found by the consistency-driven algorithm. 6.2.2.2 Classical partial order approximations for four objects The PRS s for four objects with different criteria are shown in Table 6.3. The global scores for the four objects by the property-driven classical partial order

6. Property-driven non-numerical ranking 65 G C1 (P ) + (P + ) (P ) (P ) A -2-2 -2-2 B 1 1 1 1 C 1 1 1 1 G C2 (P ) + (P + ) (P ) (P ) A 2 2 2 2 B -1-1 -1-1 C -1-1 -1-1 (a) Global scores for criterion C 1 (b) Global scores for criterion G C2 G C3 (P ) + (P + ) (P ) (P ) A -2-2 -2-2 B 2 2 2 2 C 0 0 0 0 G C4 (P ) + (P + ) (P ) (P ) A 1 1 1 1 B -2-2 -2-2 C 1 1 1 1 (c) Global scores for criterion G C3 (d) Global scores for criterion G C4 4 i=1 G C i (P ) + (P + ) (P ) (P ) A -1-1 -1-1 B 0 0 0 0 C 1 1 1 1 (e) Global scores for the first three objects based on criteria C 1, C 2, C 3 and C 4 Table 6.2: Global scores for the first three objects by classical p.o. approximations

6. Property-driven non-numerical ranking 66 P A B C D A B C D (a) P for criterion C 1 P A B C D A B C D (c) P for criterion C 3 P A B C D A B C D (b) P for criterion C 2 P A B C D A B C D (d) P for criterion C 4 Table 6.3: P for the four objects based on criteria C 1, C 2, C 3 and C 4 approximation algorithm are shown in Table 6.4. Accordingly the ranking order by partial order approximations for the four objects is C G w D G w B G w A, the same as the ranking order found by the consistent-driven algorithm. 6.3 Property-driven refined partial order approximations of arbitrary relational structures In refined partial order approximation, we approximate all the partial orders,, <,, instead of the single combined partial order, as in the property-driven classical partial order approximation algorithm (Sec. 6.2.1)

6. Property-driven non-numerical ranking 67 G C1 (P ) + (P + ) (P ) (P ) A -3-3 -3-3 B 2 2 2 2 C 2 2 2 2 D -1-1 -1-1 G C2 (P ) + (P + ) (P ) (P ) A 3 3 3 3 B -1-1 -1-1 C -1-1 -1-1 D -1-1 -1-1 (a) Global scores for criterion C 1 (b) Global scores for criterion C 2 G C3 (P ) + (P + ) (P ) (P ) A -3-3 -3-3 B 2 2 2 2 C -1-1 -1-1 D 2 2 2 2 G C4 (P ) + (P + ) (P ) (P ) A 1 1 1 1 B -3-3 -3-3 C 1 1 1 1 D 1 1 1 1 (c) Global scores for criterion C 3 (d) Global scores for criterion C 4 4 i=1 G C i (P ) + (P + ) (P ) (P ) A -2-2 -2-2 B 0 0 0 0 C 1 1 1 1 D 1 1 1 1 (e) Global scores for criteria C 1, C 2, C 3 and C 4 Table 6.4: Global scores for all four objects by classical p.o. approximations

6. Property-driven non-numerical ranking 68 6.3.1 Property-driven refined partial order approximation algorithm The property-driven refined partial order approximation algorithm takes the PRS P = (X,,,, <, ) as the input, computes one of the refined partial order approximations (P ) +, (P + ), (P ) and (P ), then calculates the global scores to produce a WRS. Property-driven refined partial order approximations algorithm. 1. Compute one of the refined partial order approximations (P ) +, (P + ), (P ) and (P ). 2. Calculate the global scores for the refined partial order approximations to obtain the WRS for (P ) +, (P + ), (P ), and (P ) written (in all 4 cases) as (X, G w, G w). Proposition 6.3.1. The time complexity of the property-driven refined partial order approximation algorithm is O(n 3 ). Proof Computing P + requires O(n 3 ), computing P requires O(n 2 ), and computing P requires O(n 3 ), and O(n 3 ) + O(n 2 ) = O(n 3 ). 6.3.2 An example We use an example of Janicki [11], shown in Figure 6.1, to illustrate the propertydriven refined partial order approximation approach.

6. Property-driven non-numerical ranking 69 P A B C D E F G A B C D E F G Figure 6.1: The structure P and its PRS 6.3.2.1 Refined partial order approximation of P by (P + ) The RS formed from P by constructing (P + ), i.e. forming the weak transitive closure of P and then removing all cycles, is shown in Table 6.5. In Table 6.5(a) the pairs with rankings revised by forming the weak transitive closure are shown in light grey cells, and in Table 6.5(b) the pairs with rankings further revised by removing all cycles are shown with dark grey cells. The structures P, P + and (P + ) are shown in Figure 6.2. (a) P (b) P + (c) (P + ) Figure 6.2: The structures P, P + and (P + )

6. Property-driven non-numerical ranking 70 P + A B C D E F G A > > B > C D < < E F G < (a) P +. The light grey cells indicate revisions formed by computing the weak transitive closure. (P + ) A B C D E F G A > > B > C D < < E F G < (b) (P + ). The dark grey cells indicate revisions formed by removing all cycles. Table 6.5: P + and (P + ) 6.3.2.2 Refined partial order approximation of P by (P ) + The RS formed from P by constructing (P ) +, i.e., removing cycles from P and then forming the weak transitive closure, is shown in Table 6.6(b). In Table 6.6(a) the pairs with rankings revised by removing all cycles are shown with dark grey cells, and in Table 6.6(b) the pairs with rankings further revised by forming the weak transitive closures are shown in light grey cells. The structures P, P and (P ) + are shown in Figure 6.3. 6.3.2.3 Refined partial order approximation of P by (P ) The RS formed from P by constructing (P ), i.e. taking the subset closure of P, and then removing all cycles, is shown in Table 6.7. In Table 6.7(a) the pairs with rankings revised by taking the subset closure are shown with dark grey cells, and in Table 6.7(b) the pairs with rankings further revised by removing all cycles are shown with dark grey cells. The structures P, P and (P ) are shown in Figure 6.4.

6. Property-driven non-numerical ranking 71 P A B C D E F G A B C D E F G (a) P. The dark grey cells indicate revisions by removing all cycles. (P ) + A B C D E F G A > B C D E F G < (b) (P ) +. The light grey cells indicate revisions by taking the weak transitive closure of P. Table 6.6: P and (P ) + (a) P (b) P (c) (P ) + Figure 6.3: The structures P, P and (P ) + (a) P (b) P (c) (P ) Figure 6.4: The structures P, P and (P )

6. Property-driven non-numerical ranking 72 P A B C D E F G A B C D E F G (a) P. The dark grey cells indicate revisions by computing the subset closure of P. (P ) A B C D E F G A B C D E F G (b) (P ). The dark grey cells indicate revisions by removing cycles from P. Table 6.7: P and (P ) 6.3.2.4 Refined partial order approximation of P by (P ) The RS formed from P by constructing (P ), i.e. removing all cycles from P and then computing the subset closure, is shown in Table 6.8. In Table 6.8(a) the pairs with rankings revised by removing all cycles are shown with dark grey cells, and in Table 6.8(b) the pairs with rankings further revised by forming the subset closure are shown in light grey cells. The structures P, P and (P ) are shown in Figure 6.5. (a) P (b) P (c) (P ) Figure 6.5: The structures P, P and (P )

6. Property-driven non-numerical ranking 73 P A B C D E F G A B C D E F G (a) P. The dark grey cells indicate revisions by removing all cycles from P. (P ) A B C D E F G A B C D E F G (b) (P ). The light grey cells indicate revisions by forming the subset closure of (P ). Table 6.8: P and (P ) 6.3.2.5 Global scores and ranking orders The global scores and ranking orders for the partial order approximations (P + ), (P ) +, (P ) and (P ) are shown in Table 6.9. We see that the ranking orders are inconsistent for the different refined partial order approximations.

6. Property-driven non-numerical ranking 74 G A B C D E F G (P + ) 6 2-2 -2-2 0-2 (P ) + 4 0-2 0 0 0-2 (P ) 1-1 0 0 0 0 0 (P ) 2 0-2 0 0 0 0 (a) Global scores Partial Order Approximation Weakly Ordered Ranking Order (P + ) A G w B G w F G w C G w D G w E G w G (P ) + A G w B G w D G w E G w F G w C G w G (P ) A G w C G w D G w E G w F G w G G w B (P ) A G w B G w D G w E G w F G w G G w C (b) Weakly ordered ranking order Table 6.9: Global scores and ranking orders for the refined p.o. approximations

Chapter 7 Property/consistency-driven non-numerical ranking In this chapter, we present the property/consistency-driven algorithm, followed by an example. 7.1 Property/consistency-driven algorithm The property/consistency-driven ranking algorithm is a combination of the propertydriven refined partial order approximation and consistency-driven ranking methods. The algorithm starts with a PRS P. First, it applies the property-driven refined partial order approximation approach to get the refined partial order approximations of P, and then applies the consistency-driven algorithm to obtain the CRS for the refined partial order approximations (recall the definition of consistent ranking system (CRS) in Sec 5.2). The property/consistency-driven algorithm [30] is illustrated in Figure 7.1. It has time complexity O(n 3 ). 75

7. Property/consistency-driven non-numerical ranking 76 Property/consistency-driven algorithm. 1. Compute one of the refined partial order approximations of P by (P ) +, (P + ), (P ), (P ). 2. For each partial order approximation, find all pairs that violate the consistency rules. If none, proceed to 6. 3. Find the pairs which violate the consistency rules most often. 4. Revise the relation between the pairs violating a rule by appropriately lowering preferences, for example from < to, etc. 5. Proceed to 2. 6. Calculate the global scores to obtain the preferred WRS. 7. End. Proposition 7.1.1. 1. The property/consistency-driven algorithm always converges. 2. Its time complexity of property/consistency-driven algorithm is O(n 3 ). 3. In the worst case, the outcome is = X X. Proof. (1) and (3). From step 4 of the algorithm, we always decrease the disorder. In the worst case, we may get = X X, but the procedure always stops. (2) Steps 1 and 2 are property-driven refined partial order approximation algorithm, which has time complexity O(n 3 ) (see Chapter 6 Proposition 6.3.1). Steps 3, 4 and 5 are consistency-driven algorithm, which has time complexity O(n 3 ) (see Chapter 5 Proposition 5.3.1). Calculating global scores is O(n 2 ). So the algorithm is O(n 3 ).

7. Property/consistency-driven non-numerical ranking 77 Figure 7.1: Property/consistency-driven algorithm

7. Property/consistency-driven non-numerical ranking 78 7.2 An example We use the example presented in Chapter 6, provided by Janicki[11], to illustrate how the property/consistency-driven algorithm. 7.2.1 Solution by refined partial order approximation of P by (P ) + The RS obtained from the property-driven refined partial order approximation of P by (P ) +, and the numbers of consistency rule violations, are shown in Table 7.1. In Table 7.1(a), the pairs in the revised relations formed by removing all cycles are shown in the dark grey cells, and the pairs in the revised relations formed by computing the weak transitive closure are shown in the light grey cells. The explanations for the numbers of consistency rule violations are given in Table 7.2. (P ) + A B C D E F G A > B C D E F G < (a) RS for (P ) +. The dark grey cells indicate revisions formed by removing all cycles from P. The light grey cells indicate revisions formed by computing the weak transitive closure. A B C D E F G A 0 0 2 0 0 0 3 B 0 0 0 0 0 0 0 C 2 0 0 0 0 0 0 D 0 0 0 0 0 0 0 E 0 0 0 0 0 0 0 F 0 0 0 0 0 0 0 G 3 0 0 0 0 0 0 (b) Numbers of consistency rule violations. The light grey cells show the most inconsistencies. Table 7.1: RS for (P ) + with the numbers of consistency rule violations Applying the automatic consistency-driven algorithm, after 2 iterations we obtain the CRS (P ) + shown in Table 7.3(b). The light grey cells show the revised pairs.

7. Property/consistency-driven non-numerical ranking 79 Violation Pair Consistency Rule Rule Label A C (A D) (D C) (A C) (A C) (A C) Rule 1 A C (A E) (E C) (A C) (A C) (A C) Rule 1 A > G (A B) (B G) (A G) (A G) Rule 8.1 A > G (A D) (D G) (A G) (A G) (A G) Rule 1 A > G (A E) (E G) (A G) (A G) (A G) Rule 1 C A (C D) (D A) (C A) (C A) (C A) Rule 1 C A (C E) (E A) (C A) (C A) (C A) Rule 1 G < A (G B) (B A) (G A) (G A) Rule 2.1 G < A (G D) (D A) (G A) (G A) (G A) Rule 1 G < A (G E) (E A) (G A) (G A) (G A) Rule 1 Table 7.2: Explanations for the violations of the consistency rules The structures P, (P ) + and consistent (P ) + are shown in Figure 7.2. (P ) + A B C D E F G A > B C D E F G < (a) RS for (P ) + (P ) + A B C D E F G A B C D E F G (b) CRS for (P ) +. After 2 iterations of revisions. This is a pairwise comparison consistent ranking. The light grey cells show revised pairs. Table 7.3: RS and CRS for (P ) +

7. Property/consistency-driven non-numerical ranking 80 (a) P (b) (P ) + (c) Consistent (P ) + Figure 7.2: The structures P, (P ) + and consistent (P ) + 7.2.2 Solution by refined partial order approximation of P by (P + ) The RS obtained from the property-driven refined partial order approximation of P by (P + ), and the numbers of consistency rule violations are shown in Table 7.4. In Table 7.4(a), the pairs in the revised relations formed by computing the weak transitive closure are shown in the light grey cells, and the pairs in the revised relations formed by removing all cycles are shown in the dark grey cells. The explanations for the consistency rule violations are given in Table 7.5. Applying the automatic consistency-driven algorithm, after 5 iterations we obtain the CRS for (P + ), shown in Table 7.6(b). Gray cells indicate pairs that are revised by the consistency rules. The structures P, (P + ) and consistent (P + ) are shown in Figure 7.3.

7. Property/consistency-driven non-numerical ranking 81 (P + ) A B C D E F G A > > B > C D < < E F G < (a) RS for (P + ). The gray cells indicate revisions formed by computing the weak transitive closure. The dark grey cells indicate the revisions formed by removing all cycles. A B C D E F G A 0 0 0 1 2 0 1 B 0 0 0 3 3 0 0 C 0 0 0 0 0 0 0 D 1 3 0 0 0 0 0 E 2 3 0 0 0 0 0 F 0 0 0 0 0 0 0 G 1 0 0 0 0 0 0 (b) Numbers of consistency rule violations. The light grey cells show the maximum number of inconsistencies. Table 7.4: RS for (P + ) with the numbers of consistency rule violations (a) P (b) (P + ) (c) Consistent (P + ) Figure 7.3: The structures P, (P + ) and consistent (P + )

7. Property/consistency-driven non-numerical ranking 82 Violation Pair Consistency Rule Rule Label A > D (A F ) (F D) (A D) (A D) Rule 8.1 A E (A C) (C E) (A E) (A > E) Rule 8.2 A E (A C) (C E) (A E) (A E) Rule 8.1 A > G (A B) (B G) (A G) (A G) Rule 8.1 B > D (B C) (C D) (B D) (B D) Rule 8.1 B > D (B F ) (F D) (B D) (B D) (B D) Rule 1 B > D (B G) (G D) (B D) (B D) (B D) Rule 1 B E (B C) (C E) (B E) (B E) Rule 8.1 B E (B F ) (F E) (B E) (B E) (B E) Rule 1 B E (B G) (G E) (B E) (B E) (B E) Rule 1 D < A (D F ) (F A) (D A) (D A) Rule 2.1 D < B (D C) (C B) (D B) (D A) Rule 2.1 D < B (D F ) (F B) (D B) (D B) (D B) Rule 1 D < B (D G) (G B) (D B) (D B) (D B) Rule 1 E A (E C) (C A) (E A) (E < A) Rule 2.2 E A (E F ) (F A) (E A) (E A) Rule 2.1 E B (E C) (C B) (E B) (E B) Rule 2.1 E B (E F ) (F B) (E B) (E B) (E B) Rule 1 E B (E G) (G B) (E B) (E B) (E B) Rule 1 G < A (G B) (B A) (G A) (G A) Rule 2.1 Table 7.5: Explanations for the consistency rule violations

7. Property/consistency-driven non-numerical ranking 83 (P + ) A B C D E F G A > > B > C D < < E F G < (a) RS for (P + ) (P + ) A B C D E F G A B C D E F G (b) CRS for (P + ). After 5 iterations of revisions. This is a pairwise comparison consistent ranking. The light grey cells show revised pairs. Table 7.6: RS and CRS for (P + ) 7.2.3 Solution by refined partial order approximation of P by (P ) The RS for the property-driven refined partial order approximation of P by (P ) and the numbers of consistency rule violations are shown in Table 7.7. In Table 7.7(a), light grey cells indicate pairs with ranking revised by computing the subset closure, and dark grey cells indicate pairs with rankings revised by removing all cycles. From the numbers of consistency rule violations presented in Table 7.7(b) (all zero), we see that the RS (P ) is consistent. The structures P, (P ) and consistent (P ) are shown in Figure 7.4.

7. Property/consistency-driven non-numerical ranking 84 (P ) A B C D E F G A B C D E F G (a) RS for (P ). The light grey cells indicate revisions by computing the subset closure of P. The dark grey cells indicate revisions by removing all cycles from P. This is a CRS. A B C D E F G A 0 0 0 0 0 0 0 B 0 0 0 0 0 0 0 C 0 0 0 0 0 0 0 D 0 0 0 0 0 0 0 E 0 0 0 0 0 0 0 F 0 0 0 0 0 0 0 G 0 0 0 0 0 0 0 (b) Numbers of consistency rule violations. Table 7.7: RS for (P ) and the numbers of consistency rule violations (a) P (b) (P ) (c) Consistent (P ) Figure 7.4: Example of a given PRS, and the RS and CRS obtained from it

Chapter 8 Converting AHP to CRS In this chapter, we present a method to convert AHP data to a CRS automatically, followed by an example. 8.1 Definition of the new numerical pairwise comparison data For the AHP method, the numerical pairwise comparisons matrix is defined as follows: A = w 1 w 1 w 1 w 2 w 2 w 1 w 2 w 2. w n w 1. w n w 2 w 1 w n w 2 w n. w n w n = a 11 a 12 a 1n a 21 a 22 a 2n... a n1 a n2 a nn 85

8. Converting AHP to CRS 86 where 0 < a ij <. In order to limit 0 < a ij 1, our new numerical pairwise comparison data matrix is defined as below [20]: A = w 1 w 1 + w 1 w 1 w 1 + w 2 w 2 w 2 + w 1 w 2 w 2 + w 2. w n w n + w 1. w n w n + w 2 w 1 w 1 + w n w 2 w 2 + w n =. w n w n + w n r 11 r 12 r 1n r 21 r 22 r 2n... r n1 r n2 r nn where r ij = 1. r ii = 0.5 2. r ij + r ji = 1 w i w i + w j. It satisfies: 3. 1 r ij 1 = ( 1 r ik 1) ( 1 r kj 1) Proof (3) Since r ij = w k w i w j w k = w j w i w i w i + w j, lhs: 1 r ij 1 = w j w i, rhs: ( 1 r ik 1) ( 1 r kj 1) = From 1 r ij 1 = ( 1 r ik 1) ( 1 r ki 1), we get r ij = r ik r kj 1 r ik r kj + 2r ik r kj. The scaled data defined in the new numerical pairwise comparison format can be converted to non-numerical pairwise comparison data. The heuristic mapping is illustrated in Table 8.1. Basically this heuristic mapping comes from the idea that we initially divide the range [0.5 1] into five equal segments for the different preferences, i.e., [0.5 0.6)

8. Converting AHP to CRS 87 for, [0.6 0.7) for, [0.7 0.8) for, [0.8 0.9) for >, [0.9 1.0] for. For our purpose, we would like to have a bigger interval for and a smaller interval for. So we adjust the initial distributed segments slightly, which leads to the mapping below. Note that the distributed segments may need to be modified so that the mapping satisfies the consistency rules. It should be emphasized that there are usually more than one such mapping. New AHP Value r ij Relative Importance(i,j) Non-numerical Value [0.5 0.56) Equi-preference [0.56 0.66) Slightly preference [0.66 0.76) Moderately preference [0.76 0.86) Strongly preference > [0.86 1] Very strongly preference Table 8.1: Heuristic mapping from new AHP data to non-numerical data Note that the non-numerical pairwise comparison data converted from the new AHP data are always consistent. The proof is shown in Table 8.2. 8.2 An example We continue with the following example [6] discussed in Chapter 5. C 1 C 2 C 3 C 4 A 1 9 1 3 B 9 1 9 1 C 8 1 4 5 D 4 1 8 5

8. Converting AHP to CRS 88 AHP Value r ik AHP Value r kj AHP Value r ij NN ik NN kj NN ij Consistency Rule [0.5 0.56) [0.5 0.56) [0.5 0.62) Rule 1 [0.56 0.66] [0.56 0.72) Rule 8.1 [0.66 0.76) [0.66 0.81) > Rule 8.2 [0.76 0.86) [0.76 0.89) > > Rule 8.3 [0.86 1] [0.86 1] Rule 8.4 [0.56 0.66) [0.5 0.56) [0.56 0.72) Rule 8.1 [0.56 0.66) [0.62 0.79) > Rule 9.1 [0.66 0.76) [0.71 0.86) > Rule 9.2 [0.76 0.86) [0.80 0.92) > > Rule 9.3 [0.86 1] [0.89 1) Rule 9.4 [0.66 0.76) [0.5 0.56) [0.66 0.80) > Rule 8.2 [0.56 0.66) [0.71 0.86) > Rule 9.2 [0.66 0.76) [0.79 0.91) > Rule 10.1 [0.76 0.86) [0.86 0.95) > Rule 10.2 [0.86 1] [0.93 1] Rule 10.3 [0.76 0.86) [0.5 0.56) [0.76 0.89) > > Rule 8.3 [0.56 0.66) [0.80 0.92) > > Rule 9.3 [0.66 0.76) [0.86 0.95) > Rule 10.2 [0.76 0.86) [0.91 0.97) > > Rule 11.1 [0.86 1] [0.95 1] > Rule 11.2 [0.86 1] [0.5 0.56) [0.86 1] Rule 8.4 [0.56 0.66) [0.89 1] Rule 9.4 [0.66 0.76) [0.92 1] Rule 10.3 [0.76 0.86) [0.95 1] > Rule 11.2 [0.86 1] [0.97 1] Rule 12.1 Table 8.2: A proof of consistency of converted data

8. Converting AHP to CRS 89 8.2.1 Automatically generated non-numerical data In Chapter 5, the PRS s were generated manually for this example, and they could be inconsistent. In this chapter, we will show how to automatically generate a CRS (consistent ranking system). 8.2.1.1 Considering three objects, A, B and C: For criterion C 1, which has w A = 1, w B = 9 and w C = 8, we get the new numerical pairwise comparison data shown in Table 8.3. By means of the heuristic mapping shown in Table 8.1, the new numerical pairwise comparison data are converted to non-numerical data, as shown in Table 8.4(a) and the numbers of consistency rule violations are shown in Table 8.4(b). From the latter (all zero), we see that the converted non-numerical pairwise comparison data for criterion C 1 are consistent. The global scores are G A = 2, G B = 1 and G C = 1. A C1 A B C A 0.5 0.1 0.11 B 0.9 0.5 0.53 C 0.89 0.47 0.5 Table 8.3: New AHP data for criterion C 1 For criterion C 2, which has w A = 9, w B = 1 and w C = 1, we get the new numerical pairwise comparison data shown in Table 8.5. The new numerical data are converted to non-numerical pairwise comparison data, as shown in Table 8.5(a). By Table 8.1, A C2 can be converted to the non-numerical pairwise comparison matrix shown in

8. Converting AHP to CRS 90 P C1 A B C A B C (a) CRS for criterion C 1 Violation A B C A 0 0 0 B 0 0 0 C 0 0 0 (b) Numbers of consistency rule violations Table 8.4: Converted CRS for criterion C 1 Table 8.6(a), and the numbers of consistency rule violations are shown in Table 8.6(b). From the latter, we see that the converted non-numerical pairwise comparison data for criterion C 2 are consistent. The global scores are G A = 2, G B = 1 and G C = 1. A C2 A B C A 0.5 0.9 0.9 B 0.1 0.5 0.5 C 0.1 0.5 0.5 Table 8.5: New AHP data for criterion C 2 P C2 A B C A B C (a) CRS for criterion C 2 Violation A B C A 0 0 0 B 0 0 0 C 0 0 0 (b) Numbers of consistency rule violations Table 8.6: Converted CRS for criterion C 2

8. Converting AHP to CRS 91 For criterion C 3, which has w A = 1, w B = 9 and w C = 4, the new numerical pairwise comparison data are shown in Table 8.7. The corresponding non-numerical pairwise comparison data, and the numbers of consistency rule violations, are shown in Table 8.8. The global scores are G A = 2, G B = 2 and G C = 0. A C3 A B C A 0.5 0.1 0.2 B 0.9 0.5 0.69 C 0.8 0.31 0.5 Table 8.7: New AHP data for criterion C 3 P C3 A B C A < B C > (a) CRS for criterion C 3 Violation A B C A 0 0 0 B 0 0 0 C 0 0 0 (b) Numbers of consistency rule violations Table 8.8: Converted CRS for criterion C 3 For criterion C 4, which has w A = 3, w B = 1 and w C = 5, the new numerical pairwise comparison data are shown in Table 8.9. The corresponding non-numerical pairwise data, and the numbers of consistency rule violations, are shown in Table 8.10. The global scores are: G A = 0, G B = 2 and G C = 2. The summary of global scores for the first three objects is given in Table 8.11. We see that the weakly ordered ranking is C G w B G w A.

8. Converting AHP to CRS 92 A C4 A B C A 0.5 0.75 0.38 B 0.25 0.5 0.17 C 0.62 0.83 0.5 Table 8.9: New AHP data for criterion C 4 P C4 A B C A B < C > (a) CRS for criterion C 4 Violation A B C A 0 0 0 B 0 0 0 C 0 0 0 (b) Numbers of consistency rule violations Table 8.10: Converted CRS for criterion C 4 G G C1 G C2 G C3 G C4 ( 4 i=1 G ) C i A -2 2-2 0-2 B 1-1 2-2 0 C 1-1 0 2 2 Table 8.11: Global scores for the first three objects

8. Converting AHP to CRS 93 8.2.1.2 Considering four objects, A, B, C and D: For criterion C 1, which has w A = 1, w B = 9, w C = 8 and w D = 4, the new numerical pairwise comparison data are shown in Table 8.12. The corresponding non-numerical pairwise comparison data, and numbers of consistency rule violations are shown in Table 8.13. The converted non-numerical pairwise comparison data are consistent, and their global scores are G A = 3, G B = 2, G C = 2 and G D = 1. A C1 A B C D A 0.5 0.1 0.11 0.2 B 0.9 0.5 0.53 0.69 C 0.89 0.47 0.5 0.67 D 0.8 0.31 0.33 0.5 Table 8.12: New AHP data for criterion C 1 P C1 A B C D A < B C D > (a) CRS for criterion C 1 Violation A B C D A 0 0 0 0 B 0 0 0 0 C 0 0 0 0 D 0 0 0 0 (b) Numbers of consistency rule violations Table 8.13: Converted CRS for criterion C 1 For criterion C 2, which has w A = 9, w B = 1, w C = 1 and w D = 1, the new numerical pairwise comparison data are shown in Table 8.14. The corresponding non-numerical pairwise comparison data, and numbers of consistency rule violations,

8. Converting AHP to CRS 94 are shown in Table 8.15. The converted non-numerical pairwise comparison data are consistent, and their global scores are G A = 3, G B = 1, G C = 1 and G D = 1. A C2 A B C D A 0.5 0.9 0.9 0.9 B 0.1 0.5 0.5 0.5 C 0.1 0.5 0.5 0.5 D 0.1 0.5 0.5 0.5 Table 8.14: New AHP data for criterion C 2 P C2 A B C D A B C D (a) CRS for criterion C 2 Violation A B C D A 0 0 0 0 B 0 0 0 0 C 0 0 0 0 D 0 0 0 0 (b) Numbers of consistency rule violations Table 8.15: Converted CRS for criterion C 2 For criterion C 3, which has w A = 1, w B = 9, w C = 4 and w D = 8, the new numerical pairwise comparison data are shown in Table 8.16. The corresponding non-numerical pairwise comparison data, and numbers of consistency rule violations, are shown in Table 8.17. The converted non-numerical pairwise comparison data are consistent and their global scores are G A = 3, G B = 2, G C = 1 and G D = 2. For criterion C 4, which has w A = 3, w B = 1, w C = 5 and w D = 5, the new numerical pairwise comparison data are shown in Table 8.18. Its corresponding non-

8. Converting AHP to CRS 95 A C3 A B C D A 0.5 0.1 0.2 0.11 B 0.9 0.5 0.69 0.53 C 0.8 0.31 0.5 0.33 D 0.89 0.47 0.67 0.5 Table 8.16: New AHP data for criterion C 3 P C3 A B C D A < B C > D (a) CRS for criterion C 3 Violation A B C D A 0 0 0 0 B 0 0 0 0 C 0 0 0 0 D 0 0 0 0 (b) Numbers of consistency rule violations Table 8.17: Converted CRS for criterion C 3

8. Converting AHP to CRS 96 numerical pairwise comparison data, and numbers of consistency rule violations, are shown in Table 8.19. The converted non-numerical pairwise comparison data are consistent, and their global scores are G A = 1, G B = 3, G C = 2 and G D = 2. A C4 A B C D A 0.5 0.75 0.38 0.38 B 0.25 0.5 0.17 0.17 C 0.62 0.83 0.5 0.5 D 0.62 0.83 0.5 0.5 Table 8.18: New AHP data for criterion C 4 P C4 A B C D A B < < C > D > (a) CRS for criterion C 4 Violation A B C D A 0 0 0 0 B 0 0 0 0 C 0 0 0 0 D 0 0 0 0 (b) Numbers of consistency rule violations Table 8.19: Converted CRS for criterion C 4 A summary of global scores for the four objects is shown in Table 8.20. We can see that the non-numerical ranking order for the four objects is C G w D G w B G w A, which is consistent with the ranking results for the first three objects. If we divide the range [0.5 1] into the following five segments, i.e., [0.5 0.65) for, [0.65 0.76) for, [0.76 0.86) for, [0.86 0.95) for >, [0.95 1.0] for, then the transformed PRS s will be the same as in Example 1 in Chapter 5, which

8. Converting AHP to CRS 97 G Object G C1 G C2 G C3 G C4 ( 4 i=1 G ) C i A -3 3-3 -1-4 B 2-1 2-3 0 C 2-1 -1 2 2 D -1-1 2 2 2 Table 8.20: Global scores for the four objects have been shown to be inconsistent. 8.2.1.3 Ranking by partial orders approximations By heuristically converting pairwise comparison data, the generated RS for P is always consistent, and the following holds: (P ) + = (P + ) = (P ) = (P ) = P Therefore, the three types of partial order approximations: property-driven classical partial order approximations, property-driven refined partial order approximations and property/consistency-driven refined partial order approximations, all produce the same ranking orders as the consistency-driven algorithm for the heuristically converted RS.

Chapter 9 Testing and Experimenting The consistency-driven, property-driven and property/consistency-driven algorithms can be used to calculate the ranking relations for a given non-numerical pairwise comparison pre-ranking data set, but how can we test whether the results of those algorithms are correct? In 2007 Waldemar Koczkodaj [17] suggested to use the method of a blindfolded person comparing the weights of stones. The person put one stone in his left hand and another in his right, and decided which of the relations,,, <, or held. The experiment was repeated for the same set of stones by various people in Summers 2007 and 2008. In this experiment the stones can be weighted using precise scale, so we have the precise results to test against. There were 60 stones, are weighed and numbered. We did 50 experiments in total. For each experiment, we randomly chose a subset of the stones, ranging from 4 to 60 stones, to compare their weights and decide the relations for every pair. For the experiments presented in this chapter, due to limited space, we chose 16 98

9. Testing and Experimenting 99 stones (objects) as the experimental set. From their weights as shown in Table 9.1, the ranking order for the 16 stones should be: K F M B N J L A P G O H I D C E Stones A B C D E F G H Weights (g) 570 910 230 250 130 1010 350 310 Stones I J K L M N O P Weights (g) 280 640 1100 590 1000 840 350 400 Table 9.1: The weights of the set of stones 9.1 Test 1: Pre-ranking with acyclicity and transitivity properties The first PRS P = (X,,,, <, ), with acyclicity and transitivity properties, is presented in Table 9.2.

9. Testing and Experimenting 100 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.2: Test 1: The first PRS with acyclicity and transitivity properties Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 B 0 0 0 0 0 3 0 0 0 0 0 7 2 3 0 10 C 0 0 0 0 0 0 1 2 1 2 0 1 0 0 1 2 D 0 0 0 0 0 0 1 2 1 2 0 1 0 0 1 2 E 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 F 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 1 2 1 0 0 0 1 H 0 0 2 2 0 0 0 0 1 0 2 0 0 5 0 1 I 0 0 1 1 0 0 0 1 0 1 2 1 0 0 0 1 J 3 0 2 2 1 0 1 0 1 0 6 1 7 7 1 0 K 0 0 0 0 0 0 2 2 2 6 0 2 0 0 2 3 L 0 7 1 1 1 0 1 0 1 1 2 0 6 5 1 0 M 0 2 0 0 0 0 0 0 0 7 0 6 0 0 0 0 N 0 3 0 0 0 0 0 5 0 7 0 5 0 0 0 0 O 0 0 1 1 0 0 0 0 0 1 2 1 0 0 0 1 P 0 10 2 2 0 0 1 1 1 0 3 0 0 0 1 0 Table 9.3: Test 1: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies.

9. Testing and Experimenting 101 9.1.1 Solution by consistency-driven algorithm Note that the first PRS from Table 9.2 is inconsistent, and the numbers of consistency rule violations are shown in Table 9.3. Iteration I From the numbers of consistency rule violations of the first pre-ranking presented in Table 9.3, we see that pairs (B, P ) and (P, B) have the most inconsistencies. After the 1st iteration, the 2nd (revised) PRS P = (X,,,, <, ) is presented in Table 9.4. The pair (B, P ) has been revised from to > and (P, B) correspondingly from to <. The 2nd pairwise comparison pre-ranking system presented in Table 9.4 is inconsistent, and the numbers of consistency rule violations are shown in Table 9.5. P A B C D E F G H I J K L M N O P A > B > > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < < Table 9.4: Test 1: The 2nd (revised) PRS. Light grey cells were revised.

9. Testing and Experimenting 102 Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 B 0 0 0 0 0 2 0 0 0 0 0 7 1 3 0 0 C 0 0 0 0 0 0 1 2 1 2 0 1 0 0 1 1 D 0 0 0 0 0 0 1 2 1 2 0 1 0 0 1 1 E 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 F 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 1 2 1 0 0 0 0 H 0 0 2 2 0 0 0 0 1 0 2 0 0 5 0 0 I 0 0 1 1 0 0 0 1 0 1 2 1 0 0 0 0 J 3 0 2 2 1 0 1 0 1 0 6 1 7 7 1 0 K 0 0 0 0 0 0 2 2 2 6 0 2 0 0 2 3 L 0 7 1 1 1 0 1 0 1 1 2 0 6 5 1 0 M 0 1 0 0 0 0 0 0 0 7 0 6 0 0 0 0 N 0 3 0 0 0 0 0 5 0 7 0 5 0 0 0 0 O 0 0 1 1 0 0 0 0 0 1 2 1 0 0 0 0 P 0 0 1 1 0 0 0 0 0 0 3 0 0 0 0 0 Table 9.5: Test 1: Numbers of consistency rule violations for the 2nd PRS. Light grey cells have the most inconsistencies. Iteration II From the numbers of consistency rule violations of the 2nd PRS shown in Table 9.5, we see that the pairs (B, L), (J, M), (J, N), and their symmetric counterpart pairs have the most inconsistencies. After the 2nd iteration, we get the 3rd revised PRS P = (X,,,, <, ), shown in Table 9.6. The pair (B, L) has been revised from > to, (J, M) from < to, and (J, N) from to, and their symmetric counterparts accordingly. The 3rd PRS shown in Table 9.6 is still inconsistent, and the numbers of consistency rule violations are shown in Table 9.7.

9. Testing and Experimenting 103 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > K > > > L > < < M > > > > > > > > N > > O < < < P < < < Table 9.6: Test 1: The 3rd (revised) PRS. Light grey cells were revised. Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 B 0 0 0 0 0 1 0 0 0 0 0 0 1 3 0 0 C 0 0 0 0 0 0 1 2 1 1 0 1 0 0 1 1 D 0 0 0 0 0 0 1 2 1 1 0 1 0 0 1 1 E 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 F 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 0 2 1 0 0 0 0 H 0 0 2 2 0 0 0 0 1 0 2 0 0 5 0 0 I 0 0 1 1 0 0 0 1 0 0 2 1 0 0 0 0 J 1 0 1 1 0 0 0 0 0 0 8 1 0 0 0 0 K 0 0 0 0 0 0 2 2 2 8 0 2 0 0 2 3 L 0 0 1 1 1 0 1 0 1 1 2 0 6 6 1 0 M 0 1 0 0 0 0 0 0 0 0 0 6 0 0 0 0 N 0 3 0 0 0 0 0 5 0 0 0 6 0 0 0 0 O 0 0 1 1 0 0 0 0 0 0 2 1 0 0 0 0 P 0 0 1 1 0 0 0 0 0 0 3 0 0 0 0 0 Table 9.7: Test 1: Numbers of consistency rule violations for the third PRS. Light grey cells have the most inconsistencies.

9. Testing and Experimenting 104 Iteration III From the numbers of consistency rule violations of the 3rd preranking, shown in Table 9.7, we see that the pairs (J, K) and (K, J) have the most inconsistencies. After the 3rd iteration, we obtain 4th PRS P = (X,,,, <, ) shown in Table 9.8. The pair (J, K) has been revised from to <, and (K, J) correspondingly from to >. The 4th PRS presented in Table 9.8 is still inconsistent, and the numbers of consistency rule violations are shown in Table 9.9. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > > L > < < M > > > > > > > > N > > O < < < P < < < Table 9.8: Test 1: The 4th (revised) PRS. Light grey cells were revised.

9. Testing and Experimenting 105 Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 B 0 0 0 0 0 1 0 0 0 0 0 0 1 3 0 0 C 0 0 0 0 0 0 1 2 1 0 0 1 0 0 1 1 D 0 0 0 0 0 0 1 2 1 0 0 1 0 0 1 1 E 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 F 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 0 2 1 0 0 0 0 H 0 0 2 2 0 0 0 0 1 0 2 0 0 5 0 0 I 0 0 1 1 0 0 0 1 0 0 2 1 0 0 0 0 J 1 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 K 0 0 0 0 0 0 2 2 2 3 0 2 0 0 2 3 L 0 0 1 1 1 0 1 0 1 0 2 0 6 6 1 0 M 0 1 0 0 0 0 0 0 0 0 0 6 0 0 0 0 N 0 3 0 0 0 0 0 5 0 0 0 6 0 0 0 0 O 0 0 1 1 0 0 0 0 0 0 2 1 0 0 0 0 P 0 0 1 1 0 0 0 0 0 0 3 0 0 0 0 0 Table 9.9: Test 1: Numbers of consistency rule violations for the 4th PRS. Light grey cells have the most inconsistencies. Iteration IV From the numbers of consistency rule violations of the 4th preranking, presented in Table 9.9, we see that the pairs (L, M), (M, L), (L, N) and (N, L) have the most inconsistencies. After the 4th iteration, we obtain the 5th PRS P = (X,,,, <, ) shown in Table 9.10. The pair (L, M) has been revised from < to, (L, N) from to, and their symmetric counterparts accordingly. The 5th PRS shown in Table 9.10 is still inconsistent, and the numbers of consistency rule violations are shown in Table 9.11.

9. Testing and Experimenting 106 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > > L > < M > > > > > > > N > > O < < < P < < < Table 9.10: Test 1: The 5th (revised) PRS. Light grey cells were revised. Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 B 0 0 0 0 0 1 0 0 0 0 0 0 1 2 0 0 C 0 0 0 0 0 0 1 2 1 0 0 0 0 0 1 1 D 0 0 0 0 0 0 1 2 1 0 0 0 0 0 1 1 E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 F 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 H 0 0 2 2 0 0 0 0 1 0 2 0 0 5 0 0 I 0 0 1 1 0 0 0 1 0 0 2 0 0 0 0 0 J 1 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 K 0 0 0 0 0 0 2 2 2 3 0 2 0 0 2 3 L 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 M 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 N 0 2 0 0 0 0 0 5 0 0 0 0 0 0 0 0 O 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 P 0 0 1 1 0 0 0 0 0 0 3 0 0 0 0 0 Table 9.11: Test 1: Numbers of consistency rule violations for the 5th PRS. Light grey cells have the most inconsistencies.

9. Testing and Experimenting 107 Iteration V From the numbers of consistency rule violations of the 5th preranking, shown in Table 9.11, we see that the pairs (H, N) and (N, H) have the most inconsistencies. After the 5th iteration, we obtain the 6th (revised) PRS P = (X,,,, <, ), shown in Table 9.12. The pair (H, N) has been revised from < to, and (N, H) correspondingly from > to. The 6th PRS shown in Table 9.12 is still inconsistent, and the numbers of consistency rule violations are shown in Table 9.13. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < I < < < J > < K > > > > L > < M > > > > > > > N > O < < < P < < < Table 9.12: Test 1: The 6th (revised) PRS. Light grey cells were revised.

9. Testing and Experimenting 108 Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 B 0 0 0 0 0 1 0 0 0 0 0 0 1 2 0 0 C 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 D 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 F 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 H 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 I 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 J 1 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 K 0 0 0 0 0 0 2 2 2 3 0 2 0 0 2 3 L 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 M 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 N 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 O 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 P 0 0 1 1 0 0 0 0 0 0 3 0 0 0 0 0 Table 9.13: Test 1: Numbers of consistency rule violations for the 6th PRS. Light grey cells have the most inconsistencies. Iteration VI From the numbers of consistency rule violations of the 6th preranking, shown in Table 9.13, we see that the pairs (J, K), (K, P ) and their symmetric counterpart pairs have the most inconsistencies. After the 6th iteration, we get the 7th (revised) PRS P = (X,,,, <, ), shown in Table 9.14. The pair (J, K) has been revised from < to, (K, P ) from to >, and their symmetric counterpart pairs accordingly. The 7th PRS shown in Table 9.14 is still inconsistent and the numbers of consistency rule violations are shown in Table 9.15.

9. Testing and Experimenting 109 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < I < < < J > K > > > > L > < M > > > > > > > N > O < < < P < < < < Table 9.14: Test 1: The 7th (revised) PRS. Light grey cells were revised. Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 B 0 0 0 0 0 1 0 0 0 0 0 0 1 2 0 0 C 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 0 D 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 0 E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 F 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 H 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 I 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 J 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 K 0 0 0 0 0 0 2 2 2 0 0 2 0 0 2 0 L 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 M 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 N 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 O 0 0 1 1 0 0 0 0 0 0 2 0 0 0 0 0 P 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Table 9.15: Test 1: Numbers of consistency rule violations for the 7th PRS. Light grey cells have the most inconsistencies.

9. Testing and Experimenting 110 Iteration VII From the 7th pre-ranking consistency rule violation numbers shown in Table 9.15, we see that the pairs (B, N), (G, K), (H, K), (I, K), (K, L), (K, O) and their symmetric counterparts have the most inconsistencies. After the 7th iteration, we obtain the 8th (revised) PRS P = (X,,,, <, ), shown in Table 9.16. The pair (B, N) has been revised from to, (G, K) from to <, (H, K) from to <, (I, K) from to <, (K, L) from > to, (K, O) from to >, and their symmetric counterparts are accordingly. From the 8th pre-ranking consistency rule violation numbers shown in Table 9.17, we see that after 7 iterations, the revised PRS, as shown in Table 9.16 is finally consistent! From the global scores shown in Table 9.18, we infer the ranking order produced by the consistency-driven algorithm: K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E, which has no rank reversal and is hence acceptable. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < < J > K > > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.16: Test 1: The 8th (revised) PRS. It is consistent. Light grey cells were revised.

9. Testing and Experimenting 111 Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 B 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 E 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 F 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 H 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 I 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 J 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 K 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 L 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 N 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 O 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 P 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Table 9.17: Test 1: Numbers of consistency rule violations for the 8th PRS Object k F M B N J A L P G O H I D C E G 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 Table 9.18: Test 1: Global scores produced by the consistency-driven algorithm 9.1.2 Solution by property-driven algorithm 9.1.2.1 Property-driven classical partial order approximations The first PRS P = (X,,,, <, ), shown in table 9.2, can be converted to combined PRS P = (X,, ) shown in Table 9.19, by substituting for,, < and.

9. Testing and Experimenting 112 P A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.19: Test 1: P (X,, ) for the first combined PRS This combined PRS P has the acyclicity and transitivity properties, and so P + = P = P Classical partial order approximation of P by (P ) + The revised PRS, formed from P by removing all cycles and then computing the transitive closure, denoted by (P ) +, is shown in Table 9.20. Since P is acyclic and transitive, (P ) + = P.

9. Testing and Experimenting 113 (P ) + A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.20: Test 1: Property-driven classical partial order approximation of P by (P ) + Classical partial order approximation of P by (P + ) The revised PRS, found from P by computing the transitive closure and then removing all cycles, denoted by (P + ), is shown in Table 9.21. Since P is acyclic and transitive, (P + ) = P.

9. Testing and Experimenting 114 (P + ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.21: Test 1: Property-driven classical partial order approximation of P by (P + ) Classical partial order approximation of P by (P ) and (P ) P is acyclic and transitive, thus (P ) = (P ) = P = P. The systems (P ) and (P ) are shown in Table 9.22.

9. Testing and Experimenting 115 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.22: Test 1: Property-driven classical partial order approximations of P by (P ) and (P ) Global scores The global scores produced by the property-driven classical partial order approximations are shown in Table 9.23. The ranking orders for the four classical partial order approximations are the same, namely: K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wi G wd G wc G we, which has no rank reversals, and is therefore acceptable. G K F M B N J A L P G O H I D C E (P ) + 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 (P + ) 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 (P ) 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 (P ) 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 Table 9.23: Test 1: Global scores produced by the property-driven classical partial order approximations

9. Testing and Experimenting 116 9.1.2.2 Property-driven refined partial order approximations The pairwise comparison pre-ranking system PRS P = (X,,,, <, ) shown in Table 9.2 is acyclic and transitive, and so (P ) + = (P + ) = (P ) = (P ) = P. Global scores The global scores produced by the property-driven refined partial order approximations are shown in Table 9.24. The ranking order produced by the property-driven refined partial order approximations is: K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E, which is the same as that produced by the classical partial order approximations, and hence acceptable. G K F M B N J A L P G O H I D C E (P ) + 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 (P + ) 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 (P ) 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 (P ) 15 12 12 9 7 5 2 2 1 4 4 7 9 11 13 15 Table 9.24: Test 1: Global scores produced by the property-driven refined p.o. approx.

9. Testing and Experimenting 117 9.2 Test 2 : Pre-ranking with cycles An example of PRS P = (X,,,, <, ) with cycles, is shown in Table 9.25. Light grey cells indicate a cycle, i.e., C D, D I and I C. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.25: Test 2: The first PRS with cycles. Light grey cells indicate a cycle. 9.2.1 Solution by consistency-driven algorithm The original PRS with cycles is shown in Table 9.25. The numbers of consistency rule violations are shown in Table 9.26.

9. Testing and Experimenting 118 Violation A B C D E F G H I J K L M N O P A 0 0 1 1 0 0 0 0 0 5 0 0 0 0 0 0 B 0 0 0 0 0 3 0 0 0 0 0 7 2 3 0 10 C 1 0 0 1 0 0 2 4 2 0 0 2 0 0 2 3 D 1 0 1 0 0 0 2 0 3 0 0 2 0 0 2 3 E 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 F 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 2 2 0 0 0 0 0 3 2 1 0 0 0 1 H 0 0 4 0 0 0 0 0 0 2 1 0 0 3 0 1 I 0 0 2 3 0 0 0 0 0 2 1 1 0 0 0 1 J 5 0 0 0 1 0 3 2 2 0 4 3 5 5 3 2 K 0 0 0 0 0 0 2 1 1 4 0 2 0 0 2 3 L 0 7 2 2 1 0 1 0 1 3 2 0 6 5 1 0 M 0 2 0 0 0 0 0 0 0 5 0 6 0 0 0 0 N 0 3 0 0 0 0 0 3 0 5 0 5 0 0 0 0 O 0 0 2 2 0 0 0 0 0 3 2 1 0 0 0 1 P 0 10 3 3 0 0 1 1 1 2 3 0 0 0 1 0 Table 9.26: Test 2: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies. Applying the consistency-driven algorithm, after 9 iterations, we obtain the PRS shown in Table 9.27. Global scores are shown in Table 9.28. The ranking order given by the consistency-driven algorithm is: K G w F G w M G w B G w N G w A G w L G w J G w P G w G G w O G w C G w D G w H G w I G w E, which has rank reversals between A and J, C and H, C and I, D and H, D and I, and J and L, hence the ranking order is not acceptable. From this experiment we see that when the PRS has cycles, the consistency-driven algorithm can produce rank reversals.

9. Testing and Experimenting 119 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < I < < < J > < K > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.27: Test 2: The 10th (revised) PRS. It is a consistent ranking. Object K F M B N A L J P G O C D H I E G 15 12 12 9 7 3 3 0 0 6 6 7 7 10 10 15 Table 9.28: Test 2: Global scores produced by the consistency-driven algorithm 9.2.2 Solution by property-driven algorithm 9.2.2.1 Property-driven classical partial order approximations The original PRS P with cycles is shown in Table 9.25 and the combined PRS P is shown in Table 9.29. The light grey cells indicate the cycle C D I C.

9. Testing and Experimenting 120 P A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.29: Test 2: Combined system P = (X,, ) for the first PRS. Light grey cells indicate a cycle. Since P, shown in Table 9.29, has cycles, after removing them we obtain the PRS P shown in Table 9.30.

9. Testing and Experimenting 121 P A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.30: Test 2: Property-driven classical partial order approximation of P by P. Light grey cells indicate the revisions by removing all cycles from P. Classical partial order approximation of P by (P ) + This is shown in Table 9.31.

9. Testing and Experimenting 122 (P ) + A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.31: Test 2: Property-driven classical partial order approximation of P by (P ) + Classical partial order approximation of P by (P + ) This is shown in Table 9.32.

9. Testing and Experimenting 123 (P + ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.32: Test 2: Property-driven classical partial order approximation of P by (P + ) Classical partial order approximation of P by (P ) This is shown in Table 9.33.

9. Testing and Experimenting 124 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.33: Test 2: Property-driven classical partial order approximation of P by (P ) Classical partial order approximation of P by (P ) This is shown in Table 9.34.

9. Testing and Experimenting 125 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.34: Test 2: Property-driven classical partial order approximation of P by (P ) Global scores The global scores and ranking orders produced by the propertydriven classical partial order approximations are shown in Tables 9.35 and 9.36 respectively. G A B C D E F G H I J K L M N O P (P ) + 2 9 11 10 15 12 4 9 10 5 15 2 12 7 4 1 (P + ) 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P ) 1 9 4 4 15 12 5 9 9 4 15 1 12 7 5 2 (P ) 1 9 4 4 15 12 5 9 10 3 15 1 12 7 5 2 Table 9.35: Test 2: Global scores produced by the classical partial order approximation algorithm

9. Testing and Experimenting 126 Approximation Ranking Order (P ) + K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w D G w I G w C G w E (P + ) K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P ) K G w F G w M G w B G w N G w A G w L G w P G w C G w D G w J G w G G w O G w H G w I G w E (P ) K G w F G w M G w B G w N G w A G w L G w P G w J G w C G w D G w G G w O G w H G w I G w E Table 9.36: Test 2: Ranking orders produced by the property-driven p.o. approximations From Table 9.36, we see that (P ) and (P ) have rank reversals between A and J, C and G, C and H, C and I, C and O, D and G, D and H, D and I, D and O, J and L, J and P. The approximations of P by both (P + ) and (P ) + have acceptable results; however (P ) + giving a better result than (P + ), as can seen by comparing the preferences among I, C and D in the two cases. 9.2.2.2 Property-driven refined partial order approximations The original PRS P = (X,,,, <, ) is shown in Table 9.25. Refined classical partial order approximation of P by (P ) + This is shown in Table 9.37.

9. Testing and Experimenting 127 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < < D < < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > > > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.37: Test 2: Property-driven refined partial order approximation of P by (P ) + Refined partial order approximation of P by (P + ) This is shown in Table 9.38.

9. Testing and Experimenting 128 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < < D < < < < < E < < < < F > > > > > > > G < < < H < < > < < I < < < < J > > > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.38: Test 2: Property-driven refined partial order approximation of P by (P + ) Refined partial order approximation of P by (P ) This is shown in Table 9.39.

9. Testing and Experimenting 129 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.39: Test 2: Property-driven refined partial order approximation of P by (P ) Refined partial order approximation of P by (P ) This is shown in Table 9.40.

9. Testing and Experimenting 130 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.40: Test 2: Property-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders produced by the propertydriven refined partial order approximations are shown in Tables 9.41 and 9.42 respectively. G A B C D E F G H I J K L M N O P (P ) + 2 9 10 10 15 12 4 10 10 5 15 2 12 7 4 1 (P + ) 2 9 11 10 15 12 4 9 10 5 15 2 12 7 4 1 (P ) 1 9 4 4 15 12 5 9 9 4 15 1 12 7 5 2 (P ) 1 9 4 4 15 12 5 9 10 3 15 1 12 7 5 2 Table 9.41: Test 2: Global scores produced by the property-driven refined partial order approximation algorithm

9. Testing and Experimenting 131 Approximation Ranking Order (P ) + K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w D G w I G w C G w E (P + ) K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P ) K G w F G w M G w B G w N G w A G w L G w P G w C G w D G w J G w G G w O G w H G w I G w E (P ) K G w F G w M G w B G w N G w A G w L G w P G w J G w C G w D G w G G w O G w H G w I G w E Table 9.42: Test 2: Ranking orders produced by the property-driven refined partial order approximation algorithm Comparing the ranking orders produced by the property-driven classical partial order approximations and refined partial order approximations, shown in Tables 9.36 and 9.42 respectively, we see that in this experiment the property-driven classical partial order approximations and refined partial order approximations produce the same ranking order. 9.2.3 Solution by property/consistency-driven algorithm Refined partial order approximation of P by (P ) + The property-driven refined partial order approximation of P by (P ) +, shown in Table 9.37, is inconsistent, as can be seen from a table of the numbers of consistency violations. Applying the consistency-driven algorithm, after 7 iterations, we obtain the revised CRS shown in Table 9.43.

9. Testing and Experimenting 132 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < J > K > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.43: Test 2: Property/consistency-driven refined partial order approximation of P by (P ) + Refined partial order approximation of P by (P + ) The property-driven refined partial order approximation of P by (P + ), shown in Table 9.38, is inconsistent. Applying the consistency-driven algorithm, after 11 iterations we obtain the CRS shown in Table 9.44.

9. Testing and Experimenting 133 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < J > K > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.44: Test 2: Property/consistency-driven refined partial order approximation of P by (P + ) Refined partial order approximation of P by (P ) The property-driven refined partial order approximation of P by (P ), shown in Table 9.39, is inconsistent. Applying the consistency-driven algorithm, after 5 iterations we obtain the CRS shown in Table 9.45.

9. Testing and Experimenting 134 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < F > > > > > > > G < < < < H < < < < I < < < < J < < K > > > > > > > > > L > < M > > > > > > > > N > O < < < < P < < < < Table 9.45: Test 2: Property/consistency-driven refined partial order approximation of P by (P ) Refined partial order approximation of P by (P ) The property-driven refined partial order approximation of P by (P ), shown in Table 9.40, is inconsistent. Applying the consistency-driven algorithm, after 6 iterations we obtain the CRS shown in Table 9.46.

9. Testing and Experimenting 135 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < F > > > > > > > G < < < < H < < < < I < < < < J < < K > > > > > > > > > L > < M > > > > > > > > N > O < < < < P < < < < Table 9.46: Test 2: Property/consistency-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders produced by the property/consistency-driven refined partial order approximations are shown in Tables 9.47 and 9.48 respectively. G A B C D E F G H I J K L M N O P (P ) + 2 9 10 10 15 12 4 10 10 5 15 2 12 7 4 1 (P + ) 2 9 11 10 15 12 4 7 11 5 15 2 12 7 4 1 (P ) 1 9 4 4 15 12 5 9 9 4 15 1 12 7 5 2 (P ) 1 9 4 4 15 12 5 9 10 3 15 1 12 7 5 2 Table 9.47: Test 2: Global scores produced by the property/consistency-driven refined partial order approximation algorithm

9. Testing and Experimenting 136 Approximation Ranking Order (P ) + K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w D G w I G w C G w E (P + ) K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P ) K G w F G w M G w B G w N G w A G w L G w P G w C G w D G w J G w G G w O G w H G w I G w E (P ) K G w F G w M G w B G w N G w A G w L G w P G w J G w C G w D G w G G w O G w H G w I G w E Table 9.48: Test 2: Ranking orders produced by the property/consistency-driven refined partial order approximation algorithm From the ranking order shown in Table 9.48, we see that the approximations of P by (P ) and (P ) produce rank reversals, but the approximations by (P ) + and (P + ) have acceptable ranking orders, and the approximation by (P + ) has better ranking results than that by (P ) + as can be seen by comparing the preferences among H, C, D and I, in the two cases.

9. Testing and Experimenting 137 9.3 Test 3: Pre-ranking with non-transitivity An example of PRS P = (X,,,, <, ) with non-transitivity property is shown in Table 9.49. The light grey cells show how the transitivity property is not satisfied, i.e., D H and H G, but D G; I H and H G, however I G. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.49: Test 3: The original PRS with non-transitivity. Light grey cells show how the transitivity property is not satisfied. 9.3.1 Solution by consistency-driven algorithm The original PRS shown in Table 9.49, is inconsistent, and the numbers of consistency rule violations are shown in Table 9.50. After 8 iterations, we obtain the CRS shown in Table 9.51. The global scores produced by the consistency-driven algorithm are shown in Table 9.52.

9. Testing and Experimenting 138 Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 B 0 0 0 0 0 3 0 0 0 0 0 7 2 3 0 10 C 0 0 0 0 0 0 1 2 1 2 0 1 0 0 1 2 D 0 0 0 0 0 0 0 3 1 2 0 1 0 0 1 2 E 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 F 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 1 0 0 0 0 2 0 1 1 1 0 0 0 1 H 0 0 2 3 0 0 2 0 2 0 2 0 0 5 0 1 I 0 0 1 1 0 0 0 2 0 1 2 1 0 0 0 1 J 3 0 2 2 1 0 1 0 1 0 6 1 7 7 1 0 K 0 0 0 0 0 0 1 2 2 6 0 2 0 0 2 3 L 0 7 1 1 1 0 1 0 1 1 2 0 6 5 1 0 M 0 2 0 0 0 0 0 0 0 7 0 6 0 0 0 0 N 0 3 0 0 0 0 0 5 0 7 0 5 0 0 0 0 O 0 0 1 1 0 0 0 0 0 1 2 1 0 0 0 1 P 0 10 2 2 0 0 1 1 1 0 3 0 0 0 1 0 Table 9.50: Test 3: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < < J > K > > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.51: Test 3: After 8 iterative revisions. This is a consistent ranking.

9. Testing and Experimenting 139 Object K F, M B N J A, L P O D, G, H, I C E G 15 12 9 7 5 2 1 4 8 12 15 Table 9.52: Test 3: Global scores produced by the consistency-driven algorithm The ranking order produced by the consistency-driven algorithm is K G wf G w M G wb G wn G wj G wa G w L G wp G wo G wd G w G G w H G w I G wc G we, which has rank reversal between G and O, and is therefore not acceptable. 9.3.2 Solution by property-driven algorithm 9.3.2.1 Property-driven classical partial order approximations The first PRS P = (X,,,, <, ) is shown in Table 9.49. Its combined PRS P (X,, ) is shown in Table 9.53. The light grey cells indicate two violations of transitivity, i.e., D H G, but D G, and I H G, but I G.

9. Testing and Experimenting 140 P A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.53: Test 3: Combined PRS P for the first PRS without transitivity property. Light grey cells show how the transitivity property is not satisfied. Classical partial order approximation of P by (P ) + The property-driven classical partial order approximation (P ) + (X,, ), formed by removing all cycles from P and then constructing the transitive closure, is shown in Table 9.54.

9. Testing and Experimenting 141 (P ) + A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.54: Test 3: Property-driven classical partial order approximation of P by (P ) + Classical partial order approximation of P by (P + ) The property-driven classical partial order approximation of P by (P + ) (X,, ), formed by constructing the transitive closure of P and then removing all cycles, is shown in Table 9.55.

9. Testing and Experimenting 142 (P + ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.55: Test 3: Property-driven classical partial order approximation of P by (P + ) Classical partial order approximation of P by (P ) The property-driven classical partial order approximation of P by (P ), formed by constructing the subset closure of P and then removing all cycles, is shown in Table 9.56.

9. Testing and Experimenting 143 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.56: Test 3: Property-driven classical partial order approximation of P by (P ) Classical partial order approximation of P by (P ) The property-driven classical partial order approximation of P by (P ), formed by removing all cycles from P and then constructing the subset closure, is shown in Table 9.57.

9. Testing and Experimenting 144 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.57: Test 3: Property-driven classical partial order approximation of P by (P ) Global scores The global scores and ranking orders produced by the propertydriven classical partial order approximation algorithm are presented in Tables 9.58 and 9.59 respectively. From Table 9.59, we see that the property-driven classical partial order approximations of P by (P ) and by (P ) both produce rank reversals between G and O, and between H and I, and the approximations of P by (P ) + and by (P + ) have the same ranking order, which is acceptable. If there are no cycles, it can be proved that (P ) + = (P + ) = P + [12]. G A B C D E F G H I J K L M N O P (P ) + 2 9 13 11 15 12 4 7 9 5 15 2 12 7 4 1 (P + ) 2 9 13 11 15 12 4 7 9 5 15 2 12 7 4 1 (P ) 2 9 13 9 15 12 7 8 7 5 15 2 12 7 4 1 (P ) 2 9 13 9 15 12 7 8 7 5 15 2 12 7 4 1 Table 9.58: Test 3: Global scores produced by the property-driven classical partial order approximation algorithm

9. Testing and Experimenting 145 Approximation Ranking Order (P ) + K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E (P + ) K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w I G w H G w D G w C G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w I G w H G w D G w C G w E Table 9.59: Test 3: Ranking order produced by the property-driven classical partial order approximation algorithm 9.3.2.2 Property-driven refined partial order approximations The original PRS P is shown in Table 9.49. Refined partial order approximation of P by (P ) + The property-driven refined partial order approximation of P by (P ) + is shown in Table 9.60.

9. Testing and Experimenting 146 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < < E < < < < F > > > > > > > G < > < > < H < < < < I < < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.60: Test 3: Property-driven refined partial order approximation of P by (P ) + Refined partial order approximation of P by (P + ) The property-driven refined partial order approximation of P by (P + ) is shown in Table 9.61.

9. Testing and Experimenting 147 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < < E < < < < F > > > > > > > G < > < > < H < < < < I < < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.61: Test 3: Property-driven refined partial order approximation of P by (P + ) Refined partial order approximation of P by (P ) The property-driven refined partial order approximation of P by (P ) is shown in Table 9.62.

9. Testing and Experimenting 148 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.62: Test 3: Property-driven refined partial order approximation of P by (P ) Refined partial order approximation of P by (P ) The property-driven refined partial order approximation of P by (P ) is shown in Table 9.63.

9. Testing and Experimenting 149 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.63: Test 3: Property-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders found by the property-driven refined partial order approximations are shown in Tables 9.64 and 9.65 respectively. Comparing tables 9.59 and 9.65, we see that the property-driven refined and classical partial order approximation algorithms produce the same ranking results for this experiment, i.e. the approximations of P by (P ) and by (P ) produce a ranking with reversals, and approximations of P by (P ) + and by (P + ) produce the same ranking, which is acceptable. G A B C D E F G H I J K L M N O P (P ) + 2 9 13 11 15 12 4 7 9 5 15 2 12 7 4 1 (P + ) 2 9 13 11 15 12 4 7 9 5 15 2 12 7 4 1 (P ) 2 9 13 9 15 12 7 8 7 5 15 2 12 7 4 1 (P ) 2 9 13 9 15 12 7 8 7 5 15 2 12 7 4 1 Table 9.64: Test 3: Global scores produced by the property-driven refined partial order approximation algorithm

9. Testing and Experimenting 150 Approximation Ranking Order (P ) + K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E (P + ) K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w I G w H G w D G w C G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w I G w H G w D G w C G w E Table 9.65: Test 3: Ranking orders produced by the property-driven refined partial order approximation algorithm 9.3.3 Solution by property/consistency-driven algorithm Property/consistency-driven refined partial order approximation of P by (P ) + This is shown in Table 9.60. It is not consistent. Applying the consistencydriven algorithm, after 11 iterations we obtain the CRS shown in Table 9.66.

9. Testing and Experimenting 151 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < < J > K > > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.66: Test 3: Property/consistency-driven refined partial order approximation of P by (P ) + Property/consistency-driven refined partial order approximation of P by (P + ) This is shown in Table 9.61. It is inconsistent. Applying the consistencydriven, after 11 iterations we obtain the CRS shown in Table 9.67.

9. Testing and Experimenting 152 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < < J > K > > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.67: Test 3: Property/consistency-driven refine approximation of P by (P + ) Property/consistency-driven refined partial order approximation of P by (P ) This is shown in Table 9.62. It is inconsistent. Applying the consistencydriven algorithm, after 7 iterations we obtain the CRS shown in Table 9.68.

9. Testing and Experimenting 153 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < < J > K > > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.68: Test 3: Property/consistency-driven refined partial order approximation of P by (P ) Property/consistency-driven refined partial order approximation of P by (P ) This is shown in Table 9.63. It is inconsistent. Applying the consistencydriven algorithm, after 7 iterations we obtain the CRS shown in Table 9.69.

9. Testing and Experimenting 154 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < < J > K > > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.69: Test 3: Property/consistency-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders found by the property/consistency-driven refined partial order approximation algorithm are shown in Tables 9.70 and 9.71 respectively. From Table 9.71, we see that the approximations of P by both (P ) and (P ) produce rank reversals between G and I, G and O, and H and I, and the approximations of P by (P ) + and by (P + ) both produce the same acceptable ranking. G A B C D E F G H I J K L M N O P (P ) + 2 9 13 11 15 12 4 7 9 5 15 2 12 7 4 1 (P + ) 2 9 13 11 15 12 4 7 9 5 15 2 12 7 4 1 (P ) 2 9 11 9 15 12 8 9 7 5 15 2 12 7 4 1 (P ) 2 9 11 9 15 12 8 9 7 5 15 2 12 7 4 1 Table 9.70: Test 3: Global scores produced by the property/consistency-driven refined partial order approximation algorithm

9. Testing and Experimenting 155 Approximation Ranking Order (P ) + K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E (P + ) K G w F G w M G w B G w N G w J G w A G w L G w P G w G G w O G w H G w I G w D G w C G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w I G w G G w H G w D G w C G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w I G w G G w H G w D G w C G w E Table 9.71: Test 3: Ranking orders produced by the property/consistency-driven refined partial order approximation algorithm

9. Testing and Experimenting 156 9.4 Test 4: Pre-ranking with cycles and nontransitivity An example of a PRS, P with cycles and non-transitivity, is shown in Table 9.72. The dark grey cells show the cycle, i.e., C D, D I and I C, and the light grey cells indicate two violations of transitivity, i.e., D H and H G, but D G, and I H and H G, but I G. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.72: Test 4: A PRS with cycles and without transitivity. Dark grey cells indicate a cycle, and light grey cells indicate violations of the transitive property. 9.4.1 Solution by consistency-driven method The original PRS shown in Table 9.72 is inconsistent, and the numbers of consistency rule violations are shown in Table 9.73. Applying the consistency-driven algorithm, after 7 iterations we obtain the CRS shown in Table 9.74.

9. Testing and Experimenting 157 Violation A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 B 0 0 0 0 0 3 0 0 0 0 0 7 2 3 0 10 C 0 0 0 1 0 0 2 2 2 2 0 1 0 0 1 2 D 0 0 1 0 0 0 0 3 2 2 0 1 0 0 1 2 E 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 F 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G 0 0 2 0 0 0 0 2 0 1 1 1 0 0 0 1 H 0 0 2 3 0 0 2 0 2 0 2 0 0 5 0 1 I 0 0 2 2 0 0 0 2 0 1 1 1 0 0 0 1 J 3 0 2 2 1 0 1 0 1 0 6 1 7 7 1 0 K 0 0 0 0 0 0 1 2 1 6 0 2 0 0 2 3 L 0 7 1 1 1 0 1 0 1 1 2 0 6 5 1 0 M 0 2 0 0 0 0 0 0 0 7 0 6 0 0 0 0 N 0 3 0 0 0 0 0 5 0 7 0 5 0 0 0 0 O 0 0 1 1 0 0 0 0 0 1 2 1 0 0 0 1 P 0 10 2 2 0 0 1 1 1 0 3 0 0 0 1 0 Table 9.73: Test 4: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies.

9. Testing and Experimenting 158 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > K > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.74: Test 4: After 7 iterative revisions. This is a CRS. Object A B C D E F G H I J K L M N O P G 2 9 11 8 15 12 8 7 10 5 15 2 12 7 4 1 Table 9.75: Test 4: Global scores produced by the consistency-driven algorithm Global scores produced by the consistency-driven algorithm are shown in Table 9.75. The ranking order is: K G wf G w M G wb G wn G wj G wa G w L G wp G wo G wh G wd G w G G wi G wc G we, which has rank reversals between D and I, G and H, and G and O, and hence the ranking order is not acceptable.

9. Testing and Experimenting 159 9.4.2 Solution by property-driven method 9.4.2.1 Property-driven classical partial order approximations The original PRS P is shown in Table 9.72. Its combined PRS P is shown in Table 9.76. The dark grey cells indicate a cycle among C, D and I, i.e., C D I C, and the light grey cells indicate non-transitivity, i.e., D < H < G but D G, and I < H < G but I G. P A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.76: Test 4: Combined PRS P (X,, ). Dark grey cells indicate a cycle and light grey cells indicate two violations of transitivity. Classical partial order approximation of P by (P ) + The PRS (P ) +, formed by removing all cycles from P and then constructing its transitive closure, is shown in Table 9.77.

9. Testing and Experimenting 160 (P ) + A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.77: Test 4: Property-driven classical partial order approximation of P by (P ) + Classical partial order approximation of P by (P + ) The PRS (P + ), formed by constructing the transitive closure of P and then removing all cycles, is shown in Table 9.78.

9. Testing and Experimenting 161 (P + ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.78: Test 4: Property-driven classical partial order approximation of P by (P + ) Classical partial order approximation of P by (P ) This is shown in Table 9.79.

9. Testing and Experimenting 162 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.79: Test 4: Property-driven classical partial order approximation of P by (P ) Classical partial order approximation of P by (P ) This is shown in Table 9.80.

9. Testing and Experimenting 163 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.80: Test 4: Property-driven classical partial order approximation of P by (P ) Global scores Global scores and ranking order found by the property-driven classical partial order approximations are shown in Tables 9.82 and 9.82 respectively. From Table 9.82, we see that the property-driven classical partial order approximations of P by (P ) and by (P ) both produce rank reversal between G and O, and the approximations of P by (P + ) and by (P ) + both produce the same acceptable ranking order. G A B C D E F G H I J K L M N O P (P ) + 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P + ) 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P ) 2 9 10 9 15 12 8 8 9 5 15 2 12 7 4 1 (P ) 2 9 11 9 15 12 7 8 9 5 15 2 12 7 4 1 Table 9.81: Test 4: Global scores produced by the property-driven classical partial order approximations

9. Testing and Experimenting 164 Approximation Ranking Order (P ) + K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P + ) K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w H G w D G w I G w C G w E (P ) K G wf G w M G wb G wn G wj G wa G w L G wp G wo G wg G wh G wd G w I G w C G w E Table 9.82: Test 4: Ranking orders produced by the property-driven classical partial order approximations 9.4.2.2 Property-driven refined partial order approximations The original PRS P is shown in Table 9.72. Refined partial order approximation of P by (P ) + The property-driven refined partial order approximation of P by (P ) + is shown in Table 9.83.

9. Testing and Experimenting 165 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < < E < < < < F > > > > > > > G < > < > < H < < < < I < < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.83: Test 4: Property-driven refined partial order approximation of P by (P ) + Refined partial order approximation of P by (P + ) This is shown in Table 9.84.

9. Testing and Experimenting 166 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < < E < < < < F > > > > > > > G < > < > < H < < < < I < < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.84: Test 4: Property-driven refined partial order approximation of P by (P + ) Refined partial order approximation of P by (P ) This is shown in Table 9.85.

9. Testing and Experimenting 167 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.85: Test 4: Property-driven refined partial order approximation of P by (P ) Refined partial order approximation of P by (P ) This is shown in Table 9.86.

9. Testing and Experimenting 168 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.86: Test 4: Property-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders found by the property-driven refined partial order approximation algorithm are shown in Tables 9.88 and 9.88 respectively. From Tables 9.82 and 9.88, we see that the property-driven classical partial order approximation and refined partial order approximation algorithms produce the same ranking result, i.e. the approximations of P by both (P ) and (P ) produce reversed ranking between G and O, and are therefore unacceptable, but the ranking result produced by both (P ) + and (P + ) is acceptable. G A B C D E F G H I J K L M N O P (P ) + 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P + ) 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P ) 2 9 10 9 15 12 8 8 9 5 15 2 12 7 4 1 (P ) 2 9 11 9 15 12 7 8 9 5 15 2 12 7 4 1 Table 9.87: Test 4: Global scores produced by the property-driven refined partial order approximations

9. Testing and Experimenting 169 Approximation Ranking Order (P ) + K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P + ) K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w H G w D G w I G w C G w E (P ) K G wf G w M G wb G wn G wj G wa G w L G wp G wo G wg G wh G wd G w I G w C G w E Table 9.88: Test 4: Ranking orders produced by the property-driven refined partial order approximations 9.4.3 Solution by property/consistency-driven method Property/consistency-driven refined partial order approximation of P by (P ) + This is shown in Table 9.83. It is inconsistent. After 9 iterations, we obtain the CRS shown in Table 9.89.

9. Testing and Experimenting 170 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < J > K > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.89: Test 4: Property/consistency-driven refined partial order approximation of P by (P ) +. It is a CRS. Property/consistency-driven refined partial order approximation of P by (P + ) This is shown in Table 9.84. It is inconsistent. Applying the consistencydriven algorithm, after 9 iterations we obtain the CRS shown in Table 9.90.

9. Testing and Experimenting 171 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < J > K > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.90: Test 4: Property/consistency-driven refined partial order approximation of P by (P + ). It is a CRS. Property/consistency-driven refined partial order approximation of P by (P ) This is shown in Table 9.85. It is inconsistent. Applying the consistencydriven algorithm, after 7 iterations we obtain the CRS shown in Table 9.91.

9. Testing and Experimenting 172 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > K > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.91: Test 4: Property/consistency-driven refined partial order approximation of P by (P ). It is a CRS. Property/consistency-driven refined partial order approximation of P by (P ) This is shown in Table 9.86. It is inconsistent. Applying the consistencydriven algorithm, after 7 iterations we obtain the CRS shown in Table 9.92.

9. Testing and Experimenting 173 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < < I < < < J > K > > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.92: Test 4: Property/consistency-driven refined partial order approximation of P by (P ). It is a CRS. Global scores The global scores and ranking orders found by the property/consistency-driven refined approximation algorithm are shown in Tables 9.93 and 9.94 respectively. From Table 9.94, we see that property/consistencydriven refined partial order approximations of P by both (P ) and (P ) produce rank reversal between G and O, and the approximations of P by both (P ) + and (P + ) produce the same acceptable ranking order. G A B C D E F G H I J K L M N O P (P ) + 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P + ) 2 9 11 11 15 12 4 7 11 5 15 2 12 7 4 1 (P ) 2 9 9 9 15 12 8 9 9 5 15 2 12 7 4 1 (P ) 2 9 9 9 15 12 8 9 9 5 15 2 12 7 4 1 Table 9.93: Test 4: Global scores produced by the property/consistency-driven refined partial order approximations

9. Testing and Experimenting 174 Approximation Ranking Order (P ) + K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P + ) K G wf G w M G wb G wn G wj G wa G w L G wp G wg G w O G wh G wc G w D G w I G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w C G w D G w H G w I G w E (P ) K G w F G w M G w B G w N G w J G w A G w L G w P G w O G w G G w C G w D G w H G w I G w E Table 9.94: Test 4: Ranking orders produced by the property/consistency-driven refined partial order approximations

9. Testing and Experimenting 175 9.5 Test 5: Pre-ranking with cycles, nontransitivity, and data errors The original PRS P with cycles, non-transitivity and some data errors is shown in Table 9.95. The dark grey cells indicate a cycle C D I C, the light grey cells indicate a failure of transitivity, i.e., I H G but I G, and the bold fonts indicate data error, i.e., F = 1050 and K = 1100, but F K ; D = 250 and H = 310 but D H. In fact, the error H D and the cycle (C, D, I) lead to a larger cycle C H D I C. P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.95: Test 5: The first PRS with cycles, without transitivity and with data errors. Dark grey cells indicate a cycle, light grey cells indicate a failure transitivity, and the bold fonts indicate data errors.

9. Testing and Experimenting 176 9.5.1 Solution by consistency-driven algorithm The PRS shown in Table 9.95 is inconsistent, and the numbers of consistency rule violations are shown in Table 9.96. After 9 iterations, we obtain the CRS shown in Table 9.97. P A B C D E F G H I J K L M N O P A 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 B 0 0 0 0 0 3 0 0 0 0 0 7 2 3 0 10 C 0 0 0 1 0 0 2 2 2 2 0 1 0 0 1 2 D 0 0 1 0 0 0 0 1 3 2 0 1 0 0 1 2 E 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 F 0 3 0 0 0 0 0 0 0 0 9 0 0 0 0 0 G 0 0 2 0 0 0 0 1 0 1 2 1 0 0 0 1 H 0 0 2 1 0 0 1 0 3 0 2 0 0 4 0 1 I 0 0 2 3 0 0 0 3 0 1 2 1 0 0 0 1 J 3 0 2 2 1 0 1 0 1 0 6 1 7 7 1 0 K 0 0 0 0 0 9 2 2 2 6 0 3 1 1 3 4 L 0 7 1 1 1 0 1 0 1 1 3 0 6 5 1 0 M 0 2 0 0 0 0 0 0 0 7 1 6 0 0 0 0 N 0 3 0 0 0 0 0 4 0 7 1 5 0 0 0 0 O 0 0 1 1 0 0 0 0 0 1 3 1 0 0 0 1 P 0 10 2 2 0 0 1 1 1 0 4 0 0 0 1 0 Table 9.96: Test 5: Numbers of consistency rule violations for the first PRS. Light grey cells have the most inconsistencies.

9. Testing and Experimenting 177 P A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < I < < < J > K > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.97: Test 5: After 9 iterative revisions. It is a CRS. The global scores are shown in Table 9.98, and the ranking order is K G w F G w M G w B G w N G w J G w A L G w P G w O G w D G G w I G w C G w H G w E, which has rank reversals between C and H, D and H, D and I, G and O, and H and I. Hence the ranking order is not acceptable. Object A B C D E F G H I J K L M N O P G 2 9 10 7 15 13 7 11 9 5 14 2 12 7 4 1 Table 9.98: Test 5: Global scores produced by the consistency-driven algorithm 9.5.2 Solution by property-driven algorithm 9.5.2.1 Property-driven classical partial order approximations Let P = (X,, ) be the combined PRS. P is shown in Table 9.99.

9. Testing and Experimenting 178 P A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.99: Test 5: Combined PRS P. Dark grey cells indicate a cycle, light grey cells indicate failure of transitivity, and bold fonts indicate data errors. Classical partial order approximation of P by (P ) + The PRS (P ) +, produced by removing all cycles from P and then constructing the transitive closure, is shown in Table 9.100.

9. Testing and Experimenting 179 (P ) + A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.100: Test 5: Property-driven classical partial order approximation of P by (P ) + Classical partial order approximation of P by (P + ) This is shown in Table 9.101.

9. Testing and Experimenting 180 (P + ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.101: Test 5: Property-driven classical partial order approximation of P by (P + ) Classical partial order approximation of P by (P ) This is shown in Table 9.102.

9. Testing and Experimenting 181 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.102: Test 5: Property-driven classical partial order approximation of P by (P ) Classical partial order approximation of P by (P ) This is shown in Table 9.103.

9. Testing and Experimenting 182 (P ) A B C D E F G H I J K L M N O P A B C D E F G H I J K L M N O P Table 9.103: Test 5: Property-driven classical partial order approximation of P by (P ) Global scores The global scores and ranking orders are shown in Tables 9.104 and 9.105 respectively. From Table 9.105, we see that all four approximations of P produce rank reversals i.e., approximation of P by (P ) + has rank reversals between D and H, F and K, G and O, and H and I; approximation of P by (P + ) has a rank reversal between F and K; approximation of P by (P ) has a rank reversal between G and O; and approximation of P by (P ) has rank reversals between D and H, G and O, and H and I. Approximation A B C D E F G H I J K L M N O P (P ) + 2 9 10 9 15 15 6 10 9 5 13 2 11 7 4 1 (P + ) 2 9 10 10 15 15 4 10 10 5 13 2 11 7 4 1 (P ) 2 9 10 9 15 13 8 8 9 5 13 2 13 7 4 1 (P ) 2 9 10 9 15 13 6 10 9 5 13 2 13 7 4 1 Table 9.104: Test 5: Global scores produced by the property-driven classical partial order approximations

9. Testing and Experimenting 183 Approximation Ranking Order (P ) + F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w D G w I G w C G w H G w E (P + ) F G w K G w M G w B G w N G w J G w A G w L G w P G w G G w O G w C G w D G w H G w I G w E (P ) F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w H G w D G w I G w C G w E (P ) F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w D G w I G w C G w H G w E Table 9.105: Test 5: Ranking orders produced by the property-driven classical partial order approximations 9.5.2.2 Property-driven refined partial order approximations The original PRS P is shown in Table 9.95. Refined partial order approximation of P by (P ) + The property-driven refined partial order approximation of P by (P ) + is shown in Table 9.106.

9. Testing and Experimenting 184 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > < > > > > > > > N > > O < < < P < < Table 9.106: Test 5: Property-driven refined partial order approximation of P by (P ) + Refined partial order approximation of P by (P + ) This is shown in Table 9.107.

9. Testing and Experimenting 185 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > > G < < > < H < < < < I < < < < J > < K > > > L < > < < M > > < > > > > > > > N > > O < < < P < < Table 9.107: Test 5: Property-driven refined partial order approximation of P by (P + ) Refined partial order approximation of P by (P ) This is shown in Table 9.108.

9. Testing and Experimenting 186 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.108: Test 5: Property-driven refined partial order approximation of P by (P ) Refined partial order approximation of P by (P ) This is shown in Table 9.109.

9. Testing and Experimenting 187 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > < K > > > L < > < < M > > > > > > > > > N > > O < < < P < < Table 9.109: Test 5: Property-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders are shown in Tables 9.110 and 9.111 respectively. Approximation A B C D E F G H I J K L M N O P (P ) + 2 9 10 9 15 15 6 10 9 5 13 2 11 7 4 1 (P + ) 2 9 10 10 15 15 4 10 10 5 13 2 11 7 4 1 (P ) 2 9 10 9 15 13 8 8 9 5 13 2 13 7 4 1 (P ) 2 9 10 9 15 13 6 10 9 5 13 2 13 7 4 1 Table 9.110: Test 5: Global scores produced by the property-driven refined partial order approximations

9. Testing and Experimenting 188 Approximation Ranking Order (P ) + F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w D G w I G w C G w H G w E (P + ) F G w K G w M G w B G w N G w J G w A G w L G w P G w G G w O G w C G w D G w H G w I G w E (P ) F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w H G w D G w I G w C G w E (P ) F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w D G w I G w C G w H G w E Table 9.111: Test 5: Ranking orders produced by the property-driven refined partial order approximations From Tables 9.111 and 9.105, we see that property-driven classical partial order approximations and refined partial order approximations produce the same ranking orders. All four property-driven refined partial order approximations have rank reversals. 9.5.3 Solution by property/consistency-driven algorithm Property/consistency-driven refined partial order approximation of P by (P ) + Property-driven refined partial order approximation of P by (P ) +, shown in Table 9.106, is inconsistent, as can be seen from a table of the numbers of consistency violations. After 9 iterations, we obtain the CRS shown in Table 9.112.

9. Testing and Experimenting 189 (P ) + A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < I < < < J > K > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.112: Test 5: Property/consistency-driven refined partial order approximation of P by (P ) +. This is a CRS. Property/consistency-driven refined partial order approximation of P by (P + ) This is shown in Table 9.107. It is inconsistent. After 10 iterations, we obtain the CRS shown in Table 9.113.

9. Testing and Experimenting 190 (P + ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < I < < < J > K > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.113: Test 5: Property/consistency-driven refined partial order approximation of P by (P + ). This is a CRS. Property/consistency-driven refined partial order approximation of P by (P ) This is shown in Table 9.108. It is inconsistent. Applying the consistencydriven algorithm, after 7 iterations we obtain the CRS shown in Table 9.114.

9. Testing and Experimenting 191 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < H < < < < I < < < J > K > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.114: Test 5: Property/consistency-driven refined partial order approximation of P by (P ). This is a CRS. Property/consistency-driven of refined partial order approximation of P by (P ) This is shown in Table 9.109. It is inconsistent. Applying the consistencydriven algorithm, after 7 iterations we obtain the CRS shown in Table 9.115.

9. Testing and Experimenting 192 (P ) A B C D E F G H I J K L M N O P A > B > > > > > > > C < < < < D < < < < E < < < < F > > > > > > > G < < < < H < < < I < < < J > K > > > > > L > M > > > > > > > N > O < < < < P < < < < Table 9.115: Test 5: Property/consistency-driven refined partial order approximation of P by (P ) Global scores The global scores and ranking orders produced by the property/consistency-driven algorithm are shown in Tables 9.116 and 9.117 respectively. From Table 9.117, we see that property/consistency-driven refined partial order approximations of P by (P ) +, (P ) and (P ) all have rank reversals, but (P + ) has acceptable ranking order. Approximation A B C D E F G H I J K L M N O P (P ) + 2 9 9 9 15 14 7 10 9 5 14 2 11 7 4 1 (P + ) 2 9 10 10 15 14 4 10 10 5 14 2 11 7 4 1 (P ) 2 9 4 4 15 13 8 9 9 5 13 2 13 7 4 1 (P ) 2 9 9 9 15 13 7 10 9 5 13 2 13 7 4 1 Table 9.116: Test 5: Global scores produced by the property/consistency-driven refined partial order approximations

9. Testing and Experimenting 193 Approximation Ranking Order (P ) + F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w C G w D G w I G w H G w E (P + ) F G w K G w M G w B G w N G w J G w A G w L G w P G w G G w O G w C G w D G w H G w I G w E (P ) F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w C G w D G w H G w I G w E (P ) F G w K G w M G w B G w N G w J G w A G w L G w P G w O G w G G w C G w D G w I G w H G w E Table 9.117: Test 5: Ranking orders produced by the property/consistency-driven refined partial order approximations

Chapter 10 Comparison and analysis of various algorithms In this section, we will compare and analyze all our non-numerical ranking algorithms, in terms of their outcomes. 10.1 Comparisons of various non-numerical ranking algorithms In Chapter 9, we presented five experiments with pre-ranking systems having different properties, and implemented various non-numerical ranking algorithms to compare the ranking results. 10.1.1 Case 1: PRS with acyclicity and transitivity Comparisons of the various non-numerical ranking algorithms, for the case when the PRS P is acyclic and transitive, can be made from the results in Table 10.1. As 194

10. Comparison and analysis of various algorithms 195 we have seen, (P ) + = (P + ) = (P ) = (P ) = P, and hence for the propertydriven algorithm, all four approximations have the same ranking orders. For the property/consistency-driven algorithm, all four approximations have the same ranking orders as the consistency-driven method. From Table 10.1, we see that all the non-numerical ranking algorithms have no rank reversals, and their ranking orders are acceptable. Algorithm Partial Order Approximation Relation Rank Time Reversal Complexity Consistency-Driven No O(n 3 ) (P ) + No O(n 3 ) Classical (P ) No O(n 3 ) (P + ) No O(n 3 ) Property-Driven (P ) No O(n 3 ) (P ) + No O(n 3 ) Refined (P + ) No O(n 3 ) (P ) No O(n 3 ) (P ) No O(n 3 ) (P ) + No O(n 3 ) Property/Consistency-Driven Refined (P + ) No O(n 3 ) (P ) No O(n 3 ) (P ) No O(n 3 ) Table 10.1: Case 1: PRS with acyclicity and transitivity 10.1.2 Case 2: PRS with cycles The comparison results for the case where the PRS is cyclic are shown in Table 10.2. We see that with the property-driven and property/consistency-driven algorithms, the approximations by (P ) + and (P + ) both have acceptable ranking results. The approximation by (P ) + has better ranking results than that by (P + ), see the example in Chapter 9, Test 2.

10. Comparison and analysis of various algorithms 196 Partial Order Rank Time Algorithm Approximation Relation Reversal Complexity Consistency-Driven Yes O(n 3 ) (P ) + No O(n 3 ) Classical (P + ) No O(n 3 ) (P ) Yes O(n 3 ) Property-Driven (P ) Yes O(n 3 ) (P ) + No O(n 3 ) Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) (P ) + No O(n 3 ) Property/Consistency-Driven Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) Table 10.2: Case 2: PRS with cycles. Gray rows indicate the algorithm can produce rank reversals. 10.1.3 Case 3: PRS with acyclicity but non-transitivity The comparison results for the case where the PRS is acyclic but non-transitive are shown in Table 10.3. We see that for the property-driven and property/consistencydriven algorithms, the approximations by both (P ) + and (P + ) have the same acceptable ranking results. If there is no cycle, then (P ) + = (P + ) = P +. 10.1.4 Case 4: PRS with cycles and non-transitivity For the case that the PRS is cyclic and non-transitive, the comparison results of various non-numerical ranking algorithms are shown in Table 10.4. We see that for the property-driven and property/consistency-driven algorithms, the approximations by both (P ) and (P ) can produce rank reversals.

10. Comparison and analysis of various algorithms 197 Partial Order Rank Time Algorithm Approximation Relation Reversal Complexity Consistency-Driven Yes O(n 3 ) (P ) + No O(n 3 ) Classical (P + ) No O(n 3 ) (P ) Yes O(n 3 ) Property-Driven (P ) Yes O(n 3 ) (P ) + No O(n 3 ) Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) (P ) + No O(n 3 ) Property-Consistency-Driven Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) Table 10.3: Case 3: PRS with acyclicity but non-transitivity. Gray rows indicate the algorithm can produce rank reversals. Partial Order Rank Time Algorithm Approximation Relation Reversal Complexity Consistency-Driven Yes O(n 3 ) (P ) + No O(n 3 ) Classical (P + ) No O(n 3 ) (P ) Yes O(n 3 ) Property-Driven (P ) Yes O(n 3 ) (P ) + No O(n 3 ) Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) (P ) + No O(n 3 ) Property/Consistency-Driven Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) Table 10.4: Case 4: PRS with cycles and non-transitivity. Gray rows indicate the algorithm can produce rank reversals

10. Comparison and analysis of various algorithms 198 10.1.5 Case 5: PRS with cycles, non-transitivity and data errors In the case that the PRS has cycles, non-transitivity and data errors as well, the comparison results of the various non-numerical ranking algorithms are shown in Table 10.5. We see that for the property-driven method, all four approximations of P by (P ) +, (P + ), (P ) and (P ) can produce rank reversals. For the property/consistency-driven algorithm, the approximations of P by (P ) +, (P ) and (P ) can produce rank reversals, while (P + ) has acceptable ranking orders. Partial Order Rank Time Algorithm Approximation Relation Reversal Complexity Consistency-Driven Yes O(n 3 ) (P ) + Yes O(n 3 ) Classical (P + ) Yes O(n 3 ) (P ) Yes O(n 3 ) Property-Driven (P ) Yes O(n 3 ) (P ) + Yes O(n 3 ) Refined (P + ) Yes O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) (P ) + Yes O(n 3 ) Property/Consistency-Driven Refined (P + ) No O(n 3 ) (P ) Yes O(n 3 ) (P ) Yes O(n 3 ) Table 10.5: Case 5: PRS with cycles, non-transitivity and data errors. Light grey cells indicate the algorithm can produce rank reversals. 10.2 Conclusions From the results of running our algorithms on our input data, we draw the following conclusions:

10. Comparison and analysis of various algorithms 199 If the PRS is acyclic and transitive, then none of the non-numerical ranking methods produces rank reversal. If the PRS is cyclic and non-transitive, then the approximations of P by (P ) + and (P + ) both give acceptable rankings, as expected. If the PRS is cyclic, non-transitive and has data errors as well, then the property/consistency-driven refined partial order approximation of P by (P + ) seems to produce the least rank reversal. However, if the original PRS is cyclic but transitive, then the approximation by (P ) + sometimes can give better ranking results than that by (P + ), seen test 2 in chapter 9.

Chapter 11 Conclusion and future work In the thesis, we presented several non-numerical ranking algorithms and tested them with an input data of pre-ranking system with various deficiencies such as cycles, non-transitivity and/or data errors. The relative preferences of the algorithms depend significantly on the presence or absence of these deficiencies. For future work, one should investigate the relationship between the non-numerical ranking model and a rough set approach. The theory of rough sets [23, 24] gives a formal approximation to classical ( crisp ) sets, where a rough set is interpreted as a pair of (classical) sets, giving a lower and upper approximation of the original set. The relational structures (P ) + and (P + ) can be viewed as upper partial order approximations of P, and the relational structures (P ), (P ) and P can be viewed as lower partial order approximations of P. More information can be found in [13]. The relationship between the non-numerical ranking model and the rough set approach is not obvious, and a direct translation into any reasonable rough set setting is problematic. This is therefore a challenging but worthwhile area of research. 200

Bibliography [1] K. Arrow, Social Choice and Individual Values. J. Wiley, New York, 1951. [2] S. Bozóki and T. Rapcsák, On Saaty s and Koczkodaj s inconsistencies of pairwise comparison matrices, Journal of Global Optimization, Vol. 42, No.2, pp. 157 175, 2008. [3] S. Burris and H. Sankappanavar, A Course in Universal Algebra. Springer, New York, 1981. [4] T. Cormen, C. Leiserson, R. Rivest, and C. Stein, Introduction to Algorithms. McGraw Hill, 2006. [5] N. Cowan, The Magic Number 4 in short-term memory. A reconsideration of mental storage capacity, Behavioural and Brain Sciences, 24, pp. 87 185, 2001. [6] J. Dyer, Remarks on the Analytic Hierarchy Process, Management Science, Vol. 36, No. 3, pp. 244 258, 1990. [7] J. Dyer and R. Wendell, A Critique of the Analytic Hierarchy Process, Working Paper 84/85-4-24, Department of Management, The University of Texas at Austin, 1985. 201

BIBLIOGRAPHY 202 [8] P. Fishburn, Interval Orders and Interval Graphs. J. Wiley, New York, 1985. [9] R. Janicki, Pairwise comparisons, incomparability and partial orders, in Proc. of ICEIS 2007(Int. Conf. on Enterprise Information Systems), pp. 297 302, 2007. [10] R. Janicki, Ranking with partial orders and pairwise comparisons, in Proc. of RSKT2008(Rough Sets and Knowledge Technology), pp. 442 451, 2008. [11] R. Janicki, Some Remarks on Approximations of Arbitrary Binary Relations by Partial Orders, Lecture Notes in Computer Science, Vol. 5306, pp. 81 91, 2008. [12] R. Janicki, Pairwise Comparisons Based Non-Numerical Ranking, Fundamenta Informaticae, 94,2, pp. 197 217, 2009. [13] R. Janicki, Approximations of Arbitrary Binary Relations by Partial Orders. Classical and Rough Set Models, Transactions on Rough Set, to appear. [14] R. Janicki and W. Koczkodaj, Weak Order Approach to Group Ranking, Computer Math. Applic., 32, pp. 51 59, 1996. [15] R. Janicki and W. Koczkodaj, A Weak Order Solution to Group Ranking and Consistency-Driven Pairwise Comparisons, Applied Mathematics and Computation, 94, pp. 227 241, 1998. [16] R. Janicki and Y. Zhai, On a consistency driven pairwise comparison based nonnumerical ranking, in Proc. of CMMSE 2010 (Int. Conf. on Computational and Mathematical Methods in Science and Engineering), 2010, to appear. [17] W. Koczkodaj, On testing subjective rankings, Unpublished Suggestion, 2007.

BIBLIOGRAPHY 203 [18] W. Koczkodaj, A new definition of consistency of pairwise comparisons, Math. Comput. Modelling 8, pp. 79 84, 1993. [19] W. Koczkodaj, M. Herman, and M. Orlowski, Using consistency-driven pairwise comparions in knowledge-based systems, in Proceedings of the Sixth International Conference on Information and Knowledge Management, pp. 91 96, 1997. [20] F. Kong and H. Liu, Applying fuzzy Analytic Hierarchy process to evaluate success factors of e-commerce, International Journal of Information and Systems Sciences, Vol. 1, No. 3-4, pp. 406 412, 2005. [21] B. Mareschal, Y. D. Smet, and P. Nemery, Rank reversal in the promethee ii method: Some new results, in Proc. of the 2008 IEEE IEEM, pp. 959 963, 2008. [22] G. A. Miller, The Magic Number Seven, Plus or Minus Two, The Psychological Reviews,63,2, pp. 81 97, 1956. [23] Z. Pawlak, Rough sets, International Journal of Computer and Information Sciences, 34, pp. 557 590, 1982. [24] Z. Pawlak, Rough Sets. Kluwer, Dordrecht, 1991. [25] T. Saaty, The Analytic Hierarchy Process. McGraw-Hill, New York, 1980. [26] T. Saaty, The Analytic Hierarchy Process. McGraw-Hill, New York, 1990. [27] T. Saaty, Fundamentals of Decision Making. RSW Publications, 1994.

BIBLIOGRAPHY 204 [28] T. Saaty, Decision making for leaders: The analytical hierarchy process for decisions in a complex world, The Analytical Hierarchy Process Series, Vol. 2, pp. 71 74, 1996. [29] Y. Zhai and R. Janicki, On consistency in pairwise comparisons based numerical and non-numerical ranking, in Proc. of FCS 2010(Int. Conf. on Foundations of Computer Science), 2010, to appear,. [30] Y. Zhai and R. Janicki, Preserving consistency and properties in pairwise comparisons based non-numerical ranking, in Proc. of SERP 2010(Int. Conf. on Software Engineering Research and Practice), 2010, to appear.

Appendix A: Automatic non-numerical ranking system manual A.1 Installation The automatic non-numerical ranking system can be download from the website: http://www.cas.mcmaster.ca/~zhaiy/ A.1.1 System requirements Windows 2000 Professional, Windows XP Professional, Windows Server 2000, Windows Server 2003 Microsoft.NET Framework 2.0 Runtime Library or later Intel Pentium III processor (or equivalent) or later 100 MB available physical memory 25MB available disk space 205

A. Automatic non-numerical ranking system manual 206 A.1.2 Automatic non-numerical ranking system installation Install the Automatic Non-numerical Ranking System to the destination hard disk. Double click Non-numericalRank.exe to run. A.2 Automatic ranking system functions A.2.1 Loading file When the application starts, Select the File Open... menu item, When presented with openfile dialog box, select the file to be processed, and click the OK button. Note that there are two types of files that can be loaded, one is full matrix and the other is half matrix with weight information. Since it is difficult to type,, and in the textpad, we use 1, 3, 5, 7, 9 to represent,,, >,, and use 1/3, 1/5, 1/7, 1/9 to represent,, <, in our input files. Type A of input file Type A of input file is full matrix. An example of type A input file is presented in Figure A.1. This example is illustrated in Chapter 5, which has 6 objects and the relations among them are A B C D E F A. Type B of input file Type B of input file is half matrix, and its file name starts with half. An example of input file having half matrix and with weight information, is shown in Figure A.2.

A. Automatic non-numerical ranking system manual 207 Figure A.1: An example of type A input file Figure A.2: An example of type B input file

A. Automatic non-numerical ranking system manual 208 A.2.2 AHP ranking As soon as an input file is loaded, the system automatically shows the AHP Ranking result. User can also select View AHP Ranking... menu item to view it. An example of AHP Ranking is illustrated in Figure A.3 Figure A.3: An example of AHP ranking User can click the Rank Reversal Information tab to check if there exists a rank reversal, shown in Figure A.4.

A. Automatic non-numerical ranking system manual 209 Figure A.4: An example of AHP rank reversal information

A. Automatic non-numerical ranking system manual 210 A.2.3 Consistency-driven non-numerical ranking Select View Non-numerical Ranking Consistency-driven... menu item to view consistency-driven ranking information, which includes the first pre-ranking, the first pre-ranking violation data, final consistent pairwise comparisons data, final ranking results and rank reversal information. An example is shown in Figure A.5. Figure A.5: An example of consistency-driven ranking Click the Rank Reversal Information tab to check if there exists a rank reversal

A. Automatic non-numerical ranking system manual 211 for the consistency-driven, displayed in Figure A.6. Figure A.6: An example of consistency-driven with rank reversal information A.2.4 Property-driven classical partial order approximations Select View Non-numerical Ranking Property-Driven Classical Partial Order Approximation... menu item to view property-driven classical partial order approximations ranking information, which includes the first combined PRS P, (P ) +,

A. Automatic non-numerical ranking system manual 212 (P + ), (P ), (P ), global scores and rank reversal information for the four classical partial order approximations (P ) +, (P + ), (P ) and (P ). An example is illustrated in Figure A.7. Figure A.7: An example of property-driven classical partial order approximations Click the Rank Reversal Information tab to check if there exists rank reversal for each of the property-driven classical partial order approximation, shown in Figure A.8.

A. Automatic non-numerical ranking system manual 213 Figure A.8: An example of property-driven classical partial order approximations with rank reversal information

A. Automatic non-numerical ranking system manual 214 A.2.5 Property-driven refined partial order approximations Select view Non-numerical Ranking Property-Driven Refined Partial Order Approximation... menu item to view property-driven refined partial order approximations ranking information, which includes the combined PRS P, refined partial order approximations (P ) +, (P + ), (P ), (P ), global scores and rank reversal information for the four refined partial order approximations (P ) +, (P + ), (P ) and (P ). An example is illustrated in Figure A.9. Click the Rank Reversal Information tab to check if there exists rank reversal for the property-driven refined partial order approximations, shown in Figure A.10. A.2.6 Property/consistency-driven refined partial order approximations Select view Non-numerical Ranking Property-Consistency Driven... menu item to view the property-consistency driven, which includes property-driven refined partial order approximations (P ) +, (P + ), (P ), (P ). An example is shown in Figure A.11. Click the Rank Reversal Information tab to check if there exists rank reversal for the property/consistency-driven refined partial order approximations, shown in Figure A.12. A.2.7 Variously driven algorithms Select view All... menu item to view all the driven algorithm outcomes, shown in Figure A.13. Click the Rank Reversal Information tab to check if there exists rank reversal for

A. Automatic non-numerical ranking system manual 215 Figure A.9: An example of property-driven refined partial order approximations

A. Automatic non-numerical ranking system manual 216 Figure A.10: An example of property-driven refined partial order approximations with rank reversal information

A. Automatic non-numerical ranking system manual 217 Figure A.11: An example of property/consistency-driven algorithm

A. Automatic non-numerical ranking system manual 218 Figure A.12: An example of property/consistency-driven algorithm with rank reversal information

A. Automatic non-numerical ranking system manual 219 Figure A.13: An example of variously driven algorithm comparisons

A. Automatic non-numerical ranking system manual 220 the non-numerical ranking algorithms, shown in Figure A.14. Figure A.14: An example of our algorithms with rank reversal information A.2.8 Consistency-driven ranking by combining and Select Advanced Non-numerical Ranking By Combination of 1/3 and 1/5 Consistency-Driven Ranking By Combination of 1/3 and 1/5... menu item to view the consistency-driven algorithm ranking by the combination of and outcome,

A. Automatic non-numerical ranking system manual 221 shown in Figure A.15. Figure A.15: An example of consistency-driven ranking by the combination of and Click the Rank Reversal Information tab to check if there exists rank reversal for the consistency-driven algorithm ranking by the combination of and, displayed in Figure A.16.

A. Automatic non-numerical ranking system manual 222 Figure A.16: An example of consistency-driven ranking by the combination of and

A. Automatic non-numerical ranking system manual 223 A.2.9 Property/consistency-driven ranking by the combination of and Select the Advanced Non-numerical Ranking By Combination of 1/3 and 1/5 Property-Consistency Driven Ranking By Combination of 1/3 and 1/5... menu item to view the property/consistency-driven algorithm ranking by the combination of and outcome, displayed in Figure A.17. Click the Rank Reversal Information tab to check if there exists rank reversal for the property/consistency-driven algorithm ranking by the combination of and, illustrated in Figure A.18. A.2.10 Our algorithms ranking by combining and Select Advanced All Ranking By Combination of 1/3 and 1/5... menu item to view all algorithms ranking by the combination of and outcome, displayed in Figure A.19. Click the Rank Reversal Information tab to check if there exists rank reversal for all algorithms ranking by the combination of and, illustrated in Figure A.20. A.2.11 Output All algorithm outcomes in detail are saved in the output folder automatically, as displayed in Figures A.21 and A.22.

A. Automatic non-numerical ranking system manual 224 Figure A.17: An example of property/consistency-driven ranking by combining and

A. Automatic non-numerical ranking system manual 225 Figure A.18: An example of property/consistency-driven with rank reversal information ranking by the combination of and

A. Automatic non-numerical ranking system manual 226 Figure A.19: An example of various driven ranking by combining and

A. Automatic non-numerical ranking system manual 227 Figure A.20: An example of variously driven ranking with rank reversal, by combining and

A. Automatic non-numerical ranking system manual 228 Figure A.21: An example of saved output

A. Automatic non-numerical ranking system manual 229 Figure A.22: An example of saved output - (continued)