A STUDY ON THE UTILIZATION OF COMPATIBILITY METRIC IN THE AHP: APPLYING TO SOFTWARE PROCESS ASSESSMENTS

Transcription

1 ISAHP 2005, Honolulu, Hawaii, July 8-10, 2003 A SUDY ON HE UILIZAION OF COMPAIBILIY MERIC IN HE AHP: APPLYING O SOFWARE PROCESS ASSESSMENS Min-Suk Yoon Yosu National University San 96-1 Dundeok-dong Yeosu City Jeonnam, Korea, msyoon@yosu.ac.kr Ho-Won Jung Korea University 5-1 Anam-dong Sungbuk-gu Seoul, Korea, hwung@korea.ac.kr Keywords: comatibility, grou decision making, software rocess assessments, conflict resolution Summary: his study intends to extend the alicability of the Analytic Hierarchy Process (AHP) to software rocess assessments. Recently an AHP aroach has been successfully alied to software rocess assessments esecially with regard to boundary roblems, ambiguities between two adacent ratings. Boundary roblems may cause a roblem in the rocess of inter-rater agreement for a rocess attribute rating among assessors. However, the aroach assumes that the assessors reach a consensus for riorities of the associated ractices. When assessors cannot reach a riority consensus, a more systematic method is required to make a consensus among those assessors having different sets of riorities. In order to solve this roblem, this study rooses a consolidating method among conflicting assessors using comatibility metric of the AHP, and shows its alication stes. 1. Introduction For about three decades, the usefulness of the Analytic Hierarchy Process (AHP) has been verified by rich alications, not only in management science areas (AHP secial issues, 1990), but also in software research areas such as software reliability alication (Zahedi and Ashrafi, 1991), software requirement analysis (Karlsson and Ryan, 1997), reusable comonent evaluation (Kontio, Caldiera, and Basili, 1996), software roduct evaluation (Jung and Choi, 1999), software quality evaluation (Yoon, 1997), Commercial- off-the-shelf software selection (Maiden, Ncube, and Moore, 1997) and others. Recently, Jung (2000) roosed a method that utilizes the AHP, in order to deal with boundary roblems in software rocess assessments based on ISO/IEC R (1998), which was develoed to meet the needs and requirements for a standardized software rocess assessment. he AHP aroach was helful in obectively ustifying the rating rocesses and in increasing the confidence of assessment results. Furthermore, the AHP aroach could be useful in tackling boundary roblems, which used to occur when some difficulties would arise in understanding the boundary between two adacent ratings: Partially" and "Largely", "Largely" and Fully", and "Not Achieved" and "Partially". Many assessors have exerienced such boundary roblems (El Emam and Jung, 2001). Under the circumstances of boundary roblems, determining the caability of a software rocess would become a redicament. he ambiguity in rating can usually lead to inter-rater disagreement; different ratings for a rocess attribute (PA) among assessors and an increase in assessment effort. Reaching a consensus in ratings among assessors was the most influential factor to assessors effort (El Emam et al.,

2 1998). herefore the AHP aroach seems be more effective in solving ambiguous roblems that require a systematic aroach, in addition to re-investigating evidence and collecting more information. he aim of this study is to extend the alicability of the AHP, when used in solving boundary roblems. he situation being focused on is inter-rater disagreement in rating a rocess, which is caused by disagreement on riorities [weights], in site of agreement on achievements, of a set of associated ractice indicators (hereafter referred to as PIs). Even though the AHP is emloyed in this kind of boundary roblem, if the assessors do not easily reach a consensus on the riorities, the critical research question still arises: How can a consensus be derived from assessors with obviously different sets of airwise comarison matrices in evaluating riorities of a set of PIs? In order to answer this question, a systematic aroach is required. his study rooses a consolidation method of the inter-rater conflicting riorities, utilizing the comatibility index of the AHP for grou udgment situations. he remainder of this aer is organized as follows: Section 2 overviews software rocess assessments utilizing the AHP. Section 3 briefly addresses the inter-rater disagreement rationale and resents an extension of the AHP aroach, which is our revised method using the comatibility of the AHP. Section 4 illustrates the alication stes of our method, and Section 5 concludes this aer with final remarks. 2. Software Process Assessment Using the AHP 2.1 A brief review of ISO/IEC he architecture of the ublished ISO/IEC standard consists of both rocess and caability dimensions. Figure 1 shows the structure of the two dimensions. In the rocess dimension, the rocesses associated with software are defined and classified in the rocess reference models such as ISO/IEC (2004) and CMMI (2002). For examle, ISO/IEC (2004) rovides five categories of software rocesses; Customer- Suly rocess, Engineering rocess, Suort rocess, Management rocess, Organization rocess categories. Caability dimension PA 5.2 : Process otimization PA 5.1 : Process innovation Level 5 PA 4.2 : Process control PA 4.1 : Process measurement Level 4 PA 3.2 : Process deloyment PA 3.1 : Process definition Level 3 PA 2.2 : Work roduct management PA 2.1 : Performance management Level 2 PA 1.1 : Process erformance Level 1 Incomlete Level 0 Process dimension Process A Process B Process C Figure 1 - wo-dimensional architecture of ISO/IEC he caability dimension is comrised of six caability levels ranging from 0 to 5. he greater the level is, the greater the rocess caability achieved. Each level of rocess caability is deicted by one or two rocess attributes (PAs) as shown in Figure 1, alicable to any software rocess, which reresent

3 measurable characteristics necessary to manage a rocess and to imrove its erformance caability. he caability level is summarized as follows: Level 0, Incomlete: he rocess is not imlemented, or fails to achieve its rocess urose. At this level there is little or no evidence of any systematic achievement of the rocess urose. Level 1, Performed: he imlemented rocess achieves its rocess urose. Level 2, Managed: he reviously described Performed rocess is now imlemented in a managed fashion (lanned, monitored and adusted) and its work roducts are aroriately established, controlled and maintained. Level 3, Established: he reviously described Managed rocess is now imlemented using a defined rocess that is based uon a standard rocess and that is caable of achieving its rocess outcomes. Level 4, Predictable: he reviously described Established rocess now oerates within defined limits to achieve its rocess outcomes. Level 5, Otimizing: he reviously described Predictable rocess is continuously imroved to meet relevant current and roected business goals. he caability level is determined by measuring its PAs and each PA is measured by an ordinal rating Fully, Largely, Partially, or Not achieved as in able 1. he ordinal rating scale is a transformation of a numerical value between 0% and 100% that reresents the extent of achievement of a PA. he achievement of caability level k indicates that all PAs rior to level k satisfy the rating Fully and then level k's attributes are rated as Fully or Largely. able 1 - he rating scale of the rocess attributes (ISO/IEC ) Acronym Achievement of the defined attribute N: 0% to 15%: here is little or no evidence of achievement of the defined Not achieved attribute in the assessed rocess. P: 16% to 50%: here is evidence of a sound systematic aroach to and Partially achieved achievement of the defined attribute in the accessed rocess. Some asects L: Largely achieved F: Fully achieved of achievement may be unredictable. 51% to 85%: here is evidence of a sound systematic aroach to and significant achievement of the defined attribute in the accessed rocess. Performance of the rocess may vary in some areas or work units. 86% to 100%: here is evidence of a comlete and systematic aroach to and full achievement of the defined attribute in the assessed rocess. No significant weaknesses exist across the defined organization unit. he existence of base ractices and work roducts rovide evidence of the erformance of the rocesses associated with them (PA 1.1). he extent of the achievement for PA 2.1 to PA 5.2 in the caability dimension have a set of associated rocess attribute indicators, which rovides an indication of the extent of achievement of the attribute in the instantiated rocess. hese indicators concern significant activities, resources or results associated with the achievement of the attribute urose by a rocess. PIs are the rincial indicators of rocess caability. Most of PIs are instances of generic ractices defined in the exemlar model of ISO (2003). 2.2 AHP alication for boundary roblems If assessors cannot easily reach consensus of rating and they agree on the use of the AHP aroach, this ste is activated. he generally outlined the AHP stes are followed in order to determine the rating of a PA with associated its indicators. he AHP starts by breaking down the decision roblem into interrelated decision elements (criteria, attributes, or factors). Note the decision roblem corresonds to rating of a PA and the AHP criteria do to PIs belonging to the PA. he comosite value of weights and achieved values of PIs is transformed into the rating of a PA as defined in able 1. In order to obtain weights for a set of n PIs, the AHP begins with construction of a

4 airwise comarison matrix according to the relative imortance of the PIs. he comarison matrix is defined as A = ( a ) n n. he scale a is an estimate for w i / w, where w i and w indicate the imortance of the ith and the th PI, resectively. he matrix A has ositive entries everywhere and satisfies the recirocal roerty, a i = 1/ a, which is called a recirocal matrix. he weight vector of PIs is generated by the following equation: Aw = λmaxw (1) where λmax is the rincile eigenvalue of the matrix A and w = ( w1, w2,, wn ) is the weight vector, corresonding to λ. o make the vector unique, let the sum of weights be 1. max If a given udgment matrix, A = ( a ) n n is erfectly consistent, i.e., a i = 1/ a and a ik ak = a for all i,, k, then the rincile eigenvalue, λ max, equals n. However, in a real decision making environment, eole's erceived relative references in airwise comarisons remain inconsistent and intransitive. A small erturbation around A leads to an eigenvalue roblem, λ max n from equation (1). Inconsistency throughout the matrix A can be catured by a single number, λ max n, which measures the deviation of the udgments from the consistent aroximation and leads to the consistency index ( CI ) as follows; CI = ( λ max n) ( n 1) (2) Random index ( RI ) is obtained as an average over CI values from large number of randomized recirocal matrices ( RI will be seen later in able 2). he ratio between the two indices ( CI / RI ) is defined as consistency ratio ( CR ). If CR is less than or equal to 0.1, Saaty (1980) recommends that the estimate of the weights be acceted. Otherwise, the comarison matrix re-created. his rocess is reeated until the threshold condition is satisfied. Finally, the AHP aroach aggregates the weights and the measured values of PIs as follows: R n = = 1 w r (3) where R is the comosite value and r is the measured value for the th PI. If all r s are measured by a value between 0 and 1, then the value R also has a value between 0 and 1. Hence, the value R is automatically transformed to a rating of Fully, Largely, Partially, or Not achieved. 3. Consolidation method 3.1 Inter-rater disagreement rationale Commonly, team-based assessors articiate in rating a software rocess assessment. he assessors make their own reliminary rocess ratings based on the interretation of their assessment record. hese are then discussed during the consolidation activity, and a consensus is made by the assessment team. he consensus is formed on the final ratings, as well as the evidence and findings for the achievements of PIs. During a team-based assessment, assessors are exosed to the same evidence. his evidence can be the result of re-onsite questionnaires, the resonses to questions during an interview, or from the document insection (SPICE trials Reort, 1999). Whereas, the imortance of each PI of a PA deends on an assessed rocess context, such as alication domain, business urose, develoment methodology, organization size, etc. Hence, each PI should have a different riority deending on its imacts on the PA (Jung, 2001). In some cases, even though the team assessors agree on the achievement of each PI, the

5 assessors would disagree on the final ratings because of differences in thinking of riorities for a set of PIs. his asect is more critical in the boundary roblems because different riorities lead to different ratings among assessors. It is logical to exect that the more disagreement exits among the assessors, the more effort will be sent on consolidation (El Emam et al., 1998). Even though the AHP method is emloyed to determine the ratings, there still remains consolidating different airwise udgments among the team assessors that lead to inter-rater disagreement on the riorities. A well-known method for such grou decision-making in the AHP is to take the geometric mean of individual udgments because the recirocal of the geometric mean of the udgments becomes the geometric mean of the recirocals (Saaty, 1980). In this method, removing outliers and taking the geometric mean of the rest hels to avoid maleficence of the outliers. However, since not many assessors articiate in rating a software rocess, removing outliers is not roer; there is not enough udgment data to waste. his method, even if ossible, requires a tool to check outliers. In this regard, we roose an alternative method to make iterative individual udgments toward a grou consensus after feedback of the grou outcome in each iteration stage. his method also requires a tool to measure how close an individual outcome is to the grou outcome and a stoing oint of iteration to insure that individual outcomes reach a consensus. he comatibility index of the AHP can satisfy those requirements. 3.2 Comatibility Metric Comatibility in the AHP is concerned with two different vectors derived from two udgments matrices. In accordance with a grou udgment of the AHP, Saaty (2001) develoed the comatibility metric that measures how mutually close two airwise recirocal matrices of two ratio vectors are. he metric was analytically derived from the relation between a matrix of udgments and the matrix of corresonding eigenvector ratios. At the beginning on comatibility metric, let Error! Obects cannot be created from editing field codes. be the Hadamard roduct in this aer and its multilication between the two given matrices W and V is defined as follows: W V = w v ), where W = w ) and V = v ) (4) ( ( ( Measuring comatibility between the two ratio scales = v = v, v,, v ) ( 1 2 n is defined below. w w, w,, w ) ( 1 2 n and e W V e, where e = ( 1,1,,1), W = ( w i w ), and V = ( v i v ) (5) 2 If two matrices are exactly the same, then e W V e = n. Comatibility between two ratio vectors can be determined using comatibility index (S.I.) which is derived from the relation between comatibility and consistency. Consistency is concerned with the comatibility of a matrix of the ratios constructed from a rincial right eigenvector with the matrix of udgments from which it is derived. Comatibility is concerned with two different vectors. Let W be the matrix of ratios of the rincile right eigenvector of the ositive recirocal matrix A, and λ max be the corresonding rincial eigenvalue of A. Using the two matrices, W and A, the Comatibility Index (S.I.) is defined as follows; S. I. = n 2 e A W e (6) S.I. becomes 1 if and only if the two matrices are exactly the same (i.e., matrix A is erfectly consistent). Otherwise, S.I. goes beyond 1.

6 Since equation (6) equals λ max / n (Saaty, ), the right hand side of (6) can be relaced by 1+ CI ( n 1) n using equation (2). From the accetance level of C.I., we can derive the significance level of S.I. to assure that two matrices of ratio vectors are comatible. able 2 gives information on comatibility and consistency for different size udgment matrices. able 2 - Relationshi between Consistency and Comatibility (Saaty, ) Size (n) C.R. R.I. C.I. λ max S.I Consolidation Method Our method can be triggered at the stage of consolidation after assuring inter-rater disagreement in rating a PA, in site of agreement on the achievements of the associated PIs, among assessors. Assuredly, this kind of boundary roblem is caused when individual assessors have different sets of PI riorities. he key idea of our method is drawn from the Delhi method, which is one of the qualitative grou decision making techniques. he Delhi method takes iterative grou rocesses, which use central tendency to make individual decision makers gradually converge toward a osition. Our method consolidates individually different udgment outcomes until all individual outcomes are comatible with the grou outcome. he latter can be obtained by taking the geometric mean of the comosite outcome of several individuals. In detail, after assessing the comatibility of the matrix of ratios of individual with that of the grou, one can suggest to each individual which of his ratios is the most incomatible with that of the grou and roose changes in his thinking to make it more comatible. hrough such revision and recalculation of the grou outcome, one may be able to obtain a grou decision that is comatible with each member. he comatibility test can be used obectively to assess how a airwise comarison matrix of an individual is close to that of grou udgments. We use derivatives of a given udgment matrix. Harker (1987) used the derivatives of a given incomlete airwise comarison to determine an element that has the greatest influence on the weight vector. Following notations describe our method and resent an examle in the next section. P = ( ) : he matrix of ratios of individual assessor W = ( w ) : he matrix of ratios of a grou udgment M : he comatibility matrix, M = P W = [ ( w / wi )] = [ w i ] f(m) : he comatibility, f ( M ) = e Me = e [ w ] e. Following stes describe our method. Ste 0: Create the matrices of ratios of each individual assessor and the grou. Ste 1: Comute comatibility (f ) between each matrix of individual udgment ratios (P) and the matrix of grou udgment ratios (W). If there is any P that gives comatibility beyond redetermined level, then go to the next ste. Else terminate rocesses. Ste 2: For such a P in the revious ste, find the (i, ) element that gives the largest absolute i

7 value of the coefficient of gradient of f with resect to. Ste 3: Guide the decision maker of P to adust the (i, ) element considering w Ste 4: Calculate new vector ' and, go back to Ste 0. Concerning ste 2 above, the derivatives of M with resect to a matrix element, of the form is an n n matrix [ M / ] kl = w w i 0 2 if if k = i, l = k =, l = otherwise i hen derivatives of comatibility with resect to each is also an n n matrix of the form, 2 [ f / ] = [ w w ] (7) In this matrix, we can find the largest gradient element that can reduce the current comatibility value. he choice of the cell to adust toward grou udgment is (i, ) that gives the largest absolute value of the coefficient of gradient of f with resect to, i.e., i ( i, ) = argmax( f / ) (8) k, l After ste 3, one can construct the udgment matrix P using new riority vector ( ' ), and comute comatibility between new P and W as follows: kl n 2 e P W e (9) According to whether or not the measured value is less than a redetermined value such as S.I. in able 2, one can sto or continue the consolidation rocesses. Concerning the comutational comlexity, the following equation lightens the comutational workload. e P W e = e = w i w i e = e e wi wi w i i = wi i w i i w (10) 4. An illustrative examle In the context of AHP, we will illustrate an examle based on the rating of a PA with four PIs. he situation assumes that three assessors articiate in the assessment. In the initial rating of able 3, the first and the second assessors' ratings are Largely and the other s is Partially, which indicates inter-rater disagreement. he achievement scores for four PIs are agreed among assessors and given as 0.75, 0.60, 0.25, and 0.40 resectively. his kind of situation is regarded as a boundary roblem. able 3 resents three airwise comarison matrices and the geometric mean matrix of the three matrices. he weight vector for each matrix is obtained through the eigenvector method. he comosite value for a

8 rating is calculated by the multilication of the weight vector with the achievement score vector of four PIs as in equation (3). It is logical to exect that a rating by the holistic evaluation should be consistent with that by the decomosed evaluation utilizing the AHP. Let SI G be the comatibility between the matrix of assessor l and that of the grou. According to equation (9), the comatibility indices between each individual matrix and the grou matrix aear as SI 1 G =1.023, SI 2 G =1.112 and SI 3 G = he third assessor shows the greatest incomatibility with the rest assessors in overall thinking of relative imortance. he corresonding significance level of comatibility is in able 2. he different riorities among assessors cause different ratings. At first iteration, according to equation (8), we can find the element that gives the largest absolute value of the coefficient of gradient to comatibility. For the udgment matrix of assessor 2, the element in the (4, 1) osition is recommended to change. Assume that the new udgment is 1/4 from 1/5, considering the value of grou mean (1/2.71). For the case of assessor 3, assume the current value of (1, 3) changes to 1 (See able 4). Calculate the new riorities vectors for both assessor 2 and assessor 3, and go back to ste 1. Since SI 3G (= 1.119) > in able 4, the second iteration is activated only for assessor 3. According to equations (7) and (8), the (1, 4) osition in the matrix for assessor 3 is to be revised. Let it be 2 from 1 in able 5. hen, since all the comatibility indices between each matrix and the grou matrix satisfy the stoing condition ( SI 1 G =1.015; SI 2 G =1.040; SI 3 G = ), we can determine the final rating. According to the final ste of our method, the comosite value becomes 54% and thus, rate the illustrated rocess attribute as Largely. able 3 - Initial rating information Assessor 1 Assessor 2 Assessor 3 Geometric Mean Matrix Weights of PI Measures of PI PI 1 = 0.70, PI 2 = 0.60, PI 3 = 0.25, PI 4 = 0.40 Comosite Value (Rating) 54% 57% 44% (Partially) SI lg SI 1G = 1.023, SI 2G = 1.112, SI 3G = able 4 - Revised rating information (1st iteration) Assessor 1 Assessor 2 Assessor 3 Geometric Mean Matrix Weights of PI Measures of PI PI 1 = 0.70, PI 2 = 0.60, PI 3 = 0.25, PI 4 = 0.40 Comosite Value (Rating) 54% 57% 48% (Partially) SI lg SI 1G = 1.034, SI 2G = 1.053, SI 3G = 1.119

9 Matrix able 5 - Revised rating information (2nd iteration) Assessor 1 Assessor 2 Assessor 3 Mean Weights of PI Measures of PI PI 1 = 0.70, PI 2 = 0.60, PI 3 = 0.25, PI 4 = 0.40 Comosite Value (Rating) 54% 57% 50% 54% SI lg SI 1G = 1.015, SI 2G = 1.040, SI 3G = Concluding Remarks In this study, we roosed a consolidating method that could systematically draw a grou consensus on a set of riorities of ractice indicators in software rocess assessments, based on ISO utilizing the AHP. Our method uses an iterative mode of udgments, which gradually narrows differences among assessors' evaluation, in order to reach the oint where the differences are so trivial that individual udgments can be comatible. he comatibility metric of the AHP is an indicator to reresent the degree of comatibility between individual udgments with a grou udgment. his study also illustrated the alication stes of our method with an aroriate examle in a software rocess assessment. he roosed method can be activated when inter-rater disagreement arises through different airwise comarison matrices for occasions such as boundary roblems that require careful and elaborate assessments. his method would be used to draw an agreement with the Organization Unit in resenting assessment results, as well as to reach a consensus among assessors. his would ultimately contribute to quality assessments. he roosed method is obviously a time-consuming technique. However, considering the trade-off between the evaluation accuracy and quality, and the time required to aly this method, we roose that this method is worthy of oerating for such occasions as boundary roblems. References AHP secial issue (1990) - Decision Making by the Analytic Hierarchy Process: heory and Alications, Euroean Journal of Oerational Research, 48/1. CMMI (2002), Caability Maturity Model Integration, SE/SW/IPPD/SS, V 1.1, CMU/SEI-2002-R-011, Software Engineering Institute, Carnegie Mellon University. El Emam K., Simon J. M., Rousseau S., Jacquet E. (1998), Cost Imlications of Interrater Agreement for Software Process Assessment, Proceedings of the 5th. International Symosium on Software Metrics, El Emam K., Jung H.-W. (2001), "An Evaluation of the ISO/IEC Assessment Model", Journal of Systems and Software. 59/1, Harker, P.. (1987), Incomlete Pairwise Comarisons in the Analytic Hierarchy Process, Mathematical Modelling, 9/

10 ISO/IEC AMD (2004), Information echnology - Software Life Cycle Processes, ISO. ISO/IEC R (1998); 5 (1999), Information echnology - Software Process Assessment: Part 2; Part 5, ISO. ISO/IEC (2003); 5 (2004), Information echnology - Process Assessment: Part 2; Part 5 (FCD), ISO. Jung H.-W. (2001), "Rating the Process Attributes Utilizing AHP in SPICE-based Process Assessments", Software Process Imrovement and Practices: International Journal 6/2, Jung, H. W. and Choi, B. J.(1999), "Otimizing Models for Quality and Cost of Modular Software Systems, Euroean Journal of Oerational Research, 112/3, Karlsson J. and Ryan, K.(1997), "A Cost-Value Aroach for Prioritizing Requirements," IEEE Software, Se./Oct., Kontio, J., Caldiera, G., and Basili, V. (1996), "Defining Factors, Goals, and Criteria for Reusable Comonent Evaluation", CASCON '96 Conference, oronto, Canada. Maiden, N., Ncube, C., and Moore, A. (1997), Lessons learned during requirements acquisition for COS systems, Communications of the ACM, 40/12, Saaty,. L. (2001), he Analytic Network Process (2 nd Edi.), RWS Publications, Pittsburgh. Saaty,. L. (1980), he Analytic Hierarchy Process, McGraw-Hill, New York. SPICE rials Reort (1999), SPICE Phase 2 rials Final Reort, Vol. 1, ISO/IEC JC1/SC7/WG10. Yoon, M. S. (1997), S/W Quality Evaluation Model Usig the AHP - Develoing a New Judgments Aggregation Method, Ph. D. Dissertation, Korea University. Zhedi, F., and Ashrafi, N. (1991), "Software Reliability Allocation Based on Structure, Utility, Price, and Cost," IEEE ransactions on Software Engineering, 17/4,