Improving subjective estimations using the paired comparisons method

Transcription

1 Improvng subectve estmatons usng the pared comparsons method Internatonal Forum on COCOMO and Software Cost Estmaton Eduardo Mranda October 24-27, 2000

2 Agenda Subectve estmates Estmatng person-hours Can we do better? - The pared comparson method The process Example Evaluaton Why does t work? Summary & further research

3 Subectve estmatons are a way of lfe for many actvtes and organzatons Lack of necessary nformaton at the begnnng of the proect Vocabulary foregn to the stakeholders Some technques are doman specfc Requre sgnfcant tme and effort

4 Estmatng person-hours LMC-Duraton LMC-Mhrs LMC-FP TEI-Duraton TEI-Mhrs TEI-FP -1.5

5 Can we do better? - The pared comparson method LOC stack queue bnary tree Actual lnked lst (a) Pared comparsons (numerc scale) reference (strng) modules lnked lst (b) balanced tree hash table Fnger n the wnd (Ad-hoc) Pared comparsons (verbal scale)

6 The pared comparsons process Artfacts to be szed Rank the artfacts from largest to smaller Ordered Lst Compare the artfacts parwse establshng ther relatve sze Judgement Matrx Revew nternal nconsstences Defnton Explanaton Relatve Recprocal Value Equal sze E / E (0~25%) Slghtly bgger 1.25< E / E (smaller) (25 75%) Bgger (smaller) 1.75 < E / E ( %) Much Bgger < E / E (smaller) ( %) Extremely bgger 5.75 < E / E (smaller) ( %) Verbal Scale (optonal) Calculate rato scale & Inconsstency Index Rato Scale Reference Artfact(s) Calculate absolute szes Szed Artfacts

7 The math - udgement matrx Artfacts g a b c d I h e f g a b c d I h 1 2 e 2 f A nxn = [ a ] a = a a = = s s = 1 1 a, How much bgger (smaller) s E, Every element has the same sze astself, If E s a (bgger) than E wthrespect to E tmes bgger (smaller) than E, then E s1/a tmes smaller

8 The math - rato and sze calculatons ( )( ) ln ln (d) (c) (b) (a) = = = = > = = n n a InconsstencyIndex r r * s s r r * s r r * s r a n n k n k k k k k k n l l n n ν ν ν ν ν Crawford and Wllams procedure. (a) Row s geometrc mean, (b) Rato scale, (c) Sze vector, (d) Inconsstency Index

9 Verbal scales Defnton Software Relatve Value AHP Relatve Values Equal sze 1 1 Slghtly bgger (smaller) Bgger (smaller) Much Bgger (smaller) 4 7 Extremely bgger (smaller) 7.5 9

10 MnmumTme - automatng the process Artfacts to be szed Rank the artfacts from largest to smaller Compare the artfacts parwse establshng ther relatve sze Revew nternal nconsstences Reference Value(s) Szed Artfacts

11 Example B G F C D E I J

12 Evaluaton Stack Queue Bnary Tree Hash Table Balanced Tree Lnked Lst (a) Lnked Lst (b) Strng Manpulaton Ad-hoc Pared Comparsons Fgure a g f c b d a g f c b Fgure a b c d h I Sze a d g e f b h c Defnton Explanaton Relatve Value Equal sze E / E (0~25%) Slghtly bgger 1.25< E / E (smaller) (25 75%) Bgger (smaller) 1.75 < E / E (75 275%) Much Bgger < E / E (smaller) ( %) Extremely bgger (smaller) 5.75 < E / E 10 6 ( %) Verbal Scale Fgure a g f c b d a g f c b

13 Evaluaton - data structures LOC stack queue bnary tree Actual lnked lst (a) Pared comparsons (numerc scale) reference (strng) modules lnked lst (b) balanced tree hash table Fnger n the wnd (Ad-hoc) Pared comparsons (verbal scale)

14 Evaluaton - geometrc fgures Actual Ad-hoc pared comparson verbal scale

15 Evaluaton - geometrc fgures reference Ad-hoc pared comparson verbal scale

16 Evaluaton - accuracy Are all the methods the same? SUMMARY Groups Count Sum Average Varance fw pc vs ANOVA Source of Varaton SS df MS F P-value F crt Between Groups Wthn Groups Are there any dfferences between PC and VS? vs pc Mean Varance Observatons 7 9 Hypotheszed Mea 0 df 10 t Stat P(T<=t) one-tal t Crtcal one-tal P(T<=t) two-tal t Crtcal two-tal Total Is PC better than FIW? fw pc Mean Varance Observatons 10 9 Pooled Varance Hypotheszed Mea 0 df 17 t Stat P(T<=t) one-tal t Crtcal one-tal P(T<=t) two-tal t Crtcal two-tal Conclusons The methods have dfferent accuracy The pared comparsons method shows better performance than the ad-hoc approach There s not sgnfcant dfference between estmatons usng numbers and those based on the verbal scale

17 Why does t work? Dfferences rather than absolute values Explct pared comparsons Redundant values The result s an average of many ndvdual observatons

18 Summary and further research The pared comparsons method provdes an structured approach for dong subectve estmatons whch s: more precse more accurate than ad-hoc approaches. Future research: Corroboraton of results Valdatng software verbal scale Applcaton nto domans others than software szng (effort, lead tme)

19 References G. Bozok, An expert udgement based software szng model, Lockheed Mssles & Space Company and Target Software J. Karlsson & K. Ryan, A Cost-Value Approach for Prortzng Requrements, IEEE Software, September/October 1997 T. Saaty, Multcrtera Decson Makng: The Analytc Herarchy Process, RWS Publcatons, 1996 G. Crawford and C. Wllams, The Analyss of subectve udgement matrces, Rand Corporaton, 1985

20 The End