Error Propagation in Software Measurement & Estimation

Transcription

1 0h IMEKO TC4 Inernaonal Symposum and 8h Inernaonal Workshop on ADC Modellng and Tesng Research on Elecrc and Elecronc Measuremen for he Economc Upurn Beneveno, Ialy, Sepember 5-, 04 Error Propagaon n Sofware Measuremen & Esmaon Luca Sanllo GUFPI-ISMA (Gruppo Uen Funcon Pon Iala Ialan Sofware Mercs Assocaon), luca.sanllo@gmal.com Absrac The real challenge n any acvy s o mnmze as much as possble he error beween an esmae and an acual value, whaever he phenomenon o be evaluaed. When dealng wh sofware, he number of proxes can be que hgh: he applcaon of an algorhm ncludng one or more ndependen varables (measures) s fnalzed o provde one or more oupu varables (esmaes) for a seres of measures ypcally abou effor, cos, me, qualy or oher aspecs of he sofware beng developed. Recenly ISO proposed also a specfc sandard on Measuremen (ISO/IEC 599), wh a glossary algned o he Merology feld and o he Inernaonal Vocabulary of Merology (VIM). Esmaon could be seem as a black ar, bu error s nrnsc n esmaes and mus be managed. Thus, regardless of he esmaon model (algorhm) beng used, praconers mus face he uncerany aspecs of such process: errors n nal measures do affec he derved mercs (or esmaed values for ndrec varables). Measuremen heory does provde an accurae way o evaluae such error propagaon for algorhmc dervaon of varable values from drec measures, as n he GUM (Gude o he expresson of Uncerany n Measuremen). Alhough some sofware esmaon models already propose confdence ranges on her resuls, he formal applcaon of error propagaon can yeld some surprsng resuls, dependng on he mahemacal funconal form underlyng he model beng examned. Movng from a prevous paper, hs one wll dscuss he propagaon of errors n sofware measuremen and shows wh some applcaon and examples based on some of he mos common sofware measuremen mehods and esmaon models as Funcon Pon Analyss (FPA) for produc szng ssues and COCOMO (Cos Consrucon Model) for effor and/or duraon ssues as well as oher ones, updang also he dscusson o new advancemen n he Sofware Qualy feld, n parcular abou produc NFRs (Non-Funconal Requremens). Few cases and examples wll be shown, n order o smulae a crcal analyss for mehods and models beng examned from a possbly new perspecve, wh regards o he accuracy hey can offer n pracce. Keywords Measuremen, esmaon, accuracy, propagaon of uncerany, error analyss. I. INTRODUCTION In scence, he erms unceranes or errors do no refer formally o msakes, bu raher o hose unceranes ha are nheren n all measuremens and can never be compleely elmnaed. A large par of a measurer s effor should be devoed o undersandng hese unceranes (error analyss) so ha approprae conclusons can de drawn from varable observaons. Measuremens can be que meanngless whou any knowledge of her assocaed errors. In scence and engneerng, numbers whou accompanyng error esmaes are suspec and possbly useless. Ths holds rue also n sofware engneerng for every measuremen, one should record he uncerany n he measured quany. Beyond he well-known and easly undersood defnon of sysemac errors and random errors (whch s referred o any sngle quany beng measured, and s no furher examned hereby), we pon our aenon o he consequences of usng measures wh unavodable errors whn he applcaon of some formula (algorhm) o derve furher quanes, as s he case of oal sze from componens sze for a sofware sysem, or of effor, me or cos esmaon from he oal sze of he sysem beng examned. In scence, hs analyss s usually denoed as error propagaon (or propagaon of uncerany). Ths paper hghlghs he general phenomenon, for whch always holds rue ha generc unceranes from wo or more dervng quanes always sum up o provde a larger uncerany on he derved quany. Ths also resuls n a formal approach o rsk analyss, n erms of wha-f scenaros where npu quanes change wh respec o her nal, planned, or desred values. II. ERROR PROPAGATION THEORY A SUMMARY When he quany o be deermned s derved from several measured quanes, and f he measured quanes are ndependen of each oher (ha s, conrbuons are no correlaed), he uncerany δ of each conrbuon may be consdered separaely []. For example, suppose ha we have measured he quanes ISBN-4:

2 me ± δ and hegh h ± δh for a fallng mass (he δ s refer o he relavely small unceranes n and h). We have deermned ha h 5. ± 0.0 cm 0.0 ± 0.00 s From physcs, we can fnd he acceleraon g from he relaon: g g(h,) h/ whch yelds g 0.09 m/s. (he well-known value for g from physcs s acually g m/s ). To fnd he uncerany n he derved value of g caused by he unceranes n h and, we consder separaely he conrbuon due o he uncerany n h and he conrbuon due o he uncerany n, combned n quadraure: δ δ + δ g g gh where for nsance we denoe he conrbuon due o he uncerany n by he symbol δ g (read as dela-g- or uncerany n g due o ). The bass of he quadraure addon s an assumpon ha he measured quanes have a normal, or Gaussan, dsrbuon abou her mean values. A ypcal mehod by whch one may calculae he conrbuons of drec measured ndependen varables n he dependen varable beng derved s he dervave mehod. The basc dea s o deermne by how much he derved quany would change f any of he ndependen varables were changed by s uncerany. Le he funcon f f(x,, x N ) be he formula used o derve he dependen varable f from he N ndependen varables x,, x N, and le δx be he esmaed error on he value measured for each varable x. Tha s, he rue value of x belongs o he nerval (x ± δx ) for each from o N. Then he rue value of f belongs o he nerval [f(x,, x N ) ± δf], where δf s gven from he dervaves formula: N f f f δ f δ x δ x δ xn x + + x K x N The dervave ( f/ x ) s a paral dervave (smply pu, when akng a paral dervave wh respec o one varable, rea any oher varable as consan). Snce each unceran varable wll ncrease, no decrease he fnal uncerany, we pu he uncerany n f due o he uncerany n x as he absolue value. In he ced example for acceleraon g g(h,) we have: δ g δ 4h δ g δ g h δ δ gh h h so ha: 4h g g cm/s 0. m/s g h + h δ δ δ δ δ For smple formulas, as addcon, subracon, mulplcaon and dvson of wo varables, one can easly fnd ha he uncerany s calculaed as n he followng. If f f(x,y) x + y, hen δ δ + δ. If f f(x,y) x - y, hen also δ δ + δ (unceranes always add). If f f(x,y) x y, hen δ y δ + x δ. + y y If f f(x,y) x / y, hen δ x δ δ. f y x Noce how he varables values are mxed ogeher n he uncerany calculaon for mulplcaon and dvson. I s also worh ponng ou ha fraconal or percenage unceranes n mulplcaon and dvson behave much lke absolue unceranes n addon. In oher words, f f(x,y) x y: y δ x + x δ y δ δ x y + δ f f xy x y wh he same resul holdng f f f(x,y) x / y. If eher x or y s a consan or has a relavely small fraconal uncerany, hen can be gnored and he oal uncerany s jus due o he remanng erm. Furhermore, f one of he measured quanes s rased o a power, he fraconal or percenage uncerany due o ha quany s merely mulpled by ha power before addng he resul n quadraure. For he orgnal example: ( hδ ) ( δh ) + δ δ y δ 4 g g h + y For he values n our example, δ h /h 0.5% and δ / % (so δ / %), so we can see ha he conrbuon from he uncerany n h s neglgble compared o he conrbuon from. We can herefore conclude (as confrmed by prevous calculaon) ha he fraconal uncerany n our measured resul for g s abou %: g 0.0 ± 0. m/s As a reference, able shows he uncerany of smple funcons, resulng from ndependen varables A, B, C, wh unceranes A, B, C, and a precsely-known consan c.

3 Funcon X A ± B X ca X c(a B) or X c(a/b) X can X ln (ca) X exp(a) Table. Example formulas. Funcon uncerany ( X)² ( A)² + ( B)² X c A ( X/X)² ( A/A)² + ( B/B)² X/X n ( A/A) X A/A X/X A III. APPLICATIONS TO SOFTWARE MEASUREMENT A SIMPLE EXAMPLE Generally, when measurng he funconal sze he measurer has o: denfy (classfy & coun) he base funconal componens; assgn a sze value o each componen; combne he nformaon above. For nsance, n he IFPUG mehod [] one has o denfy logcal fles and elemenary processes. Logcal fles are classfed as nernal (ILF) or exernal (EIF) wh respec o he boundary of he sofware beng measured, wh dfferen sze assgnmens on a low average hgh complexy scale: /0/5 and 5//0 unadjused funcon pons, respecvely. Elemenary processes are classfed as npu (EI), oupu (EO), and enqury (EQ) wh respec o he processng logc beng performed, wh dfferen sze assgnmens on a low/average/hgh complexy scale: /4/6 for npus and enqures, 4/5/ for oupus. The (unadjused) funconal sze formula for IFPUG could be expressed as: SIZE (UFP) N ILF-LOW + 0 N ILF- AVERAGE + 5 N ILF-HIGH + 5 N EIF-LOW + N EIF- AVERAGE + 0 N EIF-HIGH + N EI-LOW + 4 N EI- AVERAGE + 6 N EI-HIGH + 4 N EO-LOW + 5 N EO- AVERAGE + N EO-HIGH + N EQ-LOW + 4 N EQ- AVERAGE + 6 N EQ-HIGH Wh respec o such formula, f he measurer fals n denfyng a funcon (fle or process) and/or (s)he couns a funcon ha should NOT be couned accordng o he measuremen rules, (s)he s nroducng an error n a sngle value for N A-B where A denoes he ype of he funcon, and B he complexy rang. If he measurer s correc n denfyng a funcon, bu (s)he assgn he wrong complexy value o, (s)he nroducng a double error, snce he funcon s NOT couned under he correc fgure AND s wrongly couned under anoher fgure. For sake of readably, we rewre he formula n abbrevaed form as: S S(N, N, N,,N 5 ) N + 0N + 5N +N + 0N + 5N + N + 0N + 5N +N 4 + 0N 4 + 5N 4 +N 5 + 0N 5 + 5N 5 Ths formula s easly recognzed as a combnaon of addons and mulplcaons, where he mulplcaon coeffcens are fxed by defnon of he measuremen mehods, and he measures has o deermne ffeen quanes N AB. If we recall ha for addon: f f(x,y) x + y δ δ + δ, and for mulplcaon (by a consan): f f(x) k x δ k δ kδ, we easly oban f x x he uncerany formula for unadjused sze n he IFPUG measuremen mehod: δ S ( δ N ) + ( 0 δ N ) + ( 5 δ N ) ( δ N ) + ( 0 δ N ) + ( 5δ N ) Alhough long, hs formula s que rval, snce derves from a lnear combnaon of smple erms. Sll, proves ha an error n denfyng logcal fles s much more mpacng han an error n denfyng elemenary processes, as any experenced measurer can sae by experence, snce he convenonal mulplers for logcal fles have hgher values han for elemenary processes. For lnear combnaon of smple erms, he reader wh no experence n dervaves can sll easly derve uncerany propagaon for oher measuremen mehods. In he IFPUG case, a more complex case comes from applyng he Value Adjusmen Facor as currenly proposed by he mehod (non-iso case). The Value Adjusmen Facor (VAF) s derved from foureen General Sysem Characerscs (GSC s) by he followng formula: 4 VAF DIGSC where DI denoes he degree of nfluence of each GSC and can only have neger values from 0 o 5. The fnal funcon pon coun accordng o he IFPUG mehod s gven by he general formula: SIZE (FP) SIZE (UFP) VAF SIZE (UFP) 4 DIGSC Wha s he mpac on such formula f he measurer nroduce an error on one or more of he foureen general sysem characerscs? Recallng he fraconal or

4 percenage formula for mulplcaon, he percenage uncerany n hs case s: δ S δ ( FP) S ( UFP) δ VAF + S ( FP) S ( UFP) VAF where he uncerany on VAF s gven by: δvaf δ DI δ DI + δ DI + + δ DI ( GSC )... GSC GSC GSC4 As an example, assume ha he unadjused sze measures 00 unadjused funcon pons, wh a relave error of ±5%, and ha he degree of nfluence of any GSC s assumed o be equal o, wh an uncerany of ± per GSC. Thus: S (UFP) 00 UFP, S (UFP) 5 UFP, δs/s (UFP) 5% VAF.0, δvaf , 00 δvaf/vaf.5% and fnally: ( FP) ( UFP ) δ S δ S δvaf % S ( FP) S ( UFP ) VAF Dependng on he uncerany on boh erms (sze and value adjusmen facors), he uncerany on he fnal coun S s greaer han each of hose by a ceran amoun. Whle n a sascal (generc) sense, we would sae ha errors end o compensae, uncerany s always growng. Acually, even n he dscussed smple examples, we mus noce ha some varable could resul n beng no ndependen (several general sysem characerscs n VAF can nfluence one each oher), and ha some unceranes also could affec one each oher, as n he case where an error on some funcon fgure resuls n an error on anoher funcon fgure, n he generc formula for sze n he IFPUG model above. In such cases, he error propagaon s complcaed by muual effecs, and he concep of covarance beween varable pars should be aken no accoun, wh a more complex form of he uncerany formula (no dscussed here). IV. APPLICATION TO SOFTWARE ESTIMATION MORE COMPLEX EXAMPLES A. Consrucve Cos Model (COCOMO) The consrucve cos model [] for work effor esmaon for sofware projecs n s general form proposes some smple ranges for opmsc and pessmsc esmaes as half he orgnal esmae or wce he orgnal esmae. Such esmaon ranges are no formally proved. A more relable uncerany range can be deermned wh he descrbed approach of error propagaon. To derve he developmen effor for a sofware program, he zero-order COCOMO formula s: B y A x where y represens he expeced work effor expressed n person hours, and x he sze n lnes of code or funcons pons, wh facors A and B accordngly deermned by sascal regresson on an hsorcal sample [4]. Noe ha havng fxed sascal values for some parameers n he model does no mean necessarly ha hese values are exac: her assocaed errors can be derved from he sandard devaon of he sascal sample from he fng funcon y. To evaluae he error on y, gven he errors on he parameers A and B and on he ndependen varable x, we have o perform some paral dervaves, recallng ha: B B ( ) B B B B A x A B x, ( A x ) x, ( A x ) A x ln x x A B For nsance, we perform a approxmaed measuremen of a projec n funcon pons, and oban for x he esmaed value of,000 ± 00 FP, or a percenage uncerany of 0%. Assume also, as an example, ha A and B are equal o 0 ± and o.0 ± 0.0 respecvely, n he approprae measuremen uns. Collecng all daa and applyng he error propagaon for y, we oban: B.0 y A x 0,000 9,95.6 (person hours) B B B ( ) ( ) ( ln ) δ y A B x δx + x δa + A x x δb [ ] [ ] [ ] (person hours) , , , 05.9 Takng only he frs sgnfcan dg of he error, we oban for y he esmaed range 0,000 ± 5000, or a percenage uncerany of 5%. Noe ha he percen error on y s no he smple sum of he percen errors on A, B, and x, because he funcon assumed n hs example s no lnear. We consder now a furher sep n he COCOMO model, where he work effor derved by sascal parameers (nomnal effor, y nom ) s o be adjused by a seres of (ndependen) cos drvers, c []. Alhough for smplcy he cos drvers assume dscree values n praccal applcaon, we can rea hem as connuous varables. Here s a dervaon of he uncerany for he adjused effor esmae: y y c adj nom c ynom c c, ynom c ynom y nom c j c j For nsance, f we consder y nom 0,000 ± 5000, and a se of only facors c, for each of whch (for sake of smplcy) we assume he same value c 0.95 ± 0.05), hen fro y adj we oban: 4

5 adj nom 0, , ,966. y y c c c adj nom nom nom c K c δ y c δ y + y δ c + + y δ c c c c ( ynom ) + ( ynom ) + K + c c 0.95 ( 5,000) + ( 0, 000) + K+ 5, We herefore see ha each addonal facor n he esmaon process can apparenly make he esmaon more accurae (n he gven example, for nsance, he y adj s reduced wh respec o s nomnal value because all he facors are smaller han ), bu s percen error s ncreased ( s now abou 6%, versus he 5% of he nomnal esmae). We can hen draw a general concluson: he more we add nformaon o refne an esmae, he more we re also addng uncerany sources o he esmaon process. Thus, he measurer should locae he rgh cu-off beween accurae formulas and reasonable percenage errors applyng o hose formulas. For nsance, n some cases one could decde beween accepng he measured value of each cos drver n he model, refnng (f possble, reducng s error), or even avodng compleely some facor from he overall model because of any unaccepable mpac on he overall uncerany of he esmae. B. Jensen/Punam Model (Sofware Equaon) A general form of he sysemc model by Jensen & Punam s [5]: Sze ( Eff / β ) / Sched 4/ Prod where: Sze s he sze n SLOC ( or oher measure of amoun of funcon ). Eff (effor) s he amoun of developmen effor n person-years. β (bea) s a specal sklls facor ha vares as a funcon of sze from 0.6 o 0.9 (hs facor has he effec of reducng process producvy, for esmang purposes, as he need for negraon, esng, qualy assurance, documenaon, and managemen sklls grows wh ncreased complexy resulng from he ncrease n sze [5]). Sched (schedule) s he developmen me n years. Prod (process producvy parameer) s he producvy number used o une he model o he capably of he organzaon and he dffculy of he applcaon (by calbraon he range of values seen n pracce [for Prod] across all applcaon ypes s,94 o,9 [5]). The so-called sofware equaon by Jensen and Punam can be pu n several equvalen forms, dependng on whch varables are consdered as ndependen, and whch varable(s) s o be derved, for esmaon purposes. As a common scenaro, assume ha: Sze s measured n advance accordng o an approprae measuremen mehod and un, Sched s consraned by marke needs (alhough hs may presen srong rsks n real world applcaons), β and Prod are deermned by regresson and calbraon from known projecs. Therefore, he effor Eff s he dependen varable n hs scenaro, and s formula s: 4 β Sze Eff β Sze Prod Sched 4 Prod Sched The paral dervaves of Eff wh respec o he gven varables are: Eff Sze 4 β Prod Sched Eff β Sze 4 Sze Prod Sched Eff β Sze 4 4 Prod Prod Sched Eff 4β Sze 5 Sched Prod Sched To nsanae he example, we can consder some fgures from [5] for a real me avonc sysem (unceranes are pu by he auhor for he example o follow): Sze 40,000 SLOC n C++, wh an uncerany of ±,000, or 5%; β 0.4, wh an uncerany of ± 0.0, or approx. 5%; Sched years, wh an uncerany of ± 0., or approx. 5%; Prod,94, wh an uncerany of ± 60, or approx. 5%; As correcly repored n [5], he expeced value for effor Eff s hen equal o 4.4 person-years (approx. 500 person-monhs). For he uncerany on he compued Eff, we fnd: Eff Eff Eff Eff δ Eff δβ + δ Sze + δ Prod + δ Sched.4 py β Sze Prod Sched ha s, δeff/eff 0%. We herefore see ha mxng four parameers n he sofware equaon, each wh an uncerany of approxmaely 5%, leads o an overall uncerany of 0% on he effor fnal esmaon. From a mahemacal pon of vew, hs s due o he hghly nonlnear form of he model. If one wans o reduce he uncerany of he effor esmaon (provded ha all fgures are relable), (s)he can only ac on he unceranes of each parameer. To evaluae whch parameer s more nfluencng he error 5

6 propagaon, we make he followng assumpon we do reduce he uncerany on only one parameer per ry, leavng unchanged he unceranes on he remanng parameers. For nsance, f we pu an error of only % on Sze, we oban δeff/eff 6%. If we pu an error of only % on β, we oban δeff/eff 9%. If we pu an error of only % on Prod, we oban δeff/eff 6%. If we pu an error of only % on Sched, we oban δeff/eff %. So, gven ha s realsc o refne he esmaon on he npu varables of he sofware equaons, we should conclude ha Sched s he mos mpacng facor n he Jensen & Punam model, followed by Sze and Prod n equal measure. Such evaluaon s confrmed by he power value carred by such parameers n he sofware equaon. REFERENCES [] Boehm, B. e al., COCOMO II Model Defnon Manual, Verson.4, Unversy of Souhern Calforna, 99. URL: hp://goo.gl/jkvzr [] IFPUG, Funcon Pon Counng Pracces Manual, Verson 4.., Inernaonal Funcon Pon Users Group, 00. URL: [] ISBSG, Praccal Projec Esmaon. A oolk for esmang sofware developmen effor & duraon, Inernaonal Sofware Benchmarkng Sandards Group, 00, URL: hp://goo.gl/qynxh [4] Jensen, RW, Punam, LH, Roezhem, W, Sofware Esmang Models: Three Vewpons, Crossalk, Feb URL: hp://goo.gl/84zya4 [5] Sanllo L., Error Propagaon n Sofware Measuremen & Esmaon, Proceedngs of IWSM 006 (Inernaonal Workshop on Sofware Measuremen), November, -4, 006, Posdam (Germany) [6] Taylor, J.R., An Inroducon o Error Analyss, The Sudy of Unceranes n Physcal Measuremens, Unversy Scence Books, 98. V. CONCLUSIONS In real cases, he scope of a sofware projec s ofen no fully defned n early phases; he man cos drver, he sze of he projec, could hus be no accuraely known n he begnnng, when sofware esmaes are more useful. Dfferen values esmaed for he sze provde dfferen esmaes for he effor, of course, bu hs fac does no provde by self any consderaon of he precson of a sngle esmae. Any esmaon model canno be serously performed whou consderaon of s possble devaons beween esmaes (expeced values) and rue values. Also, when dervng mercs from drecly measured facors, we mus consder he mpac ha a (small) error n any drec measure have on he derved mercs, dependng on he algorhm or formula used o derve hem. Consderng he formal error propagaon heory can add a new perspecve n he evaluaon of sofware mercs, n he choce of a sofware measuremen mehod when compared o ohers, as well as n he choce of a sofware esmaon approach or model among he possble opons. The qualy and advanages clamed by any mehod or model can be enforced or dmnshed when an objecve analyss of such mehod or model s performed n erms of error propagaon and overall accuracy ha can offer. Error propagaon heory does provde some useful nsghs no such opc, from boh a heorecal (mehod s/model s form) and a praccal pon of vew (applcaon n real cases). 6