T E C O L O T E R E S E A R C H, I N C.

T E C O L O T E R E S E A R C H, I N C. B rdg n g En g neern g a nd Econo mcs S nce 1973 THE MINIMUM-UNBIASED-PERCENTAGE ERROR (MUPE) METHOD IN CER DEVELOPMENT Thrd Jont Annual ISPA/SCEA Internatonal Conference Dr. Shu-Png Hu 14 June 001 shu@sb.tecolote.com

Outlne Objectve Applcaton for MUPE Error Term Assumptons Addtve Multplcatve Multplcatve Error Model Defntons and Propertes Canddate Error Forms Methodologes Used n Analyzng Multplcatve CERs Log-Error, WLS (by observaton), MPE, MUPE, & ZPB/MPE Comparson between MPE and ZPB/MPE Bas Characterstcs Heurstc Goodness-of-ft Measures Analyss of Examples from USCM7 and Other Sources Conclusons Page Brdgng Engneerng and Economcs Snce 1973

Objectve Fnd a regresson technque to model a multplcatve error term wthout bas It requres no transformaton and no correcton factor Page 3 Brdgng Engneerng and Economcs Snce 1973

Applcaton for MUPE Fan-shaped pattern exsts n database, or Data pattern not notceable (especally n small samples) Data range of dependent varable over one order of magntude Management controls proportonal errors Page 4 Brdgng Engneerng and Economcs Snce 1973

Addtve Error Term Addtve Error Term : y = f(x) + Y X Note: Error dstrbuton s ndependent of the scale of the project. (OLS) Page 5 Brdgng Engneerng and Economcs Snce 1973

Addtve Error Term Addtve Error Term : y = ax^b + Y X Note: Ths requres non-lnear regresson. Page 6 Brdgng Engneerng and Economcs Snce 1973

Multplcatve Error Term Multplcatve Error Term : y = (a + bx) * UpperBound f(x) LowerBound Y X Note: Ths requres non-lnear regresson. Page 7 Brdgng Engneerng and Economcs Snce 1973

Multplcatve Error Term Multplcatve Error Term : y = ax^b * UpperBound f(x) LowerBound Y X Note: Ths equaton s lnear n log space. Page 8 Brdgng Engneerng and Economcs Snce 1973

Error Term Assumptons - Background ADDITIVE ERROR MULTIPLICATIVE ERROR Dstrn Assumptons N(0, S ), ndependent LN(0, S ), ndependent Typcal Model Form Lnear y = a + b x Exponental y = a x b Hstorcal Ratonale (Mathematcal Convenence) Legtmate Reasons Absolute Errors Proportonal Errors What should be cost errors? Cost varaton s ndependent of the scale of the project Cost varaton s proportonal to the scale of the project Model form should not drve error term assumpton Error term should not drve model form Page 9 Brdgng Engneerng and Economcs Snce 1973

Multplcatve Error Model - MUPE Defnton for cost varaton: Y = f(x)* where E( ) = 1 and V( ) = f(x) Cost Y Note: E( (Y-f(X)) / f(x) ) = 0 V( (Y-f(X)) / f(x) ) = Some Drver X Page 10 Brdgng Engneerng and Economcs Snce 1973

Bas n a Regresson Equaton a based hgh regresson equaton: f(x) based Cost Y Some Drver X Page 11 Brdgng Engneerng and Economcs Snce 1973

Canddate Error (Resdual) Forms Log-Error ( ~ LN(0, ) ) Least squares n log space Resduals of the cost functon transformed nto log-space Error = Log (y ) - Log f(x ) Weghted Resduals Least squares n unt space by the recprocal of the observaton Error y f y by the recprocal of the predcted value ( x ) Error y f ( x f ( x ) ) Page 1 Brdgng Engneerng and Economcs Snce 1973

Regresson Methods to Implement Multplcatve Errors Least squares n log space Log-Error Model Least squares n unt space wth weghted resduals Weghted by observaton - WLS Weghted by predcted value MPE (Mnmum-Percentage Error) Method MUPE (Mnmum-Unbased-Percentage Error) Method ZPB/MPE Method (Constraned MPE Method) Page 13 Brdgng Engneerng and Economcs Snce 1973

Methodologes Analyses of real examples usng four dfferent methods MPE (Mnmum-Percentage Error) Method MUPE (Mnmum-Unbased-Percentage Error) Method Log-Error Model ZPB/MPE Method (Constraned MPE Method) Page 14 Brdgng Engneerng and Economcs Snce 1973

MPE and MUPE Methods Two possble ways to perform the optmzaton for the weghted least squares usng the predcted values n USCM7 MPE hgh bas due to smultaneous mnmzaton Mnmze ( x n 1 f ( x ) y f ) MUPE bas elmnated Mnmze ( x n 1 fk-1 ( x ) y f ) where k s the teraton number Page 15 Brdgng Engneerng and Economcs Snce 1973

Brdgng Engneerng and Economcs Snce 1973 Page 16 Constraned MPE Method (ZPB/MPE) Alternatve method to remove the hgh bas n MPE equatons for the general level of the functon Constraned Excel Solver soluton 1 ) ( ) ( n x f x f y Mnmze 0 ) ( ) ( 1 n x f x f y to Subject

Comparson between ZPB/MPE and MPE For most equatons (.e.,y = a X b Z c, Y = a + bx + cz, etc.) Senstvty coeffcents (assocated wth the drver varables) are the same between MPE & ZPB/MPE equatons Only leadng term or level of functon adjusted Fndngs also proven by mathematcal dervatons For trad equatons (.e., Y = a + b X c Z d ) All coeffcents changed Page 17 Brdgng Engneerng and Economcs Snce 1973

Bas Characterstcs of Dfferent Methods Log-error equatons wll be based low Bas can be adjusted usng a correcton factor,.e., exp (1 p ) n p = number of estmated coeffcents Resduals weghted by the observaton wll be based low Bas cannot be easly justfed MPE equatons wll be based hgh Bas n equaton can be adjusted n smple cases by a correcton factor 1 ( 1 ) mpe 1 mupe MUPE (noted as IRLS) wll be asymptotcally unbased MUPE has zero proportonal error for ponts n the database, t s also proven to produce consstent parameter estmates. Page 18 Brdgng Engneerng and Economcs Snce 1973

Brdgng Engneerng and Economcs Snce 1973 Page 19 Heurstc Goodness-of-ft Measures p n y y y n 1 ˆ ˆ Multplcatve error (standard error of model): Average percentage error (APE)*: Note: *APE s termed as average bas n USCM7. n y y y n 1 ˆ ˆ n OLS y y p n SEE 1 ˆ 1

MPE Method s based Based hgh for the level of the functon Correcton for bas by a correcton factor: CF = 1 MPE Based low for varance / standard error Correcton for bas n estmate of varance by: CF = 1 / (1 MPE) average percentage error (APE) Correcton for bas n APE by: CF = 1 / (1 APE MPE ) Note: True APE = APE MPE / (1 APE MPE ) Page 0 Brdgng Engneerng and Economcs Snce 1973

Heurstc Goodness-of-ft Measures Pearson s Correlaton Squared (r ): r Cov( Y, Yˆ) Var( Y ) * Var( Yˆ) r = R n OLS Pearson s correlaton coeffcent measures the lnear assocaton between y (actual cost) and y-hat (predcted cost), t cannot explan the actual devaton between y and y-hat f the model s not OLS Page 1 Brdgng Engneerng and Economcs Snce 1973

R-Squared: Heurstc Goodness-of-ft Measures R SSE ( y yˆ ) ( yˆ y ) 1 1 ( SST ( y y ) ( y y ) SSR SST n OLS ) Adjusted R-Squared: Adj ˆ SSE ( n p ) ( y y ). R 1 1 SST ( n 1) ( y y ) ( n ( n p ) 1) Note: R-Squared s a measure of the amount of varaton about the mean explaned by the ftted equaton n OLS Page Brdgng Engneerng and Economcs Snce 1973

Analyss of Examples Comparsons of examples of complex equatons drawn from Unmanned Spacecraft Cost Model, 7th Edton (USCM7) and other sources 6 examples from USCM7 COMM/TT&C Dgtal Electroncs TT&C Dgtal Electroncs TT&C RF Dstrbuton EPS Generaton examples from other sources Analyze and compare results usng 4 dfferent methods Page 3 Brdgng Engneerng and Economcs Snce 1973

COMM/TTC Dgtal Electroncs NR CER (n = 11) MPE: Y = 11.4 * Wt.787 * NumLnks.853 ( s =.1, r =.94 ) MUPE: Y = 194.1 * Wt.804 * NumLnks.85 ( s =.1, r =.94 ) Log-Lnear: Y = 186.61 * Wt.811 * NumLnks.815 (s =.31, r =.94 ) ZPB/MPE: Y = 04.0 * Wt.787 * NumLnks.853 ( s =.1, r =.94 ) Page 4 Brdgng Engneerng and Economcs Snce 1973

COMM/TTC Dgtal Electroncs NR CER (n = 11) MPE: Y = 359.88 * Weght ( s =.58, r =.54, R = -.0 ) (+43%) MUPE: Y = 51.08 * Weght ( s =.69, r =.54, R =.5 ) Log-Error: Y = 0.48 * Weght ( s =.69, r =.54, R =.50 ) (-19%) ZPB/MPE: Y = 51.08 * Weght ( s =.69, r =.54, R =.5 ) Page 5 Brdgng Engneerng and Economcs Snce 1973

Comparson Chart for Weght based CER 45000 NR Cost (9$K) 40000 35000 30000 5000 0000 15000 10000 5000 NR COST MPE MUPE Log Error 0 10 0 30 40 50 60 70 80 90 100 110 Weght (lbs) Page 6 Brdgng Engneerng and Economcs Snce 1973

COMM/TTC Dgtal Electroncs NR CER n USCM7 MPE: Y = 345.78 * Weght ( s =.53, r =.65, R =.7 ) (+35%) average bas = 6% n USCM7 MUPE: Y = 56.88 * Weght ( s =.6, r =.65, R =.64 ) Log-Error: Y = 18.95 * Weght ( s =.58, r =.65, R =.63 ) (-15%) Y * CF = 55.34*Weght ZPB/MPE: Y = 56.88 * Weght ( s =.6, r =.65, R =.64 ) Note: The average bas measures lsted n USCM7 are based low. Page 7 Brdgng Engneerng and Economcs Snce 1973

TT&C Dgtal Electroncs Recur CER (n = 16) MPE: Y = 3.41 * Wt.9 * Nbox.659 * NLnks 1.091 ( s =.6, r =.93, R =.90 ) MUPE: Y = 19.08 * Wt.96 * Nbox.68 * NLnks 1.09 ( s =.6, r =.9, R =.91 ) Log-Lnear: Y = 17.60 * Wt.967 * Nbox.69 * NLnks 1.09 ( s =.6, r =.93, R =.91 ) ZPB/MPE: Y =.0 * Wt.9 * Nbox.659 * NLnks 1.091 ( s =.6, r =.9, R =.90 ) Page 8 Brdgng Engneerng and Economcs Snce 1973

TT&C Dgtal Electroncs Scatter Plot T 1 (9 $ M ) 10 9 8 7 6 5 4 3 1 0 0 10 0 30 40 50 60 70 80 TT&C Comp Weght (lbs) Page 9 Brdgng Engneerng and Economcs Snce 1973

TTC Dgtal Electroncs Recur CER (n = 16) MPE: Y = 103.33 * Wt ( s =.54, r =.15, R = -.8 ) (+37%) MUPE: Y = 75.4 * Wt ( s =.63, r =.15, R =.11 ) Log-Error: Y = 63.5 * Wt ( s =.59, r =.15, R =.11 ) (-16%) ZPB/MPE: Y = 75.4 * Wt ( s =.63, r =.15, R =.11 ) Page 30 Brdgng Engneerng and Economcs Snce 1973

TT&C RF Dstrbuton Comp Rec CER (n = 13) MPE: Y = -7.386 + 9.18 * Wt + 70.68 * Actve ( s =.56, r =.47, R =.4 ) (-5%, +5%, +85%) MUPE: Y = -7.043 + 7.899 *Wt + 38.0 * Actve ( s =.67, r =.46, R =.35 ) Log-Error: Y = -6.165 + 5.08 * Wt + 30.93 * Actve ( s =.60, r =.47, R =.9 ) (+1%, -10%, -0%) ZPB/MPE: Y = -5.616 +.19 * Wt + 53.74 * Actve ( s =.64, r =.47, R =.33) (+0%, -0%, +41%) Note: ZPB/MPE s 76% of MPE Page 31 Brdgng Engneerng and Economcs Snce 1973

EPS Generaton Recur CER ( n = 17 ) MPE: Y = 58.6 + 0.0979 *CellNum ( s =.39, r =.80, R =.51 ) MUPE: Y = 488.54 + 0.0864 *CellNum ( s =.41, r =.80, R =.71 ) Log-Error: Y = 437.787 + 0.080 *CellNum ( s =.47, r =.80, R =.73 ) MPE/ZPB: Y = 506.366 + 0.085 *CellNum ( 87% of MPE ) ( s =.41, r =.80, R =.71 ) Page 3 Brdgng Engneerng and Economcs Snce 1973

SEPM Factors MPE: Y =.54 * Lot# -.718 * Rate -1.0 ( s =.167, r =.84 ) MUPE: Y = 1.93 * Lot# -.713 * Rate -0.99 ( s =.169, r =.84 ) Log-Error: Y = 1.65 * Lot# -.71 * Rate -0.983 ( s =.167, r =.84 ) ZPB/MPE: Y = 1.98 * Lot# -.718 * Rate -1.0 ( s =.169, r =.84 ) (97.5% of MPE) Page 33 Brdgng Engneerng and Economcs Snce 1973

Example from ISPA MPE: Y = 377.337-79.11 * X -0.141 (s =.664, r =.4 ) MUPE: Y = - 1.145 + 53.734 * X 0.658 (s =.871, r =.36 ) Log-Error: Y = 3.51 + 30.17 * X 0.990 (s =1.06, r =.3 ) ZPB/MPE: Y = - 67.649 + 16.666 * X 0.47 (s =.89, r =.40 ) Note: The above equatons are dfferent n both magntude and sgn. Page 34 Brdgng Engneerng and Economcs Snce 1973

Conclusons MUPE method does not requre transformaton or correcton factor. Tradtonal goodness-of-ft measures may not be adequate. Tradtonal statstcs hold for lnear equatons. Revew multplcatve error, adjusted R, resdual plot, percentage error table, etc. for MUPE equatons. Pearson s correlaton s not senstve to dfferent fttng methods. Use t wth cauton. ZPB/MPE method does not change the senstvtes of MPE equaton; t only lowers the level of MPE equaton by a certan percentage *. Ths fndng s also proven by mathematcal dervatons. MPE method requres correcton factors for both standard error and average percentage error because they are based low. Log-error equatons wth correcton factors are very close to MUPE equatons n most cases. MPE and MUPE do not always converge, especally n learnng curve analyss. If no convergence, use log-error models nstead * true for most equatons Page 35 Brdgng Engneerng and Economcs Snce 1973