Why Correlation Matters in Cost Estimating

Size: px

Start display at page:

Download "Why Correlation Matters in Cost Estimating"

Shannon Lucas
5 years ago
Views:

1 Why Correlation Matters in Cost Estimating Stephen A. Book The Aerospace Corporation P.O. Box Los Angeles, CA (310) nd Annual DoD Cost Analysis Symposium Williamburg, VA 2-5 February 1999

2 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Also Impacts the Estimate! Summary 32nd DODCAS Advanced Training Session # 2

3 Space-System Cost Elements 0.0 TOTAL BRILLIANT EYES PROGRAM 1.0 SPACE BRILLIANT EYES SYSTEM System-Level Costs Space Vehicle (SV) Segment SV Program Level Space Vehicle Prime Mission Equipment Space Software Space Vehicle Space Vehicle IA&T Sensor Payload Insertion Vehicle Survivability Prototype Lot Spare Parts Technology and Producibility Aerospace Ground Equipment Launch Support Engineering Change Orders (ECOs) Other Government Costs Risk 2.0 GROUND BRILLIANT EYES SYSTEM Ground System IA&T Ground Segment Ground Segment Program Level Ground Software Ground Control Ground Military Construction Ground Operations and Support Ground ECO Ground Other Government Costs Ground Risk 3.0 LAUNCH BRILLIANT EYES SYSTEM Launch System IA&T Launch Segment Launch Segment Program Level Launch Facility LV Production Facility Launch Vehicle Launch Vehicle Integration Launch Operations Launch ECO Launch Other Government Costs Launch Risk 32nd DODCAS Advanced Training Session # 3

4 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Also Impacts the Estimate! Summary 32nd DODCAS Advanced Training Session # 4

5 Typical Cost-Estimating Roll-Up Procedure List Cost Elements in Work-Breakdown Structure (WBS) Calculate Single Best Estimate of Cost for Each WBS Element Sum All Single Best Estimates Define Result to be Single Best Estimate of Total-System Cost 32nd DODCAS Advanced Training Session # 5

6 What Does Best Estimate Signify? Most Likely Cost? ( Mode ) 50th-Percentile Cost? ( Median ) Average Cost? ( Mean ) These Three are Almost Always Different 32nd DODCAS Advanced Training Session # 6

7 Cost-Element Probability Distributions Mean, Median, Mode are Statistical Characteristics of Probability Distributions Use of These Terms Implicitly Assumes that Costs Have Probability Distributions Indeed, Even Admission that Best Estimate is not the Only Possible Estimate Implicitly Assumes That Costs Have Probability Distributions 32nd DODCAS Advanced Training Session # 7

8 Definition of Cost Risk Cost-Risk Analysis Inadequacy of Forecasted Funding Requirements to Assure that Program Can Be Completed and Meet Its Stated Objectives Element Costs Are Uncertain Quantities (i.e., Random Variables) That Have Probability Distributions Combine Element Cost Distributions to Generate Cumulative Distribution of Total System Cost Read off 70th Percentile Cost, 90th Percentile Cost, etc., From Cumulative Distribution to Estimate Amount of Extra Dollars Needed to Cover Risk Quantify Confidence in Ability to Complete Program Funded at the Best (or any other) Estimate of System Cost 32nd DODCAS Advanced Training Session # 8

9 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Also Impacts the Estimate! Summary 32nd DODCAS Advanced Training Session # 9

10 Triangular Distribution of Element Cost DENSITY L M H $ Optimistic Cost Best-Estimate Cost (Mode) Cost Implication of Technical Assessment 32nd DODCAS Advanced Training Session # 10

11 Triangular Distribution of WBS-Element Cost 50% Mode 3610 Mean 7830 Dollars 3368 Median 32nd DODCAS Advanced Training Session # 11

12 Cost-Risk Procedure COST-ELEMENT TRIANGULAR DISTRIBUTIONS MERGE ELEMENT COST DISTRIBUTIONS INTO TOTAL-COST DISTRIBUTION 70% PROBABILITY L B H $ $ L = B H $ BEST ESTIMATE COST (MOST LIKELY) RISK DOLLARS 70th PERCENTILE COST L B H $ Note: Addition of risk dollars brings confidence that total appropriation (best estimate plus risk dollars) is sufficient to fund program. 32nd DODCAS Advanced Training Session # 12

13 When WBS Elements Are Few WBS-ELEMENT TRIANGULAR COST DISTRIBUTIONS MERGE WBS-ELEMENT COST DISTRIBUTIONS INTO TOTAL-COST LOGNORMAL DISTRIBUTION Most Likely L B H $ $ Most Likely L = B H $ ROLL-UP OF MOST LIKELY WBS-ELEMENT COSTS MOST LIKELY TOTAL COST L B H Most Likely $ 32nd DODCAS Advanced Training Session # 13

14 When WBS Elements Are Many WBS-ELEMENT TRIANGULAR COST DISTRIBUTIONS MERGE WBS-ELEMENT COST DISTRIBUTIONS INTO TOTAL-COST NORMAL DISTRIBUTION Most Likely $ Most Likely... $ ROLL-UP OF MOST LIKELY WBS-ELEMENT COSTS MOST LIKELY TOTAL COST $ Most Likely $ 32nd DODCAS Advanced Training Session # 14

15 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Also Impacts the Estimate! Summary 32nd DODCAS Advanced Training Session # 15

16 Pearson Product-Moment Correlation Suppose X and Y are Two Random Variables = E( X), Y = E( Y ) ( ) ( ) ( ) E( XY ) = E( X) E( Y ) ( ) ( ) ( ) ( ) µ µ are their Expected Values ( Means ) X True Theorem: False Theorem: E X + Y = E X + E Y = µ X + µ Y Cov X, Y = E XY E X E Y = Covariance of X and Y [ ] ( ) ( ) ( ) ( ) 2 2 Var X = Cov X, X = E X E X = Variance of X [ ] ( ) (, ) ( ) ( ) 2 2 Var Y = Cov Y Y = E Y E Y = Variance of Y ( ) Cov X, Y Corr( X, Y) = = Correlation of X and Y = Var( X) Var( Y) Var X ( ) σ X Var( Y ) (, ) = ρ σ σ 2 2 σ Y =, =, Cov X Y XY X Y ρ XY 32nd DODCAS Advanced Training Session # 16

17 ( ) [ ] Variance of a Sum ( ) ( ) [ ] 2 2 Var X + Y = E X + Y E X + Y [ ] ( 2 ) ( ) ( ) = E X + XY + Y E X + E Y = E X + 2E XY + E Y [ E X ] 2E( X ) E( Y ) [ E( Y )] ( ) ( ) ( ) ( ) ( ) [ ( )] ( ) ( ) ( ) ( ) 2 (, ) = Var X + Var Y + Cov X Y ( ) ( ) 2 (, ) ( ) ( ) = Var X + Var Y + Corr X Y Var X Var Y 2 2 X Y XY X Y = σ + σ + 2ρ σ σ [ ] 2[ ( ) ( ) ( )] = E X E X + E Y E Y + E XY E X E Y 32nd DODCAS Advanced Training Session # 17

18 Variance Measures Uncertainty σ X 2 Small Area = 1.00 µ X σ X 2 Large Area = 1.00 µ X 32nd DODCAS Advanced Training Session # 18

19 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Can Also Impact the Estimate! Summary 32nd DODCAS Advanced Training Session # 19

20 Correlation Affects the Variance X 1, X 2, K, X n are Costs of WBS Elements (Random Variables) Total Cost = n X = X + X + + X k=1 k 1 2 K n Mean of Total Cost = n n n E X k = E( Xk ) = µ k= 1 k= 1 k= 1 k Variance of Total Cost = = Var n X k k= 1 n n j 1 2 σ k + 2 k= 1 j= 2 i= 1 ρ σ σ ij i j 32nd DODCAS Advanced Training Session # 20

21 Does Correlation Matter? If WBS-Element Costs are Uncorrelated (all ρ ij = 0 ), Variance of Total Cost = n 2 σ k k=1 If WBS-Element Costs are Correlated, Variance of Total Cost = n n j 1 2 σ k + 2 k= 1 j= 2 i= 1 Positive Correlations Increase Dispersion Negative Correlations Reduce Dispersion ρ σ σ ij i j If ( Worst Case) All Correlationsρ ij = 1, Variance of Total Cost = n σ 2 n >> σ k k= 1 k= 1 2 k Ignoring Correlation Issue is Tantamount to Setting all ρ ij = 0 32nd DODCAS Advanced Training Session # 21

22 Yes, Correlation Matters Suppose for Simplicity There are n Cost Elements Each Each Total Cost Var( C i ) = σ 2 Corr C ( i C j ) C, = ρ < 1 n C i k=1 = C 1, C 2, K, C n n n 1 n ( ) ( i ) ( i ) ( j ) Var C = Var C + 2ρ Var C Var C k= 1 i= 1 ( 1) 2 2 = nσ + n n ρσ ( ( n ) ρ) = nσ j= i+ 1 Correlation 0 ρ 1 Var( C) nσ 2 nσ 2 1 ( n 1) ( ρ) + n 2 σ 2 32nd DODCAS Advanced Training Session # 22

23 Magnitude of Correlation Impact Percent Underestimation of Total-Cost Sigma Var C When Correlation Assumed to be 0 instead of ρ is 100% times... nσ 1 + ( n 1) ρ nσ 1 = 1 nσ 1 + n 1 ρ 1 + n 1 ρ ( ) ( ) Percent Overestimation of Total-Cost Sigma Var C When Correlation Assumed to be 1 instead of ρ is 100% times... nσ nσ 1 + n 1 ρ ( ) ( ) ( ) n = ( n ) + ( n ) nσ ρ 1 1 ρ 1 ( ) ( ) 32nd DODCAS Advanced Training Session # 23

24 Maximum Possible Underestimation of Total-Cost Sigma Percent Underestimated When Correlation Assumed to be 0 Instead of ρ Percent Underestimated n = 1000 n = 100 n = 30 n = Actual Correlation 32nd DODCAS Advanced Training Session # 24

25 Maximum Possible Overestimation of Total-Cost Sigma Percent Overestimated When Correlation Assumed to be 1 Instead of ρ Percent Overestimated n =10 Limit as n Actual Correlation 32nd DODCAS Advanced Training Session # 25

26 Impact of Nonzero Correlation Value ( ) ( ) Percent Underestimation of Total-Cost Sigma Var C When Correlation Assumed to be ρ 1 instead of ρ > ρ 2 1 is 100% times ( n 1 ) ρ 1+ 1 ρ ( n ) ( ) ( ) Percent Overestimation of Total-Cost Sigma Var C When Correlation Assumed to be ρ 2 instead of ρ < ρ 1 2 is 100% times ( n 1 ) ρ ( n ) ρ nd DODCAS Advanced Training Session # 26

27 Percent Over/Underestimated Maximum Possible Over- and Under- Estimation of Total-Cost Sigma Percent Over/Underestimated When Correlation Assumed to be 0.1 Instead of ρ 100% 90% 80% 70% 60% 50% 40% 30% 20% n = 1000 n = 100 n = 30 n = 10 10% 0% Overestimated Underestimated Actual Correlation 32nd DODCAS Advanced Training Session # 27

28 Maximum Possible Over- and Under- Estimation of Total-Cost Sigma Percent Over/Underestimated When Correlation Assumed to be 0.2 Instead of ρ 100% Percent Over/Underestimated 90% 80% 70% 60% 50% 40% 30% 20% n = 100 n = 30 n = 10 n = % 0% Overestimated Underestimated Actual Correlation 32nd DODCAS Advanced Training Session # 28

29 Maximum Possible Over- and Under- Estimation of Total-Cost Sigma Percent Over/Underestimated When Correlation 100% Assumed to be 0.3 Instead of ρ Percent Over/Underestimated 90% 80% 70% 60% 50% 40% 30% 20% 10% n = 100 n = 30 n = 10 n = % Overestimated Underestimated Actual Correlation 32nd DODCAS Advanced Training Session # 29

30 Selection of Correlation Values Ignoring Correlation is Equivalent to Assuming that Risks are Uncorrelated, e.g., All Correlations are Zero Reasonable Choice of Nonzero Values Brings You Closer to Truth 0.2 is at Knee of Curve on Previous Charts, thereby Providing Most of the Benefits at Least Loss of Accuracy Square of Correlation Represents Percentage of Variation in one WBS Element s Cost Attributable to Influence of Another s Correlation % Influenced % ±0.10 1% ± % ± % ± % 32nd DODCAS Advanced Training Session # 30

31 A Technical Problem If WBS-Element Costs Are Correlated, Usual Monte Carlo Procedure (Summing Random Numbers Representing WBS-Element Costs) Does Not Produce Total Cost since Random Numbers are Independent Ideal Way to Simulate Total Cost Would be to Specify Inter-WBS-Element Correlations, Generate Correlated Random Numbers, and Sum Them Probability Theory Allows This to be Done Exactly only for Gaussian Cost Distributions, Not in General Not Obvious How to Generate Correlated Random Numbers that Correctly Model WBS-Element Cost Distributions Combining Process Must Provide Approximately Correct Values of Mean, Sigma, Percentiles of Total-Cost Distribution Drs. M.S. Goldberg and P.M. Lurie of IDA (Institute for Defense Analyses) have been Working on this Problem for Several Years 32nd DODCAS Advanced Training Session # 31

32 There is a Second Type of Correlation Pearson Product-Moment Linear Correlation ( ) ρ X, Y = ±1 if and only if X and Y are linearly related, i.e., the least-squares linear relationship between X and Y allows us to predict Y precisely, given X ( ) ρ 2 X, Y = proportion of variation in Y that can be explained on the basis of a least-squares linear relationship between X and Y ( ) ρ X, Y = 0 if and only if the least-squares linear relationship between X and Y provides no ability to predict Y, given X Spearman Rank Correlation ( ) ρ( X, Y ) ρ s X, Y = 1 if and only if the largest value of X corresponds to the largest value of Y, the second largest,..., etc. ρ s ( ) X, Y = 1 if and only if the largest value of X corresponds to the smallest value of Y, etc. ( ) ρ s ( X, Y ) ρ s X, Y = 0 if and only if the rank of a particular X among all X values provides no ability to predict the rank of the corresponding Y among all Y values 32nd DODCAS Advanced Training Session # 32

33 How Do The Two Types of Correlation Differ? Pearson Correlation Measures Extent of Linearity of a Relationship Between Two Random Variables Plays an Explicit, Well-defined Role in Establishing the Sigma Value (as well as the Range) of the Total-Cost Distribution Spearman Correlation Measures Extent of Monotonicity of a Relationship Between Two Random Variables Does Not Appear Explicitly in the Formula for the Total-Cost Sigma (Its impact on Sigma is Not Known) P.R. Garvey of The MITRE Corporation has Done Research on the Issue of Pearson vs. Spearman Correlation 32nd DODCAS Advanced Training Session # 33

34 Linear vs. Rank Correlation ρ = 1. 0 ρ = 0. 8 ρ s = 1. 0 ρ s = 1. 0 ρ = 0. 8 ρ = 0. 6 ρ s = 1. 0 ρ s = nd DODCAS Advanced Training Session # 34

35 Spearman Rank Correlation Coefficient Data Structure CASE RANK OF X i VALUE X j VALUE DIFFERENCE SQUARED DIFFERENCE #1 r1 c1 d 1 = c 1 - r 1 d 1 2 #2 r2 c2 d 2 = c 2 - r 2 d 2 2 #3 r3 c3 d 3 = c 3 - r 3 d #4 r n c n d n = c n - r n d n 2... SUMS n( n + 1) 2 n( n + 1) 2 d = c r = 0 d 2 ( i X j ) ρ X 6 d, = 1 n n 2 2 ( 1) Statistics Theorem: Spearman Rank Correlation Coefficient Equals Pearson (Linear) Correlation Coefficient Calculated Between the Two Sets of Ranks 32nd DODCAS Advanced Training Session # 35

36 Crystal Commercially Available Software Packages that are Addons to Additional Commercial Software Such As Windows, Excel, or Lotus on PC or Mac Crystal Ball Marketed by Decisioneering, Inc., 2530 S. Parker Road, Suite 220, Aurora, CO 80014, (800) Marketed by Palisade Corporation, 31 Decker Road, Newfield, NY 14867, (800) Inputs Parameters Defining WBS-Element Distributions Rank Correlations Among WBS-Element Cost Distributions Mathematics Monte-Carlo and Stratified Random Sampling (Latin Hypercube) Virtually All Probability Distributions That Have Names Can Be Used Suggests Adjustments to Inconsistent Input Correlation Matrix Outputs Percentiles of Program Cost Cost Probability Density and Cumulative Distribution Graphics 32nd DODCAS Advanced Training Session # 36

37 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Also Impacts the Estimate! Summary 32nd DODCAS Advanced Training Session # 37

38 Example: Triangular Cost Distribution 50% Density L M H $ Cost Probability Density Function Total Area = 1.00 Three Parameters L, M, H Completely Specify Distribution Mean, Median, Sigma, All Percentiles can be Expressed in Terms of L, M, and H 32nd DODCAS Advanced Training Session # 38

39 Statistical Metrics of Triangular Distribution Mode = M (most likely value of cost) Median = T p = Dollar Value at Which ( )( ) T. 50 = L M L H L ( )( ) = H H L H M { p} P Cost T = p ( 1 )( )( ) Tp = H p H L H M if p Tp = L + p( M L)( H L) if p M H M H L L L L ( ) if M L H L ( ) if M L H L Mean = L + M + H 3 σ = L + M + H LM LH MH 18 32nd DODCAS Advanced Training Session # 39

40 Sum of Five Uncorrelated Triangular Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 40

41 Sum of Five Correlated Triangular Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.10 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 41

42 Sum of Five Correlated Triangular Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.30 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 42

43 Sum of Five Correlated Triangular Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.50 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 43

44 Sum of Five Correlated Triangular Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.80 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 44

45 Sum of Five Correlated Triangular Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 1.00 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 45

46 Effects of Increasing Correlation on the Sum of Five Triangular Distributions Cost Mean Mode Median (50th) 70th Percentile 90th Percentile Correlation (ρ) 32nd DODCAS Advanced Training Session # 46

47 Example : Exponential Distribution Five WBS Elements WBS-Element Cost Distributions Each Have Exponential Distribution 50% Mode Median Mean Dollars 32nd DODCAS Advanced Training Session # 47

48 Exponential Cost Distribution 50% f ( x) = λe λ ( x L) L Mean Median Dollars f(x) is the Exponential Probability Density Function Total Area = 1.00 Two Parameters L, λ Completely Specify Distribution Mean, Median, Sigma, All Percentiles can be Expressed in Terms of L and λ 32nd DODCAS Advanced Training Session # 48

49 Statistical Metrics of Exponential Distribution Mode = L Median = 50th Percentile = = Dollar Value at Which L + ln 2 λ E p { p} E p = ( p) ln 1 L λ P Cost E is p Mean = L + 1 λ Q = 1 λ 32nd DODCAS Advanced Training Session # 49

50 Sum of Five Uncorrelated Exponential Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 50

51 Sum of Five Correlated Exponential Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.10 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 51

52 Sum of Five Correlated Exponential Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.30 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 52

53 Sum of Five Correlated Exponential Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.50 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 53

54 Sum of Five Correlated Exponential Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 0.80 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 54

55 Sum of Five Correlated Exponential Distributions Monte-Carlo Simulation Software Input Parameters: L = 100, M = 200, H = 500 All Pairwise Correlations are 1.00 Comparison of Rolled-Up vs. Correct Total-Cost Statistics STATISTIC EACH INPUT DISTRIBUTION ROLLED-UP VALUE CORRECT VALUE OVER-ESTIMATE: ROLL-UP, MINUS CORRECT MEAN MODE MEDIAN STANDARD DEVIATION th PERCENTILE th PERCENTILE th PERCENTILE nd DODCAS Advanced Training Session # 55

56 Effects of Increasing Correlation on the Sum of Five Exponential Distributions Cost Mean Mode Median (50th) 70th Percentile 90th Percentile Correlation (ρ) 32nd DODCAS Advanced Training Session # 56

57 Contents A Cost Analysis is a Risk Analysis Measuring Uncertainty in Cost What is Correlation? Correlation Impacts Uncertainty Correlation Also Impacts the Estimate! Summary 32nd DODCAS Advanced Training Session # 57

58 Summary Every Cost-Analysis Job Requires a Risk Analysis WBS-Element Risks(and therefore Costs) are Typically Correlated Correlations Impact the Probable Cost Range Correlations Also Impact Cost Estimates, namely the Mode, Median, and other Percentiles of the Cost Probability Distribution Your Estimate and Range will be Closer to the Truth if You Use Reasonable Nonzero Correlations Rather Than Zeroes 32nd DODCAS Advanced Training Session # 58

32nd Annual Department of Defense Cost Analysis Symposium

32nd Annual Department of Defense Cost Analysis Symposium 3nd Annual Department of Defense Analysis Symposium Do Not Use RANK CORRLATION in Risk Analysis Paul R. Garvey Chief Scientist The conomic and Decision Analysis Center The MITR Corporation From Garvey,