ECE 510 Lecture 7 Goodness of Fit, Maximum Likelihood. Scott Johnson Glenn Shirley

Transcription

1 ECE 510 Lecture 7 Goodness of Fit, Maximum Likelihood Scott Johnson Glenn Shirley

2 Confidence Limits 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 2

3 Binomial Confidence Limits (Solution 6.2) UCL: Prob of 30% units failing or less is < 0.05 LCL: Prob of 30% units failing or more is < 0.05 UCL = 60.7% LCL = 8.7% 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 3

4 Synthesizing Binomial Data 1000 units 0.416% prob of fail 4.16 units expected BINOMDIST(fails, samples, prob, TRUE) CRITBINOM(samples, prob, CDF) RAND() 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 4

5 Why Chi-Square for Exponential CL? Answer: a gamma or a chi-square distribution Confidence intervals taken from that 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 5

6 Solution 6.3 Confidence Limits CL values MC Analytic Upper CL Best estimate Lower CL Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 6

7 Confidence Limits Summary Confidence limits (UCL and LCL) are values between which (2-sided) or above or below which (1-sided) the true population value falls with <confidence level> probability Units are whatever units your data uses Confidence level is the probability that the true value lies between (or above or below) your confidence limit(s). CL can mean either confidence limit or confidence level Use context to decide Confidence limits can be calculated Analytically (best if available) Monte Carlo (will work for any distribution) Likelihood methods (coming soon) Monte Carlo confidence limits work regardless of how you calculate the best estimate 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 7

8 Goodness of Fit Tests 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 8

9 How Good Is Good Enough? 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 9

10 Pearson s Chi-Squared Test n i, expected n i, actual pass 2 i n n i, actual n i, expected i, expected 2 Chi-Sq: 2 i 2 i FAIL DoF = bins parameters 1 = 7 p-value = CHIINV(Chi-Sq, DoF) = Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 10

11 Pearson s Chi-Squared Test n i, expected n i, actual pass 2 i n n i, actual n i, expected i, expected 2 Chi-Sq: 2 i 2 i FAIL DoF = bins parameters 1 = 7 p-value = CHIINV(Chi-Sq, DoF) = Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 11

12 Exercise 7.1 Add a chi-square goodness-of-fit test for each of the 4 fits in our synthesized data sheet 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 12

13 Other Goodness-of-Fit Tests Exponential Weibull Normal 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 13

14 Maximum Likelihood Method and the Exponential Distribution 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 14

15 MLE Maximum Likelihood Estimation (MLE) is a fitting technique that is good for any model Principle We can t ask: What is the most likely model? Because we don t have some well-defined space of possible models We can ask: Given this model, how likely is this data set? (This is a fairly Bayesian approach. We are usually frequentists.) 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 15

16 PDF PDF PDF PDF Probability vs. Likelihood Maximum likelihood Probability e t Likelihood t=1 λ=0.5 t λ=1 t λ=1.5 t λ=2 t 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 16

17 Likelihood for each point Use MLE For exact values (exact times to fail), use the PDF For ranges (failed between two readout times), use CDF delta Multiply all together (or add logs) Choose a model functional form with adjustable parameters Adjust the parameters to maximize the likelihood 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 17

18 MLE for Exponential Data For a complete set of times to fail, likelihood is the PDF: t PDF i i e Take log of PDF: ln PDF ln i t i Add up likelihood for each data point: Device hours t i i L i ln PDF i ln t i N ln i i t i Then choose λ to maximize L Sample Size N 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 18

19 Ex 7.2a MLE for Exponential Maximum likelihood: lambda likelihood Log LR UCL estimate LCL guess 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 19

20 Solution 7.2a Maximum likelihood: lambda likelihood Log LR UCL estimate LCL λ = per hour = 3.2% per hour MTTF = 1/λ = 30.8 hours 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 20

21 Graphs of Likelihood vs. Lambda Maximum likelihood: lambda likelihood Log LR UCL estimate LCL 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 21

22 L N Analytic λ For exponential, can maximize analytically: ln i t i dl d N t i i 0 N t = i i Number of fails Total device hours Even works for type I censored data 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 22

23 Exercise 7.2b Calculate λ for the Ex12 data set using the analytic expression and compare it to what you got from the MLE technique 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 23

24 Solution 7.2b Same as MLE technique Analytic fail count 50 device hours lambda (fails / dev hrs) MLE Maximum likelihood: lambda likelihood Log LR UCL estimate LCL 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 24

25 Uncertainty Range of Lambda Upper Confidence Limit (UCL) Best estimate Lower Confidence Limit (LCL) UCL Best estimate LCL 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 25

26 Confidence Interval (2-Sided) TRUE 90% of random sample λ s with this confidence interval include the true population λ 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 26

27 Confidence Interval (1-Sided) TRUE 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 27

28 Uncertainties on Parameters To calculate: Monte Carlo Likelihood ratio Analytic 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 28

29 Recall Monte Carlo Lambda Uncertainty λ=0.022 MTTF = 45 hr λ=0.031 MTTF = 32 hr λ=0.042 MTTF = 24 hr 95% 5% Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 29

30 Likelihood Ratio Lambda Uncertainty L max ln L 2 max L 2 ln L max ln L L λ max λ ln L max ln L 1 CHIDIST(Log LR, 1) Number of parameters in model (=1 for exponential) 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 30

31 Exercise 7.2c Calculate UCL and LCL for lambda: Calculate Log LR for each (below) Choose lambda for each to set Log LR = 0.1 Do by hand first, then use Solver to fine-tune Likelihood of best estimate Likelihood of UCL Maximum likelihood: lambda likelihood Log LR UCL estimate LCL Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 31

32 Solution 7.2c Maximum likelihood: lambda likelihood Log LR UCL estimate LCL Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 32

33 Analytic Lambda Uncertainty for 90% CL Note N+1 vs. N UCL BE CHIINV 5%, 2N 2 N 1 LCL BE CHIINV 95%, 2N 2N 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 33

34 Venerable Calculation 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 34

35 Exercise 7.2d Calculate lambda UCL and LCL analyticall 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 35

36 Solution 7.2d Maximum likelihood: lambda UCL estimate LCL Analytic: UCL estimate LCL Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 36

37 Exercise 7.3 This is Tobias & Trindade exercise 3.1 How many units do we need to verify a 500,000 hr MTTF with 80% confidence, given that we can run a test for 2500 hours and 2 fails are allowed? Hint: you can do this by trial and error. Calculate the UCL on λ as a function of sample size SS and then adjust SS until the UCL equals the target λ. 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 37

38 Solution Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 38

39 Normal Distribution MLE and Analytic 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 39

40 sigma Li MLE for the Normal ln NORMDIST data i,,,false mu 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 40

41 2D MLE sigma sigma mu mu Uncorrelated Correlated Described by covariance matrix 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 41

42 JMP MLE Correlations 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 42

43 Analytic Normal Parameters Best estimate Uncertainty Mean µ µ = AVERAGE(data) m = NORMSINV(CL) * σ / SQRT(N) Stdev σ σ = STDEV(data) s = σ * (SQRT(CHIINV(1-CL, N-1) / (N-1)) - 1) Percentile µ + z*σ UCL = µ + z*σ + SQRT(m 2 + z 2 *s 2 ) LCL = µ + z*σ SQRT(m 2 + z 2 *s 2 ) CL = confidence level (e.g., 95%) z = probit value at which to evaluate distribution (e.g., -2) N = number of samples in data set 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 43

44 Normal Distribution Uncertainties Best estimate Confidence limits 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 44

45 Exercise 7.4a For the 9 data points given, extract mu, sigma, and their uncertainties, and calculate the 95% confidence interval for the 2-sigma point of their parent distribution. 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 45

46 Solution 7.4a 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 46

47 Exercise 7.4b Theory Best estimate Confidence limits 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 47

48 Calculation Method Flowchart Is analytic available? No Yes Use analytic Is likelihood ratio available? No Yes Use likelihood ratio Use Monte Carlo Yes Parameters correlated? No Done Add correlations to likelihood ratio Worst case Engineering judgment 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 48

49 Weibull MLE with Readout Data 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 49

50 Weibull Readout Data LIK F500 F fails L ln F F 168 F(500) F(168) Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 50

51 MLE for Weibull 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 51 R R R r r r r r r t S s t S d t F t F n L ln ln ln 1 1 t t F exp 1 t t S exp Vary these to maximize this

52 Exercise Use MLE to determine Weibull fit parameters for the readout data given below and on the Ex16 tab. Also, find (separate) 90% confidence intervals for each parameter using the likelihood ratio technique. (That is the confidence where the LR=0.1.) 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 52

53 Solution 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 53

54 Exercise Add a chi-square goodness-of-fit test to your fit from part (a). Recall that the bins should have more than about 5 fails each, so you will need to combine the first few readouts into 1 bin. So we do things the same way, combine the first 3 readouts into 1 bin, even though they only have 2 total fails. 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 54

55 Solution 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 55

56 Confidence and Figures of Merit sigma mu 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 56

57 Exercise Use the pfail at 2000 hours as the FOM for the Weibull model of Ex16. Evaluate this pfail as a function of the shape and lifetime parameters. Try various corners of the space of shape and lifetime values and find the worst case corner. Report these values as your worst case shape and lifetime values. 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 57

58 Solution 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 58

59 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 59

60 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 60

61 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 61

62 The End 30 Jan 2013 ECE 510 S.C.Johnson, C.G.Shirley 62