Commonly Used Distributions and Parameter Estimation

Size: px

Start display at page:

Download "Commonly Used Distributions and Parameter Estimation"

Merry Mills
5 years ago
Views:

1 Commoly Use Distributios a Parameter stimatio Berli Che Departmet of Computer Sciece & Iformatio gieerig Natioal Taiwa Normal Uiversity Referece:. W. Navii. Statistics for gieerig a Scietists. Chapter 4 & Teachig Material

2 How to stimate Populatio (Distributio) Parameters? Populatio Sample Iferece Parameters Statistics - Ubiase stimate? - stimatio Ucertaity? Statistics-Berli Che

3 The Beroulli Distributio We use the Beroulli istributio whe we have a experimet which ca result i oe of two outcomes Oe outcome is labele success, a the other outcome is labele l failure The probability of a success is eote by p. The probability of a failure is the p Such a trial is calle a Beroulli trial with success probability p Statistics-Berli Che 3

4 xamples. The simplest Beroulli trial is the toss of a coi. The two outcomes are heas a tails. If we efie heas to be the success outcome, the p is the probability that the coi comes up heas. For a fair coi, p = ½. Aother Beroulli trial is a selectio of a compoet from a populatio of compoets, some of which are efective. If we efie success to be a efective compoet, the p is the proportio of efective compoets i the populatiop Statistics-Berli Che 4

5 ~ Beroulli(p) For ay Beroulli trial, we efie a raom variable as follows: If the experimet results i a success, the =. Otherwise, = 0. It follows that t is a iscrete raom variable, with probability mass fuctio p(x) efie by p(0) = P( = 0) = p p() = P( = ) = p p(x) = 0 for ay value of x other tha 0 or Statistics-Berli Che 5

6 Mea a Variace of Beroulli If ~ Beroulli(p), the = 0(- p) + (p) = p (0 p)( p) ( p)( p) p( p) Statistics-Berli Che 6

7 The Biomial Distributio If a total of Beroulli trials are coucte, a The trials are iepeet ach trial has the same success probability p is the umber of successes i the trials The has the biomial istributio with parameters a p, eote ~ Bi(,p) Probability Histogram Bi(0, 0.4) Bi(0, 0.) Statistics-Berli Che 7

8 Aother Use of the Biomial Assume that a fiite populatio p cotais items of two types, successes a failures, a that a simple raom sample is raw from the populatio. The if the sample size is o more tha 5% of the populatio, the biomial i istributio may be use to moel the umber of successes Sample items ca be therefore assume to be iepeet of each other ach sample item is a Beroulli trial Statistics-Berli Che 8

9 pmf, Mea a Variace of Biomial If ~ Bi(,,p), the probability mass fuctio of is!! x x p ( p), x 0,,..., px ( ) P ( x) x!( x)! 0, otherwise Mea: = p Variace: p( p) x! x! x! Statistics-Berli Che 9

10 More o the Biomial Assume iepeet Beroulli trials are coucte ach trial has probability bilit of success p Let Y,, Y be efie as follows: Y i = if the i th trial results i success, a Y i = 0 otherwise (ach of the Y i has the Beroulli(p) istributio) Now, let represet the umber of successes amog the trials. So, = Y + + Y This shows that a biomial raom variable ca be expresse as a sum of Beroulli raom variables Samplig a sigle value from a Bi(, p) populatio is equivalet to rawig a sample of size from a Beroulli(p) with populatio, a the summig the sample value Statistics-Berli Che 0

11 stimate of p If ~ Bi(, p), the the sample proportio pˆ / umber of successes Y ˆ Y Y p umber of trials ( is use to estimate the success probability p Note: Bias is the ifferece p p. ˆp p is ubiase p ˆ ˆ 0 The ucertaity i ˆp is pˆ Y Y Y p I practice, whe computig, we substitute for p, sice p is ukow Refer to xample 4.4 (p. 0) p ˆp ) Statistics-Berli Che

12 The Poisso Distributio Oe way to thik of the Poisso istributio is as a approximatio to the biomial istributio whe is large a p is small It is the case whe is large a p is small the mass fuctio epes almost etirely o the mea p, very little o the specific values of a p We ca therefore approximate the biomial mass fuctio with a quatity λ =p; this λ is the parameter i the Poisso istributio Statistics-Berli Che

13 pmf, Mea a Variace of Poisso If ~ Poisso(λ), the probability mass fuctio of is x e, for x = 0,,,... px ( ) P ( x ) x! 0, otherwise Probability Histogram Mea: = λ Variace: Poisso() Poisso(0) Note: must be a iscrete raom variable a λ must be a positive costat Statistics-Berli Che 3

Relatioship betwee Biomial a Poisso The Poisso PMF with parameter is a goo approximatio for a biomial PMF with parameters approximatio for a biomial PMF with parameters a, provie that, is very large

14 Relatioship betwee Biomial a Poisso The Poisso PMF with parameter is a goo approximatio for a biomial PMF with parameters approximatio for a biomial PMF with parameters a, provie that, is very large a is very small p p p λ is very small k k λ p p k! lim k k k k λ λ k k λ p p λ p p k k! lim ) (!!! lim k k k k k λ k k λ k! lim! λ k x k k e k λ e x λ λ k k λ! lim ) lim (! lim Statistics-Berli Che 4 Refer to xamples 4.7, 4.9 a 4.0 (p. 9 & p. 0) k!

15 Poisso Distributio to stimate Rate Let λ eote the mea umber of evets that occur i oe uit of time or space. Let eote the umber of evets that are observe to occur i t uits of time or space If ~ Poisso(λt), we estimate λ with ˆ t Note: ˆ ˆ t t t t is ubiase ( ˆ ) ˆ The ucertaity i is ˆ t t t t I practice, we substitute ˆ for λ, sice λ is ukow t Statistics-Berli Che 5

16 Some Other Discrete Distributios Cosier a fiite populatio cotaiig two types of items, which may be calle successes a failures A simple raom sample is raw from the populatio ach item sample costitutes a Beroulli trial As each item is selecte, the probability of successes i the remaiig populatio ecreases or icreases, epeig o whether the sample item was a success or a failure For this reaso the trials are ot iepeet, so the umber of successes i the sample oes ot follow a biomial istributio The istributio that properly escribes the umber of successes is the hypergeometric istributio Statistics-Berli Che 6

17 pmf of Hypergeometric Assume a fiite populatio p cotais N items, of which R are classifie as successes a N R are classifie as failures Assume that items are sample from this populatio, a let represet the umber of successes i the sample The has a hypergeometric istributio with parameters N, R, a, which ca be eote ~ H(N,R,). The probability mass fuctio of is RN R x x, if max(0, R N) xmi(, R) px ( ) P ( x ) N 0, otherwise Statistics-Berli Che 7

18 Mea a Variace of Hypergeometric If ~ H(N,, R, ), the Mea of : R N Variace of : R R N N N N Statistics-Berli Che 8

19 Geometric Distributio Assume that a sequece of iepeet Beroulli trials is coucte, each with the same probability of success, p Let represet the umber of trials up to a icluig the first success The is a iscrete raom variable, which h is sai to have the geometric istributio with parameter p. We write ~ Geom(p). Statistics-Berli Che 9

20 pmf, Mea a Variace of Geometric If ~ Geom(p), the The pmf of is p p x px ( ) P ( x) 0, 0 otherwise x ( ),,,... The mea of is p The variace of is p p Statistics-Berli Che 0

21 Negative Biomial Distributio The egative biomial istributio is a extesio of the geometric istributio. Let r be a positive iteger. Assume that iepeet Beroulli trials, each with success probability bilit p, are coucte, a let eote the umber of trials up to a icluig the r th success The has the egative biomial istributio with parameters r a p. We write ~ NB(r,p) Note: If ~ NB(r,p), the = Y + + Y r where Y,,Y r are iepeet raom variables, each with Geom(p) )istributio tib ti 0,,, 0,,, 0,,,0,.., x trials i total y y y r Statistics-Berli Che

22 pmf, Mea a Variace of Negative Biomial If ~ NB(r,p), the The pmf of is x r xr p ( p), xr, r,... px ( ) P ( x) r 0, otherwise The mea of is r p The variace of is r ( p ) p Statistics-Berli Che

23 Multiomial Distributio A Beroulli trial is a process that results i oe of two possible outcomes. A geeralizatio of the Beroulli trial is the multiomial trial, which is a process that ca result i ay of k outcomes, where k. We eote the probabilities of the k outcomes by p,,p k (p + +p k =) Now assume that iepeet multiomial trials are coucte each with k possible outcomes a with the same probabilities p,,p k. Number the outcomes,,, k. For each outcome i, let i eote the umber of trials that t result i that t outcome. The,, k are iscrete raom variables. The collectio,, k is sai to have the multiomial istributio with parameters, p,,p k. We write,, k ~ MN(, p,,p k ) Statistics-Berli Che 3

24 pmf of Multiomial If,, k ~ MN(, p,,p k ), the the pmf of,, k is,, k (,p,,p k ), p,, k! k i x! x! x! k p( x) P( x) a x i 0, otherwise x x x p p p k, x 0,,,...,,,, Ca be viewe as a joit probability mass fuctio of,, k Note that if,, k ~ MN(, p,,p k ), the for each i, i ~ Bi(, p i ) Statistics-Berli Che 4

25 The Normal Distributio The ormal istributio (also calle the Gaussia istributio) is by far the most commoly use istributio i statistics. This istributio provies a goo moel for may, although h ot all, cotiuous populatios The ormal istributio is cotiuous rather tha iscrete. The mea of a ormal populatio may have ay value, a the variace may have ay positive value Statistics-Berli Che 5

26 pmf, Mea a Variace of Normal The probability esity fuctio of a ormal populatio p with mea a variace is give by ( x ) / f ( x) e, x If ~ N(, ), the the mea a variace of are give by Statistics-Berli Che 6

27 % Rule The above figure represets a plot of the ormal probability esity fuctio with mea a staar eviatio. Note that the curve is symmetric about, so that is the meia as well as the mea. It is also the case for the ormal populatio About 68% of the populatio is i the iterval About 95% of the populatio is i the iterval About 99.7% of the populatio is i the iterval 3 Statistics-Berli Che 7

28 Staar Uits The proportio p of a ormal populatio p that is withi a give umber of staar eviatios of the mea is the same for ay ormal populatio For this reaso, whe ealig with ormal populatios, we ofte covert from the uits i which the populatio items were origially measure to staar uits Staar uits tell how may staar eviatios a observatio is from the populatio mea Statistics-Berli Che 8

29 Staar Normal Distributio I geeral, we covert to staar uits by subtractig the mea a iviig by the staar eviatio. Thus, if x is a item sample from a ormal populatio with mea a variace, the staar uit equivalet of x is the umber z, where z = (x - )/ The umber z is sometimes calle the z-score of x. The z-score is a item sample from a ormal populatio with mea 0 a staar eviatio of. This ormal istributio is calle the staar ormal istributio Statistics-Berli Che 9

30 xamples. Q: Alumium sheets use to make beverage cas have thickesses that are ormally istribute with mea 0 a staar eviatio.3. A particular sheet is 0.8 thousaths of a ich thick. Fi the z-score: As.: z = (0.8 0)/.3 = 0.6. Q: Use the same iformatio as i. The thickess of a certai sheet has a z-score of -.7. Fi the thickess of the sheet i the origial uits of thousaths of iches: As.: -.7 = (x 0)/.3 x = -.7(.3) + 0 = 7.8 Statistics-Berli Che 30

31 Fiig Areas Uer the Normal Curve The proportio of a ormal populatio that lies withi a give iterval is equal to the area uer the ormal probability esity above that iterval. This woul suggest itegratig the ormal pf; this itegral have o close form solutio So, the areas uer the curve are approximate umerically a are available i Table A. (Z-table). This table provies area uer the curve for the staar ormal esity. We ca covert ay ormal ito a staar ormal so that we ca compute areas uer the curve The table gives the area i the left-ha tail of the curve Other areas ca be calculate l by subtractio ti or by usig the fact that the total area uer the curve is Statistics-Berli Che 3

32 Z-Table (/) Statistics-Berli Che 3

33 Z-Table (/) Statistics-Berli Che 33

xamples. Q: Fi the area uer ormal curve to the left of z = 0.47 As.: From the z table, the area is 0.6808.

34 xamples. Q: Fi the area uer ormal curve to the left of z = 0.47 As.: From the z table, the area is Q: Fi the area uer the curve to the right of z =.38 As.: From the z table, the area to the left of.38 is Therefore the area to the right is 0.96 = Statistics-Berli Che 34

35 More xamples. Q: Fi the area uer the ormal curve betwee z = 0.7 a z =8.8. As.: The area to the left of z =.8 is The area to the left of z = 0.7 is So the area betwee is = Q: What z-score correspos to the 75 th percetile of a ormal curve? As.: To aswer this questio, we use the z table i reverse. We ee to fi the z-score for which 75% of the area of curve is to the left. From the boy of the table, the closest area to 75% is , correspoig to a z-score of 0.67 Statistics-Berli Che 35

36 Liear Combiatios of Iepeet Normal RVs The liear combiatios of iepeet ormal raom variables are still ormal raom variables Let N,,, ~ N are iepeet, the Y Mea ~, c Variace c c c is ormal with Y c Y c We have to istiguish the meaig of from that of f y c f y c f Y y c c c i i Y Statistics-Berli Che 36

37 valuatig a stimator : Bias a Variace (/3) (/3) The mea square error of the estimator ca be further q ecompose ito two parts respectively compose of bias a variace, θ θ r (Mea Square rror, MS-- mea of the square error) θ θ θ θ θ θ θ θ costat costat θ θ θ 0 θ variace bias Statistics-Berli Che 37 Cf. Sectio 4.9 (cf. q a q i pp. 79)

38 valuatig a stimator : Bias a Variace (/3) (a ki of ucertaity) Statistics-Berli Che 38

39 valuatig a stimator : Bias a Variace (3/3) Bias a Variace: A xample x ifferet samples for a ukow populatio x, y F y x x x 3 y F x error of measuremet Statistics-Berli Che 39

40 stimatig the Parameters of Normal If,,,, are a raom sample from a N(, ) istributio, is estimate with the sample mea a is estimate with the sample variace s i i s i i ubiase estimator asymptotically ubiase estimator? As with ay sample mea, the ucertaity i which we replace with s/, if is ukow. The mea is a ubiase estimator of. is / Statistics-Berli Che 40

41 Sample Variace is a Asymptotically Ubiase stimator (/) Ubiase stimator (/) Sample variace is a asymptotically ubiase s p y p y estimator of the populatio variace s i i i i i i i s i i i i i i i i Statistics-Berli Che 4 i i

42 Sample Variace is a Asymptotically Ubiase stimator (/) Ubiase stimator (/) Var i i s Var Var i i i i i The size of the observe sample Statistics-Berli Che 4

43 The Logormal Distributio For ata that cotai outliers (o the right of the axis), the ormal istributio ib ti is geerally ot appropriate. The logormal istributio, which is relate to the ormal istributio, is ofte a goo choice for these ata sets If ~ N(, ), the the raom variable Y = e has the logormal istributio tio with parameters a If Y has the logormal istributio with parameters a, the the raom variable = lyy has the N(, ) istributio Probability Desity Fuctio a logormal istributio with parameters 0, Statistics-Berli Che 43

44 pf, Mea a Variace of Logormal The pf of a logormal raom variable with parameters a is f y y 0, exp l y Otherwise, y The mea (Y) ( ) a variace V(Y) ( ) are give by 0 Y ( ) e VY ( ) e e / Ca be show by avace methos Statistics-Berli Che 44

45 pf, Mea a Variace of Logormal Recall Derive Distributios exp, ~, x x f N e Y log log y F y x P y e P y Y P y F Y log log log y y y y F y y F y f Y Y log y y f log exp y y Statistics-Berli Che 45

46 Test of Logormality Trasform the ata by takig the atural logarithm (or ay logarithm) of each value Plot the histogram of the trasforme ata to etermie whether these logs come from a ormal populatio Probability Histogram takig the atural logarithm o the values of the ata Statistics-Berli Che 46

47 The xpoetial Distributio The expoetial istributio is a cotiuous istributio that is sometimes use to moel the time that elapses before a evet occurs Such a time is ofte calle a waitig time The probability esity of the expoetial istributio ivolves a parameter, which is a positive costat λ whose value etermies the esity fuctio s locatio a shape We write ~ xp(λ) Statistics-Berli Che 47

48 pf, cf, Mea a Variace of xpoetial The pf of a expoetial r.v. is x e, x 0 f( x). 0, otherwise The cf of a expoetial r.v. is 0, x 0 F( x). x e, x 0 The mea of a expoetial r.v. is /. The variace of a expoetial r.v. is /. Probability Desity Fuctio Statistics-Berli Che 48

49 Lack of Memory Property for xpoetial The expoetial istributio has a property kow as the p p p y lack of memory property: If T ~ xp(λ), a t a s are positive umbers, the P(T > t + s T > s) = P(T > t) s T P s T s t T P s T s t T P s F s t F s T P s t T P s t T T t T P t F e e e T t s s t Statistics-Berli Che 49

50 stimatig the Parameter of xpoetial If,, are a raom sample from xp(λ), the the parameter λ is estimate with ˆ /. This estimator is biase. This bias is approximately equal to λ/ (specifically, ˆ ). The ucertaity i ˆ is estimate with ˆ a ˆ This ucertaity estimate is reasoably goo whe the sample size is more tha 0 Statistics-Berli Che 50

51 The Gamma Distributio (/) Let s cosier the gamma fuctio For r > 0, the gamma fuctio is efie by r t r t e t 0 ( ) The gamma fuctio has the followig properties: If r is ay iteger, the Γ(r) ( ) = (r-)! For ay r, Γ(r+) = r Γ(r) Γ(/) = Statistics-Berli Che 5

52 The Gamma Distributio (/) The pf of the gamma istributio with parameters r > 0 a λ > 0 is r x x e, x 0 f( x) () r. 0, x 0 The mea a variace of Gamma istributio are give by r/ a r/, respectively If,, r are iepeet raom variables, each istribute as xp(λ), the the sum + + r is istribute as a gamma raom variable with parameters r a λ, eote as Γ(r, λ ) Statistics-Berli Che 5

53 The Weibull Distributio (/) The Weibull istributio is a cotiuous raom variable that is use i a variety of situatios A commo applicatio of the Weibull istributio is to moel the lifetimes of compoets The Weibull probability esity fuctio has two parameters, both positive costats, that etermie the locatio a shape. We eote these parameters a If =, the Weibull istributio is the same as the expoetial istributio with parameter λ = Statistics-Berli Che 53

54 The Weibull Distributio (/) The pf of the Weibull istributio is ( x) x e, x 0 f ( x ). 0, x 0 The mea of the Weibull is. The variace of the Weibull is. Statistics-Berli Che 54

Probability (Quatile-Quatile) Plots for Fiig a Distributio Scietists a egieers ofte work with ata that ca be thought of as a raom sample from some populatio I may cases, it is importat to etermie the

55 Probability (Quatile-Quatile) Plots for Fiig a Distributio Scietists a egieers ofte work with ata that ca be thought of as a raom sample from some populatio I may cases, it is importat to etermie the probability istributio ib ti that t approximately escribes the populatio More ofte tha ot, the oly way to etermie a appropriate istributio is to examie the sample to fi a sample istributio that fits Statistics-Berli Che 55

56 Fiig a Distributio (/4) Probability plots are a goo way to etermie a appropriate istributio Here is the iea: Suppose we have a raom sample,, We first arrage the ata i asceig orer The assig icreasig, evely space values betwee 0 a to each i There are several acceptable ways to this; the simplest is to assig the value (i 0.5)/ to i orer statistics The istributio that we are comparig the s to shoul have a mea a variace that match the sample mea a variace We wat to plot ( i, F( i )), if this plot resembles the cf of the istributio that we are itereste i, the we coclue that that is the istributio the ata came from Statistics-Berli Che 56

57 Fiig a Distributio (/4) xample: Give a sample i s arrage i icreasig orer 3.0, 3.35, 4.79, 5.96, 7.89 sample mea 5.00 sample staar eviatio s.00 Q 5 Q 4 Q 3 The curve is the cf of N(5, ). If the sample poits i s came from the istributio, they are likely to lie close to the curve. Q Q Statistics-Berli Che 57

58 Fiig a Distributio (3/4) Whe you use a software package, the it takes the (i 0.5)/ assige to each i a calculates lates the quatile (Qi) correspoig to that umber from the istributio of iterest. The it plots each (i, Qi ), or (mpirical Quatile, Quatile).g., for the previous example (ormal probability plot) Qi Percetile If this plot is a reasoably straight lie the you may coclue that the sample came from the istributio that we use to fi quatiles Statistics-Berli Che 58

59 Fiig a Distributio (4/4) A goo rule of thumb is to require at least 30 poits before relyig o a probability plot.g., a plot of the mothly prouctios of 55 gas wells mothly prouctios atural logs of mothly prouctios The mothly prouctios follow a logormal istributio! Statistics-Berli Che 59

60 The Cetral Limit Theorem (/3) The Cetral Limit Theorem Let,, be a raom sample from a populatio with mea a variace ( is large eough) Let be the sample mea Let S = + + be the sum of the sample observatios. The if is sufficietly large, ~ N, sample mea is approximately ormal! A ~ N (, ) approximately S Statistics-Berli Che 60

61 xample The Cetral Limit Theorem (/3) cotiuous Soli lies: real istributios of sample mea Dashe lies: ormal istributios iscrete somewhat skewe e to the right cotiuous highly skewe to the right Rule of Thumb For most populatios, if the sample size is greater tha 30, the Cetral Limit Theorem approximatio is goo Statistics-Berli Che 6

62 The Cetral Limit Theorem (3/3) xample 4.64: Let eotes the flaws i a i. legth of copper wire, a its correspoig pmf, mea a variace are Oe hure wires are sample from this populatio. What is the probability that the average umber of flow per wire i this sample is less tha 0.5? Followig the cetral limit theorem, we kow that the sample mea ~ N The z - score of 0.5 is 0.66, z. P 0.5 PZ Statistics-Berli Che 6

63 Law of Large Numbers Let,, be a sequece of iepeet raom variables with i a var i. Let. The, for ay, i 0 i P var i var 0, as i 0 var (sice are iepeet) which states that, i i i The esire result follows immeiately from Chebyshev's iequality, P for 0 Statistics-Berli Che 63

) approximate by N(0, 6) Recall that ~ where Y Y, Y Bi,p, the ca be represete as Y... Y,,.

64 Normal Approximatio to the Biomial If ~ Bi(,p) a if p > 0, a (-p) ( >0, the ~ N(p, p(-p)) approximately p( p) A p ˆ ~ N p, approximately Bi(00, 0.) approximate by N(0, 6) Recall that ~ where Y Y, Y Bi,p, the ca be represete as Y... Y,,..., Y is a sample from Beroullip Followig the cetral limit theorem, if is large eough the pˆ a Y Y... Y ca be approximate by N p p be approximate by N p, p, p p Statistics-Berli Che 64

65 Normal Approximatio to the Poisso Normal Approximatio to the Poisso: If ~ Poisso(λ), where λ > 0, the ~ N(λ, λ) The Poisso ca be first approximate by Biomial a the by Normal Note that variace of p biomial : p p if p Statistics-Berli Che 65

Cotiuity Correctio The biomial istributio is iscrete, while the ormal istributio ib ti

istributio with a cotiuous oe, that ca improve the accuracy of the approximatio If you

66 Cotiuity Correctio The biomial istributio is iscrete, while the ormal istributio ib ti is cotiuous The cotiuity correctio is a ajustmet, mae whe approximatig a iscrete istributio with a cotiuous oe, that ca improve the accuracy of the approximatio If you wat to iclue the epoits i your probability calculatio, the exte each epoit by 0.5. The procee with the calculatio e.g., P If you wat exclue the epoits i your probability calculatio, the iclue 0.5 less from each epoit i the calculatio e.g., P45 55 Statistics-Berli Che 66

67 Summary We cosiere various iscrete istributios: Beroulli, Biomial, Poisso, Hypergeometric, Geometric, Negative Biomial, a Multiomial The we looke at some cotiuous istributios: Normal, xpoetial, Gamma, a Weibull We leare about the Cetral Limit Theorem We iscusse Normal approximatios to the Biomial a Poisso istributios tib ti Statistics-Berli Che 67

Chapter 2 Transformations and Expectations

Chapter 2 Transformations and Expectations Chapter Trasformatios a Epectatios Chapter Distributios of Fuctios of a Raom Variable Problem: Let be a raom variable with cf F ( ) If we efie ay fuctio of, say g( ) g( ) is also a raom variable whose