Recap! Good statistics, cont.! Sufficiency! What are good statistics?! 2/20/14

Transcription

1 Recap Cramér-Rao iequality Best ubiased estimators What are good statistics? Parameter: ukow umber that we are tryig to get a idea about usig a sample X 1,,X Statistic: A fuctio of the sample. It is a radom variable Estimator: A particular statistic that is used to get a idea about the parameter. It is a radom variable Estimate: The value of the estimator for a observed sample x 1,,x. It is a umber, kow oce we have observed the sample Good statistics, cot. Sometimes mle ad mome are the same, sometimes they are differet. How do we choose betwee them? Cosider a Beroulli experimet, performed times. Typically we do ot care about the order of successes ad failures, just the umber X of successes. I what sese does X cotai all the iformatio about p? Let =5, X=3. The P(01101 X = 3) = Sufficiecy Let X 1,,X be iid with pdf/pmf f(x;θ). A statistic W=h(X 1,,X ) is sufficiet for θ if L(θ) is idepedet of θ. f W (w;θ) Ituitively, this meas that W cotais all the iformatio about θ i the sample. P(11100 X = 3) =

2 The factorizatio theorem If we have a W we ca check whether it is sufficiet. But how do we fid W? Fisher-Neyma factorizatio criterio: W=h(X 1,,X ) is sufficiet if ad oly if L(θ) = g(w;θ)s(x 1,...,x ) fuctio of parameter ad w=h(x 1,,x ) fuctio of data, ot parameter The biomial case X i X i L(p) = p (1 p) 43 The uiform case L(θ) = Is the mome a fuctio of the sufficiet statistic? Is the mle? 44 The beta case f X (x;θ) = θx θ 1, 0 < x < 1 Beta(θ,1) L(θ) = f X (x i ;θ) = θ x i θ 1 Not a statistic θ 1 = θ x i = g(w;θ)s(x 1,...,x ) ˆθ = l(w) θ = x 1 x w, a statistic θ w θ x i 1 E(X) = xθx θ 1 dx = θ θ

3 The gamma distributio Maximum likelihood Here is a very importat fact: ay mle is a fuctio of the data oly through a sufficiet statistic. This is ot ecessarily true for the method of momets. θ ( ) = l(g(w;θ)) + l(s(x 1,...,x )) ( θ) = θ g(w;θ) g(w;θ) Sums of radom variables X~Bi(,p) Y~Bi(m,p), X, Y idepedet X+Y~ X~Po(λ), Y~Po(μ), X, Y idepedet X+Y~ X~NegBi(r,p) Y~NegBi(s,p) X,Y idep. X+Y~ X~Geom(p) Y~Geom(p) X,Y idep X+Y~ 48 More sums X~Γ(α,β), Y~Γ(δ,β), X,Y idepedet X+Y~ X~χ 2 (), Y~χ 2 (m), X, Y idepedet X+Y ~ X~N(μ,σ 2 ) Y~N(η,τ 2 ) X+Y~ If Var(X) <, X 1,...,X iid as X X i E(X) N(0,1) Var(X) 49 3

4 Problems from 1/31 1. Deote the cdf for the stadard ormal Φ.. The P(X i x) = Φ x µ σ (a) P(X + µ) = Φ(0) = 2 (b) P(X µ) = (1 Φ(0)) = 2 (c) P(X µ X + ) = P(X + µ) P(X µ) = = 1 2 ( 1) 2. The mle ˆp = X / so we cosider ˆpˆq which has expected value E(X) ) E(X2 2 = (p pq p2 ) = ( 1)pq 50 So a ubiased estimator is 3. (a) L(θ) = θ (b) L(2) = 0.40 L(3) = 0.41 L(4) = 0.34, so the mle is 3. 4.(a) P(Y = x Y 1) = = λ x e λ x(1 e λ ) ( ) θ 1 (b) E(T(X)) = 2P(X eve) = 2 = 2 e λ 1 e λ 1 ˆpˆq P(Y = x) P(Y = x) = P(Y 1) 1 P(Y = 0) j=1 λ 2j e λ (2j)(1 e λ ) e λ + e λ 1 2 = 1 e λ 51 (c) t(k) λk e λ k 1 e = 1 e λ λ Review k=1 t(k) k λk = e λ (1 e λ ) 2 k=1 Expad the right had side i Taylor series ad idetify coefficiets to show that the T of (b) is the oly solutio. (d) Estimatig a probability by the umber 2 is silly. Likelihood Maximum likelihood estimator Method of momets Ubiasedess Relative efficiecy Mea squared error Cramér-Rao lower boud Efficiet estimators Sufficiecy Fisher-Neyma factorizatio theorem

5 Large samples Cosistecy desity desity =100 =25 =10 theta =1000 =100 =10 Normal mle (µ=3) Uiform mle (θ=6) A estimator θ* is cosistet if for all ε>0 lim P( θ * θ > ε) = 0 We say that θ* coverges i probability to θ. Oe way to show this is to use Chebyshev s iequality from last quarter. So, for example, ay sample average is a cosistet estimate of its expected value provided the variace of the uderlyig distributio is fiite theta Cosistecy of mles Uder fairly geeral coditios (ivolvig smoothess of the desity as a fuctio of the parameter) all mles are cosistet. Oe ca also show that if θ * P θ the for cotiuous fuctios h h(θ * ) P h(θ) This ca be used to show cosistecy of method of momets estimators Biomial case Returig to our =4 example, here are two of the ubiased estimators we cosidered: ˆp 1 = X ˆp 2 = X 2 Which (if ay) of these are cosistet?

6 Geometric distributio The mle is ˆp = 1 X. By the law of large umbers X coverges i probability to E(X) = Uiform The mle max(x i ) has pdf y 1 / θ. Hece P( ˆθ θ > ε) = P(ˆθ > θ + ε) + P(ˆθ < θ ε) = ε θ Asymptotic ormality Geometric case We will show that as gets larger, the distributio of the mle approaches a certai ormal distributio. Oe says the mle is approximately ormal, with mea θ ad variace the Cramér-Rao boud. Thus mles are asymptotically efficiet i most cases. The assumptios eeded relate to the mle havig fiite mea ad variace. Frequecy Histogram of list =100 =50 = list 61 6

7 Iterval estimates The stadard error of a estimator has two uses: (1) compariso to other estimators (2) assessmet of ucertaity A iterval estimate combies a estimate ad its estimated stadard error ito a radom iterval which covers the true (but ukow) value of θ with a give probability or cofidece coefficiet 1 α For a particular sample, the iterval either does or does ot cover θ. WE DO NOT KNOW WHICH. I the log ru it covers θ i the proportio 1 α of all data sets. 62 Ideal umber of childre I 1986, 1370 US adults were asked What do you thik is the ideal umber of childre for a family to have? #childre frequecy The sample average is 2.60, sample sd 0.97, so ese(x) = = 0.03 A cofidece iterval for the mea ideal family size is (2.52,2.68) = x ± 2.67ese(X) 63 How far off does the sample average have to be for the iterval to miss the populatio mea? More precisely, what is P( X µ > 2.67 ese(x))? The sample average is approximately ormal (why?), ad for large samples ese is about se (why?), we ca compute this probability as 2(1 Φ(2.67)) = Hece, = is the probabilty that the iterval does cover µ. We take somethig kow sample mea ad use a theoretical distributio samplig distributio to estimate somethig ukow populatio mea ad we compute the probability that we are correct cofidece coefficiet 64 The expoetial case Let X ~ exp(λ). The mle is ˆλ = 1 X. Will the iterval (0.01ˆλ,100 ˆλ) cover the true value of λ? How about (0.99 ˆλ,1.01ˆλ)? Cosider the iterval (c 1ˆλ,c2 ˆλ). How ca we determie c 1 ad c 2 to make this a 95% CI? 65 7

8 Moday s lecture Asymptotics = large sample size Cosistecy Asymptotic ormality Cofidece itervals Multiple itervals A researcher costructs CIs for 15 differet chemical reactio costats. Each iterval has 90% cofidece coefficiet, ad they are each costructed from idepedet measuremets. Some may cover the true value, some may ot. What is the probability that all itervals cover their costats? What is the most likely umber covered? About CIs The probability ivolved i computig the cofidece coefficiet has to to with the procedure. A particular iterval either covers the parameter value or ot, ad we do ot kow which. The cofidece coefficiet is NOT the probability that the parameter is i the iterval: the parameter is ot radom, the iterval is. The iterval tells us somethig about the accuracy of our estimate. The shorter the iterval, the more accurate our estimate. 68 Normal case Cosider a sample from N(µ,1). We would estimate μ by x, a observatio of the radom variable X N(µ,1 ). Now ote that X µ ~ N(0,1 ) has a distributio that does ot deped o μ. Such a quatity is called a pivot ad makes it particularly easy to create a CI. 1 α = P(z α /2 (X µ) z 1 α /2 ) = P(X z 1 α /2 µ X z α /2 ) Sice z α /2 = z 1 α /2 we get the observed CI x z 1 α /2,x + z 1 α /2 69 8

9 Beer prefereces 100 Budweiser drikers (polishig off at least two 6-packs per week) were subjected to a blid taste test betwee Schlitz ad Budweiser. 46 of the subjects preferred Schlitz. Y=# subjects preferrig Schlitz. The Y~ From the Cetral Limit Theorem Y Y p pq = p = ˆp p pq se(ˆp) N(0,1) The 1 α = P ˆp p 2 se(ˆp) z 1 α / 2 = P ˆp p 2 z 1 α / 2 pq p 2 (1+ z 2 1 α /2 ) p(2 ˆp+ z 2 1 α /2 )+ ˆp The 95% CI is (0.36,0.56). p Istead of se we ca use ese. The we get ˆp p 1 α P ese(ˆp) z 1 α / 2 = P ˆp p ˆpˆq z 1 α / = P(ˆp z 1 α / 2 ˆpˆq p ˆp + z 1 α / 2 ˆpˆq ) Numerically, this is also (0.36,0.56), so this is a easier way to do thigs (ad there will oly be a differece whe is small). Note that i this secod approach there are two approximatios: approximatig se by ese (LLN) ad approximatig the stadardized distributio of ˆp by a ormal distributio (CLT). Were the beer drikers able to tell the beers apart? Large sample CIs from mle We have see several istaces of Cis based o the mle of the form ˆθ ± z 1 α / 2 ese(ˆθ). This is based o the asymptotic ormality of the mle, ad we ofte ca use the geeral formula ese(ˆθ) = 2 θ (ˆθ)

10 The expoetial case λ (λ) = λ x i ( λ) = λ ( ( ˆλ) ) 1 2 = ˆλ 2 so the 95% approximate cofidece iterval is ( ) = lλ λ x i ˆλ(1± 1.96 ) What sample size do we eed? The US Commissio o Crime wats to estimate the proportio of crimes related to firearms i a area with oe of the highest crime rates i the coutry. They ited to draw a radom sample of files of recetly committed crimes i the area, ad wat to kow the proportio of cases with firearms to withi 5% of the true proportio with probability at least 90%. How may files do they eed to look at? (a) (b) (c) Problem solutios 0 X θ 0 U 1 P(U x) = P(X xθ) = xθ θ = x P(V x) = P(U 1 x,...,u x) = x P((1 V ) x) = P(V 1 x ) = 1 1 x 2. se( ˆλ i ) = λ 100 ad the two estimators are idepedet, so 1 e x P((θ ˆθ ) x) = P(θ(1 V ) x) 1 e x/θ 76 se( ˆλ 1 ˆλ 2 ) = se( ˆλ 1 ) 2 + se( ˆλ 2 ) 2 which ca be estimated by = 0.42 E( ˆλ 1 ˆλ 2 ) = λ λ = ˆλ 1 ˆλ 2 ad ese( ˆλ 1 ˆλ 2 ) = = 1.89 It is ot too surprisig to see this (3 ese differece would be surprisig). 3. (a) Sice the Y i cout whe there is a ew mark, the sum must be the umber marked. (b) Give m marked idividuals, the probability of drawig a marked oe is m/θ

11 (c) Give what happeed i draws 1,...,i-1 there are m i-1 marked ad we either draw a marked (y i =0) or a umarked (y i = 1) with the probability of the first give i (b) ad of the secod beig 1- that. The joit probability of all draws is the product of the coditioal probabilities. (d) Note that y 1 =1. Suppose =5 ad we draw Usig the formula i (b) we get L(θ) = 1 1 θ θ 1 θ θ 2 θ r = θ (θ 1) (θ 2) θ 4 -r-1 The geeral formula is obtaied i the same way, ad we see that r is sufficiet by the factorizatio criterio Chebyshev says P( Y c) E(Y 2 )/c 2. Let Y = (ˆθ θ) / se(ˆθ). The E(Y 2 )=1, ad P( Y c) 1-1/c 2. I other words P(ˆθ c se(ˆθ) θ ˆθ + c se(ˆθ)) 1 1 c 2 so ˆθ ± se(ˆθ) is at least a 1-1/c 2 α cofidece iterval. 79 Aother pivot X 1,,X iid desity From midterm, Y= / θ ~ X i f X (x;θ) = x θ 2 e x/θ ˆθ = 1 x i / 2 Oe-sided CIs Airplaes are ispected for corrosio every 10 years. A compay has ispected 5 of their fleet of 200 plaes, fidig o corrosio, ad would like a 95% lower cofidece boud o the probability p of o corrosio i 10 years, i.e. a value p L (x,1-α) such that P(p L (X,1-α) p) 1-α. 80 dbiom(0:6, 5, 0.99)

12 Oe-sided CIs, cot. Poisso approximatio? Aother approach (we will justify it later) is to use C(x) = {p: P p (X x) > 1 α} Bayesia methods Recall Bayes formula P(A i B) = P(B A i )P(A i ) j=1 P(B A j )P(A j ) I the cotext of cotiuous radom variables X ad Y this becomes f Y X (y x) = f X Y (x y)f Y (y) f X Y (x u)f Y (u)du Expressig ucertaity I the Bayesia approach to statistic we describe aythig that is ucertai by a probability distributio. The likelihood is the coditioal distributio of data X give the parameter (ow a radom variable) Θ: L(θ) = f X Θ (x θ) Before we collect data we assig a prior distributio to Θ: f Θ (θ) After observig data x, we compute the posterior distributio of Θ: f Θ X (θ x) Where does the prior come from? Previous experiece Expert kowledge Mathematical coveiece The posterior depeds o the prior. But if you have a lot of data, the posterior will look similar to the likelihood

13 Productio The error rate i productio of a computer chip is about 9%. The proportio P of faulty chips has a prior distributio proportioal to p 10 (1-p) 90. From a large batch, we sample 100 chips ad fid 16 defective. L(p) = f(p x) = desity prior post mode sd p What if we make aother experimet? Use the posterior from the past experimet as the prior for the ew oe. This is the same as usig the origial prior ad the performig the combied experimet. Expoetial case Let the prior be Exp(α) ad the data Exp(λ). The the posterior is f(λ x) αλ exp λ x i + α This is a gamma desity with shape parameter (+1) ad scale parameter x i +α. α is called a hyperparameter, ad is set by the statisticia based o prior expectatios

14 Cojugate priors A mathematically coveiet prior is oe where the prior ad posterior are i the same parametric family. Poisso likelihood: x i L(λ) = λ exp( λ) If we look at thigs ivolvig x s (or ) as parameters, ad thigs ivolvig λ as the dummy variable, we choose a prior of the form f(λ) λ α exp( βλ) which is a gamma desity. The the posterior desity will also be gamma, with shape parameter α + x i ad shape parameter β+. 94 Credible iterval Fid a iterval o the posterior distributio such that the probability that the parameter falls i that iterval is 95%. Commo way: high posterior desity iterval 95 Computer chips desity area 0.95 credible iterval (.9,.17) p 96 14