Topic 15: Maximum Likelihood Estimation

Size: px
Start display at page:

Download "Topic 15: Maximum Likelihood Estimation"

Transcription

1 Topic 5: Maximum Likelihood Estimatio November ad 3, 20 Itroductio The priciple of maximum likelihood is relatively straightforward. As before, we begi with a sample X (X,..., X of radom variables chose accordig to oe of a family of probabilities P θ. I additio, f(x θ, x (x,..., x will be used to deote the desity fuctio for the data whe θ is the true state of ature. The, the priciple of maximum likelihood yields a choice of the estimator ˆθ as the value for the parameter that makes the observed data most probable. Defiitio. The likelihood fuctio is the desity fuctio regarded as a fuctio of θ. The maximum likelihood estimator (MLE, L(θ x f(x θ, θ Θ. ( ˆθ(x arg max L(θ x. (2 θ We will lear that especially for large samples, the maximum likelihood estimators have may desirable properties. However, especially for high dimesioal data, the likelihood ca have may local maxima. Thus, fidig the global maximum ca be a major computatioal challege. This class of estimators has a importat property. If ˆθ(x is a maximum likelihood estimate for θ, the g(ˆθ(x is a maximum likelihood estimate for g(θ. For example, if θ is a parameter for the variace ad ˆθ is the maximum likelihood estimator, the ˆθ is the maximum likelihood estimator for the stadard deviatio. This flexibility i estimatio criterio see here is ot available i the case of ubiased estimators. Typically, maximizig the score fuctio, l L(θ x, the logarithm of the likelihood, will be easier. Havig the parameter values be the variable of iterest is somewhat uusual, so we will ext look at several examples of the likelihood fuctio. 2 Examples Example 2 (Beroulli trials. If the experimet cosists of Beroulli trial with success probability p, the L(p x p x ( p ( x p x ( p ( x p (x+ +x ( p (x+ +x. l L(p x l p( x i + l( p( x i ( x l p + ( x l( p. This equals zero whe p x. c 20 Joseph C. Watkis ( x l L(p x p p x p x p p( p 82

2 Itroductio to Statistical Methodology Maximum Likelihood Estimatio Exercise 3. Check that this is a maximum. Thus, ˆp(x x. I this case the maximum likelihood estimator is also ubiased. Example 4 (Normal data. Maximum likelihood estimatio ca be applied to a vector valued parameter. For a simple radom sample of ormal radom variables, we ca use the properties of the expoetial fuctio to simplify the likelihood fuctio. ( L(µ, σ 2 x The score fuctio 2πσ 2 exp (x µ 2 2σ 2 ( 2πσ 2 exp (x µ 2 2σ 2 l L(µ, σ 2 x 2 (l 2π + l σ2 2σ 2 µ l L(µ, σ2 x σ 2 Because the secod partial derivative with respect to µ is egative, (2πσ2 exp 2σ 2 (x i µ 2. (x i µ. ( x µ σ2 ˆµ(x x is the maximum likelihood estimator. For the derivative of the score fuctio with respect to the parameter σ 2, ( σ 2 l L(µ, σ2 x 2σ 2 + 2(σ 2 2 (x i µ 2 2(σ 2 2 σ 2 (x i µ 2. Recallig that ˆµ(x x, we obtai ˆσ 2 (x (x i x 2. Note that the maximum likelihood estimator is a biased estimator. (x i µ 2. Example 5 (Licol-Peterso method of mark ad recapture. Let s recall the variables i mark ad recapture: t be the umber captured ad tagged, k be the umber i the secod capture, r the the umber i the secod capture that are tagged, ad let N be the total populatio. Here t ad k is set by the experimetal desig; r is a observatio that may vary. The total populatio N is ukow. The likelihood fuctio for N is the hypergeometric distributio. ( t N t r( L(N r We would like to maximize the likelihood give the umber of recaptured idividuals r. Because the domai for N is the oegative itegers, we caot use calculus. However, we ca look at the ratio of the likelihood values for successive value of the total populatio. L(N r L(N r k r ( N k 83

3 l l Itroductio to Statistical Methodology Maximum Likelihood Estimatio 0.0e e-07.0e-06.5e e+00.0e-2 2.0e p p log(l log(l p p Figure : Likelihood fuctio (top row ad its logarithm, the score fuctio, (bottom row for Berouli trials. The left colum is based o 20 trials havig 8 ad successes. The right colum is based o 40 trials havig 6 ad 22 successes. Notice that the maximum likelihood is approximately 0 6 for 20 trials ad 0 2 for 40. I additio, ote that the peaks are more arrow for 40 trials rather tha 20. We shall later be able to associate this property to the variace of the maximum likelihood estimator. 84

4 Itroductio to Statistical Methodology Maximum Likelihood Estimatio N is more likely that N precisely whe this ratio is larger tha oe. The computatio below will show that this ratio is greater tha for small values of N ad less tha oe for large values. Thus, there is a place i the middle which has the maximum. We expad the biomial coefficiets i the expressio for L(N r ad simplify. L(N r L(N r ( t N t ( r( k r / N k ( t N t ( r( k r / N k ( N t ( N k r k ( N t k r ( N k (N t!(n!(n t k + r!(n k! (N t!n!(n t k + r!(n k! (N t! (N! (k r!(n t k+r! k!(n k! (N t! N! (k r!(n t k+r! k!(n k! (N t(n k N(N t k + r. Thus, the ratio exceeds if ad oly if L(N r (N t(n k L(N r N(N t k + r (N t(n k > N(N t k + r N 2 tn kn + tk > N 2 tn kn + rn tk > rn tk r > N Writig [x] for the iteger part of x, we see that L(N r > L(N r for N < [tk/r] ad L(N r L(N r for N [tk/r]. This give the maximum likelihood estimator [ ] tk ˆN. r Thus, the maximum likelihood estimator is, i this case, obtaied from the method of momets estimator by roudig dow to the ext iteger. Let look at the example of mark ad capture from the previous topic. There N 2000, the umber of fish i the populatio, is ukow to us. We tag t 200 fish i the first capture evet, ad obtai k 400 fish i the secod capture. > N<-2000 > t<-200 > fish<-c(rep(,t,rep(0,n-t > k<-400 > r<-sum(sample(fish,k > r [] 42 I this simulated example, we fid r 42 recaptured fish. For the likelihood fuctio, we look at a rage of values for N that is symmetric about Here, ˆN [ /42] 904. > N<-c(800:2200 > L<-dhyper(r,t,N-t,k > plot(n,l,type"l",ylab"l(n 42" Example 6 (Liear regressio. Our data are observatios with oe explaatory variable ad oe respose variable. The model is that y i α + βx i + ɛ i 85

5 Itroductio to Statistical Methodology Maximum Likelihood Estimatio Likelihood Fuctio for Mark ad Recapture L(N Figure 2: Likelihood fuctio L(N 42 for mark ad recapture with t 200 tagged fish, k 400 i the secod capture with r 42 havig tags ad thus recapture. Note that the maximum likelihood estimator for the total fish populatio is ˆN 904. N where the ɛ i are idepedet mea 0 ormal radom variables. The (ukow variace is σ 2. Thus, the joit desity for the ɛ i is exp ɛ2 2πσ 2 2σ 2 exp ɛ2 2 2πσ 2 2σ 2 exp ɛ2 2πσ 2 2σ 2 (2πσ2 exp 2σ 2 Sice ɛ i y i (α + βx i, the likelihood fuctio The score fuctio L(α, β, σ 2 y, x (2πσ2 exp 2σ 2 (y i (α + βx i 2. l L(α, β, σ 2 y, x 2 (l 2π + l σ2 2σ 2 (y i (α + βx i 2. Cosequetly, maximizig the likelihood fuctio for the parameters α ad β is equivalet to miimizig SS(α.β (y i (α + βx i 2. Thus, the priciple of maximum likelihood is equivalet to the least squares criterio for ordiary liear regressio. The maximum likelihood estimators α ad β give the regressio lie ŷ i ˆα + ˆβx i. Exercise 7. Show that the maximum likelihood estimator for σ 2 is ˆσ MLE 2 (y i ŷ i 2. k ɛ 2 i 86

6 Itroductio to Statistical Methodology Maximum Likelihood Estimatio Frequetly, software will report the ubiased estimator. For ordiary least square procedures, this is ˆσ 2 U 2 (y i ŷ i 2. k For the measuremets o the legths i cetimeters of the femur ad humerus for the five specimes of Archeopteryx, we have the followig R output for liear regressio. > femur<-c(38,56,59,64,74 > humerus<-c(4,63,70,72,84 > summary(lm(humerus femur Call: lm(formula humerus femur Residuals: Coefficiets: Estimate Std. Error t value Pr(> t (Itercept femur *** --- Sigif. codes: 0 *** 0.00 ** 0.0 * Residual stadard error:.982 o 3 degrees of freedom Multiple R-squared: ,Adjusted R-squared: F-statistic: 254. o ad 3 DF, p-value: The residual stadard error of.982 cetimeters is obtaied by squarig the 5 residuals, dividig by ad takig a square root. Example 8 (weighted least squares. If we kow the relative size of the variaces of the ɛ i, the we have the model y i α + βx i + γ(x i ɛ i where the ɛ i are, agai, idepedet mea 0 ormal radom variable with ukow variace σ 2. I this case, ɛ i γ(x i (y i α + βx i are idepedet ormal radom variables, mea 0 ad (ukow variace σ 2. the likelihood fuctio L(α, β, σ 2 y, x (2πσ2 exp 2σ 2 w(x i (y i (α + βx i 2 where w(x /γ(x 2. I other words, the weights are iversely proportioal to the variaces. The log-likelihood is l L(α, β, σ 2 y, x 2 l 2πσ2 2σ 2 w(x i (y i (α + βx i 2. 87

7 Itroductio to Statistical Methodology Maximum Likelihood Estimatio Exercise 9. Show that the maximum likelihood estimators ˆα w ad β w x i. have formulas ˆβ w cov w(x, y var w (x, ȳ w ˆα w + ˆβ w x w where x w ad ȳ w are the weighted meas x w w(x ix i w(x i, ȳ w The weighted covariace ad variace are, respectively, cov w (x, y The maximum likelihood estimator for σ 2 is w(x iy i w(x i. w(x i(x i x w (y i ȳ w w(x, var w (x i ˆσ 2 MLE k w(x i(y i ŷ i 2 w(x. i I the case of weighted least squares, the predicted value for the respose variable is ŷ i ˆα w + ˆβ w x i. w(x i(x i x w 2 w(x, i Exercise 0. Show that ˆα w ad ˆβ w are ubiased estimators of α ad β. I particular, ordiary (uweighted least square estimators are ubiased. I computig the optimal values usig itroductory differetial calculus, the maximum ca occur at either critical poits or at the edpoits. The ext example show that the maximum value for the likelihood ca occur at the ed poit of a iterval. Example (Uiform radom variables. If our data X (X,..., X are a simple radom sample draw from uiformly distributed radom variable whose maximum value θ is ukow, the each radom variable has desity { /θ if 0 x θ, f(x θ 0 otherwise. Therefore, the joit desity or the likelihood { /θ if 0 x f(x θ L(θ x i θ for all i, 0 otherwise. Coseqeutly, the joit desity is 0 wheever ay of the x i > θ. Restatig this i terms of likelihood, o value of θ is possible that is less tha ay of the x i. Coseuetly, ay value of θ less tha ay of the x i has likelihood 0. Symbolically, { 0 for θ < maxi x L(θ x i x (, /θ for θ max i x i x (. Recall the otatio x ( for the top order statistic based o observatios. The likelihood is 0 o the iterval (0, x ( ad is positive ad decreasig o the iterval [x (,. Thus, to maximize L(θ x, we should take the miimum value of θ o this iterval. I other words, ˆθ(x x (. Because the estimator is always less tha the parameter value it is meat to estimate, the estimator ˆθ(X X ( < θ, 88

8 Itroductio to Statistical Methodology Maximum Likelihood Estimatio 0.8 L(θ x 0.6 /θ observatios x i i this iterval θ Figure 3: Likelihood fuctio for uiform radom variables o the iterval [0, θ]. The likelihood is 0 up to max i x i ad /θ afterwards. Thus, we suspect it is biased dowwards, i. e.. E θ X ( < θ. For 0 x θ, the distributio fuctio for X ( max i X i is F X( (x P { max i X i x} P {X x, X 2 x,..., X < x} P {X x}p {X 2 x} P {X < x} each of these radom variables have the same distributio fuctio 0 for x 0, x P {X i x} θ for 0 < x θ, for θ < x. Thus, the distributio fuctio 0 ( for x 0, F X( (x x θ for 0 < x θ, for θ < x. Take the derivative to fid the desity, 0 for x 0, f X( (x x θ for 0 < x θ, 0 for θ < x. The mea θ x x 0 θ θ E θ X ( dx θ x θ dx 0 ( + θ x+ 0 + θ. This cofirms the bias of the estimator X ( ad gives us a strategy to fid a ubiased estimator. I particular, the choice is a ubiased estimator of θ. d(x + X ( 89

9 Itroductio to Statistical Methodology Maximum Likelihood Estimatio 3 Summary of Estimates Look to the text above for the defiitio of variables. parameter estimate p Beroulli trials ˆp x i x ubiased N mark recapture [ ˆN kt ] r biased upward ormal observatios µ ˆµ x i x ubiased σ 2 ˆσ mle 2 (x i x 2 biased dowward ˆσ u 2 (x i x 2 ubiased σ ˆσ mle β (x i x 2 biased dowward liear regressio ˆβ cov(x,y var(x ubiased α ˆα ȳ ˆβ x ubiased σ 2 ˆσ mle 2 (y i (ˆα + ˆβx 2 biased dowward ˆσ u 2 2 (y i (ˆα + ˆβx 2 ubiased σ ˆσ mle 4 Asymptotic Properties (y i (ˆα + ˆβx 2 biased dowward uiform [0, θ] θ ˆθ maxi x i biased dowward ˆθ + i x i ubiased Much of the attractio of maximum likelihood estimators is based o their properties for large sample sizes. We summarizes some the importat properties below, savig a more techical discussio of these properties for later.. Cosistecy. If θ 0 is the state of ature ad ˆθ (X is the maximum likelihood estimator based o observatios from a simple radom sample, the ˆθ (X θ 0 as. I words, as the umber of observatios icrease, the distributio of the maximum likelihood estimator becomes more ad more cocetrated about the true state of ature. 2. Asymptotic ormality ad efficiecy. Uder some assumptios that allows, amog several aalytical properties, the use of the delta method, a cetral limit theorem holds. Here we have (ˆθ (X θ 0 coverges i distributio as to a ormal radom variable with mea 0 ad variace /I(θ 0, the Fisher iformatio for oe observatio. Thus, Var θ0 (ˆθ (X I(θ 0, 90

10 Itroductio to Statistical Methodology Maximum Likelihood Estimatio the lowest variace possible uder the Crámer-Rao lower boud. This property is called asymptotic efficiecy. We ca write this i terms of the z-score. Let Z ˆθ(X θ 0 / I(θ 0. The, as with the cetral limit theorem, Z coverges i distributio to a stadard ormal radom variable. 3. Properties of the log likelihood surface. For large sample sizes, the variace of a maximum likelihood estimator of a sigle parameter is approximately the egative of the reciprocal of the the Fisher iformatio [ ] 2 I(θ E l L(θ X. θ2 the egative reciprocal of the secod derivative, also kow as the curvature, of the log-likelihood fuctio. The Fisher iformatio ca be approimated by the observed iformatio based o the data x, J(ˆθ 2 l L(ˆθ(x x, θ2 givig the curvature of the likelihood surface at the maximum likelihood estimate ˆθ(x If the curvature is small ear the maximum likelihood estimator, the the likelihood surface is earty flat ad the variace is large. If the curvature is large ad thus the variace is small, the likelihood is strogly curved at the maximum. We ow look at these properties i some detail by revisitig the example of the distributio of fitess effects. For this example, we have two parameters - α ad β for the gamma distributio ad so, we will wat to exted the properties above to circumstaces i which we are lookig to estimate more tha oe parameter. 5 Multidimesioal Estimatio For a multidimesioal parameter space θ (θ, θ 2,..., θ, the Fisher iformatio I(θ is ow a matrix. As with oe-dimesioal case, the ij-th etry has two alterative expressios, amely, [ I(θ ij E θ l L(θ X ] [ ] 2 l L(θ X E θ l L(θ X. θ i θ j θ i θ j Rather tha takig reciprocals to obtai a estimate of the variace, we fid the matrix iverse I(θ. This iverse will provide estimates of both variaces ad covariaces. To be precise, for observatios, let ˆθ i, (X be the maximum likelihood estimator of the i-th parameter. The Var θ (ˆθ i, (X I(θ ii Cov θ (ˆθ i, (X, ˆθ j, (X I(θ ij. Whe the i-th parameter is θ i, the asymptotic ormality ad efficiecy ca be expressed by otig that the z-score is approximately a stadard ormal. Z i, ˆθ i (X θ i I(θ ii /. Example 2. To obtai the maximum likelihood estimate for the gamma family of radom variables, write the likelihood ( ( ( β α β α β α L(α, β x Γ(α xα e βx Γ(α xα e βx (x x 2 x α e β(x+x2+ +x. Γ(α 9

11 Itroductio to Statistical Methodology Maximum Likelihood Estimatio diff alpha Figure 4: The graph of (l ˆα l x d dα l Γ(ˆα + P l x i crosses the horizotal axis at ˆα The fact that the graph of the derivative is decreasig states that the score fuctio moves from icreasig to decreasig with α ad thus is a maximum. ad the score fuctio l L(α, β x (α l β l Γ(α + (α l x i β To determie the parameters that maximize the likelihood, we solve the equatios ad α l L(ˆα, ˆβ x (l ˆβ d dα l Γ(ˆα + l x i 0 β l L(ˆα, ˆβ x ˆαˆβ x i 0, or x ˆαˆβ. Substitutig ˆβ ˆα/ x ito the first equatio results the followig relatioship for ˆα (l ˆα l x d dα l Γ(ˆα + l x i 0 which ca be solved umerically. The derivative of the logarithm of the gamma fuctio ψ(α d l Γ(α dα x i. is kow as the digamma fuctio ad is called i R with digamma. For the example for the distributio of fitess effects α 0.23 ad β 5.35 with 00, a simulated data set yields ˆα ad ˆβ for maximum likelihood estimator. (See Figure 4. To determie the variace of these estimators, we first compute the Fisher iformatio matrix. Takig the appropriate derivatives, we fid that each of the secod order derivatives are costat ad thus the expected values used to determie the etries for Fisher iformatio matrix are the egative of these costats. I(α, β 2 d2 l L(α, β x l Γ(α, α2 dα2 I(α, β 22 2 β 2 l L(α, β x α β 2, 92

12 Itroductio to Statistical Methodology Maximum Likelihood Estimatio This give a Fisher iformatio matrix 2 I(α, β 2 αβ l L(α, β x β. I(α, β ( d 2 dα 2 l Γ(α β β The secod derivative of the logarithm of the gamma fuctio ψ (α d2 l Γ(α dα2 is kow as the trigamma fuctio ad is called i R with trigamma. The iverse ( I(α, β α β α( d2 dα l Γ(α β 2 α β 2. β 2 d2 dα 2 l Γ(α For the example for the distributio of fitess effects α 0.23 ad β 5.35 ad 00, ad ( I(0.23, (0.23( ( Var (0.23,5.35 (ˆα , Var (0.23,5.35 ( ˆβ σ (0.23,5.35 (ˆα 0.00, σ (0.23,5.35 ( ˆβ ( Compare this to the empirical values of ad for the method of momets. This gives the followig table of stadard deviatios for 00 observatio method ˆα ˆβ maximum likelihood method of momets ratio Thus, the stadard deviatio for the maximum likelihood estimator is respectively 7% ad 56% that of method of momets estimator. We will look at the impact as we move o to our ext topic - iterval estimatio ad the cofidece itervals. Exercise 3. If the data are a simple radom sample of 00 observatios of a Γ(0.23, 5.35 radom variable. Use the approximate ormality of maximum likelihood estimators to estimate P {ˆα } P { ˆβ 5.690}.. 6 Choice of Estimators With all of the desirable properties of the maximum likelihood estimator, the questio arises as to why would oe choose a method of momets estimator? Oe aswer is that the use maximum likelihood techiques relies o kowig the desity fuctio explicitly i order to be able to perform the ecessary aalysis to maximize the score fuctio ad fid the Fisher iformatio. However, much less about the experimet is eed i order to compute momets. Thus far, we have computed momets usig the desity E θ X m 93 x m f X (x θ dx.

13 Itroductio to Statistical Methodology Maximum Likelihood Estimatio We could determie, for example, the (radom umber of a give protei i the cells i a tissue by givig the distributio of the umber of cells ad the the distributio of the umber of the give protei per cell. This ca be used to calculate the mea ad variace for the umber of cells with some ease. However, givig a explicit expressio for the desity ad hece the likelihood fuctio is more difficult to obtai ad ca lead to quite itricate computatios to carry out the desired aalysis of the likelihood fuctio. 7 Techical Aspects We ca use cocepts previously itroduced to obtai the properties for the maximum likelihood estimator. For example, θ 0 is more likely that a aother parameter value θ L(θ 0 X > L(θ X if ad oly if By the strog law of large umbers, this sum coverges to [ E θ0 l f(x ] θ 0. f(x θ l f(x i θ 0 f(x i θ > 0. which is greater tha 0. thus, for a large umber of observatios ad a give value of θ, the with a probability early oe, L(θ 0 X > L(θ X ad the so the maximum likelihood estimator has a high probability of beig very ear θ 0. For the asymptotic ormality ad efficiecy, we write the liear approximatio of the score fuctio d dθ l L(θ X d dθ l L(θ 0 X + (θ θ 0 d2 dθ 2 l L(θ 0 X. Now substitute θ ˆθ ad ote that d dθ l L(ˆθ X 0. The (ˆθ (X θ 0 d dθ l L(θ 0 X d 2 dθ 2 l L(θ 0 X d dθ l L(θ 0 X d 2 dθ 2 l L(θ 0 X Now assume that θ 0 is the true state of ature. The, the radom variables d l f(x i θ 0 /dθ are idepedet with mea 0 ad variace I(θ 0. Thus, the distributio of umerator d dθ l L(θ 0 X d dθ l f(x i θ 0 coverges, by the cetral limit theorem, to a ormal radom variable with mea 0 ad variace I(θ 0. For the deomiator, d 2 l f(x i θ 0 /dθ 2 are idepedet with mea I(θ 0. Thus, d 2 dθ 2 l L(θ 0 X d 2 dθ 2 l f(x i θ 0 coverges, by the law of large umbers, to I(θ 0. Thus, the distributio of the ratio, (ˆθ (X θ 0, coverges to a ormal radom variable with variace I(θ 0 /I(θ 0 2 /I(θ 0. 8 Aswers to Selected Exercises 3. We have foud that p l L(p x x p p( p 94

14 Itroductio to Statistical Methodology Maximum Likelihood Estimatio Thus p l L(p x > 0 if p < x, ad l L(p x < 0 p if p > x I words, the score fuctio l L(p x is icreasig for p < x ad l L(p x is decreasig for p > x. Thus, ˆp(x x is a maximum. 7. The log-likelihood fuctio l L(α, β, σ 2 y, x 2 (l(2π + l σ2 2σ 2 (y i (α + βx i 2 leads to the ordiary least squares equatios for the maximum likelihood estimates ˆα ad ˆβ. Take the partial derivative with respect to σ 2, σ 2 L(α, β, σ2 y, x 2σ 2 + 2(σ 2 2 (y i (α + βx i 2. This partial derivative is 0 at the maximum likelihood estimates ˆσ 2, ˆα ad ˆβ. or 0 2ˆσ 2 + 2(ˆσ 2 2 ˆσ 2 (y i (ˆα + ˆβx i 2 (y i (ˆα + ˆβx i The maximum likelihood priciple leads to a miimizatio problem for SS w (α, β ɛ 2 i w(x i (y i (α + βx i 2. Followig the steps to derive the equatios for ordiary least squares, take partial derivatives to fid that β SS w(α, β 2 w(x i x i (y i α βx i α SS w(α, β 2 w(x i (y i α βx i. Set these two equatios equal to 0 ad call the solutios ˆα w ad ˆβ w. 0 0 w(x i x i (y i ˆα w ˆβ w x i w(x i x i y i ˆα w w(x i (y i ˆα w ˆβ w x i w(x i y i ˆα w w(x i x i ˆβ w w(x i ˆβ w w(x i x 2 i ( w(x i x i (2 Multiply these equatios by the appropriate factors to obtai ( ( ( ( ( ( 0 w(x i w(x i x i y i ˆα w w(x i w(x i x i ˆβ w w(x i w(x i x 2 i (3 ( ( ( ( ( 2 0 w(x i x i w(x i y i ˆα w w(x i w(x i x i ˆβ w w(x i x i (4 95

15 Itroductio to Statistical Methodology Maximum Likelihood Estimatio Now subtract the equatio (4 from equatio (3 ad solve for ˆβ. ˆβ ( w(x i ( w(x ix i y i ( w(x ix i ( w(x iy i w(x ix 2 i ( w(x ix i 2 w(x i(x i x w (y i ȳ w w(x cov w(x, y i(x i x w 2 var w (x. Next, divide equatio (2 by w(x i to obtai ȳ w ˆα w + ˆβ w x w. (5 0. Because the ɛ i have mea zero, E (α,β y i E (α,β [α + βx i + γ(x i ɛ i ] α + βx i + γ(x i E (α,β [ɛ i ] α + βx i. Next, use the liearity property of expectatio to fid the mea of ȳ w. E (α,β ȳ w w(x ie (α,β y i w(x i w(x i(α + βx i w(x i α + β x w. (6 Take together, we have that E (α,β [y i ȳ w ] (α+βx i. (α+βx i β(x i x w. To show that ˆβ w is a ubiased estimator, we see that [ ] covw (x, y E (α,β ˆβw E (α,β E [ (α,β[cov w (x, y] var w (x var w (x var w (x E w(x ] i(x i x w (y i ȳ w (α,β w(x i var w (x w(x i(x i x w E (α,β [y i ȳ w ] w(x i β var w (x To show that ˆα w is a ubiased estimator, recall that ȳ w ˆα w + ˆβ w x w. Thus w(x i(x i x w (x i x w w(x i E (α,β ˆα w E (α,β [ȳ w ˆβ w x w ] E (α,β ȳ w E (α,β [ ˆβ w ] x w α + β x w β x w α, usig (6 ad the fact that ˆβ w is a ubiased estimator of β 3. For ˆα, we have the z-score zˆα Thus, usig the ormal approximatio, ˆα P {ˆα } P {zˆα 0.684} β. For ˆβ, we have the z-score z ˆβ ˆβ Here, the ormal approximatio gives P { ˆβ 5.690} P {z ˆβ 0.297}

16 Itroductio to Statistical Methodology Maximum Likelihood Estimatio score.eval alpha beta score score alpha beta Figure 5: (top The score fuctio ear the maximum likelihood estimators. The domai is 0. α 0.4 ad 4 β 8. (bottom Graphs of vertical slices through the score fuctio surface. (left ˆβ ad 0. α 0.4 varies. (right ˆα ad 4 β 8. The variace of the estimator is approximately the egative reciprocal of the secod derivative of the score fuctio at the maximum likelihood estimators. Note that the score fuctio is early flat as β varies. This leads to the iterpretatio that a rage of values for β are early equally likely ad that the variace for the estimator for ˆβ will be high. O the other had, the score fuctio has a much greater curvature for the α parameter ad the estimator ˆα will have a much smaller variace tha ˆβ 97

Unbiased Estimation. February 7-12, 2008

Unbiased Estimation. February 7-12, 2008 Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

Topic 10: Introduction to Estimation

Topic 10: Introduction to Estimation Topic 0: Itroductio to Estimatio Jue, 0 Itroductio I the simplest possible terms, the goal of estimatio theory is to aswer the questio: What is that umber? What is the legth, the reactio rate, the fractio

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1 EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum

More information

Random Variables, Sampling and Estimation

Random Variables, Sampling and Estimation Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig

More information

Estimation for Complete Data

Estimation for Complete Data Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

MATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4

MATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4 MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.

More information

Review Questions, Chapters 8, 9. f(y) = 0, elsewhere. F (y) = f Y(1) = n ( e y/θ) n 1 1 θ e y/θ = n θ e yn

Review Questions, Chapters 8, 9. f(y) = 0, elsewhere. F (y) = f Y(1) = n ( e y/θ) n 1 1 θ e y/θ = n θ e yn Stat 366 Lab 2 Solutios (September 2, 2006) page TA: Yury Petracheko, CAB 484, yuryp@ualberta.ca, http://www.ualberta.ca/ yuryp/ Review Questios, Chapters 8, 9 8.5 Suppose that Y, Y 2,..., Y deote a radom

More information

Statistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.

Statistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample. Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized

More information

Linear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d

Linear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y

More information

Simulation. Two Rule For Inverting A Distribution Function

Simulation. Two Rule For Inverting A Distribution Function Simulatio Two Rule For Ivertig A Distributio Fuctio Rule 1. If F(x) = u is costat o a iterval [x 1, x 2 ), the the uiform value u is mapped oto x 2 through the iversio process. Rule 2. If there is a jump

More information

Statistical Theory MT 2009 Problems 1: Solution sketches

Statistical Theory MT 2009 Problems 1: Solution sketches Statistical Theory MT 009 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. (a) Let 0 < θ < ad put f(x, θ) = ( θ)θ x ; x = 0,,,... (b) (c) where

More information

Let us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.

Let us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f. Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,

More information

TAMS24: Notations and Formulas

TAMS24: Notations and Formulas TAMS4: Notatios ad Formulas Basic otatios ad defiitios X: radom variable stokastiska variabel Mea Vätevärde: µ = X = by Xiagfeg Yag kpx k, if X is discrete, xf Xxdx, if X is cotiuous Variace Varias: =

More information

1 Inferential Methods for Correlation and Regression Analysis

1 Inferential Methods for Correlation and Regression Analysis 1 Iferetial Methods for Correlatio ad Regressio Aalysis I the chapter o Correlatio ad Regressio Aalysis tools for describig bivariate cotiuous data were itroduced. The sample Pearso Correlatio Coefficiet

More information

Topic 18: Composite Hypotheses

Topic 18: Composite Hypotheses Toc 18: November, 211 Simple hypotheses limit us to a decisio betwee oe of two possible states of ature. This limitatio does ot allow us, uder the procedures of hypothesis testig to address the basic questio:

More information

Overview of Estimation

Overview of Estimation Topic Iferece is the problem of turig data ito kowledge, where kowledge ofte is expressed i terms of etities that are ot preset i the data per se but are preset i models that oe uses to iterpret the data.

More information

Statistical Theory MT 2008 Problems 1: Solution sketches

Statistical Theory MT 2008 Problems 1: Solution sketches Statistical Theory MT 008 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. a) Let 0 < θ < ad put fx, θ) = θ)θ x ; x = 0,,,... b) c) where α

More information

6. Sufficient, Complete, and Ancillary Statistics

6. Sufficient, Complete, and Ancillary Statistics Sufficiet, Complete ad Acillary Statistics http://www.math.uah.edu/stat/poit/sufficiet.xhtml 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > 1 2 3 4 5 6 6. Sufficiet, Complete, ad Acillary

More information

Chapter 6 Principles of Data Reduction

Chapter 6 Principles of Data Reduction Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a

More information

Econ 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara

Econ 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara Poit Estimator Eco 325 Notes o Poit Estimator ad Cofidece Iterval 1 By Hiro Kasahara Parameter, Estimator, ad Estimate The ormal probability desity fuctio is fully characterized by two costats: populatio

More information

EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY

EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA, 016 MODULE : Statistical Iferece Time allowed: Three hours Cadidates should aswer FIVE questios. All questios carry equal marks. The umber

More information

Stat 421-SP2012 Interval Estimation Section

Stat 421-SP2012 Interval Estimation Section Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible

More information

This exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.

This exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam. Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the

More information

1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable

More information

7.1 Convergence of sequences of random variables

7.1 Convergence of sequences of random variables Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite

More information

Topic 10: The Law of Large Numbers

Topic 10: The Law of Large Numbers Topic : October 6, 2 If we choose adult Europea males idepedetly ad measure their heights, keepig a ruig average, the at the begiig we might see some larger fluctuatios but as we cotiue to make measuremets,

More information

Stat410 Probability and Statistics II (F16)

Stat410 Probability and Statistics II (F16) Some Basic Cocepts of Statistical Iferece (Sec 5.) Suppose we have a rv X that has a pdf/pmf deoted by f(x; θ) or p(x; θ), where θ is called the parameter. I previous lectures, we focus o probability problems

More information

TMA4245 Statistics. Corrected 30 May and 4 June Norwegian University of Science and Technology Department of Mathematical Sciences.

TMA4245 Statistics. Corrected 30 May and 4 June Norwegian University of Science and Technology Department of Mathematical Sciences. Norwegia Uiversity of Sciece ad Techology Departmet of Mathematical Scieces Corrected 3 May ad 4 Jue Solutios TMA445 Statistics Saturday 6 May 9: 3: Problem Sow desity a The probability is.9.5 6x x dx

More information

ECON 3150/4150, Spring term Lecture 3

ECON 3150/4150, Spring term Lecture 3 Itroductio Fidig the best fit by regressio Residuals ad R-sq Regressio ad causality Summary ad ext step ECON 3150/4150, Sprig term 2014. Lecture 3 Ragar Nymoe Uiversity of Oslo 21 Jauary 2014 1 / 30 Itroductio

More information

Lecture 11 and 12: Basic estimation theory

Lecture 11 and 12: Basic estimation theory Lecture ad 2: Basic estimatio theory Sprig 202 - EE 94 Networked estimatio ad cotrol Prof. Kha March 2 202 I. MAXIMUM-LIKELIHOOD ESTIMATORS The maximum likelihood priciple is deceptively simple. Louis

More information

January 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS

January 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS Jauary 25, 207 INTRODUCTION TO MATHEMATICAL STATISTICS Abstract. A basic itroductio to statistics assumig kowledge of probability theory.. Probability I a typical udergraduate problem i probability, we

More information

NANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS

NANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS NANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS STRUCTURE OF EXAMINATION PAPER. There will be oe 2-hour paper cosistig of 4 questios.

More information

CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5

CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5 CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio

More information

Econ 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.

Econ 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1. Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio

More information

Probability and Statistics

Probability and Statistics ICME Refresher Course: robability ad Statistics Staford Uiversity robability ad Statistics Luyag Che September 20, 2016 1 Basic robability Theory 11 robability Spaces A probability space is a triple (Ω,

More information

Direction: This test is worth 250 points. You are required to complete this test within 50 minutes.

Direction: This test is worth 250 points. You are required to complete this test within 50 minutes. Term Test October 3, 003 Name Math 56 Studet Number Directio: This test is worth 50 poits. You are required to complete this test withi 50 miutes. I order to receive full credit, aswer each problem completely

More information

( θ. sup θ Θ f X (x θ) = L. sup Pr (Λ (X) < c) = α. x : Λ (x) = sup θ H 0. sup θ Θ f X (x θ) = ) < c. NH : θ 1 = θ 2 against AH : θ 1 θ 2

( θ. sup θ Θ f X (x θ) = L. sup Pr (Λ (X) < c) = α. x : Λ (x) = sup θ H 0. sup θ Θ f X (x θ) = ) < c. NH : θ 1 = θ 2 against AH : θ 1 θ 2 82 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION Defiitio: et X be a radom sample with joit p.m/d.f. f X x θ. The geeralised likelihood ratio test g.l.r.t. of the NH : θ H 0 agaist the alterative AH : θ H 1,

More information

Problem Set 4 Due Oct, 12

Problem Set 4 Due Oct, 12 EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios

More information

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015 ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],

More information

Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.

Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n. Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator

More information

Exponential Families and Bayesian Inference

Exponential Families and Bayesian Inference Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where

More information

Chapter 6 Sampling Distributions

Chapter 6 Sampling Distributions Chapter 6 Samplig Distributios 1 I most experimets, we have more tha oe measuremet for ay give variable, each measuremet beig associated with oe radomly selected a member of a populatio. Hece we eed to

More information

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals

More information

Properties and Hypothesis Testing

Properties and Hypothesis Testing Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.

More information

Expectation and Variance of a random variable

Expectation and Variance of a random variable Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio

More information

Maximum Likelihood Estimation

Maximum Likelihood Estimation ECE 645: Estimatio Theory Sprig 2015 Istructor: Prof. Staley H. Cha Maximum Likelihood Estimatio (LaTeX prepared by Shaobo Fag) April 14, 2015 This lecture ote is based o ECE 645(Sprig 2015) by Prof. Staley

More information

7.1 Convergence of sequences of random variables

7.1 Convergence of sequences of random variables Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite

More information

Bayesian Methods: Introduction to Multi-parameter Models

Bayesian Methods: Introduction to Multi-parameter Models Bayesia Methods: Itroductio to Multi-parameter Models Parameter: θ = ( θ, θ) Give Likelihood p(y θ) ad prior p(θ ), the posterior p proportioal to p(y θ) x p(θ ) Margial posterior ( θ, θ y) is Iterested

More information

Economics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator

Economics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters

More information

of the matrix is =-85, so it is not positive definite. Thus, the first

of the matrix is =-85, so it is not positive definite. Thus, the first BOSTON COLLEGE Departmet of Ecoomics EC771: Ecoometrics Sprig 4 Prof. Baum, Ms. Uysal Solutio Key for Problem Set 1 1. Are the followig quadratic forms positive for all values of x? (a) y = x 1 8x 1 x

More information

Computing Confidence Intervals for Sample Data

Computing Confidence Intervals for Sample Data Computig Cofidece Itervals for Sample Data Topics Use of Statistics Sources of errors Accuracy, precisio, resolutio A mathematical model of errors Cofidece itervals For meas For variaces For proportios

More information

Topic 8: Expected Values

Topic 8: Expected Values Topic 8: Jue 6, 20 The simplest summary of quatitative data is the sample mea. Give a radom variable, the correspodig cocept is called the distributioal mea, the epectatio or the epected value. We begi

More information

MATH/STAT 352: Lecture 15

MATH/STAT 352: Lecture 15 MATH/STAT 352: Lecture 15 Sectios 5.2 ad 5.3. Large sample CI for a proportio ad small sample CI for a mea. 1 5.2: Cofidece Iterval for a Proportio Estimatig proportio of successes i a biomial experimet

More information

Questions and Answers on Maximum Likelihood

Questions and Answers on Maximum Likelihood Questios ad Aswers o Maximum Likelihood L. Magee Fall, 2008 1. Give: a observatio-specific log likelihood fuctio l i (θ) = l f(y i x i, θ) the log likelihood fuctio l(θ y, X) = l i(θ) a data set (x i,

More information

Statisticians use the word population to refer the total number of (potential) observations under consideration

Statisticians use the word population to refer the total number of (potential) observations under consideration 6 Samplig Distributios Statisticias use the word populatio to refer the total umber of (potetial) observatios uder cosideratio The populatio is just the set of all possible outcomes i our sample space

More information

Stat 139 Homework 7 Solutions, Fall 2015

Stat 139 Homework 7 Solutions, Fall 2015 Stat 139 Homework 7 Solutios, Fall 2015 Problem 1. I class we leared that the classical simple liear regressio model assumes the followig distributio of resposes: Y i = β 0 + β 1 X i + ɛ i, i = 1,...,,

More information

Lecture 2: Monte Carlo Simulation

Lecture 2: Monte Carlo Simulation STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?

More information

Direction: This test is worth 150 points. You are required to complete this test within 55 minutes.

Direction: This test is worth 150 points. You are required to complete this test within 55 minutes. Term Test 3 (Part A) November 1, 004 Name Math 6 Studet Number Directio: This test is worth 10 poits. You are required to complete this test withi miutes. I order to receive full credit, aswer each problem

More information

EE 4TM4: Digital Communications II Probability Theory

EE 4TM4: Digital Communications II Probability Theory 1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair

More information

17. Joint distributions of extreme order statistics Lehmann 5.1; Ferguson 15

17. Joint distributions of extreme order statistics Lehmann 5.1; Ferguson 15 17. Joit distributios of extreme order statistics Lehma 5.1; Ferguso 15 I Example 10., we derived the asymptotic distributio of the maximum from a radom sample from a uiform distributio. We did this usig

More information

Lecture 3: MLE and Regression

Lecture 3: MLE and Regression STAT/Q SCI 403: Itroductio to Resamplig Methods Sprig 207 Istructor: Ye-Chi Che Lecture 3: MLE ad Regressio 3. Parameters ad Distributios Some distributios are idexed by their uderlyig parameters. Thus,

More information

Introductory statistics

Introductory statistics CM9S: Machie Learig for Bioiformatics Lecture - 03/3/06 Itroductory statistics Lecturer: Sriram Sakararama Scribe: Sriram Sakararama We will provide a overview of statistical iferece focussig o the key

More information

Kurskod: TAMS11 Provkod: TENB 21 March 2015, 14:00-18:00. English Version (no Swedish Version)

Kurskod: TAMS11 Provkod: TENB 21 March 2015, 14:00-18:00. English Version (no Swedish Version) Kurskod: TAMS Provkod: TENB 2 March 205, 4:00-8:00 Examier: Xiagfeg Yag (Tel: 070 2234765). Please aswer i ENGLISH if you ca. a. You are allowed to use: a calculator; formel -och tabellsamlig i matematisk

More information

Properties of Point Estimators and Methods of Estimation

Properties of Point Estimators and Methods of Estimation CHAPTER 9 Properties of Poit Estimators ad Methods of Estimatio 9.1 Itroductio 9. Relative Efficiecy 9.3 Cosistecy 9.4 Sufficiecy 9.5 The Rao Blackwell Theorem ad Miimum-Variace Ubiased Estimatio 9.6 The

More information

First Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise

First Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >

More information

Lecture Note 8 Point Estimators and Point Estimation Methods. MIT Spring 2006 Herman Bennett

Lecture Note 8 Point Estimators and Point Estimation Methods. MIT Spring 2006 Herman Bennett Lecture Note 8 Poit Estimators ad Poit Estimatio Methods MIT 14.30 Sprig 2006 Herma Beett Give a parameter with ukow value, the goal of poit estimatio is to use a sample to compute a umber that represets

More information

Machine Learning Brett Bernstein

Machine Learning Brett Bernstein Machie Learig Brett Berstei Week Lecture: Cocept Check Exercises Starred problems are optioal. Statistical Learig Theory. Suppose A = Y = R ad X is some other set. Furthermore, assume P X Y is a discrete

More information

The variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2.

The variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2. SAMPLE STATISTICS A radom sample x 1,x,,x from a distributio f(x) is a set of idepedetly ad idetically variables with x i f(x) for all i Their joit pdf is f(x 1,x,,x )=f(x 1 )f(x ) f(x )= f(x i ) The sample

More information

1.010 Uncertainty in Engineering Fall 2008

1.010 Uncertainty in Engineering Fall 2008 MIT OpeCourseWare http://ocw.mit.edu.00 Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu.terms. .00 - Brief Notes # 9 Poit ad Iterval

More information

KLMED8004 Medical statistics. Part I, autumn Estimation. We have previously learned: Population and sample. New questions

KLMED8004 Medical statistics. Part I, autumn Estimation. We have previously learned: Population and sample. New questions We have previously leared: KLMED8004 Medical statistics Part I, autum 00 How kow probability distributios (e.g. biomial distributio, ormal distributio) with kow populatio parameters (mea, variace) ca give

More information

CSE 527, Additional notes on MLE & EM

CSE 527, Additional notes on MLE & EM CSE 57 Lecture Notes: MLE & EM CSE 57, Additioal otes o MLE & EM Based o earlier otes by C. Grat & M. Narasimha Itroductio Last lecture we bega a examiatio of model based clusterig. This lecture will be

More information

Since X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain

Since X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain Assigmet 9 Exercise 5.5 Let X biomial, p, where p 0, 1 is ukow. Obtai cofidece itervals for p i two differet ways: a Sice X / p d N0, p1 p], the variace of the limitig distributio depeds oly o p. Use the

More information

Statistical Inference Based on Extremum Estimators

Statistical Inference Based on Extremum Estimators T. Rotheberg Fall, 2007 Statistical Iferece Based o Extremum Estimators Itroductio Suppose 0, the true value of a p-dimesioal parameter, is kow to lie i some subset S R p : Ofte we choose to estimate 0

More information

SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker

SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker CHAPTER 9. POINT ESTIMATION 9. Covergece i Probability. The bases of poit estimatio have already bee laid out i previous chapters. I chapter 5

More information

Simple Linear Regression

Simple Linear Regression Simple Liear Regressio 1. Model ad Parameter Estimatio (a) Suppose our data cosist of a collectio of pairs (x i, y i ), where x i is a observed value of variable X ad y i is the correspodig observatio

More information

Statistical Intervals for a Single Sample

Statistical Intervals for a Single Sample 3/5/06 Applied Statistics ad Probability for Egieers Sixth Editio Douglas C. Motgomery George C. Ruger Chapter 8 Statistical Itervals for a Sigle Sample 8 CHAPTER OUTLINE 8- Cofidece Iterval o the Mea

More information

6.3 Testing Series With Positive Terms

6.3 Testing Series With Positive Terms 6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial

More information

ECE 901 Lecture 12: Complexity Regularization and the Squared Loss

ECE 901 Lecture 12: Complexity Regularization and the Squared Loss ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality

More information

4. Partial Sums and the Central Limit Theorem

4. Partial Sums and the Central Limit Theorem 1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems

More information

Big Picture. 5. Data, Estimates, and Models: quantifying the accuracy of estimates.

Big Picture. 5. Data, Estimates, and Models: quantifying the accuracy of estimates. 5. Data, Estimates, ad Models: quatifyig the accuracy of estimates. 5. Estimatig a Normal Mea 5.2 The Distributio of the Normal Sample Mea 5.3 Normal data, cofidece iterval for, kow 5.4 Normal data, cofidece

More information

Statistics 511 Additional Materials

Statistics 511 Additional Materials Cofidece Itervals o mu Statistics 511 Additioal Materials This topic officially moves us from probability to statistics. We begi to discuss makig ifereces about the populatio. Oe way to differetiate probability

More information

Lecture 7: Properties of Random Samples

Lecture 7: Properties of Random Samples Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ

More information

Statistics 203 Introduction to Regression and Analysis of Variance Assignment #1 Solutions January 20, 2005

Statistics 203 Introduction to Regression and Analysis of Variance Assignment #1 Solutions January 20, 2005 Statistics 203 Itroductio to Regressio ad Aalysis of Variace Assigmet #1 Solutios Jauary 20, 2005 Q. 1) (MP 2.7) (a) Let x deote the hydrocarbo percetage, ad let y deote the oxyge purity. The simple liear

More information

The standard deviation of the mean

The standard deviation of the mean Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider

More information

Mathematical Statistics - MS

Mathematical Statistics - MS Paper Specific Istructios. The examiatio is of hours duratio. There are a total of 60 questios carryig 00 marks. The etire paper is divided ito three sectios, A, B ad C. All sectios are compulsory. Questios

More information

f(x i ; ) L(x; p) = i=1 To estimate the value of that maximizes L or equivalently ln L we will set =0, for i =1, 2,...,m p x i (1 p) 1 x i i=1

f(x i ; ) L(x; p) = i=1 To estimate the value of that maximizes L or equivalently ln L we will set =0, for i =1, 2,...,m p x i (1 p) 1 x i i=1 Parameter Estimatio Samples from a probability distributio F () are: [,,..., ] T.Theprobabilitydistributio has a parameter vector [,,..., m ] T. Estimator: Statistic used to estimate ukow. Estimate: Observed

More information

Distribution of Random Samples & Limit theorems

Distribution of Random Samples & Limit theorems STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to

More information

Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 19

Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 19 CS 70 Discrete Mathematics ad Probability Theory Sprig 2016 Rao ad Walrad Note 19 Some Importat Distributios Recall our basic probabilistic experimet of tossig a biased coi times. This is a very simple

More information

Solutions: Homework 3

Solutions: Homework 3 Solutios: Homework 3 Suppose that the radom variables Y,...,Y satisfy Y i = x i + " i : i =,..., IID where x,...,x R are fixed values ad ",...," Normal(0, )with R + kow. Fid ˆ = MLE( ). IND Solutio: Observe

More information

ECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors

ECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic

More information

Chapter 3. Strong convergence. 3.1 Definition of almost sure convergence

Chapter 3. Strong convergence. 3.1 Definition of almost sure convergence Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i

More information

The picture in figure 1.1 helps us to see that the area represents the distance traveled. Figure 1: Area represents distance travelled

The picture in figure 1.1 helps us to see that the area represents the distance traveled. Figure 1: Area represents distance travelled 1 Lecture : Area Area ad distace traveled Approximatig area by rectagles Summatio The area uder a parabola 1.1 Area ad distace Suppose we have the followig iformatio about the velocity of a particle, how

More information

11 Correlation and Regression

11 Correlation and Regression 11 Correlatio Regressio 11.1 Multivariate Data Ofte we look at data where several variables are recorded for the same idividuals or samplig uits. For example, at a coastal weather statio, we might record

More information

7-1. Chapter 4. Part I. Sampling Distributions and Confidence Intervals

7-1. Chapter 4. Part I. Sampling Distributions and Confidence Intervals 7-1 Chapter 4 Part I. Samplig Distributios ad Cofidece Itervals 1 7- Sectio 1. Samplig Distributio 7-3 Usig Statistics Statistical Iferece: Predict ad forecast values of populatio parameters... Test hypotheses

More information

Linear Regression Models

Linear Regression Models Liear Regressio Models Dr. Joh Mellor-Crummey Departmet of Computer Sciece Rice Uiversity johmc@cs.rice.edu COMP 528 Lecture 9 15 February 2005 Goals for Today Uderstad how to Use scatter diagrams to ispect

More information

Hypothesis Testing. Evaluation of Performance of Learned h. Issues. Trade-off Between Bias and Variance

Hypothesis Testing. Evaluation of Performance of Learned h. Issues. Trade-off Between Bias and Variance Hypothesis Testig Empirically evaluatig accuracy of hypotheses: importat activity i ML. Three questios: Give observed accuracy over a sample set, how well does this estimate apply over additioal samples?

More information

CEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering

CEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio

More information