1.010 Uncertainty in Engineering Fall 2008

Size: px

Start display at page:

Download "1.010 Uncertainty in Engineering Fall 2008"

Augustus Hawkins
5 years ago
Views:

1 MIT OpeCourseWare Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit:

2 .00 - Brief Notes # 9 Poit ad Iterval Estimatio of Distributio Parameters (a) Some Commo Distributios i Statistics. Chi-square distributio Let Z, Z 2,..., Z be iid stadard ormal variables. The distributio of χ 2 = Z 2 i is called the Chi-square distributio with degrees of freedom. E[χ 2 ] = V ar[χ 2 ] = 2 t distributio Z t = ( ) /2 Probability desity fuctio of χ 2 for = 2, 5, 0. Let Z, Z, Z 2,..., Z be iid stadard ormal variables. The distributio of Zi 2 is called the Studet s t distributio with degrees of freedom.

3 2 E[t ] = 0, > 2 V ar[t ] = 2, 2 F distributio Probability desity fuctio of t for =, 5,. Note: t = N(0, ). Let W, W 2,..., W m, Z, Z 2,..., Z be iid stadard ormal variables. The distributio of F m, = m W 2 m i χ 2 = m m χ Z 2 2 i is called the F distributio with m ad degrees of freedom. As, 2 mf m, χ m

4 3 (b) Poit Estimatio of Distributio Parameters: Objective ad Criteria. Defiitio of (poit) estimator Let be a ukow parameter of the distributio F X of a radom variable X, for example the mea m of the variace σ 2. Cosider a radom sample of size from the statistical populatio of X, {X, X 2,..., X }. A estimator Θ of is a fuctio Θ(X, X 2,..., X ) that produces a umerical estimate of for each realizatio x, x 2,..., x of X, X 2,..., X. Notice: Θ is a radom variable whose distributio depeds o. Desirable properties of estimators. Ubiasedess: Θ is said to be a ubiased estimator of if, for ay give, E sample [Θ ] =. The bias b () of Θ is defied as: b Θ () = E sample [Θ ] Θ 2. Mea Squared Error (MSE): The mea squared error of Θ is the secod iitial momet of the estimatio error e = Θ, i.e., MSE () = E[(Θ ) 2 ] = b 2 Θ () + V ar[θ ] Θ Oe would like the mea squared error of a estimator to be as small as possible. (c) Poit Estimatio of Distributio Parameters: Methods.. Method of momets Suppose that F X has ukow parameters, 2,..., r. The idea behid the method of momets is to estimate, 2,..., r so that r selected characteristics of the distributio match their sample values. The characteristics are ofte take to be the iitial momets: µ i = E[X i ], i =,..., r The method is described below for the case r = 2.

5 4 The first ad secod iitial momets of X are, i geeral, fuctios of the ukow parameters, ad 2 : µ (, 2 ) = E[X, 2 ] = xf X, 2 (x)dx µ 2 (, 2 ) = E[X 2, 2 ] = x 2 f X, 2 (x)dx The sample values of these momets are: µ = X i = X µ 2 = X i 2 Estimators of ad 2 are obtaied by solvig the equatios for Θ ad Θ 2: µ (Θ, Θ 2) = µ µ 2 (Θ, Θ 2) = µ 2 This method is ofte simple to apply, but may produce estimators that have higher MSE tha other methods, e.g. maximum likelihood. Example: If = m ad 2 = σ 2, the: µ = m ad µ 2 = m 2 + σ 2 µ = X i = X ad µ 2 = X i m = X m = X The estimators m ad σ 2 are obtaied by solvig: m 2 + σ 2 = which gives: 2 X i 2

6 ( ) 2 2 σ 2 = X X i 5 = (X X) 2 Notice that σ 2 is a biased estimator sice its expected value is σ 2. For this reaso, oe typically uses the modified estimator: S 2 = (X i X) 2 which is ubiased. 2. Method of maximum likelihood: Cosider agai the case r = 2. The likelihood fuctio of ad 2 give a sample, L(, 2 sample), is defied as: L(, 2 sample) P [sample, 2 ] Where P is either probability or probability desity ad is regarded for a give sample as a fuctio of ad 2. I the case whe X is a cotiuous variable: P [sample, 2 ] = f X (x i, 2 ) The maximum likelihood estimators (Θ ) ML ad (Θ 2) ML are the values of ad 2 that maximize the likelihood, i.e., L(, 2 sample) is maximum for = (Θ ) ML ad 2 = (Θ 2) ML I may cases, (Θ ) ML ad ( Θ 2) ML ca be foud by imposig the statioarity coditios: L[(Θ, Θ 2) sample] = 0 ad L[(Θ, Θ 2) sample] = 0 Θ Θ 2 or, more frequetly, the equivalet coditios i terms of the log-likelihood:

7 6 (l L[(Θ, Θ 2) sample]) = 0 ad (l L[(Θ, Θ 2) sample]) = 0 Θ Θ 2 Properties of maximum likelihood estimators: As the sample size, maximum likelihood estimators:. are ubiased; 2. have the smallest possible value of MSE. Example: For X N(m, σ 2 ) with ukow parameters m ad σ 2, the maximum likelihood estimators of the parameters are: ( ) σ 2 m ML = X i = X N m, σ 2 = (X m ML ) 2 i ML σ 2 = (X i X) 2 χ 2 ( ) Notice that i this case the ML estimators m ad σ 2 are the same as the estimators produced by the method of momets. This is ot true i geeral. 3. Bayesia estimatio The previous two methods of poit estimatio are based o the classical statistical approach which assumes that the distributio parameters, 2,..., r are costats but ukow. I Bayesia estimatio,, 2,..., r are viewed as ucertai (radom variables) ad their ucertaity is quatified through probability distributios. There are 3 steps i Bayesia estimatio: Step : Quatify iitial Step 2: Use sample Step 3: Choose a sigle ucertaity o i the iformatio to update value estimate of form of a prior distributio, ucertaity posterior f distributio, f

8 7 The various steps are described below i the order 2, 3,. Step 2: How to update prior ucertaity give a sample Recall that for radom variables, f X f () f X (x) Here, f = f ad f = f X. Further, usig l( X) f X (x), oe obtais: f () f ()l( X) Step 3: How to choose Two mai methods:. Use some characteristic of f, such as the mea or the mode. The choice is rather arbitrary. Note that the mode correspods i a sese to the maximum likelihood, applied to the posterior distributio rather tha the likelihood. 2. Decisio theoretic approach: (more objective ad preferable) by. Defie a loss fuctio $( ) which is the loss if the estimate is ad the true value is. Calculate the expected posterior loss or Risk of as: R( ) = E [$( )] = $( )f ()d Choose such that R( ) is miimum. If $( ) is a quadratic fuctio of ( i i ), the R( ) is miimum for = E [] 0, if = If $( ) =, the is the mode of f c. > 0, if = Step : How to select f. Judgemetally. This approach is especially useful i egieerig desig, where subjective judgemet is ofte eccessary. This is how subjective judgemet is formally icorporated i the decisio process.

9 8 2. Based o prior data e.g. a sample of s from other data sets. 3. To reflect igorace, o-iformative prior. For example, if is a scalar parameter that ca attai values from to +, the f ()d d ( flat ) ad f () l( sample) i.e. the posterior reflects oly the likelihood. If > 0, the oe typically takes f l (l )d l d l. I this case, f (). 4. Cojugate prior. There are distributio types such that if f () is of that type, the f () f ()l() is also of the same type. Such distributios are called cojugate distributios. Example: Let: X N(m, σ 2 ) with σ 2 kow. = m ukow. Suppose: f N(m, σ 2 ) m It ca be show that l(m X,..., X ) desity of N(X, σ 2 /) From f m f ml(m sample), oe obtais ( ) m (σ 2 /) + Xσ 2 f m N m = (σ2 /) + σ 2, σ 2 = σ 2 + σ 2 I this case, f m N(m, σ 2 ) is a example of a cojugate prior, sice f m is also ormal, of the type N(m, σ 2 ). σ 2 If oe writes σ 2 =, the has the meaig of equivalet prior sample size ad m has the meaig of equivalet prior sample average. (d) Approximate Cofidece Itervals for Distributio Parameters.. Classical Approach Problem: is a ukow distributio parameter. Defie two sample statistics Θ (X,..., X ) ad Θ 2(X,..., X ) such that: P [Θ (X,..., X ) < < Θ 2(X,..., X )] = P where P is a give probability.

10 9 A iterval [Θ (X,..., X ), Θ 2(X,..., X )] with the above property is called a cofidece iterval of at cofidece level P. A simple method to obtai cofidece itervals is as follows. Cosider a poit estimatio Θ such that, exactly or i approximatio, Θ N(, σ 2 ()). If the variace σ 2 () depeds o, oe replaces σ 2 () with σ 2 ( Θ). The: Θ N(0, ) σ( Θ) P [Θ σ( Θ)Z P /2 < < Θ + σ( Θ)Z P /2] = P where Z α is the value exceeded with probability α by a stadard ormal variable. Example: = m = mea of a expoetial distributio. I this case, Θ = X Gamma(m, ), where Gamma(m, ) is the distributio of the sum of iid expoetial variables, each with mea value m. The mea ad variace of Gamma(m, ) are m ad m 2, respectively. Moreover, for large, Gamma(m, ) is close to N(m, m 2 ). Therefore, i approximatio, ( ) m 2 X N m, Usig the previous method, a approximate cofidece iterval for m at cofidece level P is [ ] X X X Z P /2, X + Z P /2 2. Bayesia Approach I Bayesia aalysis, itervals [, 2] that cotai with a give probability P are simply obtaied from the coditio that: F ( 2) F ( ) = P where F is the posterior CDF of.

Direction: This test is worth 250 points. You are required to complete this test within 50 minutes.

Direction: This test is worth 250 points. You are required to complete this test within 50 minutes. Term Test October 3, 003 Name Math 56 Studet Number Directio: This test is worth 50 poits. You are required to complete this test withi 50 miutes. I order to receive full credit, aswer each problem completely