CM9S: Machie Learig for Bioiformatics Lecture - 03/3/06 Itroductory statistics Lecturer: Sriram Sakararama Scribe: Sriram Sakararama We will provide a overview of statistical iferece focussig o the key cocepts. For more details as well as more precise statemets of the mathematical results (e.g., regularity coditios ), cosult a stadard statistical text (Casella ad Berger). Statistical iferece (X,..., X ) is the data draw from a distributio P. θ is a ukow populatio parameter, i.e., a fuctio of P. The goal of statistical iferece is to lear θ from (X,..., X ). Three broad ad related tasks i statistical iferece. Estimatio. Hypothesis testig 3. Predictio For each of these tasks, we have procedures for fidig a estimator or test or predictor ad procedures for evaluatig them. We will focus o the first two tasks. Poit estimatio A statistic is a fuctio of the data. The goal of poit estimatio is to fid a statistic t such that ˆθ = t(x,..., X ) is close to θ. ˆθ is a estimator of θ. Some procedures to fid poit estimators:. Maximum likelihood. Method of momets 3. Bayesia methods. Maximum Likelihood Estimatio Defiitio.. Give X,..., X P (x θ), the likelihood fuctio L(θ x) = i= P (x i θ). Defiitio.. A maximum likelihood estimator ˆθ(x) = argmax θ L(θ x) = argmax θ LL(θ x)
LL(θ x) = log L(θ x) = log P (x i θ) i= For a differetiable log likelihood fuctio, possible cadidates for MLE are θ that solve Why oly possible cadidates? d dθ LL(θ) = 0 Not a sufficiet coditio for a maximum (uless the log likelihood is cocave). Maximum could also be o the boudary of the parameter space. Example (Normal likelihood). X,..., X N (µ, σ ) ˆµ = x = ˆσ = i= x i (x i x) i= = S We would like to evaluate poit estimators. Geerally, we ca evaluate estimators i a fiite sample (askig how it behaves for a fixed sample size) or i a asymptotic settig (askig how it behaves as the sample size becomes large).. Fiite sample Defiitio.3. Bias of a estimator ] bias(ˆθ ) = E [ˆθ θ A estimator is ubiased if bias = 0. Defiitio.4. Variace of a estimator Defiitio.5. Mea Square Error of a estimator [ ]) ] var(ˆθ ) = E (ˆθ E [ˆθ [ ) ] mse(ˆθ ) = E (ˆθ θ The mse decomposes as mse(ˆθ ) = bias(ˆθ ) + var(ˆθ ) These defiitios assess the performace of the estimator over repeated experimets, i.e., over ew data that we do ot see i practice. Comparig estimators is ot straightforward though. Besides depedig o the evaluatio criterio, the performace of the estimator ca deped o the true parameter. Example (Normal likelihood (cotiued)).. X is a ubiased estimator of µ.
. ˆµ = X is also a ubiased estimator of µ 3. Var [X ] = σ while Var [ˆµ] = σ 4. ˆσ is ot a ubiased estimator of σ. Its bias is σ. 5. S is a ubiased estimator of σ 6. O the other had, mse( ˆσ ) < mse(s ). 7. S is ot a ubiased estimator of σ 8. X ad S are ubiased estimators of the populatio mea ad variace for ay i.i.d sample from a populatio with fiite mea ad variace..3 Asymptotics It is ofte challegig to compute fiite-sample properties. Alterately, we ca ask how the estimator behaves whe sample size goes to ifiity. To aalyze behavior whe sample size goes to ifiity, we will eed a otio of the limit of sequeces of radom variables much like the limit of sequeces of umbers. Defiitio.6 (Covergece i probability). A sequece of radom variable X,..., X coverges i probability to a radom variable X, deoted X p X, if for every ɛ > 0, lim Pr( X X > ɛ) = 0 Defiitio.7 (Covergece i distributio). A sequece of radom variable X,... coverges i distributio to a radom variable X, deoted X d X, for all x where F (x) is cotiuous. lim F X (x) = F (x) Theorem (Weak Law of Large Numbers). Let X,... be radom variables with E [X i ] = µ, Var [X i ] = σ <. X p µ. Theorem (Cetral Limit Theorem). Let X,... be radom variables with E [X i ] = µ, Var [X i ] = σ <.. (X µ) σ d N (0, ) With these cocepts, we ca ow talk about asymptotic properties of poit estimators. Defiitio.8 (Cosistecy). A sequece of estimators ˆ parameter θ if ˆθ p θ. Example (Normal likelihood (cotiued)). ˆµ = x is a cosistet estimator of µ. ˆσ is a cosistet estimator of σ. S is a cosistet estimator of σ. S is a cosistet estimator of σ. θ is a cosistet sequece of estimators of a Theorem 3 (Cosistecy of the MLE). Let X,..., X f(x θ) ad let ˆθ deote the MLE of θ. ˆθ p θ If the bias of ˆθ 0 ad its variace 0, i.e., its MSE goes to 0, the it is cosistet. 3
Theorem 4 (Asymptotic Normality of the MLE). (ˆθ θ) d N (0, I (θ)) [ ] I(θ) = E log P (X θ) θ is the Fisher iformatio associated with the model. Example. Normal likelihood Let us focus o the mea. For a sigle observatio X N (µ, σ ) log P (X θ) µ log P (X θ) µ = x µ σ = σ I(µ) = σ This makes ituitive sese. Iverse of the Fisher iformatio is the variace. Smaller the variace, larger the Fisher iformatio, ad the estimates of the mea should be more precise. 3 Hypothesis testig A hypothesis is a costrait o the parameter θ. Examples:. Offsprig has equal chace of iheritig either of two alleles from a paret (Medel s first law holds). Two gees segregate idepedetly (Medel s secod law holds) 3. A geetic mutatio has o effect o disease 4. A drug has o effect o blood pressure Defiitio 3.. Null hypothesis: H 0 : θ Θ 0. Alterate hypothesis: H : θ Θ Defiitio 3. (Hypothesis test). A rule that specifies values of the sample for which H 0 should be accepted or rejected. Typically, a hypothesis test is specified i terms of a test statistic t(x,..., X ) ad a rejectio/acceptace regio. 3. Fidig tests Oe commoly used procedure to fid tests is the Likelihood Ratio Test. Other procedures iclude the Wald Test ad Score test. Defiitio 3.3 (LRT statistic). λ(x) = sup θ Θ 0 L(θ x) sup θ Θ L(θ x) = L(ˆθ 0 x) L(ˆθ x) Θ = Θ 0 Θ Defiitio 3.4 (Likelihood Ratio Test (LRT)). A likelihood ratio test (LRT) is a test that rejects H 0 if λ(x) < c. 4
Example (Test the mea of ormal model). Give X,..., X N (µ, ), H 0 : µ = 0 versus H : µ 0. λ(x) = i= exp( xi exp( Reject H 0 if λ(x) < c equivalet to reject H 0 if x > = exp( x ) ) i= (xi ˆµ) ) log( c ). How do we choose c? 3. Evaluatig tests Decisio Truth H 0 H H 0 TN FP H FN TP FP: False positive (type-i error) FN: False egative (type-ii error) TP: True positive TN: True egative For θ Θ 0, probability of false positive for a test R: = P θ (Test rejects). For θ Θ, probability of false egative: P θ (Test accepts). To evaluate a test, we eed to tradeoff false positives ad false egatives. probability below some threshold, what is the false egative threshold? For tests with false positive Defiitio 3.5 (Level of a test). A test has level α if sup θ Θ0 P θ (Test rejects) α. α is a upper boud o the false positive probability. Remark. Ofte we set α = 0.05 For the LRT, choose c to cotrol α: sup θ Θ0 P θ (λ(x) c) = α. Trivial test is to ever reject. Istead, we would like to maximize the probability of rejectig the ull, i.e., power, if it is false while cotrollig the false positive probability. Example (Normal mea (cotiued)). For the test of the ormal mea, Θ 0 = {0}. For θ = 0, the test statistic X N (0, ) to achieve the desired sigificace level. P 0 (λ(x) c) = P 0 ( X = P ( Z log( c ) ) log( c )) Here Z N (0, ). P ( Z z α ) = α where z α is the α-quatile of the stadard ormal distributio. We set log( c ) = z α. 5
The sigificace level is a crude measure. If we reject the ull hypothesis at level α = 0.05, it does ot tell us how strog the evidece agaist H 0 is. The p-value of a test is the smallest α at which the test rejects H 0. Thus, small values of p-value implies stroger evidece agaist H 0. Defiitio 3.6 (p-value). For a statistic t(x) such that large values of t give evidece agaist H 0, the p-value p(x): p(x) = sup θ Θ0 P θ (t(x) t(x)) A p-value is valid if 0 α sup θ Θ0 P θ (p(x) α) α If p(x) is a valid p-value, a test that rejects H 0 because p(x) α is a level α test. Example (p-value for the test of ormal mea). The LRT rejects H 0 : µ = 0 if t(x) = X is large. If µ = 0, the t(x) Z(0, ). So if we observe data x, the p-value for the LRT is p(x) = P ( t(x) t(x)) = P ( Z t(x)) = Φ( t(x)) + Φ(t(x)) = ( Φ(t(x))) We ca compute the p-value for differet values of the test statistic t (Table ). t p-value 0.3 0.04 3.69e-03 4 6.33e-05 5 5.73e-07 Table : P-value for test of ormal mea as a fuctio of the test statistic t = s X To use the LRT, we eed to choose c so that the false positive probability is bouded. To compute false positive probability, we eed to kow how the statistic is distributed uder the ull (the samplig distributio of the statistic). This is easy for the test of the ormal mea but difficult more geerally. A alterative approach to evaluate the LRT (or equivaletly to choose c) relies o asymptotics. 3.. Asymptotics of the LRT Theorem 5 (Wilks theorem). Let X,..., X p(x θ). For testig θ = θ 0 versus θ θ 0 log λ(x ) d χ For testig θ Θ 0 versus θ Θ 0, if ν is the differece i the umber of free parameters betwee θ Θ ad θ Θ 0 : log λ(x ) d χ ν Reject H 0 iff log λ(x) χ ν, α gives us a asymptotic (hece approximate) level α test. Remark. Oe of the coditios for the Wilks theorem to hold is that the true parameter lies i the iterior of the parameter space. This coditio ca be violated i practice, e.g., whe the ull hypothesis restricts the parameter to the boudary of the parameter space. 6
3.. Permutatio tests I may istaces, eve asymptotic results are difficult to obtai. It is also ot clear if sample size is large eough for asymptotics to hold. Example 3 (Permutatio test). Test if distributio of two samples is the same X,..., X F Y,..., Y G H 0 : F = G = P 0 We choose the statistic: t((x,..., X ), (Y,..., Y )) = X Y. The key idea is that uder H 0, all permutatios of the data are equally likely. Use this idea to compute the samplig distributio.. Radomly choose out of elemets to belog to group. The remaiig belog to group.. Compute the differece i meas betwee the two groups. 3. How ofte does the differece i meas o the permuted data exceed the observed differece? I may cases, the umber of permutatios is too large to exhaustively eumerate. The we use a radom sample of permutatios (Mote-Carlo approximatio). 4 Iterval estimatio We would like a measure of cofidece o our estimates. Defiitio 4.. For a scalar parameter θ, give data X, a iterval estimator is a pair of fuctios L(x,..., x ) ad U(x,..., x ), L U, such that [L(X), U(X)] covers θ. Remark. The iterval is radom while θ is fixed 4. Evaluatig iterval estimators Defiitio 4. (Coverage probability). The coverage probability of a iterval estimator [L(X), U(X)] is the probability that this iterval covers the true parameter θ, deoted P θ (θ [L(X), U(X)]). Defiitio 4.3 (( α) cofidece iterval). A iterval estimator is termed a α cofidece iterval if if θ P θ (θ [L(X), U(X)]) α. 4. Fidig iterval estimators A commo method for costructig cofidece itervals is to ivert a test statistic. Example 4. Give X,..., X N (µ, ), fid ( α) cofidece iterval for µ. Set H 0 : µ = µ 0 versus H : µ µ 0. Oe test is to reject H 0 if {x : x µ 0 > z α } So H 0 is accepted for {x : x µ 0 z α x z α } or equivaletly: µ 0 x + z α 7
Sice the test has size α, P (H 0 is accepted µ = µ 0 ) = α, P ( x z α Sice this is true for all µ 0, we have So [ x z α, x + z α P µ ( x z α µ 0 x + z α µ x + z α ] is a α cofidece iterval. µ = µ 0 ) = α ) = α 8