Advanced Statistical Methods: Beyond Linear Regression

Transcription

1 Advanced Statstcal Methods: Beyond Lnear Regresson John R. Stevens Utah State Unversty Notes 2. Statstcal Methods I Mathematcs Educators Workshop 28 March

2 2 What would your students know to do wth these data? Obs Flght Temp Damage 1 STS1 66 NO 2 STS9 70 NO 3 STS51B 75 NO 4 STS2 70 YES 5 STS41B 57 YES 6 STS51G 70 NO 7 STS3 69 NO 8 STS41C 63 YES 9 STS51F 81 NO 10 STS STS41D 70 YES 12 STS51I 76 NO 13 STS5 68 NO 14 STS41G 78 NO 15 STS51J 79 NO 16 STS6 67 NO 17 STS51A 67 NO 18 STS61A 75 YES 19 STS7 72 NO 20 STS51C 53 YES 21 STS61B 76 NO 22 STS8 73 NO 23 STS51D 67 NO 24 STS61C 58 YES

3 Two Sample t-test data: Temp by Damage t = , df = 21, p-value = alternatve hypothess: true dfference n means s not equal to 0 95 percent confdence nterval: sample estmates: mean n group NO mean n group YES

4 Does the t-test make sense here? Tradtonal: Treatment Group mean vs. Control Group mean What s the response varable? Temperature? [Quanttatve, Contnuous] Damage? [Qualtatve] 4

5 Tradtonal Statstcal Model 1 Lnear Regresson: predct contnuous response from [quanttatve] predctors Y=weght, X=heght Y=ncome, X=educaton level Y=frst-semester GPA, X=parent s ncome Y=temperature, X=damage (0=no, 1=yes) Can also control for other [possbly categorcal] factors ( covarates ): Sex Maor State of Orgn Number of Sblngs 5

6 Tradtonal Statstcal Model 2 Logstc Regresson: predct bnary response from [quanttatve] predctors Y= graduate wthn 5 years =0 vs. Y= not =1 X=frst-semester GPA Y=0 (no damage) vs. Y=1 (damage) X=temperature Y=0 (survve) vs. Y=1 (death) X=dosage (dose-response model) Can also control for other factors, or covarates Race, Sex Genotype p = P(Y=1 relevant factors) = prob. that Y=1, gven state of relevant factors 6

7 Tradtonal Dose-Response Model p = Probablty of death at dose d: p log 1 p = η = β + β d Look at what affects the shape of the curve, LD50 (lethal dose for 50% effcacy), etc. 0 1 p Dose-Response Curve d 7

8 Fttng the Dose-Response Model p log 1 p = η = β + β d Why logstc regresson? β 0 = place-holder constant β 1 = effect of dosage d To estmate parameters: Newton-Raphson teratve process to maxmze the lkelhood of the model Compare Y=0 (no damage) wth Y=1 (damage) groups 0 1 8

9 Y Pr Pr Lkelhood Functon (to be maxmzed) { 0,1} ( Y = 1) ( Y = 0) = p = 1 p ( β, β ) log L( β β ) l = 0 1 0, 1 p = f ( β, β ) 0 1 =... lkelhood for obs. L ( β ) = 0, β1 p y (1 p ) 1 y multply probabltes (ndependence) 9

10 Estmaton by IRLS Iteratvely Reweghted Least Squares equvalent: Newton-Raphson algorthm for teratvely solvng score equatons l ( β β ) β 0, 1 = 0 10

11 11 Coeffcents: Estmate Std. Error z value Pr(> z ) (Intercept) * Temp * --- Sgnf. codes: 0 *** ** 0.01 *

12 12 ˆ =... p

13 What f the data were even better? Complete separaton of ponts What should happen to our slope estmate? 13

14 ˆ =... p 14 Coeffcents: Estmate Std. Error z value Pr(> z ) (Intercept) Temp

15 Falure? Shape of lkelhood functon Large Standard Errors Soluton only n Rather than maxmzng lkelhood, consder a penalty: ~ l ( β, β ) = l( β, β ) / ("magntude of varance"of ( ˆ β, ˆ β ) 0 1

16 ˆ =... p Model ftted by Penalzed ML Confdence ntervals and p-values by Profle Lkelhood 16 coef se(coef) Chsq p (Intercept) Temp

17 Beetle Data Phosphne Total Dosage Recevng Total Total Survvors Observed at Genotype (mg/l) Dosage Deaths Survvors -/B -/H -/A +/B +/H +/A , ,798 10,

18 Dose-response model Recall smple model: p log = η = β0 + β1d 1 p p = Pr(Y=1 dosage level and genotype level ) p log 1 p = η = G + D d But when s genotype (covarate G ) observed? 18

19 Coeffcents: Estmate Std. Error z value Pr(> z ) (Intercept) e e e-04 1 dose e e e-33 1 G e e e-33 1 G2B e e e-19 1 G2H e e e-33 1 dose:g e e e-33 1 dose:g2b 3.984e e e-19 1 dose:g2h 7.754e e e-33 1 G1+:G2B 1.344e e e-20 1 G1+:G2H 3.395e e e-33 1 dose:g1+:g2b e e e-19 1 dose:g1+:g2h e e e Before we fx ths, frst a lttle detour

20 A Multvarate Gaussan Mxture Component s MVN(µ,Σ ) wth proporton π 20

21 21 The Maxmum Lkelhood Approach [ ] ( ) { } ( ) ( ) ( ) ( ) Σ = = Σ Σ = = Σ = = Σ = = n J J J y Z l y y Z y y y f P MVN Y Y Y I, log ),..., (,...,,,...,,,, for MVN pdf ) (, ) ( obs. n group, ~, obs. n group µ φ π θ µ µ π π θ φ µ φ π π µ K

22 A Possble Work-Around l Keys here: = = I [ obs. n group ] (,..., ) ( Z, ) = log ( y, Σ ) θ φ µ the true group membershps are unknown (latent) nj statstcans specalze n unknown quanttes

23 A reasonable approach 1. Randomly assgn group membershps, and estmate group means µ, covarance matrces Σ, and mxng proportons π 2. Gven those values, calculate (for each obs.) ξ = E[ θ] = P(obs. n group ) 3. Update estmates for µ, Σ, and π, weghtng each observaton by these ξ : ξ 4. Repeat steps 2 and 3 to convergence µˆ = ξ y 23

24 24 Plottng character and color ndcate most lkely component

25 The EM (Baum-Welch) Algorthm - maxmzaton made easer wth Z m = latent (unobserved) data; T = (Z,Z m ) = complete data 1. Start wth ntal guesses for parameters 2. Expectaton: At the kth teraton, compute Q ( ˆ ) [ ( ) ] ( k ) ' ˆ( k ) θ, θ = E l θ T Z, θ θˆ(0) 3. Maxmzaton: Obtan estmate ˆ ( k + 1) θ ( ) by maxmzng ˆ( k ) Q θ, θ over θ 4. Iterate steps 2 and 3 to convergence ($?) 25

26 26 Beetle Data Notaton Observed values N = # recevng dosage n = # survvors at dosage wth genotype Unobserved (latent) values = # recevng dosage wth genotype N If N had been observed: n p ~ Bnomal ( N,1 p ) = Prob. of death at dosage for genotype How N can be [latently] consdered: N P ( N1, K, N6 ) ~ Multnomal ( N, P) = = prop. of populaton wth genotype

27 Lkelhood Functon Parameters θ=(p,p) and complete data T=(n,N) After smplfcaton: l ( θ T ) = log f ( n, N p, P) = [log N + n! + log 1 { N log P! log ( p ) + ( N n ) log p }] ( N n ) Mechansm of mssng data suggests EM algorthm logn! 27

28 Mssng at Random (MAR) Necessary assumpton for usual EM applcatons Covarate x s MAR f probablty of observng x does not depend on x or any other unobserved covarate, but may depend on response and other observed covarates (Ibrahm 1990) Here genotype s observed only for survvors, and for all subects at zero dosage 28

29 29 Intalzaton Step Two classes of margnal nformaton here For all dosage levels observe N At zero dosage level observe for genotype Allows estmate of P Consder margnal dstn. of mssng categorcal covarate (genotype) Usng zero dosage level: (0) N,0 P =, N ˆ (0) pˆ Ths s the key the margnal dstrbuton of the mssng categorcal covarate = N,0

30 Expectaton Step log N! logn Droppng constants and : ~ Q ( k ) =, ~ { N + ( k ) n log log 1 P L ( k ) ~ ( ) ( ( k ) p ) + N n log p! Need to evaluate: [ ]! n, θ ( ) [ ˆ ( ] k ) ( k N n, θ, L ) = E log( N n ) ~ ( k ) ˆ k = E N 30 (*)

31 Expectaton Step Bayes Formula: h ( N n, θ ) = f ( n ) ( ) N f N f ( n N ) f ( N ) N ( N ) n n, θ ~ Multnomal N n, λ λ = l P p l P p l Multnomal ~ k Pˆ pˆ ( ) N nl + ( k ) ( k ) = N k k Pˆ ( ) ( ) l pˆ l l l n 31 (*)

32 Expectaton Step [ ( ) ] k For ) ( k L ) = E log N n! n, θ : Not needed for maxmzaton only affects EM convergence rate Drect calculaton from multnomal dstn. s possble but computatonally prohbtve Need to employ some approxmaton strategy ( ˆ ~ ( k ) Second-order Taylor seres about, usng Bnet s formula ( ) ( ) ( ) ( ) 1 π log N n! N n log N n + 1 N n N n log2 32 (*)

33 Expectaton Step Consder Bnet s formula (lke Strlng s): log Have: 1 ( N n)! ( N n + 2 ) log( N n + 1) 1 ( N n + 1) + log2π E [ ˆ ] [ ( ) ] ( k ) 2,,, ˆ k N n E N n θ 2 θ N-n Use a second-order ~ Taylor seres approxmaton taken k about N ( ) n ~ ( k ) ~ ( k ) k k L as a functon of : N, Pˆ ( ) ( ), pˆ, N, n (*)

34 Maxmzaton Step ~ ( k ) Q Porton of related to : ~ Q Pˆ ( k ) P = ( k+ 1) ~ ( k ) Q, ~ N ( k ) log P P by Lagrange multplers Porton of related to : ~ Q pˆ ( k ) p ( k + 1) =, p { ( ) ( ( k ) n log 1 p + N n ) log p } by Newton-Raphson teratons, ϑ = ~ wth some parameterzaton { G, } D 34 (*)

35 Convergence EM Convergence wth Crteron 1e-12 : 1639 Iteratons n 52 Seconds 35 Expected Log Lkelhood Q EM Iteraton

36 36 Dose Response Curves (log scale) -/B -/H -/A Prob. of death Prob. of death Prob. of death Dosage Dosage Dosage +/B +/H +/A Prob. of death Prob. of death Prob. of death Dosage Dosage Dosage

37 EM Results Confdence LD50 L95 U95 t -/B /H /A /B /H /A test statstc for H 0 : no dosage effect 37 separaton of ponts

38 38 Topcs Used Here Calculus Dfferentaton & Integraton (ncludng vector dfferentaton) Lagrange Multplers Taylor Seres Expansons Lnear Algebra Determnants & Egenvalues Invertng [computatonally/nearly sngular] Matrces Postve Defnteness Probablty Dstrbutons: Multvarate Normal, Bnomal, Multnomal Bayes Formula Statstcs Logstc Regresson Separaton of Ponts [Penalzed] Lkelhood Maxmzaton EM Algorthm Bology a lttle tme and communcaton