Survival Analysis: Weeks 2-3. Lu Tian and Richard Olshen Stanford University

Transcription

1 Survival Analysis: Weeks 2-3 Lu Tian and Richard Olshen Stanford University

2 2 Kaplan-Meier(KM) Estimator Nonparametric estimation of the survival function S(t) = pr(t > t) The nonparametric estimation is more robust and does not depend on any parametric assumption.

3 3 If there is no censoring S(t) can be consistently estimated by Ŝ(t) = n n i= I(T i > t). Ŝ is a discrete distribution with mass probability of n at observed times T,, T n Ŝ(t) is the nonparametric maximum likelihood estimator (NPMLE) for S(t).

4 4 CDF as NPMLE Assuming that F ( ) is discrete with mass probability at T < T 2 < < T n, where {T, T 2, } are observed times. Let f = pr(t = T ), f 2 = pr(t = T 2 ),. Objective: estimate f, f 2,. Method: maximize n i= f i subject to n i= f i =. Solution: ˆf = ˆf 2 = = ˆf n = n.

5 5 Data and Assumptions Data: {(U i, δ i ), i =,, n} where U i = min(t i, C i ) and δ i = I(T i C i ). Assumptions:. T,, T n i.i.d. F ( ) = S( ) 2. C,, C n i.i.d. G( ) 3. T i C i, i =,, n. Noninformative censoring!

6 6 If there is censoring Assuming that F ( ) is discrete with mass probability at v < v 2 <, where {v, v 2, } are observed times. Let f = pr(t = v ), f 2 = pr(t = v 2 ),. Objective: estimate f, f 2,.

7 7 Example Obs: 2, 2, 3 +, 5, 5 +, 7, 9, 6, 6, 8 +, where + means censored v = 2; v 2 = 3, v 3 = 5, v 4 = 7, v 5 = 9, v 6 = 6, v 7 = 8, v 8 = 8 + The likelihood function in terms of (f, f 2, ) : L(F ) = f 2 (f 3 +f 4 +f 5 +f 6 +f 7 +f 8 )f 3 (f 4 +f 5 +f 6 +f 7 +f 8 )f 4 f 5 f6 2 f 8, where f + f 2 + f 3 + f 4 + f 5 + f 6 + f 7 + f 8 =

8 8 Reparametrization tricks The discrete hazard function: h = pr(t = v ) and h j = pr(t = v j T > v j ), j > 2 For t [v j, v j+ ) S(t) = pr(t > t) = pr(t > v j ) = j ( h i ). i= For t = v j j f j = f(t) = pr(t = t) = h j ( h i ). i=

9 9 Reparametrization tricks The likelihood function in terms of (h, h 2, ) : L(F ) =h 2 {( h )( h 2 )} {( h )( h 2 )h 3 } {( h )( h 2 )( h 3 )} {( h )( h 2 )( h 3 )h 4 } {( h )( h 2 )( h 3 )( h 4 )h 5 } {( h )( h 2 )( h 3 )( h 4 )( h 5 )h 6 } 2 {( h )( h 2 )( h 3 )( h 4 )( h 5 )( h 6 )( h 7 )} =h 2 ( h ) 8 ( h 2 ) 8 h 3 ( h 3 ) 6 h 4 ( h 4 ) 4 h 5 ( h 5 ) 3 h 2 6( h 6 ) ( h 7 )

10 0 KM estimation The likelihood function L(F ) = j h d j j ( h j) Y (v j) d j where n d j = δ i I(U i = v j ) = #failures at v j Y (v j ) = i= n I(U i v j ) = # at risk at v j. i=

11 KM estimation ĥj = d j /Y (v j ) Ŝ(t) = t < v j i= ( ĥi), v j t < v j+ which is the Kaplan-Meier Estimator.

12 2 Example v j Y (v j ) d j ĥ j Ŝ(v j ) = j ( ĥi) = ˆP (T > v j ) i= / /7.69 (= ) 7 5 /5.55 (= ) 9 4 /4.4 (= ) /3.4 (=.4 3 )

13 3 KM estimation Suppose that v g denotes the largest v j for which Y (v j ) > 0.. if d g = Y (v j ), then Ŝ(t) = 0 for t v g 2. if d g < Y (v j ), then Ŝ(t) > 0 but not defined for t > v g. (Not identifiable beyond v g.) The survival distribution may not be estimable with right-censored data. Implicit extrapolation is sometimes used. The KM estimator can also be used to estimate the survival function for the censoring distribution.

14 4 KM estimation KM estimator is a special MLE Ŝ(t) = argmax F L(F ) where F is the CDF for all discrete random variables (nonparametric MLE).

15 5 Self-Consistency No censoring Ŝ(t) = n n i= I(T i > t) Right censoring: Ŝ(t) = n n i= E(I(T i > t) U i, δ i ). E(I(T i > t) U i, δ i = ) = I(U i > t) 2. E(I(T i > t) U i, δ i = 0) = S(t)/S(U i )I(t U i ) + I(U i > t) Self-consistency iteration: { Ŝ new (t) = n n i= The solution is still the KM estimator. I(U i > t) + ( δ i ) Ŝold(t) Ŝ old (U i ) I(U i t) }

16 6 Redistribution of Mass Step Step Step ( 8 70 ) 5 ( 8 70 ) 5 ( 8 70 ) 5 ( 8 70 ) 5 ( 8 70 ) Total Mass }{{}}{{}}{{} Assume this is somewhere > 8

17 7 Nelson-Aalen Estimator How to estimate the cumulative hazard function? Ĥ(t) = j ĥ i for v j t < v j+ i= H(t) = log{s(t)} log{ŝ(t)} = for v j t < v j+. j { log( ĥi)} i= j ĥ i = Ĥ(t) i=

18 8 Asymptotical properties of KM estimator As n, Ŝ(t) S(t) in probability. As n, n /2 {Ŝ(t) S(t)} converges to N(0, σ2 (t)) in distribution.

19 9 Asymptotical properties of KM estimator How to estimate the variance of Ŝ(t) ĥi is an estimated probability. The variance of ĥi can be approximated by ĥ i ( ĥi) Y (v i ) = d i(y (v i ) d i ) Y (v i ) 3 ĥi and ĥj are asymptotically independent (why?).

20 20 Asymptotical variance of KM estimator For v j t < v j+ (lnŝ(t) ) : Var j Var ( ) ln( ĥi) i= δ method = j ) Var (ĥi i= ( ĥi) 2 = j i= d i Y (v i )(Y (v i ) d i ).

21 2 Asymptotical variance of KM estimator Using the δ-method ) Var (Ŝ(t) (lnŝ(t) ) ) 2 Var (e lnŝ(t) (lnŝ(t) ) = Ŝ(t)2 Var Ŝ(t)2 j = ˆσ 2 (t) i= d i Y (v i )(Y (v i ) d i ) (v j t < v j+ ) Greenwood s formula

22 22 By-Product The by-product of the Greenwood s formula is the variance estimator for Nelson-Aalen Estimator: var(ĥ(t)) = j i= d i Y (v i )(Y (v i ) d i ), v j t < v j+

23 23 Confidence Interval Ŝ(t) ±.96ˆσ(t), drawbacks? By δ-method var(log( log(ŝ(t)))) = ˆσ 2 (t) (log(ŝ(t)))2ŝ(t)2 The confidence interval for Ŝ(t) [exp{ e log( log(ŝ(t))).96σ(t) ˆ log(ŝ(t))ŝ(t) }, exp{ e log( log(ŝ(t)))+.96ˆσ(t) log(ŝ(t))ŝ(t) } ]

24 24 Median survival time How to estimate the median survival time Solving Ŝ(ˆt M ) = /2, not always solvable! How to construct the CI for the median survival time? Suppose that. pr(ŝl(t) < S(t)) = pr(ŝu(t) > S(t)) = Ŝ L (ˆt ML ) = Ŝ U (ˆt MU ) = The confidence interval for t M is [ˆt ML, ˆt MU ].

25 25 CI for median survival time

26 26 Median survival time = pr(ŝl(t M ) < S(t M )) = pr(ŝl(t M ) < 0.5) = pr(ŝl(t M ) < ŜL(ˆt ML )) = pr(t M ˆt ML ) = pr(ŝu(t M ) > S(t M )) = pr(ŝu(t M ) > 0.5) = pr(ŝu(t M ) > ŜU(ˆt MU )) = pr(t M ˆt MU )

27 27 Restricted mean survival time The area under the survival curve is a nice summary for the curve The AUC µ = τ 0 S(t)dt : = τs(τ) + τ 0 µ = ts(t) tf(t)dt = µ can be estimated as τ 0 0 τ 0 + τ 0 tf(t)dt min(t, τ)f(t)dt = E{min(t, τ)} Ŝ(t)dt.

28 28 Restricted mean survival time The restricted mean survival time E{min(T, τ)} can also be estimated as ˆµ IP W = n n i= δ i + ( δ i )I(U i τ) T i τ, Ŝ C (T i τ) where ŜC( ) is a consistent estimator of the survival function of the censoring time C. (How?) Rational [ ] I(C i τ T i ) E T i τ T i Ŝ C (T i τ) (T i τ) P (C i τ T i T i ) S C (T i τ) = T i τ This type of estimator is called the inverse probability weighting estimator

29 29 Restricted mean survival time ˆµ and ˆµ IP W are equivalent! The variance of the estimated area under the survival curve is complicated (the derivation will be given later). n τ 0 { τ t S(u)du} 2 h(t)dt. P (U t)

30 30 Area under the KM curve

31 3 Model Checking Under the exponential distribution: log(s(t)) = λt Plot t vs. pattern. log(ŝ(t)) to visually check the expected linear Under the Weibull distribution log( log(s(t))) = p log(λ) + p log(t) Plot log(t) vs. log( log(ŝ(t))) to visually check the expected linear pattern.

32 32 Model Checking log( log(s(t))) log(t)

33 33 Coming Soon The distribution of Ŝ(t) as a process.