Survival Analysis. DZHW Summer School Workshop Johannes Giesecke HU Berlin

Transcription

1 Survival Analysis DZHW Summer School Workshop Johannes Giesecke HU Berlin

2 Program Block I Basics a) Key concepts (censoring, truncation, analysis time and functions thereof, overview of potential models) b) Data structure and data management (wide and long data format, handling of dates, st-commands in Stata) Block II Some Examples using Different Models a) Non-parametric approaches b) Semi-parametric models c) Parametric models d) Outlook Survival Analysis J. Giesecke 2

3 Literature Allison, Paul D. (2014). Event History and Survival Analysis. Thousand Oaks: Sage Publications Blossfeld, Hans-Peter/Golsch, Katrin/Rohwer, Götz (2007). Event History Analysis with Stata. Mahwah: Lawrence Erlbaum Associates Box-Steffensmeier, Janet M./Jones, Bradford S. (2004). Event History Modeling. A Guide for Social Scientists. Cambridge: Cambridge University Press Cleves, Mario/Gould, William/Gutierrez, Roberto G./Marchenko Yulia V. (2010). An Introduction to Survival Analysis Using Stata. College Station: Stata Press Hosmer, David W./Lemeshow, Stanley (1999). Applied Survival Analysis. Regression Modeling of Time to Event Data. New York: John Wiley & Sons Kleinbaum, David G./Klein, Mitchel (2011). Survival Analysis: A Self- Learning Text. Berlin: Springer Verlag Survival Analysis J. Giesecke 3

4 Basics

5 Introduction aim: analysis of time until an (predefined) event occurs analysis of distribution of events investigate groups differences in time-to-event examples: job search duration, employment duration, entry into first job with permanent contract, time until first substantial pay raise etc. synonyms: hazard models, event history analysis, models for transition data, time-to-event data, survival-time data, duration data, failure time Survival Analysis J. Giesecke 5

6 Introduction data soures cross-sectional surveys collecting retrospective information (e.g. German Life History Study) panel studies collecting prospective information (e.g. German Socioeconomic Panel SOEP) administrative data (e.g. employment information collected by Federal Emplyoment Agency) Survival Analysis J. Giesecke 6

7 Alternative Models? Why not using OLS-models (dependent variable: time until an event occurs)? general problem: How to deal with censored cases? (but: censored regression models) more important: non-normally distributed dependent variable and time-varying variables Why not using logit/probit-models (dependent variable: event occurred vs. event did not occur)? loss of information (chronology is lost) time-varying variables cannot be integrated Survival Analysis J. Giesecke 7

8 Censoring and Truncation censoring: event occurs, but observational unit is not observed right censoring: event occurs after observational period has ended (observational unit no longer observed) left censoring: event occurred before observational period started (observational unit not yet observed) interval censoring: event occurred between start and end of the observational period, but no exact timing of event is known, only time interval (observational unit was not observed during time interval) Survival Analysis J. Giesecke 8

9 start of observational period end of observational period X left censoring gap interval censoring X in gap right censoring Survival Analysis J. Giesecke 9

10 Censoring and Truncation truncation: observational unit is not observed for a certain time span left truncation (delayed entry): observational unit was at risk before it was actually observed for the first time right truncation: observational unit UE is no longer observed after the end of the observational period, but event will finally occur; effectively not distinguishable from right censoring interval truncation (gaps): observational unit is not observed for a certain time span between start and end of the observational period Survival Analysis J. Giesecke 10

11 start of observational period end of observational period X left truncation gap X interval truncation right truncation Survival Analysis J. Giesecke 11

12 Continuous vs. Discrete Time continuous time: idea: continuous time axis, no time intervals but time points (infinitely small time intervals) basis of (almost) all textbooks on survival analysis discontinuous (discrete) time grouping of originally continuous time (grouping happens during or after observation phase) often neglected in textbooks, but quite important in practice Survival Analysis J. Giesecke 12

13 Types of Variables time-constant variables (e.g. sex, social origin, graduation grade) time-varying variables (e.g. number of job applications, type of search strategy) splitting of episodes might be necessary Survival Analysis J. Giesecke 13

14 Analysis Time and Functions thereof T non-negative random variable denoting the time until an event occurs 1. continuous time a) F(t)=Pr(T t) cumulative distribution function of T - probability that there is an event prior to t - monotone, nondecreasing function - also known as failure function Survival Analysis J. Giesecke 14

15 Cumulative Distribution Function F(t) F(1.517)=0.9 F(0.833)= t Weibull Distribution p=2 Survival Analysis J. Giesecke 15

16 Analysis Time and Functions thereof b) S(t)=1-F(t)= Pr(T>t) survivor function - probability that there is no event prior to t - monotone, nonincreasing function - S(0)=1, S(t)=0 for t Survival Analysis J. Giesecke 16

17 Survivor Function S(t) S(0.833)=0.5 S(1.517)= t Weibull Distribution p=2 Survival Analysis J. Giesecke 17

18 Analysis Time and Functions thereof c) f(t) density function - measure of concentration of events at time t - density function, no probabilities - note: 0 f(t), but not f(t) 1 Pr t T tt F() t S() t f() t lim t 0 t t t Survival Analysis J. Giesecke 18

19 Probability Density Function f(t) t Weibull Distribution p=2 Survival Analysis J. Giesecke 19

20 ht ( ) lim Analysis Time and Functions d) h(t) hazard function t0 thereof - measure of concentration of events at time t, conditional upon the subject having survived until t Pr t T tt T t f () t t S() t also known as: hazard rate, intensity rate, failure rate, transition rate, transition intensity, risk function, mortality rate therefore: given a certain level of f(t): the smaller S(t) the higher h(t) given a certain level of S(t): the higher f(t) the higher h(t) Survival Analysis J. Giesecke 20

21 Analysis Time and Functions thereof characteristics of hazard function: - values between 0 (no risk) to (certainty of event occurring at this particular instance) - time-constant hazard: conditional upon the event not having occurred until t, the chances of surviving at this instance or that are all the same - increasing hazard: increasing risk - falling hazard: falling risk Survival Analysis J. Giesecke 21

22 Hazard Function h(t) f (0.209) 0.4 S(0.209) h(0.209) f (1.393) 0.4 S(1.393) h(1.393) t Weibull Distribution p=2 Survival Analysis J. Giesecke 22

23 Analysis Time and Functions thereof e) H(t) cumulative hazard function - measure of total risk a subject has passed through until time t t Ht () hudu ( ) ln St () 0 Survival Analysis J. Giesecke 23

24 H(t) Cumulative Hazard Function t Weibull Distribution p=2 Survival Analysis J. Giesecke 24

25 Analysis Time and Functions thereof if one particular function is known, all other functions can be determined e.g. if ht ( ) t 0 1 then: Ht () hudu ( ) t St ( ) exp Ht ( ) exp( t) Ft ( ) 1 St ( ) 1exp Ht ( ) 1exp( t) f () t h() t S() t h()exp t H() t exp( t) Survival Analysis J. Giesecke 25

26 Hazard Function h(t) 0 2 ht ()= t Weibull Distribution p=1 Survival Analysis J. Giesecke 26

27 Cumulative Hazard Function H(t) t Ht () htdu () t t Weibull Distribution p=1 Survival Analysis J. Giesecke 27

28 Survivor Function S(t) St () exp Ht ()=exp( t) t Weibull Distribution p=1 Survival Analysis J. Giesecke 28

29 Cumulative Distribution Function F(t) F( t) 1exp H( t) =1 exp( t) t Weibull Distribution p=1 Survival Analysis J. Giesecke 29

30 Probability Density Function f(t) f () t h()exp t H() t exp( t) t Weibull Distribution p=1 Survival Analysis J. Giesecke 30

31 Analysis Time and Functions thereof 2. discontinuous time idea: T still is continuous random variable, but time axis is divided into k disjoint intervals [0=a 0,a 1 ], (a 1,a 2 ], (a 2,a 3 ],, (a k-1,a k = ] interval j: (a j-1,a j ] ] including ( excluding Survival Analysis J. Giesecke 31

32 Analysis Time and Functions thereof F(.) und S(.) are still defined as F(a j )=Pr(T a j ) - probability that an event occurs up to the end of time interval j S(a j )=1-F(a j )= Pr(T>a j ) - probability that no event occurs up to the end of time interval j (i.e., that subject survives at least until interval j+1) Survival Analysis J. Giesecke 32

33 Analysis Time and Functions thereof f(.) density function of T, here: probability function f( j) Pr a T a S( j 1) S( j) j1 j probability of the event occurring in time interval j Survival Analysis J. Giesecke 33

34 Analysis Time and Functions thereof interval hazard defined as: ha ( ) Pr a Ta Ta j j1 j j1 j1 Pr aj 1 T aj Sa ( j1) Sa ( j) Sa ( j) 1 Pr T a Sa ( ) Sa ( ) j1 j1 probability of the event occuring in time interval j, given that subject has survived to the beginning of that interval conditional probability, e.g. 0 h(a j ) 1 Survival Analysis J. Giesecke 34

35 Survivor Function S(t) t Weibull Distribution p=1 Survival Analysis J. Giesecke 35

36 Survivor Function S(t) S( j 1) 0.05 S( j 2) S( j 3) h( j 1) 0.95 f( j 1) 0.95 h( j 2) 0.95 f( j 2) h( j 2) 0.95 f( j 2) t Weibull Distribution p=1 Survival Analysis J. Giesecke 36

37 Survivor Function S(t) t Weibull Distribution p=0.5 Survival Analysis J. Giesecke 37

38 Survivor Function S(t) S( j 1) S( j 2) S( j 3) h( j 1) 0.82 f( j 1) 0.82 h( j 2) 0.51 f( j 2) 0.09 h( j 2) 0.42 f( j 2) t Weibull Distribution p=0.5 Survival Analysis J. Giesecke 38

39 Continuous vs. Discrete Time empirically, every measurement of time is discontinuous decision guidance a) ratio of units of time intervals (days, months, years) to typical length of episodes (e.g. median, mean) - the smaller this ratio the more appropriate it is to use models for continuous time b) frequency of observational units with the same T (time until an event occurs) - the smaller the number of ties the more appropriate it is to use models for continuous time Survival Analysis J. Giesecke 39

40 Overview of Potential Models we distinguish: parametric, semi-parametric and nonparametric models for event history data parametric and semi-parametric models: (strong) a-prioriassumptions about distribution of T and about the effects of covariates - parametric models: a-priori-assumption about both distribution of T and effects of covariates - semi-parametric models: a-priori-assumption effects of covariates non-parametric models: no a-priori-assumptions about distribution of T and about the effects of covariates Survival Analysis J. Giesecke 40

41 Overview of Potential Models 1. parametric and semi-parametric models models have the general form: h() t g t, x β i 0 i a) parametric models: strong a-priori-assumptions (distribution of T, effects of covariates) e.g. so-called proportional hazard-models h() t h ()exp t x β i 0 0 i Exponential model, Weibull model, Gombertz model Survival Analysis J. Giesecke 41

42 Overview of Potential Models some popular parametric models continuous time Exponential model Weibull model Log-logistic model Log-nomal model Gombertz model Generalized Gamma model discontinuous time Logistic model Complementary log-log model Survival Analysis J. Giesecke 42

43 Overview of Potential Models b) semi-parametric models: a-priori-assumptions about effects of covariates, but not about distribution of T e.g. Piecewise Constant Exponential Model h() t i h h h exp x β t(0, ] exp x β t(, ] exp x β t(, ] 01 i 1 02 i 1 2 0K i K1 K Survival Analysis J. Giesecke 43

44 Overview of Potential Models popular semi-parametric models continuous time Piecewise Constant Exponential Model Cox Model discontinuous time Piecewise Constant Logistic Model Piecewise Constant Complementary log-log Model Survival Analysis J. Giesecke 44

45 Overview of Potential Models 2. non-parametric models: no a-priori-assumption about about effects of covariates or not about distribution of T continuous time Kaplan-Meier-Estimator Nelson-Aalen-Estimator discontinuous time Life Table Survival Analysis J. Giesecke 45

46 Data Structure and Data Management

47 Data Structure intended data format: for each observational unit information on begin and end, event or cencering as well as on potential covariates should be stored PID t 0 t 1 Event x 1 x k Survival Analysis J. Giesecke 47

48 Data Structure using this data format allows to record: a)right censoring PID t 0 t 1 Event x 1 x k Survival Analysis J. Giesecke 48

49 Data Structure using this data format allows to record: b)left truncation (delayed entry) PID t 0 t 1 Event x 1 x k Survival Analysis J. Giesecke 49

50 Data Structure using this data format allows to record: c) interval truncation PID t 0 t 1 Event x 1 x k Survival Analysis J. Giesecke 50

51 Data Structure using this data format allows to record: d)time-varying covariates PID t 0 t 1 Event x 1 x k Survival Analysis J. Giesecke 51

52 Data Structure using this data format allows to record: e)multiple events PID t 0 t 1 Event x 1 x k Survival Analysis J. Giesecke 52

53 Wide vs. Long Data Format wide format: one row per observational unit PID Begin End Event x 1 x k Survival Analysis J. Giesecke 53

54 Wide vs. Long Data Format wide format: one row per observational unit, even if there are multiple episodes PID Begin1 End1 Event1 x 11 x k1 Begin2 End2 Event2 x Survival Analysis J. Giesecke 54

55 Wide vs. Long Data Format long format: multiple rows for each oberservational unit possible PID Begin End Event x 1 x k Survival Analysis J. Giesecke 55

56 Wide vs. Long Data Format long format usually necessary for survival analysis (because, for example, of interval truncation, timevarying covariates, multiple events) but also: data management much easier than in wide format switiching from long to wide format (and vice versa) is easy in Stata (command reshape) Survival Analysis J. Giesecke 56

57 Wide vs. Long Data Format. use "$home\suf97_sample.dta", clear. keep id_suf- job9best einmon einjahr. @best, i( id_suf) j(spell_nr) string (note: j = job1 job2 job3 job4 job5 job6 job7 job8 job9) Data wide -> long Number of obs > Number of variables 92 -> 21 j variable (9 values) -> spell_nr xij variables: job1manf job2manf... job9manf -> manf job1janf job2janf... job9janf -> janf job1mend job2mend... job9mend -> mend job1jend job2jend... job9jend -> jend job1lnoc job2lnoc... job9lnoc -> lnoc job1arve job2arve... job9arve -> arve job1az job2az... job9az -> az job1std job2std... job9std -> std job1best job2best... job9best -> best Survival Analysis J. Giesecke 57

58 Handling of Dates begin and end of an episode should be stored in dates format (better interaction with Stata s stcommands) conversion of existing date information into dates format: - string-to-numeric-conversion functions, e.g via gen datumsvar=date(stringvar, DMY ) - date-from-numerical-components functions, e.g. gen datumsvar=mdy(m,d,y) - see help datetime bzw. [D] datetime for more information Survival Analysis J. Giesecke 58

59 Handling of Dates. tab1 manf janf -> tabulation of manf manf Freq. Percent Cum filter 17, k.a Jan Feb März Apr Mai Juni Juli Aug Sep Okt Nov Dez Total 25, Survival Analysis J. Giesecke 59

60 Handling of Dates janf Freq. Percent Cum Filter 17, k. A , , , , Total 25, Survival Analysis J. Giesecke 60

61 Handling of Dates gen begin=ym(janf,manf) format beginn %tm gen end=ym(jend,mend) format ende %tm. list id_suf spell_nr manf janf mend jend begin end in 1/30, noobs id_suf spell_nr manf janf mend jend beginn ende Okt Sep m m Okt Juni m m Juli filter -2. Filter 2000m filter -2. Filter -2. filter -2. Filter filter -2. Filter -2. filter -2. Filter filter -2. Filter -2. filter -2. Filter filter -2. Filter -2. filter -2. Filter filter -2. Filter -2. filter -2. Filter filter -2. Filter -2. filter -2. Filter Okt Apr m m Okt Juli m m filter -2. Filter -2. filter -2. Filter..... Survival Analysis J. Giesecke 61

62 st-commands in Stata st-commands in Stata are used to analyse survialtime data before using other st-commands, we have to stset the datat see help st bzw. [st] st for more information Survival Analysis J. Giesecke 62

63 st-commands in Stata stset-command has three aims definition of onset of risk, definition of event(s), definition of entry and exit of subjects various checks if statements/data cause problems statements are kept for subsequent st-commands, data does not need to be stset again Survival Analysis J. Giesecke 63

64 stset stset beginn, failure(firstjob=1) id(persnr) origin(time ende_studium) means analysis is beginn-ende_studium, onset of risk is at t=ende_studium definition of event: firstjob=1 no multiple events (single-failure data) multiple episodes possible (within persnr) Survival Analysis J. Giesecke 64

65 stset. stset beginn, failure(firstjob=1) id(id_suf) origin(time ende_studium) id: id_suf failure event: firstjob == 1 obs. time interval: (beginn[_n-1], beginn] exit on or before: failure t for analysis: (time-origin) origin: time ende_studium total obs event time missing (beginn>=.) PROBABLE ERROR 103 multiple records at same instant PROBABLE ERROR (beginn[_n-1]==beginn) 419 obs. end on or before enter() 4527 obs. begin on or after (first) failure obs. remaining, representing 2367 subjects 2367 failures in single failure-per-subject data total analysis time at risk, at risk from t = 0 earliest observed entry t = 0 last observed exit t = 74 Survival Analysis J. Giesecke 65

66 stset. list id_suf spell_nr end_study begin firstjob _t0 _t _d _st in 1/30, noobs id_suf spell_nr end_st~y begin firstjob _t0 _t _d _st job1 1997m6 1998m job2 1997m6 1999m job3 1997m6 2000m job4 1997m job5 1997m job6 1997m job7 1997m job8 1997m job9 1997m job1 1997m8 1994m job2 1997m8 1999m job3 1997m job4 1997m job5 1997m job6 1997m Survival Analysis J. Giesecke 66

67 stset after having stset the data, we can use stcommands, for example stdes: description of survival-time data stvary: information on variables Survival Analysis J. Giesecke 67

68 stdes. stdes failure _d: ereignis == 1 analysis time _t: (beginn-origin) origin: time ende_studium id: id_suf per subject Category total mean min median max no. of subjects 2367 no. of records (first) entry time (final) exit time subjects with gap 0 time on gap if gap time at risk failures Survival Analysis J. Giesecke 68

69 stvary. stvary geschl arve failure _d: ereignis == 1 analysis time _t: (beginn-origin) origin: time ende_studium id: id_suf subjects for whom the variable is never always sometimes variable constant varying missing missing missing geschl arve Survival Analysis J. Giesecke 69

70 Some Examples using Different Models

71 Non-parametric Approaches no a-priori-assumptions about distribution of T and about the effects of covariates serve mainly descriptive purposes estimation of survivor function: Kaplan-Meier-Estimator estimation of cumulative hazard function: Nelson-Aalen- Estimator for both estimators: illustration of group differences but also: statistical tests of group differences Survival Analysis J. Giesecke 71

72 Non-parametric Approaches. sts list failure _d: firstjob == 1 analysis time _t: (begin-origin) origin: time end_study id: id_suf Beg. Net Survivor Std. Time Total Fail Lost Function Error [95% Conf. Int.] Survival Analysis J. Giesecke 72

73 Kaplan-Meier survival estimate. sts graph, graphregion(color(white)) failure _d: firstjob == 1 analysis time _t: (begin-origin) origin: time end_study id: id_suf analysis time Survival Analysis J. Giesecke 73

74 Non-parametric Approaches Question: Does survivor function dffer between groups? 1. description: illustration of group differences 2. statistical tests of group differences Survival Analysis J. Giesecke 74

75 Non-parametric Approaches 1. description: illustration of group differences sts graph, by(group) all Stata graph twoway-options can be used see help sts graph and [ST] sts graph for more options Survival Analysis J. Giesecke 75

76 Kaplan-Meier survival estimates. sts graph, by(hsart) plot1opts(recast(line) lcolor(gs12)) graphregion(color(white)) analysis time hsart = 1. FH hsart = 2. UNI Survival Analysis J. Giesecke 76

77 Non-parametric Approaches 2. statistical tests of group differences basic idea: compare number of observed events with number of events to be expected unter H 0 H 0 :h 1 (t)=h 2 (t)= =h r (t) calculation of χ² distributed test statistic Survival Analysis J. Giesecke 77

78 Non-parametric Approaches Stata-Beispiel 7b: sts test. sts test hsart Log-rank test for equality of survivor functions Events Events hsart observed expected FH UNI Total chi2(1) = Pr>chi2 = Survival Analysis J. Giesecke 78

79 Non-parametric Approaches tests of group differences can be stratified to account for the fact that risk might also differ between other groups sts test varname, strata(varlist) reduces risk of confounding option detail to display test results for single strata Survival Analysis J. Giesecke 79

80 . replace geschl=. if geschl<1 (36 real changes made, 36 to missing). sts test hsart, strata(geschl) detail Stratified log-rank test for equality of survivor functions -> geschl = 1 Events Events hsart observed expected FH UNI Total chi2(1) = Pr>chi2 = > Total Events Events hsart observed expected(*) FH UNI Total (*) sum over calculations within geschl chi2(1) = Pr>chi2 = > geschl = 2 Events Events hsart observed expected FH UNI Total chi2(1) = 3.82 Pr>chi2 = Survival Analysis J. Giesecke 80

81 Semi-parametric Models a-priori-assumptions about effects of covariates, but not about distribution of T formal model: x expxβ ht h t i 0 i 0 ht h x t i hazard at time t, given x baseline hazard x β x x... x i 1i 1 2i 2 Ki K baseline hazard does not need to be specified i Survival Analysis J. Giesecke 81

82 Semi-parametric Models advantage: no assumption about h 0 (t), therefore less error-prone disadvantage: not efficient compared to (correctly specified) parametric model In Stata: stcox Survival Analysis J. Giesecke 82

83 . stcox ib2.hsart ib2.geschl failure _d: firstjob == 1 analysis time _t: (begin-origin) origin: time end_study id: id_suf Iteration 0: log likelihood = Iteration 1: log likelihood = Iteration 2: log likelihood = Iteration 3: log likelihood = Refining estimates: Iteration 0: log likelihood = Cox regression -- Breslow method for ties No. of subjects = 2,364 Number of obs = 2,387 No. of failures = 2,364 Time at risk = LR chi2(2) = Log likelihood = Prob > chi2 = _t Haz. Ratio Std. Err. z P> z [95% Conf. Interval] hsart 1. FH geschl 1. männlich Survival Analysis J. Giesecke 83

84 Parametric Models a-priori-assumptions about effects of covariates and about distribution of T fully parameterized model two variants of parametrization: proportional hazard modelle (PH-models) accelerated failure time-models (AFT-models) Survival Analysis J. Giesecke 84

85 Parametric Models PH-models formal model: htx h texp xβ i 0 i 0 ht h t x i hazard at time t, given baseline hazard, needs to be specified z.b. h 0 t exp 0 (Exponential Model) baseline hazard cannot be left unspecified x i Survival Analysis J. Giesecke 85

86 Parametric Models AFT-models formal model: ln t x β ln i i i x β x x... x i 1i 1 2i 2 Ki K follows a certain distribution e.g. Weibull, 0 p Survival Analysis J. Giesecke 86

87 . streg ib2.hsart ib2.geschl, distribution(gompertz) failure _d: firstjob == 1 analysis time _t: (begin-origin) origin: time end_study id: id_suf Fitting constant-only model: Iteration 0: log likelihood = Iteration 1: log likelihood = Gompertz regression -- log relative-hazard form No. of subjects = 2,364 Number of obs = 2,387 No. of failures = 2,364 Time at risk = LR chi2(2) = Log likelihood = Prob > chi2 = _t Haz. Ratio Std. Err. z P> z [95% Conf. Interval] hsart 1. FH geschl 1. männlich cons /gamma Survival Analysis J. Giesecke 87

88 Outlook some important things that we did not talk about interactions between covariates time-varying effects of covariates multiple events recurrent events competing risks diagnostics for (semi-)parametric models Survival Analysis J. Giesecke 88