Elaboration d un score au départ d une base de données. Ch. Mélot, MD, PhD, MSciBiostat Service des Soins Intensifs Hôpital Universitaire Erasme

Transcription

1 Elaboration d un score au départ d une base de données Ch. Mélot, MD, PhD, MSciBiostat Service des Soins Intensifs Hôpital Universitaire Erasme ESP, le 20 février 2006

2 METHODOLOGY

3 Scientific experiment Biological Hypothesis Clinical trial or Experiment Collection of data Statistical analysis Proof of the hypothesis Data fishing,, data dredging,, data snooping Collection of data Search for statistical significance Building of hypotheses

4 Data mining,, data dredging Data mining, also known as knowledge-discovery in databases (KDD) has been defined as The nontrivial extraction of implicit, previously unknown, and potentially useful information from data and The science of extracting useful information from large data sets or databases. Used in the technical context of data warehousing and analysis, the term data mining is neutral. Data dredging (data fishing, data snooping) is the term used to refer to the inappropriate (sometimes deliberately so) search for statistically significant relationships in large quantities of data. A key point is that one should not formulate a hypothesis as a result of seeing the data, at least not, if the data are then used as proof of the hypothesis.

5 Data mining,, data dredging If you want to work from data to hypotheses while avoiding the problems of data dredging, you need to collect a data set, then partition it into two subsets, A and B, with data items randomly placed in the two subsets. One subset - say, subset B - is examined for interesting hypotheses. Once a hypothesis has been formulated by examining subset B, the hypothesis can be tested on subset A, since subset A was not used to construct the hypothesis. Only where such a hypothesis is also supported by subset A is it reasonable to believe that the hypothesis might be valid.

6 MODELS

7 MULTIVARIABLE REGRESSION

8 MULTIVARIABLE REGRESSION If y = continuous variable: multiple regression y = = o + 1 x x x 3 If y = dichotomus variable: multivariable logistic régression y = e o + 1 x x x e o + 1 x x x 3

9 MULTIVARIABLE REGRESSION If y = count of events during a given period of time : multivariable Poisson regression y = e o + 1 x x x 3 If y = time to event: multivariable Cox regression y = h 0 (t) e o + 1 x x x 3

10 Expression of the results If y = continuous variable: multiple regression y = = o + 1 x x x 3 1 = «slope» for the risk factor x 1

11 Expression of the results If y = dichotomus variable: multivariable logistic regression Logit(y) = o + 1 x x x 3 1 e = odds ratio for the risk factor x 1

12 Expression of the results If y = count of events during a given period of time (t( i ) : multivariable Poisson s regression Ln(y/t i ) = o + 1 x x x 3 e 1 = relative risk of the occurrence of the event during the period of time

13 Expression of the results If y = time to event: multivariable Cox s regression Ln(y/h 0 (t)) = = o + 1 x x x 3 1 e = hazard ratio for the risk factor x 1

14 MULTIVARIATE REGRESSION

15 MULTIVARIATE REGRESSION y 1 0j 1j 2j 3j 1 y 2 y 3 = 0j 1j 2j 0j 1j 2j 3j 3j x x 1 x 2 x 3

16 2 1.5 MULTIVARIATE ANALYSIS Belgium-Luxembourg PROPOFOL Italy Germany Austria SUFENTANIL Finland Denmark Holland Portugal Sweden UK Ireland Switzerland Spain MORPHINE Norway -1 France FENTANYL MIDAZOLAM Soliman H.M., Mélot C., et al. Br. J. Anaesth. 2001;87:

17 THE MULTICOLLINEARITY PROBLEM Presence of multicollinearity is suggested when: the parameter estimates and associated t-t and p-values are highly unstable when variables are added in the model (lack of precision of the parameter estimates) selectivity bias is present (i.e., parameter estimate for a variable will be different depending on the order of entry in the model) examination of the correlation matrix reveals a high degree of correlation between some variables and a number of significant correlation coefficients (r > 0.8, i.e. r² > 0.64)

18 THE MULTICOLLINEARITY PROBLEM Determinant of the correlation matrix: lies in the interval [0, 1] equals zero if perfect multicollinearity is present, and unity if there is no multicollinearity. Bartlett s transformation converts this determinant into a c² statistics which tests: Ho: Det = 1 2 T with 1 2K 5 ln Det 6 K(K-1)/2 degrees of freedom T = number of observations K = number of variables

19 THE MULTICOLLINEARITY PROBLEM What can be done in the presence of multicollinearity? Supplementation of the original data with more information (additional observations) Combination of variables in a single variable Reduction of the dimensionality of the data: deletion from the equation of variables whose parameter estimates are affected by multicollinearity (loose of relevant predictors -> dangerous procedure) ad hoc statistical procedure: principal components analysis

20 AN EXAMPLE OF CORRELATION MATRIX r AGE SEX Height Weight BMP FEV1 RV FRC TLC AGE 1 SEX Height Weight BMP FEV RV FRC TLC r > 0.8 r² > 0.64

21 HOW TO BUILD AND VALIDATE A CLINICAL SCORE?

22 AN EXAMPLE QUESTION: How can we build an infection score usable in the intensive care setting? How can we validate the new score?

23 BUILDING STRATEGY 1st STEP: Criteria to define infection Garner JS, Jarvis WR, Emori TG, Horan TC, Hughes JM CDC definitions for nosocomial infections, 1988 Am J Infect Control 1988;16:

24 BUILDING STRATEGY 2nd STEP: Putative predictors easily collected in the intensive care setting CRP, WBC, RR, HR, TEMP SOFA, APACHE II, DayMV, Day HD

25 BUILDING STRATEGY 3rd STEP: Collecting the data and creating the database (353 patients) Continuous variables: AGE, CRP, WBC, RR, HR, TEMP, DayMV, DayHD, SOFA, APACHE II,... Discrete variables: GENDER, INFECTION,...

26 BUILDING STRATEGY 4th STEP: Searching for predictors of infection (1 = yes, 0 = no) Simple logistic regression

27 SIMPLE LOGISTIC REGRESSION Infect 1 e e CRP CRP Predictor Constant Variable p CRP, mg/100 ml WBC, count/mm³ RR, cycle/min HR, beat/min TEMP, C SOFA, points APACHE II, points AGE,yrs DayMV, days DayHD, days

28 LOGISTIC REGRESSION PROPORTION OF INFECTION = (x) e 1 + e CRP CRP CRP, mg/100 ml

29 LOGISTIC TRANSFORMATION (x) = e 0 x 1 + e 0 x (x) Logit [ (x) ] = ln [ ] 1 - (x)

30 LOGISTIC REGRESSION LOGIT Prop. of INFECTION Logit [ (x)] = CRP CRP, mg/100ml

31 BUILDING STRATEGY 5th STEP: Defining cut-off points for the continuous variables LOWESS Smoothing Cleveland WS Robust Locally Weighted Regression and Smoothing Scatterplots J Am Stat Assoc 1979;74:

32 LOWESS Smoothing A locally weighted regression method for smoothing bivariate scattergrams. Its tension parameter indicates what percentage of the dataset s values should be included in each window for the smoothing: A higher number produces a tighter smooth (with less response to local variances) A lower number produces a looser smooth (that is more strongly influenced by local variances) Cleveland WS. Robust Locally Weighted Regression and Smoothing Scatterplots. J Am Stat Assoc 1979;74:

33 LOWESS SMOOTHING with SCATTERPLOT Tension = 66 Temperature > Infection TEMPERATURE, C

34 LOWESS SMOOTHING with SCATTERPLOT Tension = 66 WBC < 5, ,000-12,000 2 > 12,000 3 Infection ,000 12, WBC, count/mm³

35 6th STEP: BUILDING STRATEGY Creating iso-weighted dummy variables for each cut-offs values for each variable

36 CREATING DUMMY VARIABLES WBC WBC_1 WBC_2 WBC < 5, WBC 5,000-12, WBC > 12, Nomber of levels - 1 = Nomber of dummy variables

37 BUILDING STRATEGY 7th STEP: Multiple logistic regression with all variables set to 1 or 0 Each coefficient given by the logistic regression is a measure of the relative weight of the level of the variable while controlling for all other variables retained in the model

38 MULTIPLE LOGISTIC REGRESSION Infect = e TEMP_ CRP_ WBC_ WBC_ HR_ HR_ RR_ SOFA_1 1 + e TEMP_ CRP_ WBC_ WBC_ HR_ HR_ RR_ SOFA_1

39 MULTIPLE LOGISTIC REGRESSION Dummy variables Cut-offs Coefficient SE p CRP_1 > 6 mg/100ml WBC_1 < 5,000/mm³ WBC_2 > 12,000/mm³ RR_1 > 25 c/min HR_ b/min HR_2 > 140 b/min TEMP_1 > 37.5 C SOFA_1 > 5 points

40 BUILDING STRATEGY 8th STEP: Creating the INFECTION PROBABILITY SCORE Each coefficient given by the logistic regression is transformed in a natural number.

41 CREATING THE SCORE Logistic LEVEL OF CERTAINTY OF INFECTION regression coefficients TEMP CRP WBC HR RR SOFA Rounded coefficients POINTS x x /

42 INFECTION PROBABILITY SCORE (IPS) POINTS TEMP C 37.5 > 37.5 CRP mg/100ml 6 > 6 WBC cells/mm³ 5,000-12,000 > 12,000 < 5,000 HR beats/min > 140 RR cycles/min 25 > 25 SOFA points 5 > 5 IPS varies from 0 to 26

43 BUILDING STRATEGY 9th STEP: Performance of the Score: Discrimination: correct prediction by the score Calibration/reliability

44 DISCRIMINATION: ROC CURVE ON THE BUILDING DATA SET (353 patients) 100 Sensitivity (IPS > 14) SENSITIVITY = 73.6 % SPECIFICITY = 77.9 % 20 AREA = % CI: Specificity PREVALENCE OF INFECTION: 91 / 353 = 25.8 % POSITIVE PREDICTIVE VALUE (IPS > 14) = 53.6 % NEGATIVE PREDICTIVE VALUE (IPS 14) = 89.5 %

45 CALIBRATION Infected patients n Range Prob Mean Prob Expected number Observed number Observed event rate Expected event rate C stat Hosmer-Lemeshow test C= 9.4 df 8, p 0.308

46 CALIBRATION OBserved event rate Hosmer Lemeshow test (C = 9.4, p = 0.308) Expected event rate

47 PREDICTIVE VALUES OF IPS PREDICTIVE VALUE (%) PREVALENCE (%) Positive PV Negative PV

48 VALIDATION SET 140 patients

49 ROC CURVE ON THE VALIDATION DATA SET (140 patients) 100 Sensitivity (IPS > 13) AREA = % CI: SENSITIVITY = 90.2 % SPECIFICITY = 76.8 % Specificity PREVALENCE OF INFECTION: 41 / 140 = 29.3 % POSITIVE PREDICTIVE VALUE (IPS > 15) = 61.7 % NEGATIVE PREDICTIVE VALUE (IPS 15) = 95.0 %

50 Relationship between discrimination and calibration.

51 Diamond GA, J Clin Epidemiol 1992;45:85-89

52 Diamond GA, J Clin Epidemiol 1992;45:85-89