Martin L. Lesser, PhD Biostatistics Unit Feinstein Institute for Medical Research North Shore-LIJ Health System

Transcription

1 PREP Course #13: Introduction to Exploratory Data Analysis and Data Transformations (Part 2) Martin L. Lesser, PhD Biostatistics Unit Feinstein Institute for Medical Research North Shore-LIJ Health System

2 CME Disclosure Statement The North Shore LIJ Health System adheres to the ACCME s new Standards for Commercial Support. Any individuals in a position to control the content of a CME activity, including faculty, planners, and managers, are required to disclose all financial relationships with commercial interests. All identified potential conflicts of interest are thoroughly vetted by the North Shore-LIJ for fair balance and scientific objectivity and to ensure appropriateness of patient care recommendations. Course Director and Course Planners, Kevin Tracey, MD, Cynthia Hahn, Emmelyn Kim, MPH, Tina Chuck, MPH have nothing to disclose. Martin L Lesser, PhD, EMT-CC have nothing to disclose

3 Why Transform* Data? 1. Classical Inference a. To achieve homoscedasticity (ANOVA, t-test do not work with unequal variances) b. To achieve normality c. To straighten out plots d. To conform to known physical laws 2. Exploratory Data Analysis (EDA) a. To symmetrize/normalize b. To explore data c. To compare distributions d. To linearize plots e. To create confusion (??) * EDAers use the work re-express 3

4 Variance Stabilization Initial Problem: To compare locations of several distributions However, this can be problematic when the spreads of the distributions are different. We would like spread to be (fairly) independent of location. We can use transformations to create distributions where location and spread are unrelated. 4

5 Variance as a Function of the Mean Suppose Var X = f (μ) and let Y dt f(t) Then Var X Var Y f (μ) = 1 * Bartlett, 1947 Snedecor and Cochran, p.325 5

6 Well Known Examples 1. Var X = cμ (e.g. Poisson) Y = x 2. Var X = cμ 2 (or, sd(x) = kμ) Y = log X 3. Var X = cμ (A μ) (e.g. binomial) Y = sin -1 x A * Such transformations render the variance independent of μ. 6

7 Investigating Heteroscedasticity Pictures are most informative (and easy to produce)! For Instance: Plot: s vs. Mean s 2 vs. Mean Median vs. IQR Example 5: Four studies of VPC frequency: (VPC rate is given as VPC Frequency / 100,000 Beats) VPC Rates By Study Study A B C D MEAN VAR SD

8 VPC Rates among 4 Studies Study A B C D MEAN VAR SD S 2 vs. X VAR S vs. X SD 8

9 LOG (VPC+.002) among 4 Studies Study A B C D MEAN VAR SD S 2 vs. X VAR S vs. X SD 9

10 Straightening X-Y Plots Regression and Correlation Transforming X and/or Y to yield a straight line relationship makes analysis simpler. 1. Interpolation is simpler 2. Interpretation is (usually) simpler 3. Departures from fit are more clearly detected Shapes of curves of form y = x P p > 1 p < 1 10

11 Which Transformation Straightens the Plot? (How to choose p) Recall Ladder of Powers: y = x P p =, -3, -2, -1, -1/2, (0), 1, 2, 3, The Bulging Rule y y up y up x down x up x down x up y down y down 11

12 Investigating the Bulge Some Plots show a bulge clearly: Y Y up X down e.g. Y = X 1/2 might work X 12

13 Investigating the Bulge (cont d) Some Plots don t show a bulge so clearly: Y X How do we find the bulge? use half-slopes 13

14 Half-Slopes 1. Divide X-values into (approx.) thirds. 2. For each third, compute median (X) and median (Y). X L X M X R (L=Left M=Middle R= Right) Y L Y M Y R 3. Compute Half-Slopes b L Y X M M - - Y X L L b R Y X R R - - Y X M M 4. Look at R b b L R If R 1 Then XY relationship is straight. If R 1 Then a transformation may help. 14

15 Example of Half-Slopes y L y M y R x L x M x R L M R 15

16 Interpretation and Reporting Although a transformation may help to analyze the data, it may be worth the difficulty to explain or understand. Some transformations are easier to understand than others (e.g. logx, 10 x, x 2, x,1/x ) Suppose we choose 4 x, how do we explain this? 1. We could invert the transform. e.g. f 1 f ( quantile (x) ) is ok. But f -1 f (mean (x)) mean f -1 f (x) e.g. x 1 x 2 n... x n x1... n x n Note: e l o gx n i (x, 1 x 2,...,x n 1 n ) geometricmean 16

17 Inversion: An Example Peak common Bile Duct Pressure Recally -10 peak seem edfairly norm al Usual 95% CI for median (y) is Median 1.96 Note: This assumes that MedianMean and S H - SPR EAD H - SPR EAD isestimateof σ CI = (1.96) (1.48) = (-10.08, ) Inverting: 100 peak 2 y CI for median (peak) = (.98, 5.46) 17

18 What about using nonparametric procedures and forgetting about transformations? 1. EDA is not as concerned with inference as it is with description 2. For inference, non-parametrics may be fine, but description is still needed 3. Efficiency questions 18

19 19

20 Scales in Music Transformations in Everyday Experience by David C Hoaglin Chance Vol.1, No. 4, 1988, Springer-Verlag, New York Octave: If note with frequency f2 is and octave above note f1, then f2 = 2*f1. Thus, in base 2 units: logf2 logf1 = log 2 2 = 1 octave. There are 12 notes in an octave, all equispaced, resulting in 12 intervals of size Richter Scale for Earthquakes Richter scale: Strength of an earthquake is expressed in log base 10 units. The seismic energy E (in ergs) released by an earthquake of magnitude M can be estimated as log E = M. Thus, an earthquake of magnitude 7 releases about 3 times as much energy of one of magnitude 5 (10 1.5*(7-5) = 1,000).

21 Decibels Sound is actually measured by pressure (dynes/square cm). An increase in 20 db corresponds to a tenfold increase in sound pressure. L p = 20 log 10 (p/0.0002) db, where is the internationally accepted minimum audible sound pressure at 1,000 Hz (i.e., dyne/cm 2, rms). Average Speed in Auto Races Measure speed in mph. Race officials, however, measure elapsed time. Thus, average speed is reciprocal of elapsed time * distance around track * number of laps completed.

22 Gasoline Consumption Usually expressed as miles per gallon. Sometimes (e.g., in consumer testing) the number of gallons per trip (i.e., the reciprocal) is given. ph ph = -log 10 [H + ] Lenses and Cameras f-stops on aperture setting: f= ratio of focal length of lens (L) to the diameter of the aperture (d). Suppose a camera has f-stops of f= 1.4, 2.0, 2.8, 4.0, 5.6, 8.0, 11.0, This is a geometric progression, where each term is sqrt(2) times the preceding one. So 1 f-stop change from 5.6 to 8.0 halves the area of the aperture. (Because A = π (d/2) 2 = π (L/(2f)) 2 ). The f-stop involves several transformations.

23 Field Position in Football The 100-yard football field is normally described in terms of how far the line of scrimmage is from the 50-yard line, rather than how far, in absolute yards, the line is from a fixed goal line. We don t say that the Giants are 80 yards from the goal line ; we say the Giants are on the Redskins 20 yard line. Time of Goal in Hockey The time clock in hockey starts at 20 minutes and counts down to 0 minutes so that one always knows how many minutes are left in the period. However, when a goal is scored (or a penalty is assessed), the official time is announced as how many minutes into the period the goal was scored. Example: Time of the goal is 5 minutes and 30 seconds into the second period. One would have to convert this time into 14 minutes and 30 seconds left in the second period.

24 Copper Wire Gauge The larger the gauge, the smaller the wire s diameter. Gauges run from 0000, 000,00, 0, 1, 2,, 56. By definition, d gauge = d 0 r gauge and d 0 can be shown to equal mils. Shotgun Size Shotgun size is usually decribed by its gauge, 10-gauge, 12-gauge, 20-gauge, for example. Gauge (which represents the bore diameter in millimeters) was originally derived as the number of lead balls with that same diameter that together weighed one pound. Example: a bore-sized lead ball for 12-gauge shotgun weighed 1/12 pound. Thus, a close approximation for gauge is that it is proportional to the reciprocal cube of the bore diameter. Gauge Bore Diameter