Why Data Transformation? Data Transformation. Homoscedasticity and Normality. Homoscedasticity and Normality

Transcription

1 Objectives: Data Transformation Understand why we often need to transform our data The three commonly used data transformation techniques Additive effects and multiplicative effects Application of data transformation in ANOVA and regression. Why Data Transformation? The assumptions of most parametric methods: Homoscedasticity Normality Additivity Linearity Data transformation is used to make your data conform to the assumptions of the statistical methods Illustrative examples Homoscedasticity and Normality Homoscedasticity and Normality The data deviates from both homoscedasticity and normality. Won t it be nice if we would make data look this way?

2 Types of Data Transformation The logarithmic transformation The square-root ttransformation ti The arcsine transformation. Data transformation can be done conveniently in EXCEL. Alternatives: Ranks and nonparametric methods. ID Group 1 Group Var t P Equal Var.? P= Kurtosis Skewness.1.1 P(Zg1) Homoscedasticity The two groups of data seem to differ greatly in means, but a t-test shows that the means do not differ significantly from each other - a surprising result. The two groups of data differ greatly in variance, and both deviate significantly from normality. These results invalidate the t-test. We calculate two ratios: var/mean ratio and Std/mean ratio (i.e., coefficient of variation). Group1 Group Var/mean C.V P(Zg) Log-transformation ID ID Group Group 1 1 Group Group Var t -3.4 P Equal Var.? P= 0.67 Kurtosis Skewness P(Zg1) P(Zg) Log-Transformed Data NewX = ln(x+1) The transformation is successful because: The variance is now similar Deviation from normality is now nonsignificant The t-test revealed a highly significant t 34 difference in means between the two groups ID Group 1 Group Var t -3.4 P Equal Var.? P= 0.67 Kurtosis Skewness P(Zg1) P(Zg) Log-Transformed Data NewX = ln(x+1) Transform back: X = e NewX 11 Compare this mean with the original i mean. Which one is more preferable? Calculate the standard error, the degree of freedom, and 95% CL (t 0.05,16 =.47).

3 Normal but Heteroscedastic Any transformation that you use is likely to change normality. Fortunately, t-test and ANOVA are quite robust for this kind of data. Of course, you can also use nonparametric tests. Normal but Heteroscedastic ID Group 1 Group The t-test, however, dt detects t significant ifi 4 13 difference in means. You can use nonparametric methods to analyse data for comparison, and you are like to find t-test to be more powerful. 13 Var t P Equal Var.? P= Kurtosis Skewness 0 0 The two variances are significantly different. Factor B Level 1 Level Level Level Additivity What experimental design is this? Compare the group means. Is there an interaction effect? Additivity i i means that the difference between levels of one factor is consistent for different levels of another factor. Multiplicative Effects Factor B Level 1 Level Level Level Compare the group means. Is there an interaction effect? Does this data set meet the assumption of additivity? When the assumption of additivity is not met, we have difficulty in interpreting main effects. Now calculate the ratio of group means. What did you find?

4 Multiplicative Effects Factor B Level 1 Level Level Level For, we see that Level has a mean about. times as large as that for Level 1. For factor B, Level has a mean about.1 times as large as that for Level 1). If you know the value for Level 1 of, you can obtain the value for Level of by multiplying the known value by.. Similarly, you can do the same for Factor B. We say that the effect of Factors A and B are multiplicative, not additive. Factor B Level 1 Level Level Level Log-transformation Now log-transform the data. Compare the means. Is the assumption of additivity met now? Original Data Transformed data Variance Why log-transformation can change the multiplicative li effects to additive effects? Z = XY ln( Z ) = ln( X) + ln( Y ) Square-Root Transformation ID Group 1 Group The two groups of data differ much in variance Calculate two ratios: var/mean ratio and Std/mean ratio (i.e., coefficient of variation) Does your calculation suggest logtransformation? When is log transformation appropriate? Use square-root transformation when different groups have similar il Variance/ ratios Var Notice the means, which do not Var/ coincide with the most frequent Std/ observations

5 Square-Root Transformation ID Group 1 Group Var Square-root transformation: X ' = X + 3/ The variance is now almost identical between the two groups Transform the means back to the original scale and compare these means with the original means: 3 X = ( X ') Quiz on Data Transformation Group n Var SE T LowerL The data set is rightskewed for each group. Calculate the variance/mean ratio and C.V. for each group, and decide what transformation you should use. Do the transformation and convert the means back to the original UpperL scale. With Multiple Groups Confidence Limits Variance When you have multiple groups, a Variance vs or a Std vs plot can help you to decide which data transformation to use. The graph on the left shows that the Var/ ratio is almost constant. What transformation should you use?, Lo ower, Uppe er Before transformation ower, Upp per, L After transformation With the skewness in our data, do confidence limits on the right make more sense? Why?

6 Arcsine Transformation Group1 Group Group1 Group p Var SE LowerL UpperL Transform back New LowerL UpperL X ' = arcsin( X ) Used for proportions Compare the variances before and after transformation Do you know how to transform the means SE and C.L. back to the original scale? X = (sin X ') Data Transformation Using SAS Data Mydata; input x; newx=log(x); newx=sqrt(x+3/); newx=arsin(sqrt(x)); cards; Natural logarithm transfromation Square-root transformation Arcsine transformation