Performance of AIC-Selected Spatial Covariance Structures for f MRI Data

Transcription

1 Brigham Young University BYU ScholarsArchive All Theses and Dissertations Performance of AIC-Selected Spatial Covariance Structures for f MRI Data David A. Stromberg Brigham Young University - Provo Follow this and additional works at: Part of the Statistics and Probability Commons BYU ScholarsArchive Citation Stromberg, David A., "Performance of AIC-Selected Spatial Covariance Structures for fmri Data" (2005). All Theses and Dissertations This Selected Project is brought to you for free and open access by BYU ScholarsArchive. It has been accepted for inclusion in All Theses and Dissertations by an authorized administrator of BYU ScholarsArchive. For more information, please contact scholarsarchive@byu.edu, ellen_amatangelo@byu.edu.

2 PERFORMANCE OF AIC-SELECTED SPATIAL COVARIANCE STRUCTURES FOR FMRI DATA by David A. Stromberg A Project submitted to the faculty of Brigham Young University in partial fulfillment of the requirements for the degree of Master of Science Department of Statistics Brigham Young University July 2005

3

4 BRIGHAM YOUNG UNIVERSITY GRADUATE COMMITTEE APPROVAL of a Project submitted by David A. Stromberg This Project has been read by each member of the following graduate committee and by majority vote has been found to be satisfactory. Date William F. Christensen, Chair Date Gilbert W. Fellingham Date G. Bruce Schaalje

5

6 BRIGHAM YOUNG UNIVERSITY As chair of the candidate s graduate committee, I have read the Project of David A. Stromberg in its final form and have found that (1) its format, citations, and bibliographical style are consistent and acceptable and fulfill university and department style requirements; (2) its illustrative materials including figures, tables, and charts are in place; and (3) the final manuscript is satisfactory to the graduate committee and is ready for submission to the university library. Date William F. Christensen Chair, Graduate Committee Accepted for the Department G. Bruce Schaalje Graduate Coordinator Accepted for the College G. Rex Bryce Associate Dean, College of Physical and Mathematical Sciences

7

8 ABSTRACT PERFORMANCE OF AIC-SELECTED SPATIAL COVARIANCE STRUCTURES FOR FMRI DATA David A. Stromberg Department of Statistics Master of Science FMRI datasets allow scientists to assess functionality of the brain by measuring the response of blood flow to a stimulus. Since the responses from neighboring locations within the brain are correlated, simple linear models that assume independence of measurements across locations are inadequate. Mixed models can be used to model the spatial correlation between observations, however selecting the correct covariance structure is difficult. Information criteria, such as AIC are often used to choose among covariance structures. Once the covariance structure is selected, significance tests can be used to determine if a region of interest within the brain is significantly active. Through the use of simulations, this project explores the performance of AIC in selecting the covariance structure. Type I error rates are presented for the fixed effects using the the AIC chosen covariance structure. Power of the fixed effects are also discussed.

9

10 Acknowledgements I want to express my thanks to Dr. William Christensen for the many hours of work and consultation he invested with me on this project which ultimately lead to the attainment of my Master s degree. I would also like to thank all those who contributed to the progression of the project. I want to thank my family for their constant support and reassurance. I would have never made it thus far without the care and guidance of a loving mother, and never succeeded in my Master s program without a loving, and supportive wife. Most of all I want to express my gratitude to my Heavenly Father for His abundant goodness in allowing me so many wonderful opportunities such as the privilege of studying and this university.

11

12 Contents Chapter 1 Introduction 1 2 Review of Literature fmri Spatial Statistics Spatial Modeling Stationarity Isotropy Linear Mixed Models Selection of Covariance Structure Type I Error Rates Methods Data Generation Modeling the Data xi

13 4 Results Performance of AIC Type I Error Rates Power Summary Data Analysis Covariance Structure Selection Test of Significance Further Considerations Independence of Responses Across Treatments Estimating Covariance Parameters from Stationary Responses Appendix A Code Appendix 38 A.1 SAS Code for Simulations A.2 S-Plus Code for Estimation and Graphing A.3 SAS Code for Tabulating Results A.4 SAS Code for Actual fmri Analysis Bibliography 63 xii

14 Tables Table 3.1 Selected Spatial Covariance Structures. d i,j is the distance between two points, and 1 (di,j =0) is an indicator function for distance zero Covariance Parameters used to Generate Data Percents of AIC-Selected Covariance Structures Rejection Rates Based on AIC-Selected Structure Rejection Rates for Power Analysis Based on AIC-Selected Structure Least-Squares means for treatment by region levels Covariance parameter estimates from SAS xiii

15 xiv

16 Figures Figure 2.1 Hemodynamic response curve for an active voxel when using the sfmri method Generated data voxel locations within a slice. The auditory cortex is the shaded region Semi-Variograms used to generate the simulated data. Dots represent the empirical Semi-Variogram Log-Eigenvalues of the Covariance Matrices xv

17 xvi

18 Chapter 1 Introduction Functional magnetic resonance imaging (fmri) allows scientists to explore functionality of the brain by evaluating hemodynamic activity or blood flow. A primary objective is to determine whether a region of the brain becomes active when stimulated. To accomplish this, periods of stimulus or tasks are alternated with periods without stimulus, and the hemodynamic activity is recorded during the two periods. Through this method it can be determined if a specific region has a different hemodynamic response during the stimulated state than in the non-stimulated, thus indicating whether the region is associated with the stimulus or task. One region of specific interest to neuroscientists is the auditory cortex, the region of the brain responsible for hearing. To examine the functionality of the auditory cortex, an audible tone stimulus of specific intensity and frequency is presented to the subject. For subjects whose hearing is not impaired, this tone stimulus should invoke a response in the auditory cortex that can be measured by a scanner. However, the neuronal response due to the stimulus is confounded 1

19 with the response due to the noise of the scanner that records the response. Silent fmri (sfmri) has been used to overcome this difficulty of confounded responses. In sfmri, responses are recorded while the stimulus is given and then again at a resting or non-stimulated state just like in the fmri, except that the scanner is turned on and a measurement quickly recorded before there is an increase in hemodynamic activity due to the scanner, thus avoiding confounding due to the noise of the scanner. The measurement is taken after the auditory cortex has responded to the stimulus but before it responds to the scanner noise. The response is recorded for several tiny brain locations called voxels. Due to the nature of blood flow in the brain, the responses obtained from neighboring voxels cannot be considered independent but are assumed to be spatially correlated. In order to correctly identify if a region of the brain (such as the auditory cortex) is statistically active, scientists must correctly account for this dependency in the data (Christensen and Yetkin 2005; Grady and Helms 1995). To correctly account for the spatial dependency of the data, a spatial covariance structure can be included in the model. This can be implemented in SAS R with the use of the MIXED procedure. However, SAS R offers many types of spatial covariance structures and the choice of the correct structure is not an easy one. Researchers often use some criterion such as AIC or SBC to select the covariance structure that best fits the data, however these criteria do not always select the correct structure. (Keselman et al. 1998; Ferron et al. 2002; Gomez et al. 2005) Once a covariance structure has been selected, the fixed effects can 2

20 then be tested. In the case of the fmri data, the region of the auditory cortex can be tested for significant activation. This project explores, through the use of simulations, the performance of AIC in selecting the correct spatial covariance structure. Also, Type I error rates are reported for testing a region for significant activation using the AIC-selected covariance structure. Finally an actual fmri data set is analyzed. A literature review is given in Chapter 2. The literature review discusses fmri data, fundamentals of spatial data analysis and the selection of covariance structures in a mixed model setting. Finally, repercussions of covariance selection on testing a fixed effect are discussed. Chapter 3 discusses the methods used for the simulations, including how the data are generated and analyzed. Chapter 4 presents the results of the simulations, as well as some general conclusions. An example analysis of an actual data set is covered in Chapter 5. Chapter 6 briefly explores some potential directions for future research. 3

21 Chapter 2 Review of Literature 2.1 fmri Magnetic Resonance Imaging (MRI) is a tool used to view structural images of the inside of the body. In an MRI a patient is placed in a strong magnetic field which is used to magnetize the tissue. Genovese (1998) states that, carefully modulating the magnetic field affects a specific type of atomic nucleus, such as hydrogen, in the patient s tissue and produces a measurable signal in a nearby receiver coil. The signal that is recorded by the receiver differs for differing densities of tissue, thus an MRI is a device that essentially measures the hydrogen content in a given volume element in the body. These volume elements, or voxels, can be placed together to create a high-resolution, three-dimensional image of the tissue volume inside the body. Thus, MRI s are very valuable for obtaining anatomical structure images of the inside of the body. In functional Magnetic Resonance Imaging (fmri), the technology of an MRI is applied not to obtain a structural image of the body, but in an attempt to create a functional image of the body. Instead of being interested in the actual 4

22 image, the researcher is interested in the activity of structures inside the body. An activity image can be obtained by exploring the subtle changes in the measured signal across time. These changes are caused by physiological effects related to neural activity (Genovese 1998). This neural activity may give indications of regions that are responsible for different physical or mental tasks or other stimuli. In an fmri experiment a patient is placed in the scanner and then given some sequence of tasks or stimuli while the images of the subject s brain are acquired at regular time intervals. The data set that results from this type of experiment is a time series of three dimensional images. Many of these experimental designs consist of two conditions: one in which the subject is given a stimulus or task and then a control or resting period. It is assumed that the control task involves all aspects of the stimulus task except for those stimuli being specifically investigated. An example of an fmri experiment would be playing an audible tone for a few seconds into the ear of the subject during the MRI scan and then turning the tone off. By repeating this process many times the scanner would produce images of the brain that could be associated with the time of the stimulus or control. This time-series image could then be used to classify voxels as active or inactive via a statistical hypothesis test (Genovese 1998; Lange 1996). One might use the t-test to compare the average signal levels in the two conditions. This classification technique provides a good map of which regions of the brain are more active when the stimulus is given. One major problem in assessing the effects of a audible tone given during 5

23 the fmri is that the scanner itself is very noisy. The neuronal response due to tone stimulus is confounded with the response to the noise of the scanner. The silent fmri (sfmri) technique is designed to decrease the problem of the scanner noise (Yang et al. 2000). When a stimulus is given, there is a brief lag between the neuronal activity and the resulting hemodynamic response; this lag is exploited in the sfmri (see Figure 2.1 (Christensen and Yetkin 2005)). The pure tone response is given and then the scanner is turned on. A measure of the hemodynamic activity is quickly recorded. Since the hemodynamic response due to the noise of the scanner will occur after a brief amount of time, the response recorded by the scanner immediately after turning the scanner on represents only the hemodynamic response due to the pure-tone stimulus. After a resting period in silence, the scanner is once again turned on and a measurement is quickly recorded before the brain has time to react to the noise of the scanner. This response represents the control response (Yetkin et al. 2003). The data obtained from the sfmri can be analyzed with a statistical model to determine if a region of interest is statistically active. Since the data represent a three dimensional map of activation the responses are probably spatially correlated and the use of spatial statistics is recommended. 2.2 Spatial Statistics Cressie (1993, pg. 2-3) states that beginning classes in Statistics (and many of the more advanced ones) always assume that observations on a phe- 6

24 Figure 2.1: Hemodynamic response curve for an active voxel when using the sfmri method. 7

25 nomenon are taken under identical conditions and that each observation is taken independently of any other. The data then form a random sample [i.e., are independent and identically distributed (i.i.d)]; standard statistical techniques can be applied to build a statistical model and to estimate the model s parameters. He goes on to say, models that involve statistical dependence are often more realistic; two classes of models that have commonly been used involve intraclasscorrelation structures and serial-correlation structures. These offer little scope for spatial data, where dependence is present in all directions and becomes weaker as data locations become more dispersed. The complex spatial nature of fmri data makes the modeling of covariance structures an important part of the data analysis process Spatial Modeling Spatial data can be thought of as a realization from a random process that varies continuously throughout a region. Symbolically, Cressie (1993) refers to this as {Z(s) : s D}, where D is a subset of IR d with positive d-dimensional volume, and s is the continuous spatial index of the process Z. It is of interest to model this random process given the data {Z(s 1 ),..., Z(s n )} at know spatial locations {s 1,..., s n }. One of the key tools in modeling this process is the variogram. Matheron (1962) defines the 8

26 variogram, 2γ(.), as 2γ(s 1 s 2 ) = var(z(s 1 ) Z(s 2 )), for all s 1, s 2 D. It should be noted that γ(.) is called the semivariogram and is commonly used in place of the variogram. From the variogram definition it is easy to see that 2γ(0) = 0; however, if the observed value of the variogram does not approach zero but approaches some other value c 0 > 0 as the distance between two points approaches zero, then c 0 is called the nugget effect (Matheron 1962). Notationally, this can be written lim h 0 γ(h) = c 0 > 0. The nugget effect can be considered microscale variation in the process and can be used to explain the discontinuity in the variogram as the distance approaches zero. This nugget can also be due to measurement error in the process Stationarity In order to model the random process Z, assumptions must be made about the process. If no assumptions are made the data would represent an incomplete sampling of a single realization of the process, making any kind of statistical inference impossible. One assumption often made is that the mean of the overall process is the same regardless of the spatial location s, that is E(Z(s)) = µ for all s D. Assumptions about the covariance must also be made in order to make predictions. One such assumption is cov(z(s 1 ), Z(s 2 )) = C(s 1 s 2 ), for all s 1, s 2 D. 9

27 The function C(.) is called a covariogram and a process satisfying both the constant mean and the constant covariogram function assumptions is called secondorder stationary. The stationarity assumption allows for estimation and prediction within the space (Cressie 1993) Isotropy The value of the variogram function is found using the distance between two spatial locations s 1 and s 2. If the variogram value is the same for all s 1, s 2 pairs for which the distance between s 1 and s 2 is constant, regardless of their locations, then the variogram is called isotropic (Journel and Huijbregts 1979). Isotropy is a simplifying assumption that is often made in the analysis of spatial data. If a process is stationary and the variogram is only a function of s 1 s 2, it is isotropic. If the value of the variogram is not only a function of the distance between s 1 and s 2 but also a function of their direction then the process is said to be anisotropic. The MIXED procedure in SAS R can be used to analyze isotropic and certain anisotropic spatial covariance functions (SAS 1999). 2.3 Linear Mixed Models Linear mixed models can be used for the analysis of correlated data, such as fmri data. With the use of the MIXED procedure in SAS R, spatial covariance structures can be fitted to the correlated data with the use of the repeated 10

28 statement. The repeated statement is used because the observed measurements are recorded on the same subject over different locations. The linear mixed model can be written as: y = Xβ + Zu + ɛ where X and Z are design matrices, β and u are vectors of coefficients for the fixed and random effects respectively, and ɛ is a vector of random errors. The vector u is assumed to be normally distributed with mean vector zero and variancecovariance matrix G. ɛ is also assumed to be normally distributed with mean 0 and variance-covariance matrix R. Also u and ɛ are assumed independent; consequently, Cov(u, ɛ) = 0. In general, V ar(y) for linear mixed models is ZGZ + R = V. Estimation of the parameters in mixed models is more difficult than in fixed linear models (Littell et al. 2002). Not only must β be estimated but also the unknown parameters in V. Also, the correlated nature of the data must be taken into account. In such a case, least-squares is not usually the best method for finding ˆβ. Instead, we use generalized least-squares (GLS), which minimizes (y Xβ) V 1 (y Xβ). Thus the (GLS) estimate for β is ˆβ GLS = (X V 1 X) 1 X V 1 y. However, since V is unknown, estimated generalized least-squares (EGLS) is used by inserting a reasonable estimate of V into the estimating equation. This esti- 11

29 mate of V can be obtained through likelihood-based approaches such as maximum likelihood (ML) or restricted maximum likelihood (REML) (Littell et al. 2002). Now with the estimate ˆV, the estimate for β becomes ˆβ EGLS = (X ˆV 1 X) 1 X ˆV 1 y. It should be noted that although REML is a sensible method for parameter estimation, and is the default method in SAS R, Warnes and Ripley (1987) claim that maximum likelihood estimators are of limited value for spatial problems. They offer no better method of parameter estimation but simply suggest that the estimates be interpreted with caution. If G and R (which affect V) were known, ˆβ would be the best linear unbiased estimator (BLUE) of β. However, since they are not known, ˆβ is no longer BLUE but is referred to as being the empirical best linear unbiased estimator (EBLUE), where best indicates the least squared error. (Littell et al. 2002; Searle 1971) Selection of Covariance Structure The modeling of a covariance structure refers to representing V (Y) as a function of a relatively small number of parameters (Littell et al. 2000). Since observations on different subjects are assumed independent, but repeated measurements from the same subject cannot be considered independent, the covariance structure of the data can be modeled by placing a functional specification on R. In their analysis of repeated measures data over time, Littell et al. (2000) use 12

30 a model in which the covariance between two observations on the same subject depends only on the length of the time interval between measurements (called the lag), and the variance is constant over time. In spatial analysis the covariance is a function of distance rather than time, and in more complex spatial modeling it is also a function of direction and space (Cressie 1993). Since the observations obtained from an fmri cannot be assumed to be independent, the correlation between measurements must be modeled. Grady and Helms (1995) discuss how an unstructured covariance structure is useful for exploring trends in the covariance matrix. Littell et al. (2000) made use of the correlogram (Cressie 1993) as a graphical tool to explore the nature of the correlation and Dawson et al. (1997) offer other graphical techniques. Often the researcher s prior knowledge can be a useful tool in exploring the covariance structure in an effort to obtain the correct structure. Littell et al. (2000) observed that if the covariance structure is incorrectly modeled the estimates of the effects and inferences made may be invalid (Ferron et al. 2002). In the analysis of an fmri dataset, since the measurements are correlated over space it seems practical to use a covariance structure specifically designed for spatial data, but it may be difficult to choose which structure is correct. Many researchers use information criteria such as the Akaike Information Criteria (AIC) (Akaike 1973) or Schwarz s Bayesian criteria (SBC) (Schwarz 1978) as tools for choosing between different covariance structures. Marx and Stroup (1993) state that AIC are SBC are goodness-of-fit indices that may be used to 13

31 compare models with the same fixed effects but different covariance structures. AIC and SBC in SAS R are computed by taking -2 times the minimized restricted log-likelihood (REML) and then penalizing for the number of covariance parameter estimates. SBC imposes a greater penalty for covariance parameter estimates. If two different covariance structures contained the same number of parameters, such as compound symmetry and auto-regressive(1) or spatial Gaussian and spatial spherical, then AIC and SBC would simply indicate the better fit to the data since both structures receive the same penalty. Grady and Helms (1995) also note that if covariance structures have different numbers of parameters, and if one is a special case of the other, likelihood ratio tests can be used to choose between the covariance structures. Covariance structure modeling offers many benefits in a data analysis process. Grady and Helms (1995) suggest that by modeling the covariance structure of the data a more complete understanding of the process is obtained. Modeling a covariance structure allows for unbalanced and missing data without having to delete the subject (Littell et al. 2000). By modeling the covariance structure the number of parameters that must be estimated can be drastically reduced. This is advantageous when the selected covariance structure is adequate, that is, it produces good estimates and test statistics. 14

32 2.3.2 Type I Error Rates Several simulation studies have been performed to investigate Type I error rates of F-tests in mixed linear models (Schaalje et al. 2002). Most of these studies include covariance structures such as: auto-regressive of order 1 (AR(1)), compound symmetry (CS), toeplitz (TOEP), and similar structures with heterogeneity of variance. The influence of the complexity of the covariance structure on the error rates were explored. Different ways of computing degrees of freedom have also be investigated (Schaalje et al. 2002; Kenward and Roger 1997). In an fmri it is of interest to find regions of activity in the brain. If these regions are previously identified then the location can be considered a fixed effect and F-tests can be used to evaluate if the responses in that region significantly differ from the other responses. However, as previously stated, if the covariance structure is not adequately modeled, the tests on fixed effects may be invalid. 15

33 Chapter 3 Methods This chapter describes the methods used for the simulation study. The simulation study entails the generation of data that resemble fmri data, where observations taken at spatial locations are correlated according to some known spatial structure. Then mixed models with several spatially correlated error structures are fitted to the data. A spatial error structure is chosen using AIC as a measure of fit after which statistical properties of the model parameter estimates will be observed. An explanation of the data generation process is outlined in the first section. The second section discusses the use of simulations to investigate the performance of the covariance structure selection process, the Type I error rates, and power. 3.1 Data Generation The data generation was designed to represent an fmri data set. Responses were generated for each voxel within a single slice of an fmri scan. Since the 390 observed locations on a slice are all taken from the same subject, they are spatially 16

34 correlated repeated measures. No fixed effects were added to the simulated data set. The data consisted of 390 observations from a grid similar to half of an fmri scan (Figure 3.1). Two sets of generated data were created for the same slice, one to represent the responses from the control fmri scan and one to represent the responses from the stimulated scan. The region intended to represent the auditory cortex region contained 34 of the 390 observed locations. The region of the auditory cortex was identified in the grid of points but no effect was added to the responses in the region (Figure 3.1). Due to the absence of a fixed effect associated with the region of interest it is possible to consider Type I error rates associated with the test for region activation. The four covariance structures used to generate the repeated measures were exponential, Gaussian, power, and spherical. Details of these four covariance structures are given in Table 3.1 (SAS 1999; Journel and Huijbregts 1979). Each of these covariance structures has three parameters; the variance (σ 2 ), the range of dependency (θ) or correlation coefficient (ρ), and the nugget (η). Note that d i,j represents the euclidean distance between the i th and j th elements in the grid of observed data locations. The nugget (η) is multiplied by an indicator variable where 1 (di,j =0) is equal to one when d i,j = 0 and is equal to zero otherwise. The parameters for each of the four structures were chosen from the estimates of an actual data set. The data set was slice 10 from an fmri scan. The observations were trimmed in order to only have responses for regions that actually contain brain tissue. The estimates were obtained by fitting a semi-variogram 17

35 y x Figure 3.1: Generated data voxel locations within a slice. The auditory cortex is the shaded region. Structure (i, j) th element Exponential Gaussian σ 2 [exp( d i,j /θ Exp )] + η1 (di,j =0) σ [ 2 exp( d 2 i,j/θgau 2 )] + η1 (di,j =0) Power σ 2 ρ d i,j [ P ow + ] η1 (d i,j =0) Spherical σ 2 1 3d ij 2θ Sph + d3 i,j 1 2θSph 3 (di,j θ Sph ) + η1 (di,j =0) Table 3.1: Selected Spatial Covariance Structures. d i,j is the distance between two points, and 1 (di,j =0) is an indicator function for distance zero. 18

36 Parameter Structure σ 2 θ or ρ η Exponential Gaussian Power Spherical Table 3.2: Covariance Parameters used to Generate Data. to the data with each of the four structures. Weighted least squares was used to fit the semi-variograms and obtain estimates of the parameters. The fitted semivariograms are shown in Figure 3.2 and Table 3.2 gives the values of the covariance parameters for each structure. To ensure that the data from each of the four structures were similar in terms of generalized variance, the eigenvalues for each of the four structures were computed and the log eigenvalues are shown in Figure 3.3. Since the eigenvalues of the covariance matrices are similar for the four structures, the generalized variances are also similar. This is important to insure that the results of the AIC selection of the covariance structure is not simply confounded with the fact that one structure yields a much larger generalized variance than another. Once the data generation parameters were chosen, ten thousand datasets for each structure were generated following a two-step process outlined by Ripley (1987). The first step of this process is to generate a random multivariate normal vector y with mean vector zero and variance-covariance matrix as an identity, y N n (0, I). In the second step the random vector y is multiplied by the Cholesky decomposition of the covariance matrix of the desired structure. The response vec- 19

37 gamma Exponential Gaussian Power Spherical objective = distance Figure 3.2: Semi-Variograms used to generate the simulated data. Dots represent the empirical Semi-Variogram. 20

38 Log(Eigenvalues) Exponential Gaussian Power Spherical Index Figure 3.3: Log-Eigenvalues of the Covariance Matrices 21

39 tor that results has a zero mean and variance-covariance matrix which follows the structure of R as in Table 3.1. Recall that R is the variance-covariance matrix for repeated measurements that are correlated so V ar(y) = R. Since the actual data had an overall mean of 11, this amount was added to every generated response; thus the response vector has a mean vector of 11 and a variance-covariance matrix of R. Since the entire region has the same expected value and the covariance depends only on the distance between two voxels, the data are stationary and isotropic. However, for the power study, a fixed amount of 2 was added to all of the responses from the stimulated state in the predefined region. All of the data generation was done using Proc IML of SAS. Due to memory problems, the data were generated in small portions and then added together to create the total 40,000 data sets. Proc MIXED was used to analyze all of the data sets following a similar pattern as the data generation, in small portions. Generating the data, fitting the model, and evaluating Type I error rates for the 40,000 data sets took about 1000 hours on a Penguin Computing Dual Opteron Altus 1000E Linux machine with SAS v. 9. By default, Proc MIXED uses the iterative process of REML to estimate the variance-covariance parameters of the mixed model (Littell et al. 2002). 3.2 Modeling the Data The model was fitted in two steps. Since it is assumed that the repeated measures are stationary, the control or non-stimulated set of responses was used 22

40 to fit the covariance structure with no fixed effects in the model. Recall that stationarity implies that the mean and the covariance of the process do not differ for different locations but are constant. The stimulated set of responses are assumed to not be stationary since it is believed that the responses in the auditory cortex differ from the others. So the MODEL statement of SAS Proc Mixed for the control was y=. The four covariance structures were fitted to each of the data sets by changing the REPEATED statement (Listing 3.1 for example code). The DDFM option on the MODEL statement was changed to KENWARDROGER. This was used instead of the default BETWITHIN (between-within) because it involves modifying the estimated variance-covariance matrix of the fixed effects which should result in better Type I error rates (Kenward and Roger 1997; SAS 1999). Listing 3.1: Example of Pseudo SAS Mixed Code 1 Mixed code f o r s t a t i o n a r y p r o c e s s to f i n d c o v a r i a n c e s t r u c t r e and parameters ; 2 Proc mixed data=fmri ; 3 class treatment r e g i o n ; 4 model y= / ddfm=kenwardroger; 5 repeated / subject=treatment type=sp(exp) ( x y ) local ; 6 where treatment =1; 7 8 Mixed code f o r a n a l y s i s o f treatment by r e g i o n e f f e c t, using parameter v a l u e s from s t a t i o n a r y n o n s t i u l a t e d r e s p o n s e s ; 9 Proc mixed data=frmi n o p r o f i l e ; 10 class treatment r e g i o n ; 11 model y= treatment r e g i o n / ddfm=kenwardroger; 12 repeated / subject=i n t e r c e p t type=sp(exp) ( x y ) local ; 13 parms / n o i t e r ; 14 AIC was recorded for each fitted model and then used to determine which covariance structure best fit the data. Recall that AIC is computed by taking 23

41 -2 log likelihood and then penalizing by adding twice the number of parameter estimates. However, since the four structures in question all have three parameters (the variance, the range, and the nugget), AIC is being penalized equally for all structures and simply represents the best fit to the data with the largest likelihood. The lowest AIC model for each data set was tabulated to examine the success rates of AIC in choosing the correct covariance structure. Chapter 4 will show the results of the AIC selection in a two-way table, with data structure on one margin and AIC chosen structure on the other. If the iterative process of REML were performing perfectly in finding the true parameter values, all of the AIC chosen structures would be the same as the actual data structure. Once the best covariance structure was chosen for each data set, the model was re-fit with the covariance structure and covariance parameter estimates obtained from only fitting the control responses. This time the MODEL statement was changed to y=treatment region treatment*region, where the treatment represents the control or stimulated responses, region is a indicator variable of locations previously identified as inside the auditory cortex, and treatment*region represents the interaction between treatment and region. In the second call of the MIXED procedure the F-test on the treatment by region interaction is of special interest to determine if the auditory cortex is becoming active during the stimulated state. So, the p-values for the treatment by region interaction were recorded for the AIC chosen model, and then for simulation purposes p-values were also recorded for the correct covariance structure with the 24

42 true covariance parameters. The proportion of times that the p-values were less than or equal to α = 0.05 was calculated, which should be close to.05 since no fixed effects were added. The rejection rates for this fixed effect are shown in Chapter 4. The table shows the structure with which the data was generated, the AIC chosen structure(model used to fit the data) and the rejection rates. Since the rejection rates are not exact with only 10,000 simulations, ranges for the rates were computed. The standard deviation of this proportion is given by the equation p(1 p) n, which for the 10,000 observed datasets is A 95% probability region for the proportion is ( ). Therefore, if the Kenward-Roger method were performing correctly for these spatial structures, the Type I error rates should fall within the probability region in 95% of cases. 25

43 Chapter 4 Results This chapter presents the results of the simulation study. First the performance of AIC in selecting a spatial covariance structure is discussed. Then type I error rates and power are addressed. 4.1 Performance of AIC For each of the 40,000 total data sets, AIC was recorded for all four modeled covariance structures. The structure that returned the lowest AIC was selected as the structure that best fit the data. The results of the AIC selection are shown in Table 4.1. Each row in the table represents the covariance structure from which the data was generated and the columns represent the AIC-selected structure. If AIC were performing perfectly at selecting the correct covariance structure then this table would have 100% down the diagonal and zero everywhere else. It is clear to see from Table 4.1 that AIC performs very poorly at selecting the correct covariance structure. The Gaussian covariance structure is the most commonly selected structure, chosen for 33.61% of the 40,000 total data 26

44 AIC Selected Structure Data Structure Exponential Gaussian Power Spherical Total(Counts) Exponential 19.73% 33.02% 19.26% 27.99% Gaussian 4.18% 48.19% 43.68% 3.95% Power 20.41% 31.88% 19.30% 28.41% Spherical 24.08% 21.36% 44.05% 10.51% Total 17.10% 33.61% 31.57% 17.72% Table 4.1: Percents of AIC-Selected Covariance Structures. sets. Power is the next most common structure followed by spherical and exponential. Further exploration of the table reveals some interesting patterns. For the exponential, Gaussian, and power data structures, the structure most commonly chosen by AIC was the Gaussian structure. For the spherical data, the power structure was most commonly selected by AIC. However, even when the actual data structure was Gaussian, AIC chose the Gaussian structure less than half the time. In general, AIC is a very poor selector of the correct spatial covariance structures. 4.2 Type I Error Rates To evaluate type I error rates for the treatment by region interaction, the p-values from the fixed-effects tests were recorded. These tests were done using the AIC-selected covariance structure. Recall that the treatment by region interaction is of primary interest because it indicates if the region identified as the auditory cortex becomes active during the stimulated state. Since no fixed effect was added to the responses in the region of interest, the type I error rates should be equal 27

45 AIC Selected Structure Actual Data Structure Exponential Gaussian Power Spherical Overall Exponential * * * Gaussian * * * * * Power * * * Spherical * * 0.076* * * Overall * * * * * Table 4.2: Rejection Rates Based on AIC-Selected Structure. to α = The type I error rates are reported in Table 4.2. The row labeled Overall represents the marginal type I error rates for each modeled covariance structure regardless of the actual data structure. The Overall column represents the marginal type I error rates for each data structure. The type I error rates marked with an asterisk (*) are significantly different than 0.05 based on the number of data types modeled with a particular structure. For example, 1973 of the 10,000 exponentially generated data sets were modeled with an exponential covariance structure so the type I error rates should be between and (0.05 ± (1 0.05) 1973 ). Of the 16 different combinations of actual and selected data structures, 12 of them have type I error rates that are significantly different than expected. Gaussian structures seem to yield the highest inflation to the type I error rates. The exponential model fitted to the exponential and power data have reasonable type I error rates, as well as the spherical model to the exponential data and power model to the power data. The type I error rate for the power model of the exponential data is significantly lower than expected. Overall the type I error 28

46 rates are inflated even when using a spatial covariance structure to account for the dependency in the data. During the data analysis process, researchers are unaware of the correct structure from which their data came and can only select the structure used to model the data. From the information presented in Table 4.2 it can be clear to the researcher that regardless of the structure chosen to model the data, the type I error rates will be inflated. If the Gaussian structure is selected by AIC, it also yields the highest type I error rates, so the results should be reported cautiously. 4.3 Power In the fmri analysis it is of primary interest to determine if a region of the brain is being activated when stimulated. In the results discussed to this point no fixed-effects where added to the responses so the responses of all regions would have an equal mean. In order to address the goal of evaluating region activity, a power analysis was also run. The simulations of the power analysis follow a similar pattern as the rest of the simulations. The parameters for all of the structures were the same as before however, a small constant was added to all of the responses identified as being in the region of interest in the stimulated state. The amount added was chosen so that the rejection rate of the fixed-effects test would be about When the data were generated, the overall mean of the responses was 11 and for the power analysis 2 was added to the responses in the region of interest. Thus, the mean of 29

47 AIC Selected Structure Actual Data Structure Exponential Gaussian Power Spherical Overall Exponential Gaussian Power Spherical Overall Table 4.3: Rejection Rates for Power Analysis Based on AIC-Selected Structure. the responses in the region of interest during the stimulated state is approximately 13, while the mean of the rest of the responses are approximately 11. Ignoring the spatial correlation, the standard deviation of the 390 responses in the simulated data is around 2.5, so less than one standard deviation has been added to the 34 responses. The rejection rates for the treatment by region interaction are presented in Table 4.3, the rejection rates are from the AIC-selected covariance structure. The rejection rates seem to be similar for all structures of data and models. The data structure that results in the highest overall power is the power data structure. The modeled structure that yields the highest overall power is Gaussian. For the exponential data structure the Gaussian models had the highest power with 48.75% of the models rejecting the null hypothesis; of the 16 combinations of data and modeled structures this is the one with the highest power. Since all four covariance structures have similar power, the structure with an optimal type I error rate would be prefered over the others, indicating that Gaussian would be a poor choice. However, since the only results reported in this study are from AIC-selected structures, it is unclear if the same pattern would hold for all data 30

48 sets and structures. 4.4 Summary AIC appears to be a very poor selector of spatial covariance structures. Incorrect structures are generally selected more frequently than the correct one. The Gaussian covariance structure is most commonly selected by AIC. Type I error rates are generally inflated for testing fixed effects when using the AIC-selected structure, which is sometimes the correct structure. 31

49 Chapter 5 Data Analysis In Chapter 4, the overall performance of SAS R in analyzing spatial data resembling an actual fmri was investigated. We now consider an analysis of the fmri data set from which this project was motivated. From the results of the simulations, we know that Gaussian covariance structures are most commonly selected by AIC regardless of the true structure of the data. Type I error rates are inflated, and a difference in the mean responses in the auditory cortex of about 2 results in concluding significant activation in the region of interest about half the time. 5.1 Covariance Structure Selection In order to select which covariance structure should be used to model the data, all four structures where fit to the data and AIC recorded for each one. The model with the lowest AIC is chosen as the one that should be used. The AIC values are as follows, 541.0(Exp), 517.1(Gau), and 916.7(Sph). The power model did not converge so the value of AIC is not comparable. As expected, the Gaussian 32

50 structure returned the lowest value of AIC. However, from the results of the simulation study it is unclear whether the data really come from a Gaussian process or if it is just an artifact of the poor performance of AIC. Regardless of the lack of confidence in the AIC selection process, a structure must be chosen and since we have no better method, the AIC chosen structure will be used, but we also consider the results associated with the exponential and spherical models. The covariance parameter estimates obtained for the Gaussian model are nugget(η)=0.0023, range(θ Gaussian )=1.316, and variance(σ 2 )= These estimates along with the Gaussian structure should correctly account for the dependency in the data so that accurate inference can be made. 5.2 Test of Significance Now that a covariance structure has been chosen and covariance parameters estimated, the tests on the fixed-effects can be examined. The F-statistic for the region by treatment interaction is with 1 numerator and 387 denominator degrees of freedom. This statistics yields a p-value of <0.0001, thus we would conclude that the mean of the defined region is different during the two stages. Printing out the least-squares means reveals that even though the difference in the responses is small it is sufficient enough to be significantly detected. The leastsquares means and standard errors associated with the Gaussian model are given in Table 5.1. Even thought AIC selected Gaussian as the best fitting structure, it is of 33

51 Treatment Region Estimate Standard Error Stimulated Auditory Cortex Stimulated Non-Auditory Cortex Non-Stimulated Auditory Cortex Non-Stimulated Non-Auditory Cortex Table 5.1: Least-Squares means for treatment by region levels interest to examine the results from the other structures. Since the power structure did not converge it is not meaningful to explore the results of that model. The F-statistics for the fixed-effect test of the treatment by region interaction were 17.57, 17.56, and for exponential, Gaussian and spherical, respectively. The parameter estimates for the other three structures are shown in Table 5.2. Despite yielding unrealistic parameter estimates for σ 2 and θ, the F-statistic associated with the exponential model is similar to the F-statistic associated with the more realistic-looking Gaussian model. The spherical model concludes that the range of spatial dependency is essentially zero, yet it too has an F-statistic very close to the Gaussian model F-statistic. All three F-statistics would result in the same conclusion so it appears that adjusting for the spatial correlation, whether with the correct structure or just a good one, is beneficial. Parameter Structure σ 2 θ η Exponential Gaussian Spherical E Table 5.2: Covariance parameter estimates from SAS. 34

52 Chapter 6 Further Considerations During the course of this project many different ideas have been discussed and this chapter presents a few of those ideas which were not incorporated into this project but which may be useful in further research. 6.1 Independence of Responses Across Treatments In Chapter 3 the method for generating the data sets for the simulation study was explained. There were two sets of responses created with the spatial covariance structure, these two sets represented the stimulated and non-stimulated responses for an individual. In the data generation, these two sets were created independent of one another which in actuality is not reasonable because the responses are for the same subject so even across treatments the responses should be correlated for a given voxel. For future studies, the responses from the two treatments should not be generated independent of one another, but should be generated so that for a given voxel, the responses are correlated. This can be incorporated by creating two sets of responses, Z 1 and Z 3 such that they are indepen- 35

53 dent. The set of correlated responses (Z 2 ) can be obtained from Z 2 = az 1 + bz 3, where a and b are chosen to induce correlation. When the simulated data was analyzed, the treatments were considered independent but that the responses from both treatments had the same covariance structure and covariance parameters. The non-stimulated set of data was used to find the lowest AIC model structure and also to obtain estimates for the covariance parameters. These covariance parameter estimates were used for both treatments when evaluating the type I error rates and for the power analysis. This was specified in SAS R in the REPEATED statement by specifying SUBJECT=treatment. Recall that the repeated statement specifies the variance-covariance structure of ɛ, which is called R, and that the variance-covariance matrix of the response vector V is equal to R only if a REPEATED statement is used and there are no random effects. However, in specifying the SUBJECT=treatment option of the REPEATED statement, the variance-covariance structure R was placed on both sets of treatment responses so that V was really a block diagonal matrix with two blocks of the same R matrix. Both blocks of the R matrix were the same since the covariance parameters were estimated from the non-stimulated responses and then forced into the model with both treatments, not allowing SAS R to further iterate in finding new covariance parameter estimates. Before the actual fmri data set was analyzed, this inconsistency of considering the two treatments independent of one another was discovered and corrected for the actual fmri analysis. Simply by changing the SUBJECT= option to SUB- 36