A stationarity test on Markov chain models based on marginal distribution

Transcription

1 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 646 A stationarity test on Markov chain models based on marginal distribution Mahboobeh Zangeneh Sirdari 1, M. Ataharul Islam 2, and Norhashidah Awang 1 1 School of Mathematical Sciences, Universiti Sains Malaysia, USM, Pulau Pinang zangeneh m@yahoo.com, 2 Department of Statistics, University of Dhaka, Dhaka 1000 Abstract. A stationarity test on Markov chain models is proposed in this paper. Most of the previous test procedures for Markov chain models have been done based on conditional probabilities of transition matrix. The likelihood ratio test and chi-square test have been used for test procedures such as stationarity, order of Markov chain, and goodness of fit test, for which all the parameters need to be estimated. This paper uses the efficient score test, an extension of Tsiatis model, for testing the stationarity of Markov chain model based on marginal distribution as obtained (Azzalini, 1994). For testing the suitability of the proposed method, a numerical example of real life data is given Introduction Markov chain models are used in various applied fields such as time series analysis, longitudinal studies, real life time data, and environmental problems. The behavior of a Markov chain depends on the transition matrix, which contains transitional probabilities. In most practical studies the transition matrix is unknown and needs to be estimated. There are several methods for estimation and test procedure of transition probabilities. However, most of the researchers have worked on estimation of parameters. Yet, reports on test procedures are hardly found. One of the most important tests on Markov chain models is stationarity of transition probabilities which is interested to work on. In this section a brief summary of procedure tests on Markov chain is presented. Anderson and Goodman (1957) obtained maximum likelihood estimates and their asymptotic distribution in a Markov chain of arbitrary order when there are repeated observations of the chain. Likelihood ratio tests and χ 2 -tests are considered for testing stationary and order of higher-order Markov chains. Billingsley (1961) used Whittle s formula, chi-square and maximum likelihood methods to estimate and test the parameters. A sample {x 1, x 2,..., x n } from a first order Markov process with transition probabilities p ij and initial probabilities p i was considered. If s s matrix F {f ij } is defined as the transition count of the sequence, then it can be shown that (f ij f i p ij ) 2 /(f i p ij ) (1) ij

2 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 647 is asymptotically chi-square in distribution with s(s 1) degrees of freedom. This chi-square statistic is useful for testing whether the transition probabilities of the process have specified values p ij. Then the natural problem of testing whether these transition probabilities have a specified form p ij (t) arises, where t is an unknown from the sample. Bartlett (1951) constructed a likelihood ratio test for the goodness of fit by proving that the asymptotic distribution in Markov chains was normal. For testing whether a sequence of observations is at most r-dependent, it is assumed that the transition probabilities are known, or at least depend on a limited number of parameters which can be estimated. If the transition probabilities are completely unknown, a different test is needed and this test is presented (Hoel, 1954) where the derivation depended upon Bartlett s results and methods. However, the previous methods of testing parameters were based on transition probabilities and statistic test was depended on transition probabilities. In recent decades, more research on estimating and test procedures of parameters of Markov chain model are extended to the new methods where covariates and link functions are used and repeated measures are considered. For example, Muenz and Rubinstein (1985) proposed a model for Markov chain based on covariates and showed how the covariates relate to changes in state. An extensive covariate-dependent for higher order Markov models was improved (Islam and Chowdhury, 2006). An influence of time-dependent covariates on the marginal distribution of binary response has been studied (Azzalini, 1994). It has been shown that the covariates relate only to the mean value of the process, independently of the association parameter. An application of Markov models based on marginal distribution is provided (Shafiqur Rahman and Islam, 2007). A goodness of fit test for the logistic regression model based on binary data was employed (Tsiatis, 1980). He modified the model related to the probability of responses with a set of covariates. In this paper, Tsiatis (1980) method is used for testing the stationarity of binary Markov chain model based on marginal distribution, modified (Azzalini, 1994). The efficient score test is used for testing null hypothesis, which only requires the estimate of parameters under the true null hypothesis. 2 Stationarity test A single stationary process (y 1,..., y T ) generated by a binary Markov chain taking values 0 and 1 is considered. The transition matrix is defined by [ ] [ ] p00 p P 01 1 p01 p 01 p 10 p 11 1 p 11 p 11

3 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 648 where p ijt Pr(Y t j Y t 1 i); i, j 0, 1. The denoted mean θ E(Y t ) is for the case of stationary while θ t E(Y t ) stands for non-stationary process. The odds ratio between successive observations is defined as ψ p 11/(1 p 11 ) p 01 /(1 p 01 ) Pr(Y t 1 Y t 1) Pr(Y t 1 Y t 0) Pr(Y t 1 0, Y t 1) Pr(Y t 1 1, Y t 0). The range of possible values for ψ is independent of the value of θ. The relationship between mean and the probabilities can be presented as θ θp 11 + (1 θ)p 01 and for generalization, in non-stationary case θ t θ t 1 p 11 + (1 θ t 1 )p 01 In this case, θ t E(Y t ) varies with t via logit function resulting to logit(θ t ) χ tβ; θ t exp(χ tβ)/(1 + exp(χ tβ)) (2) where χ t is p-dimensional vector of time-dependent covariates and β is a p-dimensional parameter. The transition probabilities have been obtained in terms of the odds ratio and mean for observations as derived (Azzalini, 1994) { θt for ψ 1 p jt δ 1+(ψ 1)(θ t θ t 1 ) 2(ψ 1)(1 θ t 1 + j 1 δ+(ψ 1)(θt+θ t 1 2θ tθ t 1 ) ) 2(ψ 1)θ t 1 (1 θ t 1 for ψ 1 ) t 2,..., T. Where, δ 2 1+(ψ 1){(θ t θ t 1 ) 2 ψ (θ t +θ t 1 ) 2 2(θ t +θ t 1 )}. It is assumed that a sequence of observed data y 1,..., y T is available for inference. The likelihood function would be L i0 j0 p y ijt ijt Thus, the log-likelihood function is (1 p 01t ) 1 y 01t p y 01t 01t (1 p 11t ) 1 y 11t p y 11t 11t t0 ln L {(1 y 01t ) ln (1 p 01t )+y 01t ln p 01t +(1 y 11t ) ln (1 p 11t )+y 11t ln p 11t } p 01t {y 01t ln (1 p 01t ) +ln (1 p p 11t 01t)}+ {y 11t ln (1 p 11t ) +ln (1 p 11t)}

4 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 649 {y 01t logit(p 01t ) + ln (1 p 01t )} + {y 11t logit(p 11t ) + ln (1 p 11t )} {y 01t logit(θ 01t )+ln (1 θ 01t )}+ {y 11t logit(θ 11t )+ln (1 θ 11t )}. Since the logit functions for conditional means are logit(θ 01t ) χ 01tβ 01, and logit(θ 11t ) χ 11tβ 11, ln L {y 01t χ 01tβ 01 ln (1 + exp (χ 01tβ 01 ))} + {y 11t χ 11tβ 11 ln (1 + exp (χ 11tβ 11 ))}. (3) The likelihood function for marginal model, as obtained from Azzalini (1994) would be L i0 p y it it (1 p 1t ) 1 yt p yt 1t. t0 Then the log-likelihood function is defined as p 1t l ln L {(1 y t ) ln (1 p 1t ) + y t ln p 1t } {y t ln (1 p 1t ) + ln (1 p 1t)} {y t logit(p 1t ) + ln (1 p 1t )} {y t logit(θ t ) + ln (1 θ t )} {y t χ tβ ln (1 + exp (χ tβ))}. (4) The estimate of parameters can be computed from the following equation ln L β ln L p 01t ln p 01t θ 01t ln θ 01t β + ln L p 11t ln p 11t θ 11t ln θ 11t β 0. Via equation (4), ln L β q {y t χ tq χ tq exp (χ tβ)/(1 + exp (χ tβ))} 0. And by exploding equation (3) the following equations can be written.

5 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 650 ln L 0 β 01q {y 01t χ 01tq χ 01tq exp (χ 01tβ 01 )/(1 + exp (χ 01tβ 01 ))} 0, ln L 0 {y 11t χ 11tq χ 11tq exp (χ β 11tβ 11 )/(1 + exp (χ 11tβ 11 ))} 0, 11q where, ln L ln L 0 + ln L 1. To test the stationarity of binary Markov model based on marginal distribution, by considering Tsiatis model (Tsiatis, 1980), it is assumed that the space of covariate (χ 1,..., χ p ) is partitioned into G distinct regions in p-dimensional space denoted by R 1,..., R G. The indicator functions I (k) t defined by I (k) t considered as follow 1 if (χ 1,..., χ p ) R k and I (k) t (k 1,..., G) are 0 otherwise. The model is logit(θ t ) χ β + γ I t ; θ t exp (χ β + γ I t )/(1 + exp (χ β + γ I t )) (5) where I t (I (1) t,..., I (G) t ) and γ (γ 1,..., γ G ). The null hypothesis test is H 0 : γ 1... γ G 0, based on partitioning the space of time-dependent covariates into distinct regions. The related test statistic is a quadratic form of observed counts minus the expected counts which has asymptotic chi-square distribution with G degrees of freedom, as proven (Rao, 1973). Both the efficient scores test or likelihood ratio test can be used for testing the hypothesis. At this point, the efficient score test was used and the test statistic is defined by T Z V Z, (6) where Z is the G-dimensional vector ( l/ γ 1,..., l/ γ G ). And the matrix V is where, V A BC 1 B A jj 2 l/ γ j γ j (j, j 1,..., k), B jj 2 l/ γ j β j (j 1,..., k; j 0,..., p), C jj 2 l/ β j β j (j, j 1,..., p), All above terms were evaluated at γ 0 and β ˆβ, where ˆβ is the maximum likelihood estimate of the parameters when H 0 is true. The log-likelihood based on model (5) is

6 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 651 l ln L {y t (χ β + γ I t ) ln (1 + exp (χ β + γ I t ))}. where I t is the row vector of indicator variables for the tth observation. The j th element of vector Z used in the computation of the statistic (6) is the partial derivative of l with respect to γ at γ 0 and β ˆβ, y t I (j) t I (j) t exp (χ β)/(1 + exp (χ β)) O j E j, where O j and E j are the observed and expected numbers of responses in the j th region. Therefore the statistic test (6) is a quadratic form of the vector of observed counts minus expected counts. Quantities necessary for computing the covariance matrix V are presented in A jj { ξ j ˆθt (1 ˆθ t ) j j 0 j j ; j, j 1,..., k, B jj ξ j χ jtˆθt (1 ˆθ t ) (j 1,..., k; j 0,..., p), C jj ξ j χ jt χ j tˆθ t (1 ˆθ t ) (j, j 0,..., p), where ξ i denotes the set of indices j such that (χ i1,..., χ ip ) R j, ˆθt exp (χ ˆβ)/(1 + exp (χ ˆβ)). The second derivatives of log-likelihood function for computing the statistic (6) with respect to γ q and β q, under null hypothesis are 2 ln L β q β q 2 ln L γ q β q 2 ln L γ q γ q exp(χ β + γ I t ) χ tq χ tq [ 1 + exp(χ β + γ I t ) ][ exp(χ β + γ I t ) ] χ tq χ tq θ t (1 θ t ), I (q) exp(χ β + γ I t ) t χ tq [ 1 + exp(χ β + γ I t ) ][ exp(χ β + γ I t ) ] I (q) t χ tq θ t (1 θ t ), I (q) t I (q ) exp(χ β + γ I t ) t [ 1 + exp(χ β + γ I t ) ][ exp(χ β + γ I t ) ]

7 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 652 I (q) t I (q) t θ t (1 θ t ). 3 Extension the model for second order Markov chains The second order Markov model for times t 2, t 1 and t is considered. The related transition matrix is shown by p 000 p p 001 p 001 p P 100 p 101 p 010 p p 101 p p 011 p 011 p 110 p p 111 p 111 where p ljit Pr(Y t i Y t 2 l, Y t 1 j); i, j, l 0, 1,t 1,..., T. Marginal mean is defined by θ t E(Y t ), which is θ t E(Y t ) Pr(Y t 1) Pr(Y t 1 Y t 2 0, Y t 1 0) Pr(Y t 2 0, Y t 1 0) + Pr(Y t 1 Y t 2 1, Y t 1 0) Pr(Y t 2 1, Y t 1 0) + Pr(Y t 1 Y t 2 0, Y t 1 1) Pr(Y t 2 0, Y t 1 1) + Pr(Y t 1 Y t 2 1, Y t 1 1) Pr(Y t 2 1, Y t 1 1) θ 001t + θ 101t + θ 011t + θ 111t where, θ ljit E(Y ljit ); i, j, l 0, 1 are called conditional means. For an available sequence observed data, y 1,..., y T, the likelihood function can be written as L i0 j0 l0 p y ljit ljit Thus, the log-likelihood function is (1 p lj1t ) 1 y lj1t p y lj1t lj1t. t0 j0 l0 1 1 ln L {(1 y lj1t ) ln (1 p lj1t ) + y lj1t ln p lj1t j1 l1 1 1 p lj1t {y lj1t ln j1 l1 (1 p lj1t ) + ln (1 p lj1t)} 1 1 {y lj1t logit(p lj1t ) + ln (1 p lj1t )} j1 l1

8 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia {y lj1t logit(θ lj1t ) + ln (1 θ lj1t )}. j1 l1 The logit functions for conditional means are logit(θ lj1t ) χ lj1tβ lj1 ; l, j 0, 1. Hence, ln L {y 001t χ 001tβ 001 ln (1 + exp (χ 001tβ 001 ))} + {y 101t χ 101tβ 101 ln (1 + exp (χ 101tβ 101 ))} + {y 011t χ 011tβ 011 ln (1 + exp (χ 011tβ 011 ))} + {y 111t χ 111tβ 111 ln (1 + exp (χ 111tβ 111 ))} ln L 1 + ln L 2 + ln L 3 + ln L 4. The likelihood function for marginal model would be L i0 p y it it (1 p 1t ) 1 yt p yt 1t. t0 where, p it Pr(Y t i); i 0, 1. Then, the log-likelihood function is defined as p 1t ln L {(1 y t ) ln (1 p 1t ) + y t ln p 1t } {y t ln (1 p 1t ) + ln (1 p 1t)} {y t logit(p 1t ) + ln (1 p 1t )} {y t logit(θ t ) + ln (1 θ t )}. Via equation (2) ln L {y t χ tβ ln (1 + exp (χ tβ))}. The estimate of parameters can be computed from the following equation ln L β ln L ln p 001t ln θ 001t + ln L ln p 101t ln θ 101t p 001t θ 001t β p 101t θ 101t β + ln L ln p 011t ln θ 011t + ln L ln p 111t ln θ 111t p 011t θ 011t β p 111t θ 111t β 0.

9 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 654 Hence, ln L 1 β 001q ln L 2 β 101q ln L 3 β 011q ln L 4 β 111q {y 001t χ 001tq χ 001tq exp (χ 001tβ 001 )/(1 + exp (χ 001tβ 001 ))} 0, {y 101t χ 101tq χ 101tq exp (χ 101tβ 101 )/(1 + exp (χ 101tβ 101 ))} 0, {y 011t χ 011tq χ 011tq exp (χ 011tβ 011 )/(1 + exp (χ 011tβ 011 ))} 0, {y 111t χ 111tq χ 111tq exp (χ 111tβ 111 )/(1 + exp (χ 111tβ 111 ))} 0, Model (5) can be used for testing of stationarity on the second order Markov model with the null hypothesis H 0 : γ 1... γ G 0, and the test statistic is T Z V Z. 4 Example The proposed test procedure in this paper is applied for the Health and Retirement Study (HRS) data, which is about retirement and health among the elderly in the United States. The data were collected from 1992 to 2006 by the RAND Centre for people in 8 waves, for considering repeated measures. In this case, only individuals who attended to the program in 1992 and the follow up until 2006 have been considered. The study is about the affective factors of depression during the elderly. Depression (0 for no depress and 1 for depress) is considered as dependent variable, and age (in year), gender (0 for male and 1 for female), body mass index (BMI), and drink (0 for no drink and 1 for drink) as covariate variables. The space of covariate (χ 1,..., χ p ) is partitioned into 4 distinct regions, (male and no drink), (male and drink), (female and no drink), and (female and drink). Some of variables were contained missing values because reference person did not respond to the all waves. Thus, these individuals are dropped completely from studying if there were missing value in the covariate variables, but were kept if the value of dependent variable (depression) was missing. There were 668 missing values in the covariate variables that included 353 IDs, i.e. in these individuals there was respond for depression variable but not for covariate variables; so 353 IDs have dropped from data in this work. For estimating the parameters of model, S-Plus program which has been modified by Chowdhury et al. (2005), is developed and used. The result of estimation parameters and test statistics for first and second order of Markov chain model based on conditional probabilities is showed in Table 1

10 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 655 and 2. Table 3 shows the result for marginal model. Billingsley chi-square statistics is computed by equation (1), and Tsiatis statistics is estimated by equation (6). The results in table 3 show that data satisfy the model for first and second order Markov chain based on marginal distribution. The estimates of parameters for the first and second order transitions specify negative association between depression and age (non-significant) and drink (non-significant); and positive association with BMI and sex. 5 Conclusion In the previous works most of test procedures for stationarity and order of higher order Markov chain have been based on likelihood ratio test and usual chi-square test. In this paper a stationarity test for first and second order Markov chain model was developed based on marginal probabilities by considering repeated measures. It was an extension of Tsiatis test procedure for logistic regression models which is improved for Markov chain models by using logistic regression function. The test is also done by using Billingsley chi-square test. At this point, the Tsiatis test considered the efficient score test which only requires the estimate of parameters under the null hypothesis. The results of tests showed satisfied stationarity of the model. But the important points are, the estimate of Tsiatis statistic is easier than Billingsley statistic, the number of estimated parameters is smaller, and extension of the model for Tsiatis is easier. The utility of the proposed test has been examined with an example for real life data. The results indicate the suitability of the techniques. In addition the proposed test procedure can be extending for higher order and test of order of Markov chain. References 1. Anderson, T. W. and Goodman, L. A.: Statistical inference about Markov chains. The Annals of Mathematical Statistics 28, (1957) 2. Azzalini, A.: Logistic regression for autocorrelated data with application to repeated measures. Biometrika 81, (1994) 3. Bartlett, M. S.: The frequency goodness of fit test for probability chains. Proc. Camb. Phil. Soc. 47, 86 (1951) 4. Billingsley, P.: Statistical methods in Markov chains. The Annals of Mathematical Statistics 32, (1961) 5. Bonney, G. E.: Regressive logistic models for familial disease and other binary traits. Biometrics 42, (1986) 6. Chowdhury, R. I., Islam, M. A., Shah, M. A. and Al-Enezi, N.: A computer program to estimate the parameters of covariate dependence higher order Markov model. Computer Methods and Program in Biomedicine 77, (2005) 7. Cox, D. R.: The Analysis of Binary Data. London: Methuen (1970) 8. Hoel, G.: A test for Markoff chains. Biometrika 41, (1954) 9. Islam, M. A. and Chowdhury, R. I.: A higher order Markov model for analyzing covariate dependence. Applied Mathematical 30, (2006)

11 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia Islam, M. A., Chowdhury, R. I. and Huda, Shahriar: Markov Models with Covariate Dependence for Repeated Measures. New York: Nova Science (2008) 11. Muenz, L.R. and Rubinstein, L.V.: Markov models for covariate dependence of binary sequences. Biometrics 41, (1985) 12. Rao, C. R.: Linear Statistical Inference and its Applications. 2nd edition, New York: Wiley (1973) 13. Shafiqur M. Rahman and Islam, M. A.: Markov structure based logistic regression for repeated measures: An application to diabetes mellitus data. Statistical Methodology 4, (2007) 14. Tsiatis, Anastasios A.: A note on a goodness-of-fit test for the logistic regression model. Biometrika 67, (1980) Table 1. Transition counts of Markov chain of depression data. Transition 0 1 First order Second order

12 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 657 Table 2. Estimates of parameters of covariate-dependent Markov models for depression data based on conditional probabilities. Model ˆβ s.e. p-value First order 0 1 Constant Age Sex BMI Drink Constant Age Sex BMI Drink Billingsley Chi-square 3.94E Tsiatis test Second order Constant Age Sex BMI Drink Constant o.003 Age Sex BMI Drink Constant Age Sex BMI Drink Constant Age Sex BMI Drink Billingsley Chi-square 2.13E Tsiatis test

13 Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 658 Table 3. Estimates of parameters of covariate-dependent Markov models for depression data based on marginal probabilities. Model ˆβ s.e. p-value First order Constant Age Sex BMI Drink Billingsley Chi-square 4.217E Tsiatis test Second order Constant Age Sex BMI Drink Billingsley Chi-square 4.552E Tsiatis test