A note on a Marčenko-Pastur type theorem for time series. Jianfeng. Yao

Transcription

1 A note on a Marčenko-Pastur type theorem for time series Jianfeng Yao Workshop on High-dimensional statistics The University of Hong Kong, October 2011

2 Overview 1 High-dimensional data and the sample covariance matrix 2 Variations on the Marčenko-Pastur theorem 3 Seek for a similar theorem for time series Model and Main result Main idea of the proof Application to a ARMA process

3 1 High-dimensional data and the sample covariance matrix 2 Variations on the Marčenko-Pastur theorem 3 Seek for a similar theorem for time series Model and Main result Main idea of the proof Application to a ARMA process

4 The sample covariance matrix S n Consider a sequence of p-dimensional random vectors, x 1,..., x n with population covariance matrix Σ = Cov(x 1); The sample covariance matrix is S n = 1 n n x j x T j, or S n = 1 n j=1 n (x j x)(x j x) T. j=1 Classical theory: p is fixed and n, S n Σ, almost surely. In particular, the p (random) eigenvalues of S n converge to the eigenvalues of Σ. λ Sn 1 λ Sn p,

5 High-dimensional data Some typical data dimensions : data dimension p sample size n c = p/n portfolio climate survey speech analysis a 10 2 b ORL face data base micro-arrays Important: the dimension-to-sample ratio c = p/n is not always close to 0 ; frequently larger than 1

6 An effect of high-dimensions: S n Σ (x i ) N p(0, I p) i.i.d.; Σ = I p; p = 40, n = 160, c = p/n = Warning:!! λ Sn j 1!! Blue curve: MP law of index Répartition des v.p., p=40, n= Histogram of eigenvalues {λ Sn j } of S n

7 Répartition des v.p., p=40, n=320 p = 40, n = 320, c = p/n = 0.125, Blue curve: MP law of index Histogram of eigenvalues of S n

8 The Marčenko-Pastur distributions Theorem. (a simplified version) Assume : Marčenko & Pastur, 1967 X = (X ij ) = (x 1,..., x n) be a p n array of i.i.d. variables (0, 1), (so we have Σ = Cov(x 1) = I p); not necessarily Gaussian, but with finite 4-th moment; p, n, p/n c (0, 1]; Then, the empirical distribution of the eigenvalues of S n = 1 n XXT, p F n = 1 p j=1 δ λ Sn j converges (in probability) to the distribution with density function where f (x) = 1 (x a)(b x), a x b, 2πcx a = (1 c) 2, b = (1 + c) 2.

9 The Marčenko-Pastur distribution f (x) = 1 (x a)(b x), (1 c) 2 = a x b = (1 + c) 2. 2πcx Densités de la loi de Marcenko Pastur 1.0 c p/n a b 1/ / / c=1/4 c=1/8 c=1/2

10 1 High-dimensional data and the sample covariance matrix 2 Variations on the Marčenko-Pastur theorem 3 Seek for a similar theorem for time series Model and Main result Main idea of the proof Application to a ARMA process

11 Marčenko-Pastur s original version; Silverstein 1995 s version; Bai and Zhou 2008 s version.

12 Marčenko-Pastur 1967

13 8VERWPEXIHERHTYFPMWLIHF]%17

14 Marčenko-Pastur Cont d Consider where (τ i ) are i.i.d. real ; n B p = A p + τ i x i x i, i=1 A p is non random and Hermitian; (x i ) is a sequence of independent complex-valued vectors; from (τ i ); independent

15 Marčenko-Pastur Cont d Assume I. Concentration : n/p c > 0; II. The ESD of A p tends to a (possibly defective) nonrandom measure ν 0; III. The coordinates (q 1,..., q p) of each (column) vector x i satisfy 1 E[q k ] = 0, E q k 2 = 1, E q k 4 < ; 2 E[q i q j ] = p 1 δ ij + a ij (p), E[q i q j q l q m] = p 2 [δ ij δ lm + δ im δ jl ] + ϕ il (p)ϕ jm (p) + b ijlm (p), with p ij 1/2 a ij (p) 2 0, ϕ ij (p) 2 0, p 1/2 b ijlm (p) 2 0. ij ijlm IV. (τ i ) are i.i.d. with distribution function H(x) Note. Independent columns with nearly uncorrelated components.

16 Marčenko-Pastur Cont d Then, the ESD function of B p converges (in probability) to a non decreasing function ν(λ) at all points of continuity; its Stieldjes transform s(z) is the unique solution in the region {I(z) > 0} to the equation ) τ s(z) = s 0 (z c 1 + τs(z) dh(τ), where s 0 is the ST of ν 0 (LSD of (A p)). Special cases: 1 Assume A p 0, then s 0(z) = 1/z, z = 1 s(z) + c τ 1 + τs(z) dh(τ). 2 If moreover, τ i 1, then H = δ 1, z = 1 s(z) s(z), which can be solved and leads to the MP distribution with index c.

17 The paper in MathScinet: Silverstein 1995

18 The paper: Silverstein Cont d

19 And the theorem: Silverstein 1995 s theorem

20 Silverstein versus Marčenko-Pastur Silverstein: Write X n = (x 1,..., x N ) and set y j = Tn 1/2 x j, then Silverstein s matrix B n is B n = 1 N T n 1/2 X nxn Tn 1/2 = 1 N N y j yj, i.e. the SCM of i.i.d. random vectors (y j ) with a special structure; = : more general correlations T n within the coordinates The convergence of the ESD is stronger (almost sure); Matrix entries are square-integrable only. j=1 Marčenko-Pastur: The final equations appeared already in Marčenko-Pastur; also those matrices have an extra additional term A n. (Note however there is a similar paper by Silverstein and Bai where this additional matrix A n is present).

21 Yet another MP type theorem The problem with Silverstein s theorem is that various within-components correlations could not be always put into the form where x j has i.i.d. coordinates. y j = T 1/2 n x j, Unfortunately, random vectors are not all Gaussian...

22 Bai and Zhou 2008 theorem

23

24 Note. Independent columns; and 1). specifies a very general correlation pattern within the components.

25 1 High-dimensional data and the sample covariance matrix 2 Variations on the Marčenko-Pastur theorem 3 Seek for a similar theorem for time series Model and Main result Main idea of the proof Application to a ARMA process

26 What s next? What about dependent columns, for example a p-dimensional time series, x 1,..., x n, with stationary population covariance matrix Σ = Cov(x t)? The problem is open for linear time series with general Σ. A solution below for linear time series with Σ = I p.

27 SP 500 daily stock prices ; p = 488 stocks; An example n = 1000 daily returns r t(i) = log p t(i)/p t 1(i) from to ;

28 The sample correlation matrix Let the SCM ( ) S n = 1 n n (r t r)(r t r) T. t=1 We consider the sample correlation matrix R n with R n(i, j) = S n(i, j) [S n(i, i)s n(j, j)] 1/2. The 10 largest and 10 smallest eigenvalues of R n are:

29 Sample eigenvalues of stock returns The important questions here are: Give an explanation for the largest eigenvalues (using theory of spikes, factor models) ; Find the (population) correlation structure between the 488 returns from the bulk eigenvalues.

30 A Marčenko-Pastur type theorem for time series A very special case of time series; The ST of the LSD characterised by an equation depending on the spectral density of the time series.

31 The model consider an univariate real-valued linear process z t = φ k ε t k, t Z, (1) k=0 where (ε k ) is a real-valued i. i. d. noise with mean zero and variance 1. The p-dimensional process (X t) considered in this paper will be made by p independent copies of the linear process (z t), i.e. for X t = (X 1t,..., X pt), X it = φ k ε i,t k, t Z, k=0 where the p coordinate processes {(ε 1,t,..., ε p,t)} are independent copies of the univariate error process {ε t} in (1).

32 The SCM S n Let X 1,..., X n be the observations of the time series; we consider the ESD of the SCM S n = 1 n n X j X j. (2) j=1 Note. A previous paper by Jing at al. (2009) considers a similar question, but our result here is much more general (although not enough!).

33 Theorem Assume that the following conditions hold: 1 The dimensions p, n and p/n c (0, ); 2 The error process has a fourth moment: Eε 4 t < ; 3 The linear filter (φ k ) is absolutely summable, i.e. φ k <. k=0 Y. (2011) Then almost surely the ESD of S n tends to a non-random probability distribution F. Moreover, the Stieltjes transform s = s(z) of F (as a mapping from C + into C + ) satisfies the equation z = 1 s + 1 2π 1 dλ, (3) 2π 0 cs + {2πf (λ)} 1 where f (λ) is the spectral density of the linear process (z t): f (λ) = 1 2 φ k e ikλ, λ [0, 2π). (4) 2π k=0

34 Main idea of the proof The data matrix (X 1,..., X n) have p i.i.d. rows; the correlations in a row are the autocovariances of the base linear process (z t): for all 1 i p, Cov(X is, X it ) = Cov(z s, z t) = γ t s, 1 s, t n. We can then adapt Bai and Zhou s theorem to this case with an evaluation of these autocovariances and exchanging the roles of columns/rows.

35 Application of Bai and Zhou s theorem It remains to evaluate the LSD, say H, of the (deterministic) covariance matrix T n of each coordinate process (X i1,..., X in ); This equals to the n-th order Toeplitz matrix associated to f = 2πf : T n(s, t) = γ t s, 1 s, t n, and f (λ) = k= γ k e ikλ, λ [0, 2π).

36 Cont d Assume that we can apply Bai and Zhou s theorem: first we identify the ESD H of the Toeplitz matrices T n; second, the ESD of n Sn converges a.s nonrandom probability distribution p whose Stieltjes transform m solves the equation z = 1 m + 1 c x 1 + mx dh(x). Need a theorem from Gabor Szegö to compute the last integral.

37 A theorem of Szegö The Fourier coefficients (γ k ) of the function f are absolutely summable; f is smooth with defined minimum a and maximum b on [0, 2π]; By the fundamental eigenvalue distribution theorem of Szegö for Toeplitz matrices: for any function ϕ continuous on [a, b] and denoting the eigenvalues of T n by σ (n) 1,..., σ(n) n, it holds that 1 lim n n n k=1 ϕ(σ (n) k ) = 1 2π 2π 0 ϕ( f (λ))dλ.

38 Cont d Consequently, the ESD of T n weakly converges to a nonrandom distribution H with support [a, b] and we have 0 ϕ(x)dh(x) = 1 2π 2π 0 ϕ( f (λ))dλ. (5) Hence we get z = 1 m + 1 c = 1 m + 1 2πc x 1 + mx dh(x) 2π 0 1 m + 1/ f dλ. The final equation is obtained by observing the relation s(z) = 1 c m(z/c).

39 Application to p-dimensional ARMA(1,1) series The base process is z t = φz t 1 + ε t + θε t 1, t Z, where φ < 1 and θ is real. The general equation (3) for ST reduces to with z = 1 s + θ csθ φ (φ + θ)(1 + φθ) (csθ φ) 2 ɛ(α) α2 4. (6) α = cs(1 + θ2 ) φ 2, ɛ(α) = sgn(iα). (7) csθ φ

40 Density plots of the LSD phi=0.4 theta=0 c=0.2 phi=0.4 theta=0.2 c= phi=0.4 theta=0.6 c= phi=0.8 theta=0.2 c= Densities of the LSD from ARMA(1,1) model. Left to right and top to bottom: (φ, θ, c) = (0.4, 0, 0.2) (0.4, 0.2, 0.2) (0.4, 0.6, 0.2) (0.8, 0.2, 0.2).

41 Conclusions Marčenko-Pastur theorems are a fundamental tool for the understanding of the spectral distributions of large sample covariance matrices; For dependent observations (time series for example), much work need to be done; existing results are partial answers only.