Cleaning correlation matrices, Random Matrix Theory & HCIZ integrals

Transcription

1 Cleaning correlation matrices, Random Matrix Theory & HCIZ integrals J.P Bouchaud with: M. Potters, L. Laloux, R. Allez, J. Bun, S. Majumdar

2 Portfolio theory: Basics Portfolio weights w i, Asset returns X t i If expected/predicted gains are g i then the expected gain of the portfolio is G = i w i g i Let risk be defined as: variance of the portfolio returns (maybe not a good definition!) R 2 = ij w i σ i C ij σ j w j where σ 2 i is the variance of asset i, and C ij is the correlation matrix.

3 Markowitz Optimization Find the portfolio with maximum expected return for a given risk or equivalently, minimum risk for a given return (G) In matrix notation: w C = G C 1 g g T C 1 g where all gains are measured with respect to the risk-free rate and σ i = 1 (absorbed in g i ). Note: in the presence of non-linear contraints, e.g. i w i A a spin-glass problem! (see [JPB,Galluccio,Potters])

4 Markowitz Optimization More explicitly: w α λ 1 α (Ψ α g)ψ α = g + α (λ 1 α 1)(Ψ α g)ψ α Compared to the naive allocation w g: Eigenvectors with λ 1 are projected out Eigenvectors with λ 1 are overallocated Very important for stat. arb. strategies (for example)

5 Empirical Correlation Matrix Before inverting them, how should one estimate/clean correlation matrices? Empirical Equal-Time Correlation Matrix E E ij = 1 T t X t i Xt j σ i σ j Order N 2 quantities estimated with NT datapoints. When T < N, E is not even invertible. Typically: N = ; T = days (10 years Beware of high frequencies) q := N/T = O(1)

6 Risk of Optimized Portfolios In-sample risk (for G = 1): R 2 in = wt E Ew E = 1 g T E 1 g True minimal risk R 2 true = wt C Cw C = 1 g T C 1 g Out-of-sample risk R 2 out = wt E Cw E = gt E 1 CE 1 g (g T E 1 g) 2

7 Risk of Optimized Portfolios Let E be a noisy, unbiased estimator of C. Using convexity arguments, and for large matrices: R 2 in R2 true R2 out In fact, using RMT: R 2 out = R2 true (1 q) 1 = R 2 in (1 q) 2, indep. of C! (For large N) If C has some time dependence (beyond observation noise) one expects an even worse underestimation

8 In Sample vs. Out of Sample 150 Return Raw in-sample Cleaned in-sample Cleaned out-of-sample Raw out-of-sample Risk

9 Rotational invariance hypothesis (RIH) In the absence of any cogent prior on the eigenvectors, one can assume that C is a member of a Rotationally Invariant Ensemble RIH Surely not true for the market mode v 1 (1,1,...,1)/ N, with λ 1 Nρ but OK in the bulk (see below) A more plausible assumption: factor model hierarchical, block diagonal C s ( Parisi matrices ) Cleaning E within RIH: keep the eigenvectors, play with eigenvalues The simplest, classical scheme, shrinkage: C = (1 α)e + αi λ C = (1 α)λ E + α, α [0,1]

10 RMT: from ρ C (λ) to ρ E (λ) Solution using different techniques (replicas, diagrams, free matrices) gives the resolvent G E (z) = N 1 Tr(E zi) as: G E (z) = Note: One should work from ρ C G E 1 dλ ρ C (λ) z λ(1 q + qzg E (z)), Example 1: C = I (null hypothesis) Marcenko-Pastur [67] (λ+ λ)(λ λ ) ρ E (λ) = 2πqλ, λ [(1 q) 2,(1 + q) 2 ] Suggests a second cleaning scheme (Eigenvalue clipping, [Laloux et al. 1997]): any eigenvalue beyond the Marcenko-Pastur edge can be trusted, the rest is noise.

11 Eigenvalue clipping λ < λ + are replaced by a unique one, so as to preserve TrC = N.

12 RMT: from ρ C (λ) to ρ E (λ) Solution using different techniques (replicas, diagrams, free matrices) gives the resolvent G E (z) as: G E (z) = Note: One should work from ρ C G E 1 dλ ρ C (λ) z λ(1 q + qzg E (z)), Example 2: Power-law spectrum (motivated by data) ρ C (λ) = µa (λ λ 0 ) 1+µΘ(λ λ min) Suggests a third cleaning scheme (Eigenvalue substitution, Potters et al. 2009, El Karoui 2010): λ E is replaced by the theoretical λ C with the same rank k

13 Empirical Correlation Matrix ρ(λ) Data Dressed power law (µ=2) Raw power law (µ=2) Marcenko-Pastur κ rank λ MP and generalized MP fits of the spectrum

14 Eigenvalue cleaning Classical Shrinkage Ledoit-Wolf Shrinkage Power Law Substitution Eigenvalue Clipping 2 R α Out-of sample risk for different 1-parameter cleaning schemes

15 A RIH Bayesian approach All the above schemes lack a rigorous framework and are at best ad-hoc recipes A Bayesian framework: suppose C belongs to a RIE, with P(C) and assume Gaussian returns. Then one needs: C X t = i with DCCP(C {X t i }) P(C {Xi t }) = Z 1 exp [ NTrV (C, {Xi t })] ; where (Bayes): V (C, {X t i }) = 1 2q [ logc + EC 1 ] + V 0 (C)

16 A Bayesian approach: a fully soluble case V 0 (C) = (1 + b)lnc + bc 1, b > 0: Inverse Wishart ρ C (λ) (λ+ λ)(λ λ ) λ 2 ; λ ± = (1 + b ± (1 + b) 2 b 2 /4)/b In this case, the matrix integral can be done, leading exactly to the Shrinkage recipe, with α = f(b, q) Note that b can be determined from the empirical spectrum of E, using the generalized MP formula

17 The general case: HCIZ integrals A Coulomb gas approach: integrate over the orthogonal group C = OΛO, where Λ is diagonal. DOexp [ N2q Tr [ logλ + EO Λ 1 O + 2qV 0 (Λ) ]] Can one obtain a large N estimate of the HCIZ integral F(ρ A, ρ B ) = lim N 2 [ ] N ln DOexp N 2q TrAO BO in terms of the spectrum of A and B?

18 The general case: HCIZ integrals Can one obtain a large N estimate of the HCIZ integral F(ρ A, ρ B ) = lim N 2 [ ] N ln DOexp N 2q TrAO BO in terms of the spectrum of A and B? When A (or B) is of finite rank, such a formula exists in terms of the R-transform of B [Marinari, Parisi & Ritort, 1995]. When the rank of A,B are of order N, there is a formula due to Matytsin [94](in the unitary case), later shown rigorously by Zeitouni & Guionnet, but its derivation is quite obscure...

19 An instanton approach to large N HCIZ Consider Dyson s Brownian motion matrices. The eigenvalues obey: dx i = 2 βn dw + 1 N dt j i 1 x i x j, Constrain x i (t = 0) = λ Ai and x i (t = 1) = λ Bi. The probability of such a path is given by a large deviation/instanton formula, with: d 2 x i dt 2 = 2 N 2 l =i 1 (x i x l ) 3.

20 An instanton approach to large N HCIZ Constrain x i (t = 0) = λ Ai and x i (t = 1) = λ Bi. The probability of such a path is given by a large deviation/instanton formula, with: d 2 x i dt 2 = 2 N 2 l =i 1 (x i x l ) 3. This can be interpreted as the motion of particles interacting through an attractive two-body potential φ(r) = (Nr) 2. Using the virial formula, one finally gets Matytsin s equations: t ρ + x [ρv] = 0, t v + v x v = π 2 ρ x ρ.

21 An instanton approach to large N HCIZ Finally, the action associated to these trajectories is: S 1 2 dxρ [ v 2 + π2 3 ρ2 ] 1 2 [ dxdyρ Z (x)ρ Z (y)ln x y ] Z=B Z=A Now, the link with HCIZ comes from noticing that the propagator of the Brownian motion in matrix space is: P(B A) exp [ N 2 Tr(A B)2 ] = exp N 2 [TrA2 +TrB 2 2TrAOBO ] Disregarding the eigenvectors of B (i.e. integrating over O) leads to another expression for P(λ Bi λ Aj ) in terms of HCIZ that can be compared to the one using instantons The final result for F(ρ A, ρ B ) is exactly Matytsin s expression, up to details (!)

22 Back to eigenvalue cleaning... Estimating HCIZ at large N is only the first step, but......one still needs to apply it to B = C 1, A = E = X CX and to compute also correlation functions such as with the HCIZ weight O 2 ij E C 1 As we were working on this we discovered the work of Ledoit- Péché that solves the problem exactly using tools from RMT...

23 The Ledoit-Péché magic formula The Ledoit-Péché [2011] formula is a non-linear shrinkage, given by: λ C = λ E 1 q + qλ E lim ǫ 0 G E (λ E iǫ) 2. Note 1: Independent of C: only G E is needed (and is observable)! Note 2: When applied to the case where C is inverse Wishart, this gives again the linear shrinkage Note 3: Still to be done: reobtain these results using the HCIZ route (many interesting intermediate results to hope for!)

24 Eigenvalue cleaning: Ledoit-Péché Fit of the empirical distribution with V 0 (z) = a/z+b/z2 +c/z 3.

25 What about eigenvectors? Up to now, most results using RMT focus on eigenvalues What about eigenvectors? What natural null-hypothesis beyond RIH? Are eigen-values/eigen-directions stable in time? Important source of risk for market/sector neutral portfolios: a sudden/gradual rotation of the top eigenvectors!..a little movie...

26 What about eigenvectors? Correlation matrices need a certain time T to be measured Even if the true C is fixed, its empirical determination fluctuates: E t = C + noise What is the dynamics of the empirical eigenvectors induced by measurement noise? Can one detect a genuine evolution of these eigenvectors beyond noise effects?

27 What about eigenvectors? More generally, can one say something about the eigenvectors of randomly perturbed matrices: H = H 0 + ǫh 1 where H 0 is deterministic or random (e.g. GOE) and H 1 random.

28 Eigenvectors exchange An issue: upon pseudo-collisions of eigenvectors, eigenvalues exchange Example: 2 2 matrices H 11 = a, H 22 = a + ǫ, H 21 = H 12 = c, λ ± ǫ 0 a + ǫ 2 ± c 2 + ǫ2 4 Let c vary: quasi-crossing for c 0, with an exchange of the top eigenvector: (1, 1) (1, 1) For large matrices, these exchanges are extremely numerous labelling problem

29 Subspace stability An idea: follow the subspace spanned by P-eigenvectors: ψ k+1, ψ k+2,... ψ k+p ψ k+1, ψ k+2,... ψ k+p Form the P P matrix of scalar products: G ij = ψ k+i ψ k+j The determinant of this matrix is insensitive to label permutations and is a measure of the overlap between the two P-dimensional subspaces D = P 1 ln detg is a measure of how well the first subspace can be approximated by the second

30 Intermezzo Non equal time correlation matrices E τ ij = 1 T t X t i Xt+τ j σ i σ j N N but not symmetrical: leader-lagger relations General rectangular correlation matrices G αi = 1 T T t=1 Y t αx t i N input factors X; M output factors Y Example: Yα t = Xt+τ j, N = M

31 Intermezzo: Singular values Singular values: Square root of the non zero eigenvalues of GG T or G T G, with associated eigenvectors u k α and vk i 1 s 1 > s 2 >...s (M,N) 0 Interpretation: k = 1: best linear combination of input variables with weights vi 1, to optimally predict the linear combination of output variables with weights u 1 α, with a crosscorrelation = s 1. s 1 : measure of the predictive power of the set of Xs with respect to Y s Other singular values: orthogonal, less predictive, linear combinations

32 Intermezzo: Benchmark Null hypothesis: No correlations between Xs and Y s: G true 0 But arbitrary correlations among Xs, C X, and Y s, C Y, are possible Consider exact normalized principal components for the sample variables Xs and Y s: and define Ĝ = Ŷ ˆX T. ˆX t i = 1 λi j U ij X t j ; Ŷ t α =...

33 Intermezzo: Random SVD Final result:([wachter] (1980); [Laloux,Miceli,Potters,JPB]) ρ(s) = (m + n 1) + (s 2 γ )(γ + s 2 ) δ(s 1) + πs(1 s 2 ) with γ ± = n + m 2mn ± 2 mn(1 n)(1 m), 0 γ ± 1 Analogue of the Marcenko-Pastur result for rectangular correlation matrices Many applications; finance, econometrics ( large models), genomics, etc. and subspace stability!

34 Back to eigenvectors Extend the target subspace to avoid edge effects: ψ k+1, ψ k+2,... ψ k+p ψ k Q+1, ψ k+2,... ψ k+q Form the P Q matrix of scalar products: G ij = ψ k+i ψ k+j The singular values of G indicates how well the Q perturbed vectors approximate the initial ones D = 1 P i ln s i

35 Null hypothesis Note: if P and Q are large, D can be accidentally small One can compute D exactly in the limit P, Q, N, with fixed p = P/N, q = Q/N: Final result: (same problem as above!) with: and ρ(s) = D = 1 0 dsln s ρ(s) (s 2 γ )(γ + s 2 ) πs(1 s 2 ) γ ± = p + q 2pq ± 2 pq(1 p)(1 q), 0 γ ± 1

36 Back to eigenvectors: perturbation theory Consider a randomly perturbed matrix: H = H 0 + ǫh 1 Perturbation theory to second order in ǫ yields: D ǫ2 2P i {k+1,...,k+p } j {k Q+1,...,k+Q} ( ψi H 1 ψ j λ i λ j ) 2. The full distribution of s can again be computed exactly (in some limits) using free random matrix tools.

37 GOE: the full SV spectrum Initial eigenspace: spanned by [a, b] [ 2, 2], b a = Target eigenspace: spanned by [a δ, b + δ] [ 2,2] Two cases (set s = ǫ 2 ŝ): Weak fluctuations δ 1 ρ(ŝ) is a semi circle centered around δ 1, of width Strong fluctuations δ 1 ρ(ŝ) ŝ min /ŝ 2, with ŝ min 1 and ŝ max δ 1.

38 The case of correlation matrices Consider the empirical correlation matrix: E = C + η η = 1 T T t=1 (X t X t C) The noise η is correlated as: ηij η kl = 1 T (C ikc jl + C il C jk ) from which one derives: D 1 2TP P N i=1 j=q+1 λ i λ j (λ i λ j ) 2 (and a similar equation for eigenvalues).

39 Stability of eigenvalues: Correlations Eigenvalues clearly change: well known correlation crises

40 Stability of eigenspaces: Correlations D(τ) for a given T, P = 5, Q = 10

41 Stability of eigenspaces: Correlations D(τ = T) for P = 5, Q = 10

42 Conclusion Many RMT tools available to understand the eigenvalue spectrum and suggest cleaning schemes The understanding of eigenvectors is comparatively poorer The dynamics of the top eigenvector (aka market mode) is relatively well understood A plausible, realistic model for the true evolution of C is still lacking (many crazy attempts Multivariate GARCH, BEKK, etc., but second generation models are on their way)

43 Bibliography J.P. Bouchaud, M. Potters, Financial Applications of Random Matrix Theory: a short review, in The Oxford Handbook of Random Matrix Theory (2011) R. Allez and, Eigenvectors dynamics: general theory & some applications, arxiv P.-A. Reigneron, R. Allez and, Principal regression analysis and the index leverage effect, Physica A, Volume 390 (2011)