Separable covariance arrays via the Tucker product - Final

Transcription

1 Separable covariance arrays via the Tucker product - Final by P. Hoff Kean Ming Tan June 4, / 28

2 International Trade Data set Yearly change in log trade value (in 2000 dollars): Y = {y i,j,k,t } i {1,..., 30} indexes the exporting nation j {1,..., 30} indexes the importing nation k {1,..., 6} indexes the commodity type t {1,..., 10} indexes the year Interested in modeling the mean M ijk = µ i,j,k across t measurements What can we do? 2 / 28

3 International Trade data set Interested in the model Y = M 1 n + E iid error model: ɛ i,j,k,l normal(0, σ 2 ) multivariate error model: ɛ i,j,k multivariate normal(0, Σ) matrix-variate error model: ɛ i,j matrix normal(0, Σ 1, Σ 2 ) But all four dimensions are correlated! E??? 3 / 28

4 Definition outline Vectorize Inner Product Matricization Array-matrix Product 4 / 28

5 Definition - k-mode product of an array Suppose Y R m 1 m 2 m 3 and A R m 1 m 1, the 1-mode product is Z = Y 1 A if and only if Z (1) = AY (1) Array-matrix multiplication: Y 1 A 1. Matricization: Y (1) R m 1 m 2 m 3 2. Multiply: AY (1) 3. Reform: Y 1 A = array(vec(ay (1) ), m 1, m 2, m 3 ) 5 / 28

6 Remember this picture? We managed to find the Array Normal Model Y array normal(m, Σ 1,..., Σ K ) 6 / 28

7 Array normal model 1 Array normal model, y R m 1 m 2 m 3 : Z = {z i,j,k } iid normal(0, 1) Y = M + Z {A, B, C} iid array normal(m, Σ 1 = AA T, Σ 2 = BB T, Σ 3 = CC T ) array normal(m, Σ 1 Σ 2 Σ 3 ) 1 Adapted from Hoff s slides 7 / 28

8 Probability density function Multivariate normal (2π) p/2 Σ 1/2 exp( (y µ) T Σ 1 (y µ)/2) Array normal: Note that Σ = {Σ 1, Σ 2, Σ 3 } 3 (2π) m/2 ( Σ k m/(2mk) )exp( (Y M) Σ 1/2 2 /2) k=1 where m = m 1 m 2 m 3 8 / 28

9 Estimation: Frequentist ˆM = Ȳ = Y i /n Let E = Y Ȳ 1 n and repeat the following for k = 1, 2, 3 Compute Ẽ = E {Σ 1/2 1, I mk, Σ 1/2 3, I n } and S k = Ẽ(k)ẼT (k) Σ k = S k /n k where n k = n j k m j Analogy for univariate case σ 2 = (y ȳ) 2 /n 9 / 28

10 Estimation: Bayesian Assume the following prior: M Σ anorm(m 0, Σ 1 Σ 2 Σ 3 /κ 0 ) Σ k inverse-wishart(s 1 0k, v 0k) After A LOT of calculation, the posteriors are M Y, Σ anorm([κ 0 M 0 +nȳ]/[κ 0 +n], Σ 1 Σ 2 Σ 3 /[κ 0 +n]) Σ k Y, Σ k inverse-wishart([s 0k +S k +R (k) R T (k) ] 1, v 0k +n k ) where R = κ0 n κ 0 +n (Ȳ M 0) {Σ 1/2 1, I mk, Σ 1/2 3 } 10 / 28

11 Finally!!! We model this data set! Want to model Y = M 1 n + E Understand correlation among exporters, importers and commodities of E Consider two models: 1. Y anorm(m 1 n, I m1 I m2 Σ 3 Σ 4 ) 2. Y anorm(m 1 n, Σ 1 Σ 2 Σ 3 Σ 4 ) 11 / 28

12 MCMC 205,000 iterations, 5,000 as burn-in period Parameter values are kept only every 40th iteration We have 5,000 samples But samples are matrices, how to check for convergence? Jon Wakefield: Calculate a summary measure of the matrix, for instance, the determinant 12 / 28

13 Trace plots 13 / 28

14 Model Comparison Posterior predictive evaluation: t 1 (Y) = log I/m 1 log S 1 where S 1 = E (1) E T (1) and S 1 = S 1 /tr(s 1 ) Discrepancies between t 1 (Y) and t 1 (Ỹ) indicates model does not capture some aspects of the data 14 / 28

15 Results 15 / 28

16 Understand Correlation Understand relationship among exporters, importers, and commodities, respectively Analyze posterior mean estimates of Σ 1, Σ 2, and Σ 3 Plot first two eigenvectors of each estimate 16 / 28

17 Exporter 17 / 28

18 Importer 18 / 28

19 Commodity 19 / 28

20 Discussion & Critique - 1 Estimation of parameters: Bayesian approach preferable MLE might not exist Estimation of high-dimensional parameters benefits from regularization 20 / 28

21 Discussion & Critique - 1 cont Impose regularization on the inverse covariance matrix Σ 1 k l(y Σ) K k=1 n k 2 log Σ 1 k 1 2 i=1 As usual, ˆM = Ȳ Closed form solution for Σ 1 = {Σ 1 1,..., Σ 1 K } n K (Y i Ȳ) Σ 1/2 2 λ k Σ 1 k 1 DOES NOT EXIST!!! k=1 21 / 28

22 Discussion & Critique -1 cont How about an iterative approach? Given ˆM = Ȳ l(y Σ) Σ 1 k n k 2 log Σ 1 k 1 ( 2 tr Block coordinate descent approach! Σ 1 k Ẽ(k)ẼT (k) ) λ k Σ 1 k 1 22 / 28

23 Discussion & Critique - 2 Josh Keller: Isn t normality assumption too stringent for an array? 23 / 28

24 Discussion & Critique - 2 Josh Keller: Isn t normality assumption too stringent for an array? Normality assumption Central Limit Theorem, no worries:) 23 / 28

25 Discussion & Critique - 2 Josh Keller: Isn t normality assumption too stringent for an array? Normality assumption Central Limit Theorem, no worries:) Not quite true in array case Some other types of array models are clearly needed! 23 / 28

26 Discussion & Critique - 2 Working towards array t-distribution x R p has elliptical contoured distribution with mean µ and dispersion parameter Σ, call it E p (µ, Σ, h( )) ( ) f (x) = c n Σ 1/2 h (x µ) T Σ 1 (x µ) Special Cases: Multivariate normal distribution: h(z) = exp( 1 2 z) Multivariate t-distribution: Under some transformation and rotation 24 / 28

27 Discussion & Critique - 2 X R m 1 m K has array elliptical contoured distribution with mean M and covariance matrices Σ = {Σ 1,..., Σ K }, E m1,...,m K (M, Σ, h( )) f (X) = c n Special Cases: K k=1 Σ k m/2m k h( (Y M) Σ 1/2 ) 2 ) Array normal distribution: h(z) = exp( 1 2 Z) Array t-distribution:??? 25 / 28

28 Array t-distribution 26 / 28

29 Conclusion Construct the array normal distribution Model the International Trade Data Set 27 / 28

30 Questions? 28 / 28