Mean and covariance models for relational arrays

Size: px

Start display at page:

Download "Mean and covariance models for relational arrays"

Iris Hood
5 years ago
Views:

1 Mean and covariance models for relational arrays Peter Hoff Statistics, Biostatistics and the CSSS University of Washington

2 Outline Introduction and examples Models for multiway mean structure Models for separable covariance arrays

3 Array-valued data y i,j,k = jth measurement on ith subject under condition k (psychometrics) type-k relationship between i and j (relational data/network) relationship between i and j at time t (dynamic network) y 123 y 124 y 125 y 122 y 121

4 Array-valued data y i,j,k = jth measurement on ith subject under condition k (psychometrics) type-k relationship between i and j (relational data/network) relationship between i and j at time t (dynamic network) y 123 y 124 y 125 y 122 y 121

5 Array-valued data y i,j,k = jth measurement on ith subject under condition k (psychometrics) type-k relationship between i and j (relational data/network) relationship between i and j at time t (dynamic network) y 123 y 124 y 125 y 122 y 121

6 Dynamic network example Cold war cooperation and conflict 66 countries USA 8 years (1950,1955,..., 1980, 1985) y i,j,t =relation between i, j in year t also have data on gdp polity Y is a data array UKG Model for the mean structure of the data: Y M + E Task: Estimate a low-rank M ROK AUL NEW PHI EGY TAW THI TUR IND BUL HUN RUM CAN JOR SPN LEB FRN HON BRA GDR COL DEN NEP GFR SAF ETH ARG VEN IRQ BEL OMA COS ALB HAI DOMSAL NOR ITA NTH IRE AFG PAN LBR CHL CZE SWD GUA SAU SRI INS CUB NIC POR PER AUS ECU IRN GRC YUG MYA USR ISR PRK CHN

7 Dynamic network example Cold war cooperation and conflict 66 countries USA 8 years (1950,1955,..., 1980, 1985) y i,j,t =relation between i, j in year t also have data on gdp polity Y is a data array UKG Model for the mean structure of the data: Y M + E Task: Estimate a low-rank M ROK AUL NEW PHI EGY TAW THI TUR IND BUL HUN RUM CAN JOR SPN LEB FRN HON BRA GDR COL DEN NEP GFR SAF ETH ARG VEN IRQ BEL OMA COS ALB HAI DOMSAL NOR ITA NTH IRE AFG PAN LBR CHL CZE SWD GUA SAU SRI INS CUB NIC POR PER AUS ECU IRN GRC YUG MYA USR ISR PRK CHN

8 Dynamic network example Cold war cooperation and conflict 66 countries USA 8 years (1950,1955,..., 1980, 1985) y i,j,t =relation between i, j in year t also have data on gdp polity Y is a data array UKG Model for the mean structure of the data: Y M + E Task: Estimate a low-rank M ROK AUL NEW PHI EGY TAW THI TUR IND BUL HUN RUM CAN JOR SPN LEB FRN HON BRA GDR COL DEN NEP GFR SAF ETH ARG VEN IRQ BEL OMA COS ALB HAI DOMSAL NOR ITA NTH IRE AFG PAN LBR CHL CZE SWD GUA SAU SRI INS CUB NIC POR PER AUS ECU IRN GRC YUG MYA USR ISR PRK CHN

9 Multivariate relational data example Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Replications over time: Y = {Y 1,..., Y 10} Y t = M + E t M R , constant over time; E t R , changing over time. How should the covariance among {E 1,..., E 10} be described?

10 Multivariate relational data example Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Replications over time: Y = {Y 1,..., Y 10} Y t = M + E t M R , constant over time; E t R , changing over time. How should the covariance among {E 1,..., E 10} be described?

11 Reduced rank models for mean structure Y = Θ + E Θ contains the main features we hope to recover, E is patternless. Matrix decomposition: If Θ is a rank-r matrix, then RX RX RX θ i,j = u i, v j = u i,r v j,r Θ = u r v T r = u r v r r=1 r=1 r=1 Array decomposition: If Θ is a rank-r array, then RX RX θ i,j,k = u i, v j, w k = u i,r v j,r w k,r Θ = u r v r w r r=1 r=1 (PARAFAC: Harshman[1970], Kruskal[1976,1977], Harshman and Lundy[1984], Kruskal[1989])

12 Reduced rank models for mean structure Y = Θ + E Θ contains the main features we hope to recover, E is patternless. Matrix decomposition: If Θ is a rank-r matrix, then RX RX RX θ i,j = u i, v j = u i,r v j,r Θ = u r v T r = u r v r r=1 r=1 r=1 Array decomposition: If Θ is a rank-r array, then RX RX θ i,j,k = u i, v j, w k = u i,r v j,r w k,r Θ = u r v r w r r=1 r=1 (PARAFAC: Harshman[1970], Kruskal[1976,1977], Harshman and Lundy[1984], Kruskal[1989])

13 A model-based approach For a K-way array Y, u (k) 1,..., u(k) m k Y = Θ + E RX Θ = r=1 u (1) r u (K) r U (1),..., U (K) iid multivariate normal(µ k, Ψ k ), with {µ k, Ψ k, k = 1,..., K} to be estimated. Some motivation: shrinkage: Θ contains lots of parameters. hierarchical: covariance among columns of U (k) is identifiable. estimation: p(y U (1),..., U (K) ) multimodal, MCMC stochastic search adaptability: incorporate reduced rank arrays as a model component multilinear predictor in a GLM multilinear effects for regression parameters

14 A model-based approach For a K-way array Y, u (k) 1,..., u(k) m k Y = Θ + E RX Θ = r=1 u (1) r u (K) r U (1),..., U (K) iid multivariate normal(µ k, Ψ k ), with {µ k, Ψ k, k = 1,..., K} to be estimated. Some motivation: shrinkage: Θ contains lots of parameters. hierarchical: covariance among columns of U (k) is identifiable. estimation: p(y U (1),..., U (K) ) multimodal, MCMC stochastic search adaptability: incorporate reduced rank arrays as a model component multilinear predictor in a GLM multilinear effects for regression parameters

15 A model-based approach For a K-way array Y, u (k) 1,..., u(k) m k Y = Θ + E RX Θ = r=1 u (1) r u (K) r U (1),..., U (K) iid multivariate normal(µ k, Ψ k ), with {µ k, Ψ k, k = 1,..., K} to be estimated. Some motivation: shrinkage: Θ contains lots of parameters. hierarchical: covariance among columns of U (k) is identifiable. estimation: p(y U (1),..., U (K) ) multimodal, MCMC stochastic search adaptability: incorporate reduced rank arrays as a model component multilinear predictor in a GLM multilinear effects for regression parameters

16 A model-based approach For a K-way array Y, u (k) 1,..., u(k) m k Y = Θ + E RX Θ = r=1 u (1) r u (K) r U (1),..., U (K) iid multivariate normal(µ k, Ψ k ), with {µ k, Ψ k, k = 1,..., K} to be estimated. Some motivation: shrinkage: Θ contains lots of parameters. hierarchical: covariance among columns of U (k) is identifiable. estimation: p(y U (1),..., U (K) ) multimodal, MCMC stochastic search adaptability: incorporate reduced rank arrays as a model component multilinear predictor in a GLM multilinear effects for regression parameters

17 Simulation study K = 3, R = 4, (m 1, m 2, m 3) = (10, 8, 6) 1. Generate M, a random array of roughly full rank 2. Set Θ = ALS 4(M) 3. Set Y = Θ + E, {e i,j,k } iid normal(0, v(θ)/4). For each of 100 such simulated datasets, we obtain ˆΘ LS and ˆΘ HB. Questions: How well do ˆΘ LS and ˆΘ HB recover the truth Θ? represent Y?

18 Simulation study K = 3, R = 4, (m 1, m 2, m 3) = (10, 8, 6) 1. Generate M, a random array of roughly full rank 2. Set Θ = ALS 4(M) 3. Set Y = Θ + E, {e i,j,k } iid normal(0, v(θ)/4). For each of 100 such simulated datasets, we obtain ˆΘ LS and ˆΘ HB. Questions: How well do ˆΘ LS and ˆΘ HB recover the truth Θ? represent Y?

19 Simulation study: known rank least squares MSE Bayesian MSE mode mean least squares RSS Bayesian RSS

20 Simulation study: misspecified rank least squares hierarchical Bayes log RSS log MSE assumed rank assumed rank

21 Mean structure for dynamic networks {Y 1,..., Y T } = Y = M + E y 125 M = X u r v r w r y 124 What if each Y t is symmetric? y 123 M = X r u r u r w r, u r R n, w r R T y 122 y 121 (INDSCAL (Carroll and Chang, 1970)) Under this Model, M t = UΛ tu T M = {M 1,..., M T }, Λ t = diag(w 1,t,..., w R,t ).

22 Mean structure for dynamic networks {Y 1,..., Y T } = Y = M + E y 125 M = X u r v r w r y 124 What if each Y t is symmetric? y 123 M = X r u r u r w r, u r R n, w r R T y 122 y 121 (INDSCAL (Carroll and Chang, 1970)) Under this Model, M t = UΛ tu T M = {M 1,..., M T }, Λ t = diag(w 1,t,..., w R,t ).

23 Dynamic network example y i,j,t { 5, 4,..., +1, +2}, the level of military conflict/cooperation x i,j,t,1 = log gdp i + log gdp j, the sum of the log gdps of the two countries; x i,j,t,2 = (log gdp i ) (log gdp j ), the product of the log gdps; x i,j,t,3 = polity i polity j, where polity i { 1, 0, +1}; x i,j,t,4 = (polity i > 0) (polity j > 0). Model: y i,j,t = f (z i,j,t, c 5,..., c +2) = max{y : z i,j,t > c y } z i,j,t = β T x i,j,t + u T i Λ tu j + ɛ i,j,t Z = {z i,j,t β T x i,j,t } = UΛ tu T + E

24 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

25 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

26 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

27 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

28 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

29 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

30 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

31 Longitudinal network example u 2 USA UKG ROK AUL NEW PHI THI TUR CAN JOR EGY TAW IND BUL HUN RUM FRN SPN LEB HON COL ARG BEL BRA DEN GDR GFR SAF NEP ETH COS ALB DOM IRQ OMA VEN NOR ITA HAI SAL NTH AFGPAN LBR GUA IRE SWDCHL CZE SAU SRI INS POR CUB NIC PER AUS ECU IRN GRC YUG MYA ISR CHN PRK USR v v u

32 Covariance structure of multiple relational arrays Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Replications over time: Y = {Y 1,..., Y 10} Y t = M + E t M R , constant over time; E t R , changing over time. How should the covariance among {E 1,..., E 10} be described?

33 Separable covariance via Tucker products Y = Θ + E Decompose Θ using the Tucker decomposition (Tucker 1964,1966): RX SX TX θ i,j,k = z r,s,ta i,r b j,r c k,r r=1 s=1 t=1 Θ = Z {A, B, C} Z is the R S T core array A, B, C are R m 1, S m 2, T m 3 matrices. R, S and T are the 1-rank, 2-rank and 3-rank of Θ is array-matrix multiplication (De Lathauwer et al., 2000)

34 Separable covariance via Tucker products Multivariate normal model: z = {z j : j = 1,..., m} Matrix normal model: Z = {z i,j } m 1,m 2 i=1,j=1 iid normal(0, 1) y = µ + Az multivariate normal(µ, Σ = AA T ) iid normal(0, 1) Y = M + AZB T matrix normal(m, Σ 1 = AA T, Σ 2 = BB T ) NOTE: AZB T = Z {A, B} Array normal model: Z = {z i,j,k } m 1,m 2,m 3 i=1,j=1,k=1 iid normal(0, 1) Y = M + Z {A, B, C} array normal(m, Σ 1 = AA T, Σ 2 = BB T, Σ 3 = CC T )

35 Separable covariance via Tucker products Multivariate normal model: z = {z j : j = 1,..., m} Matrix normal model: Z = {z i,j } m 1,m 2 i=1,j=1 iid normal(0, 1) y = µ + Az multivariate normal(µ, Σ = AA T ) iid normal(0, 1) Y = M + AZB T matrix normal(m, Σ 1 = AA T, Σ 2 = BB T ) NOTE: AZB T = Z {A, B} Array normal model: Z = {z i,j,k } m 1,m 2,m 3 i=1,j=1,k=1 iid normal(0, 1) Y = M + Z {A, B, C} array normal(m, Σ 1 = AA T, Σ 2 = BB T, Σ 3 = CC T )

36 Separable covariance via Tucker products Multivariate normal model: z = {z j : j = 1,..., m} Matrix normal model: Z = {z i,j } m 1,m 2 i=1,j=1 iid normal(0, 1) y = µ + Az multivariate normal(µ, Σ = AA T ) iid normal(0, 1) Y = M + AZB T matrix normal(m, Σ 1 = AA T, Σ 2 = BB T ) NOTE: AZB T = Z {A, B} Array normal model: Z = {z i,j,k } m 1,m 2,m 3 i=1,j=1,k=1 iid normal(0, 1) Y = M + Z {A, B, C} array normal(m, Σ 1 = AA T, Σ 2 = BB T, Σ 3 = CC T )

37 Separable covariance structure For the matrix normal model: Cov[Y] = Σ 1 Σ 2 Cov[vec(Y)] = Σ 2 Σ 1 E[YY T ] = Σ 1 tr(σ 2) E[Y T Y] = Σ 2 tr(σ 1) For the array normal model: Cov[Y] = Σ 1 Σ 2 Σ 3 Cov[vec(Y)] = Σ K Σ 1 E[Y (k) Y T (k)] = Σ k Y tr(σ j ) j k

38 International trade example Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Full cell means model: y i,j,k,l = µ i,j,k + e i,j,k,l Let E = {e i,j,k,l } iid error model: E array normal(0, I, I, I, σ 2 I) vector normal error model: E array normal(0, I, I, Σ 3, I) matrix normal error model: E array normal(0, I, I, Σ 3, Σ 4) array normal model: E array normal(0, Σ 1, Σ 2, Σ 3, Σ 4}

39 International trade example Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Full cell means model: y i,j,k,l = µ i,j,k + e i,j,k,l Let E = {e i,j,k,l } iid error model: E array normal(0, I, I, I, σ 2 I) vector normal error model: E array normal(0, I, I, Σ 3, I) matrix normal error model: E array normal(0, I, I, Σ 3, Σ 4) array normal model: E array normal(0, Σ 1, Σ 2, Σ 3, Σ 4}

40 International trade example Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Full cell means model: y i,j,k,l = µ i,j,k + e i,j,k,l Let E = {e i,j,k,l } iid error model: E array normal(0, I, I, I, σ 2 I) vector normal error model: E array normal(0, I, I, Σ 3, I) matrix normal error model: E array normal(0, I, I, Σ 3, Σ 4) array normal model: E array normal(0, Σ 1, Σ 2, Σ 3, Σ 4}

41 International trade example Yearly change in log exports (2000 dollars) : Y = {y i,j,k,l } R i {1,..., 30} indexes exporting nation j {1,..., 30} indexes importing nation k {1,..., 6} indexes commodity l {1,..., 10} indexes year Full cell means model: y i,j,k,l = µ i,j,k + e i,j,k,l Let E = {e i,j,k,l } iid error model: E array normal(0, I, I, I, σ 2 I) vector normal error model: E array normal(0, I, I, Σ 3, I) matrix normal error model: E array normal(0, I, I, Σ 3, Σ 4) array normal model: E array normal(0, Σ 1, Σ 2, Σ 3, Σ 4}

42 International trade example Model comparison: reduced: array normal(0, I, I, Σ 3, Σ 4) full: array normal(0, Σ 1, Σ 2, Σ 3, Σ 4) density density density t 1(Y) t 2(Y) t 3(Y)

43 International trade example IDN MYS THA SINKOR CHN HKG MEX JPN BRATUR USA NZL AUS IRE CZE NLD ITA NOR GRC CAN SWI DEN SPN UKG FRNDEU SWE FINAUT IRE GRC AUT SPN FRN DEN ITA SWE DEU FIN UKG MEX CANNLD NOR SWI AUS CZE USA BRA TUR NZL CHN HKG SIN JPN IDN THA MYS KOR machinery mfg goods food textiles chemicals crude materials

44 Discussion Scientific studies increasingly involve data with multiway array structure often this structure is unrecognized array structure may be present in data, latent data, or parameters Model-based versions of array decompositions offer several benefits applicability of multiway methods is broadened parameter estimates have reduced MSE Applicability to dynamic, multivariate relational data reduced-rank methods can capture mean structure array-matrix transformations can model covariance

45 Discussion Scientific studies increasingly involve data with multiway array structure often this structure is unrecognized array structure may be present in data, latent data, or parameters Model-based versions of array decompositions offer several benefits applicability of multiway methods is broadened parameter estimates have reduced MSE Applicability to dynamic, multivariate relational data reduced-rank methods can capture mean structure array-matrix transformations can model covariance

46 Discussion Scientific studies increasingly involve data with multiway array structure often this structure is unrecognized array structure may be present in data, latent data, or parameters Model-based versions of array decompositions offer several benefits applicability of multiway methods is broadened parameter estimates have reduced MSE Applicability to dynamic, multivariate relational data reduced-rank methods can capture mean structure array-matrix transformations can model covariance

Hierarchical models for multiway data

Hierarchical models for multiway data Peter Hoff Statistics, Biostatistics and the CSSS University of Washington Array-valued data y i,j,k = jth variable of ith subject under condition k (psychometrics).