A criterion-based PLS approach to SEM. Michel Tenenhaus (HEC Paris) Arthur Tenenhaus (SUPELEC)

Transcription

1 A criterion-based PLS approach to SEM Michel Tenenhaus (HEC Paris) Arthur Tenenhaus (SUPELEC)

2 Economic inequality and political instability Data from Russett (1964), in GIFI Economic inequality Agricultural inequality GINI : Inequality of land distributions FARM : % farmers that own half of the land (> 50) RENT : % farmers that rent all their land Industrial development GNPR : Gross national product per capita ($ 1955) LABO : % of labor force employed in agriculture Political instability INST : Instability of executive (45-61) ECKS : Nb of violent internal war incidents (46-61) DEAT : Nb of people killed as a result of civic group violence (50-62) D-STAB : Stable democracy D-UNST : Unstable democracy DICT : Dictatorship 2

3 Economic inequality and political instability (Data from Russett, 1964) Gini Farm Rent Gnpr Labo Inst Ecks Deat Demo Argentine Australie * Autriche France Yougoslavie = Stable democracy 2 = Unstable democracy 3 = Dictatorship 3

4 Agricultural inequality (X 1 ) A SEM model GINI INST FARM RENT ξ 1 C 12 = 0 C 13 = 1 ξ 3 ECKS DEAT D-STB GNPR LABO ξ 2 C 23 = 1 D-INS DICT Industrial development (X 2 ) Political instability (X 3 ) 4

5 Latent Variable outer estimation Y1 = X1w1 = w11gini + w12farm + w13rent Y2 = X 2w2 = w21gnpr + w22labo Y = X w = w INST + w ECKS + w DEATH w D-STB + w D-UNST w DICT 36 5

6 Some modified multi-block methods for SEM c jk = 1 if blocks are linked, 0 otherwise and c jj = 0 SUMCOR (Horst, 1961) SSQCOR (Mathes, 1993, Hanafi, 2004) SABSCOR (Mathes, 1993, Hanafi, 2004) MAXDIFF (Van de Geer, 1984) [SUMCOV] MAXDIFF B (Hanafi & Kiers, 2006) [SSQCOV] SABSCOV (Krämer, 2007) Max c Cor( X w, X w ) jk, jk, jk j j k k GENERALIZED CANONICAL CORRELATION ANALYSIS Max c Cor ( X w, X w ) jk, 2 jk j j k k Max c Cor( X w, X w ) All wj = 1 jk, jk j j k k Max c Cov( X w, X w ) GENERALIZED PLS REGRESSION All wj = 1 jk, jk j j k k Max c Cov ( X w, X w ) All wj = 1 jk, 2 jk j j k k Max c Cov( X w, X w ) jk j j k k 6

7 Covariance-based criteria for SEM c jk = 1 if blocks are linked, 0 otherwise and c jj = 0 SUMCOR-PLSPM SSQCOR-PLSPM SABSCOR-PLSPM SUMCOV-PLSPM SSQCOV-PLSPM SABSCOV-PLSPM Max c Cov( X w, X w ) All Var( X w ) = 1 j j jk, All Var( X w ) = 1 j j jk, jk j j k k Max c Cov ( X w, X w ) All Var( X w ) = 1 j j jk, 2 jk j j k k Max c Cov( X w, X w ) All = 1 jk, jk j j k k Max c Cov( X w, X w ) w j All = 1 jk, jk j j k k Max c Cov ( X w, X w ) w j All = 1 jk, 2 jk j j k k Max c Cov( X w, X w ) w j jk j j k k 7

8 A continuum approach j< k Maximize c g(cov( X w, X w )) subject to the constraints: 2 jk j j k k τ w + (1 τ ) Var( X w ) = 1, with 0 τ 1, i = 1,..., J i i i i i i where gx = x (Horst scheme) 2 ( ) x (Factorial scheme) x (Centroid scheme) 8

9 A general procedure to obtain critical points of the criteria Construct the Lagrangian function related to the optimization problem. Cancel the derivative of the Lagrangian function with respect to each w i. Use the Wold s procedure to solve the stationary equations ( Lohmöller s procedure). This procedure converges to a critical point of the criterion. The criterion increases at each step of the algorithm. 9

10 The general algorithm Y j = ± X j w j Initial step w j τ w Outer Estimation 2 + (1 τ ) Var( X w ) = 1 j j j j j Iterate until convergence Y 1 Y 2 Y J e j1 e j2 e jj Z j Inner estimation w j = 1 ' 1 ' [( τ ji+ (1 τ j) XX j j] XZ j j n 1 ZX[( τ I+ (1 τ ) XX] XZ n ' ' 1 ' j j j j j j j j Choice of weights e jh : - Horst : e jh = c jh - Centroid : e jh = c jh sign(cor(y h, Y j ) - Factorial : e jh = c jh Cov(Y h, Y j ) c 10 jh = 1 if blocks are linked, 0 otherwise and c jj = 0

11 Specific cases j< k Maximize c g(cov( X w, X w )) subject to the constraints: 2 jk j j k k τ w + (1 τ ) Var( X w ) = 1, with 0 τ 1, i = 1,..., J i i i i i Criterion SUMCOR SSQCOR SABSCOR Horst Factorial Centroid Scheme (g(x) = x) (g(x) = x 2 ) (g(x) = x ) Value of τ i i PLS Mode B With usual PLS-PM constraints: Var( X w ) = 1 i i 11

12 Specific cases j< k Maximize c g(cov( X w, X w )) subject to the constraints: 2 jk j j k k τ w + (1 τ ) Var( X w ) = 1, with 0 τ 1, i = 1,..., J i i i i i Criterion SUMCOV SSQCOV SABSCOV Horst Factorial Centroid Scheme (g(x) = x) (g(x) = x 2 ) (g(x) = x ) Value of τ i i PLS New Mode A With usual PLS regression constraint: w i = 1 12

13 I. PLS approach : 2 blocks X 1 ξ 1 ξ 2 X 2 Mode for weight calculation Y 1 = X 1 w 1 Y 2 = X 2 w 2 Method Deflation A A PLS regression of X 2 on X 1 On X 1 only (*) B A Redundancy analysis of X 2 with respect to X 1 On X 1 only A A Tucker Inter-Battery Factor Analysis On X 1 and X 2 B B Canonical correlation Analysis On X 1 and X 2 (*) Deflation: Working on residuals of the regression of X on the previous LV s in order to obtain orthogonal LV s. 13

14 PLS regression (2 components) Use of PLS-GRAPH dim 1 - Mode A for X - Mode A for Y - Deflate only X Max Cov( Xa, Yb) a = b = 1 = Max Cor( Xa, Yb)* Var( Xa)* Var( Yb) a = b = 1 dim 2 14

15 PLS Regression in SIMCA-P : PLS Scores australie 3 royaume-uni t[2] suisse canada états-unis belgique nouvelle zélande pays-bas suède luxembourg france West Germany norvège finlande danemark irlande argentine uruguay venezuela cuba espagne irak italie chili équateur pèrou autriche colombie grèce rép. dominicaine costa-rica salvador guatémala taiwan brésil panama honduras nicaragua égypte sud vietnam philippines inde libye bolivie -2 japon -3-4 pologne yougoslavie t[1] 15

16 Correlation loadings RENT 0.60 GINI FARM 0.40 GNPR DEMOSTB pc(corr)[comp. 2] INST DEMOINST DEAT ECKS DICTATURE LABO pc(corr)[comp. 1] 16

17 Redundancy analysis of X on Y (2 components) - Mode A for X - Mode B for Y - Deflate only X dim 1 Max a = Var ( Yb) = 1 = a = Var ( Yb) = 1 Cov( Xa, Yb) Max Cor( Xa, Yb)* Var( Xa) dim 2 2 Max Cor x j Yb Var ( Yb) = 1 j (, ) 17

18 Inter-battery factor analysis (2 components) - Mode A for X - Mode A for Y - Deflate both X and Y dim 1 Max Cov( Xa, Yb) a = b = 1 = Max Cor( Xa, Yb)* Var( Xa)* Var( Yb) a = b = 1 dim 2 18

19 Canonical correlation analysis (2 components) - Mode B for X - Mode B for Y - Deflate both X and Y dim 1 Max Var ( Xa) = Var ( Yb) = 1 Cov( Xa, Yb) dim 2 Max Var ( Xa) = Var ( Yb) = 1 Cor( Xa, Yb) 19

20 Barker & Rayens PLS DA - Mode A for X - Mode B for Y - Deflate only on X a Max = Var ( Yb) = 1 Cov( Xa, Yb) = a Max Cor( Xa, Yb)* Var( Xa) = Var ( Yb) = 1 Redundancy analysis of X with respect to Y 20

21 Barker & Rayens PLS DA Economic inequality vs Political regime Separation between political regimes is improved compared to PLS-DA. 21

22 II. Hierarchical model : J blocs X 1 ξ 1... ξ X 1.. X J X J ξ J Scheme for computation of the inner components Z j Computation of outer weights w j Horst Centroid Factorial Mode A SUMCOV SABSCOV SSQCOV Mode B SUMCOR SUMCOR SSQCOR (Carroll GCCA) Deflation : On original blocks and/or the super-block Generalized PLS regression Generalized CCA 22

23 III. Multi-block data analysis SABSCOR : PLS Mode B + Centroid scheme Use of XLSTAT-PLSPM Max Cor( X1w1, X 2w2) + Cor( X1w1, X 3w3) + Cor( X 2w2, X 3w3) = =

24 Multiblock data analysis SSQCOR : PLS Mode B + Factorial scheme Max Cor ( X1w1, X 2w2) + Cor ( X1w1, X 3w3) + Cor ( X 2w2, X 3w3) = + + =

25 Practice supports theory Mode B + Centroid Mode B + Factorial Agricultural inequality <--> Industrial development Agricultural inequality <--> Political Instability Industrial development <--> Political Instability SABSCOR * SSQCOR * * Criterion optimized by the method (checked on random initial weights) 25

26 IV. Structural Equation Modeling Agricultural inequality (X 1 ) GINI INST FARM RENT ξ 1 C 12 =0 C 13 =1 ξ 3 ECKS DEAT D-STB GNPR LABO ξ 2 C 23 =1 D-INS DICT Industrial development (X 2 ) C ij = 1 if ξ i and ξ j are connected = otherwise Political instability (X 3 ) 26

27 SABSCOR-PLSPM Mode B + Centroid scheme Y 1 = X 1 w 1 Y 3 = X 3 w 3 Y 2 = X 2 w 2 Max Cor( X1w1, X 3w3) + Cor( X 2w2, X 3w3) = =

28 SSQCOR-PLSPM Mode B + Factorial scheme Y 1 = X 1 w 1 Y 3 = X 3 w 3 Y 2 = X 2 w Max Cor ( X1w1, X 3w3) + Cor ( X 2w2, X 3w3) = =

29 Comparison between methods Mode B + Centroid scheme Mode B + Factorial scheme Cor( Y, Y ) + Cor( Y, Y ) * Cor ( Y, Y ) + Cor ( Y, Y ) * * Criterion optimized by the method (checked on random initial weights) Practice supports theory 29

30 Mode A + Centroid scheme The criterion optimized by the algorithm, if any, is unknown. 30

31 SABSCOV-PLSPM New Mode A + Centroid scheme weight Cor=.429 Cov= Cov=-1.69 Cor= Max Cov( X1w1, X 3w3) + Cov( X 2w2, X 3w3) = 2.69 One-step hierarchical PLS Regression 31

32 SABSCOV-PLSPM New Mode A + Factorial scheme weight Cor=.4276 Cov= Cov=-1.69 Cor= Max Cov ( X1w1, X 3w3) + Cov ( X 2w2, X 3w3) = 3.86 One-step hierarchical PLS Regression 32

33 Generalized Barker & Rayens PLS-DA SSQCOV-PLSPM New Mode A for X 1 and X 2 and Mode B for Y weight Cov Max Cor ( X1w1, X 3w3)* Var( X1w1) + Cor ( X 2w2, X 3w3)* Var( X 2w2) = 1.39 One-step hierarchical B&R PLS-DA

34 Generalized Barker & Rayens PLS-DA Industrial development Agricultural inequality

35 Conclusion In the PLS approach of Herman Wold, the constraint is: Var( X w ) = 1 j In the PLS regression of Svante Wold, the constraint is: This presentation unifies both approaches. j w j = 1 35