Sample Geometry. Edps/Soc 584, Psych 594. Carolyn J. Anderson

Transcription

1 Sample Geometry Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees, University of Illinois Spring 2017

2 Outline Motivation Variable Space Observation Space What it is. Mean Covariance Global summary statistic for Ë Reading: Johnson & Wichern, Chapter 3 C.J. Anderson (Illinois) Sample Geometry Spring / 49

3 Motivation The sample Ü, Ë and Ê have geometric interpretations. The geometric interpretation shows how Ü, Ë, and Ê are related to n p. Studying these relationships provides insight into multivariate methods. Introduce a summary statistic that describes the variability in the data. This is based on geometry. C.J. Anderson (Illinois) Sample Geometry Spring / 49

4 Assumptions The geometric interpretation of sample descriptive statistics (i.e., Ü, Ë, and Ê) is based on two assumptions. (1) Each row of the data is unrelated to all of the others (i.e., independent observations). Each row corresponds to a case, individual, sampling unit. Each row is a multivariate observation (point) in p-dimensional space. X 11 X 12 X 1p X 21 X 22 X 2p n p = X n1 X. n2 Xnp (2) The joint distribution of all p variables is the same for all cases (individuals, rows). We are drawing multivariate observations from the same unchanging population. C.J. Anderson (Illinois) Sample Geometry Spring / 49

5 The Variable Space Each row of is a point in p-space or variable space. The variables (columns of ) define the axes. Consider the n points in the p dimensional space. The center of the point cloud is Ü. The variability and covariability is measured by Ë. C.J. Anderson (Illinois) Sample Geometry Spring / 49

6 The data The Variable Space: Example Ü = = ( ) = ( x 2 1 ) տ Ü What would happen to picture if multiply by 2? x 1 C.J. Anderson (Illinois) Sample Geometry Spring / 49

7 The Observation Space Important alternative geometric representation of that considers the data as p points in an n-dimensional space. Each observation (case, individual) defines an axis or dimension of the space. Each column of is a vector in the n-space. X 11 X 12 X 1p X 21 X 22 X 2p n p = = (Ý 1, Ý 2,...,Ý p ) X n1 X n2. X np where Ý k = x 1k x 2k. for k = 1,...p. x nk C.J. Anderson (Illinois) Sample Geometry Spring / 49

8 The data The Observation Space: Example 2 4 = ( ) row 2 4 Ý Ý 2 Ý row 1 1 Ý 4 In this space, we talk about the lengths of the vector variables and about the angle between them. C.J. Anderson (Illinois) Sample Geometry Spring / 49

9 The Observation Space: The mean Question: How does Ü relate to the vectors in the observation space? Answer: The projection of the vector variable onto 1 n. Quick recall: The projection of Ü onto Ý equals ( )( ) Ü = Ü Ý Ý Ý Ý = Ý Ü Ý L Ý L Ý Ü L Ü θ Ý x C.J. Anderson (Illinois) Sample Geometry Spring / 49

10 The Mean & Projection The equal angular vector is an (n 1) vector n =. 1 Projection of Ý k = (x 1k,x 2k,...,x nk ) on the the vector 1 n = 1 1 (x 1k,x 2k,...,x nk ). ( 1 Ý k 1 n 1 n1 n ) 1 = = ( ) }{{} n j=1 x jk 1 n = x k 1 n n C.J. Anderson (Illinois) Sample Geometry Spring / 49 n 1 n

11 The Mean in the Observation Space The sample mean x k of the k th variable corresponds to the multiple of 1 n required to give the projection of Ý k onto the vector 1 n. Projection of Ý k onto 1 n : Ýk This vector is k = Ý k Ük1 n 1 n x k 1 n k is the Deviation vector. We have the decomposition of Ý k into two parts: Ý k = x }{{} k 1 n + k }{{} mean = {}}{ x k 1 n + deviation from mean {}}{ Ý k Ük1 n C.J. Anderson (Illinois) Sample Geometry Spring / 49

12 Back to our little example case 2 Ý 1 Ý 3 Ý 2 3 In the direction of 1 = (1,1) case 1 Ý 4 x 1 1 = ([0(1)+4(1)]/2)1 = (2.0)1 x 1 = 2 x 2 1 = ([1(1)+1(1)]/2)1 = (1.0)1 x 2 = 1 x 3 1 = ([2(1)+3(1)]/2)1 = (2.5)1 x 3 = 2.5 x 4 1 = ([4(1) 1(1)]/2)1 = (1.5)1 x 4 = 1.5 C.J. Anderson (Illinois) Sample Geometry Spring / 49

13 2-dimensional example continued For our 2-case example with 4 variables ( ) = Note: Ê = What is the rank of this correlation matrix? C.J. Anderson (Illinois) Sample Geometry Spring / 49

14 Deviation Vectors: Variances & Covariance Sample variance: The squared lengths of deviation vectors, n = k k = (x jk x k ) 2, L 2 k j=1 are sums of squared deviation from the mean, so s kk = 1 n L2 = 1 n (x jk x k ) 2. k n Sample covariance: The inner products between two deviation vectors, n k i = (x jk x k )(x ji x i ), j=1 j=1 are sums of cross products, so s ik = 1 n k i = 1 n (x jk x k )(x ji x i ) n j=1 C.J. Anderson (Illinois) Sample Geometry Spring / 49

15 Deviation Vectors: Correlation The angle between two deviation vectors equals cos(θ ik ) = = i k L i L k n j=1 (x jk x k )(x ji x i ) n j=1 (x n ji x i ) 2 j=1 (x jk x k ) 2 = r ik The cosine of the angle between two deviation vectors is the sample correlation coefficient of the variables. If two deviation vectors have the same orientation (same direction), θ = 0 cos(θ) = 1 = r ik. If two deviation vectors have the opposite orientation, θ = 180 cos(θ) = 1 = r ik. If two deviation vectors are perpendicular (orthogonal), θ = 90 cos(θ) = 0 = r ik. Back to simple example (pages 12-13) C.J. Anderson (Illinois) Sample Geometry Spring / 49

16 Correlation: Simple Example 2 4 = Ê = ( ) 1 case case 1 4 C.J. Anderson (Illinois) Sample Geometry Spring / 49

17 3-Dimensional Example Suppose that p = 3, n = 3 and the data are = So Ý 1 = Ý 2 = Ý 3 = x 1 = = 620 x 2 = = 650 x 3 = = 680 C.J. Anderson (Illinois) Sample Geometry Spring / 49

18 3-Dimensional Example: Variable 1 case = (1,1,1) Ý 1 case 1 case 2 C.J. Anderson (Illinois) Sample Geometry Spring / 49

19 3-Dimensional Example: Graph of Variables case = (1,1,1) Ý 2 Ý 1 Ý 3 case 1 case 2 C.J. Anderson (Illinois) Sample Geometry Spring / 49

20 3-Dimensional Example Means are the multiples of 1 that give projection of Ý i onto Using the first variable for illustration Ý = x 11 = = = Ý 1 x 1 1 = = is perpendicular to x 1 as it should be: 1 x 1 1 = (80, 120,40) = 620( ) = 0 C.J. Anderson (Illinois) Sample Geometry Spring / 49

21 Deviation Vectors Notes (continued): Ý 1 is decomposed into two parts: If i = 0, then Ý i = x i 1. Ý 1 = x 1 1 n + 1 All of them 80 1 = = = C.J. Anderson (Illinois) Sample Geometry Spring / 49

22 Graph of 3 deviation vectors 2 case case 2 C.J. Anderson (Illinois) Sample Geometry Spring / 49

23 Variances Variance are proportional to the lengths of deviation vectors Variable 1: L 1 = = = = ns 11 = 3s 11 So s 11 = 22400/3 = and s 11 = Variable 2: L 2 = = = = 3s 22 So s 22 = 20000/3 = and s 22 = Variable 3: L 3 = = 7400 = = 3s 33 So s 33 = 7400/3 = and s 33 = C.J. Anderson (Illinois) Sample Geometry Spring / 49

24 Correlations Correlations are the cosines of the angles between deviation vectors r 12 = (0) 120( 100) +40(100) = = L 1 L 2 (149.67)(141.42) =.756 and the angle between 1 and 2 = cos 1 (.756) = r 13 = 1 3 = 80(70) 120( 40)+40( 30) = 9200 L 1 L 3 (149.67)(86.02) =.715 and the angle between 1 and 3 = cos 1 (.715) = r 23 = 2 3 = 0(70) 100( 40)+100( 30) = 1000 L 2 L 3 (141.42)(86.02) =.082 and the angle between 2 and 3 = cos 1 (.082) = C.J. Anderson (Illinois) Sample Geometry Spring / 49

25 Summary: Observation Space The axes of the observation space are defined by cases (individuals) and variables are represented as vectors. The projection of the column Ý k of the data matrix n p onto the equal angular vector 1 n is the vector x k 1. The information contained in Ë is obtained from the deviation vectors: k = Ý k x k 1 n = {(x jk x k )} The variance: L 2 k = k k = ns kk The covariance: i k = ns ik The sample correlation coefficient r ik is the cosine of the angle between i and k. C.J. Anderson (Illinois) Sample Geometry Spring / 49

26 Random Sample and Expected Values We sample from a population to learn about a phenomenon of interest We ll think about data (n cases) as a random sample of cases from some large population (real or hypothetical). For every case in the sample, we measure p variables. So The measurements of the p variables for a single case will usually be correlated or dependent. Multivariate analysis constitutes techniques designed to account for and/or study these correlations/dependencies. The measurements from different cases must be independent. This assumption can be violated a number of ways (e.g., collect observations over time, poor measurement or experimental procedures, repeated observations of same case). Lack of independence can be a big problem. C.J. Anderson (Illinois) Sample Geometry Spring / 49

27 where DATA = 1 2. n j = (X j1,x j2,...,x jp ) Each j is a random vector containing p measurements. Distances between the n points in p-space are determined by the joint probability function governing each and every j, f(üj) = f(x j1,x j2,...,x jp ) C.J. Anderson (Illinois) Sample Geometry Spring / 49

28 Important Result Let 1, 2,..., n be a random sample from the joint distribution f(üj), which has mean vector µ and covariance matrix Σ. Then is an unbiased estimator of µ has covariance matrix (1/n)Σ E( ) = µ and Cov( ) = 1 n Σ C.J. Anderson (Illinois) Sample Geometry Spring / 49

29 Estimating Covariance Matrix To estimate Σ, first note So E(Ën) = n 1 n Σ = Σ 1 n Σ n n 1 Ë n = S = 1 n 1 is an unbiased estimator of Σ. n (j Ü)(j Ü) Ë w/o a subscript has divisor (n 1) and is unbiased. Ën w/ a subscript has divisor n, biased, but is the MLE, ˆΣ. The (i,k) th element of the Ë is an unbiased estimator of σ ik : s ik = 1 n (x ji x i )(x jk x k ) n 1 j=1 j=1 s ik is not an unbiased estimator of σ ik. r ik is not an unbiased estimator of ρ ik. The amount of bias is small for large n. C.J. Anderson (Illinois) Sample Geometry Spring / 49

30 Generalized Variance The Sample covariance matrix describes the variation and covariation in and between the p variables, s 11 s 12 s 1p s 12 s 22 s 2p S = s p1 s p2 s pp where s ik = (1/(n 1)) n j=1 (x ji x i )(x jk x k ). With just one variable, we usually only need just one statistics (the sample variance) to describe the variability in the data. With p variable,need p variances and p(p 1)/2 covariances. It would be nice to have a single statistic that summarizes the information in Ë (i.e., reflects all the variances and covariance). C.J. Anderson (Illinois) Sample Geometry Spring / 49

31 Generalized Sample Variance GSV = Generalized Sample Variance = determinant of Ë = Ë Recall that the determinant of an A k k matrix is a scalar = a 11 for k = 1 = a 11 a 22 a 12 a 21 for k = 2 k = a ij A ij ( 1) i+j for k > 1 j=1 where A ij is the (k 1) (k 1) sub-matrix of where the i th row and i th column of have been deleted. C.J. Anderson (Illinois) Sample Geometry Spring / 49

32 Example: GSV Ë = GSV = 3(4(9) 3(3)) ( 1)( 1(9) 2(3))+2( 1(3) 4(2)) = 3(27)+1( 15)+2( 11) = 44 What does this mean? Two ways to interpret it (geometrically): Using observation space (n dimensional) Using variable space (p-dimensional) C.J. Anderson (Illinois) Sample Geometry Spring / 49

33 GSV: n-space interpretation The GSV = Ë is related to the volume of the parallelpiped (geometric figure) defined by the p deviation vectors. e.g., p = 2 and n = whatever The volume ( area for p = 2) 2 1 Remember i = Ý i x i 1 (Ý i is column vector of matrix ) Specifically, GSV = Ë = (n p) p (volume) 2 = (volume)2 (n p) p or volume = Ë (n p) p/2. C.J. Anderson (Illinois) Sample Geometry Spring / 49

34 Implications for Size of GSV GSV = Ë = (n p) p (volume) 2 = (volume)2 (n p) p Does the GSV ( Ë ) increase or decrease when n decreases? The length of any i = Ý i = x i 1 increases? p decreases? The angles (θ) between i s get smaller (i.e., r ik s get closer to 1)? r ik = cos(θ ik ) = i k L i L k Draw an example where θ ik = 90 versus θ ik very small. C.J. Anderson (Illinois) Sample Geometry Spring / 49

35 Variable space Interpretation of GSV In the variable space, we consider the spread of the n points in the p dimensional space around the sample mean Ü = ( x 1, x 2,..., x p ) For this, we need some more definitions/concepts: The statistical distance of a point P = (x 1,x 2,...,x p ) from the origin is the distance of the point P = (x1,x 2,...,x p ) = ( x 1 x 2 x p,,, ) s11 s22 spp in standardized coordinates; that is, D(0,P) = x1 2 +x x 2 p x1 2 = + x2 2 +,+ x2 p s 11 s 22 s C.J. Anderson (Illinois) Sample Geometry pp Spring / 49

36 Constant Statistical Distance All points that have coordinates (x 1,x 2,,x p ) and are a constant (squared) statistical distance from the origin must satisfy x 2 1 s 11 + x2 2 s x2 p s pp = c 2. This is the equation for an ellipsoid with center at the origin (zero) and major and minor axes coinciding with the coordinate axes. e.g., For P = (x 1,x 2 ), (0,c s 22 ) ց ւ P = (x 1,x 2 ) ( c s 11,0) ր տ (c s11,0) (0, c s 22 ) ր C.J. Anderson (Illinois) Sample Geometry Spring / 49

37 Statistical Distance between two Points The statistical distance of a point P = (x 1,x 2 ) from point Q = (y 1,y 2 ) is d(p,q) = a 11 (x 1 y 1 ) 2 +2a 12 (x 1 y 1 )(x 2 y 2 )+a 22 (x 2 y 2 ) 2 where a 11, a 12, and a 22 are some constants. For now, we ll suppose that the variables are correlated (i.e., the variable vectors are not at 90 angles). The square distance d(p,q) 2 is the equation of an ellipse centered at Q: x 2 Q P x 1 C.J. Anderson (Illinois) Sample Geometry Spring / 49

38 Statistical Distance between two Points The statistical distance of a point P = (x 1,x 2 ) from point Q = (y 1,y 2 ) is d(p,q) = a 11 (x 1 y 1 ) 2 +2a 12 (x 1 y 1 )(x 2 y 2 )+a 22 (x 2 y 2 ) 2 where a 11, a 12, and a 22 are some constants. For now, we ll suppose that the variables are correlated (i.e., the variable vectors are not at 90 angles). The square distance d(p,q) 2 is the equation of an ellipse centered at Q: x 2 Q P x 1 C.J. Anderson (Illinois) Sample Geometry Spring / 49

39 Rotation of axes When variable are correlated, we have a rotation of the axes through and angle θ: Namely, x 1 x 1 and x 2 x 2 x 2 x 1 = x 1 cos(θ)+x 2 sin(θ) x 2 = x 1 sin(θ)+x 2 cos(θ) x 2 x 1 θ Q P x 1 C.J. Anderson (Illinois) Sample Geometry Spring / 49

40 Back to Interpretation of GSV The following is an equation of an ellipsoid (Ü Ü) Ë 1 (Ü Ü) = c 2 where Ü is like point P and Ü is like point Q. (A quadratic form) The above is the (squared) statistical distance of points Ü from the point Ü in the p dimensional space that are a constant distance c from Ü. e.g., p = 1 : (x 1 x 1 ) 2 s 11 = c 2 p = 2 : a 11 (x 1 x 1 ) 2 +2a 12 (x 1 x 1 )(x 2 x 2 )+a 22 (x 2 x 2 ) 2 = c 2 where the a 11, a 12 and a 22 depend on ( ) ( ) s11 s Ë = 12 Ë 1 a11 a = 12 s 21 s 22 a 21 a 22 C.J. Anderson (Illinois) Sample Geometry Spring / 49

41 GSV Interpretation The Volume of this ellipsoid is volume{ü : (Ü Ü) Ë 1 (Ü Ü) c 2 } = (constant) Ë 1/2 c p Note: Ë = GSV So the GSV is proportional to the volume of the ellipsoid where the ellipsoid represents statistical distances of observations from the vector of means. C.J. Anderson (Illinois) Sample Geometry Spring / 49

42 When does GSV = 0 Consider matrix of deviations, x 11 x 1 x 12 x 2 x 1p x p x 21 x 1 x 22 x 2 x 2p x p 1 n x = x n1 x 1 x n2 x 2 x np x p GSV = 0 means that at least one column of ( 1 n x) can be expresses as a linear combination of the others. One (or more) of the deviation vectors j lies in the (hyper)plane defined by the others. (Remember of simple example). The GSV is zero if and only if at least one deviation vector lies in the (hyper)plane formed by all linear combinations of the others; i.e., the columns of the matrix 1 n x are linearly dependent. C.J. Anderson (Illinois) Sample Geometry Spring / 49

43 Implications & Limitations of GSV If the (sample size) (number of variables) n p then Ë = GSV = 0 for all samples. e.g., Example where p = 4 and n = 2 More on the meaning & interpretation of GSV that we ll elaborate on: The GSV represents both variability and covariability. You can have very different patterns of variability and association (correlational structure) but have the same value for GSV. C.J. Anderson (Illinois) Sample Geometry Spring / 49

44 Different Patterns S, Same GSV Problem with GSV = Ë is that different patterns of variability and covariability can give the same value for GSV. eg., Ë0 = Ë1 = Ë2 = ( ) Ë0 = ( ) Ë1 = ( ) Ë2 = Part of the problem is that GSV reflects both variability and covariability. C.J. Anderson (Illinois) Sample Geometry Spring / 49

45 The GSV: variances and covariances Even if the strength of the linear relationship between variables is constant (i.e., covariance) the GSV could be larger simply be increasing s ii s. larger parellepiped (ellipsoid) larger volume larger GSV. If you want a measure that reflects only the covariability, then first standardize all of the variables such that they have the same length (i.e., variance): x ji = x ji x i sii The sample covariance matrix for x ji s (i.e., Ë ) is the sample correlation matrix Ë = Ê. GSV of standardized variables = Ê = det(ê). The residual/deviation vectors of the x s all have length C.J. Anderson (Illinois) Sample Geometry Spring / 49

46 Proof that L d i = n 1 L 2 i = i i = = ( (x1i x i ), (x 2i x i ),, (x ) ni x i ) sii sii sii n (x ji x i ) 2 j=1 s ii = 1 n (x ji x i ) 2 s ii = 1 n 1 j=1 1 n n j=1 (x (x ji x i ) 2 ji x i ) 2 = n 1 (x 1i x i) sii (x 2i x i) sii. (x ni x i) sii j=1 C.J. Anderson (Illinois) Sample Geometry Spring / 49

47 GSV using R Ê focuses only on angles between the i s. When angles= 90, the i s are perpendicular and Ê reaches a maximum: r ik = cos(90)/1 = 0 and Ê = 1. When angles= 0, the i go in the same direction and Ê reaches a minimum: r ik = cos(0)/1 = ±1 and Ê = 0 e.g., Ê0 = Ê1 = Ê2 = Ê3 = ( ) Ê0 = ( ) Ê1 = ( ) Ê2 = ( ) Ê3 = C.J. Anderson (Illinois) Sample Geometry Spring / 49

48 Relationship between det(s) and det(r) The two ways of computing the GSV (one on unstandardized and the other on standardized variables) are functionally related: Ë = (s 11 s 22 s pp ) Ê = p s ii R This emphasizes the fact that Ë depends on the s ii s but Ê doesn t. e.g., and i=1 Ë1 = 9 = (5)(5)0.36 = 9 Ë0 = (3)(3) Ê = (3)(3)(1) = 9 C.J. Anderson (Illinois) Sample Geometry Spring / 49

49 Another Global Statistic While GSV = Ê focuses just on covariability, there is another measure that focuses just summarizing the variance in Ë: Total Sample Variance = p s ii = trace(ë) = tr(ë) We ll use this one later when we talk about principal components analysis. i=1 C.J. Anderson (Illinois) Sample Geometry Spring / 49