Textbook: Methods of Multivariate Analysis 2nd edition, by Alvin C. Rencher

Transcription

1 Lecturer: James Degnan Office: SMLC 342 Office hours: MW 1:00-3:00 or by appointment Please include STAT476 or STAT576 in the subject line of the to make sure I don t overlook your . Textbook: Methods of Multivariate Analysis 2nd edition, by Alvin C. Rencher Assessment: Grading will be based homework (roughly 4 assignments in the semester) (30%), two in-class tests (20% each), and a final exam (30%). SAS Programming January 16, / 43

2 We will mostly use R for computing, but may also consider SAS depending on student interest. Solutions to homework can be done in any software package, but it will be easier for me to grade and give partial credit for homework done in R and SAS. Homework For turning in computer-based homework, turn in all computer code used as an appendix only. Do not include computer code as part of your solutions. Figures and tables can be generated from computer output, but solutions must be discussed separately from the output, and the results in the Figures and Tables should be cited in the homework solutions. This will be discussed further in class. Late homework will be penalized 10% per day. All homework must be printed (not ed) and turned in either in class or to my office. Sliding homework under my door is fine. SAS Programming January 16, / 43

3 Topics We ll cover mostly topics from the listed book. Some examples include Reviews of matrix algebra mostly as needed Multivariate distributions, joint PDFs and CDFs, multivariate normal Hotelling s T 2 MANOVA (multivariate ANOVA) MANCOVA (multivariate ANCOVA) Multivariate regression Correlation analysis Cluster Analysis Principal Components Analysis (PCA) Multdimensional Scaling (MDS) Factor Analysis SAS Programming January 16, / 43

4 What does multivariate statistics refer to? How is multivariate analysis different from say, multiple regression? Many statistical models are of the form Response = Predictors + error where the Response is a single (i.e., univariate) random variable and the Predictors consist of multiple variables. Typically the predictors are treated as non-random. This is especially clear in designed experiements, where the predictors are controlled by the experimenter, and the only thing random is the response. SAS Programming January 16, / 43

5 What does multivariate statistics refer to? Often, however, there are covariates among the predictors which are not directly in the experimenter s control. In analysis of variance, you might have multiple treatment groups, such as patients being assigned placebo versus various drugs. The response might be, say total cholesterol level. Covariates might include things like sex and age of the patients in the study, which are not under the experimenter s control, and would lead to an analysis of covariance (ANCOVA). Usually in a model like this, even though the covariates are not controlled, a linear model treats the response as random, but not the predictors, including sex and age of the patients. SAS Programming January 16, / 43

6 What does multivariate statistics refer to? In the previous example, the total cholesterol level was the variable of interest, and understanding variation in cholesterol requires accounting for effects in the covariates, but otherwise the covariates are not of interest. In multivariate techniques, we are usually interested two or more random variables and understanding how they co-vary, their joint distribution, and possibly how they are related to predictors and covariates that are not directly of interest. To take the cholesterol example, we might distinguish between good cholesterol (high density lipoprotein, or HDL), and bad cholesterol (low density lipoprotein, or LDL). If both of these are measured simultaneously, we might want a model that predicts both values as a function of the predictors. SAS Programming January 16, / 43

7 What does multivariate statistics refer to? To continue the cholesterol example, we might have LDL and HDL both as response variables, so that the model looks like this: (HDL,LDL) = Drug + Dose + Age + Sex + error Alternatively, consider this model LDL = HDL + Drug + Dose + Age + Sex + error The first model is multivariate, while the second is not using this terminology, even though both models use the same variables. SAS Programming January 16, / 43

8 What does multivariate statistics refer to? In many statistics problems, multiple measuresments are made on individuals and a somewhat arbitrary decision is made regarding which is the response (and therefore treated randomly) and which are the predictors (treated nonrandomly). In simple linear regression, we think of the x values as nonrandom and the y values as random, but for many examples, particularly in observational data as opposed to designed experiments, both variables are equally random. Historically, regression was invented by Francis Galton (cousin of Charles Darwin of Origin of Species fame) in England. He coined the term regression in analyzing data on heights of parents compared to sons. SAS Programming January 16, / 43

9 Regression of Son s height s on parent s heights SAS Programming January 16, / 43

10 Considering the two heights in a multivariate way, we instead think about the joint distribution of the two variables and consider them both random. SAS Programming January 16, / 43 What does multivariate statistics refer to? In the case of parent s heights versus son s heights, it seems silly to think of one person s height as random while another person s height is nonrandom. On the other hand, it might make more sense to think of parent s heights as predicting their children s heights rather than children s heights as predicting their parents heights. Other examples might be more symmetric, such as heights of husbands versus heights of wives (Galton also analyzes this case, using a 3x3 contingency table with heights classified as Tall, Medium, and Short). A regression here needs to make an arbitrary decision about which of the husband or the wife is considered the random response. Generally, a regression of the husbands heights on the wive s heights will lead to a different trendline than a regression of the wives heights on the husbands heights.

11 Why use multivariate statistics? Apart from the example just given, a justifcation for multivariate techniques in statistics is that we can sometimes get more information from the joint distribution of random variables than by looking at marginal distributions (i.e., the distribution of Y given different levels of X ). Multivariate techniques can be more computationally intensive than univariate techniques, and often matrix methods are used. Techniques such as computing the inverse of a matrix, eigenvalues and eigenvectors, the determinant of a matrix, and singular value decompositions often arise. If data is high-dimensional (lots of variables), it might be particularly difficult to use multivariate techniques. We will often look at cases with fairly low dimensionality. Later in the course, we will also consider methods of dimension reduction. While much of statistics is oriented around statistical inference, a lot of multivariate techniques are concerned with visualization and gaining qualitative insights into data. Consequently, a number of multivariate techniques are descriptive and exploratory rather than inferential. SAS Programming January 16, / 43

12 Matrix review: notation A note on notation. A common convention in statistics is to use upper case Roman letters for random variables and lower case Roman letters for values of random variables (i.e., X versus x in the expression P(X = x)). A very strong convention in mathematics is to use upper case letters for matrices. Depending on the book, the two most common conventions are to either use upper case, (mostly) Roman, bold, nonitalic letters (e.g., A) or to use upper case, (mostly) Roman, italic, nonbold letters (e.g., A). If bold characters are used for matrices, then lower case, bold, nonitalic letters are typically used for vectors (e.g., y). The mathematical convention seems to be the stronger, and in multivariate settings, there is often no notational difference between random and nonrandom objects. SAS Programming January 16, / 43

13 Matrix review: notation Our textbook uses bold, nonitalic letters for matrices and vectors, whether they are random or not, and that is the convention I will follow also. Note that Christensen s books use nonbold, italic characters for matrices and vectors. Greek letters are also sometimes used for vectors to indicate a parameter that is also a vector, such as µ versus µ. SAS Programming January 16, / 43

14 Matrix review Note that chapter 2 of the book has a fairly thorough matrix review which I am following. As an undergraduate, I took Math 321 at UNM rather than Math 314. I found that Math 321 didn t prepare me very well for the matrix algebra needed in statistics in graduate school, although it was good preparation for some other concepts like linear spaces and vector spaces needed for Advanced Linear Models. At some point in grad school, I picked up the textbook being used for Math 314 to give me more practice with more matrix manipuation (as opposed to proofs) and for certain concepts like eigenvectors and eigenvalues. SAS Programming January 16, / 43

15 Matrix review A matrix is a rectangular array of numbers with n rows and p columns. These variables are often used in statistics with n representing the sample size and p the number of parameters (for example, in a regression problem). We index a matrix A by elements a ij where 1 i n and 1 j p. Often we write A = (a ij ) to emphasize the notation. We can think of a column vector as an n 1 matrix and a row vector as a 1 p matrix. If something is written as a vector without specifying whether it is a column vector or row vector, the default assumption is that it is a column vector unless the context requires it to be a row vector. SAS Programming January 16, / 43

16 Matrix review: Transpose A column vector would be written as (for example) x 1 x = x 2 x 3 Just as matrix elements can be indexed by subscripts (and using a nonbold character), so can vectors, such as if x 3 referring to the third element of a vector. The transpose of a matrix or a vector is written with an apostraphe, such as x = (x 1 x 2 x 3 ), x = (x 1 x 2 x 3 ) The transpose of a matrix changes an n p matrix into a p n matrix. If B = A, then b ji = a ij for all 1 i n and 1 j p. SAS Programming January 16, / 43

17 Matrix review: special matrices The transpose of a matrix changes the first column of the old matrix to be the first row of the new matrix, and generally the ith row of the first matrix to be the ith column of the second matrix. Note that (A ) = A. A matrix is square if n = p. A square matrix A is symmetric if and only if a ij = a ji for all i and j. Similarly, A is symmetric if and only if A = A. SAS Programming January 16, / 43

18 Matrix review The diagonal of a square matrix refers to the elements a ii, 1 i n. A diagonal matrix, often written D has 0s for all nondiagonal entries. That is, if D is diagonal, then d ij = 0 for i j. A special function called diag() is used to deal with diagonal matrices. For example, diag(1, 2, 4) = You can also use diag to set all nondiagonal entries to 0. E.g., diag = SAS Programming January 16, / 43

19 Matrix review: special matrices The identiy matrix I plays a role similar to the number 1 in arithmetic and is a square, diagonal matrix with 1s on the diagonal. For any n p matrix A, we have I n n A = A and AI p p. A vector of 1s is denote by j and a matrix of all 1s is denoted by J. The dimension might be clear from context or you can write things like I n n and J 3 3 to help keep track of the dimension. A matrix of 0s is denoted by O. A square matrix is upper triangular if all entries below the diagonal are 0, i.e. if a ij = 0 for i > j. Zeroes on the diagonal are ok. A square matrix is lower triangular if all entries above the diagonal are 0. SAS Programming January 16, / 43

20 Matrix review: matrix operations The most important operations are addition (and substraction), matrix multiplication, and scalar multiplication. If two matrices A and B have the same dimension, then C = A + B is defined by c ij = a ij + b ij. Note that if two matrices, A and B are identical, then a ij = b ij for each i and j, and therefore A B = 0, rather than 0. SAS Programming January 16, / 43

21 Matrix review: matrix operations To multiply two matrices A n p and B p q, the number of columns of the first matrix must equal the number of rows of the second matrix (they are called conformable). The resulting matrix C = A B is defined by c ij = p a ik b kj k=1 You can think of this as the dot product of the ith row of A with the jth column of B. The resulting matrix has dimension n q. This takes a bit of getting used to and can be slow to perform by hand. Even if the product AB is defined, then the product BA is not defined unless the number of columns of B matches the number of rows of A. For example, if A is 2 3 and B is 3 2, then AB is 2 2 and BA is 3 3. If A and B are both n n, then so is their product; however, typically AB is still not equal to BA. SAS Programming January 16, / 43

22 Matrix review: matrix operations In statistics, it is common to work with a design matrix X. Often instead of working with this matrix directly, the matrix X X is used. A useful feature of this matrix is that it is symmetric. A useful review is to give an example and try to prove the general case. As an example, in a regression problem you might have the following: ) 5 2 = ( β0 β 1 + In this problem the design matrix X is the 3 2 matrix. Part of the solution of the regression problem involves X X. We ll compute this product for this example e 1 e 2 e 3 SAS Programming January 16, / 43

23 Matrix review: matrix operations We have C = X X = ( ) SAS Programming January 16, / 43

24 Matrix review: matrix operations Thus, c 11 = = 3 c 12 = = 8 c 21 = = 8 c 22 = = 22 Finally, C = X X = ( ) SAS Programming January 16, / 43

25 Matrix review: matrix operations Something to notice about this answer is that the matrix is square and symmetric (since x 12 = x 21 ) even though X was not square. Can you show that this is true for any matrix X with any number of dimensions? SAS Programming January 16, / 43

26 Matrix review: matrix operations To show that X X is symmetric, if X is n p, then X is p n, and therefore X X is p p., so it is square. We d like to also show that for C = X X, C = C, meaning that C is symmetric. If we look at the elements of C, it is symmetric if c ij = c ji for an arbitrary choice of i and j. Let B = X so that b ij = x ji. Applying the rules for matrix multiplication, Also, c ji = c ij = n b ik x kj = k=1 n b jk x ki = k=1 Thus, C = C, so X X is symmetric. n x ki x kj k=1 n x kj x ki = c ij k=1 SAS Programming January 16, / 43

27 Matrix review: matrix operations A more general property of transposes is that for two conformable matrices, A and B, (AB) = B A Thus, (X X) = X (X ) = X X SAS Programming January 16, / 43

28 Properties of matrix multiplication Most properties of addition and multiplication carry over to matrices, with a few differences. Here are some properties: A(B + C) = AB + AC (Distributive laws) (A + B)C = AC + BC (A + B)(C + D) = AC + AD + BC + BD A(BC) = (AB)C (associativity) for vectors x, y, z (x y) (x y) = x x 2x y + y y The last property relies on the fact that for n 1 vectors, x y = y x, since these products results in single numbers (scalars). SAS Programming January 16, / 43

29 Matrix review Note that if a and b are n 1 vectors, then a b is a scalar (single number): a b = n a i b i i=1 Consequently, a b = (a b) = b a while C = ab is n n. An example is that jj = J n n The length of a vector is a a = n i=1 a2 i. SAS Programming January 16, / 43

30 The inverse of a matrix Another important concept is the inverse of a matrix. If you have a matrix equation such as A = BC You might need to solve for C in terms A and C. For matrices, division isn t defined the way it is for numbers, so we have to do something else. For the equation x = yz where the values are numbers, we normally devide both sides by y. We can think of this as multiplying both sides by the multiplicative inverse of y, which is y 1. This is the approach to take with matrices. SAS Programming January 16, / 43

31 The inverse of a matrix Suppose there is a matrix X such that XB = I, the identity matrix. Then A = BC XA = XBC XA = IC = C We say that X is an inverse of B and write X = B 1. Similar to transposes, (AB) 1 = B 1 A 1. The general idea is that the product of a matrix and it s inverse is the identity matrix with 1s on the diagonal. We won t review general methods of computing inverses by hand, which is tedious except for 2 2 matrices (which we might use later). But you should be able to compute inverses in software. SAS Programming January 16, / 43

32 Matrix operations in R You should be able to do some matrix operations in R, but you should also be to do them in some computer program. We ll go over how to do them in R. First you need to have access to R. This can be downloaded and installed for free on a personal computer, or used in some of the computer pods. You can also use R online (without downloading) at r online.php SAS Programming January 16, / 43

33 Matrix operations in R We ll practice using the regression example from before with y = (452) and predictor x = (3, 2, 3). The design matrix X also has a column of 1s. SAS Programming January 16, / 43

34 Matrix operations in R Operator or Function Description A * B Element-wise multiplication A %*% B Matrix multiplication A %o% B Outer product. AB crossprod(a,b) crossprod(a) A B and A A respectively. t(a) Transpose diag(x) Creates diagonal matrix with elements of x in the principal diag(a) Returns a vector containing the elements of the principal d diag(k) If k is a scalar, this creates a k x k identity matrix. Go solve(a, b) Returns vector x in the equation b = Ax (i.e., A-1b) solve(a) Inverse of A where A is a square matrix. ginv(a) Moore-Penrose Generalized Inverse of A. ginv(a) requires loading the MASS package. y<-eigen(a) y$val are the eigenvalues of A y$vec are the eigenvectors of A y<-svd(a) Single value decomposition of A. SAS Programming January 16, / 43

35 Back to matrix review Another bit of notation that is sometimes conventient is to use a i to denote the ith row of matrix A and b j to denote the jth column of B. Then we can write the ijth element of C = AB as a i b j. The jth column of C is Ab j, and the ith row of C is a i B. SAS Programming January 16, / 43

36 Matrix review: scalar multiplication Multiplying by a scalar results in multiplying each element by the same amount. For ca, the ijth element is ca ij. Since scalar multiplication is commutative, ca = Ac. SAS Programming January 16, / 43

37 Quadratic form A quadratic form is a product of the form y Ay = i a ii y 2 i + i j a ij y i y j A bilinear form is a product of the form x Ay = ij a ij x i y j Since these are both scalars, you can have expressions such as 1 x Ay even though an expression like 1/A is undefined. SAS Programming January 16, / 43

38 Partitioned matrices It s frequently convenient to partitition matrices, like this This can also be represented a 11 a 12 b 11 a 21 a 22 b 21 c 11 c 12 d 11 ( ) A11 A 12 A 21 A 22 where A 11 is 2 2, A 12 is 2 1, A 21 is 1 2, and A 22 is 1 1. SAS Programming January 16, / 43

39 Partitioned matrices If A and B are conformable and are both partitioned so that their partitions are conformable (i.e., A ij and B ij are conformable), then the product can be performed on the submatrices. That is C = AB will have a submatrix C ij which is defined analogously to c ij in the usual matrix multiplication. For example, (Example on board...) SAS Programming January 16, / 43

40 Rank of a matrix The rank of a matrix is the number of linearly independent columns or linearly independent rows. A set of vectors y 1,..., y k is linearly independent if c 1 y c k y k = 0 if and only if c 1 = c k = 0. In other words, if you can find constants c 1,..., c k where at least one of them is nonzero, yet the linear combination is 0, then the vectors are not linearly independent. This occurs for example if two vectors are identical, one vector is the sum of the other two, etc. In statistics, the design matrix X might not have linearly independent columns, in which case we say that X is not of full rank. This can also lead to X X not being invertible, which leads to the need for generalized inverses linear model theory. SAS Programming January 16, / 43

41 Positive definite A matrix A is positive definite if y Ay > 0 for all y 0 and positive semi-definite if y Ay 0 for all y 0. A nice thing about matrices of the form X X is that they are positive definite. To show this, let A = X X. Assume that X is n p and of full rank p < n. Then y Ay = y X Xy = (Xy) (Xy) = z z for z = Xy. (Note that if X is n p and y is p 1, then z is n 1.) Here Xy 0 because Xy is a linear combination of the columns of y, and since X is full rank, then linear combination cannot equal 0. If X is not of full rank, then X X is positive semidefinite instead since you could have Xy = 0. SAS Programming January 16, / 43

42 Cholesky decomposition Matrices like X X are often analogous to squares in algebra. If you go in the other direction, you can think of factoring a square matrix into a product of two matrices, such as A = T T, in which case T is analogous to a square root of A. This can be done using the Cholesky decomposition. SAS Programming January 16, / 43

43 Cholesky Decomposition t 11 = a 11, t 1j = a 1j t11 i 1 t ii = aii t ij = tki 2 k=1 { aij i 1 k=1 t ki t kj t ii 2 j n i < j 0 otherwise SAS Programming January 16, / 43