Lecture 13: Simple Linear Regression in Matrix Format
|
|
- Anthony Scott
- 6 years ago
- Views:
Transcription
1 See updates and corrections at Lecture 13: Simple Linear Regression in Matrix Format , Section B, Fall October 2015 Contents 1 Least Squares in Matrix Form The Basic Matrices Mean Squared Error Minimizing the MSE Fitted Values and Residuals Residuals Expectations and Covariances Sampling Distribution of Estimators 8 4 Derivatives with Respect to Vectors Second Derivatives Maxima and Minima Expectations and Variances with Vectors and Matrices 12 6 Further Reading 13 1
2 2 So far, we have not used any notions, or notation, that goes beyond basic algebra and calculus (and probability). This has forced us to do a fair amount of book-keeping, as it were by hand. This is just about tolerable for the simple linear model, with one predictor variable. It will get intolerable if we have multiple predictor variables. Fortunately, a little application of linear algebra will let us abstract away from a lot of the book-keeping details, and make multiple linear regression hardly more complicated than the simple version 1. These notes will not remind you of how matrix algebra works. However, they will review some results about calculus with matrices, and about expectations and variances with vectors and matrices. Throughout, bold-faced letters will denote matrices, as a as opposed to a scalar a. 1 Least Squares in Matrix Form Our data consists of n paired observations of the predictor variable X and the response variable Y, i.e., (x 1, y 1 ),... (x n, y n ). We wish to fit the model Y = β 0 + β 1 X + ɛ (1) where E [ɛ X = x] = 0, Var [ɛ X = x] = σ 2, and ɛ is uncorrelated across measurements The Basic Matrices Group all of the observations of the response into a single column (n 1) matrix y, y 1 y 2 y = (2). y n Similarly, we group both the coefficients into a single vector (i.e., a 2 1 matrix) [ ] β0 β = (3) β 1 We d also like to group the observations of the predictor variable together, but we need something which looks a little unusual at first: 1 x 1 1 x 2 x =. (4). 1 x n 1 Historically, linear models with multiple predictors evolved before the use of matrix algebra for regression. You may imagine the resulting drudgery. 2 When I need to also assume that ɛ is Gaussian, and strengthen uncorrelated to independent, I ll say so.
3 3 1.2 Mean Squared Error This is an n 2 matrix, where the first column is always 1, and the second column contains the actual observations of X. We have this apparently redundant first column because of what it does for us when we multiply x by β: xβ = β 0 + β 1 x 1 β 0 + β 1 x 2. β 0 + β 1 x n That is, xβ is the n 1 matrix which contains the point predictions. The matrix x is sometimes called the design matrix. 1.2 Mean Squared Error At each data point, using the coefficients β results in some error of prediction, so we have n prediction errors. These form a vector: (5) e(β) = y xβ (6) (You can check that this subtracts an n 1 matrix from an n 1 matrix.) When we derived the least squares estimator, we used the mean squared error, MSE(β) = 1 n e 2 i (β) (7) n How might we express this in terms of our matrices? I claim that the correct form is MSE(β) = 1 n et e (8) i=1 To see this, look at what the matrix multiplication really involves: e 1 e 2 [e 1 e 2... e n ]. e n (9) This, clearly equals i e2 i, so the MSE has the claimed form. Let us expand this a little for further use. MSE(β) = 1 n et e (10) = 1 n (y xβ)t (y xβ) (11) = 1 n (yt β T x T )(y xβ) (12) = 1 n ( y T y y T xβ β T x T y + β T x T xβ ) (13)
4 4 1.3 Minimizing the MSE Notice that (y T xβ) T = β T x T y. Further notice that this is a 1 1 matrix, so y T xβ = β T x T y. Thus MSE(β) = 1 ( y T y 2β T x T y + β T x T xβ ) (14) n 1.3 Minimizing the MSE First, we find the gradient of the MSE with respect to β: MSE(β = 1 ( y T y 2 β T x T y + β T x T xβ ) n (15) = 1 ( 0 2x T y + 2x T xβ ) n (16) = 2 ( x T xβ x T y ) n (17) We now set this to zero at the optimum, β: x T x β x T y = 0 (18) This equation, for the two-dimensional vector β, corresponds to our pair of normal or estimating equations for ˆβ 0 and ˆβ 1. Thus, it, too, is called an estimating equation. Solving, β = (x T x) 1 x T y (19) That is, we ve got one matrix equation which gives us both coefficient estimates. If this is right, the equation we ve got above should in fact reproduce the least-squares estimates we ve already derived, which are of course and ˆβ 1 = c XY s 2 X = xy xȳ x 2 x 2 (20) ˆβ 0 = y ˆβ 1 x (21) Let s see if that s right. As a first step, let s introduce normalizing factors of 1/n into both the matrix products: β = (n 1 x T x) 1 (n 1 x T y) (22) Now let s look at the two factors in parentheses separately, from right to left. y 1 y 2 1 n xt y = 1 [ ] n x 1 x 2... x n = 1 [ i y ] i n i x iy i [ ] y = xy. y n (23) (24) (25)
5 5 Similarly for the other factor: 1 n xt x = 1 [ n n i x i [ ] 1 x = x x 2 i x i i x2 i ] (26) (27) Now we need to take the inverse: ( ) 1 [ 1 1 x 2 x n xt x = x 2 x 2 x 1 = 1 [ ] x 2 x s 2 X x 1 ] (28) (29) Let s multiply together the pieces. (x T x) 1 x T y = 1 [ ] [ ] x 2 x y s 2 X x 1 xy = 1 [ ] x2 y xxy s 2 X xy + xy = 1 [ ] (s 2 X + x 2 )y x(c XY + xȳ) s 2 c X XY = 1 [ ] s 2 x y + x 2 y xc XY x 2 ȳ = s 2 X [ ] y c XY x s 2 X c XY s 2 X c XY (30) (31) (32) (33) (34) which is what it should be. So: n 1 x T y is keeping track of y and xy, and n 1 x T x keeps track of x and x 2. The matrix inversion and multiplication then handles all the bookkeeping to put these pieces together to get the appropriate (sample) variances, covariance, and intercepts. We don t have to remember that any more; we can just remember the one matrix equation, and then trust the linear algebra to take care of the details. 2 Fitted Values and Residuals Remember that when the coefficient vector is β, the point predictions for each data point are xβ. Thus the vector of fitted values, m(x), or m for short, is Using our equation for β, m = x β (35) m = x(x T x) 1 x T y (36)
6 6 Notice that the fitted values are linear in y. The matrix H x(x T x) 1 x T (37) does not depend on y at all, but does control the fitted values: m = Hy (38) If we repeat our experiment (survey, observation... ) many times at the same x, we get different y every time. But H does not change. The properties of the fitted values are thus largely determined by the properties of H. It thus deserves a name; it s usually called the hat matrix, for obvious reasons, or, if we want to sound more respectable, the influence matrix. Let s look at some of the properties of the hat matrix. 1. Influence Since H is not a function of y, we can easily verify that m i / y j = H ij. Thus, H ij is the rate at which the i th fitted value changes as we vary the j th observation, the influence that observation has on that fitted value. 2. Symmetry It s easy to see that H T = H. 3. Idempotency A square matrix a is called idempotent 3 when a 2 = a (and so a k = a for any higher power k). Again, by writing out the multiplication, H 2 = H, so it s idempotent. Idemopotency, Projection, Geometry Idempotency seems like the most obscure of these properties, but it s actually one of the more important. y and m are n-dimensional vectors. If we project a vector u on to the line in the direction of the length-one vector v, we get vv T u (39) (Check the dimensions: u and v are both n 1, so v T 1 1.) If we group the first two terms together, like so, is 1 n, and v T u is (vv T )u (40) where vv T is the n n project matrix or projection operator for that line. Since v is a unit vector, v T v = 1, and (vv T )(vv T ) = vv T (41) so the projection operator for the line is idempotent. The geometric meaning of idempotency here is that once we ve projected u on to the line, projecting its image on to the same line doesn t change anything. Extending this same reasoning, for any linear subspace of the n-dimensional space, there is always some n n matrix which projects vectors in arbitrary 3 From the Latin idem, same, and potens, power.
7 7 2.1 Residuals position down into the subspace, and this projection matrix is always idempotent. It is a bit more convoluted to prove that any idempotent matrix is the projection matrix for some subspace, but that s also true. We will see later how to read off the dimension of the subspace from the properties of its projection matrix. 2.1 Residuals The vector of residuals, e, is just Using the hat matrix, Here are some properties of I H: 1. Influence e i / y j = (I H) ij. 2. Symmetry (I H) T = I H. e y x β (42) e = y Hy = (I H)y (43) 3. Idempotency (I H) 2 = (I H)(I H) = I H H + H 2. But, since H is idempotent, H 2 = H, and thus (I H) 2 = (I H). Thus, simplifies to MSE( β) = 1 n yt (I H) T (I H)y (44) MSE( β) = 1 n yt (I H)y (45) 2.2 Expectations and Covariances We can of course consider the vector of random variables Y. By our modeling assumptions, Y = xβ + ɛ (46) where ɛ is an n 1 matrix of random variables, with mean vector 0, and variancecovariance matrix σ 2 I. What can we deduce from this? First, the expectation of the fitted values: E [HY] = HE [Y] (47) = Hxβ + HE [ɛ] (48) = x(x T x) 1 x T xβ + 0 (49) = xβ (50) which is as it should be, since the fitted values are unbiased.
8 8 Next, the variance-covariance of the fitted values: Var [HY] = Var [H(xβ + ɛ)] (51) using, again, the symmetry and idempotency of H. Similarly, the expected residual vector is zero: = Var [Hɛ] (52) = HVar [ɛ] H T (53) = σ 2 HIH (54) = σ 2 H (55) E [e] = (I H)(xβ + E [ɛ]) = xβ xβ = 0 (56) The variance-covariance matrix of the residuals: Var [e] = Var [(I H)(xβ + ɛ)] (57) = Var [(I H)ɛ] (58) = (I H)Var [ɛ] (I H)) T (59) = σ 2 (I H)(I H) T (60) = σ 2 (I H) (61) Thus, the variance of each residual is not quite σ 2, nor (unless H is diagonal) are the residuals exactly uncorrelated with each other. Finally, the expected MSE is [ ] 1 E n et e (62) which is 1 n E [ ɛ T (I H)ɛ ] (63) We know (because we proved it in the exam) that this must be (n 2)σ 2 /n; we ll see next time how to show this. 3 Sampling Distribution of Estimators Let s now turn on the Gaussian-noise assumption, so the noise terms ɛ i all have the distribution N(0, σ 2 ), and are independent of each other and of X. The vector of all n noise terms, ɛ, is an n 1 matrix. Its distribution is a multivariate Gaussian or multivariate normal 4, with mean vector 0, and 4 Some people write this distribution as MV N, and others also as N. I will stick to the former.
9 9 variance-covariance matrix σ 2 I. We may use this to get the sampling distribution of the estimator β: β = (x T x) 1 x T Y (64) = (x T x) 1 x T (xβ + ɛ) (65) = (x T x) 1 x T xβ + (x T x) 1 x T ɛ (66) = β + (x T x) 1 x T ɛ (67) Since ɛ is Gaussian and is being multiplied by a non-random matrix, (x T x) 1 x T ɛ is also Gaussian. Its mean vector is while its variance matrix is E [ (x T x) 1 x T ɛ ] = (x T x) 1 x T E [ɛ] = 0 (68) Var [ (x T x) 1 x T ɛ ] = (x T x) 1 x T Var [ɛ] ( (x T x) 1 x T ) T (69) = (x T x) 1 x T σ 2 Ix(x T x) 1 (70) = σ 2 (x T x) 1 x T x(x T x) 1 (71) = σ 2 (x T x) 1 (72) ] Since Var [ β = Var [ (x T x) 1 x T ɛ ] (why?), we conclude that Re-writing slightly, β MV N(β, σ 2 (x T x) 1 ) (73) β MV N(β, σ2 n (n 1 x T x) 1 ) (74) will make it easier to prove to yourself that, according [ ] to this, β 0 and β 1 are both unbiased (which we know is right), that Var ˆβ1 = σ2 n s2 X (which we know [ ] is right) and that Var ˆβ0 = σ2 n (1 + x2 /s 2 X ) (which we know is right). This will [ also give us Cov beta ˆ 0, ˆβ 1 ], which otherwise would be tedious to calculate. I will leave you to show, in a similar way, that the fitted values Hy are multivariate Gaussian, as are the residuals e, and to find both their mean vectors and their variance matrices. 4 Derivatives with Respect to Vectors This is a brief review, not intended as a full course in vector calculus. Consider some scalar function of a vector, say f(x), where x is represented as a p 1 matrix. (Here x is just being used as a place-holder or generic variable; it s not necessarily the design matrix of a regression.) We would like to think about the derivatives of f with respect to x.
10 10 Derivatives are rates of change; they tell us how rapidly the function changes in response to minute changes in its arguments. Since x is a p 1 matrix, we could also write f(x) = f(x 1, x 1, x p ) (75) This makes it clear that f will have a partial derivative with respect to each component of x. How much does f change when we vary the components? We can find this out by using a Taylor expansion. If we pick some base value of the matrix x 0 and expand around it, f(x) f(x 0 ) + p (x x 0 f ) i x i i=1 x 0 (76) = f(x 0 ) + (x x 0 ) T f(x 0 ) (77) where we define the gradient, f, to be the vector of partial derivatives, f f x 1 f x 2. f x p (78) Notice that this defines f to be a one-column matrix, just as x was taken to be. You may sometimes encounter people who want it to be a one-row matrix; it comes to the same thing, but you may have to track a lot of transposes to make use of their math. All of the properties of the gradient can be proved using those of partial derivatives. Here are some basic ones we ll need. 1. Linearity (af(x) + bg(x)) = a f(x) + b g(x) (79) Proof: Directly from the linearity of partial derivatives. 2. Linear forms If f(x) = x T a, with a not a function of x, then (x T a) = a (80) Proof: f(x) = i x ia i, so f/ x i = a i. Notice that a was already a p 1 matrix, so we don t have to transpose anything to get the derivative. 3. Linear forms the other way If f(x) = bx, with b not a function of x, then (bx) = b T (81) Proof: Once again, f/ x i = b i, but now remember that b was a 1 p matrix, and f is p 1, so we need to transpose.
11 Second Derivatives 4. Quadratic forms Let c be a p p matrix which is not a function of x, and consider the quadratic form x T cx. (You can check that this is scalar.) The gradient is (x T cx) = (c + c T )x (82) Proof: First, write out the matrix multiplications as explicit sums: x T cx = p j=1 x j p c jk x k = k=1 Now take the derivative with respect to x i. f x i = p p j=1 k=1 p j=1 k=1 p x j c jk x k (83) x j c jk x k x i (84) If j = k = i, the term in the inner sum is 2c ii x i. If j = i but k i, the term in the inner sum is c ik x k. If j i but k = i, we get x j c ji. Finally, if j i and k i, we get zero. The j = i terms add up to (cx) i. The k = i terms add up to (c T x) i. (This splits the 2c ii x i evenly between them.) Thus and f x i = ((c + c T x) i (85) f = (c + c T )x (86) (You can, and should, double check that this has the right dimensions.) 5. Symmetric quadratic forms If c = c T, then 4.1 Second Derivatives x T cx = 2cx (87) The p p matrix of second partial derivatives is called the Hessian. I won t step through its properties, except to note that they, too, follow from the basic rules for partial derivatives. 4.2 Maxima and Minima We need all the partial derivatives to be equal to zero at a minimum or maximum. This means that the gradient must be zero there. At a minimum, the Hessian must be positive-definite (so that moves away from the minimum always increase the function); at a maximum, the Hessian must be negative definite (so moves away always decrease the function). If the Hessian is neither positive nor negative definite, the point is neither a minimum nor a maximum, but a saddle (since moving in some directions increases the function but moving in others decreases it, as though one were at the center of a horse s saddle).
12 12 5 Expectations and Variances with Vectors and Matrices If we have p random variables, Z 1, Z 2,... Z p, we can grow them into a random vector Z = [Z 1 Z 2... Z p ] T. (That is, the random vector is an n 1 matrix of random variables.) This has an expected value: E [Z] zp(z)dz (88) and a little thought shows E [Z] = E [Z 1 ] E [Z 2 ]. E [Z p ] (89) Since expectations of random scalars are linear, so are expectations of random vectors: when a and b are non-random scalars, If a is a non-random matrix, E [az + bw] = ae [Z] + be [W] (90) E [az] = ae [Z] (91) Every coordinate of a random vector has some covariance with every other coordinate. The variance-covariance matrix of Z is the p p matrix which stores these: Var [Z] Var [Z 1 ] Cov [Z 1, Z 2 ]... Cov [Z 1, Z p ] Cov [Z 2, Z 1 ] Var [Z 2 ]... Cov [Z 2, Z p ].... Cov [Z p, Z 1 ] Cov [Z p, Z 2 ]... Var [Z p ] (92) This inherits properties of ordinary variances and covariances. Just Var [Z] = E [ Z 2] (E [Z]) 2, Var [Z] = E [ ZZ T ] E [Z] (E [Z]) T (93) For a non-random vector a and a non-random scalar b, For a non-random matrix c, Var [a + bz] = b 2 Var [Z] (94) Var [cz] = cvar [Z] c T (95)
13 13 (Check that the dimensions all conform here: if c is q p, Var [cz] should be q q, and so is the right-hand side.) For a quadratic form, Z T cz, with non-random c, the expectation value is E [ Z T cz ] = E [Z] T ce [Z] + tr cvar [Z] (96) where tr is of course the trace of a matrix, the sum of its diagonal entries. To see this, notice that Z T cz = tr Z T cz (97) because it s a 1 1 matrix. But the trace of a matrix product doesn t change when we cyclic permute the matrices, so Therefore Z T cz = tr czz T (98) E [ Z T cz ] = E [ tr czz T ] (99) = tr E [ czz T ] (100) = tr ce [ ZZ T ] (101) = tr c(var [Z] + E [Z] E [Z] T ) (102) = tr cvar [Z] + tr ce [Z] E [Z] T ) (103) = tr cvar [Z] + tr E [Z] T ce [Z] (104) = tr cvar [Z] + E [Z] T ce [Z] (105) using the fact that tr is a linear operation so it commutes with taking expectations; the decomposition of Var [Z]; the cyclic permutation trick again; and finally dropping tr from a scalar. Unfortunately, there is generally no simple formula for the variance of a quadratic form, unless the random vector is Gaussian. 6 Further Reading Linear algebra is a pre-requisite for this class; I strongly urge you to go back to your textbook and notes for review, if any of this is rusty. If you desire additional resources, I recommend Axler (1996) as a concise but thorough course. Petersen and Pedersen (2012), while not an introduction or even really a review, is an extremely handy compendium of matrix, and matrix-calculus, identities. References Axler, Sheldon (1996). Linear Algebra Done Right. Undergraduate Texts in Mathematics. Berlin: Springer-Verlag.
14 14 REFERENCES Petersen, Kaare Brandt and Michael Syskind Pedersen (2012). The Matrix Cookbook. Tech. rep., Technical University of Denmark, Intelligent Signal Processing Group. URL publication_details.php?id=3274.
Lecture 13: Simple Linear Regression in Matrix Format. 1 Expectations and Variances with Vectors and Matrices
Lecture 3: Simple Linear Regression in Matrix Format To move beyond simple regression we need to use matrix algebra We ll start by re-expressing simple linear regression in matrix form Linear algebra is
Linear Models Review
Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign
Lecture 9 SLR in Matrix Form
Lecture 9 SLR in Matrix Form STAT 51 Spring 011 Background Reading KNNL: Chapter 5 9-1 Topic Overview Matrix Equations for SLR Don t focus so much on the matrix arithmetic as on the form of the equations.
MATH 320, WEEK 7: Matrices, Matrix Operations
MATH 320, WEEK 7: Matrices, Matrix Operations 1 Matrices We have introduced ourselves to the notion of the grid-like coefficient matrix as a short-hand coefficient place-keeper for performing Gaussian
Basic Linear Algebra in MATLAB
Basic Linear Algebra in MATLAB 9.29 Optional Lecture 2 In the last optional lecture we learned the the basic type in MATLAB is a matrix of double precision floating point numbers. You learned a number
Linear Algebra Review
Linear Algebra Review Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Linear Algebra Review 1 / 45 Definition of Matrix Rectangular array of elements arranged in rows and
A primer on matrices
A primer on matrices Stephen Boyd August 4, 2007 These notes describe the notation of matrices, the mechanics of matrix manipulation, and how to use matrices to formulate and solve sets of simultaneous
Getting Started with Communications Engineering. Rows first, columns second. Remember that. R then C. 1
1 Rows first, columns second. Remember that. R then C. 1 A matrix is a set of real or complex numbers arranged in a rectangular array. They can be any size and shape (provided they are rectangular). A
22 Approximations - the method of least squares (1)
22 Approximations - the method of least squares () Suppose that for some y, the equation Ax = y has no solutions It may happpen that this is an important problem and we can t just forget about it If we
Linear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,
Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,
A primer on matrices
A primer on matrices Stephen Boyd August 4, 2007 These notes describe the notation of matrices, the mechanics of matrix manipulation, and how to use matrices to formulate and solve sets of simultaneous
Math (P)Review Part II:
Math (P)Review Part II: Vector Calculus Computer Graphics Assignment 0.5 (Out today!) Same story as last homework; second part on vector calculus. Slightly fewer questions Last Time: Linear Algebra Touched
The Hilbert Space of Random Variables
The Hilbert Space of Random Variables Electrical Engineering 126 (UC Berkeley) Spring 2018 1 Outline Fix a probability space and consider the set H := {X : X is a real-valued random variable with E[X 2
. a m1 a mn. a 1 a 2 a = a n
Biostat 140655, 2008: Matrix Algebra Review 1 Definition: An m n matrix, A m n, is a rectangular array of real numbers with m rows and n columns Element in the i th row and the j th column is denoted by
Chapter 5 Matrix Approach to Simple Linear Regression
STAT 525 SPRING 2018 Chapter 5 Matrix Approach to Simple Linear Regression Professor Min Zhang Matrix Collection of elements arranged in rows and columns Elements will be numbers or symbols For example:
Math 308 Midterm Answers and Comments July 18, Part A. Short answer questions
Math 308 Midterm Answers and Comments July 18, 2011 Part A. Short answer questions (1) Compute the determinant of the matrix a 3 3 1 1 2. 1 a 3 The determinant is 2a 2 12. Comments: Everyone seemed to
Inverse of a Square Matrix. For an N N square matrix A, the inverse of A, 1
Inverse of a Square Matrix For an N N square matrix A, the inverse of A, 1 A, exists if and only if A is of full rank, i.e., if and only if no column of A is a linear combination 1 of the others. A is
M. Matrices and Linear Algebra
M. Matrices and Linear Algebra. Matrix algebra. In section D we calculated the determinants of square arrays of numbers. Such arrays are important in mathematics and its applications; they are called matrices.
Approximations - the method of least squares (1)
Approximations - the method of least squares () In many applications, we have to consider the following problem: Suppose that for some y, the equation Ax = y has no solutions It could be that this is an
Dot Products, Transposes, and Orthogonal Projections
Dot Products, Transposes, and Orthogonal Projections David Jekel November 13, 2015 Properties of Dot Products Recall that the dot product or standard inner product on R n is given by x y = x 1 y 1 + +
Math 291-2: Lecture Notes Northwestern University, Winter 2016
Math 291-2: Lecture Notes Northwestern University, Winter 2016 Written by Santiago Cañez These are lecture notes for Math 291-2, the second quarter of MENU: Intensive Linear Algebra and Multivariable Calculus,
The Multivariate Gaussian Distribution [DRAFT]
The Multivariate Gaussian Distribution DRAFT David S. Rosenberg Abstract This is a collection of a few key and standard results about multivariate Gaussian distributions. I have not included many proofs,
Topic 7 - Matrix Approach to Simple Linear Regression. Outline. Matrix. Matrix. Review of Matrices. Regression model in matrix form
Topic 7 - Matrix Approach to Simple Linear Regression Review of Matrices Outline Regression model in matrix form - Fall 03 Calculations using matrices Topic 7 Matrix Collection of elements arranged in
Coordinate systems and vectors in three spatial dimensions
PHYS2796 Introduction to Modern Physics (Spring 2015) Notes on Mathematics Prerequisites Jim Napolitano, Department of Physics, Temple University January 7, 2015 This is a brief summary of material on
Regression. Oscar García
Regression Oscar García Regression methods are fundamental in Forest Mensuration For a more concise and general presentation, we shall first review some matrix concepts 1 Matrices An order n m matrix is
Stat 206: Sampling theory, sample moments, mahalanobis
Stat 206: Sampling theory, sample moments, mahalanobis topology James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Notation My notation is different from the book s. This is partly because
Stat 206: Linear algebra
Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two
Business Statistics. Lecture 9: Simple Regression
Business Statistics Lecture 9: Simple Regression 1 On to Model Building! Up to now, class was about descriptive and inferential statistics Numerical and graphical summaries of data Confidence intervals
Final Review Sheet. B = (1, 1 + 3x, 1 + x 2 ) then 2 + 3x + 6x 2
Final Review Sheet The final will cover Sections Chapters 1,2,3 and 4, as well as sections 5.1-5.4, 6.1-6.2 and 7.1-7.3 from chapters 5,6 and 7. This is essentially all material covered this term. Watch
Review from Bootcamp: Linear Algebra
Review from Bootcamp: Linear Algebra D. Alex Hughes October 27, 2014 1 Properties of Estimators 2 Linear Algebra Addition and Subtraction Transpose Multiplication Cross Product Trace 3 Special Matrices
REVIEW OF DIFFERENTIAL CALCULUS
REVIEW OF DIFFERENTIAL CALCULUS DONU ARAPURA 1. Limits and continuity To simplify the statements, we will often stick to two variables, but everything holds with any number of variables. Let f(x, y) be
Sometimes the domains X and Z will be the same, so this might be written:
II. MULTIVARIATE CALCULUS The first lecture covered functions where a single input goes in, and a single output comes out. Most economic applications aren t so simple. In most cases, a number of variables
Math 123, Week 2: Matrix Operations, Inverses
Math 23, Week 2: Matrix Operations, Inverses Section : Matrices We have introduced ourselves to the grid-like coefficient matrix when performing Gaussian elimination We now formally define general matrices
POL 213: Research Methods
Brad 1 1 Department of Political Science University of California, Davis April 15, 2008 Some Matrix Basics What is a matrix? A rectangular array of elements arranged in rows and columns. 55 900 0 67 1112
Lecture 10: F -Tests, ANOVA and R 2
Lecture 10: F -Tests, ANOVA and R 2 1 ANOVA We saw that we could test the null hypothesis that β 1 0 using the statistic ( β 1 0)/ŝe. (Although I also mentioned that confidence intervals are generally
Partial Fractions. June 27, In this section, we will learn to integrate another class of functions: the rational functions.
Partial Fractions June 7, 04 In this section, we will learn to integrate another class of functions: the rational functions. Definition. A rational function is a fraction of two polynomials. For example,
3 Fields, Elementary Matrices and Calculating Inverses
3 Fields, Elementary Matrices and Calculating Inverses 3. Fields So far we have worked with matrices whose entries are real numbers (and systems of equations whose coefficients and solutions are real numbers).
STAT Homework 8 - Solutions
STAT-36700 Homework 8 - Solutions Fall 208 November 3, 208 This contains solutions for Homework 4. lease note that we have included several additional comments and approaches to the problems to give you
[Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty.]
Math 43 Review Notes [Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty Dot Product If v (v, v, v 3 and w (w, w, w 3, then the
Linear Algebra for Beginners Open Doors to Great Careers. Richard Han
Linear Algebra for Beginners Open Doors to Great Careers Richard Han Copyright 2018 Richard Han All rights reserved. CONTENTS PREFACE... 7 1 - INTRODUCTION... 8 2 SOLVING SYSTEMS OF LINEAR EQUATIONS...
Finite Mathematics : A Business Approach
Finite Mathematics : A Business Approach Dr. Brian Travers and Prof. James Lampes Second Edition Cover Art by Stephanie Oxenford Additional Editing by John Gambino Contents What You Should Already Know
1 Matrices and matrix algebra
1 Matrices and matrix algebra 1.1 Examples of matrices A matrix is a rectangular array of numbers and/or variables. For instance 4 2 0 3 1 A = 5 1.2 0.7 x 3 π 3 4 6 27 is a matrix with 3 rows and 5 columns
STAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method.
STAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method. Rebecca Barter May 5, 2015 Linear Regression Review Linear Regression Review
Lecture 6 Positive Definite Matrices
Linear Algebra Lecture 6 Positive Definite Matrices Prof. Chun-Hung Liu Dept. of Electrical and Computer Engineering National Chiao Tung University Spring 2017 2017/6/8 Lecture 6: Positive Definite Matrices
Linear Algebra, Summer 2011, pt. 2
Linear Algebra, Summer 2, pt. 2 June 8, 2 Contents Inverses. 2 Vector Spaces. 3 2. Examples of vector spaces..................... 3 2.2 The column space......................... 6 2.3 The null space...........................
Lecture 24: Weighted and Generalized Least Squares
Lecture 24: Weighted and Generalized Least Squares 1 Weighted Least Squares When we use ordinary least squares to estimate linear regression, we minimize the mean squared error: MSE(b) = 1 n (Y i X i β)
Answers in blue. If you have questions or spot an error, let me know. 1. Find all matrices that commute with A =. 4 3
Answers in blue. If you have questions or spot an error, let me know. 3 4. Find all matrices that commute with A =. 4 3 a b If we set B = and set AB = BA, we see that 3a + 4b = 3a 4c, 4a + 3b = 3b 4d,
a11 a A = : a 21 a 22
Matrices The study of linear systems is facilitated by introducing matrices. Matrix theory provides a convenient language and notation to express many of the ideas concisely, and complicated formulas are
Math101, Sections 2 and 3, Spring 2008 Review Sheet for Exam #2:
Math101, Sections 2 and 3, Spring 2008 Review Sheet for Exam #2: 03 17 08 3 All about lines 3.1 The Rectangular Coordinate System Know how to plot points in the rectangular coordinate system. Know the
Diagonalization. MATH 1502 Calculus II Notes. November 4, 2008
Diagonalization MATH 1502 Calculus II Notes November 4, 2008 We want to understand all linear transformations L : R n R m. The basic case is the one in which n = m. That is to say, the case in which the
Statistics 910, #5 1. Regression Methods
Statistics 910, #5 1 Overview Regression Methods 1. Idea: effects of dependence 2. Examples of estimation (in R) 3. Review of regression 4. Comparisons and relative efficiencies Idea Decomposition Well-known
Notes on Linear Algebra I. # 1
Notes on Linear Algebra I. # 1 Oussama Moutaoikil Contents 1 Introduction 1 2 On Vector Spaces 5 2.1 Vectors................................... 5 2.2 Vector Spaces................................ 7 2.3
Tangent spaces, normals and extrema
Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent
[y i α βx i ] 2 (2) Q = i=1
Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation
MATH 310, REVIEW SHEET 2
MATH 310, REVIEW SHEET 2 These notes are a very short summary of the key topics in the book (and follow the book pretty closely). You should be familiar with everything on here, but it s not comprehensive,
Example: 2x y + 3z = 1 5y 6z = 0 x + 4z = 7. Definition: Elementary Row Operations. Example: Type I swap rows 1 and 3
Linear Algebra Row Reduced Echelon Form Techniques for solving systems of linear equations lie at the heart of linear algebra. In high school we learn to solve systems with or variables using elimination
Getting Started with Communications Engineering
1 Linear algebra is the algebra of linear equations: the term linear being used in the same sense as in linear functions, such as: which is the equation of a straight line. y ax c (0.1) Of course, if we
1 GSW Sets of Systems
1 Often, we have to solve a whole series of sets of simultaneous equations of the form y Ax, all of which have the same matrix A, but each of which has a different known vector y, and a different unknown
STA 2101/442 Assignment Four 1
STA 2101/442 Assignment Four 1 One version of the general linear model with fixed effects is y = Xβ + ɛ, where X is an n p matrix of known constants with n > p and the columns of X linearly independent.
STAT 350: Geometry of Least Squares
The Geometry of Least Squares Mathematical Basics Inner / dot product: a and b column vectors a b = a T b = a i b i a b a T b = 0 Matrix Product: A is r s B is s t (AB) rt = s A rs B st Partitioned Matrices
Reference: Davidson and MacKinnon Ch 2. In particular page
RNy, econ460 autumn 03 Lecture note Reference: Davidson and MacKinnon Ch. In particular page 57-8. Projection matrices The matrix M I X(X X) X () is often called the residual maker. That nickname is easy
Designing Information Devices and Systems I Fall 2018 Lecture Notes Note Positioning Sytems: Trilateration and Correlation
EECS 6A Designing Information Devices and Systems I Fall 08 Lecture Notes Note. Positioning Sytems: Trilateration and Correlation In this note, we ll introduce two concepts that are critical in our positioning
MS&E 226. In-Class Midterm Examination Solutions Small Data October 20, 2015
MS&E 226 In-Class Midterm Examination Solutions Small Data October 20, 2015 PROBLEM 1. Alice uses ordinary least squares to fit a linear regression model on a dataset containing outcome data Y and covariates
1 Review of the dot product
Any typographical or other corrections about these notes are welcome. Review of the dot product The dot product on R n is an operation that takes two vectors and returns a number. It is defined by n u
Image Registration Lecture 2: Vectors and Matrices
Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this
Chapter 7. Extremal Problems. 7.1 Extrema and Local Extrema
Chapter 7 Extremal Problems No matter in theoretical context or in applications many problems can be formulated as problems of finding the maximum or minimum of a function. Whenever this is the case, advanced
1 Least Squares Estimation - multiple regression.
Introduction to multiple regression. Fall 2010 1 Least Squares Estimation - multiple regression. Let y = {y 1,, y n } be a n 1 vector of dependent variable observations. Let β = {β 0, β 1 } be the 2 1
Gaussian Quiz. Preamble to The Humble Gaussian Distribution. David MacKay 1
Preamble to The Humble Gaussian Distribution. David MacKay Gaussian Quiz H y y y 3. Assuming that the variables y, y, y 3 in this belief network have a joint Gaussian distribution, which of the following
Lecture Notes 1: Vector spaces
Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
Math 4310 Solutions to homework 1 Due 9/1/16
Math 0 Solutions to homework Due 9//6. An element [a] Z/nZ is idempotent if [a] 2 [a]. Find all idempotent elements in Z/0Z and in Z/Z. Solution. First note we clearly have [0] 2 [0] so [0] is idempotent
STEP Support Programme. STEP 2 Matrices Topic Notes
STEP Support Programme STEP 2 Matrices Topic Notes Definitions............................................. 2 Manipulating Matrices...................................... 3 Transformations.........................................
In the bivariate regression model, the original parameterization is. Y i = β 1 + β 2 X2 + β 2 X2. + β 2 (X 2i X 2 ) + ε i (2)
RNy, econ460 autumn 04 Lecture note Orthogonalization and re-parameterization 5..3 and 7.. in HN Orthogonalization of variables, for example X i and X means that variables that are correlated are made
Vector calculus background
Vector calculus background Jiří Lebl January 18, 2017 This class is really the vector calculus that you haven t really gotten to in Calc III. Let us start with a very quick review of the concepts from
Topics in linear algebra
Chapter 6 Topics in linear algebra 6.1 Change of basis I want to remind you of one of the basic ideas in linear algebra: change of basis. Let F be a field, V and W be finite dimensional vector spaces over
1 Matrices and Systems of Linear Equations. a 1n a 2n
March 31, 2013 16-1 16. Systems of Linear Equations 1 Matrices and Systems of Linear Equations An m n matrix is an array A = (a ij ) of the form a 11 a 21 a m1 a 1n a 2n... a mn where each a ij is a real
Linear Methods for Prediction
Chapter 5 Linear Methods for Prediction 5.1 Introduction We now revisit the classification problem and focus on linear methods. Since our prediction Ĝ(x) will always take values in the discrete set G we
MAT2342 : Introduction to Applied Linear Algebra Mike Newman, fall Projections. introduction
MAT4 : Introduction to Applied Linear Algebra Mike Newman fall 7 9. Projections introduction One reason to consider projections is to understand approximate solutions to linear systems. A common example
Lecture 9: Elementary Matrices
Lecture 9: Elementary Matrices Review of Row Reduced Echelon Form Consider the matrix A and the vector b defined as follows: 1 2 1 A b 3 8 5 A common technique to solve linear equations of the form Ax
Ordinary Least Squares Linear Regression
Ordinary Least Squares Linear Regression Ryan P. Adams COS 324 Elements of Machine Learning Princeton University Linear regression is one of the simplest and most fundamental modeling ideas in statistics
Some Notes on Linear Algebra
Some Notes on Linear Algebra prepared for a first course in differential equations Thomas L Scofield Department of Mathematics and Statistics Calvin College 1998 1 The purpose of these notes is to present
Gaussian elimination
Gaussian elimination October 14, 2013 Contents 1 Introduction 1 2 Some definitions and examples 2 3 Elementary row operations 7 4 Gaussian elimination 11 5 Rank and row reduction 16 6 Some computational
Topic 15 Notes Jeremy Orloff
Topic 5 Notes Jeremy Orloff 5 Transpose, Inverse, Determinant 5. Goals. Know the definition and be able to compute the inverse of any square matrix using row operations. 2. Know the properties of inverses.
The Matrix Algebra of Sample Statistics
The Matrix Algebra of Sample Statistics James H. Steiger Department of Psychology and Human Development Vanderbilt University James H. Steiger (Vanderbilt University) The Matrix Algebra of Sample Statistics
Peter Hoff Linear and multilinear models April 3, GLS for multivariate regression 5. 3 Covariance estimation for the GLM 8
Contents 1 Linear model 1 2 GLS for multivariate regression 5 3 Covariance estimation for the GLM 8 4 Testing the GLH 11 A reference for some of this material can be found somewhere. 1 Linear model Recall
Linear Regression 9/23/17. Simple linear regression. Advertising sales: Variance changes based on # of TVs. Advertising sales: Normal error?
Simple linear regression Linear Regression Nicole Beckage y " = β % + β ' x " + ε so y* " = β+ % + β+ ' x " Method to assess and evaluate the correlation between two (continuous) variables. The slope of
Notes on Random Variables, Expectations, Probability Densities, and Martingales
Eco 315.2 Spring 2006 C.Sims Notes on Random Variables, Expectations, Probability Densities, and Martingales Includes Exercise Due Tuesday, April 4. For many or most of you, parts of these notes will be
Vectors and Matrices Statistics with Vectors and Matrices
Vectors and Matrices Statistics with Vectors and Matrices Lecture 3 September 7, 005 Analysis Lecture #3-9/7/005 Slide 1 of 55 Today s Lecture Vectors and Matrices (Supplement A - augmented with SAS proc
Mathematics for Graphics and Vision
Mathematics for Graphics and Vision Steven Mills March 3, 06 Contents Introduction 5 Scalars 6. Visualising Scalars........................ 6. Operations on Scalars...................... 6.3 A Note on
APPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
Precalculus idea: A picture is worth 1,000 words
Six Pillars of Calculus by Lorenzo Sadun Calculus is generally viewed as a difficult subject, with hundreds of formulas to memorize and many applications to the real world. However, almost all of calculus
Vector, Matrix, and Tensor Derivatives
Vector, Matrix, and Tensor Derivatives Erik Learned-Miller The purpose of this document is to help you learn to take derivatives of vectors, matrices, and higher order tensors (arrays with three dimensions
Factoring. there exists some 1 i < j l such that x i x j (mod p). (1) p gcd(x i x j, n).
18.310 lecture notes April 22, 2015 Factoring Lecturer: Michel Goemans We ve seen that it s possible to efficiently check whether an integer n is prime or not. What about factoring a number? If this could
Solving Quadratic Equations
Concepts: Solving Quadratic Equations, Completing the Square, The Quadratic Formula, Sketching Quadratics Solving Quadratic Equations Completing the Square ax + bx + c = a x + ba ) x + c Factor so the
5 More on Linear Algebra
14.102, Math for Economists Fall 2004 Lecture Notes, 9/23/2004 These notes are primarily based on those written by George Marios Angeletos for the Harvard Math Camp in 1999 and 2000, and updated by Stavros
CS168: The Modern Algorithmic Toolbox Lecture #7: Understanding Principal Component Analysis (PCA)
CS68: The Modern Algorithmic Toolbox Lecture #7: Understanding Principal Component Analysis (PCA) Tim Roughgarden & Gregory Valiant April 0, 05 Introduction. Lecture Goal Principal components analysis
Regression, part II. I. What does it all mean? A) Notice that so far all we ve done is math.
Regression, part II I. What does it all mean? A) Notice that so far all we ve done is math. 1) One can calculate the Least Squares Regression Line for anything, regardless of any assumptions. 2) But, if
MAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012
(Homework 1: Chapter 1: Exercises 1-7, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions
The General Linear Model in Functional MRI
The General Linear Model in Functional MRI Henrik BW Larsson Functional Imaging Unit, Glostrup Hospital University of Copenhagen Part I 1 2 Preface The General Linear Model (GLM) or multiple regression
EXAM 2 REVIEW DAVID SEAL
EXAM 2 REVIEW DAVID SEAL 3. Linear Systems and Matrices 3.2. Matrices and Gaussian Elimination. At this point in the course, you all have had plenty of practice with Gaussian Elimination. Be able to row
Stat751 / CSI771 Midterm October 15, 2015 Solutions, Comments. f(x) = 0 otherwise
Stat751 / CSI771 Midterm October 15, 2015 Solutions, Comments 1. 13pts Consider the beta distribution with PDF Γα+β ΓαΓβ xα 1 1 x β 1 0 x < 1 fx = 0 otherwise, for fixed constants 0 < α, β. Now, assume