Vectors Vectors and the scalar multiplication and vector addition operations: x 1 x 1 y 1 2x 1 + 3y 1 x x n 1 = 2 x R n, 2 2 y + 3 2 2x = 2 + 3y 2............ x n x n y n 2x n + 3y n I ll use the two terms vector and point interchangeable: any point R n corresponds to a vector starting from the origin and ending at that point. 1
The inner (dot or cross) product of two vectors is defined to be u t v = i u i v i = u v cos(θ), where u denotes the norm of a vector u = u t u = u 2 i, i and θ is the angle between the two vectors. A unit vector is a vector whose norm is 1. When two vectors are orthogonal, we mean cos(θ) = 0, therefore u t v = 0, denoted by u v. The Euclidean distance between two vectors u and v is u v. 2
Linear Combinations & Spans A linear combination of vectors x 1,..., x p is b 1 x 1 + b 2 x 2 + + b p x p, b 1,..., b p R. Span(x 1,..., x p ) denotes the set of all linear combinations of the p vectors, x 1,..., x p. Concatenate the p vectors into a matrix X n p = (x 1 x p ). The column space of X is C(X) = Span(x 1,..., x p ). Any vector in C(X) can be written as X n p b p 1, where b = (b 1,..., b p ) t. 3
Linear Subspace Let M be a collection of vectors from R n. M is a vector space if M is closed under linear combinations, that is, if u, v M, then au + bv M, where a, b R. You can image a linear subspace as a bag of vectors, and for any two vectors in of that bag, say u and v (the two vectors could be the same, i.e., you are allowed to create copies of vectors in that bag), their linear combination, say u 2v, should also be in that bag. Apparently, we have u u = 0, so 0 is in any linear subspace. (i.e., any linear subspace should pass the origin). 4
Examples of vector spaces 0 (the smallest subspace) R n (the largest one) Collection of all points on a line passing through the origin A span of (v 1,..., v m ) is a linear subspace The column space of a matrix X n p, C(X). Apparently, if we change the order of the columns of X, the column space C(X) remains the same. M and N are vector spaces, so are M N and M + N = {u + v, u M, v N }. But M N may not. 5
Replacement Rule Given p vectors: x 1,..., x p, define z 1 = a 1 x 1 + a 2 x 2 + + a p x p. If a 1 0, then the two follow subspaces are the same Span(x 1, x 2,..., x p ) = Span(z 1, x 2,..., x p ). Similarly, let X be a new matrix which is the same as X except that we replace the jth column by a linear combination, X[, j] = a j X[, j] + i j a i X[, i]. If a j 0, then C( X) = C(X). 6
Orthogonality Vector Vector: u v if u t v = 0. Vector Subspace: u M, if u is orthogonal to any vector from M. For example, if u is orthogonal to each column of a matrix X, then we have u C(X). Subspace Subspace: Similarly we can define M N, if any vector from M is orthogonal to any vector from N. Define the orthogonal complement of subspace V as M = {x R n : x M}. It is easy to show that M is a subspace. Apparently M M. 7
Linear Independence A set of vectors v 1,..., v m is said to be linear independent, if c 1 v 1 + + c m v m = 0 iff c 1 = = c m = 0. Otherwise they are linear dependent. In other words, if a set of vectors are linear independent, then no one can be expressed as a linear combination of the others; if they are linear dependent, then there exists one vector, say v 2, which can be written as a linear combination of v 1, v 3,..., v m. 8
Linear Independence and Bases A set of vectors {u 1,..., u m } is a basis for a subspace M, if 1. Span(u 1,..., u m ) = M, and 2. u 1,..., u m are linear independent. That is, a basis is a set of vectors that spans a linear subspace M without redundancy. An orthonormal basis (ONB), if in addition u j s are unit vectors and orthogonal to each other. 9
If {u 1,..., u m } is a basis for a subspace M, then any vector in M can be uniquely represented by the linear combination of u i s. That is, if we can write a vector v as v = c 1 u 1 + c 2 u 2 + + c m u m, and also v = a 1 u 1 + a 2 u 2 + + a m u m, then c i = a i for all i = 1 : m. Bases are not unique. That is, a linear space M has more than one bases, but the number of vectors in each basis is the same, which is the rank/dim of M. If M has dim p, then any set of more than p vectors from M is linear dependent. If M has dim p, then any set of p linearly independent vectors in M form a basis for M. 10
Gram-Schmidt Procedure Given a basis {x 1,..., x p }, we can use the Gram-Schmidt algorithm to obtain an ONB. u 1 = x 1, e 1 = u 1 u 1 u 2 = x 2 (x t 2e 1 )e 1, e 2 = u 2 u 2 u k+1 = x k+1 k j=1 (xt j e j)e j, e k+1 = u k+1 u k+1 The resulting ONB is {e 1,..., e p }. 11
OLS Solution Consider a linear model y i = x i1 β 1 + + x ip β p + err i, i = 1,..., n Using the LS principal, we aim to find β = (β 1,..., β p ) t, which minimizes n (y i x i1 β 1 x ip β p ) 2. i=1 Using the matrix form, we can write the linear model as and solve y n 1 = X n p β p 1 + e, min y β R Xβ 2. (1) p 12
The LS optimization min y β R Xβ 2. p is equivalent to finding a vector z in C(X), such that y z = min y z C(X) z 2. Once we solve z, we then go back to find its representation β. 13