Projection. Ping Yu. School of Economics and Finance The University of Hong Kong. Ping Yu (HKU) Projection 1 / 42
|
|
- Rebecca Jane Strickland
- 6 years ago
- Views:
Transcription
1 Projection Ping Yu School of Economics and Finance The University of Hong Kong Ping Yu (HKU) Projection 1 / 42
2 1 Hilbert Space and Projection Theorem 2 Projection in the L 2 Space 3 Projection in R n Projection Matrices 4 Partitioned Fit and Residual Regression Projection along a Subspace Ping Yu (HKU) Projection 2 / 42
3 Overview Whenever we discuss projection, there must be an underlying Hilbert space since we must define "orthogonality". We explain projection in two Hilbert spaces (L 2 and R n ) and integrate many estimators in one framework. Projection in the L 2 space: linear projection and regression (linear regression is a special case) Projection in R n : Ordinary Least Squares (OLS) and Generalized Least Squares (GLS) One main topic of this course is the (ordinary) least squares estimator (LSE). Although the LSE has many interpretations, e.g., as a MLE or a MoM estimator, the most intuitive interpretation is that it is a projection estimator. Ping Yu (HKU) Projection 2 / 42
4 Hilbert Space and Projection Theorem Hilbert Space and Projection Theorem Ping Yu (HKU) Projection 3 / 42
5 Hilbert Space and Projection Theorem Hilbert Space Definition (Hilbert Space) A complete inner product space is called a Hilbert space. a An inner product is a bilinear operator h,i : H H! R, where H is a real vector space, satisfying for any x,y,z 2 H and α 2 R, (i) hx + y,zi = hx,zi + hy,zi; (ii) hαx,zi = α hx,zi; (iii) hx,zi = hz,xi; (iv) hx,xi 0 with equal if and only if x = 0. We denote this Hilbert space as (H,h,i). a A metric space (H,d) is complete if every Cauchy sequence in H converges in H, where d is a metric on H. A sequence fx n g in a metric space is called a Cauchy sequence if for any ε > 0, there is a positive integer N such that for all natural numbers m,n > N, d(x m,x n ) < ε. Ping Yu (HKU) Projection 4 / 42
6 Hilbert Space and Projection Theorem Angle and Orthogonality An important inequality in the inner product space is the Cauchy Schwarz inequality: jhx,yij kxk kyk, where kk p h,i is the norm induced by h,i. Due to this inequality, we can define angle(x,y) = arccos hx,yi kxk kyk. We assume the value of the angle is chosen to be in the interval [0,π]. [Figure Here] If hx,yi = 0, angle(x,y) = π 2 ; we call x is orthogonal to y and denote it as x? y. Ping Yu (HKU) Projection 5 / 42
7 Hilbert Space and Projection Theorem Figure: Angle in Two-dimensional Euclidean Space Ping Yu (HKU) Projection 6 / 42
8 Hilbert Space and Projection Theorem Projection and Projector The ingredients of a projection are fy,m,(h,h,i)g, where M is a subspace of H. Note that the same H endowed with different inner products are different Hilbert spaces, so the Hilbert space is denoted as (H,h,i) rather than H. Our objective is to find some Π(y) 2 M such that Π(y) = arg min h2m ky hk2. (1) Π(): H! M is called a projector, and Π(y) is called a projection of y. Ping Yu (HKU) Projection 7 / 42
9 Hilbert Space and Projection Theorem Direct Sum, Orthogonal Space and Orthogonal Projector Definition Let M 1 and M 2 be two disjoint subspaces of H so that M 1 \ M 2 = f0g. The space V = fh 2 Hjh = h 1 + h 2,h 1 2 M 1,h 2 2 M 2 g is called the direct sum of M 1 and M 2 and it is denoted by V = M 1 M 2. Definition Let M be a subspace of H. The space M? fh 2 Hjhh,Mi = 0g is called the orthogonal space or orthogonal complement of M, where hh,mi = 0 means h is orthogonal to every element in M. Definition Suppose H = M 1 M 2. Let h 2 H so that h = h 1 + h 2 for unique h i 2 M i, i = 1,2. Then P is a projector onto M 1 along M 2 if Ph = h 1 for all h. In other words, PM 1 = M 1 and PM 2 = 0. When M 2 = M 1?, we call P as an orthogonal projector. Ping Yu (HKU) Projection 8 / 42
10 Hilbert Space and Projection Theorem Figure: Projector and Orthogonal Projector What is M 2? [Back to Lemma 9] Ping Yu (HKU) Projection 9 / 42
11 Hilbert Space and Projection Theorem Hilbert Projection Theorem Theorem (Hilbert Projection Theorem) If M is a closed subspace of a Hilbert space H, then for each y 2 H, there exists a unique point x 2 M for which ky xk is minimized over M. Moreover, x is the closest element in M to y if and only if hy x,mi = 0. The first part of the theorem states the existence and uniqueness of the projector. The second part of the theorem states something related to the first order conditions (FOCs) of (1) or, simply, orthogonal conditions. From the theorem, given any closed subspace M of H, H = M M?. Also, the closest element in M to y is determined by M itself, not the vectors generating M since there may be some redundancy in these vectors. Ping Yu (HKU) Projection 10 / 42
12 Hilbert Space and Projection Theorem Figure: Projection Ping Yu (HKU) Projection 11 / 42
13 Hilbert Space and Projection Theorem Sequential Projection Theorem (Law of Iterated Projections or LIP) If M 1 and M 2 are closed subspaces of a Hilbert space H, and M 1 M 2, then Π 1 (y) = Π 1 (Π 2 (y)), where Π j (), j = 1,2, is the orthogonal projector of y onto M j. Proof. Write y = Π 2 (y) + Π? 2 (y). Then Π 1 (y) = Π 1 (Π 2 (y) + Π? 2 (y)) = Π 1(Π 2 (y)) + Π 1 (Π? 2 (y)) = Π 1(Π 2 (y)), where the last equality is because Π? 2 (y),x = 0 for any x 2 M 2 and M 1 M 2. We first project y onto a larger space M 2, and then project the projection of y (in the first step) onto a smaller space M 1. The theorem shows that such a sequential procedure is equivalent to projecting y onto M 1 directly. We will see some applications of this theorem below. Ping Yu (HKU) Projection 12 / 42
14 Projection in the L 2 Space Projection in the L 2 Space Ping Yu (HKU) Projection 13 / 42
15 Projection in the L 2 Space Linear Projection A random variable x 2 L 2 (P) if E[x 2 ] <. L 2 (P) endowed with some inner product is a Hilbert space. y 2 L 2 (P), x 1,,x k 2 L 2 (P), M = span (x 1,,x k ) span(x), 1 H = L 2 (P) with h,i defined as hx,yi = E [xy]. h Π(y) = arg min E (y h) 2i h2m = x 0 arg min β2r k E h(y x 0 β ) 2i (2) is called the best linear predictor (BLP) of y given x, or the linear projection of y onto x. 1 span(x) = z 2 L 2 (P)jz = x 0 α,α 2 R k. Ping Yu (HKU) Projection 14 / 42
16 Projection in the L 2 Space continue... Since this is a concave programming problem, FOCs are sufficient 2 : where u = y 2E x y x 0 β 0 = 0 ) E [xu] = 0 (3) h Π(y) is the error, and β 0 = arg min E (y x 0 β ) 2i. β2r k Π(y) always exists and is unique, but β 0 needn t be unique unless x 1,,x k are linearly independent, that is, there is no nonzero vector a 2 R k such that a 0 x = 0 almost surely (a.s.). h Why? If 8 a 6= 0, a 0 x 6= 0, then E (a 0 x) 2i > 0 and a 0 E [xx 0 ]a > 0, thus E [xx 0 ] > 0. So from (3), and Π(y) = x 0 (E [xx 0 ]) 1 E [xy]. β 0 = E xx 0 1 E [xy] (why?) (4) In the literature, β with a subscript 0 usually represents the true value of β. 2 x (a0 x) = x (x0 a) = a Ping Yu (HKU) Projection 15 / 42
17 Projection in the L 2 Space Regression The setup is the same as in linear projection except that M = L 2 (P,σ(x)), where L 2 (P,σ(x)) is the space spanned by any function of x (not only the linear function of x) as long as it is in L 2 (P). h Π(y) = arg min E (y h) 2i (5) h2m Note that E h(y h) 2i = E h(y E[yjx] + E[yjx] h) 2i h = E (y E[yjx]) 2i + 2E [(y E[yjx]) (E[yjx] h)] + E h(e[yjx] h) 2i h? = E (y E[yjx]) 2i h + E (E[yjx] h) 2i h E (y E[yjx]) 2i E[u 2 ], so Π(y) = E [yjx], which is called the population regression function (PRF), where the error u satisfies E[ujx] = 0 (why?). We can use variation to characterize the FOCs: h 0 = argmin E (y (Π(y) + εh(x))) 2i ε2r 2 E [h(x) (y (Π(y) + εh(x)))]j ε=0 = 0 ) E [h(x)u] = 0, 8 h(x) 2 L 2 (P,σ(x)) (6) Ping Yu (HKU) Projection 16 / 42
18 Projection in the L 2 Space Relationship Between the Two Projections Π 1 (y) is the BLP of Π 2 (y) given x, i.e., the BLPs of y and Π 1 (y) given x are the same. This is a straightforward application of the law of iterated projections. Explicitly, define h β o = arg min E β2r k E [yjx] The FOCs for this minimization problem are x 0 β i Z h 2 = arg min E [yjx] β2r k E [ 2x(E [yjx] x 0 β o )] = 0 ) E [xx 0 ]β o = E [xe [yjx]] = E [xy] ) β o = (E [xx 0 ]) 1 E [xy] = β 0 x 0 β 2 i df (x). In other words, β 0 is a (weighted) least squares approximation to the true model. If E [yjx] is not linear in x, β o depends crucially on the weighting function F (x) or the distribution of x. The weighting function ensures that frequently drawn x i will yield small approximation errors at the cost of larger approximation errors for less frequently drawn x i. Ping Yu (HKU) Projection 17 / 42
19 Projection in the L 2 Space Figure: Linear Approximation of Conditional Expectation (I) Ping Yu (HKU) Projection 18 / 42
20 Projection in the L 2 Space Figure: Linear Approximation of Conditional Expectation (II) Ping Yu (HKU) Projection 19 / 42
21 Projection in the L 2 Space Linear Regression Linear regression is a special case of regression with E[yjx] = x 0 β. Regression and linear projection are implied by the definition of projection, but linear regression is a "model" where some structure (or restriction) is imposed. In the following figure, when we project y onto a larger space M 2 = L 2 (P,σ(x)), Π(y) falls into a smaller space M 1 = span (x) by coincidence, so there must be a restriction on the joint distribution of (y, x) (what kind of restriction?). In summary, the linear regression model is y = x 0 β + u, E[ujx] = 0. E[ujx] = 0 is necessary for a causal interpretation of β. Ping Yu (HKU) Projection 20 / 42
22 Projection in the L 2 Space Figure: Linear Regression Ping Yu (HKU) Projection 21 / 42
23 Projection in R n Projection in R n Ping Yu (HKU) Projection 22 / 42
24 Projection in R n The LSE The projection in the L 2 space is treated as the population version. The projection in R n is treated as the sample counterpart of the population version. The LSE is defined as n bβ = arg min SSR(β ) = arg min β2r k β2r k i=1 y i h x 0 i β 2 = arg min E n y x 0 β i 2, β2r k where E n [] is the expectation under the empirical distribution of the data, and n SSR(β ) y i i=1 x 0 n i β 2 = yi 2 2β 0 n x i y i + β 0 i=1 i=1 is the sum of squared residuals as a function of β. n x i x 0 i β i=1 Ping Yu (HKU) Projection 23 / 42
25 Projection in R n Figure: Objective Functions of OLS Estimation: k = 1, 2 Ping Yu (HKU) Projection 24 / 42
26 Projection in R n Normal Equations SSR(β ) is a quadratic function of β, so the FOCs are also sufficient to determine the LSE. Matrix calculus 3 gives the FOCs for b β: 0 = β SSR(b β ) = 2 = 2X 0 y + 2X 0 X b β, which is equivalent to the normal equations n i=1 X 0 X b β = X 0 y. x i y i + 2 n x i x 0 iβ b i=1 So bβ = (X 0 X) 1 X 0 y. 3 x (a0 x) = x (x0 a) = a, and x (x0 Ax) = (A + A 0 )x. Ping Yu (HKU) Projection 25 / 42
27 Projection in R n Notations Matrices are represented using uppercase bold. In matrix notation the sample (data, or dataset) is (y,x), where y is an n 1 vector with ith entry y i and X is a matrix with ith row x 0 i, i.e., y (n1) 0 B y 1.. y n 1 C A and 0 X = (nk) The first column of X is assumed to be ones if without further specification, i.e., the first column of X is 1 = (1,,1) 0. The bold zero, 0, denotes a vector or matrix of zeros. Reexpress X as X = X 1 X k, where different from x i, X j, j = 1,,k, represents the jth column of X and is all the observations for jth variable. The linear regression model upon stacking all n observations is then y = Xβ + u, where u is an n 1 column vector with ith entry u i. Ping Yu (HKU) Projection 26 / 42 x x 0 n 1 C A,
28 Projection in R n LSE as a Projection The above derivation of b β expresses the LSE using rows of the data matrices y and X. The following expresses the LSE using columns of y and X. y 2 R n, X 1,,X k 2 R n are linearly independent, M = span (X 1,,X k ) span(x), 4 H = R n with the Euclidean inner product. 5 (7) Π(y) = arg min h2m hk2 = X arg min ky β2r k Xβk 2 n = X arg min β2r k i=1 y i x 0 i β 2, where n i=1 y i x 0 i β 2 is exactly the objective function of OLS. 4 span(x) = z 2 R n jz = Xα,α 2 R k is called the column space or range space of X. 5 Recall that for x = (x 1,,x n ), and z = (z 1,,z n ), the Euclidean inner product of x and z is hx,zi = n i=1 x i z i, so kxk 2 = hx,xi = n i=1 x 2 i. Ping Yu (HKU) Projection 27 / 42
29 Projection in R n continue... As Π(y) = X b β, we can solve out b β by premultiplying both sides by X, that is, X 0 Π(y) = X 0 X b β ) b β = (X 0 X) 1 X 0 Π(y), where (X 0 X) 1 exists because X is full rank. On the other hand, orthogonal conditions for this optimization problem are where bu = y Π(y). X 0 bu = 0, Since these orthogonal conditions are equivalent to normal equations (or the FOCs), b β = (X 0 X) 1 X 0 y. These two b β s are the same since (X 0 X) 1 X 0 y (X 0 X) 1 X 0 Π(y) = (X 0 X) 1 X 0 bu = 0. Finally, Π(y) = X(X 0 X) 1 X 0 y = P X y, where P X is called the projection matrix. Ping Yu (HKU) Projection 28 / 42
30 Projection in R n Multicollinearity In the above calculation, we first project y on span(x) and then find b β by solving Π(y) = X b β. The two steps involve very different operations: optimization versus solving linear equations. Furthermore, although Π(y) is unique, b β may not be. When rank(x) < k or X is rank deficient, there are more than one (actually, infinite) b β such that X b β = Π(y). This is called multicollinearity and will be discussed in more details in the next chapter. In the following discussion, we always assume rank(x) = k or X is full-column rank; otherwise, some columns of X can be deleted to make it so. Ping Yu (HKU) Projection 29 / 42
31 Projection in R n Generalized Least Squares All are the same as in the last example except hx,zi W = x 0 Wz, where the weight matrix W is positive definite and denoted as W > 0. The projection FOCs are where eu = y Thus X e β, that is, Π(y) = X arg min β2r k ky Xβk 2 W. (8) X, eu W = 0 (orthogonal conditions) hx,xi W e β = hx,yiw ) e β = (X 0 WX) 1 X 0 Wy. Π(y) = X(X 0 WX) 1 X 0 Wy = P X?WX y where the notation P X?WX will be explained later. Ping Yu (HKU) Projection 30 / 42
32 Projection in R n Projection Matrices Projection Matrices Since Π(y) = P X y is the orthogonal projection onto span(x), P X is the orthogonal projector onto span(x). Similarly, bu = y Π(y) = (I n P X )y M X y is the orthogonal projection onto span? (X), so M X is the orthogonal projector onto span? (X), where I n is the n n identity matrix. Since P X X = X(X 0 X) 1 X 0 X = X, M X X = (I n P X )X = 0; we say P X preserves span(x), M X annihilates span(x), and M X is called the annihilator. This implies another way to express bu: bu = M X y = M X (Xβ + u) = M X u. Also, it is easy to check M X P X = 0, so M X and P X are orthogonal. Ping Yu (HKU) Projection 31 / 42
33 Projection in R n Projection Matrices continue... P X is symmetric: P 0 X = X(X 0 X) 1 X 0 0 = X(X 0 X) 1 X 0 = P X. P X is idempotent 6 (intuition?): P 2 X = X(X 0 X) 1 X 0 X(X 0 X) 1 X 0 = P X. P X is positive semidefinite: for any α 2 R n, α 0 P X α = (X 0 α) 0 (X 0 X) 1 X 0 α 0, "Positive semidefinite" cannot be strengthen to "positive definite". Why? For an idempotent matrix, the rank equals the trace 7. and tr(p X ) = tr(x(x 0 X) 1 X 0 ) = tr((x 0 X) 1 X 0 X) = tr(i k ) = k < n, tr(m X ) = tr(i n P X ) = tr(i n ) tr(p X ) = n k < n. For a general "nonorthogonal" projector P, it is still unique and idempotent, but need not be symmetric (let alone positive semidefiniteness). For example, P X?WX in the GLS estimation is not symmetric. 6 A square matrix A is idempotent if A 2 = AA = A. 7 Trace of a square matrix is the sum of its diagonal elements. tr(a + B) =tr(a)+tr(b) and tr(ab) =tr(ba). Ping Yu (HKU) Projection 32 / 42
34 Partitioned Fit and Residual Regression Partitioned Fit and Residual Regression Ping Yu (HKU) Projection 33 / 42
35 Partitioned Fit and Residual Regression Partitioned Fit It is of interest to understand the meaning of part of b β, say, b β 1 in the partition of bβ = ( b β 0 1, b β 0 2 )0, where we partition with rank(x) = k. Xβ = " X 1...X2 # β 1 β 2 We will show that b β 1 is the "net" effect of X 1 on y when the effect of X 2 is removed from the system. This result is called the Frisch-Waugh-Lovell (FWL) theorem due to Frisch and Waugh (1933) and Lovell (1963). The FWL theorem is an excellent implication of the projection property of least squares. To simplify notation, P j P Xj, M j M Xj, Π j (y) = X j b β j, j = 1,2. Ping Yu (HKU) Projection 34 / 42
36 Partitioned Fit and Residual Regression The FWL Theorem Theorem bβ 1 could be obtained when the residuals from a regression of y on X 2 alone are regressed on the set of residuals obtained when each column of X 1 is regressed on X 2. In mathematical notations, bβ 1 = X 0 1?2 X 1?2 1 X 0 1?2 y?2 = X 0 1 M 2X 1 1 X 0 1 M 2 y. where X 1?2 = (I P 2 )X 1 = M 2 X 1, y?2 = (I P 2 )y = M 2 y. This theorem states that b β 1 can be calculated by the OLS regression of ey = M 2 y on e X 1 = M 2 X 1. This technique is called residual regression. Corollary Π 1 (y) X 1 b β 1 = X 1 X 0 1?2 X 1 1 X 0 1?2 y P 12 y = P 12 (Π(y)). Ping Yu (HKU) Projection 35 / 42
37 Partitioned Fit and Residual Regression Figure: The FWL Theorem Ping Yu (HKU) Projection 36 / 42
38 Partitioned Fit and Residual Regression P 12 P 12 = X 1 {z} trailing term X 0 1 (I P 1 2)X 1 X 0 1 (I P 2 ). {z } leading term I P 2 in the leading term annihilates span(x 2 ) so that P 12 (Π 2 (y)) = 0. The leading term sends Π(y) toward span? (X 2 ). But the trailing X 1 ensures that the final result will lie in span(x 1 ). The rest of the expression for P 12 ensures that X 1 is preserved under the transformation: P 12 X 1 = X 1. Why P 12 y = P 12 (Π(y))? We can treat the projector P 12 as a sequential projector: first project y onto span(x) to get Π(y), and then project Π(y) to span(x 1 ) along span(x 2 ) to get Π 1 (y). bβ 1 is calculated from Π 1 (y) by bβ 1 = (X 0 1 X 1) 1 X 0 1 Π 1(y). Ping Yu (HKU) Projection 37 / 42
39 Partitioned Fit and Residual Regression Figure: Projection by P 12 Ping Yu (HKU) Projection 38 / 42
40 Partitioned Fit and Residual Regression Proof I of the FWL Theorem (brute-force) Calculate b β 1 explicitly in the residual regression and check whether it is equal to the LSE of β 1. Residual regression includes the following three steps. Step 1: Projecting y on X 2, we have the residuals bu y = y X 2 (X 0 2 X 2) 1 X 0 2 y = M 2y. Step 2: Projecting X 1 on X 2, we have the residuals bu x1 = X 1 X 2 (X 0 2 X 2) 1 X 0 2 X 1 = M 2 X 1. Step 3: Projecting bu y on b U x1, we get the residual regression estimator of β 1 1 eβ 1 = bu 0 b x1ux1 bu 0 x1 bu y = X 0 1 M 1 2X 1 X 0 1 M 2 y h i 1 h i = X 0 1 X 1 X 0 1 X 2(X 0 2 X 2) 1 X 0 2 X 1 X 0 1 y X0 1 X 2 X 0 2 X 1 2 X 0 2 y h i W 1 X 0 1 y X0 1 X 2 X 0 2 X 1 2 X 0 2 y Ping Yu (HKU) Projection 39 / 42
41 Partitioned Fit and Residual Regression continue... On the other hand, bβ = = and X 0 1 X 1 X 0 1 X 1 2 X 0 1 y X 0 2 X 1 X 0 2 X 2 X 0 2 y W 1 W 1 X 0 1 X 2(X 0 2 X 2) 1 X 0 1 y X 0 2 y, bβ 1 = W 1 X 0 1 y W 1 X 0 1 X 2(X 0 2 X 2) 1 X 0 h 2 i y = W 1 X 0 1 y X0 1 X 2 X 0 2 X 1 2 X 0 2 y = β e 1. The partitioned inverse formula: A11 A 12 A 21 A 22 1 = ea 1 11 ea 1 11 A 12A 1 22 A 1 22 A 21A e 1 11 A A 1 22 A 21A e 1 11 A 12A 1 22! (9) where e A 11 = A 11 A 12 A 1 22 A 21. Ping Yu (HKU) Projection 40 / 42
42 Partitioned Fit and Residual Regression Proof II of the FWL Theorem To show b β 1 = X 0 1?2 X 1?2 1 X 0 1?2 y?2, we need only show that X 0 1 M 2y = X 0 1 M 2X 1 bβ 1. Multiplying y = X 1 b β 1 + X 2 b β 2 + bu by X 0 1 M 2 on both sides, we have X 0 1 M 2y = X 0 1 M 2X 1 b β 1 + X 0 1 M 2X 2 b β 2 + X 0 1 M 2 bu = X0 1 M 2X 1 b β 1, where the last equality is from M 2 X 2 = 0, and X 0 1 M 2 bu = X0 bu 1 = 0 (why the first equality hold? bu = Mu and M 2 M = M). Ping Yu (HKU) Projection 41 / 42
43 Partitioned Fit and Residual Regression Projection along a Subspace P 12 as a Projector along a Subspace Lemma Define P X?Z as the projector onto span(x) along span? (Z), where X and Z are n k matrices and Z 0 X is nonsingular. Then P X?Z is idempotent, and P X?Z = X(Z 0 X) 1 Z 0. For orthogonal projectors, P X = P X?X. To see the difference between P X and P X?Z, we check Figure 2 again. In the left panel, X = (1,0) 0 and Z = (1,1) 0 ; in the right panel, X = (1,0) 0. (why?) It is easy to check that P X = and P X?Z = 0 0 So an orthogonal projector must be symmetric, while an projector need not be. P 12 = P X1?X 1?2.. Ping Yu (HKU) Projection 42 / 42
Least Squares Estimation-Finite-Sample Properties
Least Squares Estimation-Finite-Sample Properties Ping Yu School of Economics and Finance The University of Hong Kong Ping Yu (HKU) Finite-Sample 1 / 29 Terminology and Assumptions 1 Terminology and Assumptions
More informationMultiple Regression Model: I
Multiple Regression Model: I Suppose the data are generated according to y i 1 x i1 2 x i2 K x ik u i i 1...n Define y 1 x 11 x 1K 1 u 1 y y n X x n1 x nk K u u n So y n, X nxk, K, u n Rks: In many applications,
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationMath Camp II. Basic Linear Algebra. Yiqing Xu. Aug 26, 2014 MIT
Math Camp II Basic Linear Algebra Yiqing Xu MIT Aug 26, 2014 1 Solving Systems of Linear Equations 2 Vectors and Vector Spaces 3 Matrices 4 Least Squares Systems of Linear Equations Definition A linear
More informationLecture 6: Geometry of OLS Estimation of Linear Regession
Lecture 6: Geometry of OLS Estimation of Linear Regession Xuexin Wang WISE Oct 2013 1 / 22 Matrix Algebra An n m matrix A is a rectangular array that consists of nm elements arranged in n rows and m columns
More information3. For a given dataset and linear model, what do you think is true about least squares estimates? Is Ŷ always unique? Yes. Is ˆβ always unique? No.
7. LEAST SQUARES ESTIMATION 1 EXERCISE: Least-Squares Estimation and Uniqueness of Estimates 1. For n real numbers a 1,...,a n, what value of a minimizes the sum of squared distances from a to each of
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationLecture Notes 1: Vector spaces
Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More informationThe Hilbert Space of Random Variables
The Hilbert Space of Random Variables Electrical Engineering 126 (UC Berkeley) Spring 2018 1 Outline Fix a probability space and consider the set H := {X : X is a real-valued random variable with E[X 2
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationThe Geometry of Linear Regression
Chapter 2 The Geometry of Linear Regression 21 Introduction In Chapter 1, we introduced regression models, both linear and nonlinear, and discussed how to estimate linear regression models by using the
More informationLecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε,
2. REVIEW OF LINEAR ALGEBRA 1 Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, where Y n 1 response vector and X n p is the model matrix (or design matrix ) with one row for
More informationLinear models. Linear models are computationally convenient and remain widely used in. applied econometric research
Linear models Linear models are computationally convenient and remain widely used in applied econometric research Our main focus in these lectures will be on single equation linear models of the form y
More informationExercise Sheet 1.
Exercise Sheet 1 You can download my lecture and exercise sheets at the address http://sami.hust.edu.vn/giang-vien/?name=huynt 1) Let A, B be sets. What does the statement "A is not a subset of B " mean?
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini June 9, 2018 1 / 56 Table of Contents Multiple linear regression Linear model setup Estimation of β Geometric interpretation Estimation of σ 2 Hat matrix Gram matrix
More informationLinear Models Review
Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign
More informationContents. Appendix D (Inner Product Spaces) W-51. Index W-63
Contents Appendix D (Inner Product Spaces W-5 Index W-63 Inner city space W-49 W-5 Chapter : Appendix D Inner Product Spaces The inner product, taken of any two vectors in an arbitrary vector space, generalizes
More informationLecture 1: Systems of linear equations and their solutions
Lecture 1: Systems of linear equations and their solutions Course overview Topics to be covered this semester: Systems of linear equations and Gaussian elimination: Solving linear equations and applications
More informationAlgebra II. Paulius Drungilas and Jonas Jankauskas
Algebra II Paulius Drungilas and Jonas Jankauskas Contents 1. Quadratic forms 3 What is quadratic form? 3 Change of variables. 3 Equivalence of quadratic forms. 4 Canonical form. 4 Normal form. 7 Positive
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationorthogonal relations between vectors and subspaces Then we study some applications in vector spaces and linear systems, including Orthonormal Basis,
5 Orthogonality Goals: We use scalar products to find the length of a vector, the angle between 2 vectors, projections, orthogonal relations between vectors and subspaces Then we study some applications
More informationFinite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product
Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )
More informationSystems of Linear Equations
Systems of Linear Equations Math 108A: August 21, 2008 John Douglas Moore Our goal in these notes is to explain a few facts regarding linear systems of equations not included in the first few chapters
More informationLinear Normed Spaces (cont.) Inner Product Spaces
Linear Normed Spaces (cont.) Inner Product Spaces October 6, 017 Linear Normed Spaces (cont.) Theorem A normed space is a metric space with metric ρ(x,y) = x y Note: if x n x then x n x, and if {x n} is
More informationAnalysis and Linear Algebra. Lectures 1-3 on the mathematical tools that will be used in C103
Analysis and Linear Algebra Lectures 1-3 on the mathematical tools that will be used in C103 Set Notation A, B sets AcB union A1B intersection A\B the set of objects in A that are not in B N. Empty set
More information12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x)
1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 11 The estimate ^ is unbiased, but E(^ 2 ) = n?1 n 2 and is thus biased. An unbiased estimate is ^ 2 = 1 (x i? ^) 2 : n? 1 In x?? we show that the linear
More informationMath 413/513 Chapter 6 (from Friedberg, Insel, & Spence)
Math 413/513 Chapter 6 (from Friedberg, Insel, & Spence) David Glickenstein December 7, 2015 1 Inner product spaces In this chapter, we will only consider the elds R and C. De nition 1 Let V be a vector
More informationSTAT 540: Data Analysis and Regression
STAT 540: Data Analysis and Regression Wen Zhou http://www.stat.colostate.edu/~riczw/ Email: riczw@stat.colostate.edu Department of Statistics Colorado State University Fall 205 W. Zhou (Colorado State
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini April 27, 2018 1 / 1 Table of Contents 2 / 1 Linear Algebra Review Read 3.1 and 3.2 from text. 1. Fundamental subspace (rank-nullity, etc.) Im(X ) = ker(x T ) R
More informationMAT Linear Algebra Collection of sample exams
MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system
More informationMODULE 8 Topics: Null space, range, column space, row space and rank of a matrix
MODULE 8 Topics: Null space, range, column space, row space and rank of a matrix Definition: Let L : V 1 V 2 be a linear operator. The null space N (L) of L is the subspace of V 1 defined by N (L) = {x
More informationThe following definition is fundamental.
1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic
More information1. Subspaces A subset M of Hilbert space H is a subspace of it is closed under the operation of forming linear combinations;i.e.,
Abstract Hilbert Space Results We have learned a little about the Hilbert spaces L U and and we have at least defined H 1 U and the scale of Hilbert spaces H p U. Now we are going to develop additional
More informationSTAT 151A: Lab 1. 1 Logistics. 2 Reference. 3 Playing with R: graphics and lm() 4 Random vectors. Billy Fang. 2 September 2017
STAT 151A: Lab 1 Billy Fang 2 September 2017 1 Logistics Billy Fang (blfang@berkeley.edu) Office hours: Monday 9am-11am, Wednesday 10am-12pm, Evans 428 (room changes will be written on the chalkboard)
More informationAppendix A: Matrices
Appendix A: Matrices A matrix is a rectangular array of numbers Such arrays have rows and columns The numbers of rows and columns are referred to as the dimensions of a matrix A matrix with, say, 5 rows
More informationLINEAR ALGEBRA BOOT CAMP WEEK 1: THE BASICS
LINEAR ALGEBRA BOOT CAMP WEEK 1: THE BASICS Unless otherwise stated, all vector spaces in this worksheet are finite dimensional and the scalar field F has characteristic zero. The following are facts (in
More information4 Linear Algebra Review
Linear Algebra Review For this topic we quickly review many key aspects of linear algebra that will be necessary for the remainder of the text 1 Vectors and Matrices For the context of data analysis, the
More informationMATH 304 Linear Algebra Lecture 19: Least squares problems (continued). Norms and inner products.
MATH 304 Linear Algebra Lecture 19: Least squares problems (continued). Norms and inner products. Orthogonal projection Theorem 1 Let V be a subspace of R n. Then any vector x R n is uniquely represented
More informationElements of Convex Optimization Theory
Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a self-contained overview of elements of convex optimization
More informationChapter 4 Euclid Space
Chapter 4 Euclid Space Inner Product Spaces Definition.. Let V be a real vector space over IR. A real inner product on V is a real valued function on V V, denoted by (, ), which satisfies () (x, y) = (y,
More informationLecture 7. Econ August 18
Lecture 7 Econ 2001 2015 August 18 Lecture 7 Outline First, the theorem of the maximum, an amazing result about continuity in optimization problems. Then, we start linear algebra, mostly looking at familiar
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationRecall the convention that, for us, all vectors are column vectors.
Some linear algebra Recall the convention that, for us, all vectors are column vectors. 1. Symmetric matrices Let A be a real matrix. Recall that a complex number λ is an eigenvalue of A if there exists
More informationMa 3/103: Lecture 24 Linear Regression I: Estimation
Ma 3/103: Lecture 24 Linear Regression I: Estimation March 3, 2017 KC Border Linear Regression I March 3, 2017 1 / 32 Regression analysis Regression analysis Estimate and test E(Y X) = f (X). f is the
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationEcon 2120: Section 2
Econ 2120: Section 2 Part I - Linear Predictor Loose Ends Ashesh Rambachan Fall 2018 Outline Big Picture Matrix Version of the Linear Predictor and Least Squares Fit Linear Predictor Least Squares Omitted
More information1 Dirac Notation for Vector Spaces
Theoretical Physics Notes 2: Dirac Notation This installment of the notes covers Dirac notation, which proves to be very useful in many ways. For example, it gives a convenient way of expressing amplitudes
More informationNumerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??
Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement
More informationThis model of the conditional expectation is linear in the parameters. A more practical and relaxed attitude towards linear regression is to say that
Linear Regression For (X, Y ) a pair of random variables with values in R p R we assume that E(Y X) = β 0 + with β R p+1. p X j β j = (1, X T )β j=1 This model of the conditional expectation is linear
More informationSTAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method.
STAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method. Rebecca Barter May 5, 2015 Linear Regression Review Linear Regression Review
More informationDuke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014
Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document
More information3 Multiple Linear Regression
3 Multiple Linear Regression 3.1 The Model Essentially, all models are wrong, but some are useful. Quote by George E.P. Box. Models are supposed to be exact descriptions of the population, but that is
More informationChapter 2. Vectors and Vector Spaces
2.1. Operations on Vectors 1 Chapter 2. Vectors and Vector Spaces Section 2.1. Operations on Vectors Note. In this section, we define several arithmetic operations on vectors (especially, vector addition
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More informationThe Linear Regression Model
The Linear Regression Model Carlo Favero Favero () The Linear Regression Model 1 / 67 OLS To illustrate how estimation can be performed to derive conditional expectations, consider the following general
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 26, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 55 High dimensional
More informationDepartment of Biostatistics Department of Stat. and OR. Refresher course, Summer Linear Algebra
Department of Biostatistics Department of Stat. and OR Refresher course, Summer 2017 Linear Algebra Original Author: Oleg Mayba (UC Berkeley, 2006) Modified By: Eric Lock (UNC, 2010 & 2011) Gen Li (UNC,
More informationLinear Algebra for Machine Learning. Sargur N. Srihari
Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it
More informationVector spaces. DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis.
Vector spaces DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Vector space Consists of: A set V A scalar
More informationj=1 u 1jv 1j. 1/ 2 Lemma 1. An orthogonal set of vectors must be linearly independent.
Lecture Notes: Orthogonal and Symmetric Matrices Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Orthogonal Matrix Definition. Let u = [u
More information7. Dimension and Structure.
7. Dimension and Structure 7.1. Basis and Dimension Bases for Subspaces Example 2 The standard unit vectors e 1, e 2,, e n are linearly independent, for if we write (2) in component form, then we obtain
More information[y i α βx i ] 2 (2) Q = i=1
Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation
More informationMathematical Foundations of Applied Statistics: Matrix Algebra
Mathematical Foundations of Applied Statistics: Matrix Algebra Steffen Unkel Department of Medical Statistics University Medical Center Göttingen, Germany Winter term 2018/19 1/105 Literature Seber, G.
More informationA Primer in Econometric Theory
A Primer in Econometric Theory Lecture 1: Vector Spaces John Stachurski Lectures by Akshay Shanker May 5, 2017 1/104 Overview Linear algebra is an important foundation for mathematics and, in particular,
More informationLINEAR ALGEBRA BOOT CAMP WEEK 4: THE SPECTRAL THEOREM
LINEAR ALGEBRA BOOT CAMP WEEK 4: THE SPECTRAL THEOREM Unless otherwise stated, all vector spaces in this worksheet are finite dimensional and the scalar field F is R or C. Definition 1. A linear operator
More informationAdvanced Econometrics I
Lecture Notes Autumn 2010 Dr. Getinet Haile, University of Mannheim 1. Introduction Introduction & CLRM, Autumn Term 2010 1 What is econometrics? Econometrics = economic statistics economic theory mathematics
More informationSeminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1
Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Session: 15 Aug 2015 (Mon), 10:00am 1:00pm I. Optimization with
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationProjections and Least Square Solutions. Recall that given an inner product space V with subspace W and orthogonal basis for
Math 57 Spring 18 Projections and Least Square Solutions Recall that given an inner product space V with subspace W and orthogonal basis for W, B {v 1, v,..., v k }, the orthogonal projection of V onto
More information3 (Maths) Linear Algebra
3 (Maths) Linear Algebra References: Simon and Blume, chapters 6 to 11, 16 and 23; Pemberton and Rau, chapters 11 to 13 and 25; Sundaram, sections 1.3 and 1.5. The methods and concepts of linear algebra
More informationAnalytic Geometry. Orthogonal projection. Chapter 4 Matrix decomposition
1541 3 Analytic Geometry 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 In Chapter 2, we studied vectors, vector spaces and linear mappings at a general but abstract level. In this
More informationIntroduction to Empirical Processes and Semiparametric Inference Lecture 22: Preliminaries for Semiparametric Inference
Introduction to Empirical Processes and Semiparametric Inference Lecture 22: Preliminaries for Semiparametric Inference Michael R. Kosorok, Ph.D. Professor and Chair of Biostatistics Professor of Statistics
More informationLecture 1 and 2: Random Spanning Trees
Recent Advances in Approximation Algorithms Spring 2015 Lecture 1 and 2: Random Spanning Trees Lecturer: Shayan Oveis Gharan March 31st Disclaimer: These notes have not been subjected to the usual scrutiny
More informationSTAT200C: Review of Linear Algebra
Stat200C Instructor: Zhaoxia Yu STAT200C: Review of Linear Algebra 1 Review of Linear Algebra 1.1 Vector Spaces, Rank, Trace, and Linear Equations 1.1.1 Rank and Vector Spaces Definition A vector whose
More informationMath 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces.
Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,...,
More informationREFLECTIONS IN A EUCLIDEAN SPACE
REFLECTIONS IN A EUCLIDEAN SPACE PHILIP BROCOUM Let V be a finite dimensional real linear space. Definition 1. A function, : V V R is a bilinear form in V if for all x 1, x, x, y 1, y, y V and all k R,
More informationSemidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 2
Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 2 Instructor: Farid Alizadeh Scribe: Xuan Li 9/17/2001 1 Overview We survey the basic notions of cones and cone-lp and give several
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 23, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 47 High dimensional
More informationIntroduction to Real Analysis Alternative Chapter 1
Christopher Heil Introduction to Real Analysis Alternative Chapter 1 A Primer on Norms and Banach Spaces Last Updated: March 10, 2018 c 2018 by Christopher Heil Chapter 1 A Primer on Norms and Banach Spaces
More informationXβ is a linear combination of the columns of X: Copyright c 2010 Dan Nettleton (Iowa State University) Statistics / 25 X =
The Gauss-Markov Linear Model y Xβ + ɛ y is an n random vector of responses X is an n p matrix of constants with columns corresponding to explanatory variables X is sometimes referred to as the design
More informationLecture 13: Orthogonal projections and least squares (Section ) Thang Huynh, UC San Diego 2/9/2018
Lecture 13: Orthogonal projections and least squares (Section 3.2-3.3) Thang Huynh, UC San Diego 2/9/2018 Orthogonal projection onto subspaces Theorem. Let W be a subspace of R n. Then, each x in R n can
More informationReference: Davidson and MacKinnon Ch 2. In particular page
RNy, econ460 autumn 03 Lecture note Reference: Davidson and MacKinnon Ch. In particular page 57-8. Projection matrices The matrix M I X(X X) X () is often called the residual maker. That nickname is easy
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationAdditional Topics on Linear Regression
Additional Topics on Linear Regression Ping Yu School of Economics and Finance The University of Hong Kong Ping Yu (HKU) Additional Topics 1 / 49 1 Tests for Functional Form Misspecification 2 Nonlinear
More informationMAT 570 REAL ANALYSIS LECTURE NOTES. Contents. 1. Sets Functions Countability Axiom of choice Equivalence relations 9
MAT 570 REAL ANALYSIS LECTURE NOTES PROFESSOR: JOHN QUIGG SEMESTER: FALL 204 Contents. Sets 2 2. Functions 5 3. Countability 7 4. Axiom of choice 8 5. Equivalence relations 9 6. Real numbers 9 7. Extended
More informationLECTURE 2 LINEAR REGRESSION MODEL AND OLS
SEPTEMBER 29, 2014 LECTURE 2 LINEAR REGRESSION MODEL AND OLS Definitions A common question in econometrics is to study the effect of one group of variables X i, usually called the regressors, on another
More informationInverse of a Square Matrix. For an N N square matrix A, the inverse of A, 1
Inverse of a Square Matrix For an N N square matrix A, the inverse of A, 1 A, exists if and only if A is of full rank, i.e., if and only if no column of A is a linear combination 1 of the others. A is
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationLECTURE 7. Least Squares and Variants. Optimization Models EE 127 / EE 227AT. Outline. Least Squares. Notes. Notes. Notes. Notes.
Optimization Models EE 127 / EE 227AT Laurent El Ghaoui EECS department UC Berkeley Spring 2015 Sp 15 1 / 23 LECTURE 7 Least Squares and Variants If others would but reflect on mathematical truths as deeply
More informationTopic 7 - Matrix Approach to Simple Linear Regression. Outline. Matrix. Matrix. Review of Matrices. Regression model in matrix form
Topic 7 - Matrix Approach to Simple Linear Regression Review of Matrices Outline Regression model in matrix form - Fall 03 Calculations using matrices Topic 7 Matrix Collection of elements arranged in
More informationLecture 23: 6.1 Inner Products
Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such
More informationStatistics 612: L p spaces, metrics on spaces of probabilites, and connections to estimation
Statistics 62: L p spaces, metrics on spaces of probabilites, and connections to estimation Moulinath Banerjee December 6, 2006 L p spaces and Hilbert spaces We first formally define L p spaces. Consider
More informationLinear Algebra Review
January 29, 2013 Table of contents Metrics Metric Given a space X, then d : X X R + 0 and z in X if: d(x, y) = 0 is equivalent to x = y d(x, y) = d(y, x) d(x, y) d(x, z) + d(z, y) is a metric is for all
More informationANALYSIS OF VARIANCE AND QUADRATIC FORMS
4 ANALYSIS OF VARIANCE AND QUADRATIC FORMS The previous chapter developed the regression results involving linear functions of the dependent variable, β, Ŷ, and e. All were shown to be normally distributed
More informationidentity matrix, shortened I the jth column of I; the jth standard basis vector matrix A with its elements a ij
Notation R R n m R n m r R n s real numbers set of n m real matrices subset of R n m consisting of matrices with rank r subset of R n n consisting of symmetric matrices NND n subset of R n s consisting
More informationBoolean Inner-Product Spaces and Boolean Matrices
Boolean Inner-Product Spaces and Boolean Matrices Stan Gudder Department of Mathematics, University of Denver, Denver CO 80208 Frédéric Latrémolière Department of Mathematics, University of Denver, Denver
More informationLinear algebra I Homework #1 due Thursday, Oct Show that the diagonals of a square are orthogonal to one another.
Homework # due Thursday, Oct. 0. Show that the diagonals of a square are orthogonal to one another. Hint: Place the vertices of the square along the axes and then introduce coordinates. 2. Find the equation
More information