# Lecture 6: Geometry of OLS Estimation of Linear Regession

Save this PDF as:
Size: px
Start display at page:

## Transcription

1 Lecture 6: Geometry of OLS Estimation of Linear Regession Xuexin Wang WISE Oct / 22

2 Matrix Algebra An n m matrix A is a rectangular array that consists of nm elements arranged in n rows and m columns A typical element of A might be denoted by either A ij or a ij, where i = 1,, n and j = 1,, m If a matrix has only one column or only one row, it is called a vector There are two types of vectors, column vectors and row vectors If a matrix has the same number of columns and rows, it is said to be square A square matrix A is symmetric ifa ij = A ji for all i and j A square matrix is said to be diagonal if A ij = 0 for all i j The transpose of A is obtained by interchanging its row and column subscripts, denote A T or A 2 / 22

3 Arithmetic Operations on Matrices Addition and subtraction of matrices works exactly the way it does for scalars Matrix multiplication: A B = C n mm l AB BA except in special cases Identity matrix: I, IB = BI n l Assuming that the dimensions of the matrices are conformable for the various operations Distributive properties Associative properties A(B + C) = AB + AC (B + C)A = BA + CA (A + B) + C = A + (B + C) (AB)C = A(BC) 3 / 22

4 Transpose and Inverse (AB) T = B T A T If A is invertible, then it has an inverse matrix A 1 with the property that AA 1 = A 1 A = I If A is symmetric, then so is A 1 If A is triangular, then so is A 1 If an n n square matrix A is invertible, then its rank is n Such a matrix is said to have full rank If a square matrix does not have full rank, and therefore is not invertible, it is said to be singular for matrices that are not necessarily square, the rank is the largest number m for which an m m nonsingular matrix can be constructed by omitting some rows and some columns from the original matrix 4 / 22

5 Regression Models and Matrix Notation Model with one regressor y 1 = β 0 + β 0 x 11 + u 1 y 2 = β 0 + β 0 x 12 + u 2 y n = β 0 + β 0 x 1n + u n where y = Matrix notation: y 1 y 2 y n 1 x 11 1 x 12 y = Xβ + u,, X =, u = 1 x 1n u 1 u 2 u n, β = ( β0 β 1 ) 5 / 22

6 Multiple Regression Model Matrix notation: where y 1 y 2 y =, X = y n y = Xβ + u, x 11 x k1 x 12 x k2 x 1n x kn Incorporate the case of intercept, β = β 1 β k 6 / 22

7 Partitioned Matrices [ X = x 1 n k n 1 X = n k X = n k X 1 X 2 X n [ k1 x 2 n 1 x k n 1 1 k 1 k 1 k ] X 11 k 2 X 12 X 21 X 22 ], n 1 n 2 If two matrices A and B of the same dimensions are partitioned in exactly the same way, they can be added or subtracted block by block A + B = [ A 1 +B 1 A 2 +B 2 ] 7 / 22

8 OLS Estimation of One Linear Regressor Model First derivative: 1 n 1 n n (y i β 0 β 1 x 1i ) = 0 i=1 n x 1i (y i β 0 β 1 x 1i ) = 0 i=1 [ n n i=1 x 1i Matrix form n i=1 x 1i n i=1 x2 1i ] [ β0 β 1 ] = X T Xβ = X T y [ n i=1 y ] i n i=1 x 1iy i ˆβ = ( X T X ) 1 X T y 8 / 22

9 OLS Estimation of Multiple Linear Regressor Model SSR(β) = (y Xβ) T (y Xβ) First Derivative X T (y Xβ) = 0 Remark: Method of Moments E (x i u) = 0, i = 1,, k X T Xβ = X T y 9 / 22

10 The Geometry of Linear Regression: Introduction n observations of a linear regression model with k regressors y = Xβ + u, where y and u are n vectors, X is an n k matrix ˆβ = ( X T X ) 1 X T y Numerical properties of these OLS estimates they have nothing to do with how the data were actually generated Euclidean geometry 10 / 22

11 The Geometry of Vector Spaces an n vector was defined as a column vector with n elements, that is, an n 1 matrix Euclidean space in n dimensions, which we will denote as E n Scalar or inner product: For any two vectors x, y E n, their scalar product is x, y = x T y Comutative: x, y = y, x norm of a vector x is x = ( x T x ) 1/2 11 / 22

12 Vector Geometry in Two Dimensions Cartesian coordinates: x = (x 1, x 2 ), y = (y 1, y 2 ) Adding: x + y O A C B x x y x + y y x 2 y 2 y 1 x 1 addition of vectors 12 / 22

13 The Geometry of Scalar Products Multiplying: αx = α x x, y = x y cos θ cos θ = 0, θ = π 2, x, y are said to be orthogonal Cauchy-Schwartz inequality: x, y x y 13 / 22

14 Subspaces of Euclidean Space Defining a subspace of E n is in terms of a set of basis vectors A subspace that is of particular interest to us is the one for which the columns of X provide the basis vectors We may denote the k columns of X as x 1, x 2,, x k Then the subspace associated with these k basis vectors will be denoted by (X) or (x 1, x 2,, x k ) { } k (x 1, x 2,, x k ) z E n z = b i x i, b i R The subspace defined above is called the subspace spanned by the x 1, x 2,, x k or the column space of X The orthogonal complement of (X) is denoted as (X) i=1 (X) { w E n w T z =0, z (X) } If the dimension of (X) is k, then the dimension of (X) is n k 14 / 22

15 The Geometry of OLS Estimation ] X = [x 1 x 2 x k ] Xβ = [x 1 x 2 x k β 1 β 2 β k = k β i x i i=1 OLS estimator ˆβ ( X T y X ˆˆβ ) = 0 15 / 22

16 The Geometry of OLS Estimation Pythagoras Theorem y 2 = X ˆβ 2 + û 2 y T y = ˆβ T X T X ˆβ + û T û T SS = ESS + SSR O y θ û X ˆβ Residuals and fitted values 16 / 22

17 The Geometry of OLS Estimation x 2 O θ y X ˆβ B û A S(x 1, x 2) x 1 a) y projected on two regressors x 2 A ˆβ 2x 2 X ˆβ O ˆβ 1x 1 x 1 O θ y X ˆβ B û A b) The span S(x 1, x 2) of the regressors c) The vertical plane through y Linear regression in three dimensions 17 / 22

18 Orthogonal Projections A projection is a mapping that takes each point of E n into a point in a subspace of E n, while leaving all points in that subspace unchanged An orthogonal projection maps any point into the point of the subspace that is closest to it OLS is an example of orthogonal projection Projection matrix P X = X ( X T X ) 1 X T M X = I X ( X T X ) 1 X T = I P X X ˆβ = P X y M X y = û 18 / 22

19 Orthogonal Projections P X P X = P X, M X M X = M X The pair of projections P X and M X are said to be complementary projections, since the sum of P X y and M X y restores the original vector y y 2 = P X y 2 + M X y 2 P X y y P Z would be the matrix that projects on to (Z), P X,W would be the matrix that projects of (X, W) 19 / 22

20 Linear Transformations of Regressors Nonsingular linear transformation: k k A XA = X [ a 1 a 2 ] [ a k = Xa1 Xa 2 ] Xa k (X) = (XA) Xβ = XAA 1 β Fitted values and residuals are invariant to any nonsingular linear transformation of the columns of X, even though ˆβ will change Special Case: Units of measurement of the regressors 20 / 22

21 The Frisch-Waugh-Lovell Theorem Two Groups of Regressors: y = X 1 β 1 + X 2 β 2 + u, where X 1 is n k 1 matrix, X 2 is n k 2 matrix, X = [ X 1 X 2 ] with k = k1 + k 2 X T 2 X 1 = O : OLS estimator of β 2 in y = X 2 β 2 + u 2 is the same as the OLS estimator β 2 in y = X 1 β 1 + X 2 β 2 + u(second condition in the omitted variable bias) P 1 = P X1 = X 1 ( X T 1 X 1 ) 1 X T 1 P 1 P X = P X P 1 = P 1 M 1 = I P 1 M 1 M X = M X 21 / 22

22 The Frisch-Waugh-Lovell Theorem Consider two regression model y = X 1 β 1 + X 2 β 2 + u M 1 y = M 1 X 2 β 2 + residual Theorem (The Frisch-Waugh-Lovell Theorem) The OLS estimates of β 2 from the two regressions above are numerically identical The residuals from regressions above are numerically identical 22 / 22

### The Geometry of Linear Regression

Chapter 2 The Geometry of Linear Regression 21 Introduction In Chapter 1, we introduced regression models, both linear and nonlinear, and discussed how to estimate linear regression models by using the

### Math Camp II. Basic Linear Algebra. Yiqing Xu. Aug 26, 2014 MIT

Math Camp II Basic Linear Algebra Yiqing Xu MIT Aug 26, 2014 1 Solving Systems of Linear Equations 2 Vectors and Vector Spaces 3 Matrices 4 Least Squares Systems of Linear Equations Definition A linear

### . a m1 a mn. a 1 a 2 a = a n

Biostat 140655, 2008: Matrix Algebra Review 1 Definition: An m n matrix, A m n, is a rectangular array of real numbers with m rows and n columns Element in the i th row and the j th column is denoted by

### Linear Algebra V = T = ( 4 3 ).

Linear Algebra Vectors A column vector is a list of numbers stored vertically The dimension of a column vector is the number of values in the vector W is a -dimensional column vector and V is a 5-dimensional

### Phys 201. Matrices and Determinants

Phys 201 Matrices and Determinants 1 1.1 Matrices 1.2 Operations of matrices 1.3 Types of matrices 1.4 Properties of matrices 1.5 Determinants 1.6 Inverse of a 3 3 matrix 2 1.1 Matrices A 2 3 7 =! " 1

### Lecture 3: Matrix and Matrix Operations

Lecture 3: Matrix and Matrix Operations Representation, row vector, column vector, element of a matrix. Examples of matrix representations Tables and spreadsheets Scalar-Matrix operation: Scaling a matrix

### CS100: DISCRETE STRUCTURES. Lecture 3 Matrices Ch 3 Pages:

CS100: DISCRETE STRUCTURES Lecture 3 Matrices Ch 3 Pages: 246-262 Matrices 2 Introduction DEFINITION 1: A matrix is a rectangular array of numbers. A matrix with m rows and n columns is called an m x n

### Properties of Matrices and Operations on Matrices

Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 1 x 2. x n 8 (4) 3 4 2

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS SYSTEMS OF EQUATIONS AND MATRICES Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Introduction to Matrix Algebra

Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

### Quantum Computing Lecture 2. Review of Linear Algebra

Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces

### Math Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p

Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### STAT 350: Geometry of Least Squares

The Geometry of Least Squares Mathematical Basics Inner / dot product: a and b column vectors a b = a T b = a i b i a b a T b = 0 Matrix Product: A is r s B is s t (AB) rt = s A rs B st Partitioned Matrices

### Review of Linear Algebra

Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=

### Matrix & Linear Algebra

Matrix & Linear Algebra Jamie Monogan University of Georgia For more information: http://monogan.myweb.uga.edu/teaching/mm/ Jamie Monogan (UGA) Matrix & Linear Algebra 1 / 84 Vectors Vectors Vector: A

### In the bivariate regression model, the original parameterization is. Y i = β 1 + β 2 X2 + β 2 X2. + β 2 (X 2i X 2 ) + ε i (2)

RNy, econ460 autumn 04 Lecture note Orthogonalization and re-parameterization 5..3 and 7.. in HN Orthogonalization of variables, for example X i and X means that variables that are correlated are made

### Linear Models Review

Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign

### Matrix Algebra Determinant, Inverse matrix. Matrices. A. Fabretti. Mathematics 2 A.Y. 2015/2016. A. Fabretti Matrices

Matrices A. Fabretti Mathematics 2 A.Y. 2015/2016 Table of contents Matrix Algebra Determinant Inverse Matrix Introduction A matrix is a rectangular array of numbers. The size of a matrix is indicated

### Matrix Algebra. Matrix Algebra. Chapter 8 - S&B

Chapter 8 - S&B Algebraic operations Matrix: The size of a matrix is indicated by the number of its rows and the number of its columns. A matrix with k rows and n columns is called a k n matrix. The number

### Knowledge Discovery and Data Mining 1 (VO) ( )

Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory

### Reference: Davidson and MacKinnon Ch 2. In particular page

RNy, econ460 autumn 03 Lecture note Reference: Davidson and MacKinnon Ch. In particular page 57-8. Projection matrices The matrix M I X(X X) X () is often called the residual maker. That nickname is easy

### Linear Algebra. The analysis of many models in the social sciences reduces to the study of systems of equations.

POLI 7 - Mathematical and Statistical Foundations Prof S Saiegh Fall Lecture Notes - Class 4 October 4, Linear Algebra The analysis of many models in the social sciences reduces to the study of systems

### Chapter 4 - MATRIX ALGEBRA. ... a 2j... a 2n. a i1 a i2... a ij... a in

Chapter 4 - MATRIX ALGEBRA 4.1. Matrix Operations A a 11 a 12... a 1j... a 1n a 21. a 22.... a 2j... a 2n. a i1 a i2... a ij... a in... a m1 a m2... a mj... a mn The entry in the ith row and the jth column

### 2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian

FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian

### Linear Algebra, Vectors and Matrices

Linear Algebra, Vectors and Matrices Prof. Manuela Pedio 20550 Quantitative Methods for Finance August 2018 Outline of the Course Lectures 1 and 2 (3 hours, in class): Linear and non-linear functions on

### Multiplying matrices by diagonal matrices is faster than usual matrix multiplication.

7-6 Multiplying matrices by diagonal matrices is faster than usual matrix multiplication. The following equations generalize to matrices of any size. Multiplying a matrix from the left by a diagonal matrix

### Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Lecture 3 Linear Algebra Background

Lecture 3 Linear Algebra Background Dan Sheldon September 17, 2012 Motivation Preview of next class: y (1) w 0 + w 1 x (1) 1 + w 2 x (1) 2 +... + w d x (1) d y (2) w 0 + w 1 x (2) 1 + w 2 x (2) 2 +...

### Linear Algebra Review

Linear Algebra Review Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Linear Algebra Review 1 / 45 Definition of Matrix Rectangular array of elements arranged in rows and

### POLI270 - Linear Algebra

POLI7 - Linear Algebra Septemer 8th Basics a x + a x +... + a n x n b () is the linear form where a, b are parameters and x n are variables. For a given equation such as x +x you only need a variable and

### Matrix Operations. Linear Combination Vector Algebra Angle Between Vectors Projections and Reflections Equality of matrices, Augmented Matrix

Linear Combination Vector Algebra Angle Between Vectors Projections and Reflections Equality of matrices, Augmented Matrix Matrix Operations Matrix Addition and Matrix Scalar Multiply Matrix Multiply Matrix

### Linear Algebra Review. Vectors

Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors

### Vectors and matrices: matrices (Version 2) This is a very brief summary of my lecture notes.

Vectors and matrices: matrices (Version 2) This is a very brief summary of my lecture notes Matrices and linear equations A matrix is an m-by-n array of numbers A = a 11 a 12 a 13 a 1n a 21 a 22 a 23 a

### Chapter 3 Transformations

Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases

### Elementary Row Operations on Matrices

King Saud University September 17, 018 Table of contents 1 Definition A real matrix is a rectangular array whose entries are real numbers. These numbers are organized on rows and columns. An m n matrix

### Appendix A: Matrices

Appendix A: Matrices A matrix is a rectangular array of numbers Such arrays have rows and columns The numbers of rows and columns are referred to as the dimensions of a matrix A matrix with, say, 5 rows

### ELEMENTS OF MATRIX ALGEBRA

ELEMENTS OF MATRIX ALGEBRA CHUNG-MING KUAN Department of Finance National Taiwan University September 09, 2009 c Chung-Ming Kuan, 1996, 2001, 2009 E-mail: ckuan@ntuedutw; URL: homepagentuedutw/ ckuan CONTENTS

### 1 Multiply Eq. E i by λ 0: (λe i ) (E i ) 2 Multiply Eq. E j by λ and add to Eq. E i : (E i + λe j ) (E i )

Direct Methods for Linear Systems Chapter Direct Methods for Solving Linear Systems Per-Olof Persson persson@berkeleyedu Department of Mathematics University of California, Berkeley Math 18A Numerical

### Lecture 20: 6.1 Inner Products

Lecture 0: 6.1 Inner Products Wei-Ta Chu 011/1/5 Definition An inner product on a real vector space V is a function that associates a real number u, v with each pair of vectors u and v in V in such a way

### Homework 1 Elena Davidson (B) (C) (D) (E) (F) (G) (H) (I)

CS 106 Spring 2004 Homework 1 Elena Davidson 8 April 2004 Problem 1.1 Let B be a 4 4 matrix to which we apply the following operations: 1. double column 1, 2. halve row 3, 3. add row 3 to row 1, 4. interchange

### Matrices and Vectors

Matrices and Vectors James K. Peterson Department of Biological Sciences and Department of Mathematical Sciences Clemson University November 11, 2013 Outline 1 Matrices and Vectors 2 Vector Details 3 Matrix

### Chapter 1: Systems of linear equations and matrices. Section 1.1: Introduction to systems of linear equations

Chapter 1: Systems of linear equations and matrices Section 1.1: Introduction to systems of linear equations Definition: A linear equation in n variables can be expressed in the form a 1 x 1 + a 2 x 2

### Linear Algebra March 16, 2019

Linear Algebra March 16, 2019 2 Contents 0.1 Notation................................ 4 1 Systems of linear equations, and matrices 5 1.1 Systems of linear equations..................... 5 1.2 Augmented

### ELE/MCE 503 Linear Algebra Facts Fall 2018

ELE/MCE 503 Linear Algebra Facts Fall 2018 Fact N.1 A set of vectors is linearly independent if and only if none of the vectors in the set can be written as a linear combination of the others. Fact N.2

### Mathematics. EC / EE / IN / ME / CE. for

Mathematics for EC / EE / IN / ME / CE By www.thegateacademy.com Syllabus Syllabus for Mathematics Linear Algebra: Matrix Algebra, Systems of Linear Equations, Eigenvalues and Eigenvectors. Probability

### Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J Olver 3 Review of Matrix Algebra Vectors and matrices are essential for modern analysis of systems of equations algebrai, differential, functional, etc In this

### Matrices Gaussian elimination Determinants. Graphics 2009/2010, period 1. Lecture 4: matrices

Graphics 2009/2010, period 1 Lecture 4 Matrices m n matrices Matrices Definitions Diagonal, Identity, and zero matrices Addition Multiplication Transpose and inverse The system of m linear equations in

### Lecture 23: 6.1 Inner Products

Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such

### Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε,

2. REVIEW OF LINEAR ALGEBRA 1 Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, where Y n 1 response vector and X n p is the model matrix (or design matrix ) with one row for

### MAT Linear Algebra Collection of sample exams

MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system

### Index. book 2009/5/27 page 121. (Page numbers set in bold type indicate the definition of an entry.)

page 121 Index (Page numbers set in bold type indicate the definition of an entry.) A absolute error...26 componentwise...31 in subtraction...27 normwise...31 angle in least squares problem...98,99 approximation

### MATH 240 Spring, Chapter 1: Linear Equations and Matrices

MATH 240 Spring, 2006 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 8th Ed. Sections 1.1 1.6, 2.1 2.2, 3.2 3.8, 4.3 4.5, 5.1 5.3, 5.5, 6.1 6.5, 7.1 7.2, 7.4 DEFINITIONS Chapter 1: Linear

### EE731 Lecture Notes: Matrix Computations for Signal Processing

EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten

### Section 12.4 Algebra of Matrices

244 Section 2.4 Algebra of Matrices Before we can discuss Matrix Algebra, we need to have a clear idea what it means to say that two matrices are equal. Let's start a definition. Equal Matrices Two matrices

### Elementary maths for GMT

Elementary maths for GMT Linear Algebra Part 2: Matrices, Elimination and Determinant m n matrices The system of m linear equations in n variables x 1, x 2,, x n a 11 x 1 + a 12 x 2 + + a 1n x n = b 1

### Matrices. Chapter Definitions and Notations

Chapter 3 Matrices 3. Definitions and Notations Matrices are yet another mathematical object. Learning about matrices means learning what they are, how they are represented, the types of operations which

### Linear Algebra and Matrix Inversion

Jim Lambers MAT 46/56 Spring Semester 29- Lecture 2 Notes These notes correspond to Section 63 in the text Linear Algebra and Matrix Inversion Vector Spaces and Linear Transformations Matrices are much

### n n matrices The system of m linear equations in n variables x 1, x 2,..., x n can be written as a matrix equation by Ax = b, or in full

n n matrices Matrices Definitions Diagonal, Identity, and zero matrices Addition Multiplication Transpose and inverse The system of m linear equations in n variables x 1, x 2,..., x n a 11 x 1 + a 12 x

### 10. Linear Systems of ODEs, Matrix multiplication, superposition principle (parts of sections )

c Dr. Igor Zelenko, Fall 2017 1 10. Linear Systems of ODEs, Matrix multiplication, superposition principle (parts of sections 7.2-7.4) 1. When each of the functions F 1, F 2,..., F n in right-hand side

### 2. Review of Linear Algebra

2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear

### INNER PRODUCT SPACE. Definition 1

INNER PRODUCT SPACE Definition 1 Suppose u, v and w are all vectors in vector space V and c is any scalar. An inner product space on the vectors space V is a function that associates with each pair of

### LINEAR ALGEBRA REVIEW

LINEAR ALGEBRA REVIEW JC Stuff you should know for the exam. 1. Basics on vector spaces (1) F n is the set of all n-tuples (a 1,... a n ) with a i F. It forms a VS with the operations of + and scalar multiplication

### Matrix operations Linear Algebra with Computer Science Application

Linear Algebra with Computer Science Application February 14, 2018 1 Matrix operations 11 Matrix operations If A is an m n matrix that is, a matrix with m rows and n columns then the scalar entry in the

### Lecture II: Linear Algebra Revisited

Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function

### Matrix Basic Concepts

Matrix Basic Concepts Topics: What is a matrix? Matrix terminology Elements or entries Diagonal entries Address/location of entries Rows and columns Size of a matrix A column matrix; vectors Special types

### MATH 304 Linear Algebra Lecture 18: Orthogonal projection (continued). Least squares problems. Normed vector spaces.

MATH 304 Linear Algebra Lecture 18: Orthogonal projection (continued). Least squares problems. Normed vector spaces. Orthogonality Definition 1. Vectors x,y R n are said to be orthogonal (denoted x y)

### ORTHOGONALITY AND LEAST-SQUARES [CHAP. 6]

ORTHOGONALITY AND LEAST-SQUARES [CHAP. 6] Inner products and Norms Inner product or dot product of 2 vectors u and v in R n : u.v = u 1 v 1 + u 2 v 2 + + u n v n Calculate u.v when u = 1 2 2 0 v = 1 0

### Matrices and Determinants

Chapter1 Matrices and Determinants 11 INTRODUCTION Matrix means an arrangement or array Matrices (plural of matrix) were introduced by Cayley in 1860 A matrix A is rectangular array of m n numbers (or

### Lecture 7: Vectors and Matrices II Introduction to Matrices (See Sections, 3.3, 3.6, 3.7 and 3.9 in Boas)

Lecture 7: Vectors and Matrices II Introduction to Matrices (See Sections 3.3 3.6 3.7 and 3.9 in Boas) Here we will continue our discussion of vectors and their transformations. In Lecture 6 we gained

### ECS130 Scientific Computing. Lecture 1: Introduction. Monday, January 7, 10:00 10:50 am

ECS130 Scientific Computing Lecture 1: Introduction Monday, January 7, 10:00 10:50 am About Course: ECS130 Scientific Computing Professor: Zhaojun Bai Webpage: http://web.cs.ucdavis.edu/~bai/ecs130/ Today

### MATRIX ALGEBRA. or x = (x 1,..., x n ) R n. y 1 y 2. x 2. x m. y m. y = cos θ 1 = x 1 L x. sin θ 1 = x 2. cos θ 2 = y 1 L y.

as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations

### A matrix over a field F is a rectangular array of elements from F. The symbol

Chapter MATRICES Matrix arithmetic A matrix over a field F is a rectangular array of elements from F The symbol M m n (F ) denotes the collection of all m n matrices over F Matrices will usually be denoted

### 4 Linear Algebra Review

4 Linear Algebra Review For this topic we quickly review many key aspects of linear algebra that will be necessary for the remainder of the course 41 Vectors and Matrices For the context of data analysis,

### Raphael Mrode. Training in quantitative genetics and genomics 30 May 10 June 2016 ILRI, Nairobi. Partner Logo. Partner Logo

Basic matrix algebra Raphael Mrode Training in quantitative genetics and genomics 3 May June 26 ILRI, Nairobi Partner Logo Partner Logo Matrix definition A matrix is a rectangular array of numbers set

### MATH2210 Notebook 2 Spring 2018

MATH2210 Notebook 2 Spring 2018 prepared by Professor Jenny Baglivo c Copyright 2009 2018 by Jenny A. Baglivo. All Rights Reserved. 2 MATH2210 Notebook 2 3 2.1 Matrices and Their Operations................................

### Introduc)on to linear algebra

Introduc)on to linear algebra Vector A vector, v, of dimension n is an n 1 rectangular array of elements v 1 v v = 2 " v n % vectors will be column vectors. They may also be row vectors, when transposed

### orthogonal relations between vectors and subspaces Then we study some applications in vector spaces and linear systems, including Orthonormal Basis,

5 Orthogonality Goals: We use scalar products to find the length of a vector, the angle between 2 vectors, projections, orthogonal relations between vectors and subspaces Then we study some applications

### Glossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB

Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the

### Chapter 5. Linear Algebra. A linear (algebraic) equation in. unknowns, x 1, x 2,..., x n, is. an equation of the form

Chapter 5. Linear Algebra A linear (algebraic) equation in n unknowns, x 1, x 2,..., x n, is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b where a 1, a 2,..., a n and b are real numbers. 1

### Systems of Linear Equations and Matrices

Chapter 1 Systems of Linear Equations and Matrices System of linear algebraic equations and their solution constitute one of the major topics studied in the course known as linear algebra. In the first

### Lecture Notes in Linear Algebra

Lecture Notes in Linear Algebra Dr. Abdullah Al-Azemi Mathematics Department Kuwait University February 4, 2017 Contents 1 Linear Equations and Matrices 1 1.2 Matrices............................................

### Introduction. Vectors and Matrices. Vectors [1] Vectors [2]

Introduction Vectors and Matrices Dr. TGI Fernando 1 2 Data is frequently arranged in arrays, that is, sets whose elements are indexed by one or more subscripts. Vector - one dimensional array Matrix -

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and

Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Matrix Operations: Determinant

Matrix Operations: Determinant Determinants Determinants are only applicable for square matrices. Determinant of the square matrix A is denoted as: det(a) or A Recall that the absolute value of the determinant

### Review Packet 1 B 11 B 12 B 13 B = B 21 B 22 B 23 B 31 B 32 B 33 B 41 B 42 B 43

Review Packet. For each of the following, write the vector or matrix that is specified: a. e 3 R 4 b. D = diag{, 3, } c. e R 3 d. I. For each of the following matrices and vectors, give their dimension.

### (, ) : R n R n R. 1. It is bilinear, meaning it s linear in each argument: that is

17 Inner products Up until now, we have only examined the properties of vectors and matrices in R n. But normally, when we think of R n, we re really thinking of n-dimensional Euclidean space - that is,

### ELEMENTARY LINEAR ALGEBRA

ELEMENTARY LINEAR ALGEBRA K R MATTHEWS DEPARTMENT OF MATHEMATICS UNIVERSITY OF QUEENSLAND First Printing, 99 Chapter LINEAR EQUATIONS Introduction to linear equations A linear equation in n unknowns x,

### Section I. Define or explain the following terms (3 points each) 1. centered vs. uncentered 2 R - 2. Frisch theorem -

First Exam: Economics 388, Econometrics Spring 006 in R. Butler s class YOUR NAME: Section I (30 points) Questions 1-10 (3 points each) Section II (40 points) Questions 11-15 (10 points each) Section III

### Introduction to Matrices

POLS 704 Introduction to Matrices Introduction to Matrices. The Cast of Characters A matrix is a rectangular array (i.e., a table) of numbers. For example, 2 3 X 4 5 6 (4 3) 7 8 9 0 0 0 Thismatrix,with4rowsand3columns,isoforder

### POL 213: Research Methods

Brad 1 1 Department of Political Science University of California, Davis April 15, 2008 Some Matrix Basics What is a matrix? A rectangular array of elements arranged in rows and columns. 55 900 0 67 1112

### Math 360 Linear Algebra Fall Class Notes. a a a a a a. a a a

Math 360 Linear Algebra Fall 2008 9-10-08 Class Notes Matrices As we have already seen, a matrix is a rectangular array of numbers. If a matrix A has m columns and n rows, we say that its dimensions are

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and

Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### Introduction to Quantitative Techniques for MSc Programmes SCHOOL OF ECONOMICS, MATHEMATICS AND STATISTICS MALET STREET LONDON WC1E 7HX

Introduction to Quantitative Techniques for MSc Programmes SCHOOL OF ECONOMICS, MATHEMATICS AND STATISTICS MALET STREET LONDON WC1E 7HX September 2007 MSc Sep Intro QT 1 Who are these course for? The September

### Numerical Linear Algebra

Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix

### MATH 106 LINEAR ALGEBRA LECTURE NOTES

MATH 6 LINEAR ALGEBRA LECTURE NOTES FALL - These Lecture Notes are not in a final form being still subject of improvement Contents Systems of linear equations and matrices 5 Introduction to systems of