# Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

Save this PDF as:

Size: px
Start display at page:

Download "Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =" ## Transcription

1 30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a a 1N a 21 a a 2N a M1 a M2... a MN A matrix can also be written as A = (a nm ) where n = 1, 2,..., N and m = 1, 2,..., M. Matrices are usually written as boldface, upper-case letters. Definition of Matrix Transpose. The transpose of A, denoted by A t, is the NxM matrix (a mn ) or A t = a 11 a a 1M a 21 a a 2M a N1 a N2... a NM Example. A example of a matrix A and it s transpose A t is: 1 2 [ ] A = 3 4 and A t = Definition of Vectors.. A column vector is a Bx1 matrix and a row vector is a 1xB matrix, where B 1. Column and row vectors are usually simply referred to as vectors and they are assumed to be column vectors unless they are explicitly identified as row vector or if it is clear from the context. Vectors are denoted by boldface, lower-case letters. x = [x 1, x 2,..., x B ] t = Definition of Dimension. The integer B is called the dimension of the vector x in the definition above. The phrase x is B dimensional is equivalent to stating that the dimension of x is B. Definition of 0 and 1 vectors.. The vectors x 1 x 2. x B 0 = (0, 0,..., 0) t and 1 = (1, 1,..., 1) t are called the Origin or Zero Vector and the One Vector, respectively. Definition of Vector Addition and Scalar Multiplication.. The addition of two vectors x 1, x 2 R B is defined as: x 3 = [x 11 + x 21, x 12 + x 22,..., x 1B + x 2B ] t. The word

2 LINEAR ALGEBRA 31 scalars is another word for numbers. Scalar multiplication is the product of a number α and a vector x, defined by αx = [αx 1, αx 2,..., αx B ] t. Example. If x 1 = [1, 2] and x 2 = [3, 4] then x 3 = x 1 + x 2 = [1 + 3, 2 + 4] = [4, 6]. If α = 3, then αx 1 = [3, 6]. Definition of Matrix Multiplication. If A and B are MxN and NxP matrices, then the matrix product, C = AB, is the MxP matrix defined by c mp = N a mn b np n=1 If A 3 2 and B 2 2 are the following matrices: 1 2 [ ] A = 3 4 and B = then C = AB is the 3 2 matrix: C = For example, c 2,1 = (3)(11) + (4)(13) = = 85. Definition of Dot, or Inner, Product. If x and y are B-dimensional vectors, then the matrix product x t y is referred to as the Dot or Inner Product of x and y. Example. If x = [ 2, 0, 2] t and y = [4, 1, 2] t, then x t y = (-2)(4) + (0)(1) + (2)(2) = -4. Note that a linear system of equations can be represented as a matrix multiplication. For example, the system and be written in matrix-vector form as a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 Ax = b. Definition of Diagonal Matrix. A diagonal matrix D = (d mn ) is an NxN matrix with the property that d mn = 0 if m n. Definition of Identity Matrix. The NxN Identity Matrix, denoted by I N or just I if N is known, is the NxN diagonal matrix with d nn = 1 for n = 1, 2,..., N. Notice that if A is any NxN matrix, then AI N = I N A = A. Definition of Inverse Matrix. Let A be an NxN square matrix. If there is a matrix B with the property that AB = BA = I N, then B is called the inverse of A or A inverse.

3 32 MATHEMATICS REVIEW G It is denoted by A 1. The inverse of a square matrix A does not always exist. If it does exist, then A is invertible. A.1.2 Vector Spaces Definition of Vector, or Linear, Space. Let B denote a positive integer. A finite-dimensional Vector, or Linear, Space with dimension B is a collection of B dimensional vectors, V that satisfies commutative, associative, and distributive laws and with the properties that: 1. For every x V and y V, the sum (x + y) V. 2. For every real number s and x V, the product sx V. 3. For every x V, the additive inverse x V, which implies that 0 V. Notation. A B-dimensional vector space is often denoted by R B. Definition of Subspace. A Subspace is a subset of a vector space that is also a vector space. Notice that subspaces of vector spaces always include the origin. Although it is not common, we us the notation S V to state that S is a subspace of a vector space V. Figure A.1 Subspace and non-subspace Subspaces are often used in algorithms for analyzing image spectrometer data. The motivation is that spectra with certain characteristics might all be contained in a subspace. Definition of Linear Combination. If y, {x d } D d=1 RB and y = a 1 x 1 + a 2 x a D x D then y is a linear combination of {x 1, x 2,..., x D }. The numbers a 1, a 2,..., a D are called the coefficients of the linear combination. Notice that the linear combination can be written in matrix-vector form as: y = Xa

4 LINEAR ALGEBRA 33 where X is the BxD matrix with x d as the d th column and a is the vector of coefficients. Definition of Linear Transformation.. Let V and W denote two vector spaces. A function f : V W is called a Linear Transformation if a, b R and x, y V, the statement f (ax + by) = af (x) + bf (y) is true. If A is an m n matrix and x R m, then f (x) = Ax is a linear transformation. Definition of Linear Independence. If B={x 1, x 2,..., x D } is a set of vectors with the property that a 1 x 1 + a 2 x a D x D = 0 = a 1 = a 2 =... = a D = 0 then the vectors in the set B are called Linearly Independent. Informally, no vector in a set of linearly independent vectors can be written as a linear combination of the other vectors in the set. Fact.There can be no more than B linearly independent vectors in a B dimensional vector space. If B = {x 1, x 2,..., x B } is a set of B linearly independent vectors in a B dimensional space V, then every x V can be written uniquely as a linear combination of the vectors in B. Definition of Basis. A Basis of a subspace of B-dimensional space, S, is a set of vectors {v 1, v 2,..., v D } with the property that every vector x S can be written one and exactly one way as a linear combination of the elements of the basis. In other words, if x S then there is a unique set of coefficients, a 1, a 2,..., a D such that y = a 1 v 1 + a 2 + v 2, + a D v D. It must be the case that D B. Fact.There are infinitely many bases for any subspace but they all have the same number of elements. The number of elements is called the dimension of the subspace. Definition of Standard Basis. The standard basis of a D-dimensional subspace is the set of vectors {s 1, s 2,..., s D } R D where the d th element of the d th vector s d, s dd, is equal to one and the rest of the elements are equal to zero. That is s dj = 1 if d = j and s dj = 0 if d j. Example.. The standard basis for R 3 is s 1 = 0 and s 2 = 0 1 and s 3 = Any 3-Dimensional vector can be represented as a linear combination of the standard basis vectors: [ x1 ] x 2 = x 1 s 1 + x 2 s 2 + x 3 s 3 = x x x 3 0 x Fact.If the columns of an NxN matrix A form a basis for R N, then A is invertible. Bases can be though of as coordinate systems. It can be useful to represent vectors in terms of different bases, or coordinate systems. Therefore, the method for changing the

5 34 MATHEMATICS REVIEW G coordinates used to represent a vector from one basis to another is first described. Then, a method for determining a linear transformation for mapping one basis to another basis is described. The two methods are referred to as Change of Coordinates or Change of Basis. Definition of Change of Basis. The process of changing the representation of a vector x from a representation in terms of one basis to a representation in terms of a different basis is called a Change of Basis transformation. Calculating a change of basis transformation may require solving a linear system. To see this, let U = {u 1, u 2,..., u B } and V = {v 1, v 2,..., v B } be two bases for R B. Suppose the representation of a vector x in terms of the basis U is known to be x = a 1 u 1 + a 2 u a B u B = Ua and that the representation of x in terms of V is unknown x = c 1 v 1 + c 2 v c B v B = Vc. In other words, a = (a 1, a 2,..., a B ) is known and c = (c 1, c 2,..., c B ) is unknown. Then c = V 1 Ua. In particular, if U is the standard basis, then U = I and x = [a 1, a 2,..., a B ] t, so c = V 1 x. The matrix W = V 1 U is called the change of basis matrix. Definition of Pseudo-inverse. The definition of a pseudo-inverse is motivated by linear systems of equations, Ax = b. If A is not square, then A 1 is not defined so one cannot write x = A 1 b. However, one can write (A t A) x = A t b. Note that this is not equivalent to the original linear system of equations. The matrix, A t A is N N so, assuming that N < M, it is almost certainly invertible. Therefore, x = (A t A) 1 A t b is a solution of (A t A) x = A t b. The matrix (A t A) 1 A t b is called the pseudo-inverse of A and is denoted by A +.

6 LINEAR ALGEBRA 35 A.1.3 Norms, Metrics, and Dissimilarities Definition of norm. A norm produces a quantity computed from vectors that somehow measures the size of the vector. More formally, a norm is a function, N : R B R + with the properties that x, y R B and a R: 1. N (x) 0 and N (x) = 0 if and only if x = N (ax) = a N (x). 3. N (x + y) N (x) + N (y) Notation. The norm is usually written with the vertical bar notation: N (x) = x. Definition of L p norm, or just p-norm.. Assume p 1. The p-norm is defined by ( B ) 1 x p = b=1 x b p p. Definition of Euclidean Norm.. The Euclidean Norm is the p-norm with p = 2, that is, the L 2 norm. If p is not specified, then it is assumed that x denotes the Euclidean norm. Note that the Euclidean norm can be written as x = x t x. Definition of L norm, or just -norm.. The norm is x = max B b=1 x b. Definition of Unit Norm Vector. A vector x is said to have unit norm if x = 1. Note that if x is any vector then has unit norm. x x Example. The set of all vectors x R 2 with unit norm are equivalent to a circle, called the unit circle, since the set of all points with x x 2 2 = 1 is a circle of radius 1. Definition of Generalized Unit Circle.. Let N be a norm. The set U = {x N (x) = 1} is called a Generalized Unit Circle. Often, for simplicity, the set Uis called a Unit Circle even though it is only a geometric circle of the Euclidean norm. It is sometimes useful to visualize the unit circle for other norms. Some examples are shown in A.2. Fact.If x and y are vectors, then x t y = x y cos θ, where θ is the angle between x and y. If x and y are normalized, then the inner product of the vectors is equal to the cosine of the angle between the vectors. x t ( ) t ( ) y x y x y = = cos θ. x y Definition of Distance Function or Metric. A distance or metric is a function defined on pairs of vectors, d : R B xr B R + and has the following properties: 1. d (x, y) 0 and d (x, y) = 0 if and only if x = y 2. d (x, y) = d (y, x) 3. For any z, d (x, y) d (x, z) + d (z, y)

7 36 MATHEMATICS REVIEW G Figure A.2 Some Unit Circles. Note that as p increases, the L p norm approaches the L norm. Definition of L p distance. d p (x, y) = x y p. Note that the Euclidean distance squared can be written as d 2 (x, y) 2 = x y 2 = (x y) t (x y) = x 2 + y 2 2x t y. Therefore, if x and y are unit vectors, then measuring Euclidean distance between vectors is equivalent to measuring the cosine of the angle between the two vectors: d 2 (x, y) 2 = x y 2 = (x y) t (x y) = x t y = 2 2 cos θ which implies that, cos θ = x y 2. Definition of Orthogonality and Orthonormal Bases. Two non-zero vectors x and y are called orthogonal if x t y = 0. Note that orthogonal is another word for perpendicular since (for non-zero vectors) x t y = x y cos θ = 0 only happens if cos θ = 0. An orthonormal basis is a basis U = {u 1, u 2,..., u B } with the property that u i u j = δ ij where δ ij is the Kronecker delta function.

8 LINEAR ALGEBRA 37 Example. Computing Coefficients of Orthonormal Bases. Suppose that U = {u 1, u 2,..., u B } is an orthonormal basis of R B, and x R B. Then there exists coefficients c 1, c 2,..., c B such that x = c 1 u 1 + c 2 u c B u B. In general, one must solve a linear system of equations to find the coefficients. However, with an orthonormal basis, it is much easier. For example, to find c 1, transpose x x t = (c 1 u 1 + c 2 u c B u B ) t = c 1 u t 1 + c 2 u t c B u t B and then multiply on the right by u 1 x t u 1 = c 1 u t 1u 1 + c 2 u t 2u c B u t Bu 1 = c 1. In fact, for every b = 1, 2,..., B, c b = x t u b. Definition of Projections. The projection of a vector x onto a vector y is given by P roj (x, y) = x t y y. Figure A.3 Projection of a vector x onto a vector y

9 38 MATHEMATICS REVIEW G A.1.4 Eigenanalysis and Symmetric Matrices Eigenvalues, Eigenvectors, and Symmetric Matrices play a fundamental role in describing properties of spectral data obtained from imaging spectrometers. The definitions given here are not the most general but are sufficient for understanding and devising algorithms for imaging spectroscopy. Definition of Eigenvalues and Eigenvectors. Let A be an n n matrix. A number λ R and corresponding vector v λ R n are called an eigenvalue and eigenvector, respectively, of A if Av λ = λv λ. It is common to write v in place of v λ if there is no ambiguity. Definition of Symmetric Matrix. A square matrix A is called symmetric if A = A t. Definition of Positive Definite and Semi-definite Matrices. A square matrix A is called positive definite if, x : x t Ax > 0. A is positive semi-definite if x : x t Ax 0. In these cases, if λ and v are an eigenvalue and corresponding eigenvector pair and v 0, then v t Av = v t λv = λ x 2 > 0. Since x 2 > 0, it must be true that λ > 0. Thus, eigenvalues corresponding to nonzero eigenvectors of a positive definite matrix are positive. The same argument can be used for positive semi-definite matrices (in which case the eigenvalues are non-negative). A Orthonormal Basis Theorem If A is an B B positive-definite, symmetric matrix, then the eigenvectors of A form an orthonormal basis for R B. This theorem is presented without proof; the interested reader can find it in many linear algebra texts. Suppose V = {v 1, v 2,..., v B } is an orthonormal basis of eigenvectors of A. Let V = [v 1 v 2... v B ] be the matrix with columns equal to the elements of the basis V. Then AV = [λ 1 v 1 λ 1 v 2... λ B v B ] = V Λ where Λ = diag (λ 1, λ 2,..., λ B ) is the diagonal B B matrix with the eigenvalues along the diagonal and zeros elsewhere. Thus A = V ΛV t. This relationship is called diagonalization. Often V is taken to be the matrix with rows equal to the transposes of the elements of V. In this case, diagonalization is written as A = V t ΛV. Thus, a consequence of the Orthonormal Basis Theorem is that any positive-definite, symmetric matrix A can be diagonalized by the matrix V whose columns are the orthonormal basis of eigenvectors of A and a diagonal matrix Λ with diagonal elements equal to the eigenvalues of A. Example. Suppose A is the matrix The reader can verify that the eigenvalues of A are λ 1 = , λ 2 = , λ 3 = and the corresponding eigenvectors are the columns of the matrix V given by

10 LINEAR ALGEBRA 39 A Singular Value Decomposition The Singular Value Decomposition (SVD) extends the concept of diagonalization to some non-square and non-invertible square matrices. There are very stable numerical algorithms for computing the SVD and is therefore often at the core of computational algorithms for tasks such as linear system solvers and computing inverses and pseudo-inverses of matrices. Definition of Singular Value Decomposition (SVD). Let A be an M N with M > N. The SVD of A is given by A = V ΣU t where V is an M M orthogonal matrix whose columns are eigenvectors of AA t, U is an M M orthogonal matrix of eigenvectors of A t A, and Σ is an M N matrix that is as diagonal as possible, that is, Σ is of the form: Σ = σ σ σ N σ N. (A.1) The first N rows and N columns of Σ consist of a diagonal matrix and the remaining M N rows are all zeros. The pseudo-inverse of an M N with M > N matrix provides motivation for the definition, although the derivation is omitted here. Recall that the pseudo-inverse is given by A + = (A t A) 1 A t. It can be shown that A + = UΣ + V t where: 1 σ σ Σ +. =... (A.2) σ N σ N Note that Σ is M N and Σ + is N M. Therefore, ΣΣ + is M M and Σ + Σ is N N. Furthermore, Σ + Σ =

11 40 MATHEMATICS REVIEW G σ σ σ σ σ N σ N = If M < N and rank (A) = M, then σ σ Σ =.... (A.3) σ M σ M Calculating (A t A) 1 is often numerically unstable. The SVD is numerically stable. Example. Suppose Then the SVD of A is given by: 1 2 A = [ ] V = Σ = U = One can verify that A + := (A t A) 1 A t = UΣ + V t. A Simultaneous Diagonalization Let C and C n denote symmetric, positive semidefinite matrices. Then, there exists a matrix, W, that simultaneously diagonalizes C and Cn. The matrix W is constructed here. First, note that there exist matrices U and Λ such that Λ is of the form: C = U t ΛU.

12 LINEAR ALGEBRA 41 Λ = λ λ λ B λ B where λ b 0. Hence, Λ = Λ 1 2 Λ 1 2. Therefore Let Λ 1 2 UCU t Λ 1 2 = I. C nt = Λ 1 2 UCn U t Λ 1 2. Then C t n t = C nt and C nt is positive semi-definite. Hence, there exists V such that V t V = V V t = I and C nt = V t D nt V or V C nt V t = D nt where D nt is a diagonal matrix with non-negative entries. Let W = V Λ 1 2 U. Then and W CW t = V Λ 1 2 UCU t Λ 1 2 V t = V IV t = V V t = I W C n W t = V Λ 1 2 UCn U t Λ 1 2 V t = V C nt V t = D nt. Therefore, W simultaneously diagonalizes C and C n.

13 42 MATHEMATICS REVIEW G A.1.5 Determinants, Ranks, and Numerical Issues In this section, some quantities related to solutions of linear systems are defined and used to get a glimpse at some numerical issues that occur in Intelligent Systems all too frequently. A more thorough study requires a course in Numerical Linear Algebra. It is often assumed that solutions are estimated numerically and so there is error due to finite precision arithmetic. The determinant is a number associated with a square matrix. The precise definition is recursive and requires quite a few subscripts so it is first motivated by the definition using a 2 2 matrix A. The definition is then given and some important properties are discussed. Consider the problem of solving a linear system of two equations and two unknowns, Ax = b, using standard row reduction: [ a11 a ] [ ] 12 x1 = a 21 a 22 x 2 [ b1 b 2 ] a 21 0 a21 a 11 a 12 a 11a 22 a 21a 12 a 11 ] [ = x 2 [ x1 a21b1 a 11 a21b1+a11b2 a 11 ] so, after a little more algebra, the solutions for x 1 and x 2 are found to be x 2 = a 11b 2 a 21 b 1 a 11 a 22 a 21 a 12 and x 1 = a 22b 1 a 12 b 2 a 22 a 11 a 12 a 21 The denominator of the solutions are called the determinant of A which is often denoted by det (A) or by A. Notice that the numerators of the solutions for x 1 and x 2 are the determinants of the matrices [ ] [ ] b1 a 12 a11 b 1 A 1 = and A 2 = b 2 a 22 a 21 b 2 Definition of Rank of a matrix. Let A denote an M N matrix. It is a fact that the number of linearly independent rows of A is equal to the number of linearly independent columns of A. The Rank of A, often denoted by rank (A), is the number of linearly independent rows (or columns). Note that, if M > N, then rank (A) N. It is almost always true that rank (A) = N but one must take care to make sure it is true. Similarly, if A is a square matrix of size N N, then it is almost always true that rank (A) = N. In the latter case, A 1 exists and the matrix A is referred to as Invertible or Nonsingular. If rank (A) < N, then A is non-invertible, also referred to as Singular. Definition of Null Space. Assume x 1, x 2 0. Note that, if Ax 1 =Ax 2 = 0 then α, β R, A (αx 1 + βx 2 ) = αa (x 1 ) + βa (x 2 )= 0. Therefore, the set N A with the property that n N A An = 0 is a subspace of R N called the Null Space of A. Of course, 0 N A. In fact, 0 may be the only vector in N A, particularly if M > N. However, if M < N, which is often the case for deep learning networks, then it is certain that there are nonzero vectors x N A R N. Example. Null Spaces and Singular Value Decomposition. If M < N and rank(a) = M, then the SVD of A is of the form A = VΣU t where Σ is defined in Eq. A.3. Since V and U are square, nonsingular matrices, N V = N W = {0}. Therefore,

14 LINEAR ALGEBRA 43 N A = { x n = U t x and n = (0, 0,..., 0, n M+1, n M+2,..., n N ) t}. since some elements of the first M columns of Σ are nonzero and all the elements of the last N M + 1 columns of Σ are zeros. If M > N and rank(a) = N, then Σ is defined in Eq. A.3. Since there is a nonzero element in each column of Σ, N A = {0}. A Condition Number Consider the following linear system: [ 2 1 ] [ ] [ ] x1 1 Ax = b : = x 2 1 Problems with this system are depicted in Fig. A.4 (A.4) Figure A.4 A linear system associated with two lines (Red and Blue) that are almost parallel. The solution of the system is x 1 = 0 and x 2 = 1. Plot A depicts the lines. Plot B depicts the Mean Squared Error(MSE) of the system when x 1 ranges from -0.1 to 0.2 and x 2 is held constant at x 2=1. So, for example, if x 1 < 0.03 then the MSE < For some applications, 0.03 can be a very significant error. Of course, We can make the MSE much smaller for a wider range of values of x 1 by letting a 21 get close and close to 2, e.g. 2.01, 2.001, , etc. Error can be looked at relatively in addition to the MSE. For any square, nonsingular matrix A and linear system Ax = b let x t denote a solution of the system and x f an estimated solution, which will have some error. Thus Ax t = b but Ax f = b f and b b f. A couple of definitions are required. These will lead to the notion of the condition number of a matrix, which is a very useful quantity. Definition of Residual and Relative Residual for Linear Systems. The Residual is r = b b f = b Ax f and the Relative Residual is r b. Definition of Error and Relative Error for Linear Systems. The Error is e = x t x f and the Relative Error is e x t. Definition of Condition Number of a Matrix. The Condition Number of a square matrix is cond (A) = A A 1. If A is not square, then bfa 1 is replaced by the pseudo-inverse A +. It is a fact that cond (A) 1.

### 3 a 21 a a 2N. 3 a 21 a a 2M APPENDIX: MATHEMATICS REVIEW G 12.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers 2 A = 6 4 a 11 a 12... a 1N a 21 a 22... a 2N. 7..... 5 a M1 a M2...

### Glossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the

### Image Registration Lecture 2: Vectors and Matrices Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this

### Review of Linear Algebra Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=

### Linear Algebra Review. Vectors Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors

### Linear Algebra - Part II Linear Algebra - Part II Projection, Eigendecomposition, SVD (Adapted from Sargur Srihari s slides) Brief Review from Part 1 Symmetric Matrix: A = A T Orthogonal Matrix: A T A = AA T = I and A 1 = A T

### Foundations of Matrix Analysis 1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the

### Jim Lambers MAT 610 Summer Session Lecture 1 Notes Jim Lambers MAT 60 Summer Session 2009-0 Lecture Notes Introduction This course is about numerical linear algebra, which is the study of the approximate solution of fundamental problems from linear algebra

### Review of Some Concepts from Linear Algebra: Part 2 Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set

### Large Scale Data Analysis Using Deep Learning Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset

### Elementary maths for GMT Elementary maths for GMT Linear Algebra Part 2: Matrices, Elimination and Determinant m n matrices The system of m linear equations in n variables x 1, x 2,, x n a 11 x 1 + a 12 x 2 + + a 1n x n = b 1

### 14 Singular Value Decomposition 14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing

### Linear Algebra for Machine Learning. Sargur N. Srihari Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### Knowledge Discovery and Data Mining 1 (VO) ( ) Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory

### Properties of Matrices and Operations on Matrices Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

### 10-701/ Recitation : Linear Algebra Review (based on notes written by Jing Xiang) 10-701/15-781 Recitation : Linear Algebra Review (based on notes written by Jing Xiang) Manojit Nandi February 1, 2014 Outline Linear Algebra General Properties Matrix Operations Inner Products and Orthogonal

### Matrices and Linear Algebra Contents Quantitative methods for Economics and Business University of Ferrara Academic year 2017-2018 Contents 1 Basics 2 3 4 5 Contents 1 Basics 2 3 4 5 Contents 1 Basics 2 3 4 5 Contents 1 Basics 2

### CS 143 Linear Algebra Review CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see

### This appendix provides a very basic introduction to linear algebra concepts. APPENDIX Basic Linear Algebra Concepts This appendix provides a very basic introduction to linear algebra concepts. Some of these concepts are intentionally presented here in a somewhat simplified (not

### Mathematical foundations - linear algebra Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar

### MAT 610: Numerical Linear Algebra. James V. Lambers MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic

### Math Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra. DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Basic Concepts in Linear Algebra Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear

### 3 (Maths) Linear Algebra 3 (Maths) Linear Algebra References: Simon and Blume, chapters 6 to 11, 16 and 23; Pemberton and Rau, chapters 11 to 13 and 25; Sundaram, sections 1.3 and 1.5. The methods and concepts of linear algebra

### Deep Learning Book Notes Chapter 2: Linear Algebra Deep Learning Book Notes Chapter 2: Linear Algebra Compiled By: Abhinaba Bala, Dakshit Agrawal, Mohit Jain Section 2.1: Scalars, Vectors, Matrices and Tensors Scalar Single Number Lowercase names in italic

### Chapter 5. Linear Algebra. A linear (algebraic) equation in. unknowns, x 1, x 2,..., x n, is. an equation of the form Chapter 5. Linear Algebra A linear (algebraic) equation in n unknowns, x 1, x 2,..., x n, is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b where a 1, a 2,..., a n and b are real numbers. 1

### Chapter 6: Orthogonality Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products

### Linear Algebra Massoud Malek CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Chapter 3 Transformations Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases

### Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column

### Review of Basic Concepts in Linear Algebra Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra

### Elementary linear algebra Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The

### CS 246 Review of Linear Algebra 01/17/19 1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector

### Linear Algebra and Eigenproblems Appendix A A Linear Algebra and Eigenproblems A working knowledge of linear algebra is key to understanding many of the issues raised in this work. In particular, many of the discussions of the details

### Conceptual Questions for Review Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.

### Linear Algebra V = T = ( 4 3 ). Linear Algebra Vectors A column vector is a list of numbers stored vertically The dimension of a column vector is the number of values in the vector W is a -dimensional column vector and V is a 5-dimensional

### 2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian

### Lecture 1: Review of linear algebra Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations

### Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document

### Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017 Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...

### 15 Singular Value Decomposition 15 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing

### Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v ) Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

### NOTES on LINEAR ALGEBRA 1 School of Economics, Management and Statistics University of Bologna Academic Year 207/8 NOTES on LINEAR ALGEBRA for the students of Stats and Maths This is a modified version of the notes by Prof Laura

### Stat 159/259: Linear Algebra Notes Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

### Lecture notes: Applied linear algebra Part 1. Version 2 Lecture notes: Applied linear algebra Part 1. Version 2 Michael Karow Berlin University of Technology karow@math.tu-berlin.de October 2, 2008 1 Notation, basic notions and facts 1.1 Subspaces, range and

### The Singular Value Decomposition The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will

### 1. What is the determinant of the following matrix? a 1 a 2 4a 3 2a 2 b 1 b 2 4b 3 2b c 1. = 4, then det What is the determinant of the following matrix? 3 4 3 4 3 4 4 3 A 0 B 8 C 55 D 0 E 60 If det a a a 3 b b b 3 c c c 3 = 4, then det a a 4a 3 a b b 4b 3 b c c c 3 c = A 8 B 6 C 4 D E 3 Let A be an n n matrix

### TBP MATH33A Review Sheet. November 24, 2018 TBP MATH33A Review Sheet November 24, 2018 General Transformation Matrices: Function Scaling by k Orthogonal projection onto line L Implementation If we want to scale I 2 by k, we use the following: [

### Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.

### Introduction to Matrix Algebra Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

### [POLS 8500] Review of Linear Algebra, Probability and Information Theory [POLS 8500] Review of Linear Algebra, Probability and Information Theory Professor Jason Anastasopoulos ljanastas@uga.edu January 12, 2017 For today... Basic linear algebra. Basic probability. Programming

### Definitions for Quizzes Definitions for Quizzes Italicized text (or something close to it) will be given to you. Plain text is (an example of) what you should write as a definition. [Bracketed text will not be given, nor does

### Throughout these notes we assume V, W are finite dimensional inner product spaces over C. Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal

### Chapter 1. Matrix Algebra ST4233, Linear Models, Semester 1 2008-2009 Chapter 1. Matrix Algebra 1 Matrix and vector notation Definition 1.1 A matrix is a rectangular or square array of numbers of variables. We use uppercase boldface

### Matrix Algebra: Summary May, 27 Appendix E Matrix Algebra: Summary ontents E. Vectors and Matrtices.......................... 2 E.. Notation.................................. 2 E..2 Special Types of Vectors.........................

### A Review of Matrix Analysis Matrix Notation Part Matrix Operations Matrices are simply rectangular arrays of quantities Each quantity in the array is called an element of the matrix and an element can be either a numerical value

### j=1 u 1jv 1j. 1/ 2 Lemma 1. An orthogonal set of vectors must be linearly independent. Lecture Notes: Orthogonal and Symmetric Matrices Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Orthogonal Matrix Definition. Let u = [u

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko

### MATRIX ALGEBRA. or x = (x 1,..., x n ) R n. y 1 y 2. x 2. x m. y m. y = cos θ 1 = x 1 L x. sin θ 1 = x 2. cos θ 2 = y 1 L y. as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations

### 235 Final exam review questions 5 Final exam review questions Paul Hacking December 4, 0 () Let A be an n n matrix and T : R n R n, T (x) = Ax the linear transformation with matrix A. What does it mean to say that a vector v R n is an

### 1. General Vector Spaces 1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### Review problems for MA 54, Fall 2004. Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on

### STA141C: Big Data & High Performance Statistical Computing STA141C: Big Data & High Performance Statistical Computing Numerical Linear Algebra Background Cho-Jui Hsieh UC Davis May 15, 2018 Linear Algebra Background Vectors A vector has a direction and a magnitude

### Stat 206: Linear algebra Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two

### MATH 315 Linear Algebra Homework #1 Assigned: August 20, 2018 Homework #1 Assigned: August 20, 2018 Review the following subjects involving systems of equations and matrices from Calculus II. Linear systems of equations Converting systems to matrix form Pivot entry

### 1 Linearity and Linear Systems Mathematical Tools for Neuroscience (NEU 34) Princeton University, Spring 26 Jonathan Pillow Lecture 7-8 notes: Linear systems & SVD Linearity and Linear Systems Linear system is a kind of mapping f( x)

### B553 Lecture 5: Matrix Algebra Review B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

### Maths for Signals and Systems Linear Algebra in Engineering Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

### 1 Last time: least-squares problems MATH Linear algebra (Fall 07) Lecture Last time: least-squares problems Definition. If A is an m n matrix and b R m, then a least-squares solution to the linear system Ax = b is a vector x R n such that

### 7. Symmetric Matrices and Quadratic Forms Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value

### MAT Linear Algebra Collection of sample exams MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88 Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### Math Linear Algebra Final Exam Review Sheet Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of

### 1. Background: The SVD and the best basis (questions selected from Ch. 6- Can you fill in the exercises?) Math 35 Exam Review SOLUTIONS Overview In this third of the course we focused on linear learning algorithms to model data. summarize: To. Background: The SVD and the best basis (questions selected from

### Review of linear algebra Review of linear algebra 1 Vectors and matrices We will just touch very briefly on certain aspects of linear algebra, most of which should be familiar. Recall that we deal with vectors, i.e. elements of

### A matrix over a field F is a rectangular array of elements from F. The symbol Chapter MATRICES Matrix arithmetic A matrix over a field F is a rectangular array of elements from F The symbol M m n (F ) denotes the collection of all m n matrices over F Matrices will usually be denoted

### Applied Linear Algebra in Geoscience Using MATLAB Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in

### Final Review Sheet. B = (1, 1 + 3x, 1 + x 2 ) then 2 + 3x + 6x 2 Final Review Sheet The final will cover Sections Chapters 1,2,3 and 4, as well as sections 5.1-5.4, 6.1-6.2 and 7.1-7.3 from chapters 5,6 and 7. This is essentially all material covered this term. Watch

### Lecture 6: Geometry of OLS Estimation of Linear Regession Lecture 6: Geometry of OLS Estimation of Linear Regession Xuexin Wang WISE Oct 2013 1 / 22 Matrix Algebra An n m matrix A is a rectangular array that consists of nm elements arranged in n rows and m columns

### Linear Algebra: Matrix Eigenvalue Problems CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

### Solutions to Final Practice Problems Written by Victoria Kala Last updated 12/5/2015 Solutions to Final Practice Problems Written by Victoria Kala vtkala@math.ucsb.edu Last updated /5/05 Answers This page contains answers only. See the following pages for detailed solutions. (. (a x. See

### Chapter Two Elements of Linear Algebra Chapter Two Elements of Linear Algebra Previously, in chapter one, we have considered single first order differential equations involving a single unknown function. In the next chapter we will begin to

### Math Camp II. Basic Linear Algebra. Yiqing Xu. Aug 26, 2014 MIT Math Camp II Basic Linear Algebra Yiqing Xu MIT Aug 26, 2014 1 Solving Systems of Linear Equations 2 Vectors and Vector Spaces 3 Matrices 4 Least Squares Systems of Linear Equations Definition A linear

### Background Mathematics (2/2) 1. David Barber Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and

### Review of Linear Algebra Review of Linear Algebra Dr Gerhard Roth COMP 40A Winter 05 Version Linear algebra Is an important area of mathematics It is the basis of computer vision Is very widely taught, and there are many resources

### MATH 2331 Linear Algebra. Section 2.1 Matrix Operations. Definition: A : m n, B : n p. Example: Compute AB, if possible. MATH 2331 Linear Algebra Section 2.1 Matrix Operations Definition: A : m n, B : n p ( 1 2 p ) ( 1 2 p ) AB = A b b b = Ab Ab Ab Example: Compute AB, if possible. 1 Row-column rule: i-j-th entry of AB:

### Lecture notes on Quantum Computing. Chapter 1 Mathematical Background Lecture notes on Quantum Computing Chapter 1 Mathematical Background Vector states of a quantum system with n physical states are represented by unique vectors in C n, the set of n 1 column vectors 1 For

### Notes on Linear Algebra and Matrix Theory Massimo Franceschet featuring Enrico Bozzo Scalar product The scalar product (a.k.a. dot product or inner product) of two real vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) is not a vector but a

### 1. Linear systems of equations. Chapters 7-8: Linear Algebra. Solution(s) of a linear system of equations (continued) 1 A linear system of equations of the form Sections 75, 78 & 81 a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2 a m1 x 1 + a m2 x 2 + + a mn x n = b m can be written in matrix

### The following definition is fundamental. 1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic

### Parallel Singular Value Decomposition. Jiaxing Tan Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector

### Linear Algebra and Matrix Inversion Jim Lambers MAT 46/56 Spring Semester 29- Lecture 2 Notes These notes correspond to Section 63 in the text Linear Algebra and Matrix Inversion Vector Spaces and Linear Transformations Matrices are much Linear Combination Vector Algebra Angle Between Vectors Projections and Reflections Equality of matrices, Augmented Matrix Matrix Operations Matrix Addition and Matrix Scalar Multiply Matrix Multiply Matrix Linear Algebra Review Fei-Fei Li 1 / 51 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector