Cheng Soon Ong & Christian Walder. Canberra February June 2017
|
|
- Anthony Parsons
- 5 years ago
- Views:
Transcription
1 Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2017 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 141
2 Part III Linear Algebra 91of 141
3 t Linear Curve Fitting - Input Specification N = 10 x (x 1,..., x N ) T t (t 1,..., t N ) T x i R i = 1,..., N t i R i = 1,..., N x 92of 141
4 Strategy in this course Estimate best predictor = training = learning Given data (x 1, y 1 ),..., (x n, y n ), find a predictor f w ( ). 1 Identify the type of input x and output y data 2 Propose a (linear) mathematical model for f w 3 Design an objective function or likelihood 4 Calculate the optimal parameter (w) 5 Model uncertainty using the Bayesian approach 6 Implement and compute (the algorithm in python) 7 Interpret and diagnose results 93of 141
5 t Linear Curve Fitting N = 10 x (x 1,..., x N ) T t (t 1,..., t N ) T x i R i = 1,..., N t i R i = 1,..., N x 94of 141
6 t Linear Curve Fitting - Least Squares N = 10 x (x 1,..., x N ) T t (t 1,..., t N ) T x i R t i R y(x, w) = w 1 x + w 0 X [x 1] i = 1,..., N i = 1,..., N w = (X T X) 1 X T t x 95of 141
7 Intuition Arithmetic zero, positive/negative numbers addition and multiplication Geometry Points and Lines Vector addition and scaling Humans have experience with 3 dimensions (less with 1, 2 though) Generalisation to N dimensions (possibly N ) Line vector space V Point vector x V Example : X R n m Space of matrices R n m and the space of vectors R n m are isomorphic 96of 141
8 Vector Space V, underlying Field F Given vectors u, v, w V and scalars α, β F, the following holds: Associativity of addition : u + (v + w) = (u + v) + w. Commutativity of addition : v + w = w + v. Identity element of addition : 0 such that v + 0 = v, v V. Inverse of addition : For all v V, w V, such that v + w = 0. The additive inverse is denoted v. Distributivity of scalar multiplication with respect to vector addition : α(v + w) = αv + αw. Distributivity of scalar multiplication with respect to field addition : (α + β)v = αv + βv. Compatibility of scalar multiplication with field multiplication : α(βv) = (αβ)v. Identity element of scalar multiplication : 1v = v, where 1 is the identity in F. 97of 141
9 Matrix-Vector Multiplication 1 for i in range (m) : R[ i ] = 0. 0 ; for j in range ( n ) : R[ i ] = R[ i ] + A [ i, j ] V [ j ] Listing 1: Code for elementwise matrix multiplication. 98of 141
10 Matrix-Vector Multiplication 1 for i in range (m) : R[ i ] = 0. 0 ; for j in range ( n ) : R[ i ] = R[ i ] + A [ i, j ] V [ j ] Listing 2: Code for elementwise matrix multiplication. A V = R a 11 a a 1n v 1 a 11 v 1 + a 12 v a 1n v n a 21 a a 2n v = a 21 v 1 + a 22 v a 2n v n... a m1 a m2... a mn v n a m1 v 1 + a m2 v a mn v n 99of 141
11 Matrix-Vector Multiplication A V = R a 11 a a 1n v 1 a 11 v 1 + a 12 v a 1n v n a 21 a a 2n v = a 21 v 1 + a 22 v a 2n v n... a m1 a m2... a mn v n a m1 v 1 + a m2 v a mn v n 100of 141
12 Matrix-Vector Multiplication A V = R a 11 a a 1n v 1 a 11 v 1 + a 12 v a 1n v n a 21 a a 2n v = a 21 v 1 + a 22 v a 2n v n... a m1 a m2... a mn v n a m1 v 1 + a m2 v a mn v n R = A [ :, 0 ] V [ 0 ] ; 2 for j in range ( 1, n ) : R += A [ :, j ] V [ j ] ; Listing 4: Code for columnwise matrix multiplication. 101of 141
13 Matrix-Vector Multiplication Denote the n columns of A by a 1, a 2,..., a n Each a i is now a (column) vector A V = R v 1 a 1 a 2... a n v 2... = a 1 v 1 + a 2 v a n v n v n 102of 141
14 Transpose of R Given R = AV, R = a 1 v 1 + a 2 v a n v n What is R T? 103of 141
15 Transpose of R Given R = AV, R = a 1 v 1 + a 2 v a n v n What is R T? R T = [ v 1 a T 1 + v 2a T v ] na T n NOT equal to A T V T! (In fact, A T V T is not even defined.) 104of 141
16 Transpose Reverse order rule : (AV) T = V T A T 105of 141
17 Transpose Reverse order rule : (AV) T = V T A T V T A T = R T [ ] v1 v 2... v n a T 1 a T 2... a T n = [ v 1 a T 1 + v 2a T v ] na T n 106of 141
18 The trace of a square matrix A is the sum of all diagonal elements of A. n tr {A} = k=1 A kk Example A = tr {A} = = The trace does not exist for a non-square matrix. 107of 141
19 vec(x) operator Define vec(x) as the vector which results from stacking all columns of a matrix A on top of each other. Example A = vec(a) = Given two matrices X, Y R n p, the trace of the product X T Y can be written as tr { X T Y } = vec(x) T vec(y) 108of 141
20 Mapping from two vectors x, y V to a field of scalars F ( F = R or F = C ): Conjugate Symmetry : x, y = y, x, : V V F Linearity : ax, y = a x, y, and x + y, z = x, z + y, z Positive-definitness : x, x 0, and x, x = 0 for x = 0 only. 109of 141
21 Canonical s Inner product for real numbers x, y R: x, y = x y 110of 141
22 Canonical s Inner product for real numbers x, y R: x, y = x y Dot product between two vectors: Given x, y R n x, y = x T y n x k y k = x 1 y 1 + x 2 y x n y n k=1 Canonical inner product for matrices : Given X, Y R n p X, Y = tr { X T Y } = = p k=1 l=1 p (X T Y) kk = k=1 n X lk Y lk p k=1 l=1 n (X T ) kl (Y) lk 111of 141
23 Calculations with Matrices Denote (i, j) th element of matrix X by X ij Transpose : (X T ) ij = X ji Product : (XY) ij =... k=1 X iky kj Proof of the linearity of the canonical matrix inner product : X + Y, Z = tr { (X + Y) T Z } p = ((X + Y) T Z) kk = = = = p k=1 l=1 p k=1 l=1 p k=1 l=1 k=1 n ((X + Y) T ) kl Z lk = n X lk Z lk + Y lk Z lk p n (X T ) kl Z lk + (Y T ) kl Z lk p (X T Z) kk + (Y T Z) kk k=1 k=1 l=1 n (X lk + Y lk )Z lk = tr { X T Z } + tr { Y T Z } = X, Z + Y, Z 112of 141
24 In linear algebra and functional analysis, a projection is a linear transformation P from a vector space V to itself such that P 2 = P 113of 141
25 In linear algebra and functional analysis, a projection is a linear transformation P from a vector space V to itself such that P 2 = P y (a,b) (a-b,0) x 114of 141
26 In linear algebra and functional analysis, a projection is a linear transformation P from a vector space V to itself such that P 2 = P A y [ ] a = b (a-b,0) [ ] a b 0 (a,b) A = x [ ] of 141
27 Orthogonal x y Px Py y - Py Orthogonality : need an inner product x, y Choose two arbitrary vectors x and y. Then Px and y Py are orthogonal. 0 = Px, y Py = (Px) T (y Py) = x T (P T P T P)y 116of 141
28 Orthogonal Orthogonal P 2 = P P = P T Example : Given some unit vector u R n characterising a line through the origin in R n project an arbitrary vector x R n onto this line by Proof : P = P T, and P = uu T P 2 x = (uu T )(uu T )x = uu T x = Px 117of 141
29 Orthogonal Given a matrix A R n p and a vector x. What is the closest point x to x in the column space of A? Orthogonal into the column space of A matrix Proof x = A(A T A) 1 A T x P = A(A T A) 1 A T P 2 = A(A T A) 1 A T A(A T A) 1 A T = A(A T A) 1 A T = P Orthogonal projection? P T = (A(A T A) 1 A T ) T = A(A T A) 1 A T = P 118of 141
30 t Linear Curve Fitting - Least Squares N = 10 x (x 1,..., x N ) T t (t 1,..., t N ) T x i R t i R y(x, w) = w 1 x + w 0 X [x 1] i = 1,..., N i = 1,..., N w = (X T X) 1 X T t x 119of 141
31 Linear Independence of Vectors A set of vectors is linearly independent if none of them can be written as a linear combination of finitely many other vectors in the set. Given a vector space V and k vectors v 1,..., v k V and scalars a 1,..., a k. The vectors are linearly independent, if and only if 0 0 a 1 v 1 + a 2 v a k v k = 0 has only the trivial solution a 1 = a 2 = a k = of 141
32 Rank The column rank of a matrix A is the maximal number of linearly independent columns of A. The row rank of a matrix A is the maximal number of linearly independent rows of A. For every matrix A, the row rank and column rank are equal, called the rank of A. 121of 141
33 Determinant Let S n be the set of all permutation of the numbers {1,..., n}. The determinant of the square matrix A is then given by det {A} = σ S n sgn(σ) n i=1 A i,σ(i) where sgn is the signature of the permutation ( +1 if the permutation is even, 1 if the permutation is odd). det {AB} = det {A} det {B} for A, B square matrices. det { A 1} = det {A} 1. det { A T} = det {A}. det {c A} = c n det {A} for A R n n and c R. 122of 141
34 Identity Matrix I I = AA 1 = A 1 A The matrix inverse A 1 does only exist for square matrices which are NOT singular. Singular matrix at least one eigenvalue is zero, determinant A = 0. Inversion reverses the order (like transposition) (AB) 1 = B 1 A 1 (Only then is (AB)(AB) 1 = (AB)B 1 A 1 = AA 1 = I.) 123of 141
35 and Transpose (A 1 ) T? 124of 141
36 and Transpose (A 1 ) T? need to assume that (A 1 ) exists rule : (A 1 ) T = (A T ) 1 125of 141
37 Useful Identity (A 1 + B T C 1 B) 1 B T C 1 = AB T (BAB T + C) 1 126of 141
38 Useful Identity (A 1 + B T C 1 B) 1 B T C 1 = AB T (BAB T + C) 1 How to analyse and prove such an equation? A 1 must exist, therefore A must be square, say A R n n. Same for C 1, so let s assume C R m m. Therefore, B R m n. 127of 141
39 Prove the Identity (A 1 + B T C 1 B) 1 B T C 1 = AB T (BAB T + C) 1 Multiply by (BAB T + C) (A 1 + B T C 1 B) 1 B T C 1 (BAB T + C) = AB T Simplify the left-hand side (A 1 + B T C 1 B) 1 [B T C 1 B(AB T ) + B T ] = (A 1 + B T C 1 B) 1 [B T C 1 B + A 1 ]AB T = AB T 128of 141
40 Woodbury Identity (A + BD 1 C) 1 = A 1 A 1 B(D + CA 1 B) 1 CA 1 Useful if A is diagonal and easy to invert, and B is very tall (many rows, but only a few columns) and C is very wide (few rows, many columns). 129of 141
41 Manipulating Matrix Equations Don t multiply by a matrix which does not have full rank. Why? In the general case, you will lose solutions. Don t multiply by nonsquare matrices. Don t assume a matrix inverse exists because a matrix is square. The matrix might be singular and you are making the same mistake as if deducing a = b from a 0 = b of 141
42 Every square matrix A R n n has an Eigenvector decomposition Ax = λx where x R n and λ C. Example: [ ] 0 1 x = λx 1 0 λ = { ı, ı} {[ [ ı ı x =, 1] 1]} 131of 141
43 How many eigenvalue/eigenvector pairs? Ax = λx is equivalent to (A λi)x = 0 Has only non-trivial solution for det {A λi} = 0 polynom of nth order; at most n distinct solutions 132of 141
44 Real Eigenvalues How can we enforce real eigenvalues? Let s look at matrices with complex entries A C n n. Transposition is replaced by Hermitian adjoint, e.g. [ ] H 1 + ı2 3 + ı4 = 5 + ı6 7 + ı8 [ 1 ı2 ] 5 ı6 3 ı4 7 ı8 Denote the complex conjugate of a complex number λ by λ. 133of 141
45 Real Eigenvalues How can we enforce real eigenvalues? Let s assume A C n n, Hermitian (A H = A). Calculate x H Ax = λx H x for an eigenvector x C n of A. Another possibility to calculate x H Ax and therefore x H Ax = x H A H x (A is Hermitian) = (x H Ax) H (reverse order) = (λx H x) H (eigenvalue) = λx H x λ = λ (λ is real). If A is Hermitian, then all eigenvalues are real. Special case: If A has only real entries and is symmetric, then all eigenvalues are real. 134of 141
46 Every matrix A R n p can be decomposed into a product of three matrices A = UΣV T where U R n n and V R p p are orthogonal matrices ( U T U = I and V T V = I ), and Σ R n p has nonnegative numbers on the diagonal. 135of 141
47 t Linear Curve Fitting - Least Squares N = 10 x (x 1,..., x N ) T t (t 1,..., t N ) T x i R t i R y(x, w) = w 1 x + w 0 X [x 1] i = 1,..., N i = 1,..., N w = (X T X) 1 X T t x 136of 141
48 Inverse of a matrix Assume a full rank symmetric real matrix A. Then A = U T ΛU where Λ is a diagonal matrix with real eigenvalues U contains the eigenvectors A 1 = (U T ΛU) 1 = U 1 Λ 1 U T inverse changes order = U T Λ 1 U U T U = I The inverse of a diagonal matrix is the inverse of its elements. 137of 141
49 How to calculate a gradient? Given a vector space V, and a function f : V R. Example: X R n p, arbitrary C R n n, f (X) = tr { X T CX }. How to calculate the gradient of f? 138of 141
50 How to calculate a gradient? Given a vector space V, and a function f : V R. Example: X R n p, arbitrary C R n n, f (X) = tr { X T CX }. How to calculate the gradient of f? Ill-defined question. There is no gradient in a vector space. Only a Directional Derivative at point X V in direction ξ V. f (X + hξ) f (X) Df (X)(ξ) = lim h 0 h For the example f (X) = tr { X T CX } Df (X)(ξ) = lim h 0 tr { (X + hξ) T C(X + hξ) } tr { X T CX } h = tr { ξ T CX + X T Cξ } = tr { X T (C T + C)ξ } 139of 141
51 How to calculate a gradient? Given a vector space V, and a function f : V R. How to calculate the gradient of f? 1 Calculate the Directional Derivative 2 Define an inner product A, B on the vector space 3 The gradient is now defined as Df (X)(ξ) = grad f, ξ For the example f (X) = tr { X T CX } Define the inner product A, B = tr { A T B }. Write the Directional Derivative as an inner product with ξ Df (X)(ξ) = tr { X T (C T + C)ξ } = (C + C T )X, ξ = grad f, ξ 140of 141
52 Gilbert Strang, "Introduction to Linear Algebra", Wellesley Cambridge, David C. Lay, Steven R. Lay, Judi J. McDonald, "Linear Algebra and Its Applications", Pearson, Jonathan S. Golan, "The Linear Algebra a Beginning Graduate Student Ought to Know", Springer, 2012 Kaare B. Petersen, Michael S. Pedersen, "The Matrix Cookbook", of 141
Knowledge Discovery and Data Mining 1 (VO) ( )
Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationCS 246 Review of Linear Algebra 01/17/19
1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector
More informationBackground Mathematics (2/2) 1. David Barber
Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationB553 Lecture 5: Matrix Algebra Review
B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationMath Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p
Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the
More informationLinear Algebra Review. Fei-Fei Li
Linear Algebra Review Fei-Fei Li 1 / 37 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector
More informationMath Camp II. Basic Linear Algebra. Yiqing Xu. Aug 26, 2014 MIT
Math Camp II Basic Linear Algebra Yiqing Xu MIT Aug 26, 2014 1 Solving Systems of Linear Equations 2 Vectors and Vector Spaces 3 Matrices 4 Least Squares Systems of Linear Equations Definition A linear
More informationQuick Tour of Linear Algebra and Graph Theory
Quick Tour of Linear Algebra and Graph Theory CS224W: Social and Information Network Analysis Fall 2014 David Hallac Based on Peter Lofgren, Yu Wayne Wu, and Borja Pelato s previous versions Matrices and
More informationIntroduction to Matrix Algebra
Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p
More informationLinear Algebra Review
Chapter 1 Linear Algebra Review It is assumed that you have had a course in linear algebra, and are familiar with matrix multiplication, eigenvectors, etc. I will review some of these terms here, but quite
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationFoundations of Matrix Analysis
1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the
More informationMATRIX ALGEBRA. or x = (x 1,..., x n ) R n. y 1 y 2. x 2. x m. y m. y = cos θ 1 = x 1 L x. sin θ 1 = x 2. cos θ 2 = y 1 L y.
as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More informationReview of Linear Algebra
Review of Linear Algebra Dr Gerhard Roth COMP 40A Winter 05 Version Linear algebra Is an important area of mathematics It is the basis of computer vision Is very widely taught, and there are many resources
More informationIr O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )
Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O
More informationMATH 583A REVIEW SESSION #1
MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),
More informationReview of Linear Algebra
Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=
More informationLINEAR ALGEBRA BOOT CAMP WEEK 4: THE SPECTRAL THEOREM
LINEAR ALGEBRA BOOT CAMP WEEK 4: THE SPECTRAL THEOREM Unless otherwise stated, all vector spaces in this worksheet are finite dimensional and the scalar field F is R or C. Definition 1. A linear operator
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More information22.3. Repeated Eigenvalues and Symmetric Matrices. Introduction. Prerequisites. Learning Outcomes
Repeated Eigenvalues and Symmetric Matrices. Introduction In this Section we further develop the theory of eigenvalues and eigenvectors in two distinct directions. Firstly we look at matrices where one
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationMathematical Methods wk 2: Linear Operators
John Magorrian, magog@thphysoxacuk These are work-in-progress notes for the second-year course on mathematical methods The most up-to-date version is available from http://www-thphysphysicsoxacuk/people/johnmagorrian/mm
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko
More informationMath Linear Algebra Final Exam Review Sheet
Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationIntroduction to Matrices
POLS 704 Introduction to Matrices Introduction to Matrices. The Cast of Characters A matrix is a rectangular array (i.e., a table) of numbers. For example, 2 3 X 4 5 6 (4 3) 7 8 9 0 0 0 Thismatrix,with4rowsand3columns,isoforder
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationLarge Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset
More information18.06SC Final Exam Solutions
18.06SC Final Exam Solutions 1 (4+7=11 pts.) Suppose A is 3 by 4, and Ax = 0 has exactly 2 special solutions: 1 2 x 1 = 1 and x 2 = 1 1 0 0 1 (a) Remembering that A is 3 by 4, find its row reduced echelon
More informationFurther Mathematical Methods (Linear Algebra) 2002
Further Mathematical Methods (Linear Algebra) 00 Solutions For Problem Sheet 0 In this Problem Sheet we calculated some left and right inverses and verified the theorems about them given in the lectures.
More informationLinear Algebra Review. Fei-Fei Li
Linear Algebra Review Fei-Fei Li 1 / 51 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector
More informationMaths for Signals and Systems Linear Algebra in Engineering
Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE
More informationChapter 4 - MATRIX ALGEBRA. ... a 2j... a 2n. a i1 a i2... a ij... a in
Chapter 4 - MATRIX ALGEBRA 4.1. Matrix Operations A a 11 a 12... a 1j... a 1n a 21. a 22.... a 2j... a 2n. a i1 a i2... a ij... a in... a m1 a m2... a mj... a mn The entry in the ith row and the jth column
More informationA = 3 B = A 1 1 matrix is the same as a number or scalar, 3 = [3].
Appendix : A Very Brief Linear ALgebra Review Introduction Linear Algebra, also known as matrix theory, is an important element of all branches of mathematics Very often in this course we study the shapes
More informationRepeated Eigenvalues and Symmetric Matrices
Repeated Eigenvalues and Symmetric Matrices. Introduction In this Section we further develop the theory of eigenvalues and eigenvectors in two distinct directions. Firstly we look at matrices where one
More informationGRE Subject test preparation Spring 2016 Topic: Abstract Algebra, Linear Algebra, Number Theory.
GRE Subject test preparation Spring 2016 Topic: Abstract Algebra, Linear Algebra, Number Theory. Linear Algebra Standard matrix manipulation to compute the kernel, intersection of subspaces, column spaces,
More informationA VERY BRIEF LINEAR ALGEBRA REVIEW for MAP 5485 Introduction to Mathematical Biophysics Fall 2010
A VERY BRIEF LINEAR ALGEBRA REVIEW for MAP 5485 Introduction to Mathematical Biophysics Fall 00 Introduction Linear Algebra, also known as matrix theory, is an important element of all branches of mathematics
More informationFoundations of Computer Vision
Foundations of Computer Vision Wesley. E. Snyder North Carolina State University Hairong Qi University of Tennessee, Knoxville Last Edited February 8, 2017 1 3.2. A BRIEF REVIEW OF LINEAR ALGEBRA Apply
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More informationLinGloss. A glossary of linear algebra
LinGloss A glossary of linear algebra Contents: Decompositions Types of Matrices Theorems Other objects? Quasi-triangular A matrix A is quasi-triangular iff it is a triangular matrix except its diagonal
More informationLinear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0.
Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the
More informationlinearly indepedent eigenvectors as the multiplicity of the root, but in general there may be no more than one. For further discussion, assume matrice
3. Eigenvalues and Eigenvectors, Spectral Representation 3.. Eigenvalues and Eigenvectors A vector ' is eigenvector of a matrix K, if K' is parallel to ' and ' 6, i.e., K' k' k is the eigenvalue. If is
More informationLINEAR ALGEBRA SUMMARY SHEET.
LINEAR ALGEBRA SUMMARY SHEET RADON ROSBOROUGH https://intuitiveexplanationscom/linear-algebra-summary-sheet/ This document is a concise collection of many of the important theorems of linear algebra, organized
More informationSymmetric and anti symmetric matrices
Symmetric and anti symmetric matrices In linear algebra, a symmetric matrix is a square matrix that is equal to its transpose. Formally, matrix A is symmetric if. A = A Because equal matrices have equal
More information18.06 Problem Set 8 - Solutions Due Wednesday, 14 November 2007 at 4 pm in
806 Problem Set 8 - Solutions Due Wednesday, 4 November 2007 at 4 pm in 2-06 08 03 Problem : 205+5+5+5 Consider the matrix A 02 07 a Check that A is a positive Markov matrix, and find its steady state
More informationNOTES on LINEAR ALGEBRA 1
School of Economics, Management and Statistics University of Bologna Academic Year 207/8 NOTES on LINEAR ALGEBRA for the students of Stats and Maths This is a modified version of the notes by Prof Laura
More informationChapter 3. Matrices. 3.1 Matrices
40 Chapter 3 Matrices 3.1 Matrices Definition 3.1 Matrix) A matrix A is a rectangular array of m n real numbers {a ij } written as a 11 a 12 a 1n a 21 a 22 a 2n A =.... a m1 a m2 a mn The array has m rows
More informationQuantum Computing Lecture 2. Review of Linear Algebra
Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces
More informationNotes on Linear Algebra and Matrix Theory
Massimo Franceschet featuring Enrico Bozzo Scalar product The scalar product (a.k.a. dot product or inner product) of two real vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) is not a vector but a
More information10-701/ Recitation : Linear Algebra Review (based on notes written by Jing Xiang)
10-701/15-781 Recitation : Linear Algebra Review (based on notes written by Jing Xiang) Manojit Nandi February 1, 2014 Outline Linear Algebra General Properties Matrix Operations Inner Products and Orthogonal
More information1. What is the determinant of the following matrix? a 1 a 2 4a 3 2a 2 b 1 b 2 4b 3 2b c 1. = 4, then det
What is the determinant of the following matrix? 3 4 3 4 3 4 4 3 A 0 B 8 C 55 D 0 E 60 If det a a a 3 b b b 3 c c c 3 = 4, then det a a 4a 3 a b b 4b 3 b c c c 3 c = A 8 B 6 C 4 D E 3 Let A be an n n matrix
More informationLinear Algebra and Matrices
Linear Algebra and Matrices 4 Overview In this chapter we studying true matrix operations, not element operations as was done in earlier chapters. Working with MAT- LAB functions should now be fairly routine.
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 89 Part II
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More informationMath 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008
Math 520 Exam 2 Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008 Exam 2 will be held on Tuesday, April 8, 7-8pm in 117 MacMillan What will be covered The exam will cover material from the lectures
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationq n. Q T Q = I. Projections Least Squares best fit solution to Ax = b. Gram-Schmidt process for getting an orthonormal basis from any basis.
Exam Review Material covered by the exam [ Orthogonal matrices Q = q 1... ] q n. Q T Q = I. Projections Least Squares best fit solution to Ax = b. Gram-Schmidt process for getting an orthonormal basis
More informationA matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and
Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.
More informationLecture II: Linear Algebra Revisited
Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function
More information3 Matrix Algebra. 3.1 Operations on matrices
3 Matrix Algebra A matrix is a rectangular array of numbers; it is of size m n if it has m rows and n columns. A 1 n matrix is a row vector; an m 1 matrix is a column vector. For example: 1 5 3 5 3 5 8
More informationNumerical Linear Algebra Homework Assignment - Week 2
Numerical Linear Algebra Homework Assignment - Week 2 Đoàn Trần Nguyên Tùng Student ID: 1411352 8th October 2016 Exercise 2.1: Show that if a matrix A is both triangular and unitary, then it is diagonal.
More informationNumerical Methods for Solving Large Scale Eigenvalue Problems
Peter Arbenz Computer Science Department, ETH Zürich E-mail: arbenz@inf.ethz.ch arge scale eigenvalue problems, Lecture 2, February 28, 2018 1/46 Numerical Methods for Solving Large Scale Eigenvalue Problems
More informationLinear Algebra - Part II
Linear Algebra - Part II Projection, Eigendecomposition, SVD (Adapted from Sargur Srihari s slides) Brief Review from Part 1 Symmetric Matrix: A = A T Orthogonal Matrix: A T A = AA T = I and A 1 = A T
More informationJim Lambers MAT 610 Summer Session Lecture 1 Notes
Jim Lambers MAT 60 Summer Session 2009-0 Lecture Notes Introduction This course is about numerical linear algebra, which is the study of the approximate solution of fundamental problems from linear algebra
More informationImage Registration Lecture 2: Vectors and Matrices
Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this
More informationChapter 2. Matrix Arithmetic. Chapter 2
Matrix Arithmetic Matrix Addition and Subtraction Addition and subtraction act element-wise on matrices. In order for the addition/subtraction (A B) to be possible, the two matrices A and B must have the
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationA matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and
Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.
More informationMATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)
MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m
More informationLINEAR ALGEBRA BOOT CAMP WEEK 2: LINEAR OPERATORS
LINEAR ALGEBRA BOOT CAMP WEEK 2: LINEAR OPERATORS Unless otherwise stated, all vector spaces in this worksheet are finite dimensional and the scalar field F has characteristic zero. The following are facts
More informationLinear Algebra. Workbook
Linear Algebra Workbook Paul Yiu Department of Mathematics Florida Atlantic University Last Update: November 21 Student: Fall 2011 Checklist Name: A B C D E F F G H I J 1 2 3 4 5 6 7 8 9 10 xxx xxx xxx
More informationMatrices and Linear Algebra
Contents Quantitative methods for Economics and Business University of Ferrara Academic year 2017-2018 Contents 1 Basics 2 3 4 5 Contents 1 Basics 2 3 4 5 Contents 1 Basics 2 3 4 5 Contents 1 Basics 2
More informationLinear Algebra (Review) Volker Tresp 2017
Linear Algebra (Review) Volker Tresp 2017 1 Vectors k is a scalar (a number) c is a column vector. Thus in two dimensions, c = ( c1 c 2 ) (Advanced: More precisely, a vector is defined in a vector space.
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More information11 a 12 a 21 a 11 a 22 a 12 a 21. (C.11) A = The determinant of a product of two matrices is given by AB = A B 1 1 = (C.13) and similarly.
C PROPERTIES OF MATRICES 697 to whether the permutation i 1 i 2 i N is even or odd, respectively Note that I =1 Thus, for a 2 2 matrix, the determinant takes the form A = a 11 a 12 = a a 21 a 11 a 22 a
More informationMATRICES ARE SIMILAR TO TRIANGULAR MATRICES
MATRICES ARE SIMILAR TO TRIANGULAR MATRICES 1 Complex matrices Recall that the complex numbers are given by a + ib where a and b are real and i is the imaginary unity, ie, i 2 = 1 In what we describe below,
More informationQuick Tour of Linear Algebra and Graph Theory
Quick Tour of Linear Algebra and Graph Theory CS224w: Social and Information Network Analysis Fall 2012 Yu Wayne Wu Based on Borja Pelato s version in Fall 2011 Matrices and Vectors Matrix: A rectangular
More information33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM
33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM (UPDATED MARCH 17, 2018) The final exam will be cumulative, with a bit more weight on more recent material. This outline covers the what we ve done since the
More informationConceptual Questions for Review
Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.
More informationComputational Methods CMSC/AMSC/MAPL 460. Eigenvalues and Eigenvectors. Ramani Duraiswami, Dept. of Computer Science
Computational Methods CMSC/AMSC/MAPL 460 Eigenvalues and Eigenvectors Ramani Duraiswami, Dept. of Computer Science Eigen Values of a Matrix Recap: A N N matrix A has an eigenvector x (non-zero) with corresponding
More informationNotes on singular value decomposition for Math 54. Recall that if A is a symmetric n n matrix, then A has real eigenvalues A = P DP 1 A = P DP T.
Notes on singular value decomposition for Math 54 Recall that if A is a symmetric n n matrix, then A has real eigenvalues λ 1,, λ n (possibly repeated), and R n has an orthonormal basis v 1,, v n, where
More informationLecture notes: Applied linear algebra Part 1. Version 2
Lecture notes: Applied linear algebra Part 1. Version 2 Michael Karow Berlin University of Technology karow@math.tu-berlin.de October 2, 2008 1 Notation, basic notions and facts 1.1 Subspaces, range and
More informationLinear Algebra. Min Yan
Linear Algebra Min Yan January 2, 2018 2 Contents 1 Vector Space 7 1.1 Definition................................. 7 1.1.1 Axioms of Vector Space..................... 7 1.1.2 Consequence of Axiom......................
More informationMAT 2037 LINEAR ALGEBRA I web:
MAT 237 LINEAR ALGEBRA I 2625 Dokuz Eylül University, Faculty of Science, Department of Mathematics web: Instructor: Engin Mermut http://kisideuedutr/enginmermut/ HOMEWORK 2 MATRIX ALGEBRA Textbook: Linear
More informationMATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators.
MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators. Adjoint operator and adjoint matrix Given a linear operator L on an inner product space V, the adjoint of L is a transformation
More informationGlossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB
Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the
More informationMassachusetts Institute of Technology Department of Economics Statistics. Lecture Notes on Matrix Algebra
Massachusetts Institute of Technology Department of Economics 14.381 Statistics Guido Kuersteiner Lecture Notes on Matrix Algebra These lecture notes summarize some basic results on matrix algebra used
More informationMathematics Department Stanford University Math 61CM/DM Inner products
Mathematics Department Stanford University Math 61CM/DM Inner products Recall the definition of an inner product space; see Appendix A.8 of the textbook. Definition 1 An inner product space V is a vector
More information