Lecture 3: Review of Linear Algebra


 Giles Oliver
 11 months ago
 Views:
Transcription
1 ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms, etc) as matrices This lecture reviews basic concepts from linear algebra that will be useful Linear Vector Space Definition A linear vector space X is a collection of elements satisfying the following properties: addition: x, y, z X a) x + y X b) x + y = y + x c) (x + y) + z = x + (y + z) d) X, such that x + = x e) x X, x X such that x + ( x) = multiplication: x, y X and a, b R a) a x X b) a(b x) = (ab) x c) x = x, x = d) a(x + y) = ax + ay Example Here are two examples of linear vector spaces The familiar ddimensional Euclidean space R d and the space of finite energy signals/functions supported on the interval [, T ] { } L 2 ([, T ]) := x : T It is easy to verify the properties above for both examples x 2 (t) dt < Definition 2 A subset M X is subspace if x, y M ax + by M, scalars a, b Definition 3 An inner product is a mapping from X X to R The inner product between any x, y X is denoted by x, y and it satisfies the following properties for all x, y, z X : a) x, y = y, x b) ax, y = a x, y, scalars a c) x + y, z = x, z + y, z d) x, x A space X equipped with an inner product is called an inner product space
2 Lecture 3: Review of Linear Algebra 2 Example 2 Let X = R n Then x, y := x T y = n i= x iy i Example 3 Let X = L 2 ([, ]) Then x, y := x(t)y(t) dt The inner product measures the alignment of the two vectors The inner product induces a norm defined as x := x, x The norm measures the length/size of x The inner product x, y = x y cos(θ), where θ is the angle between x and y Thus, in general, for every x, y X we have x, y x y, with equality if and only if x and y are linearly dependent or parallel ; ie, θ = This is called the CauchySchwarz Inequality Two vectors x, y are orthogonal if x, y = Example 4 Let X = R 2, then x = and y = are orthogonal, as are u = and v = Definition 4 An inner product space that contains all its limits is called a Hilbert Space and in this case we often denote the space by H; ie, if x, x 2, are in H and lim n x n exists, then the limit is also in H It is easy to verify that R n, L 2 ([, T ]), and l 2 (Z), the set of all finite energy sequences (eg, discretetime signals), are all Hilbert Spaces 2 Bases and Representations Definition 5 A collection of vectors {x,, x k } are said to be linearly independent if none of them can be represented as a linear combination of the others That is, for any x i and every set of scalar weights {θ j } we have x i j i θ jx j Definition 6 The set of all vectors that can be generated by taking linear combinations of {x,, x k } have the form k v = θ i x i, is called the span of {x,, x k }, denoted span(x,, x k ) i= Definition 7 A set of linearly independent vectors {φ i } i is a basis for H if every x H can be represented as a unique linear combination of {φ i } That is, every x H can be expressed as x = i θ i φ i for a certain unique set of scalar weights {θ i } [ Example 5 Let H = R 2 Then and are a basis (since they are orthogonal) Also, [ ] are a basis because they are linearly independent (although not orthogonal) ] and Definition 8 An orthonormal basis is one satisfying φ i, φ j = δ ij := {, i = j, i j
3 Lecture 3: Review of Linear Algebra 3 Every x H can be represented in terms of an orthonormal basis {φ i } i (or orthobasis for short) according to: x = i x, φ i φ i This is easy to see as follows Suppose x has a representation i θ iφ i Then x, φ j = i θ iφ i, φ j = i θ i φ i, φ j = i θ iδ i,j = θ j Example 6 Here is an orthobasis for L 2 ([, ]): for i =, 2, Doesn t it look familiar? φ 2i (t) := 2 cos(2π(i )t) φ 2i (t) := 2 sin(2πit) Any basis can be converted into an orthonormal basis using GramSchmidt Orthogonalization Let {φ i } be a basis for a vector space X An orthobasis for X can be constructed as follows GramSchmidt Orthogonalization ψ := φ / φ 2 ψ 2 := φ 2 ψ, φ 2 ψ ; ψ 2 = ψ 2/ ψ 2 k ψ k = φ k k i= ψ i, φ k ψ i ; ψ k = ψ k / ψ k 3 Orthogonal Projections and Filters One of the most important tools that we will use from linear algebra is the notion of an orthogonal projection Most linear filters used in signal processing (eg, bandpass filters, averaging filters, etc) may all be interpreted as or related to orthogonal projections Let H be a Hilbert space and let M H be a subspace Every x H can be written as x = y + z, where y M and z M, which is shorthand for z orthogonal to M; that is v M, v, z = The vector y is the optimal approximation to x in terms of vectors in M in the following sense: x y = min x v v M The vector y is called the projection of x onto M Here is an application of this concept Let M H and let {φ i } r i= be an orthobasis for M We say that the subspace M is spanned by {φ i } r i= Note that this implies that M is an rdimensional subspace of H (and it is isometric to R r ) For any x H, the projection of x onto M is given by y = r φ i, x φ i i= and this projection can be viewed as a sort of filter that removes all components of the signal that are orthogonal to M
4 Lecture 3: Review of Linear Algebra 4 [ ] [ Example 7 Let H = R 2 Consider the canonical coordinate system φ = and φ 2 = the subspace spanned by φ The projection of any x = [x x 2 ] T R 2 onto this subspace is ( [ ]) x P x = x, = [x x 2 ] = ] Consider The projection operator P is just a matrix and it is given by P := φ φ T = [ ] = [ ] [ ] / 2 It is also easy to check that φ = / / 2 and φ 2 = 2 / is an orthobasis for R 2 What is the 2 projection operator onto the span of φ in this case? More generally suppose we are considering R n and we have an orthobasis {φ i } r i= for some rdimensional, r < n, subspace M of R n Then the projection matrix is given by P M = r i= φ iφ T i Moreover, if {φ i} r i= is a basis for M, but not necessarily orthonormal, then P M = Φ(Φ T Φ) Φ T, where Φ = [φ,, φ r ], a matrix whose columns are the basis vectors Example 8 Let H = L 2 ([, ]) and let M = {linear functions on [, ]} Since all linear functions have the form at + b, for t [, ], here is a basis for M: φ (t) =, φ 2 (t) = t Note that this means that M is twodimensional That makes sense since every line is defined by its slope and intercept (two real numbers) Using the GramSchmidt procedure we can construct the orthobasis ψ (t) =, ψ 2 (t) = t /2 Now, consider any signal/function x L 2 ([, ]) The projection of x onto M is 4 Eigenanalysis P M x = x, + x, t /2 (t /2) = x(τ) dτ + (t /2) (τ /2)x(t) dτ Suppose A is an m n matrix with entries from a field (eg, R or C, the latter being the complex numbers) Then there exists a factorization of the form A = U D V where U = [u u m ] is m m with orthonormal columns, V = [v v n ] is n n with orthonormal columns and the superscript means transposition and conjugation (if complexvalued), and the matrix D is m n and has the form σ σ 2 D = σ m The values σ,, σ m are called the singular values of A and the factorization is called the singular value decomposition (SVD) Because of the orthonormality of the columns of U and V we have Av i = σ i u i and A u i = σ i v i, i =,, m A vector u is called an eigenvector of A if A u = λ u for some scalar λ The scalar λ is called the eigenvalue associated with u Symmetric matrices (which are always square) always have real eigenvalues and have an eigendecomposition of the form A = UDU, where the columns of U the orthonormal eigenvectors of A, D is a diagonal matrix written D = diag(λ,, λ n ), and diagonal entries are the eigenvalues This is just a special case of the SVD A symmetric positivesemidefinite matrix satisfies the property v T Av for all v This implies that the eigenvalues of symmetric positivesemidefinite matrices are nonnegative
5 Lecture 3: Review of Linear Algebra 5 Example 9 Let X be a random vector taking values in R n and recall the definition of the covariance matrix: Σ := E[(X µ)(x µ) T ] It is easy to see that v T Σv, and of course Σ is symmetric Therefore, every covariance matrix has an eigendecomposition of the form Σ = UDU, where D = diag(λ,, λ n ) and λ i for i =,, n The KarhunenLoève Transform (KLT), which is also called Principal Component Analysis (PCA), is based on transforming a random vector X into the coordinate system associated with the eigendecomposition of the covariance of X Let X be an ndimensional random vector with covariance matrix Σ = UDU Let u,, u n be the eigenvectors Assume that the eigenvectors and eigenvalues are ordered such that λ λ 2 λ n The KLT or PCA representation of the random vector X is given by X = n (u T i X)u i i= The coefficients in this representation can be arranged in a vector as θ = U T X, where U is as defined above The vector θ is called the KLT or PCA of X Using this representation we can define the approximation to X in the span of the first r < n eigenvectors X r = r (u T i X)u i i= Note that this approximation involves only r scalar random variables {(u T i X)}r i= rather than n In fact, it is easy to show that among all rterm linear approximations of X in terms of r random variables, X r has the smallest mean square error; that is if we let S r denote all rterm linear approximations to X, then E[ X X r 2 ] = min E[ X Y r ] Y r S r
Lecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More information2. Review of Linear Algebra
2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationMATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)
MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m
More information(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =
. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)
More informationIr O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )
Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O
More informationBasic Concepts in Matrix Algebra
Basic Concepts in Matrix Algebra An column array of p elements is called a vector of dimension p and is written as x p 1 = x 1 x 2. x p. The transpose of the column vector x p 1 is row vector x = [x 1
More informationMath 443 Differential Geometry Spring Handout 3: Bilinear and Quadratic Forms This handout should be read just before Chapter 4 of the textbook.
Math 443 Differential Geometry Spring 2013 Handout 3: Bilinear and Quadratic Forms This handout should be read just before Chapter 4 of the textbook. Endomorphisms of a Vector Space This handout discusses
More information1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)
Notes for 20170130 Most of mathematics is best learned by doing. Linear algebra is no exception. You have had a previous class in which you learned the basics of linear algebra, and you will have plenty
More informationElementary Linear Algebra Review for Exam 2 Exam is Monday, November 16th.
Elementary Linear Algebra Review for Exam Exam is Monday, November 6th. The exam will cover sections:.4,..4, 5. 5., 7., the class notes on Markov Models. You must be able to do each of the following. Section.4
More informationDef. The euclidian distance between two points x = (x 1,...,x p ) t and y = (y 1,...,y p ) t in the pdimensional space R p is defined as
MAHALANOBIS DISTANCE Def. The euclidian distance between two points x = (x 1,...,x p ) t and y = (y 1,...,y p ) t in the pdimensional space R p is defined as d E (x, y) = (x 1 y 1 ) 2 + +(x p y p ) 2
More informationThere are two things that are particularly nice about the first basis
Orthogonality and the GramSchmidt Process In Chapter 4, we spent a great deal of time studying the problem of finding a basis for a vector space We know that a basis for a vector space can potentially
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a ToolBox Linear Equation Systems Discretization of differential equations: solving linear equations
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More information5.) For each of the given sets of vectors, determine whether or not the set spans R 3. Give reasons for your answers.
Linear Algebra  Test File  Spring Test # For problems  consider the following system of equations. x + y  z = x + y + 4z = x + y + 6z =.) Solve the system without using your calculator..) Find the
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationThe Eigenvalue Problem: Perturbation Theory
Jim Lambers MAT 610 Summer Session 200910 Lecture 13 Notes These notes correspond to Sections 7.2 and 8.1 in the text. The Eigenvalue Problem: Perturbation Theory The Unsymmetric Eigenvalue Problem Just
More informationI. Multiple Choice Questions (Answer any eight)
Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam  Course Instructor : Prashanth L.A. Date : Sep24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY
More informationInner products. Theorem (basic properties): Given vectors u, v, w in an inner product space V, and a scalar k, the following properties hold:
Inner products Definition: An inner product on a real vector space V is an operation (function) that assigns to each pair of vectors ( u, v) in V a scalar u, v satisfying the following axioms: 1. u, v
More informationCOMP6237 Data Mining Covariance, EVD, PCA & SVD. Jonathon Hare
COMP6237 Data Mining Covariance, EVD, PCA & SVD Jonathon Hare jsh2@ecs.soton.ac.uk Variance and Covariance Random Variables and Expected Values Mathematicians talk variance (and covariance) in terms of
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a twodimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a twodimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationMATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2.
MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2. Diagonalization Let L be a linear operator on a finitedimensional vector space V. Then the following conditions are equivalent:
More informationLECTURE VI: SELFADJOINT AND UNITARY OPERATORS MAT FALL 2006 PRINCETON UNIVERSITY
LECTURE VI: SELFADJOINT AND UNITARY OPERATORS MAT 204  FALL 2006 PRINCETON UNIVERSITY ALFONSO SORRENTINO 1 Adjoint of a linear operator Note: In these notes, V will denote a ndimensional euclidean vector
More informationMA 265 FINAL EXAM Fall 2012
MA 265 FINAL EXAM Fall 22 NAME: INSTRUCTOR S NAME:. There are a total of 25 problems. You should show work on the exam sheet, and pencil in the correct answer on the scantron. 2. No books, notes, or calculators
More informationSingular Value Decomposition
Singular Value Decomposition Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A =
More informationOptimization Theory. A Concise Introduction. Jiongmin Yong
October 11, 017 16:5 wsbook9x6 Book Title Optimization Theory 01708Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 01708Lecture Notes page Optimization
More informationSingular Value Decomposition
Chapter 6 Singular Value Decomposition In Chapter 5, we derived a number of algorithms for computing the eigenvalues and eigenvectors of matrices A R n n. Having developed this machinery, we complete our
More information2. Matrix Algebra and Random Vectors
2. Matrix Algebra and Random Vectors 2.1 Introduction Multivariate data can be conveniently display as array of numbers. In general, a rectangular array of numbers with, for instance, n rows and p columns
More informationLinear Algebra for Machine Learning. Sargur N. Srihari
Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it
More informationDSGA 1002 Lecture notes 10 November 23, Linear models
DSGA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.
More informationCHAPTER VIII HILBERT SPACES
CHAPTER VIII HILBERT SPACES DEFINITION Let X and Y be two complex vector spaces. A map T : X Y is called a conjugatelinear transformation if it is a reallinear transformation from X into Y, and if T (λx)
More informationInequalities in Hilbert Spaces
Inequalities in Hilbert Spaces Jan Wigestrand Master of Science in Mathematics Submission date: March 8 Supervisor: Eugenia Malinnikova, MATH Norwegian University of Science and Technology Department of
More informationThe Spectral Theorem for normal linear maps
MAT067 University of California, Davis Winter 2007 The Spectral Theorem for normal linear maps Isaiah Lankham, Bruno Nachtergaele, Anne Schilling (March 14, 2007) In this section we come back to the question
More informationPseudoinverse & MoorePenrose Conditions
ECE 275AB Lecture 7 Fall 2008 V1.0 c K. KreutzDelgado, UC San Diego p. 1/1 Lecture 7 ECE 275A Pseudoinverse & MoorePenrose Conditions ECE 275AB Lecture 7 Fall 2008 V1.0 c K. KreutzDelgado, UC San Diego
More informationLecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set.
Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. Definition A set of k vectors {u, u 2,..., u k }, where each u i R n, is said to be an
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: Any book on linear algebra! [HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More informationReview and problem list for Applied Math I
Review and problem list for Applied Math I (This is a first version of a serious review sheet; it may contain errors and it certainly omits a number of topic which were covered in the course. Let me know
More informationICS 6N Computational Linear Algebra Symmetric Matrices and Orthogonal Diagonalization
ICS 6N Computational Linear Algebra Symmetric Matrices and Orthogonal Diagonalization Xiaohui Xie University of California, Irvine xhx@uci.edu Xiaohui Xie (UCI) ICS 6N 1 / 21 Symmetric matrices An n n
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (nonzero) vector
More informationwhich arises when we compute the orthogonal projection of a vector y in a subspace with an orthogonal basis. Hence assume that P y = A ij = x j, x i
MODULE 6 Topics: GramSchmidt orthogonalization process We begin by observing that if the vectors {x j } N are mutually orthogonal in an inner product space V then they are necessarily linearly independent.
More informationLecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε,
2. REVIEW OF LINEAR ALGEBRA 1 Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, where Y n 1 response vector and X n p is the model matrix (or design matrix ) with one row for
More informationL3: Review of linear algebra and MATLAB
L3: Review of linear algebra and MATLAB Vector and matrix notation Vectors Matrices Vector spaces Linear transformations Eigenvalues and eigenvectors MATLAB primer CSCE 666 Pattern Analysis Ricardo GutierrezOsuna
More informationKernel Method: Data Analysis with Positive Definite Kernels
Kernel Method: Data Analysis with Positive Definite Kernels 2. Positive Definite Kernel and Reproducing Kernel Hilbert Space Kenji Fukumizu The Institute of Statistical Mathematics. Graduate University
More information10725/36725: Convex Optimization Prerequisite Topics
10725/36725: Convex Optimization Prerequisite Topics February 3, 2015 This is meant to be a brief, informal refresher of some topics that will form building blocks in this course. The content of the
More informationLinear Algebra Final Exam Study Guide Solutions Fall 2012
. Let A = Given that v = 7 7 67 5 75 78 Linear Algebra Final Exam Study Guide Solutions Fall 5 explain why it is not possible to diagonalize A. is an eigenvector for A and λ = is an eigenvalue for A diagonalize
More informationNumerical Methods I Singular Value Decomposition
Numerical Methods I Singular Value Decomposition Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATHGA 2011.003 / CSCIGA 2945.003, Fall 2014 October 9th, 2014 A. Donev (Courant Institute)
More informationLecture Note 1: Background
ECE5463: Introduction to Robotics Lecture Note 1: Background Prof. Wei Zhang Department of Electrical and Computer Engineering Ohio State University Columbus, Ohio, USA Spring 2018 Lecture 1 (ECE5463 Sp18)
More informationNormed & Inner Product Vector Spaces
Normed & Inner Product Vector Spaces ECE 174 Introduction to Linear & Nonlinear Optimization Ken KreutzDelgado ECE Department, UC San Diego Ken KreutzDelgado (UC San Diego) ECE 174 Fall 2016 1 / 27 Normed
More informationComputational math: Assignment 1
Computational math: Assignment 1 Thanks Ting Gao for her Latex file 11 Let B be a 4 4 matrix to which we apply the following operations: 1double column 1, halve row 3, 3add row 3 to row 1, 4interchange
More informationChapter Two Elements of Linear Algebra
Chapter Two Elements of Linear Algebra Previously, in chapter one, we have considered single first order differential equations involving a single unknown function. In the next chapter we will begin to
More informationLecture 10: Vector Algebra: Orthogonal Basis
Lecture 0: Vector Algebra: Orthogonal Basis Orthogonal Basis of a subspace Computing an orthogonal basis for a subspace using GramSchmidt Orthogonalization Process Orthogonal Set Any set of vectors that
More informationSupplementary Notes on Linear Algebra
Supplementary Notes on Linear Algebra Mariusz Wodzicki May 3, 2015 1 Vector spaces 1.1 Coordinatization of a vector space 1.1.1 Given a basis B = {b 1,..., b n } in a vector space V, any vector v V can
More informationSignal Analysis. Principal Component Analysis
Multi dimensional Signal Analysis Lecture 2E Principal Component Analysis Subspace representation Note! Given avector space V of dimension N a scalar product defined by G 0 a subspace U of dimension M
More informationMath 18, Linear Algebra, Lecture C00, Spring 2017 Review and Practice Problems for Final Exam
Math 8, Linear Algebra, Lecture C, Spring 7 Review and Practice Problems for Final Exam. The augmentedmatrix of a linear system has been transformed by row operations into 5 4 8. Determine if the system
More informationMath Linear Algebra Final Exam Review Sheet
Math 151 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a componentwise operation. Two vectors v and w may be added together as long as they contain the same number n of
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationLecture 7 Spectral methods
CSE 291: Unsupervised learning Spring 2008 Lecture 7 Spectral methods 7.1 Linear algebra review 7.1.1 Eigenvalues and eigenvectors Definition 1. A d d matrix M has eigenvalue λ if there is a ddimensional
More informationLecture 23: 6.1 Inner Products
Lecture 23: 6.1 Inner Products WeiTa Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationSolutions to Final Practice Problems Written by Victoria Kala Last updated 12/5/2015
Solutions to Final Practice Problems Written by Victoria Kala vtkala@math.ucsb.edu Last updated /5/05 Answers This page contains answers only. See the following pages for detailed solutions. (. (a x. See
More informationThe Gram Schmidt Process
u 2 u The Gram Schmidt Process Now we will present a procedure, based on orthogonal projection, that converts any linearly independent set of vectors into an orthogonal set. Let us begin with the simple
More informationThe Gram Schmidt Process
The Gram Schmidt Process Now we will present a procedure, based on orthogonal projection, that converts any linearly independent set of vectors into an orthogonal set. Let us begin with the simple case
More information1. Select the unique answer (choice) for each problem. Write only the answer.
MATH 5 Practice Problem Set Spring 7. Select the unique answer (choice) for each problem. Write only the answer. () Determine all the values of a for which the system has infinitely many solutions: x +
More informationProblem Set 1. Homeworks will graded based on content and clarity. Please show your work clearly for full credit.
CSE 151: Introduction to Machine Learning Winter 2017 Problem Set 1 Instructor: Kamalika Chaudhuri Due on: Jan 28 Instructions This is a 40 point homework Homeworks will graded based on content and clarity
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2
EE/ACM 150  Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko
More informationFunctional Analysis Review
Functional Analysis Review Lorenzo Rosasco slides courtesy of Andre Wibisono 9.520: Statistical Learning Theory and Applications September 9, 2013 1 2 3 4 Vector Space A vector space is a set V with binary
More informationCS168: The Modern Algorithmic Toolbox Lecture #8: How PCA Works
CS68: The Modern Algorithmic Toolbox Lecture #8: How PCA Works Tim Roughgarden & Gregory Valiant April 20, 206 Introduction Last lecture introduced the idea of principal components analysis (PCA). The
More informationAgenda: Understand the action of A by seeing how it acts on eigenvectors.
Eigenvalues and Eigenvectors If Av=λv with v nonzero, then λ is called an eigenvalue of A and v is called an eigenvector of A corresponding to eigenvalue λ. Agenda: Understand the action of A by seeing
More informationALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA
ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA Kent State University Department of Mathematical Sciences Compiled and Maintained by Donald L. White Version: August 29, 2017 CONTENTS LINEAR ALGEBRA AND
More informationDimensionality Reduction and Principal Components
Dimensionality Reduction and Principal Components Nuno Vasconcelos (Ken KreutzDelgado) UCSD Motivation Recall, in Bayesian decision theory we have: World: States Y in {1,..., M} and observations of X
More informationMATH 304 Linear Algebra Lecture 10: Linear independence. Wronskian.
MATH 304 Linear Algebra Lecture 10: Linear independence. Wronskian. Spanning set Let S be a subset of a vector space V. Definition. The span of the set S is the smallest subspace W V that contains S. If
More informationOrthogonality. Orthonormal Bases, Orthogonal Matrices. Orthogonality
Orthonormal Bases, Orthogonal Matrices The Major Ideas from Last Lecture Vector Span Subspace Basis Vectors Coordinates in different bases Matrix Factorization (Basics) The Major Ideas from Last Lecture
More informationProposition 42. Let M be an m n matrix. Then (32) N (M M)=N (M) (33) R(MM )=R(M)
RODICA D. COSTIN. Singular Value Decomposition.1. Rectangular matrices. For rectangular matrices M the notions of eigenvalue/vector cannot be defined. However, the products MM and/or M M (which are square,
More informationv = v 1 2 +v 2 2. Two successive applications of this idea give the length of the vector v R 3 :
Length, Angle and the Inner Product The length (or norm) of a vector v R 2 (viewed as connecting the origin to a point (v 1,v 2 )) is easily determined by the Pythagorean Theorem and is denoted v : v =
More informationBASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x
BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,
More information5. Orthogonal matrices
L Vandenberghe EE133A (Spring 2017) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 51 Orthonormal
More informationHomework 2. Solutions T =
Homework. s Let {e x, e y, e z } be an orthonormal basis in E. Consider the following ordered triples: a) {e x, e x + e y, 5e z }, b) {e y, e x, 5e z }, c) {e y, e x, e z }, d) {e y, e x, 5e z }, e) {
More informationVAR Model. (kvariate) VAR(p) model (in the Reduced Form): Y t2. Y t1 = A + B 1. Y t + B 2. Y tp. + ε t. + + B p. where:
VAR Model (kvariate VAR(p model (in the Reduced Form: where: Y t = A + B 1 Y t1 + B 2 Y t2 + + B p Y tp + ε t Y t = (y 1t, y 2t,, y kt : a (k x 1 vector of time series variables A: a (k x 1 vector
More informationAN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES
AN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES JOEL A. TROPP Abstract. We present an elementary proof that the spectral radius of a matrix A may be obtained using the formula ρ(a) lim
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2
MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2 1 Ridge Regression Ridge regression and the Lasso are two forms of regularized
More informationMath 224, Fall 2007 Exam 3 Thursday, December 6, 2007
Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007 You have 1 hour and 20 minutes. No notes, books, or other references. You are permitted to use Maple during this exam, but you must start with a blank
More informationFace Recognition and Biometric Systems
The Eigenfaces method Plan of the lecture Principal Components Analysis main idea Feature extraction by PCA face recognition Eigenfaces training feature extraction Literature M.A.Turk, A.P.Pentland Face
More informationHOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION)
HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) PROFESSOR STEVEN MILLER: BROWN UNIVERSITY: SPRING 2007 1. CHAPTER 1: MATRICES AND GAUSSIAN ELIMINATION Page 9, # 3: Describe
More informationInformation About Ellipses
Information About Ellipses David Eberly, Geometric Tools, Redmond WA 9805 https://www.geometrictools.com/ This work is licensed under the Creative Commons Attribution 4.0 International License. To view
More informationLecture 8: Signal Detection and Noise Assumption
ECE 830 Fall 0 Statistical Signal Processing instructor: R. Nowak Lecture 8: Signal Detection and Noise Assumption Signal Detection : X = W H : X = S + W where W N(0, σ I n n and S = [s, s,..., s n ] T
More informationElements of Positive Definite Kernel and Reproducing Kernel Hilbert Space
Elements of Positive Definite Kernel and Reproducing Kernel Hilbert Space Statistical Inference with Reproducing Kernel Hilbert Space Kenji Fukumizu Institute of Statistical Mathematics, ROIS Department
More informationPrincipal Components Analysis (PCA) and Singular Value Decomposition (SVD) with applications to Microarrays
Principal Components Analysis (PCA) and Singular Value Decomposition (SVD) with applications to Microarrays Prof. Tesler Math 283 Fall 2015 Prof. Tesler Principal Components Analysis Math 283 / Fall 2015
More informationLecture Note 1: Background
ECE5463: Introduction to Robotics Lecture Note 1: Background Prof. Wei Zhang Department of Electrical and Computer Engineering Ohio State University Columbus, Ohio, USA Spring 2018 Lecture 1 (ECE5463 Sp18)
More informationLinear algebra 2. Yoav Zemel. March 1, 2012
Linear algebra 2 Yoav Zemel March 1, 2012 These notes were written by Yoav Zemel. The lecturer, Shmuel Berger, should not be held responsible for any mistake. Any comments are welcome at zamsh7@gmail.com.
More information1. Elements of linear algebra
Elements of linear algebra Contents Solving systems of linear equations 2 Diagonal form of a square matrix 3 The Jordan normal form of a square matrix 4 The GramSchmidt orthogonalization process 5 The
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationPCA, Kernel PCA, ICA
PCA, Kernel PCA, ICA Learning Representations. Dimensionality Reduction. MariaFlorina Balcan 04/08/2015 Big & HighDimensional Data HighDimensions = Lot of Features Document classification Features per
More informationPreliminary Linear Algebra 1. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 100
Preliminary Linear Algebra 1 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics 611 1 / 100 Notation for all there exists such that therefore because end of proof (QED) Copyright c 2012
More informationPolar Form of a Complex Number (I)
Polar Form of a Complex Number (I) The polar form of a complex number z = x + iy is given by z = r (cos θ + i sin θ) where θ is the angle from the real axes to the vector z and r is the modulus of z, i.e.
More informationPh.D. Katarína Bellová Page 1 Mathematics 2 (10PHYBIPMA2) EXAM  Solutions, 20 July 2017, 10:00 12:00 All answers to be justified.
PhD Katarína Bellová Page 1 Mathematics 2 (10PHYBIPMA2 EXAM  Solutions, 20 July 2017, 10:00 12:00 All answers to be justified Problem 1 [ points]: For which parameters λ R does the following system
More information1. The Polar Decomposition
A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION MATAN GAVISH Part. Theory. The Polar Decomposition In what follows, F denotes either R or C. The vector space F n is an inner product space with
More informationIn English, this means that if we travel on a straight line between any two points in C, then we never leave C.
Convex sets In this section, we will be introduced to some of the mathematical fundamentals of convex sets. In order to motivate some of the definitions, we will look at the closest point problem from
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra PerOlof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More information