# EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2

Save this PDF as:

Size: px
Start display at page:

Download "EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2"

## Transcription

1 EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

2 Outline 1 Linear Algebra / Matrix Analysis Notation & Definitions 2 Basic Vector Space Results 3 Overview of Matrix Analysis Concepts 4 Special Types of Matrices 5 Inner Products & Norms Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

3 Linear Algebra / Matrix Analysis Notation & Definitions Terminology List Common fields and sets R field of real scalars C field of complex scalars F general field (will be either R or C here) R n, C n, F n set of n 1 vectors over R, C, or F, respectively R m n, C m n, F m n set of m n matrices over R, C, or F, respectively Special vector/matrix sets R n +, R n ++ set of n 1 real vectors whose components are nonnegative or positive, respectively S n set of n n real symmetric matrices H n set of n n Hermitian matrices S n +, S n ++ set of n n real symmetric positive semidefinite or positive definite matrices, respectively H n +, H n ++ set of n n Hermitian positive semidefinite or positive definite matrices, respectively Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

4 Linear Algebra / Matrix Analysis Notation & Definitions Terminology List (Continued) Vector space quantities dim(v) dimension of vector space V R(A) range space of matrix A N (A) null space of matrix A rank(a) rank of matrix A (i.e., dim(r(a))) nullity(a) nullity of matrix A (i.e., dim(n (A))) Common matrix operators and quantities complex conjugate operator (i.e., a or A ) T transpose operator (i.e., a T or A T ) complex conjugate transpose operator (i.e., a or A ) det(a) determinant of square matrix A A 1 inverse of square matrix A (if it exists) diag(a) column vector formed from diagonal components of matrix A diag(a) diagonal matrix formed from components of vector a tr(a) trace of matrix A (i.e., the sum of diagonal components) Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

5 Linear Algebra / Matrix Analysis Notation & Definitions Terminology List (Continued) Inner product / norm quantities x, y inner product of x and y (x and y can be either vectors or matrices) a, A norm of vector a or matrix A, respectively a p l p-norm of vector a A a,b operator norm of A induced by vector norms a and b A F Frobenius norm of matrix A z dual norm of z associated with z (z can be either a vector or a matrix) Useful vectors & matrices and miscellaneous terminology 1 column vector of all ones 0 m n m n matrix of zeros I n n n identity matrix [a] k k-th element of vector a [A] k,l (k, l)-th element of matrix A Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

6 Basic Vector Space Results Linear Dependence and Independence A set of n vectors {v 1, v 2,..., v n } from a vector space V is said to be linearly dependent if and only if there are n scalars a 1, a 2,..., a n not all zero such that n a k v k = 0. In other words, there is at least one vector, say v l, which depends linearly on the other vectors, i.e., v l = 1 a l n k l a k v k. The vectors v 1, v 2,..., v n are said to be linearly independent if they are not linearly dependent. Equivalently, the set {v 1, v 2,..., v n } is linearly independent when n a k v k = 0, if and only if a 1 = a 2 = = a n = 0. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

7 Basic Vector Space Results Span, Basis, & Dimension Let S {v 1,..., v n } denote a subset of vectors from a vector space V defined over F. The span of S (denoted span(s)) is the set of all linear combinations of elements in S, i.e., span(s) {c 1 v c n v n : c 1,..., c n F}. If S is a linearly independent set of vectors which spans V (meaning span(s) = V), then S is said to be a basis for V. Any vector w V can be represented in terms of a basis S in one and only one way. As such, a basis effectively defines a coordinate system for V. A vector space V can be characterized by several different bases (bases are nonunique). However, all bases for a given vector space will have the same number of elements. This common number is call the dimension of the vector space V and is denoted dim(v). Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

8 Basic Vector Space Results Range & Null Space, Rank & Nullity Let A F m n. Then, the range space R(A) and null space N (A) are defined as follows. R(A) {y = Ax : x F n } N (A) {x F n : Ax = 0}. The rank and nullity of A are the dimensions of the range and null spaces, respectively, i.e., rank(a) dim(r(a)) nullity(a) dim(n (A)). Equality of row and column ranks: ( ) rank(a) = rank A T. Rank-nullity theorem: rank(a) + nullity(a) = n. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

9 Basic Vector Space Results Systems of Linear Equations Let A F m n, x F n, and b F m. Consider the following system of linear equations which we would like to solve for x. Ax = b. This system can behave in any one of three possible ways: 1 It can have a single unique solution. (b R(A), nullity(a) = 0) 2 It can have infinitely many solutions. (b R(A), nullity(a) > 0) 3 It can have no solution. (b / R(A)) If a solution exists (i.e., b R(A)), then the set of solutions can be characterized as follows. Let p be a particular solution to Ax = b, i.e., Ap = b. Then, the solution set is given by {p + v : Av = 0}, that is, the solution set is a translation of the solution set of the homogeneous system Ax = 0. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

10 Overview of Matrix Analysis Concepts Matrix Arithmetic Operations Addition and scalar multiplication: If α, β F, A F m n, and B F m n, then C αa + βb is such that C F m n with [C] k,l = α [A] k,l + β [B] k,l. Matrix multiplication: If A F m n and B F n p, then C AB is such that C F m p with n [C] k,l = [A] k,i [B] i,l, 1 k m, 1 l p. i=1 Block matrix multiplication: If A F m 1 n 1, B F m 1 n 2, C F m 2 n 1, D F m 2 n 2, E F n 1 p 1, F F n 1 p 2, G F n 2 p 1, and H F n 2 p 2, then we have [ ] [ ] [ ] A B E F AE + BG AF + BH =. C D G H CE + DG CF + DH Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

11 Overview of Matrix Analysis Concepts Determinant of a Square Matrix The determinant of an n n square matrix A, denoted det(a), is a scalar quantity used to help construct the inverse of A (if it exists), calculate the eigenvalues of A, and determine the volume of the parallelepiped spanned by the columns of A (via its absolute value). It can be determined recursively as n n det(a) = ( 1) k+l [A] k,l M k,l = ( 1) k+l [A] l,k M l,k, where l is a fixed integer in 1 l n and M k,l is the minor of [A] k,l, which is the determinant of the (n 1) (n 1) submatrix formed by deleting the k-th row and m-th column of A. (The determinant of a scalar is the scalar itself.) Properties: If A and B are n n and C AB, then det(c) = det(a) det(b). If A, B, C, and D are m m, m n, n m, and n n, respectively, then ([ ]) ([ ]) A 0m n A B det = det = det(a) det(d). C D 0 n m D If A is an n n triangular matrix (either upper or lower), then n det(a) = [A] k,l. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

12 Matrix Inverse Overview of Matrix Analysis Concepts The inverse of an n n square matrix A, denoted A 1, is one for which AA 1 = A 1 A = I n. If such a matrix exists, A is said to be invertible. Otherwise, A is said to be singular. It can be shown that A is invertible if and only if det(a) 0. In this case, [ A 1 ] k,l = 1 det(a) ( 1)l+k M l,k, 1 k n, 1 l n, where M l,k is the minor of [A] l,k. The quantity C l,k ( 1) l+k M l,k is the cofactor of [A] l,k. Matrix Inversion Lemma: If A, B, C, and D are m m, m n, n m, and n n, respectively, and A and D are nonsingular, then (A BD 1 C ) 1 = A 1 + A 1 B ( D CA 1 B ) 1 CA 1. This lemma can be proved by considering the inverse of the block matrix M given by [ ] A B M. C D In this case, the matrix S D;M A BD 1 C is called the Schur complement of D in M. Similarly, S A;M D CA 1 B is the Schur complement of A in M. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

13 Overview of Matrix Analysis Concepts Eigenvalues & Eigenvectors For an n n square matrix A, an n 1 nonzero vector v such that Av = λv is said to be an eigenvector of A with eigenvalue λ. The eigenvalues of A can be obtained as the roots of its characteristic polynomial p(λ) given by p(λ) det(λi n A). Note that there are exactly n eigenvalues (counting multiplicity). Properties of eigenvalues & eigenvectors: If {λ 1,..., λ n} denote the set of eigenvalues of A, then it can be shown that det(a) = n λ k, tr(a) = n λ k. The eigenvalues of a triangular matrix (either upper or lower) are the diagonal elements. Suppose A is n n with n linearly independent eigenvectors v 1,..., v n corresponding to eigenvalues λ 1,..., λ n. If V [ v 1 v n ] and Λ diag ( [ λ1 λ n ] T ), then we have A = VΛV 1, and we say that A is diagonalizable. If the eigenvalues of a matrix A are distinct, then A is diagonalizable. Otherwise, A may or may not be diagonalizable. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

14 Overview of Matrix Analysis Concepts Useful Miscellaneous Identities Conjugate, transpose, and conjugate transpose: Suppose that α and β are scalars, that U is k l, V is k l, X is m n, and Y is n p, and that A is m 1 n 1, B is m 1 n 2, C is m 2 n 1, and D is m 2 n 2. Then we have [ ] A B [ A (αu + βv) = α U + β V, (XY) = X Y B ], = C D C D. (αu + βv) T = αu T + βv T, (XY) T = Y T X T, (αu + βv) = α U + β V, (XY) = Y X, [ A B C D [ A B C D ] T = ] = [ A T B T [ A C T D T C B D Trace: Suppose that α and β are scalars, that U is k k, V is k k, X is m n, and Y is n m, and that A is m m, B is m n, C is n m, and D is n n. Then we have ([ ]) A B tr(αu + βv) = αtr(u) + βtr(v), tr(xy) = tr(yx), tr = tr(a) + tr(d). C D Inverse: Suppose that c is a nonzero scalar and that A and B are invertible n n matrices. Then we have (ca) 1 = 1 c A 1, ( A 1) 1 = A, (AB) 1 = B 1 A 1. (A T ) 1 = (A 1) T, (A ) 1 = (A 1) (, det A 1) 1 = det(a). ] ].. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

15 Special Types of Matrices Unitary and Normal Matrices Unitary Matrices: A matrix U C m n (with m n) is said to be unitary if U U = I n. Similarly, if U R m n, then U is unitary if U T U = I n. If m = n, then we have U U = UU = I m, (U C m m ). U T U = UU T = I m, (U R m m ). Normal Matrices: A matrix A C n n is said to be normal if A A = AA. Similarly, if A R n n, then A is normal if A T A = AA T. It can be shown that A is normal if and only if it is diagonalizable by a unitary matrix. More specifically, A is diagonalizable if and only if A = UΛU, (A C n n ). A = UΛU T, (A R n n ). Here, U is unitary and Λ = diag(λ 1,..., λ n ) is a diagonal matrix of eigenvalues of A. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

16 Special Types of Matrices Symmetric and Hermitian Matrices Symmetric matrices: An n n matrix A is said to be symmetric if A = A T. The set of n n real symmetric matrices is a special one and will be denoted S n. Any real symmetric matrix A is normal and as such, is unitarily diagonalizable, i.e., admits a decomposition of the form A = UΛU T as described above. Hermitian matrices: An n n matrix A is said to be Hermitian if A = A. The set of n n Hermitian matrices is a special one and will be denoted H n. As any real symmetric matrix is also Hermitian (yet there are Hermitian matrices which are not real symmetric), it follows that S n H n. Any Hermitian matrix is normal and as such, is unitarily diagonalizable, i.e., admits a decomposition of the form A = UΛU as described above. Properties of real symmetric and Hermitian matrices: The eigenvalues of any real symmetric or any Hermitian matrix are always real. If A S n, v R n, B H n, and w C n, then we have v T Av R, w Bw R. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

17 Special Types of Matrices Positive Semidefinite / Positive Definite Matrices Positive semidefinite matrices: An n n real symmetric matrix A is said to be positive semidefinite if v T Av 0 for all v R n. Similarly, an n n Hermitian matrix B is said to be positive semidefinite if w Bw 0 for all w C n. In either case, we will write A 0 and B 0. The sets of positive semidefinite real symmetric and Hermitian matrices are special ones and will be denoted S n + and H n +, respectively. Positive definite matrices: An n n real symmetric matrix A is said to be positive definite if v T Av > 0 for all v R n, v 0. Similarly, an n n Hermitian matrix B is said to be positive definite if w Bw > 0 for all w C n, w 0. In either case, we will write A 0 and B 0. The sets of positive semidefinite real symmetric and Hermitian matrices are special ones and will be denoted S n ++ and H n ++, respectively. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

18 Special Types of Matrices Properties of Positive Semidefinite/Definite Matrices (Definiteness of Diagonal Elements and Eigenvalues:) The diagonal entries of a positive semidefinite (definite) matrix are always nonnegative (positive). A real symmetric or Hermitian matrix is positive semidefinite (definite) if and only if all of the eigenvalues are nonnegative (positive). (Partial Ordering:) For arbitrary square matrices A and B, we will write A B if (A B) 0 and write A B if (A B) 0. This defines a partial ordering on the set of all square matrices. (Matrix Square Roots:) If A S n +, ρ A = rank(a), B H n +, and ρ B = rank(b), then there exists a P R ρ A n and Q C ρ B n such that A = P T P, B = Q Q. (Cholesky Decomposition:) If A S n ++ and B H n ++, then there exist lower triangular matrices L A R n n and L B C n n with strictly positive diagonal entries such that A = L A L T A, B = L B L B. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

19 Inner Products & Norms Inner Products One way to measure the correlation or coherence between two vectors or matrices is through the use of an inner product. An inner product x, y maps two vectors or matrices x and y (defined over a field F) to the underlying field F and satisfies the following properties. Conjugate symmetry: x, y = y, x. Linearity in the first argument: Positive definiteness: αx + βy, z = α x, z + β y, z for all α, β F. x, x 0 with equality if and only if x = 0. Common inner products: y T x, for x, y R n (standard inner product on R n ) x, y = y x, for x, y C n (standard inner product on C n ). y Px, for x, y C n, P H n ++ tr ( Y T X ), for X, Y R m n (standard inner product on R m n ) X, Y = tr ( Y X ), for X, Y C m n (standard inner product on C m n ) tr ( Y PX ), for X, Y C m n, P H n ++. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

20 Inner Products & Norms Norms: Definition and Examples of Vector Norms One way to measure the length of a vector or matrix in some sense is through the use of a norm. A norm x maps a vector or matrix x (defined over a field F) to R + and satisfies the following. Positive definiteness: Homogeneity: Triangle inequality: x 0 with equality if and only if x = 0. αx = α x for all α F. x + y x + y for all x, y. Common vector norms: For all examples considered here, we assume x F n and x k = [x] k. Euclidean norm: (l 2 -norm) x 2 = n x k 2 = x x. Chebyshev norm: (l -norm) x = max { x 1,..., x n }. l p-norm: ( n ) 1 p x p = x k p (valid only for p 1). Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

21 Inner Products & Norms Matrix Norms: Operator and Entrywise Norms There are conventionally three varieties of matrix norms: operator norms, entrywise norms, and Schatten norms (which will be discussed at another time). Operator Norms: If a and b are vector norms defined over F m and F n, respectively, then the operator norm of X F m n, induced by a and b, is defined as X a,b sup { Xu a : u b 1 }. Example: When both a and b are the Euclidean norm, the resulting operator norm is called the spectral norm or the l 2 -norm and is given by ( X 2 = λ max X X ), ( where λ max X X ) denotes the maximum eigenvalue of X X. Entrywise Norms: If we treat the matrix X F m n as a vector of size mn and apply a familiar vector norm, we obtain an entrywise norm. Assuming X k,l = [X] k,l, some examples are as follows. X F = m n Xk,l 2 = tr ( X X ) (Frobenius norm). l=1 ( 1 m n p X p = Xk,l p) (l p-norm). l=1 Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

22 The Dual Norm Inner Products & Norms The concept of duality occurs frequently throughout the study of convex optimization. One way in which duality manifests itself is through the dual norm. If z is some norm for either a vector or matrix z, the associated dual norm z is defined as follows. z sup{re[ x, z ] : x 1}. The dual norm can be expressed in the following equivalent forms, which are more convenient for analysis. { x z = sup{ x, z : x = 1} = sup x, z : x 0}. Properties: The dual norm z is indeed a norm. The primal norm and dual norm satisfy the inequality x y x, y = y, x for all x, y. The dual of the dual norm, denoted z, is the original norm z, i.e., z = z. Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

23 Inner Products & Norms Examples of Dual Norms To calculate the dual norm z, typically an upper bound on x, z is computed and x is chosen so as to achieve the upper bound, if possible. Using this approach leads to the following examples of dual norms. Primal Norm: Dual Norm: Euclidean norm n z k 2 Euclidean norm n z k 2 l 1 -norm l p -norm n z k l -norm max{ z 1,..., z n } ( n z k p ) 1 p l q -norm ( n z k q ) 1 q spectral norm nuclear norm ) Z 2 = λ max (Z Z) Z 2 = tr( Z Z (q = p/(p 1)) Andre Tkacenko (JPL) EE/ACM Lecture 2 April 5, / 23

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 4

EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 4 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 12, 2012 Andre Tkacenko

### 1. General Vector Spaces

1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### MA 265 FINAL EXAM Fall 2012

MA 265 FINAL EXAM Fall 22 NAME: INSTRUCTOR S NAME:. There are a total of 25 problems. You should show work on the exam sheet, and pencil in the correct answer on the scantron. 2. No books, notes, or calculators

### Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )

Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

### c c c c c c c c c c a 3x3 matrix C= has a determinant determined by

Linear Algebra Determinants and Eigenvalues Introduction: Many important geometric and algebraic properties of square matrices are associated with a single real number revealed by what s known as the determinant.

### Linear Algebra in Actuarial Science: Slides to the lecture

Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### and let s calculate the image of some vectors under the transformation T.

Chapter 5 Eigenvalues and Eigenvectors 5. Eigenvalues and Eigenvectors Let T : R n R n be a linear transformation. Then T can be represented by a matrix (the standard matrix), and we can write T ( v) =

### Recall the convention that, for us, all vectors are column vectors.

Some linear algebra Recall the convention that, for us, all vectors are column vectors. 1. Symmetric matrices Let A be a real matrix. Recall that a complex number λ is an eigenvalue of A if there exists

### IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET

IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

### Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε,

2. REVIEW OF LINEAR ALGEBRA 1 Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, where Y n 1 response vector and X n p is the model matrix (or design matrix ) with one row for

### ft-uiowa-math2550 Assignment OptionalFinalExamReviewMultChoiceMEDIUMlengthForm due 12/31/2014 at 10:36pm CST

me me ft-uiowa-math255 Assignment OptionalFinalExamReviewMultChoiceMEDIUMlengthForm due 2/3/2 at :3pm CST. ( pt) Library/TCNJ/TCNJ LinearSystems/problem3.pg Give a geometric description of the following

### Chapters 5 & 6: Theory Review: Solutions Math 308 F Spring 2015

Chapters 5 & 6: Theory Review: Solutions Math 308 F Spring 205. If A is a 3 3 triangular matrix, explain why det(a) is equal to the product of entries on the diagonal. If A is a lower triangular or diagonal

### Linear Algebra & Analysis Review UW EE/AA/ME 578 Convex Optimization

Linear Algebra & Analysis Review UW EE/AA/ME 578 Convex Optimization January 9, 2015 1 Notation 1. Book pages without explicit citation refer to [1]. 2. R n denotes the set of real n-column vectors. 3.

### Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET

IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

### Review of some mathematical tools

MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical

### ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA

ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA Kent State University Department of Mathematical Sciences Compiled and Maintained by Donald L. White Version: August 29, 2017 CONTENTS LINEAR ALGEBRA AND

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and

Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### Preliminary Linear Algebra 1. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 100

Preliminary Linear Algebra 1 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics 611 1 / 100 Notation for all there exists such that therefore because end of proof (QED) Copyright c 2012

### Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

### MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

### 1. Select the unique answer (choice) for each problem. Write only the answer.

MATH 5 Practice Problem Set Spring 7. Select the unique answer (choice) for each problem. Write only the answer. () Determine all the values of a for which the system has infinitely many solutions: x +

### Quiz ) Locate your 1 st order neighbors. 1) Simplify. Name Hometown. Name Hometown. Name Hometown.

Quiz 1) Simplify 9999 999 9999 998 9999 998 2) Locate your 1 st order neighbors Name Hometown Me Name Hometown Name Hometown Name Hometown Solving Linear Algebraic Equa3ons Basic Concepts Here only real

### I. Multiple Choice Questions (Answer any eight)

Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY

### Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors week -2 Fall 26 Eigenvalues and eigenvectors The most simple linear transformation from R n to R n may be the transformation of the form: T (x,,, x n ) (λ x, λ 2,, λ n x n

### Matrices A brief introduction

Matrices A brief introduction Basilio Bona DAUIN Politecnico di Torino September 2013 Basilio Bona (DAUIN) Matrices September 2013 1 / 74 Definitions Definition A matrix is a set of N real or complex numbers

### Basic Concepts in Linear Algebra

Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear

### Linear algebra and applications to graphs Part 1

Linear algebra and applications to graphs Part 1 Written up by Mikhail Belkin and Moon Duchin Instructor: Laszlo Babai June 17, 2001 1 Basic Linear Algebra Exercise 1.1 Let V and W be linear subspaces

### Notes on Linear Algebra and Matrix Analysis

Notes on Linear Algebra and Matrix Analysis Maxim Neumann May 2006, Version 0.1.1 1 Matrix Basics Literature to this topic: [1 4]. x y < y,x >: standard inner product. x x = 1 : x is normalized x y = 0

### Linear Algebra- Final Exam Review

Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.

### Chapter 2. General Vector Spaces. 2.1 Real Vector Spaces

Chapter 2 General Vector Spaces Outline : Real vector spaces Subspaces Linear independence Basis and dimension Row Space, Column Space, and Nullspace 2 Real Vector Spaces 2 Example () Let u and v be vectors

### EE263: Introduction to Linear Dynamical Systems Review Session 5

EE263: Introduction to Linear Dynamical Systems Review Session 5 Outline eigenvalues and eigenvectors diagonalization matrix exponential EE263 RS5 1 Eigenvalues and eigenvectors we say that λ C is an eigenvalue

### Linear Algebra. Workbook

Linear Algebra Workbook Paul Yiu Department of Mathematics Florida Atlantic University Last Update: November 21 Student: Fall 2011 Checklist Name: A B C D E F F G H I J 1 2 3 4 5 6 7 8 9 10 xxx xxx xxx

### Linear Algebra Primer

Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................

### Solutions to Final Practice Problems Written by Victoria Kala Last updated 12/5/2015

Solutions to Final Practice Problems Written by Victoria Kala vtkala@math.ucsb.edu Last updated /5/05 Answers This page contains answers only. See the following pages for detailed solutions. (. (a x. See

### Math 18, Linear Algebra, Lecture C00, Spring 2017 Review and Practice Problems for Final Exam

Math 8, Linear Algebra, Lecture C, Spring 7 Review and Practice Problems for Final Exam. The augmentedmatrix of a linear system has been transformed by row operations into 5 4 8. Determine if the system

### Mathematical Foundations

Chapter 1 Mathematical Foundations 1.1 Big-O Notations In the description of algorithmic complexity, we often have to use the order notations, often in terms of big O and small o. Loosely speaking, for

### MATRICES The numbers or letters in any given matrix are called its entries or elements

MATRICES A matrix is defined as a rectangular array of numbers. Examples are: 1 2 4 a b 1 4 5 A : B : C 0 1 3 c b 1 6 2 2 5 8 The numbers or letters in any given matrix are called its entries or elements

### 1 9/5 Matrices, vectors, and their applications

1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric

### ANSWERS. E k E 2 E 1 A = B

MATH 7- Final Exam Spring ANSWERS Essay Questions points Define an Elementary Matrix Display the fundamental matrix multiply equation which summarizes a sequence of swap, combination and multiply operations,

### Introduction to Matrix Algebra

Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

### ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS. 1. Linear Equations and Matrices

ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS KOLMAN & HILL NOTES BY OTTO MUTZBAUER 11 Systems of Linear Equations 1 Linear Equations and Matrices Numbers in our context are either real numbers or complex

### (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =

. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)

### The University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.

The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational

### Chapter Two Elements of Linear Algebra

Chapter Two Elements of Linear Algebra Previously, in chapter one, we have considered single first order differential equations involving a single unknown function. In the next chapter we will begin to

### SUMMARY OF MATH 1600

SUMMARY OF MATH 1600 Note: The following list is intended as a study guide for the final exam. It is a continuation of the study guide for the midterm. It does not claim to be a comprehensive list. You

### 18.06 Problem Set 8 - Solutions Due Wednesday, 14 November 2007 at 4 pm in

806 Problem Set 8 - Solutions Due Wednesday, 4 November 2007 at 4 pm in 2-06 08 03 Problem : 205+5+5+5 Consider the matrix A 02 07 a Check that A is a positive Markov matrix, and find its steady state

### MATRICES. knowledge on matrices Knowledge on matrix operations. Matrix as a tool of solving linear equations with two or three unknowns.

MATRICES After studying this chapter you will acquire the skills in knowledge on matrices Knowledge on matrix operations. Matrix as a tool of solving linear equations with two or three unknowns. List of

### Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007

Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007 You have 1 hour and 20 minutes. No notes, books, or other references. You are permitted to use Maple during this exam, but you must start with a blank

### Math 102 Final Exam - Dec 14 - PCYNH pm Fall Name Student No. Section A0

Math 12 Final Exam - Dec 14 - PCYNH 122-6pm Fall 212 Name Student No. Section A No aids allowed. Answer all questions on test paper. Total Marks: 4 8 questions (plus a 9th bonus question), 5 points per

### Computational math: Assignment 1

Computational math: Assignment 1 Thanks Ting Gao for her Latex file 11 Let B be a 4 4 matrix to which we apply the following operations: 1double column 1, halve row 3, 3add row 3 to row 1, 4interchange

### Generalized eigenvector - Wikipedia, the free encyclopedia

1 of 30 18/03/2013 20:00 Generalized eigenvector From Wikipedia, the free encyclopedia In linear algebra, for a matrix A, there may not always exist a full set of linearly independent eigenvectors that

### (a) II and III (b) I (c) I and III (d) I and II and III (e) None are true.

1 Which of the following statements is always true? I The null space of an m n matrix is a subspace of R m II If the set B = {v 1,, v n } spans a vector space V and dimv = n, then B is a basis for V III

### Extra Problems for Math 2050 Linear Algebra I

Extra Problems for Math 5 Linear Algebra I Find the vector AB and illustrate with a picture if A = (,) and B = (,4) Find B, given A = (,4) and [ AB = A = (,4) and [ AB = 8 If possible, express x = 7 as

### 18.06SC Final Exam Solutions

18.06SC Final Exam Solutions 1 (4+7=11 pts.) Suppose A is 3 by 4, and Ax = 0 has exactly 2 special solutions: 1 2 x 1 = 1 and x 2 = 1 1 0 0 1 (a) Remembering that A is 3 by 4, find its row reduced echelon

### BASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x

BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,

### Theorem A.1. If A is any nonzero m x n matrix, then A is equivalent to a partitioned matrix of the form. k k n-k. m-k k m-k n-k

I. REVIEW OF LINEAR ALGEBRA A. Equivalence Definition A1. If A and B are two m x n matrices, then A is equivalent to B if we can obtain B from A by a finite sequence of elementary row or elementary column

### Problems in Linear Algebra and Representation Theory

Problems in Linear Algebra and Representation Theory (Most of these were provided by Victor Ginzburg) The problems appearing below have varying level of difficulty. They are not listed in any specific

### Advanced Digital Signal Processing -Introduction

Advanced Digital Signal Processing -Introduction LECTURE-2 1 AP9211- ADVANCED DIGITAL SIGNAL PROCESSING UNIT I DISCRETE RANDOM SIGNAL PROCESSING Discrete Random Processes- Ensemble Averages, Stationary

### Lecture 23: 6.1 Inner Products

Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such

### MATH 323 Linear Algebra Lecture 12: Basis of a vector space (continued). Rank and nullity of a matrix.

MATH 323 Linear Algebra Lecture 12: Basis of a vector space (continued). Rank and nullity of a matrix. Basis Definition. Let V be a vector space. A linearly independent spanning set for V is called a basis.

### Functional Analysis Review

Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### SAMPLE OF THE STUDY MATERIAL PART OF CHAPTER 1 Introduction to Linear Algebra

1.1. Introduction SAMPLE OF THE STUDY MATERIAL PART OF CHAPTER 1 Introduction to Linear algebra is a specific branch of mathematics dealing with the study of vectors, vector spaces with functions that

### Chapter 4 & 5: Vector Spaces & Linear Transformations

Chapter 4 & 5: Vector Spaces & Linear Transformations Philip Gressman University of Pennsylvania Philip Gressman Math 240 002 2014C: Chapters 4 & 5 1 / 40 Objective The purpose of Chapter 4 is to think

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 1 x 2. x n 8 (4) 3 4 2

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS SYSTEMS OF EQUATIONS AND MATRICES Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Math 443 Differential Geometry Spring Handout 3: Bilinear and Quadratic Forms This handout should be read just before Chapter 4 of the textbook.

Math 443 Differential Geometry Spring 2013 Handout 3: Bilinear and Quadratic Forms This handout should be read just before Chapter 4 of the textbook. Endomorphisms of a Vector Space This handout discusses

### Linear algebra 2. Yoav Zemel. March 1, 2012

Linear algebra 2 Yoav Zemel March 1, 2012 These notes were written by Yoav Zemel. The lecturer, Shmuel Berger, should not be held responsible for any mistake. Any comments are welcome at zamsh7@gmail.com.

### Mathematical Methods wk 2: Linear Operators

John Magorrian, magog@thphysoxacuk These are work-in-progress notes for the second-year course on mathematical methods The most up-to-date version is available from http://www-thphysphysicsoxacuk/people/johnmagorrian/mm

### Properties of Matrices and Operations on Matrices

Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

### 4. Determinants.

4. Determinants 4.1. Determinants; Cofactor Expansion Determinants of 2 2 and 3 3 Matrices 2 2 determinant 4.1. Determinants; Cofactor Expansion Determinants of 2 2 and 3 3 Matrices 3 3 determinant 4.1.

### Eigenvalues and Eigenvectors

Chapter 1 Eigenvalues and Eigenvectors Among problems in numerical linear algebra, the determination of the eigenvalues and eigenvectors of matrices is second in importance only to the solution of linear

### Linear Algebra: Characteristic Value Problem

Linear Algebra: Characteristic Value Problem . The Characteristic Value Problem Let < be the set of real numbers and { be the set of complex numbers. Given an n n real matrix A; does there exist a number

### Optimization Theory. A Concise Introduction. Jiongmin Yong

October 11, 017 16:5 ws-book9x6 Book Title Optimization Theory 017-08-Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 017-08-Lecture Notes page Optimization

### Linear Algebra Practice Problems

Linear Algebra Practice Problems Math 24 Calculus III Summer 25, Session II. Determine whether the given set is a vector space. If not, give at least one axiom that is not satisfied. Unless otherwise stated,

### Algorithms to Compute Bases and the Rank of a Matrix

Algorithms to Compute Bases and the Rank of a Matrix Subspaces associated to a matrix Suppose that A is an m n matrix The row space of A is the subspace of R n spanned by the rows of A The column space

### Matrix Operations: Determinant

Matrix Operations: Determinant Determinants Determinants are only applicable for square matrices. Determinant of the square matrix A is denoted as: det(a) or A Recall that the absolute value of the determinant

### 1 Matrices and Systems of Linear Equations

March 3, 203 6-6. Systems of Linear Equations Matrices and Systems of Linear Equations An m n matrix is an array A = a ij of the form a a n a 2 a 2n... a m a mn where each a ij is a real or complex number.

### Math Linear Algebra Final Exam Review Sheet

Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of

### Chapter 7. Linear Algebra: Matrices, Vectors,

Chapter 7. Linear Algebra: Matrices, Vectors, Determinants. Linear Systems Linear algebra includes the theory and application of linear systems of equations, linear transformations, and eigenvalue problems.

### Since the determinant of a diagonal matrix is the product of its diagonal elements it is trivial to see that det(a) = α 2. = max. A 1 x.

APPM 4720/5720 Problem Set 2 Solutions This assignment is due at the start of class on Wednesday, February 9th. Minimal credit will be given for incomplete solutions or solutions that do not provide details

### Chapter 0 Miscellaneous Preliminaries

EE 520: Topics Compressed Sensing Linear Algebra Review Notes scribed by Kevin Palmowski, Spring 2013, for Namrata Vaswani s course Notes on matrix spark courtesy of Brian Lois More notes added by Namrata

### MATH 304 Linear Algebra Lecture 33: Bases of eigenvectors. Diagonalization.

MATH 304 Linear Algebra Lecture 33: Bases of eigenvectors. Diagonalization. Eigenvalues and eigenvectors of an operator Definition. Let V be a vector space and L : V V be a linear operator. A number λ

### SAMPLE OF THE STUDY MATERIAL PART OF CHAPTER 1 Introduction to Linear Algebra

SAMPLE OF THE STUDY MATERIAL PART OF CHAPTER 1 Introduction to 1.1. Introduction Linear algebra is a specific branch of mathematics dealing with the study of vectors, vector spaces with functions that

### NONCOMMUTATIVE POLYNOMIAL EQUATIONS. Edward S. Letzter. Introduction

NONCOMMUTATIVE POLYNOMIAL EQUATIONS Edward S Letzter Introduction My aim in these notes is twofold: First, to briefly review some linear algebra Second, to provide you with some new tools and techniques

### Numerical Methods I Solving Square Linear Systems: GEM and LU factorization

Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,

### Math 314/ Exam 2 Blue Exam Solutions December 4, 2008 Instructor: Dr. S. Cooper. Name:

Math 34/84 - Exam Blue Exam Solutions December 4, 8 Instructor: Dr. S. Cooper Name: Read each question carefully. Be sure to show all of your work and not just your final conclusion. You may not use your

### 5.6. PSEUDOINVERSES 101. A H w.

5.6. PSEUDOINVERSES 0 Corollary 5.6.4. If A is a matrix such that A H A is invertible, then the least-squares solution to Av = w is v = A H A ) A H w. The matrix A H A ) A H is the left inverse of A and

### Topic 2 Quiz 2. choice C implies B and B implies C. correct-choice C implies B, but B does not imply C

Topic 1 Quiz 1 text A reduced row-echelon form of a 3 by 4 matrix can have how many leading one s? choice must have 3 choice may have 1, 2, or 3 correct-choice may have 0, 1, 2, or 3 choice may have 0,

### Math 225 Linear Algebra II Lecture Notes. John C. Bowman University of Alberta Edmonton, Canada

Math 225 Linear Algebra II Lecture Notes John C Bowman University of Alberta Edmonton, Canada March 23, 2017 c 2010 John C Bowman ALL RIGHTS RESERVED Reproduction of these lecture notes in any form, in

### av 1 x 2 + 4y 2 + xy + 4z 2 = 16.

74 85 Eigenanalysis The subject of eigenanalysis seeks to find a coordinate system, in which the solution to an applied problem has a simple expression Therefore, eigenanalysis might be called the method

### Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

### Notes on Mathematics

Notes on Mathematics - 12 1 Peeyush Chandra, A. K. Lal, V. Raghavendra, G. Santhanam 1 Supported by a grant from MHRD 2 Contents I Linear Algebra 7 1 Matrices 9 1.1 Definition of a Matrix......................................

### HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION)

HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) PROFESSOR STEVEN MILLER: BROWN UNIVERSITY: SPRING 2007 1. CHAPTER 1: MATRICES AND GAUSSIAN ELIMINATION Page 9, # 3: Describe

### Math Linear Algebra II. 1. Inner Products and Norms

Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,

### Stat 159/259: Linear Algebra Notes

Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

### Lecture 2 INF-MAT : , LU, symmetric LU, Positve (semi)definite, Cholesky, Semi-Cholesky

Lecture 2 INF-MAT 4350 2009: 7.1-7.6, LU, symmetric LU, Positve (semi)definite, Cholesky, Semi-Cholesky Tom Lyche and Michael Floater Centre of Mathematics for Applications, Department of Informatics,

### Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 2

Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 2 Instructor: Farid Alizadeh Scribe: Xuan Li 9/17/2001 1 Overview We survey the basic notions of cones and cone-lp and give several