Linear Algebra and Graphs

Size: px
Start display at page:

Download "Linear Algebra and Graphs"

Transcription

1 Linear Algebra and Graphs IGERT Data and Network Science Bootcamp Victor Amelkin UC Santa Barbara September 11, 2015

2 Table of Contents Linear Algebra: Review of Fundamentals Matrix Arithmetic Inversion and Linear Systems Vector Spaces Geometry Eigenproblem Linear Algebra and Graphs Graphs: Definitions, Properties, Representation Spectral Graph Theory

3 Materials Copy of These Slides MATLAB Source Code for Examples

4 Vectors and Matrices Notation x C scalar (often, R is good enough) v C n n-dimensional (column-)vector A C n m matrix with n rows and m columns A ij = A i,j element of A in i th row and j th column A T transpose of A (A T ij = A ji) A H Hermitian transpose of A (A H ij = s A ji) 1 Examples A = [ ] R 2 3 A T = R a + i b = a i b C

5 Block Matrices A = m 1... m p n 1 A A 1p n q A q1... A qp Definition Block matrix a matrix of matrices. A ij block at i th row and j th column of partitioned matrix A. Blocks of the same row (column) have the same number of rows (columns).

6 Matrix Arithmetic (1) Multiplication by scalar Performed elementwise: α A n m = {α A ij} n m Addition Performed elementwise: A n m + B n m = {A ij + B ij} n m Examples [ ] [ ] = [ ]

7 Matrix Arithmetic (2) Multiplication A n mb m k = C n k [ A11 A 12 A 21 A 22 [ ] 1 0[ ] 0 1 = [0] 1 1 [a] 1 1[b] 1 1 = [a b] 1 1 (a, b C) ] [ ] [ B11 B 12 A11B 11 + A 12B 21 A 11B 12 + A 12B 22 = B 21 B 22 A 21B 11 + A 22B 21 A 21B 12 + A 22B 22 ] Corollary { m } A n mb m k = A il B lj l=1 n k

8 Matrix Arithmetic (2) Examples Example 1 (matrix-matrix (MM) multiplication): [ ] 0 1 [ = Example 2 (matrix-vector (MV) multiplication): [ ] 1 [ = Example 3 (MV block multiplication): ] ] [ ] = [ [ 1 4 ] [ 2 [1] + 5 ] [ 3 [3] + 6 ] [2] ] 1 1

9 Matrix Arithmetic (2) More Examples Example 4 (MV block multiplication): [ ] = [ ] Example 5 (row scaling 2 ): d d n n n [ A 1. A n Example 6 (permutation of rows 3 ): A A A 3 ] n n 3 m 2 For column scaling, apply diagonal matrix from the right. 3 For permutation of columns, apply P T from the right. = = 1 2 [ [ 13 = 25 d 1A 1. d na n A 3 A 1 A 2 ] [ m n n ] ]

10 Inversion Left Inverse A L is a left inverse of A if A L A = I, where I is the identity matrix (I ii = 1, I ij = 0 if i j). Right Inverse A R is a right inverse of A if AA R = I. Inverse A 1 is the inverse of A if A 1 = A L = A R. If inverse exists, it is unique; if it does not, then the (Moore-Penrose) pseudoinverse is the closest substitute.

11 Inversion Examples (matrix inversion.m) Left Inverse (no right inverse for skinny matrices): L [ Right Inverse (no left inverse for fat matrices): [ ] R Inverse (may exist only for square matrices): ]

12 Linear Systems Here, a linear system = a system of linear algebraic equations. Solving Ax = b w.r.t. x R n is one of two fundamental problems of linear algebra (the other one is eigenproblem). Unique solution exists iff A is non-singular (det(a) 0). Problem is related to matrix inversion (i.e., x = A 1 b). A system with singular A either has no or infinitely many solutions.

13 LU Factorization The method to directly solve linear systems (a kind of) Gaussian elimination. Bad ideas: Cramer s rule; inversion followed by multiplication. One kind of Gaussian elimination LU factorization / decomposition (a.k.a. Gaussian elimination with partial pivoting). Theorem (LU factorization) For any n-by-m matrix A, there exist a permutation matrix P such that P A = LU, where L is lower-triangular with units on the main diagonal and U is a block-matrix of the form r (m r) [ ] r U 11 U 12 U = (n r) 0 0 where U 11 is upper-triangular with non-zero diagonal entries. Integer r is the rank of A.

14 Solving Linear Systems with LU (linear systems.m) Problem: solve Ax = b w.r.t. x R n 1. Step 1: decompose A = P 1 LU (O(n 3 )). Step 2: (P 1 LU)x = b (LU)x = b, where b = P b (O(n)). Step 3: solve Ly = b w.r.t. y using forward substitution (O(n 2 )). Step 4: solve Ux = y w.r.t. x using back substitution (O(n 2 )).

15 Vector Space A vector space consists of a set (a field) of scalars F (often, C or R), a set of vectors V (sequences, matrices, functions,... ) and a pair of operations, vector addition + : V V V and scalar multiplication : F V V, such that α, β F x, y, z V: x + y = y + x (commutativity of addition), x + (y + z) = (x + y) + z (associativity of addition), 0 V : x + 0 = x (existence of additive identity), x V : x + ( x) = 0 (existence of additive inverse), α(βx) = (αβ)x (multiplicative associativity), 1x = x (unit scaling), α(x + y) = αx + αy (distributivity), (α + β)x = αx + βx (distributivity).

16 Vector Space A vector space consists of a set (a field) of scalars F (often, C or R), a set of vectors V (sequences, matrices, functions,... ) and a pair of operations, vector addition + : V V V and scalar multiplication : F V V, such that α, β F x, y, z V: x + y = y + x (commutativity of addition), x + (y + z) = (x + y) + z (associativity of addition), 0 V : x + 0 = x (existence of additive identity), x V : x + ( x) = 0 (existence of additive inverse), α(βx) = (αβ)x (multiplicative associativity), 1x = x (unit scaling), α(x + y) = αx + αy (distributivity), (α + β)x = αx + βx (distributivity). (Notation abuse: instead of V, F, +,, we usually refer to a vector space simply as V or V over, when we want to emphasize how the operations + and are defined.)

17 Vector Space A vector space consists of a set (a field) of scalars F (often, C or R), a set of vectors V (sequences, matrices, functions,... ) and a pair of operations, vector addition + : V V V and scalar multiplication : F V V, such that α, β F x, y, z V: x + y = y + x (commutativity of addition), x + (y + z) = (x + y) + z (associativity of addition), 0 V : x + 0 = x (existence of additive identity), x V : x + ( x) = 0 (existence of additive inverse), α(βx) = (αβ)x (multiplicative associativity), 1x = x (unit scaling), α(x + y) = αx + αy (distributivity), (α + β)x = αx + βx (distributivity). (Notation abuse: instead of V, F, +,, we usually refer to a vector space simply as V or V over, when we want to emphasize how the operations + and are defined.) A subset W of V is a subspace of V if W is a vector space on its own. Alternatively, W is a subspace iff it is closed under +,.

18 Linear Independence Linear combination: a 1x a nx n (where a i F, x j V) linear combination of vectors {x 1,..., x n} with coefficients {a 1,..., a n}.

19 Linear Independence Linear combination: a 1x a nx n (where a i F, x j V) linear combination of vectors {x 1,..., x n} with coefficients {a 1,..., a n}. Important observation: a 1x a nx n = [ ] x 1... x n a 1. a n If we think about x i not as abstract elements of V, but as (column-)vectors, it becomes clear that the result of a matrix-vector multiplication is a linear combination of the matrix columns.

20 Linear Independence Linear combination: a 1x a nx n (where a i F, x j V) linear combination of vectors {x 1,..., x n} with coefficients {a 1,..., a n}. Important observation: a 1x a nx n = [ ] x 1... x n a 1. a n If we think about x i not as abstract elements of V, but as (column-)vectors, it becomes clear that the result of a matrix-vector multiplication is a linear combination of the matrix columns. span(x 1,..., x n) = {a 1x a nx n a i F} a span of a set of vectors is the set of all their possible linear combinations.

21 Linear Independence Linear combination: a 1x a nx n (where a i F, x j V) linear combination of vectors {x 1,..., x n} with coefficients {a 1,..., a n}. Important observation: a 1x a nx n = [ ] x 1... x n a 1. a n If we think about x i not as abstract elements of V, but as (column-)vectors, it becomes clear that the result of a matrix-vector multiplication is a linear combination of the matrix columns. span(x 1,..., x n) = {a 1x a nx n a i F} a span of a set of vectors is the set of all their possible linear combinations. {x 1,..., x n} are linearly independent if a 1x a nx n = 0 iff [ a1... a n ] = 0.

22 Linear Independence Linear combination: a 1x a nx n (where a i F, x j V) linear combination of vectors {x 1,..., x n} with coefficients {a 1,..., a n}. Important observation: a 1x a nx n = [ ] x 1... x n a 1. a n If we think about x i not as abstract elements of V, but as (column-)vectors, it becomes clear that the result of a matrix-vector multiplication is a linear combination of the matrix columns. span(x 1,..., x n) = {a 1x a nx n a i F} a span of a set of vectors is the set of all their possible linear combinations. {x 1,..., x n} are linearly independent if a 1x a nx n = 0 iff [ a1... a n ] = 0. A set {x 1,..., x n} of linearly independent vectors is a basis of subspace W if W = span(x 1,..., x n). W s dimension dim W is n.

23 Fundamental Subspaces of a Matrix The nullspace (kernel) of A n m is N (A) = {x Ax = 0}. The range (column space, image) of A n m is N (A H ) left nullspace of A. R(A H ) row space of A. R(A) = colspan(a) = {y y = Ax}

24 Fundamental Subspaces of a Matrix The Big Picture row space column space nullspace left nullspace (from Gilbert Strang s Introduction to Linear Algebra, 4 th edition, 2009)

25 Norm Lengths in a vector space are measured using a norm. A vector space augmented with a norm is a normed (vector) space. A norm defined on a vector space V over field F is a mapping : V R, such that α F x, y V the following norm axioms hold x V : x 0 (non-negativity 4 ), x = 0 x = 0 (positive definiteness), αx = α x (homogeneity), x + y x + y (subadditivity / triangle inequality). Examples: l p-norm: x p = ( i x i p) 1/p, ( 1/p L p-norm: f p = f dµ) p <. D 4 Non-negativity axiom is redundant, as it can be derived from other axioms of a norm.

26 Convexity and Norm A unit sphere is defined as {x x = 1}. A set S V is convex if x, y S 0 λ 1 : λx + (1 λ)y S. In other words, for any two points of S, all the points on the line between x and y are also in S. A function f : V R is convex if x, y V 0 λ 1 : f(λx + (1 λ)y) λf(x) + (1 λ)f(y) Norms are convex.

27 Convexity and Norm l p (norms and convexity.m) ( ) 1/p Function f p(x) = x i p mapping R n to R is a norm iff 1 p. i Alternatively, f p is a norm iff the unit sphere {x f p(x) = 1} induced by f p is convex. 1 Unit spheres induced by fp(x) = ( P i jxijp ) 1=p p=5.0 p= p=1.0 p= p=0.5 x2 0 p= x1

28 Common Vector Norms Euclidean norm x 2 = x x2 n Taxicab / Manhattan norm x 1 = x x n Chebyshev norm x = max{ x 1,..., x n }

29 Norm Equivalence Norm Equivalence Two norms α and β are equivalent if there exist two positive constants c 1, c 2 < such that x V: c 1 x α x β c 2 x α. Theorem In finite-dimensional vector spaces, all norms are equivalent. Examples for x C n x 1 n x 2, x 1 n x, x 2 n x, and, more generally, for 0 < p < q x q x p n (1/p 1/q) x q.

30 Matrix Norms (operator norm.m) Frobenius Norm: A F = n i,j=1 A 2 ij = trace(ah A), where trace() of a matrix is the sum of the elements on its main diagonal. Ax Operator Norm (Induced p-norm): A = sup p x p = sup Ax p x 0 x p=1 An operator norm measures the maximum degree of distortion / amount of stretch of a unit sphere under transformation by A :0 3:0.... : 3:696465!2:0 0:5 x original unit sphere distorted unit sphere x 1 For A, B C n n, AB A B (submultiplicativity).

31 Inner Product, Angle, Projection Inner product (scalar product) of x, y V: x, y = x H y = sx iy i. i

32 Inner Product, Angle, Projection Inner product (scalar product) of x, y V: x, y = x H y = i sx iy i. Hölder s inequality: x, y x p y q for 1 p + 1 q = 1. Cauchy-Bunyakovsky-Schwarz (CBS) inequality: x, y x 2 y 2

33 Inner Product, Angle, Projection Inner product (scalar product) of x, y V: x, y = x H y = i sx iy i. Hölder s inequality: x, y x p y q for 1 p + 1 q = 1. Cauchy-Bunyakovsky-Schwarz (CBS) inequality: x, y x 2 y 2 CBS inequality inspires the following definition of an angle θ between vectors x, y R n : x, y cos θ = x y Vectors x and y are orthogonal (x y) if the cosine of the angle between them is 0.

34 Inner Product, Angle, Projection Inner product (scalar product) of x, y V: x, y = x H y = i sx iy i. Hölder s inequality: x, y x p y q for 1 p + 1 q = 1. Cauchy-Bunyakovsky-Schwarz (CBS) inequality: x, y x 2 y 2 CBS inequality inspires the following definition of an angle θ between vectors x, y R n : x, y cos θ = x y Vectors x and y are orthogonal (x y) if the cosine of the angle between them is 0. The length of the orthogonal projection of x upon y is x, y y

35 Inner Product, Angle, Projection Example Problem: Given a matrix A R n n, find the amount of stretch caused by A to vectors along a direction defined by a vector x in l 2. relative length?

36 Inner Product, Angle, Projection Example Problem: Given a matrix A R n n, find the amount of stretch caused by A to vectors along a direction defined by a vector x in l 2. relative length? Given an arbitrary x, normalize it, so that its length is 1: ( x = x/ x x = x x = x ) x = 1

37 Inner Product, Angle, Projection Example Problem: Given a matrix A R n n, find the amount of stretch caused by A to vectors along a direction defined by a vector x in l 2. relative length? Given an arbitrary x, normalize it, so that its length is 1: ( x = x/ x x = x x = x ) x = 1 The amount of stretch caused by A along x: new size original size = proj x A x x, Ax = proj x A x = x, A x = = xt Ax x x 2 x T x

38 Inner Product, Angle, Projection Example Problem: Given a matrix A R n n, find the amount of stretch caused by A to vectors along a direction defined by a vector x in l 2. relative length? Given an arbitrary x, normalize it, so that its length is 1: ( x = x/ x x = x x = x ) x = 1 The amount of stretch caused by A along x: new size original size = proj x A x x, Ax = proj x A x = x, A x = = xt Ax x x 2 x T x Derived Rayleigh quotient (generally, x H is used instead of x T ).

39 Linear Systems (Revisited) If the columns of matrix A of a linear system Ax = b span the entire space, then b can be uniquely explained in terms of these columns (b has a unique representation in this basis). If the columns of A span a subspace, then either b has infinitely many representations (if it belongs to the column (sub)space) or it has no precise representation in terms of A s columns. Even if b is out of A s range, we can replace b by the next best thing its projection upon the column (sub)space. UU is a projector upon subspace spanned by U s columns, where U = (U H U) 1 U H is U s pseudo-inverse.

40 Determinants Definition Permutation p 1,..., p n of numbers 1,..., n is their rearrangement. Sign σ(p) of permutation p is 1 if p has an even number of element interchanges; otherwise, it is 1 (e.g., σ( 1, 3, 2 = 1), σ( 3, 1, 2 = 1)). Determinant det(a n n) = A = σ(p)a 1,p1... A n,pn C (Leibniz). p Simple determinants: det(a 1 1) = A 11, det(a 2 2) = A 11A 22 A 12A 21 Expansion along a row (similarly, along a column): = (+1) ( 1) (+1) = 1 ( ) 2 ( ) + 0 ( ) = 5. =

41 Determinants Properties, Computation Properties det(a T ) = det(a), Adding rows to each other does not change det(a). Multiplying any row by α 0 scales det(a) by α. Even # of row swaps does not change det(); odd number changes sign. det(ab) = det(a) det(b) For (block-)triangular matrices A 11 A A 1n 0 A A 2n n = det(a.. ii). i= A nn Computing det(a) for large A det(a) = det(p LU) = det(p ) det(l) det(u) }{{}}{{}}{{} σ(p ) 1 ni=1 U ii

42 Determinants Important Facts Matrix Singularity An invertible A n n is called non-singular. Otherwise, it is singular. A is singular iff det(a) = 0. (det(a) 0 does not mean almost singular.)

43 Determinants Important Facts Matrix Singularity Thus, An invertible A n n is called non-singular. Otherwise, it is singular. A is singular iff det(a) = 0. (det(a) 0 does not mean almost singular.) All the columns (rows) of A are linearly independent iff det(a) 0. (In this case, we say that matrix A is full-rank, i.e., rank(a n n) = n). Linear system Ax = b has a unique solution iff det(a) 0. Homogeneous linear system Ax = 0 has non-trivial solutions iff det(a) = 0.

44 Eigenproblem Definition For a square matrix A n n, we are interested in those non-trivial vectors x 0 that do not change their direction under transformation by A: Ax = λx, λ C. These x are eigenvectors 5 of A, and the corresponding scaling factors λ are eigenvalues of A. Pairs λ, x of corresponding eigenvalues and eigenvectors are eigenpairs. Distinct eigenvalues of matrix A comprise its spectrum σ(a). Spectral radius ρ(a) = max{ σ i(a) } 5 If x is an eigenvector of A, then α x is also an eigenvector. Thus, an eigenvector defines an entire direction or, more generally, a subspace, referred to as eigenspace, whose elements do not change direction when transformed by A. Thus, an eigenspace is an invariant subspace of its matrix.

45 Characteristic Polynomial, Its Roots and Coefficients Definition Goal: find non-trivial solutions of A n nx = λx (A λi)x = 0, where I is the n-by-n identity matrix (I ii = 1, I ij = 0(i j).

46 Characteristic Polynomial, Its Roots and Coefficients Definition Goal: find non-trivial solutions of A n nx = λx (A λi)x = 0, where I is the n-by-n identity matrix (I ii = 1, I ij = 0(i j). Homogeneous system (A λi)x = 0 has non-trivial solutions iff its matrix is singular, that is, if det(a λi) = 0.

47 Characteristic Polynomial, Its Roots and Coefficients Definition Goal: find non-trivial solutions of A n nx = λx (A λi)x = 0, where I is the n-by-n identity matrix (I ii = 1, I ij = 0(i j). Homogeneous system (A λi)x = 0 has non-trivial solutions iff its matrix is singular, that is, if det(a λi) = 0. p(λ) = det(a λi) is the characteristic polynomial (in λ, of degree n) of matrix A; its roots are A s eigenvalues, and multiplicity of each root is the algebraic multiplicity of the corresponding eigenvalue. p(λ) = 0 is the characteristic equation for A.

48 Characteristic Polynomial, Its Roots and Coefficients Definition Goal: find non-trivial solutions of A n nx = λx (A λi)x = 0, where I is the n-by-n identity matrix (I ii = 1, I ij = 0(i j). Homogeneous system (A λi)x = 0 has non-trivial solutions iff its matrix is singular, that is, if det(a λi) = 0. p(λ) = det(a λi) is the characteristic polynomial (in λ, of degree n) of matrix A; its roots are A s eigenvalues, and multiplicity of each root is the algebraic multiplicity of the corresponding eigenvalue. p(λ) = 0 is the characteristic equation for A. Useful Facts From the fundamental theorem of algebra, any n-by-n square matrix always has n (not necessarily distinct) complex eigenvalues. From the complex conjugate root theorem, if a + i b C is an eigenvalue of A R n n, then a i b is also its eigenvalue. From Vieta s theorem applied to p(λ) = λ n + c 1λ n c n 1λ + c n, λ 1 + λ λ n = c 1 = trace(a) λ 1 λ 2... λ n = ( 1) n c n = det(a)

49 Resolving Eigenproblem Directly Algorithm Step 0: estimate where eigenvalues are located. Step 1: solve the characteristic equation det(a λi) = 0 using the estimates from Step 0, and find eigenvalues. Step 2: for each eigenvalue λ i, find the corresponding eigenspace by solving homogeneous system (A λ ii)x = 0. This eigenspace is comprised of non-trivial members of N (A λ ii).

50 Resolving Eigenproblem Directly Example (eigen direct.m) A = , p(λ) = det(a λi) = λ 3 + 6λ 2 11λ + 6, λ 1,2,3 = 1, 2, 3, λ 1 = 1 : N (A λ 1I) = span( [ ] T ), }{{} e 1 λ 2 = 2 : N (A λ 2I) = span( [ ] T ), }{{} e 2 λ 3 = 3 : N (A λ 3I) = span( [ ] T ). }{{} e 3

51 Resolving Eigenproblem What Actually Works (eigen arnoldi.m) Bad news: solving an equation p(λ) = 0 for high-degree p is very hard.

52 Resolving Eigenproblem What Actually Works (eigen arnoldi.m) Bad news: solving an equation p(λ) = 0 for high-degree p is very hard. Most real-world eigensolvers use the idea of Krylov sequences {x, Ax, A 2 x,... } and subspaces spanned by them.

53 Resolving Eigenproblem What Actually Works (eigen arnoldi.m) Bad news: solving an equation p(λ) = 0 for high-degree p is very hard. Most real-world eigensolvers use the idea of Krylov sequences {x, Ax, A 2 x,... } and subspaces spanned by them. A popular eigensolver for sparse matrices Arnoldi/Lancsoz iteration (an advanced version of the power method). It allows to quickly compute several (largest, smallest, closest to a given value) eigenvalues and the corresponding eigenvectors of a sparse matrix, mostly, using matrix-vector multiplication. This method is used by MATLAB s eigs and by Python s scipy.sparse.linalg.eigs. For dense matrices, eigensolvers based on Schur or Cholesky decomposition may be used.

54 Eigenvalue Localization Sometimes, it may be enough to have a good estimate of where eigenvalues are, without actually computing them. That estimation is referred to as eigenvalue localization. Tools Crude bound: λ(a) < A. Cauchy s [ Interlacing ] Theorem: for real symmetric n-by-n matrix B c A = c T, where δ R, δ λ n(a) λ n 1(B)... λ k (B) λ k (A) λ k 1 (B) λ 1(B) λ 1(A). Gerschgorin Circles The eigenvalues of A C n n are trapped inside the union of Gerschgorin circles z A ii < r i, where r i = min { n n A ij, A ji, }, i = 1,..., n. A k Gerschgorin circles j=1 j i j=1 j i disjoint from others contain exactly k eigenvalues.

55 Gerschgorin Circles Example * * * *

56 Eigendecomposition (eigendecomp.m) A C n n is diagonalizable if there is an invertible P such that P 1 AP is diagonal. If a real-valued matrix is symmetric, then it is diagonalizable. (Though, invertible matrices do not have to be symmetric in general.)

57 Eigendecomposition (eigendecomp.m) A C n n is diagonalizable if there is an invertible P such that P 1 AP is diagonal. If a real-valued matrix is symmetric, then it is diagonalizable. (Though, invertible matrices do not have to be symmetric in general.) Each diagonalizable A permits (eigen)decomposition: A = q 1... q n } {{ } Q λ λ λ n } {{ } Λ q 1... q n = QΛQ 1. Analog for non-diagonalizable matrices Jordan normal form.

58 Eigendecomposition MV Multiplication for Real Symmetric Matrices Assume A R n n symmetric. A is diagonalizable, and its eigenvectors are orthogonal. For orthogonal matrix A, A 1 = A T. ( n ) Ax = (QΛQ 1 )x = λ iq i q T i x = i=1 n λ iq i (q T i x) = i=1 n i=1 λ i x, q i }{{} q i proj qi x

59 Singular Value Decomposition (SVD) Theorem For every (rectangular) matrix A C n m, there are two unitary 6 matrices U C m m and V C n n, as well as a matrix Σ R m n of the form σ Σ =. 0 σ 2....,.... with σ 1 σ 2 σ min (n,m) 0, such that A = UΣV H. Diagonal values of Σ singular values of A, columns of U and V left and right singular vectors of A, respectively. (Notice redundant columns in U and rows (or columns) in Σ.) 6 A matrix A C n n is unitary if A H A = AA H = I. Unitary matrices play a role similar to the role a scalar 1 plays ( a size-preserving transform.)

60 Table of Contents Linear Algebra: Review of Fundamentals Matrix Arithmetic Inversion and Linear Systems Vector Spaces Geometry Eigenproblem Linear Algebra and Graphs Graphs: Definitions, Properties, Representation Spectral Graph Theory

61 Graphs: definitions, properties An (edge-weighted) graph is a tuple G = V, E, w, where V is a set of nodes, E V V is a set of edges between the nodes, and w : E R defined edge weights. If w is not specified, then edge weights can assumed to be equal 1. If E is a symmetric relation (and w, if specified, is a symmetric function), then G is said to be undirected; otherwise, it is directed. A graph may have weights on its nodes rather than the edges (or on both). A node-weighted graph can be transformed into an edge-weighted graph, or vice versa. A graph G = V, E, w is (weakly) connected if for any v 1, v 2 V, v 1 v 2, we can reach v 2 from v 1 by walking along the adjacent edges E (ignoring their direction). A (weakly) disconnected graph G consists of connected components (CC), which are maximal (weakly) connected subgraphs. When we take into account direction of edges, the notion of connectedness extends to the notion of strong connectedness (strongly connected components (SCC) are, then, defined similarly to weakly connected components).

62 Graphs Examples

63 Graphs Special Graphs

64 Representation of Graphs Adjacency Matrix All definitions are given for a graph G = V, E, w having V = n nodes and E = m edges. The most popular representation of a graph is its adjacency matrix A: { w( i, j ) = w ij if i, j E, A ij = 0 otherwise. If weights w are not specified, then A is a binary matrix.

65 Representation of Graphs Adjacency Matrices of Special Graphs

66 Representation of Graphs Degree Matrix The in-degree d in i of node v i is the sum of the weights on all of its incoming edges. The out-degree d out i of node v i is similarly defined via v i s outgoing edges. For undirected graphs, both in-degree and out-degree are equal d i referred to as simply the degree of node v i. For unweighted graphs, the degree measures the number of a node s (in-, our-, or all) neighbors. The degree matrix D = diag({d i} 1...n) of a graph G is a diagonal matrix with node degrees on its main diagonal. For directed graphs, in-degree and out-degree matrices can be similarly defined using the appropriate degree definitions.

67 Laplacian of Undirected Graphs (Combinatorial) Laplacian Weighted graph: L ij = Unweighted graph: Other Laplacians L = D A { Dii = d i = i,l E w i,l if i = j, w ij if i, j E, 0 otherwise; L ij = { di = i,l E 1 if i = j, 1 if i, j E, 0 otherwise. L sym = D 1/2 LD 1/2 = I D 1/2 AD 1/2, L rw = D 1 A.

68 Spectral Graph Theory Spectral Graph Theory Graphs are usually represented with matrices 7. Spectral graph theory attempts to connect spectral properties of these matrices with the corresponding graphs structural properties. Limitations Most results of spectral graph theory are obtained for undirected and unweighted graphs, i.e., graphs having binary symmetric adjacency matrices. If a result applies to weighted graphs, it will be explicitly stated. 7 Some may even go as far as to claim that graphs and matrices are the same thing.

69 Spectrum of Adjacency Matrix Walks in Graphs A {0, 1} n n adjacency matrix of an undirected unweighted graph G. A ij number of walks of length 1 in G between nodes v i and v j. (A k ) ij number of walks of length k in G between nodes v i and v j. (A k 1) i number of walks of length k ending at v i. 1 T A k 1 number of walks of length k in G.

70 Largest 8 Eigenvalue of Adjacency Matrix µ 1 = µ max Connection to µ max (undirected, unweighted, connected G): 1 T A k 1 = (since A is real and symmetric) = 1 T (Q diag (µ i)q 1 ) k 1 = = (since Q is orthogonal) = 1 T Q diag (µ k i )Q 1 1 = ( n ) = 1 T µ k i q i q T i 1 = i=1 n µ k i (1 T q i )(1 T q i ) T = i=1 = (1 = α 1q α nq n ) = k µmax n i=1 i max µ k i n µ k i q i, 1 2 = i=1 ( n ) 2 α j qi, q j = j=1 ( ( n ) 1/k 1/k lim 1 T A 1) k = lim µ k i αi 2 = k k i=1 = lim αmax 2 + ( ) k µi α 2 µ max i 1/k n µ k i (α i) 2 ; i=1 = µ max(= A ). Thus, µ k max = A(G) k is the number of walks of length k in G. 8 The largest eigenvalue is such w.r.t. its absolute value.

71 Largest Eigenvalue of Adjacency Matrix µ 1 = µ max Summary Derived µ k max = A(G) k is the number of walks of length k in G. For directed A, the meaning of lim k A k 1 = q max is close to the one of PageRank.

72 Largest Eigenvalue of Adjacency Matrix µ 1 = µ max Summary Derived µ k max = A(G) k is the number of walks of length k in G. For directed A, the meaning of lim k A k 1 = q max is close to the one of PageRank. Beyond Walks (( ) applies to weighted graphs) ( ) If graph G is connected, µ max has multiplicity 1, and its eigenvector is positive (all its entries are strictly positive). ( ) d avg µ 1 d max (d avg, d max mean and maximum node degrees). max {d avg, d max} µ max d max. ( ) If G is connected, and µ max = d max, then i : d i = d max. ( ) A connected graph is bipartite iff µ min = µ max. A graph is bipartite iff its spectrum is symmetric about 0. χ(g) 1 + µ min/µ max.

73 Smallest Eigenvalues of Combinatorial Laplacian L = D A. Eigenvalues of L are non-negative: 0 λ 1 λ 2 λ n.

74 Smallest Eigenvalues of Combinatorial Laplacian L = D A. Eigenvalues of L are non-negative: 0 λ 1 λ 2 λ n. L is singular 0 is its eigenvalue corresponding to eigenvector 1. If G is connected, λ 1 = 0 has multiplicity 1.

75 Smallest Eigenvalues of Combinatorial Laplacian L = D A. Eigenvalues of L are non-negative: 0 λ 1 λ 2 λ n. L is singular 0 is its eigenvalue corresponding to eigenvector 1. If G is connected, λ 1 = 0 has multiplicity 1. If G has k connected components, λ 1 = 0 s multiplicity is k.

76 Smallest Eigenvalues of Combinatorial Laplacian L = D A. Eigenvalues of L are non-negative: 0 λ 1 λ 2 λ n. L is singular 0 is its eigenvalue corresponding to eigenvector 1. If G is connected, λ 1 = 0 has multiplicity 1. If G has k connected components, λ 1 = 0 s multiplicity is k. The harder it is to disconnect G by removing its edges, the larger the gap between λ 1 = 0 and λ 2 > 0 is.

77 Smallest Eigenvalues of Combinatorial Laplacian L = D A. Eigenvalues of L are non-negative: 0 λ 1 λ 2 λ n. L is singular 0 is its eigenvalue corresponding to eigenvector 1. If G is connected, λ 1 = 0 has multiplicity 1. If G has k connected components, λ 1 = 0 s multiplicity is k. The harder it is to disconnect G by removing its edges, the larger the gap between λ 1 = 0 and λ 2 > 0 is. λ 2 algebraic connectivity (a.k.a. Fiedler value, spectral gap) a measure of graph connectedness. q 2 Fiedler vector eigenvector associated with λ 2 solution to a relaxed min-cut (sparsest cut) in G. The same eigenvector of a normalized Laplacian L sym solution to a relaxed normalized min-cut ( edge-balanced sparsest cut ) in G.

78 Spectral Bisection (spectral bisection.m) Spectral bisection using 7 2 of Combinatorial Laplacian (6 2 = ) 18 cut edges Figure : Example of spectral bisection with Fiedler vector. (The Tapir graph as well as the plotting functions come from meshpart toolbox by John R. Gilbert and Shang-Hua Teng.)

79 Spectral Clustering (spectral clustering.m) Partitioning into 2 clusters Partitioning into 3 clusters Partitioning into 4 clusters Partitioning into 5 clusters Figure : Example of spectral clustering using normalized Laplacian and k-means. (The Tapir graph as well as the plotting functions come from meshpart toolbox by John R. Gilbert and Shang-Hua Teng.)

80 What is next Relevant Courses at UCSB ECE/CS211A Matrix Analysis a decent overview of most fundamentals of linear algebra, from the definition of block-matrix arithmetic to spectral theory. CS290H Graph Laplacians and Spectra this course is focused on the study of spectra of graph Laplacians as well as on the accompanying computational problems (extracting eigenpairs of Laplacians, solving Laplacian linear systems). Reading Linear Algebra Core Matrix Analysis by Shiv Chandrasekaran a textbook for ECE/CS211A. Provides an overview of most necessary fundamentals. Introduction to Linear Algebra (any edition) by Gilbert Strang an entry-level book about fundamentals of linear algebra; great exposition. Matrix Analysis and Applied Linear Algebra by Carl Meyer an advanced linear algebra textbook; pick this one if Strang s textbook feels too easy to read.

81 What is next Reading Linear Algebra of Graphs Spectral Graph Theory by Fan Chung (1997). Complex Graphs and Networks by Fan Chung (2006). Dan Spielman s course on spectral graph theory. Eigenvalues of Graphs by László Lovász (2007). Luca Trevisan s course on spectral graph theory. Algebraic connectivity of graphs by Miroslav Fiedler (1973). A tutorial on spectral clustering by Ulrike von Luxburg (2007).

82 Thanks Copyright (c) Victor Amelkin, 2015

Networks and Their Spectra

Networks and Their Spectra Networks and Their Spectra Victor Amelkin University of California, Santa Barbara Department of Computer Science victor@cs.ucsb.edu December 4, 2017 1 / 18 Introduction Networks (= graphs) are everywhere.

More information

Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination

Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column

More information

Review of Some Concepts from Linear Algebra: Part 2

Review of Some Concepts from Linear Algebra: Part 2 Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set

More information

Elementary linear algebra

Elementary linear algebra Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The

More information

MATH 240 Spring, Chapter 1: Linear Equations and Matrices

MATH 240 Spring, Chapter 1: Linear Equations and Matrices MATH 240 Spring, 2006 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 8th Ed. Sections 1.1 1.6, 2.1 2.2, 3.2 3.8, 4.3 4.5, 5.1 5.3, 5.5, 6.1 6.5, 7.1 7.2, 7.4 DEFINITIONS Chapter 1: Linear

More information

1. General Vector Spaces

1. General Vector Spaces 1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

More information

Foundations of Matrix Analysis

Foundations of Matrix Analysis 1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the

More information

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.

More information

OHSx XM511 Linear Algebra: Solutions to Online True/False Exercises

OHSx XM511 Linear Algebra: Solutions to Online True/False Exercises This document gives the solutions to all of the online exercises for OHSx XM511. The section ( ) numbers refer to the textbook. TYPE I are True/False. Answers are in square brackets [. Lecture 02 ( 1.1)

More information

Linear algebra and applications to graphs Part 1

Linear algebra and applications to graphs Part 1 Linear algebra and applications to graphs Part 1 Written up by Mikhail Belkin and Moon Duchin Instructor: Laszlo Babai June 17, 2001 1 Basic Linear Algebra Exercise 1.1 Let V and W be linear subspaces

More information

1 9/5 Matrices, vectors, and their applications

1 9/5 Matrices, vectors, and their applications 1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric

More information

Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors Chapter 1 Eigenvalues and Eigenvectors Among problems in numerical linear algebra, the determination of the eigenvalues and eigenvectors of matrices is second in importance only to the solution of linear

More information

Knowledge Discovery and Data Mining 1 (VO) ( )

Knowledge Discovery and Data Mining 1 (VO) ( ) Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory

More information

Linear Algebra in Actuarial Science: Slides to the lecture

Linear Algebra in Actuarial Science: Slides to the lecture Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations

More information

Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )

Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v ) Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

More information

Lecture 2: Linear Algebra Review

Lecture 2: Linear Algebra Review EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1

More information

A Brief Outline of Math 355

A Brief Outline of Math 355 A Brief Outline of Math 355 Lecture 1 The geometry of linear equations; elimination with matrices A system of m linear equations with n unknowns can be thought of geometrically as m hyperplanes intersecting

More information

Linear Algebra Highlights

Linear Algebra Highlights Linear Algebra Highlights Chapter 1 A linear equation in n variables is of the form a 1 x 1 + a 2 x 2 + + a n x n. We can have m equations in n variables, a system of linear equations, which we want to

More information

Review of some mathematical tools

Review of some mathematical tools MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical

More information

LinGloss. A glossary of linear algebra

LinGloss. A glossary of linear algebra LinGloss A glossary of linear algebra Contents: Decompositions Types of Matrices Theorems Other objects? Quasi-triangular A matrix A is quasi-triangular iff it is a triangular matrix except its diagonal

More information

The Eigenvalue Problem: Perturbation Theory

The Eigenvalue Problem: Perturbation Theory Jim Lambers MAT 610 Summer Session 2009-10 Lecture 13 Notes These notes correspond to Sections 7.2 and 8.1 in the text. The Eigenvalue Problem: Perturbation Theory The Unsymmetric Eigenvalue Problem Just

More information

Linear Algebra Primer

Linear Algebra Primer Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................

More information

Review problems for MA 54, Fall 2004.

Review problems for MA 54, Fall 2004. Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on

More information

DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra. DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

More information

Online Exercises for Linear Algebra XM511

Online Exercises for Linear Algebra XM511 This document lists the online exercises for XM511. The section ( ) numbers refer to the textbook. TYPE I are True/False. Lecture 02 ( 1.1) Online Exercises for Linear Algebra XM511 1) The matrix [3 2

More information

MAT 610: Numerical Linear Algebra. James V. Lambers

MAT 610: Numerical Linear Algebra. James V. Lambers MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic

More information

Lecture Summaries for Linear Algebra M51A

Lecture Summaries for Linear Algebra M51A These lecture summaries may also be viewed online by clicking the L icon at the top right of any lecture screen. Lecture Summaries for Linear Algebra M51A refers to the section in the textbook. Lecture

More information

Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88 Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

More information

MATH 583A REVIEW SESSION #1

MATH 583A REVIEW SESSION #1 MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),

More information

A Review of Linear Algebra

A Review of Linear Algebra A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose

More information

ANSWERS. E k E 2 E 1 A = B

ANSWERS. E k E 2 E 1 A = B MATH 7- Final Exam Spring ANSWERS Essay Questions points Define an Elementary Matrix Display the fundamental matrix multiply equation which summarizes a sequence of swap, combination and multiply operations,

More information

Basic Elements of Linear Algebra

Basic Elements of Linear Algebra A Basic Review of Linear Algebra Nick West nickwest@stanfordedu September 16, 2010 Part I Basic Elements of Linear Algebra Although the subject of linear algebra is much broader than just vectors and matrices,

More information

Math Linear Algebra Final Exam Review Sheet

Math Linear Algebra Final Exam Review Sheet Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of

More information

Chapter 5 Eigenvalues and Eigenvectors

Chapter 5 Eigenvalues and Eigenvectors Chapter 5 Eigenvalues and Eigenvectors Outline 5.1 Eigenvalues and Eigenvectors 5.2 Diagonalization 5.3 Complex Vector Spaces 2 5.1 Eigenvalues and Eigenvectors Eigenvalue and Eigenvector If A is a n n

More information

Large Scale Data Analysis Using Deep Learning

Large Scale Data Analysis Using Deep Learning Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset

More information

Numerical Methods - Numerical Linear Algebra

Numerical Methods - Numerical Linear Algebra Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear

More information

MAT Linear Algebra Collection of sample exams

MAT Linear Algebra Collection of sample exams MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system

More information

Linear Algebra Massoud Malek

Linear Algebra Massoud Malek CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

More information

MATH 532: Linear Algebra

MATH 532: Linear Algebra MATH 532: Linear Algebra Chapter 5: Norms, Inner Products and Orthogonality Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2015 fasshauer@iit.edu MATH 532 1 Outline

More information

Introduction to Matrix Algebra

Introduction to Matrix Algebra Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

More information

HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION)

HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) PROFESSOR STEVEN MILLER: BROWN UNIVERSITY: SPRING 2007 1. CHAPTER 1: MATRICES AND GAUSSIAN ELIMINATION Page 9, # 3: Describe

More information

A Review of Linear Algebra

A Review of Linear Algebra A Review of Linear Algebra Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab: Implementations

More information

5.6. PSEUDOINVERSES 101. A H w.

5.6. PSEUDOINVERSES 101. A H w. 5.6. PSEUDOINVERSES 0 Corollary 5.6.4. If A is a matrix such that A H A is invertible, then the least-squares solution to Av = w is v = A H A ) A H w. The matrix A H A ) A H is the left inverse of A and

More information

Math Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p

Math Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the

More information

LINEAR ALGEBRA SUMMARY SHEET.

LINEAR ALGEBRA SUMMARY SHEET. LINEAR ALGEBRA SUMMARY SHEET RADON ROSBOROUGH https://intuitiveexplanationscom/linear-algebra-summary-sheet/ This document is a concise collection of many of the important theorems of linear algebra, organized

More information

Linear Algebra Formulas. Ben Lee

Linear Algebra Formulas. Ben Lee Linear Algebra Formulas Ben Lee January 27, 2016 Definitions and Terms Diagonal: Diagonal of matrix A is a collection of entries A ij where i = j. Diagonal Matrix: A matrix (usually square), where entries

More information

2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian

2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian

More information

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko

More information

Stat 159/259: Linear Algebra Notes

Stat 159/259: Linear Algebra Notes Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

More information

ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS. 1. Linear Equations and Matrices

ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS. 1. Linear Equations and Matrices ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS KOLMAN & HILL NOTES BY OTTO MUTZBAUER 11 Systems of Linear Equations 1 Linear Equations and Matrices Numbers in our context are either real numbers or complex

More information

Linear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0.

Linear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0. Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the

More information

Econ Slides from Lecture 7

Econ Slides from Lecture 7 Econ 205 Sobel Econ 205 - Slides from Lecture 7 Joel Sobel August 31, 2010 Linear Algebra: Main Theory A linear combination of a collection of vectors {x 1,..., x k } is a vector of the form k λ ix i for

More information

Computational Methods. Eigenvalues and Singular Values

Computational Methods. Eigenvalues and Singular Values Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

More information

Linear Algebra: Matrix Eigenvalue Problems

Linear Algebra: Matrix Eigenvalue Problems CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

More information

Conceptual Questions for Review

Conceptual Questions for Review Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.

More information

Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors /88 Chia-Ping Chen Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Eigenvalue Problem /88 Eigenvalue Equation By definition, the eigenvalue equation for matrix

More information

The following definition is fundamental.

The following definition is fundamental. 1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic

More information

Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017

Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017 Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...

More information

LINEAR ALGEBRA REVIEW

LINEAR ALGEBRA REVIEW LINEAR ALGEBRA REVIEW SPENCER BECKER-KAHN Basic Definitions Domain and Codomain. Let f : X Y be any function. This notation means that X is the domain of f and Y is the codomain of f. This means that for

More information

CS 246 Review of Linear Algebra 01/17/19

CS 246 Review of Linear Algebra 01/17/19 1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector

More information

Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

More information

Some notes on Linear Algebra. Mark Schmidt September 10, 2009

Some notes on Linear Algebra. Mark Schmidt September 10, 2009 Some notes on Linear Algebra Mark Schmidt September 10, 2009 References Linear Algebra and Its Applications. Strang, 1988. Practical Optimization. Gill, Murray, Wright, 1982. Matrix Computations. Golub

More information

Notes on Linear Algebra and Matrix Theory

Notes on Linear Algebra and Matrix Theory Massimo Franceschet featuring Enrico Bozzo Scalar product The scalar product (a.k.a. dot product or inner product) of two real vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) is not a vector but a

More information

(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =

(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax = . (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)

More information

CS 143 Linear Algebra Review

CS 143 Linear Algebra Review CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see

More information

Linear Algebra. Session 12

Linear Algebra. Session 12 Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)

More information

Notes on Eigenvalues, Singular Values and QR

Notes on Eigenvalues, Singular Values and QR Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square

More information

Numerical Linear Algebra

Numerical Linear Algebra Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix

More information

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP) MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

More information

ANSWERS (5 points) Let A be a 2 2 matrix such that A =. Compute A. 2

ANSWERS (5 points) Let A be a 2 2 matrix such that A =. Compute A. 2 MATH 7- Final Exam Sample Problems Spring 7 ANSWERS ) ) ). 5 points) Let A be a matrix such that A =. Compute A. ) A = A ) = ) = ). 5 points) State ) the definition of norm, ) the Cauchy-Schwartz inequality

More information

Chapter 0 Miscellaneous Preliminaries

Chapter 0 Miscellaneous Preliminaries EE 520: Topics Compressed Sensing Linear Algebra Review Notes scribed by Kevin Palmowski, Spring 2013, for Namrata Vaswani s course Notes on matrix spark courtesy of Brian Lois More notes added by Namrata

More information

MTH 464: Computational Linear Algebra

MTH 464: Computational Linear Algebra MTH 464: Computational Linear Algebra Lecture Outlines Exam 2 Material Prof. M. Beauregard Department of Mathematics & Statistics Stephen F. Austin State University March 2, 2018 Linear Algebra (MTH 464)

More information

Glossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB

Glossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the

More information

Linear Algebra - Part II

Linear Algebra - Part II Linear Algebra - Part II Projection, Eigendecomposition, SVD (Adapted from Sargur Srihari s slides) Brief Review from Part 1 Symmetric Matrix: A = A T Orthogonal Matrix: A T A = AA T = I and A 1 = A T

More information

G1110 & 852G1 Numerical Linear Algebra

G1110 & 852G1 Numerical Linear Algebra The University of Sussex Department of Mathematics G & 85G Numerical Linear Algebra Lecture Notes Autumn Term Kerstin Hesse (w aw S w a w w (w aw H(wa = (w aw + w Figure : Geometric explanation of the

More information

Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014

Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document

More information

Quantum Computing Lecture 2. Review of Linear Algebra

Quantum Computing Lecture 2. Review of Linear Algebra Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces

More information

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

More information

and let s calculate the image of some vectors under the transformation T.

and let s calculate the image of some vectors under the transformation T. Chapter 5 Eigenvalues and Eigenvectors 5. Eigenvalues and Eigenvectors Let T : R n R n be a linear transformation. Then T can be represented by a matrix (the standard matrix), and we can write T ( v) =

More information

Equality: Two matrices A and B are equal, i.e., A = B if A and B have the same order and the entries of A and B are the same.

Equality: Two matrices A and B are equal, i.e., A = B if A and B have the same order and the entries of A and B are the same. Introduction Matrix Operations Matrix: An m n matrix A is an m-by-n array of scalars from a field (for example real numbers) of the form a a a n a a a n A a m a m a mn The order (or size) of A is m n (read

More information

Linear Algebra for Machine Learning. Sargur N. Srihari

Linear Algebra for Machine Learning. Sargur N. Srihari Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it

More information

forms Christopher Engström November 14, 2014 MAA704: Matrix factorization and canonical forms Matrix properties Matrix factorization Canonical forms

forms Christopher Engström November 14, 2014 MAA704: Matrix factorization and canonical forms Matrix properties Matrix factorization Canonical forms Christopher Engström November 14, 2014 Hermitian LU QR echelon Contents of todays lecture Some interesting / useful / important of matrices Hermitian LU QR echelon Rewriting a as a product of several matrices.

More information

Numerical Methods I Eigenvalue Problems

Numerical Methods I Eigenvalue Problems Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 2nd, 2014 A. Donev (Courant Institute) Lecture

More information

MATH 581D FINAL EXAM Autumn December 12, 2016

MATH 581D FINAL EXAM Autumn December 12, 2016 MATH 58D FINAL EXAM Autumn 206 December 2, 206 NAME: SIGNATURE: Instructions: there are 6 problems on the final. Aim for solving 4 problems, but do as much as you can. Partial credit will be given on all

More information

Math Introduction to Numerical Analysis - Class Notes. Fernando Guevara Vasquez. Version Date: January 17, 2012.

Math Introduction to Numerical Analysis - Class Notes. Fernando Guevara Vasquez. Version Date: January 17, 2012. Math 5620 - Introduction to Numerical Analysis - Class Notes Fernando Guevara Vasquez Version 1990. Date: January 17, 2012. 3 Contents 1. Disclaimer 4 Chapter 1. Iterative methods for solving linear systems

More information

SUMMARY OF MATH 1600

SUMMARY OF MATH 1600 SUMMARY OF MATH 1600 Note: The following list is intended as a study guide for the final exam. It is a continuation of the study guide for the midterm. It does not claim to be a comprehensive list. You

More information

Matrix Factorization and Analysis

Matrix Factorization and Analysis Chapter 7 Matrix Factorization and Analysis Matrix factorizations are an important part of the practice and analysis of signal processing. They are at the heart of many signal-processing algorithms. Their

More information

Math 302 Outcome Statements Winter 2013

Math 302 Outcome Statements Winter 2013 Math 302 Outcome Statements Winter 2013 1 Rectangular Space Coordinates; Vectors in the Three-Dimensional Space (a) Cartesian coordinates of a point (b) sphere (c) symmetry about a point, a line, and a

More information

Bindel, Fall 2016 Matrix Computations (CS 6210) Notes for

Bindel, Fall 2016 Matrix Computations (CS 6210) Notes for 1 Logistics Notes for 2016-08-29 General announcement: we are switching from weekly to bi-weekly homeworks (mostly because the course is much bigger than planned). If you want to do HW but are not formally

More information

4. Determinants.

4. Determinants. 4. Determinants 4.1. Determinants; Cofactor Expansion Determinants of 2 2 and 3 3 Matrices 2 2 determinant 4.1. Determinants; Cofactor Expansion Determinants of 2 2 and 3 3 Matrices 3 3 determinant 4.1.

More information

Lecture 1: Review of linear algebra

Lecture 1: Review of linear algebra Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations

More information

Review of Linear Algebra

Review of Linear Algebra Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=

More information

Linear Algebra M1 - FIB. Contents: 5. Matrices, systems of linear equations and determinants 6. Vector space 7. Linear maps 8.

Linear Algebra M1 - FIB. Contents: 5. Matrices, systems of linear equations and determinants 6. Vector space 7. Linear maps 8. Linear Algebra M1 - FIB Contents: 5 Matrices, systems of linear equations and determinants 6 Vector space 7 Linear maps 8 Diagonalization Anna de Mier Montserrat Maureso Dept Matemàtica Aplicada II Translation:

More information

Properties of Matrices and Operations on Matrices

Properties of Matrices and Operations on Matrices Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

More information

APPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.

APPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2. APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product

More information

EE731 Lecture Notes: Matrix Computations for Signal Processing

EE731 Lecture Notes: Matrix Computations for Signal Processing EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten

More information

IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET

IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET IMPORTANT DEFINITIONS AND THEOREMS REFERENCE SHEET This is a (not quite comprehensive) list of definitions and theorems given in Math 1553. Pay particular attention to the ones in red. Study Tip For each

More information

Linear Algebra Review. Vectors

Linear Algebra Review. Vectors Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors

More information

1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)

1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220) Notes for 2017-01-30 Most of mathematics is best learned by doing. Linear algebra is no exception. You have had a previous class in which you learned the basics of linear algebra, and you will have plenty

More information

Linear Algebra Review

Linear Algebra Review Chapter 1 Linear Algebra Review It is assumed that you have had a course in linear algebra, and are familiar with matrix multiplication, eigenvectors, etc. I will review some of these terms here, but quite

More information