Introduction to Numerical Linear Algebra II
|
|
- Gabriel Brown
- 5 years ago
- Views:
Transcription
1 Introduction to Numerical Linear Algebra II Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA 1 / 49
2 Overview We will cover this material in relatively more detail, but will still skip a lot... 1 Norms {Measuring the length/mass of mathematical quantities} General norms Vector p norms Matrix norms induced by vector p norms Frobenius norm 2 Singular Value Decomposition (SVD) The most important tool in Numerical Linear Algebra 3 Least Squares problems Linear systems that do not have a solution 2 / 49
3 Norms
4 General Norms How to measure the mass of a matrix or length of a vector Norm is function R m n R with 1 Non-negativity A 0, A = 0 A = 0 2 Triangle inequality A + B A + B 3 Scalar multiplication α A = α A for all α R. Properties Minus signs A = A Reverse triangle inequality A B A B New norms For norm on R m n, and nonsingular M R m m def A M = M A is also a norm 4 / 49
5 Vector p Norms For x R n and integer p 1 x p def = ( n ) 1/p x i p i=1 One norm x 1 = n j=1 x j Euclidean (two) norm x 2 = n j=1 x j 2 = x T x Infinity (max) norm x = max 1 j n x j 5 / 49
6 Exercises 1 Determine 11 p for 11 R n and p = 1, 2, 2 For x R n with x j = j, 1 j n, determine closed-form expressions for x p for p = 1, 2, 6 / 49
7 Inner Products and Norm Relations For x, y R n Cauchy-Schwartz inequality Hölder inequality x T y x 2 y 2 Relations x T y x 1 y x T y x y 1 x x 1 n x x 2 x 1 n x 2 x x 2 n x 7 / 49
8 Exercises 1 Prove the norm relations on the previous slide 2 For x R n show x 2 x x 1 8 / 49
9 Vector Two Norm Theorem of Pythagoras For x, y R n x T y = 0 x ± y 2 2 = x y 2 2 Two norm does not care about orthonormal matrices For x R n and V R m n with V T V = I n V x 2 = x 2 9 / 49
10 Vector Two Norm Theorem of Pythagoras For x, y R n x T y = 0 x ± y 2 2 = x y 2 2 Two norm does not care about orthonormal matrices For x R n and V R m n with V T V = I n V x 2 = x 2 V x 2 2 = (V x)t (V x) = x T V T V x = x T x = x / 49
11 Exercises For x, y R n show 1 Parallelogram equality x + y x y 2 2 = 2 ( x y 2 ) 2 2 Polarization identity x T y = 1 4 ( x + y 2 2 x y 2 2) 10 / 49
12 Matrix Norms (Induced by Vector p Norms) For A R m n and integer p 1 def Ax p A p = max x 0 x p = max y p=1 A y p One Norm: Maximum absolute column sum A 1 = max 1 j n m i=1 a ij = max 1 j n A e j 1 Infinity Norm: Maximum absolute row sum A = max 1 i m n j=1 a ij = max 1 i m AT e i 1 11 / 49
13 Matrix Norm Properties Every norm realized by some vector y 0 A p = Ay p y p = Az p where z y y p z p = 1 {Vector y is different for every A and every p} Submultiplicativity Matrix vector product: For A R m n and y R n Ay p A p y p Matrix product: For A R m n and B R n l AB p A p B p 12 / 49
14 More Matrix Norm Properties For A R m n and permutation matrices P R m m, Q R n n Permutation matrices do not matter P A Q p = A p Submatrices have smaller norms than parent matrix ( ) B A12 If P A Q = then B A 21 A p A p / 49
15 Exercises 1 Different norms realized by different vectors Find y and z so that A 1 = Ay 1 and A = Az when ( ) 1 4 A = If Q R n n is permutation then Q p = 1 3 Prove the two types of submultiplicativity 4 If D = diag ( d 11 d nn ) then D p = max 1 j n d jj 5 If A R n n nonsingular then A p A 1 p 1 14 / 49
16 Norm Relations and Transposes For A R m n Relations between different norms 1 n A A 2 m A 1 m A 1 A 2 n A 1 Transposes A T 1 = A A T = A 1 A T 2 = A 2 15 / 49
17 Proof: Two Norm of Transpose {True for A = 0, so assume A 0} Let A 2 = Az 2 for some z R n with z 2 = 1 Reduce to vector norm ( ) A 2 2 = Az 2 2 = (Az) T (Az) = z T A T Az = z T A T Az }{{} vector Cauchy-Schwartz inequality and submultiplicativity imply ( ) z T A T Az z 2 A T Az 2 A T 2 A 2 Thus A 2 2 AT 2 A 2 and A 2 A T 2 Reversing roles of A and A T gives A T 2 A 2 16 / 49
18 Matrix Two Norm A R m n Orthonormal matrices do not change anything If U R k m with U T U = I m, V R l n with V T V = I n U A V T 2 = A 2 Gram matrices For A R m n A T A 2 = A 2 2 = A A T 2 Outer products For x R m, y R n xy T 2 = x 2 y 2 17 / 49
19 Proof: Norm of Gram Matrix Submultiplicativity and transpose A T A 2 A 2 A T 2 = A 2 A 2 = A 2 2 Thus A T A 2 A 2 2 Let A 2 = Az 2 for some z R n with z 2 = 1 Cauchy-Schwartz inequality and submultiplicativity imply ( ) A 2 2 = Az 2 2 = (Az) T (Az) = z T A T A z }{{} vector z 2 A T A z 2 A T A 2 Thus A 2 2 AT A 2 18 / 49
20 Exercises 1 Infinity norm of outer products For x R m and y R n, show x y T = x y 1 2 If A R n n with A 0 is idempotent then (i) A p 1 (ii) A 2 = 1 if A also symmetric 3 Given A R n n Among all symmetric matrices, 1 2 (A + AT ) is a (the?) matrix that is closest to A in the two norm 19 / 49
21 Frobenius Norm The true mass of a matrix For A = ( a 1 a n ) R m n A F def = n n m a j 2 2 = a ij 2 = j=1 j=1 i=1 trace(a T A) 20 / 49
22 Frobenius Norm The true mass of a matrix For A = ( a 1 a n ) R m n A F def = n n m a j 2 2 = a ij 2 = j=1 j=1 i=1 trace(a T A) Vector If x R n then x F = x 2 Transpose Identity A T F = A F I n F = n 20 / 49
23 More Frobenius Norm Properties A R m n Orthonormal invariance If U R k m with U T U = I m, V R l n with V T V = I n Relation to two norm U A V T F = A F A 2 A F rank(a) A 2 min{m, n} A 2 Submultiplicativity A B F A 2 B F A F B F 21 / 49
24 Exercises 1 Show the orthonormal invariance of the Frobenius norm 2 Show the submultiplicativity of the Frobenius norm 3 Frobenius norm of outer products For x R m and y R n show x y T F = x 2 y 2 22 / 49
25 Singular Value Decomposition (SVD)
26 Full SVD Given: A R m n Tall and skinny: m n ( ) 1 Σ A = U V T where Σ σ... 0 Short and fat: m n A = U ( Σ 0 ) V T where Σ σ 1... σ n 0 σ m 0 U R m m and V R n n are orthogonal matrices 24 / 49
27 Names and Conventions A R m n with SVD ( ) Σ A = U V T 0 or A = U ( Σ 0 ) V T Singular values (svalues): Diagonal elements σ j of Σ Left singular vector matrix: U Right singular vector matrix: V Svalue ordering σ 1 σ 2... σ min{m, n} 0 Svalues of matrices B, C: σ j (B), σ j (C) 25 / 49
28 SVD Properties A R m n Number of svalues equal to small dimension A R m n has min{m, n} singular values σ j 0 Orthogonal invariance If P R m m and Q R n n are orthogonal matrices then P A Q has same svalues as A Gram Product Nonzero svalues (= eigenvalues) of A T A are squares of svalues of A Inverse A R n n nonsingular σ j > 0 for 1 j n If A = U Σ V T then A 1 = V Σ 1 U T SVD of inverse 26 / 49
29 Exercises 1 Transpose A T has same singular values as A 2 Orthogonal matrices All singular values of A R n n are equal to 1 A is orthogonal matrix 3 If A R n n is symmetric and idempotent then all singular values are 0 and/or 1 4 For A R m n and α > 0 express svalues of (A T A + α I) 1 A T in terms of α and svalues of A ( ) 5 If A R m n In with m n then singular values of are A equal to 1 + σj 2 for 1 j n 27 / 49
30 Singular Values A R m n with svalues σ 1 σ p p min{m, n} Two norm A 2 = σ 1 Frobenius norm A F = σ σ2 p Well conditioned in absolute sense Product σ j (A) σ j (B) A B 2 1 j p σ j (A B) σ 1 (A) σ j (B) 1 j p 28 / 49
31 Matrix Schatten Norms A R m n, with singular values σ 1 σ ρ > 0, and integer p 0, the family of the Schatten p-norms is defined as ( ρ ) 1/p def A p = σ p i. i=1 Different than the vector-induced matrix p-norms 1. Schatten zero norm 2 : equal to the matrix rank. Schatten one norm: the sum of the singular values of the matrix, also called the nuclear norm. Schatten two norm: the Frobenius norm. Schatten infinity norm: the spectral (or two) norm. Schatten p-norms are unitarily invariant, submultiplicative, satisfy Hölder s inequality, etc. 1 Notation is, unfortunately, confusing. 2 Not really a norm / 49
32 Exercises 1 Norm of inverse If A R n n nonsingular with svalues σ 1 σ n then A 1 2 = 1/σ n 2 Appending a column to a tall and skinny matrix If A R m n with m > n, z R m, B = ( A z ) then σ n+1 (B) σ n (A) σ 1 (B) σ 1 (A) 3 Appending a row to a tall and skinny matrix If A R m n with m n, z R n, B T = ( A T z ) then σ n (B) σ n (A) σ 1 (A) σ 1 (B) σ 1 (A) 2 + z / 49
33 Rank rank(a) = number of nonzero (positive) svalues of A Zero matrix rank(0) = 0 Rank bounded by small dimension If A R m n then rank(a) min{m, n} Transpose rank(a T ) = rank(a) Gram product rank(a T A) = rank(a) = rank(aa T ) General product rank(a B) min{rank(a), rank(b)} If A nonsingular then rank(a B) = rank(b) 31 / 49
34 SVDs of Full Rank Matrices All svalues of A R m n are nonzero Full column-rank rank(a) = n ( ) Σ A = U V T Σ R n n nonsingular 0 {Linearly independent columns} Full row-rank rank(a) = m {Linearly independent rows} A = U ( Σ 0 ) V T Σ R m m nonsingular Nonsingular rank(a) = n = m {Lin. indep. rows & columns} A = U Σ V T Σ R n n nonsingular 32 / 49
35 Exercises 1 Rank of outer product If x R m and y R n then rank(xy T ) 1 2 If A R n n nonsingular then ( A B ) has full row-rank for any B R n k 3 Orthonormal matrices If A R m n has orthonormal columns then rank(a) = n and all svalues of A are equal to 1 4 Gram products For A R m n (i) rank(a) = n A T A nonsingular (ii) rank(a) = m AA T nonsingular 33 / 49
36 Thin SVD A R m n with A = U ( ) Σ V T or A = U ( Σ 0 ) V T 0 Singular values σ 1 σ p 0, p min{m, n} Singular vectors U = ( u 1 u m ) V = ( v 1 v n ) If rank(a) = r then thin (reduced) SVD {only non zero svalues} 1 A = ( u 1 ) u r σ... σ r v T 1. v T r = r σ j u j vj T j=1 34 / 49
37 Optimality of SVD Given A R m n, rank(a) = r, thin SVD r j=1 σ j u j v T j Approximation from k dominant svalues A k k σ j u j vj T j=1 1 k < r 35 / 49
38 Optimality of SVD Given A R m n, rank(a) = r, thin SVD r j=1 σ j u j v T j Approximation from k dominant svalues A k k σ j u j vj T j=1 1 k < r Absolute distance of A to set of rank-k matrices min rank(b)=k A B 2 = A A k 2 = σ k+1 min A B F = A A k F = rank(b)=k r j=k+1 σ 2 j 35 / 49
39 Exercises 1 Find an example to illustrate that a closest matrix of rank k is not unique in the two norm 36 / 49
40 Moore Penrose Inverse Given A R m n with rank(a) = r 1 and SVD A = U ( ) Σr 0 V T = 0 0 r σ j u j vj T j=1 Moore Penrose inverse ( ) A def Σ 1 r 0 = V U T = 0 0 r j=1 1 σ j v j u T j Zero matrix 0 m n = 0 n m 37 / 49
41 Special Cases of Moore Penrose Inverse Nonsingular If A R n n with rank(a) = n then A = A 1 Inverse A 1 A = I n = A A 1 Full column rank If A R m n with rank(a) = n then A = (A T A) 1 A T Left inverse A A = I n Full row rank If A R m n with rank(a) = m then A = A T (AA T ) 1 Right inverse A A = I m 38 / 49
42 Necessary and Sufficient Conditions A is Moore Penrose inverse of A A satisfies 1 A A A = A 2 A A A = A 3 (A A ) T = A A {symmetric} 4 (A A) T = A A {symmetric} 39 / 49
43 Exercises 1 If x R m and y R n with y 0 then x y 2 = x 2 / y 2 2 For A R m n the following matrices are idempotent: A A A A I m A A I n A A 3 If A R m n and A 0 then A A 2 = A A 2 = 1 4 If A R m n then (I m A A ) A = 0 m n A (I n A A) = 0 m n 5 If A R m n with rank(a) = n then (A T A) 1 2 = A If A = B C where B R m n has rank(b) = n and C R n n is nonsingular then A = C 1 B 40 / 49
44 Matrix Spaces and Singular Vectors: A A R m n Column space range(a) = {b : b = A x for some x R n } R m Null space (kernel) null(a) = {x : A x = 0} R n If rank(a) = r 1 A = ( ) ( ) ( ) Σ U r U r 0 V T r m r 0 0 Vn r T range(a) = range(u r ) null(a) = range(v n r ) 41 / 49
45 Matrix Spaces and Singular Vectors: A T A R m n Row space range(a T ) = {d : d = A T y for some y R m } R n Left null space null(a T ) = {y : A T y = 0} R m If rank(a) = r 1 A = ( ) ( ) ( ) Σ U r U r 0 V T r m r 0 0 Vn r T range(a T ) = range(v r ) null(a T ) = range(u m r ) 42 / 49
46 Fundamental Theorem of Linear Algebra A R m n range(a) null(a T ) = R m implies m = rank(a) + dim null(a T ) range(a) null(a T ) range(a T ) null(a) = R n implies n = rank(a) + dim null(a) range(a T ) null(a) 43 / 49
47 Spaces of the Moore Penrose Inverse {Need this for least squares} Column space range(a ) = range(a T A) = range(a T ) null(a) Null space null(a ) = null(a A T ) = null(a T ) range(a) 44 / 49
48 Least Squares (LS) Problems
49 General LS Problems Given A R m n and b R m min x Ax b 2 General LS solution y = A b + q for any q null(a) All solutions have same LS residual r b A y {A y = A A b since q null(a)} LS residual orthogonal to column space A T r = 0 LS solution of minimal two norm y = A b Computation: SVD 46 / 49
50 Exercises 1 Use properties of Moore Penrose inverses to show that the LS residual is orthogonal to the column space of A 2 Determine the minimal norm solution for min x A x b 2 if A = 0 m n 3 If y is the minimal norm solution to min x A x b 2 and A T b = 0, then what can you say about y? 4 Determine the minimal norm solution for min x A x b 2 if A = c d T where c R m and d R n? 47 / 49
51 Full Column Rank LS Problems Given A R m n with rank(a) = n and b R m min x Ax b 2 Unique LS solution y = A b Computation: QR decomposition 1 Factor A = Q R where Q T Q = I n and R is 2 Multiply c = Q T b 3 Solve R y = c Do NOT solve A T A y = A T b 48 / 49
52 Exercises 1 If A R m n with rank(a) = n has thin QR factorization A = Q R where Q T Q = I n and R is then A = R 1 Q T 2 If A R m n has orthonormal columns then A = A T 3 If A R m n has rank(a) = n then I m A A 2 = min{1, m n} 49 / 49
MATH36001 Generalized Inverses and the SVD 2015
MATH36001 Generalized Inverses and the SVD 201 1 Generalized Inverses of Matrices A matrix has an inverse only if it is square and nonsingular. However there are theoretical and practical applications
More informationb 1 b 2.. b = b m A = [a 1,a 2,...,a n ] where a 1,j a 2,j a j = a m,j Let A R m n and x 1 x 2 x = x n
Lectures -2: Linear Algebra Background Almost all linear and nonlinear problems in scientific computation require the use of linear algebra These lectures review basic concepts in a way that has proven
More informationChapter 1. Matrix Algebra
ST4233, Linear Models, Semester 1 2008-2009 Chapter 1. Matrix Algebra 1 Matrix and vector notation Definition 1.1 A matrix is a rectangular or square array of numbers of variables. We use uppercase boldface
More informationLinear Algebra (Review) Volker Tresp 2018
Linear Algebra (Review) Volker Tresp 2018 1 Vectors k, M, N are scalars A one-dimensional array c is a column vector. Thus in two dimensions, ( ) c1 c = c 2 c i is the i-th component of c c T = (c 1, c
More informationIndex. book 2009/5/27 page 121. (Page numbers set in bold type indicate the definition of an entry.)
page 121 Index (Page numbers set in bold type indicate the definition of an entry.) A absolute error...26 componentwise...31 in subtraction...27 normwise...31 angle in least squares problem...98,99 approximation
More informationChapter 0 Miscellaneous Preliminaries
EE 520: Topics Compressed Sensing Linear Algebra Review Notes scribed by Kevin Palmowski, Spring 2013, for Namrata Vaswani s course Notes on matrix spark courtesy of Brian Lois More notes added by Namrata
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationMath Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p
Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the
More informationσ 11 σ 22 σ pp 0 with p = min(n, m) The σ ii s are the singular values. Notation change σ ii A 1 σ 2
HE SINGULAR VALUE DECOMPOSIION he SVD existence - properties. Pseudo-inverses and the SVD Use of SVD for least-squares problems Applications of the SVD he Singular Value Decomposition (SVD) heorem For
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationELE/MCE 503 Linear Algebra Facts Fall 2018
ELE/MCE 503 Linear Algebra Facts Fall 2018 Fact N.1 A set of vectors is linearly independent if and only if none of the vectors in the set can be written as a linear combination of the others. Fact N.2
More informationSection 3.9. Matrix Norm
3.9. Matrix Norm 1 Section 3.9. Matrix Norm Note. We define several matrix norms, some similar to vector norms and some reflecting how multiplication by a matrix affects the norm of a vector. We use matrix
More informationReview of Some Concepts from Linear Algebra: Part 2
Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set
More informationLinear Algebra Formulas. Ben Lee
Linear Algebra Formulas Ben Lee January 27, 2016 Definitions and Terms Diagonal: Diagonal of matrix A is a collection of entries A ij where i = j. Diagonal Matrix: A matrix (usually square), where entries
More informationKnowledge Discovery and Data Mining 1 (VO) ( )
Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory
More informationReview of Basic Concepts in Linear Algebra
Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra
More informationMATH 240 Spring, Chapter 1: Linear Equations and Matrices
MATH 240 Spring, 2006 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 8th Ed. Sections 1.1 1.6, 2.1 2.2, 3.2 3.8, 4.3 4.5, 5.1 5.3, 5.5, 6.1 6.5, 7.1 7.2, 7.4 DEFINITIONS Chapter 1: Linear
More informationMath 407: Linear Optimization
Math 407: Linear Optimization Lecture 16: The Linear Least Squares Problem II Math Dept, University of Washington February 28, 2018 Lecture 16: The Linear Least Squares Problem II (Math Dept, University
More informationSingular Value Decompsition
Singular Value Decompsition Massoud Malek One of the most useful results from linear algebra, is a matrix decomposition known as the singular value decomposition It has many useful applications in almost
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationLinear Algebra (Review) Volker Tresp 2017
Linear Algebra (Review) Volker Tresp 2017 1 Vectors k is a scalar (a number) c is a column vector. Thus in two dimensions, c = ( c1 c 2 ) (Advanced: More precisely, a vector is defined in a vector space.
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2017 LECTURE 5
STAT 39: MATHEMATICAL COMPUTATIONS I FALL 17 LECTURE 5 1 existence of svd Theorem 1 (Existence of SVD) Every matrix has a singular value decomposition (condensed version) Proof Let A C m n and for simplicity
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationApplied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization
More informationDesigning Information Devices and Systems II
EECS 16B Fall 2016 Designing Information Devices and Systems II Linear Algebra Notes Introduction In this set of notes, we will derive the linear least squares equation, study the properties symmetric
More informationorthogonal relations between vectors and subspaces Then we study some applications in vector spaces and linear systems, including Orthonormal Basis,
5 Orthogonality Goals: We use scalar products to find the length of a vector, the angle between 2 vectors, projections, orthogonal relations between vectors and subspaces Then we study some applications
More informationPreliminary Linear Algebra 1. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 100
Preliminary Linear Algebra 1 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics 611 1 / 100 Notation for all there exists such that therefore because end of proof (QED) Copyright c 2012
More informationORTHOGONALITY AND LEAST-SQUARES [CHAP. 6]
ORTHOGONALITY AND LEAST-SQUARES [CHAP. 6] Inner products and Norms Inner product or dot product of 2 vectors u and v in R n : u.v = u 1 v 1 + u 2 v 2 + + u n v n Calculate u.v when u = 1 2 2 0 v = 1 0
More informationMATH 350: Introduction to Computational Mathematics
MATH 350: Introduction to Computational Mathematics Chapter V: Least Squares Problems Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2011 fasshauer@iit.edu MATH
More informationarxiv: v1 [cs.ds] 24 Dec 2017
Lectures on Randomized Numerical Linear Algebra * Petros Drineas Michael W. Mahoney arxiv:1712.08880v1 [cs.ds] 24 Dec 2017 Contents 1 Introduction 2 2 Linear Algebra 3 2.1 Basics..............................................
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationLOW-RANK MATRIX APPROXIMATIONS DO NOT NEED A SINGULAR VALUE GAP
LOW-RANK MATRIX APPROXIMATIONS DO NOT NEED A SINGULAR VALUE GAP PETROS DRINEAS AND ILSE C. F. IPSEN Abstract. Low-rank approximations to a real matrix A can be conputed from ZZ T A, where Z is a matrix
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More informationThe Singular Value Decomposition
The Singular Value Decomposition An Important topic in NLA Radu Tiberiu Trîmbiţaş Babeş-Bolyai University February 23, 2009 Radu Tiberiu Trîmbiţaş ( Babeş-Bolyai University)The Singular Value Decomposition
More informationELEMENTS OF MATRIX ALGEBRA
ELEMENTS OF MATRIX ALGEBRA CHUNG-MING KUAN Department of Finance National Taiwan University September 09, 2009 c Chung-Ming Kuan, 1996, 2001, 2009 E-mail: ckuan@ntuedutw; URL: homepagentuedutw/ ckuan CONTENTS
More informationMathematical Foundations of Applied Statistics: Matrix Algebra
Mathematical Foundations of Applied Statistics: Matrix Algebra Steffen Unkel Department of Medical Statistics University Medical Center Göttingen, Germany Winter term 2018/19 1/105 Literature Seber, G.
More informationHomework 2 Solutions
Homework 2 Solutions EE 263 Stanford University Summer 2018 July 5, 2018 1. Halfspace. Suppose a, b R n are two given points. Show that the set of points in R n that are closer to a than b is a halfspace,
More informationG1110 & 852G1 Numerical Linear Algebra
The University of Sussex Department of Mathematics G & 85G Numerical Linear Algebra Lecture Notes Autumn Term Kerstin Hesse (w aw S w a w w (w aw H(wa = (w aw + w Figure : Geometric explanation of the
More informationRecall the convention that, for us, all vectors are column vectors.
Some linear algebra Recall the convention that, for us, all vectors are column vectors. 1. Symmetric matrices Let A be a real matrix. Recall that a complex number λ is an eigenvalue of A if there exists
More informationCheat Sheet for MATH461
Cheat Sheet for MATH46 Here is the stuff you really need to remember for the exams Linear systems Ax = b Problem: We consider a linear system of m equations for n unknowns x,,x n : For a given matrix A
More informationDot product and linear least squares problems
Dot product and linear least squares problems Dot Product For vectors u,v R n we define the dot product Note that we can also write this as u v = u,,u n u v = u v + + u n v n v v n = u v + + u n v n The
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationReview of Linear Algebra
Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=
More informationOrthogonalization and least squares methods
Chapter 3 Orthogonalization and least squares methods 31 QR-factorization (QR-decomposition) 311 Householder transformation Definition 311 A complex m n-matrix R = [r ij is called an upper (lower) triangular
More informationMAT 610: Numerical Linear Algebra. James V. Lambers
MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic
More informationThe Singular Value Decomposition and Least Squares Problems
The Singular Value Decomposition and Least Squares Problems Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 27, 2009 Applications of SVD solving
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationlinearly indepedent eigenvectors as the multiplicity of the root, but in general there may be no more than one. For further discussion, assume matrice
3. Eigenvalues and Eigenvectors, Spectral Representation 3.. Eigenvalues and Eigenvectors A vector ' is eigenvector of a matrix K, if K' is parallel to ' and ' 6, i.e., K' k' k is the eigenvalue. If is
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationLinear Algebra Methods for Data Mining
Linear Algebra Methods for Data Mining Saara Hyvönen, Saara.Hyvonen@cs.helsinki.fi Spring 2007 1. Basic Linear Algebra Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki Example
More informationThe University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.
The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational
More informationLinear Algebra, part 3. Going back to least squares. Mathematical Models, Analysis and Simulation = 0. a T 1 e. a T n e. Anna-Karin Tornberg
Linear Algebra, part 3 Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2010 Going back to least squares (Sections 1.7 and 2.3 from Strang). We know from before: The vector
More informationVector Spaces, Orthogonality, and Linear Least Squares
Week Vector Spaces, Orthogonality, and Linear Least Squares. Opening Remarks.. Visualizing Planes, Lines, and Solutions Consider the following system of linear equations from the opener for Week 9: χ χ
More informationMATH 612 Computational methods for equation solving and function minimization Week # 2
MATH 612 Computational methods for equation solving and function minimization Week # 2 Instructor: Francisco-Javier Pancho Sayas Spring 2014 University of Delaware FJS MATH 612 1 / 38 Plan for this week
More informationLarge Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset
More informationBasic Concepts in Matrix Algebra
Basic Concepts in Matrix Algebra An column array of p elements is called a vector of dimension p and is written as x p 1 = x 1 x 2. x p. The transpose of the column vector x p 1 is row vector x = [x 1
More informationLinear Algebra in Actuarial Science: Slides to the lecture
Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More information1 Linear Algebra Problems
Linear Algebra Problems. Let A be the conjugate transpose of the complex matrix A; i.e., A = A t : A is said to be Hermitian if A = A; real symmetric if A is real and A t = A; skew-hermitian if A = A and
More informationNotes on Linear Algebra and Matrix Theory
Massimo Franceschet featuring Enrico Bozzo Scalar product The scalar product (a.k.a. dot product or inner product) of two real vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) is not a vector but a
More informationbe a Householder matrix. Then prove the followings H = I 2 uut Hu = (I 2 uu u T u )u = u 2 uut u
MATH 434/534 Theoretical Assignment 7 Solution Chapter 7 (71) Let H = I 2uuT Hu = u (ii) Hv = v if = 0 be a Householder matrix Then prove the followings H = I 2 uut Hu = (I 2 uu )u = u 2 uut u = u 2u =
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationLeast Squares. Tom Lyche. October 26, Centre of Mathematics for Applications, Department of Informatics, University of Oslo
Least Squares Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo October 26, 2010 Linear system Linear system Ax = b, A C m,n, b C m, x C n. under-determined
More informationLecture II: Linear Algebra Revisited
Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function
More informationThe Singular Value Decomposition
CHAPTER 6 The Singular Value Decomposition Exercise 67: SVD examples (a) For A =[, 4] T we find a matrixa T A =5,whichhastheeigenvalue =5 Thisprovidesuswiththesingularvalue =+ p =5forA Hence the matrix
More informationLinGloss. A glossary of linear algebra
LinGloss A glossary of linear algebra Contents: Decompositions Types of Matrices Theorems Other objects? Quasi-triangular A matrix A is quasi-triangular iff it is a triangular matrix except its diagonal
More informationIr O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )
Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O
More information5. Orthogonal matrices
L Vandenberghe EE133A (Spring 2017) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal
More informationOrthonormal Bases; Gram-Schmidt Process; QR-Decomposition
Orthonormal Bases; Gram-Schmidt Process; QR-Decomposition MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 205 Motivation When working with an inner product space, the most
More informationMAT Linear Algebra Collection of sample exams
MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system
More informationSUMMARY OF MATH 1600
SUMMARY OF MATH 1600 Note: The following list is intended as a study guide for the final exam. It is a continuation of the study guide for the midterm. It does not claim to be a comprehensive list. You
More informationUNIT 6: The singular value decomposition.
UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T
More informationBasic Elements of Linear Algebra
A Basic Review of Linear Algebra Nick West nickwest@stanfordedu September 16, 2010 Part I Basic Elements of Linear Algebra Although the subject of linear algebra is much broader than just vectors and matrices,
More informationMatrix decompositions
Matrix decompositions Zdeněk Dvořák May 19, 2015 Lemma 1 (Schur decomposition). If A is a symmetric real matrix, then there exists an orthogonal matrix Q and a diagonal matrix D such that A = QDQ T. The
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions
More informationSpring 2014 Math 272 Final Exam Review Sheet
Spring 2014 Math 272 Final Exam Review Sheet You will not be allowed use of a calculator or any other device other than your pencil or pen and some scratch paper. Notes are also not allowed. In kindness
More informationEE731 Lecture Notes: Matrix Computations for Signal Processing
EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten
More informationLecture Notes to be used in conjunction with. 233 Computational Techniques
Lecture Notes to be used in conjunction with 233 Computational Techniques István Maros Department of Computing Imperial College London V29e January 2008 CONTENTS i Contents 1 Introduction 1 2 Computation
More informationDS-GA 1002 Lecture notes 10 November 23, Linear models
DS-GA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.
More informationLecture Notes to be used in conjunction with. 233 Computational Techniques
Lecture Notes to be used in conjunction with 233 Computational Techniques István Maros Department of Computing Imperial College London V2.9e January 2008 CONTENTS i Contents 1 Introduction 1 2 Computation
More information1 9/5 Matrices, vectors, and their applications
1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric
More informationProblem set 5: SVD, Orthogonal projections, etc.
Problem set 5: SVD, Orthogonal projections, etc. February 21, 2017 1 SVD 1. Work out again the SVD theorem done in the class: If A is a real m n matrix then here exist orthogonal matrices such that where
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationPart 1a: Inner product, Orthogonality, Vector/Matrix norm
Part 1a: Inner product, Orthogonality, Vector/Matrix norm September 19, 2018 Numerical Linear Algebra Part 1a September 19, 2018 1 / 16 1. Inner product on a linear space V over the number field F A map,
More informationLecture 5. Ch. 5, Norms for vectors and matrices. Norms for vectors and matrices Why?
KTH ROYAL INSTITUTE OF TECHNOLOGY Norms for vectors and matrices Why? Lecture 5 Ch. 5, Norms for vectors and matrices Emil Björnson/Magnus Jansson/Mats Bengtsson April 27, 2016 Problem: Measure size of
More informationSTAT200C: Review of Linear Algebra
Stat200C Instructor: Zhaoxia Yu STAT200C: Review of Linear Algebra 1 Review of Linear Algebra 1.1 Vector Spaces, Rank, Trace, and Linear Equations 1.1.1 Rank and Vector Spaces Definition A vector whose
More informationLinear Algebra, part 3 QR and SVD
Linear Algebra, part 3 QR and SVD Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2012 Going back to least squares (Section 1.4 from Strang, now also see section 5.2). We
More informationLecture 6: Geometry of OLS Estimation of Linear Regession
Lecture 6: Geometry of OLS Estimation of Linear Regession Xuexin Wang WISE Oct 2013 1 / 22 Matrix Algebra An n m matrix A is a rectangular array that consists of nm elements arranged in n rows and m columns
More informationMultiplicative Perturbation Bounds of the Group Inverse and Oblique Projection
Filomat 30: 06, 37 375 DOI 0.98/FIL67M Published by Faculty of Sciences Mathematics, University of Niš, Serbia Available at: http://www.pmf.ni.ac.rs/filomat Multiplicative Perturbation Bounds of the Group
More informationAPPLIED LINEAR ALGEBRA
APPLIED LINEAR ALGEBRA Giorgio Picci November 24, 2015 1 Contents 1 LINEAR VECTOR SPACES AND LINEAR MAPS 10 1.1 Linear Maps and Matrices........................... 11 1.2 Inverse of a Linear Map............................
More information