12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x)
|
|
- Dayna Osborne
- 5 years ago
- Views:
Transcription
1 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 11 The estimate ^ is unbiased, but E(^ 2 ) = n?1 n 2 and is thus biased. An unbiased estimate is ^ 2 = 1 (x i? ^) 2 : n? 1 In x?? we show that the linear least squares problem arises out of a maximum likelihood estimation problem Inner Products,Vector Norms, and Matrix Norms We briey review basic results for inner products, vector norms, and matrix norms. In x1.4.1 basic properties of norms and inner products are given. In x1.4.2, we discusses the properties of the norms that are based on the Euclidean norm Basic Denitions and Lemmas First, we give the denition of an inner product. Denition An inner product in < n is a function (; ) mapping < n < n into < n that satises the following four axioms 1. (x; x) 0; (x; x) = 0; if and only if x = 0; x 2 < n, 2. (x; y) = (x; y); x; y 2 < n ; 2 <, 3. (x; y) = (y; x); x; y 2 < n, 4. (x + z; y) = (x; y) + (z; y); x; y; z 2 < n. We note that if we dene an inner product for complex vectors, the third axiom becomes (x; y) = (y; x) x; y 2 C n (1.5) where a denotes the complex conjugate of a. The inner product that is used most often is the Euclidean dot product (x; y) = x T y = x i y i : Others are introduced in Chapter??. The Cauchy-Schwarz inequality given below is quite useful for all inner products.
2 12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x) 1=2 (y; y) 1=2 : (1.6) Moreover, equality in (1.6) holds if and only if x = y for some 2 <. This inequality leads to the following denition of the angle between two vectors relative to an inner product. Denition The angle between the two nonzero vectors x; y 2 < n with respect to an inner product (; ) is given by (x; y) cos = (x; x) 1=2 (y; y) 1=2 : (1.7) The next important denition is that of a vector norm. Denition A norm in < n is a function k k mapping < n into < satisfying the following three axioms 1. kxk 0; kxk = 0 if and only if x = 0; x 2 < n. 2. kxk = jjkxk; x 2 < n ; 2 <. 3. kx + yk kxk + kyk; x; y 2 < n : Denition and the Cauchy-Schwarz inequality give us that for any inner product (; ) we can dene a norm k k by kxk = (x; x) 1=2 : (1.8) The most well known of these norms is the Euclidean norm given by! 1=2 kxk2 =? x T x 1=2 = n X x 2 i : (1.9) It is also referred to as the two-norm. This is the most important vector norm in the book. The subscript \2" is explained below. The two-norm is one of the class of p-norms. These are given by kxk p = ( jx i j p ) 1=p ; p 1: (1.10)
3 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 13 Clearly, p = 2 leads to the denition (1.9). Except for the two-norm, the only norms we will use from this class are the one-norm given by and the 1-norm given by kxk1 = jx i j; (1.11) kxk 1 = max 1in jx ij = lim p!1 kxk p: (1.12) The next lemma, the Holder inequality, relates the p-norms to the Euclidean inner product. Lemma 1.4 (Holder inequality) Let k k p and k k q be norms in < n from the class (1.10) such that p?1 + q?1 = 1. Then for all x; y 2 < n we have jx T yj kxk p kyk q : (1.13) The interesting cases for us are p = q = 2, p = 1; q = 1, and p = 1; q = 1. For p = q = 2, the Holder inequality is just the Cauchy-Schwarz inequality. The next inequality states that if we can bound kxk, for a given x 2 < n in any norm kk, the quantity kxk can be bounded in any other norm kk. Lemma 1.5 All norms on < n are uniformly equivalent, meaning that for any two norms k k and k k there are constants c1 and c2 such that for all x 2 < n. c1kxk kxk c2kxk (1.14) For the two-norm, the one-norm, and the 1?norm the uniform equivalence relations are summarized by 1 p n kxk1 kxk2 kxk1; (1.15) We will also need norms for matrices. kxk 1 kxk2 p nkxk 1 ; (1.16) kxk 1 kxk1 nkxk 1 : (1.17) Denition A norm in < mn is a function k k mapping < mn into < satisfying the following three axioms
4 14 CHAPTER 1. PRELIMINARIES 1. kxk 0; kxk = 0 if and only if X = 0; X 2 < mn 2. kxk = jjkxk X 2 < mn ; 2 < 3. kx + Y k kxk + ky k X; Y 2 < mn. This denition is isomorphic to the denition of a vector norm on < mn. For example, the Frobenius norm dened by kxk F = m X x 2 ij j=1 1 A 1=2 is isomorphic to the two-norm on < mn. For semantic reasons, we dene a family of norms. (1.18) Denition Let k k ;m;n be a norm on < mn and let it be well-dened for every positive integer m and n. Then the set N = fk k ;m;n : m; n positive integers g is called a family of norms. For any positive integers m and n, and any X 2 < mn, we denote kxk ;m;n by kxk. That is, for any matrix X, the quantity kxk is the appropriate member of N applied to X. The set V N given by V = fk k ;m;1 : m positive integer g is the associated family of vector norms. For any x 2 < m, the quantity kxk = k(x)k is the appropriate member of V applied to x. Since X represents a linear operator from < n to < m, it is appropriate to dene the induced norm k k on < mn by It is a simple matter to show that kxyk kxk = sup : (1.19) y6=0 kyk kxk = max kyk =1 kxyk : (1.20) Note that the maximum is taken over a closed, bounded set, thus we have that kxk = kxy k (1.21)
5 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 15 for some y such that ky k = 1. The above denition leads to the very useful bound kxyk kxk kyk (1.22) where equality occurs for every vector of the form y ; 2 <. If X 2 < m1 then (1.22) appears to give a second denition for kxk. However, it is easily veried that if X = (x), for some x 2 < m, then kxk, dened from the matrix norm (1.22), and kxk, dened from the vector norm, are the same value. Thus the notation k k is unambiguous for all m and n. For any induced norm k k, the identity matrix I n for < nn satises However, for the Frobenius norm ki n k = 1: (1.23) ki n k F = p n; thus it is not an induced norm for any vector norm. For the one-norm and the 1-norm there are formulas for the corresponding matrix norms and for a vector y satisfying (1.21). The one-norm formula is kxk1 = max 1jn If j max is the index of a column such that mx mx kxk1 = jx i;jmax j jx ij j: (1.24) then y = e jmax, the corresponding column of the identity matrix. The 1-norm formula is kxk 1 = max If i max is the index of a row such that 1im kxk 1 = j=1 jx ij j: (1.25) j=1 jx imax;jj then the vector y = (y 1 ; : : : ; y n) T with components y j = sign(x imax;j) satises (1.21). Note that kxk 1 = kx T k1.
6 16 CHAPTER 1. PRELIMINARIES The matrix two-norm does not have a formula like (1.24) or (1.25) and all other formulations are really equivalent to (1.19). Moreover, computing the vector y in (1.21) is a nontrivial task that we will discuss in Chapters?? and??. The induced norms have a convenient property that is important in understanding matrix computations. For X 2 < mn and Y 2 < ns consider kxy k. We have that Thus kxy k = max kzk =1 kxy zk max kzk =1 kxk ky zk = kxk max kzk =1 ky zk = kxk ky k : kxy k kxk ky k : (1.26) A norm k k (or really family of norms) that satises the property (1.26) is said to be consistent. Since they are induced norms the two-norm, one-norm, and the 1-norm are all consistent. The Frobenius norm also satises (1.26). An example of a matrix norm that is not consistent is given below. Example 1.1 Consider the norm k k on < mn given by kxk = max (i;j) jx ijj: This is simply the 1-norm applied to X written out as vector in < mn. For m = n = 2, consider 1 1 X = Y = : 1 1 Note that 2 2 XY = 2 2 and thus kxy k = 2 > kxk ky k = 1. Clearly, kk norm is not consistent. Henceforth, we use only consistent families of norms. The above dened matrix norms satisfy the following useful inequalities for all X 2 < mn and Y 2 < ns : (mn)? 1 4 (kxk 1kXk 1 ) 1=2 kxk2 (kxk1kxk 1 ) 1=2 ; (1.27) 1 1 max pn ; p kxk F kxk2 kxk F ; (1.28) m kxy k F kxk2ky k F ; (1.29) kxy k F kxk F ky k2: (1.30)
7 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 17 For X 6= 0, the lower bound in (1.28) can be tightened into 1 p rank(x) kxk F kxk2: (1.31) Since rank(x) is often more expensive to compute than a good approximation to kxk2, the practical use of (1.31) is limited. For any diagonal matrix = diag() and any p-norm k k p kk p = kk 1 = max 1in j ij 1 p 1: For any matrix X 2 < mn we have the following relations for the matrix jxj: kxk2 k jxj k2 p rank(x)kxk2 minf p m; p ngkxk2; kxk1 = k jxj k1; kxk 1 = k jxj k 1 ; kxk F = k jxj k F : In the context of linear least squares, our interest will be in the two-norm or in the Frobenius norm. The other norms will be used to bound them. Some special properties of these norms are given in the next section The Two-Norm, the Frobenius Norm, and Orthogonality We begin by dening orthogonality. We then relate orthogonality to the matrix two-norm and Frobenius norm. Denition Two vectors x; y 2 < n are orthogonal with respect to an inner product (; ) if (x; y) = 0. The set S1 < n is orthogonal to the set S2 < n, if for each x 2 S1 and y 2 S2, we have (x; y) = 0. We write x? y; S1? S2 to mean \x is orthogonal to y and \S1 is orthogonal to S2." The zero vector is orthogonal to any vector and the set f0g is orthogonal to any set. An orthogonal set of vectors is dened as follows. Denition A set of k vectors fx1; x2; : : : ; x k g, where each x i 2 < n, is said to be an orthogonal with respect to the inner product (; ) if (x i ; x j ) = 0 for i 6= j. The set is said to be orthonormal if it is orthogonal and (x i ; x i ) = 1 for i = 1; 2; : : : ; k The denition of an orthogonal matrix is related to the denition for vectors.
Lecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set.
Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. Definition A set of k vectors {u, u 2,..., u k }, where each u i R n, is said to be an
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationINNER PRODUCT SPACE. Definition 1
INNER PRODUCT SPACE Definition 1 Suppose u, v and w are all vectors in vector space V and c is any scalar. An inner product space on the vectors space V is a function that associates with each pair of
More informationInner products and Norms. Inner product of 2 vectors. Inner product of 2 vectors x and y in R n : x 1 y 1 + x 2 y x n y n in R n
Inner products and Norms Inner product of 2 vectors Inner product of 2 vectors x and y in R n : x 1 y 1 + x 2 y 2 + + x n y n in R n Notation: (x, y) or y T x For complex vectors (x, y) = x 1 ȳ 1 + x 2
More information3 a 21 a a 2N. 3 a 21 a a 2M
APPENDIX: MATHEMATICS REVIEW G 12.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers 2 A = 6 4 a 11 a 12... a 1N a 21 a 22... a 2N. 7..... 5 a M1 a M2...
More informationj=1 [We will show that the triangle inequality holds for each p-norm in Chapter 3 Section 6.] The 1-norm is A F = tr(a H A).
Math 344 Lecture #19 3.5 Normed Linear Spaces Definition 3.5.1. A seminorm on a vector space V over F is a map : V R that for all x, y V and for all α F satisfies (i) x 0 (positivity), (ii) αx = α x (scale
More informationLinear Algebra: Characteristic Value Problem
Linear Algebra: Characteristic Value Problem . The Characteristic Value Problem Let < be the set of real numbers and { be the set of complex numbers. Given an n n real matrix A; does there exist a number
More informationProf. M. Saha Professor of Mathematics The University of Burdwan West Bengal, India
CHAPTER 9 BY Prof. M. Saha Professor of Mathematics The University of Burdwan West Bengal, India E-mail : mantusaha.bu@gmail.com Introduction and Objectives In the preceding chapters, we discussed normed
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationA theorem on summable families in normed groups. Dedicated to the professors of mathematics. L. Berg, W. Engel, G. Pazderski, and H.- W. Stolle.
Rostock. Math. Kolloq. 49, 51{56 (1995) Subject Classication (AMS) 46B15, 54A20, 54E15 Harry Poppe A theorem on summable families in normed groups Dedicated to the professors of mathematics L. Berg, W.
More informationChater Matrix Norms and Singular Value Decomosition Introduction In this lecture, we introduce the notion of a norm for matrices The singular value de
Lectures on Dynamic Systems and Control Mohammed Dahleh Munther A Dahleh George Verghese Deartment of Electrical Engineering and Comuter Science Massachuasetts Institute of Technology c Chater Matrix Norms
More informationHilbert Spaces. Hilbert space is a vector space with some extra structure. We start with formal (axiomatic) definition of a vector space.
Hilbert Spaces Hilbert space is a vector space with some extra structure. We start with formal (axiomatic) definition of a vector space. Vector Space. Vector space, ν, over the field of complex numbers,
More informationFunctional Analysis: Assignment Set # 3 Spring 2009 Professor: Fengbo Hang February 25, 2009
duardo Corona Functional Analysis: Assignment Set # 3 Spring 9 Professor: Fengbo Hang February 5, 9 C6. Show that a norm that satis es the parallelogram identity: comes from a scalar product. kx + yk +
More informationRank, Trace, Determinant, Transpose an Inverse of a Matrix Let A be an n n square matrix: A = a11 a1 a1n a1 a an a n1 a n a nn nn where is the jth col
Review of Linear Algebra { E18 Hanout Vectors an Their Inner Proucts Let X an Y be two vectors: an Their inner prouct is ene as X =[x1; ;x n ] T Y =[y1; ;y n ] T (X; Y ) = X T Y = x k y k k=1 where T an
More informationthen kaxk 1 = j a ij x j j ja ij jjx j j: Changing the order of summation, we can separate the summands, kaxk 1 ja ij jjx j j: let then c = max 1jn ja
Homework Haimanot Kassa, Jeremy Morris & Isaac Ben Jeppsen October 7, 004 Exercise 1 : We can say that kxk = kx y + yk And likewise So we get kxk kx yk + kyk kxk kyk kx yk kyk = ky x + xk kyk ky xk + kxk
More informationVector spaces. DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis.
Vector spaces DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Vector space Consists of: A set V A scalar
More informationMath 413/513 Chapter 6 (from Friedberg, Insel, & Spence)
Math 413/513 Chapter 6 (from Friedberg, Insel, & Spence) David Glickenstein December 7, 2015 1 Inner product spaces In this chapter, we will only consider the elds R and C. De nition 1 Let V be a vector
More informationLinear Algebra (Review) Volker Tresp 2017
Linear Algebra (Review) Volker Tresp 2017 1 Vectors k is a scalar (a number) c is a column vector. Thus in two dimensions, c = ( c1 c 2 ) (Advanced: More precisely, a vector is defined in a vector space.
More informationIntroduction and Preliminaries
Chapter 1 Introduction and Preliminaries This chapter serves two purposes. The first purpose is to prepare the readers for the more systematic development in later chapters of methods of real analysis
More informationReview of Linear Algebra
Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=
More informationPart 1a: Inner product, Orthogonality, Vector/Matrix norm
Part 1a: Inner product, Orthogonality, Vector/Matrix norm September 19, 2018 Numerical Linear Algebra Part 1a September 19, 2018 1 / 16 1. Inner product on a linear space V over the number field F A map,
More informationL p Functions. Given a measure space (X, µ) and a real number p [1, ), recall that the L p -norm of a measurable function f : X R is defined by
L p Functions Given a measure space (, µ) and a real number p [, ), recall that the L p -norm of a measurable function f : R is defined by f p = ( ) /p f p dµ Note that the L p -norm of a function f may
More informationChapter 1. Preliminaries. The purpose of this chapter is to provide some basic background information. Linear Space. Hilbert Space.
Chapter 1 Preliminaries The purpose of this chapter is to provide some basic background information. Linear Space Hilbert Space Basic Principles 1 2 Preliminaries Linear Space The notion of linear space
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationDot product and linear least squares problems
Dot product and linear least squares problems Dot Product For vectors u,v R n we define the dot product Note that we can also write this as u v = u,,u n u v = u v + + u n v n v v n = u v + + u n v n The
More informationVector Spaces. Vector space, ν, over the field of complex numbers, C, is a set of elements a, b,..., satisfying the following axioms.
Vector Spaces Vector space, ν, over the field of complex numbers, C, is a set of elements a, b,..., satisfying the following axioms. For each two vectors a, b ν there exists a summation procedure: a +
More informationChapter 4 Euclid Space
Chapter 4 Euclid Space Inner Product Spaces Definition.. Let V be a real vector space over IR. A real inner product on V is a real valued function on V V, denoted by (, ), which satisfies () (x, y) = (y,
More informationIntroduction to Numerical Linear Algebra II
Introduction to Numerical Linear Algebra II Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA 1 / 49 Overview We will cover this material in
More informationProjection. Ping Yu. School of Economics and Finance The University of Hong Kong. Ping Yu (HKU) Projection 1 / 42
Projection Ping Yu School of Economics and Finance The University of Hong Kong Ping Yu (HKU) Projection 1 / 42 1 Hilbert Space and Projection Theorem 2 Projection in the L 2 Space 3 Projection in R n Projection
More informationLecture Notes 1: Vector spaces
Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More information1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)
Notes for 2017-01-30 Most of mathematics is best learned by doing. Linear algebra is no exception. You have had a previous class in which you learned the basics of linear algebra, and you will have plenty
More informationFinite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product
Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )
More informationELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS. 1. Linear Equations and Matrices
ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS KOLMAN & HILL NOTES BY OTTO MUTZBAUER 11 Systems of Linear Equations 1 Linear Equations and Matrices Numbers in our context are either real numbers or complex
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More informationNorms and Perturbation theory for linear systems
CHAPTER 7 Norms and Perturbation theory for linear systems Exercise 7.7: Consistency of sum norm? Observe that the sum norm is a matrix norm. This follows since it is equal to the l -norm of the vector
More informationVectors in Function Spaces
Jim Lambers MAT 66 Spring Semester 15-16 Lecture 18 Notes These notes correspond to Section 6.3 in the text. Vectors in Function Spaces We begin with some necessary terminology. A vector space V, also
More informationMTH 503: Functional Analysis
MTH 53: Functional Analysis Semester 1, 215-216 Dr. Prahlad Vaidyanathan Contents I. Normed Linear Spaces 4 1. Review of Linear Algebra........................... 4 2. Definition and Examples...........................
More information1 Linear Algebra Problems
Linear Algebra Problems. Let A be the conjugate transpose of the complex matrix A; i.e., A = A t : A is said to be Hermitian if A = A; real symmetric if A is real and A t = A; skew-hermitian if A = A and
More informationInner Product and Orthogonality
Inner Product and Orthogonality P. Sam Johnson October 3, 2014 P. Sam Johnson (NITK) Inner Product and Orthogonality October 3, 2014 1 / 37 Overview In the Euclidean space R 2 and R 3 there are two concepts,
More informationReview of Basic Concepts in Linear Algebra
Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra
More informationIntroduction to Linear Algebra. Tyrone L. Vincent
Introduction to Linear Algebra Tyrone L. Vincent Engineering Division, Colorado School of Mines, Golden, CO E-mail address: tvincent@mines.edu URL: http://egweb.mines.edu/~tvincent Contents Chapter. Revew
More informationFunctional Analysis Exercise Class
Functional Analysis Exercise Class Week 9 November 13 November Deadline to hand in the homeworks: your exercise class on week 16 November 20 November Exercises (1) Show that if T B(X, Y ) and S B(Y, Z)
More informationThe Transpose of a Vector
8 CHAPTER Vectors The Transpose of a Vector We now consider the transpose of a vector in R n, which is a row vector. For a vector u 1 u. u n the transpose is denoted by u T = [ u 1 u u n ] EXAMPLE -5 Find
More informationMethods of Mathematical Physics X1 Homework 3 Solutions
Methods of Mathematical Physics - 556 X Homework 3 Solutions. (Problem 2.. from Keener.) Verify that l 2 is an inner product space. Specifically, show that if x, y l 2, then x, y x k y k is defined and
More informationContents. Appendix D (Inner Product Spaces) W-51. Index W-63
Contents Appendix D (Inner Product Spaces W-5 Index W-63 Inner city space W-49 W-5 Chapter : Appendix D Inner Product Spaces The inner product, taken of any two vectors in an arbitrary vector space, generalizes
More informationsatisfying ( i ; j ) = ij Here ij = if i = j and 0 otherwise The idea to use lattices is the following Suppose we are given a lattice L and a point ~x
Dual Vectors and Lower Bounds for the Nearest Lattice Point Problem Johan Hastad* MIT Abstract: We prove that given a point ~z outside a given lattice L then there is a dual vector which gives a fairly
More informationBasic Calculus Review
Basic Calculus Review Lorenzo Rosasco ISML Mod. 2 - Machine Learning Vector Spaces Functionals and Operators (Matrices) Vector Space A vector space is a set V with binary operations +: V V V and : R V
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More informationLecture 23: 6.1 Inner Products
Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such
More informationSection 3.9. Matrix Norm
3.9. Matrix Norm 1 Section 3.9. Matrix Norm Note. We define several matrix norms, some similar to vector norms and some reflecting how multiplication by a matrix affects the norm of a vector. We use matrix
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More information1. Subspaces A subset M of Hilbert space H is a subspace of it is closed under the operation of forming linear combinations;i.e.,
Abstract Hilbert Space Results We have learned a little about the Hilbert spaces L U and and we have at least defined H 1 U and the scale of Hilbert spaces H p U. Now we are going to develop additional
More informationIr O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )
Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O
More informationChapter 3 Least Squares Solution of y = A x 3.1 Introduction We turn to a problem that is dual to the overconstrained estimation problems considered s
Lectures on Dynamic Systems and Control Mohammed Dahleh Munther A. Dahleh George Verghese Department of Electrical Engineering and Computer Science Massachuasetts Institute of Technology 1 1 c Chapter
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationSection 7.5 Inner Product Spaces
Section 7.5 Inner Product Spaces With the dot product defined in Chapter 6, we were able to study the following properties of vectors in R n. ) Length or norm of a vector u. ( u = p u u ) 2) Distance of
More informationA Review of Linear Algebra
A Review of Linear Algebra Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab: Implementations
More informationDistances and similarities Based in part on slides from textbook, slides of Susan Holmes. October 3, Statistics 202: Data Mining
Distances and similarities Based in part on slides from textbook, slides of Susan Holmes October 3, 2012 1 / 1 Similarities Start with X which we assume is centered and standardized. The PCA loadings were
More informationLecture 1 Introduction
L. Vandenberghe EE236A (Fall 2013-14) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation
More information1 More concise proof of part (a) of the monotone convergence theorem.
Math 0450 Honors intro to analysis Spring, 009 More concise proof of part (a) of the monotone convergence theorem. Theorem If (x n ) is a monotone and bounded sequence, then lim (x n ) exists. Proof. (a)
More informationElementary Linear Algebra
Instructor s Manual with Sample Tests for Elementary Linear Algebra Fourth Edition Stephen Andrilli and David Hecker Dedication To all the instructors who have used the various editions of our book over
More informationLINEAR ALGEBRA W W L CHEN
LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008. This chapter is available free to all individuals, on the understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,
More informationMath Introduction to Numerical Analysis - Class Notes. Fernando Guevara Vasquez. Version Date: January 17, 2012.
Math 5620 - Introduction to Numerical Analysis - Class Notes Fernando Guevara Vasquez Version 1990. Date: January 17, 2012. 3 Contents 1. Disclaimer 4 Chapter 1. Iterative methods for solving linear systems
More informationMetric Spaces. DEF. If (X; d) is a metric space and E is a nonempty subset, then (E; d) is also a metric space, called a subspace of X:
Metric Spaces DEF. A metric space X or (X; d) is a nonempty set X together with a function d : X X! [0; 1) such that for all x; y; and z in X : 1. d (x; y) 0 with equality i x = y 2. d (x; y) = d (y; x)
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationLOWELL WEEKLY JOURNAL
KY Y 872 K & q $ < 9 2 q 4 8 «7 K K K «> 2 26 8 5 4 4 7»» 2 & K q 4 [«5 «$6 q X «K «8K K88 K 7 ««$25 K Q ««q 8 K K Y & 7K /> Y 8«#»«Y 87 8 Y 4 KY «7««X & Y» K ) K K 5 KK K > K» Y Y 8 «KK > /» >» 8 K X
More informationFunctional Analysis Review
Functional Analysis Review Lorenzo Rosasco slides courtesy of Andre Wibisono 9.520: Statistical Learning Theory and Applications September 9, 2013 1 2 3 4 Vector Space A vector space is a set V with binary
More informationThe following definition is fundamental.
1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic
More informationMathematical Methods wk 1: Vectors
Mathematical Methods wk : Vectors John Magorrian, magog@thphysoxacuk These are work-in-progress notes for the second-year course on mathematical methods The most up-to-date version is available from http://www-thphysphysicsoxacuk/people/johnmagorrian/mm
More informationMathematical Methods wk 1: Vectors
Mathematical Methods wk : Vectors John Magorrian, magog@thphysoxacuk These are work-in-progress notes for the second-year course on mathematical methods The most up-to-date version is available from http://www-thphysphysicsoxacuk/people/johnmagorrian/mm
More informationQUASI-UNIFORMLY POSITIVE OPERATORS IN KREIN SPACE. Denitizable operators in Krein spaces have spectral properties similar to those
QUASI-UNIFORMLY POSITIVE OPERATORS IN KREIN SPACE BRANKO CURGUS and BRANKO NAJMAN Denitizable operators in Krein spaces have spectral properties similar to those of selfadjoint operators in Hilbert spaces.
More informationAN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES
AN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES JOEL A. TROPP Abstract. We present an elementary proof that the spectral radius of a matrix A may be obtained using the formula ρ(a) lim
More informationLinear Analysis Lecture 5
Linear Analysis Lecture 5 Inner Products and V Let dim V < with inner product,. Choose a basis B and let v, w V have coordinates in F n given by x 1. x n and y 1. y n, respectively. Let A F n n be the
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationLinear Algebra (Review) Volker Tresp 2018
Linear Algebra (Review) Volker Tresp 2018 1 Vectors k, M, N are scalars A one-dimensional array c is a column vector. Thus in two dimensions, ( ) c1 c = c 2 c i is the i-th component of c c T = (c 1, c
More informationFunctional Analysis MATH and MATH M6202
Functional Analysis MATH 36202 and MATH M6202 1 Inner Product Spaces and Normed Spaces Inner Product Spaces Functional analysis involves studying vector spaces where we additionally have the notion of
More information(, ) : R n R n R. 1. It is bilinear, meaning it s linear in each argument: that is
17 Inner products Up until now, we have only examined the properties of vectors and matrices in R n. But normally, when we think of R n, we re really thinking of n-dimensional Euclidean space - that is,
More informationExercise Solutions to Functional Analysis
Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n
More informationMetric Spaces Math 413 Honors Project
Metric Spaces Math 413 Honors Project 1 Metric Spaces Definition 1.1 Let X be a set. A metric on X is a function d : X X R such that for all x, y, z X: i) d(x, y) = d(y, x); ii) d(x, y) = 0 if and only
More informationChapter 8 Integral Operators
Chapter 8 Integral Operators In our development of metrics, norms, inner products, and operator theory in Chapters 1 7 we only tangentially considered topics that involved the use of Lebesgue measure,
More informationlinearly indepedent eigenvectors as the multiplicity of the root, but in general there may be no more than one. For further discussion, assume matrice
3. Eigenvalues and Eigenvectors, Spectral Representation 3.. Eigenvalues and Eigenvectors A vector ' is eigenvector of a matrix K, if K' is parallel to ' and ' 6, i.e., K' k' k is the eigenvalue. If is
More informationFrames inr n. Brody Dylan Johnson Department of Mathematics Washington University Saint Louis, Missouri
Frames inr n Brody Dylan Johnson Department of Mathematics Washington University Saint Louis, Missouri 63130 e-mail: brody@math.wustl.edu February, 00 Abstract These notes provide an introduction to the
More informationSpectral Properties of Elliptic Operators In previous work we have replaced the strong version of an elliptic boundary value problem
Spectral Properties of Elliptic Operators In previous work we have replaced the strong version of an elliptic boundary value problem L u x f x BC u x g x with the weak problem find u V such that B u,v
More information1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r
DAMTP 2002/NA08 Least Frobenius norm updating of quadratic models that satisfy interpolation conditions 1 M.J.D. Powell Abstract: Quadratic models of objective functions are highly useful in many optimization
More informationPh 219b/CS 219b. Exercises Due: Wednesday 21 November 2018 H = 1 ( ) 1 1. in quantum circuit notation, we denote the Hadamard gate as
h 29b/CS 29b Exercises Due: Wednesday 2 November 208 3. Universal quantum gates I In this exercise and the two that follow, we will establish that several simple sets of gates are universal for quantum
More informationFUNCTIONAL ANALYSIS LECTURE NOTES: ADJOINTS IN HILBERT SPACES
FUNCTIONAL ANALYSIS LECTURE NOTES: ADJOINTS IN HILBERT SPACES CHRISTOPHER HEIL 1. Adjoints in Hilbert Spaces Recall that the dot product on R n is given by x y = x T y, while the dot product on C n is
More informationAnalysis-3 lecture schemes
Analysis-3 lecture schemes (with Homeworks) 1 Csörgő István November, 2015 1 A jegyzet az ELTE Informatikai Kar 2015. évi Jegyzetpályázatának támogatásával készült Contents 1. Lesson 1 4 1.1. The Space
More informationj=1 u 1jv 1j. 1/ 2 Lemma 1. An orthogonal set of vectors must be linearly independent.
Lecture Notes: Orthogonal and Symmetric Matrices Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Orthogonal Matrix Definition. Let u = [u
More informationDuke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014
Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document
More informationIt follows from the above inequalities that for c C 1
3 Spaces L p 1. In this part we fix a measure space (, A, µ) (we may assume that µ is complete), and consider the A -measurable functions on it. 2. For f L 1 (, µ) set f 1 = f L 1 = f L 1 (,µ) = f dµ.
More information(K + L)(c x) = K(c x) + L(c x) (def of K + L) = K( x) + K( y) + L( x) + L( y) (K, L are linear) = (K L)( x) + (K L)( y).
Exercise 71 We have L( x) = x 1 L( v 1 ) + x 2 L( v 2 ) + + x n L( v n ) n = x i (a 1i w 1 + a 2i w 2 + + a mi w m ) i=1 ( n ) ( n ) ( n ) = x i a 1i w 1 + x i a 2i w 2 + + x i a mi w m i=1 Therefore y
More informationChapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices
Chapter 7 Iterative methods for large sparse linear systems In this chapter we revisit the problem of solving linear systems of equations, but now in the context of large sparse systems. The price to pay
More informationTechinical Proofs for Nonlinear Learning using Local Coordinate Coding
Techinical Proofs for Nonlinear Learning using Local Coordinate Coding 1 Notations and Main Results Denition 1.1 (Lipschitz Smoothness) A function f(x) on R d is (α, β, p)-lipschitz smooth with respect
More informationConvexity and Smoothness
Capter 4 Convexity and Smootness 4. Strict Convexity, Smootness, and Gateaux Di erentiablity Definition 4... Let X be a Banac space wit a norm denoted by k k. A map f : X \{0}!X \{0}, f 7! f x is called
More informationAnalysis Preliminary Exam Workshop: Hilbert Spaces
Analysis Preliminary Exam Workshop: Hilbert Spaces 1. Hilbert spaces A Hilbert space H is a complete real or complex inner product space. Consider complex Hilbert spaces for definiteness. If (, ) : H H
More informationDoubly Indexed Infinite Series
The Islamic University of Gaza Deanery of Higher studies Faculty of Science Department of Mathematics Doubly Indexed Infinite Series Presented By Ahed Khaleel Abu ALees Supervisor Professor Eissa D. Habil
More informationMetric Spaces Math 413 Honors Project
Metric Spaces Math 413 Honors Project 1 Metric Spaces Definition 1.1 Let X be a set. A metric on X is a function d : X X R such that for all x, y, z X: i) d(x, y) = d(y, x); ii) d(x, y) = 0 if and only
More informationFunctional Analysis Exercise Class
Functional Analysis Exercise Class Week: December 4 8 Deadline to hand in the homework: your exercise class on week January 5. Exercises with solutions ) Let H, K be Hilbert spaces, and A : H K be a linear
More informationKernel Method: Data Analysis with Positive Definite Kernels
Kernel Method: Data Analysis with Positive Definite Kernels 2. Positive Definite Kernel and Reproducing Kernel Hilbert Space Kenji Fukumizu The Institute of Statistical Mathematics. Graduate University
More informationMTH 2032 SemesterII
MTH 202 SemesterII 2010-11 Linear Algebra Worked Examples Dr. Tony Yee Department of Mathematics and Information Technology The Hong Kong Institute of Education December 28, 2011 ii Contents Table of Contents
More information