Representation of objects. Representation of objects. Transformations. Hierarchy of spaces. Dimensionality reduction
|
|
- Justin Gilbert
- 5 years ago
- Views:
Transcription
1 Representation of objects Representation of objects For searching/mining, first need to represent the objects Images/videos: MPEG features Graphs: Matrix Text document: tf/idf vector, bag of words Kernels How to compare objects Distance measure Similarity measure Vector/metric spaces Composition of spaces Variety of data Dynamic data 2 Transformations Hierarchy of spaces Dimensionality reduction Fourier, wavelet, SVD,.. Embedding of objects into Euclidean spaces Typically from metric spaces Vector space Metric space Normed linear space Inner product space E.g., spectral analysis of graphs R n 3 4
2 Vector spaces Vector addition Commutative, associative, identity, inverse Scalar multiplication (allow complex numbers) Associative: (cd)x = c(dx) Identity: x = x Distributive: (c + d)x = cx + dx c(x+y) = cx + cy Subspace Closed under vector addition and scalar multiplication References Linear algebra and its applications, G. Strang, Brooks/Cole, 3 rd edition Definitions Vectors v, v 2,.., v k are linearly independent iff c v + c 2 v c k v k = 0 only when c = c 2 =..= c k = 0. Given a set of vectors v, v 2,.., v k, their span is the vector space generated by their linear combinations. A basis for a vector space V is a set of vectors that are linearly independent and that spans V. The cardinality of the basis of a vector space is called its dimension. Bases can be different but they all have the same dimension 5 6 Normed linear (vector) space The norm of a vector is a measure of its size, denoted v. It has the following properties: v 0, v = 0 iff v = 0 cv = c. v Note: c is the magnitude of scalar c. u+v u + v, triangle inequality A normed linear space is a vector space with a norm. Examples Vector space of matrices with A = maximum absolute value of elements of A Vector space of infinite sequences satisfying the convergence condition (Σ x k p ) /p <, p 7 Inner product space Can be used to compare objects An inner product <u,v> is a binary operator from complex (real) vectors to a complex (real) scalar such that <u,u> is real and at least 0 <u,u> = 0 iff u = 0 <u,v> = <v,u> <bu+cv, w> = b<u,w> + c<v,w> An inner product space is vector space with an inner product. Inner product defines a norm = <v,v> = v. Satisfies the three conditions on previous slide (proof follows). Also satisfies Schwarz inequality: <u,v> u v (proof follows). 8 2
3 Proof of Schwarz inequality 0 <u-xv, u-xv> = <u,u-xv> - x<v,u-xv> = <u-xv,u> - x<u-xv,v> = <u,u> - x<v,u> - x (<u,v> - x<v,v>) = <u,u> - x<u,v> - x<u,v> + x 2 <v,v> Choose x = <u,v> / <v,v> Then, 0 <u,u> - <u,v> 2 / <v,v> Or, <u,v> u v Proof of triangle inequality u+v 2 = u 2 + v 2 + <u,v> + <u,v> = u 2 + v Re <u,v> u 2 + v <u,v> u 2 + v u v, Schwarz ineq = ( u + v ) 2 Note that if <u,v> = 0 then u+v 2 = u 2 + v 2, Pythagorus theorem u-v 2 = u 2 + v 2, defn of u+v, complex numbers, complex numbers 9 0 Metric space A metric space comes with a non-negative metric function d such that d(x,x) = 0 d(x,y) = d(y,x) d(x,y) d(x,z) + d(z,y), triangle inequality Every normed linear space is a metric space with d(x,y) = x-y. Every inner product space is a normed linear space. Inner product space is also a metric space Complete spaces A sequence f, f 2,.. converges to the limit f provided for any ε, there exists N such that for all n > N, d(f n,f) < ε. A sequence is called a Cauchy sequence provided for any ε, there exists N such that for all m,n > N, d(f m,f n ) < ε. Every convergent sequence is a Cauchy sequence. d(f m,f n ) d(f m,f) + d(f n,f) The limit of a Cauchy sequence may not exist in the metric space Rational numbers A complete metric space is a metric space for which the limit of every Cauchy sequence lies in the metric space. Similar completeness: Inner product space Hilbert space Normed linear space Banach space 2 3
4 Real vector space: R n n basis vectors Inner product <u,v> is the dot product = u T v. Length of a vector v = v T v = v Angle θ between two vectors u and v is given by cos θ = <u,v> / u v u-v 2 = u 2 + v 2 2 u v cos θ, property of triangles u-v 2 = (u-v) T (u-v) = u 2 + v 2 2 u T v Schwarz inequality: u T v u v u T v u p v q provided /p + /q = and < p,q < (Holder s inequality) L p (Minkowski) distance metrics: x = (Σ x k p ) /p is a norm for p. p-mean: ((Σ x k p )/n) /p, n is the size of the vector p = -: Harmonic mean; p = 0: Geometric mean; p = : Arith mean; p = 2: Euclidean mean; p = : max; p = - : min 3 Generalized p-means b Ha + blê2 ab a log a (Generalization of AM-GM inequality) Let x i > 0, λ i in [0,], Σ λ i =. Then Π x i λi Σ λ i x i. Proof follows from convexity of exponential function. 2 y á e x Hlog a + log bl log b Figure 4: A visual proof that p ab < (a + b)/2. 4 Generalized p-means Properties of Minkowski norms Let x i > 0, λ i in [0,], Σ λ i =. If p q then (Σ λ i x ip ) /p (Σ λ i x iq ) /q for all non-zero p,q. If p = and q > then the inequality becomes (Σ λ i x i ) q (Σ λ i x iq ) and the proof follows from convexity of x q. If 0 < p < q then consider the fraction q/p in the above inequality. Similar analysis for other cases. (a) k= (b) k=2 (c) k=3 (d) k=inf Figure: Illustration of Minkowski distance For p q <, x p x q. Proof: x p x q Note the difference = (Σ x k p ) /p (Σ x k q ) /q with p-means = (Σ x k p ) q/p Σ ( x k p ) q/p Therefore, it suffices to show that (Σ u k ) r Σ u r k, r, u k 0, which follows from the expansion of LHS
5 Properties of Minkowski norms If p then x p satisfies triangle inequality. Need to show that u+v p u p + v p Since x p is convex, (-λ)u + λv p p = Σ (-λ)u i + λv i p Σ (-λ) u i p + λ v i p = (-λ) u p p + λ v pp. So, for unit vectors u and v, (-λ)u + λv p. u+v p / ( u p + v p ) = u / ( u p + v p ) + v / ( u p + v p ) p = u p / ( u p + v p ) (u / u p ) + v p / ( u p + v p ) (v / v p ) p. 7 Other spaces of interest l p = set of all infinite sequences of real (or complex) numbers such that their p-norm is finite. Extension from finite vector spaces to countably infinite dimensions. These spaces are nested, i.e., l l 2 l 3... l L p = set of functions whose pth powers are integrable: p f : X R such that f dµ <. and the p-norm of the function is defined as ( f The norm for l p generalizes the finite p-norm while the norm for L p generalizes the p-mean: for p q <, f p f q. Each L p is complete. p dµ) /p 8 Metric for color spaces Color similarity using a real, symmetric, positive semi-definite matrix A (no negative eigenvalues) Matrix A considers cross-talk among color bins d(u,v) = ((u-v) T A(u-v) ) Quadratic form distance Example: Three bins: red, orange, blue Vectors u(,0,0), v(0,,0), w(0,0,) Matrix A = Compute the distances 9 Is d(u,v) a metric? x T Ax = Σ λ i y 2 orthonormal eigenvectors, real eigenvalues i A = Σ λ i q i q T i /* spectral decomposition for real and symmetric A*/ λ i is the i th eigenvalue and q i is the corresponding normalized eigenvector y i = x T q i = length of projection of x along q i d(u,v) = ((u-v) T A(u-v) ) = (Σ λ i (u i -v i ) 2 ) d(u,v) 0, d(u,u) = 0, d(u,v) = d(v,u) d(u,w) d(u,v) + d(v,w), provided (Σ λ i (u i -w i ) 2 ) (Σ λ i (u i -v i ) 2 ) + (Σ λ i (v i -w i ) 2 ) (Σ ( λ i u i - λ i w i ) 2 ) (Σ ( λ i u i - λ i v i ) 2 ) + (Σ( λ i v i - λ i w i ) 2 ) This is the triangle inequality for L 2 norm 20 5
6 Predicting class membership: Normalized Euclidean distance Given point x(x,,x k ) and a set of points with centroid c(c,,c k ): Normalize in each dimension: y i = (x i -c i )/σ i Distance of point x from set = y: y 2 = Σ ((x i -c i )/σ i ) 2 = (x-c) T S(x-c) S is a diagonal matrix with entries of the form /σ i 2 Points with the same y-value lie on an axesoriented ellipsoid centered at c. 2 Predicting class membership: Mahalanobis distance Distance of point x from set = (x-c) T S - (x-c) S is the covariance matrix: real, symmetric, and positive semi-definite (its inverse also has these properties) S(i,j) = E [(Z i µ i )(Z j µ j )] S(i,i) = σ 2 i = E [(Zi µ i )(Z i µ i )] If each data is placed in a row then the covariance of dimensions i and j = (Z i µ i ) T (Z j µ j ) / n, n is the number of data items or rows. As compared to normalized Euclidean, Mahalanobis distance allows the axes of the ellipsoid to rotate. The eigenvectors determine the axes of the ellipsoid. 22 Orthonormal basis A basis for an inner product space is orthonormal provided <f,g> = 0 for any pair of distinct vectors f and g in the basis. <f,f > = Only the first condition is needed for an orthogonal basis. Orthogonal matrix: square matrix whose columns are orthonormal. Gram-Schmidt orthogonalization A set of linearly independent vectors {u i } can be transformed into a set {w i } of orthonormal vectors. v = u, w = v / v v 2 = (u 2 - <u 2,w > w ), w 2 = v 2 / v 2 k- v k = (u k - <u k,w i > w i ), w k = v k / v k i =
7 Properties of orthogonal matrices Q T Q = I (by definition) Multiplication by an orthogonal matrix preserves length Qx = x Proof: Qx 2 = (Qx) T (Qx) = x T Q T Qx = x T Ix = x T x = x 2 Multiplication by an orthogonal matrix preserves inner products and angles (Qx) T (Qy) = x T Q T Qy = x T Iy = x T y cos θ = <x,y> / x y Hilbert space Complete Inner Product Space Finite dimensions R n with inner product as the dot product of u and v C n with inner product as the dot product of u and the conjugate of v Infinite dimensions L 2 (R) denotes the collection of all measurable functions f s.t. f(x) 2 dx <. - Space of square-integrable functions: vector is a function here. Can be generalized to L p (R). Define <f,g> = f(x)g(x) dx L 2 (0,2π) denotes the collection of all measurable functions f defined over the interval (0,2π) s.t. 2π f(x) 2 dx <. 0 Space of 2π-periodic square-integrable functions Properties of Hilbert Spaces Given a Hilbert space with orthonormal basis {e i } (Fourier expansion): v Σ <v,e i > e i (Plancherel s theorem): <v,w> = Σ <v,e i > <w,e i > (Parseval s theorem): <v,v> = v 2 = <v,e i > 2 = <v,e i > e i 2 Dimensionality reduction Reduce the number of dimensions of data Reduced storage and computation Focus on the main trends Project the d-dimensional points in a k-dimensional space so that: k << d distances are preserved as well as possible Example: use the first few Fourier coefficients
8 Embeddings Given a distance d, embed the objects into a space of smaller dimensions using a mapping F and a distance d such that d(i,j) is close to d (F(i),F(j)) Isometric mapping: exact preservation of distance Contractive mapping: d (F(i),F(j)) d(i,j) NN and range queries in reduced space Examples d(a,b) = d(a,c) = d(b,c) = 2, d(a,e) = d(b,e) = d(c,e) =. Isometric embedding in a 3-d space using L 2? Isometric embedding in a 3-d space using L? Isometric embedding of n points in n- dimensional space using L F(i) = distance vector of object i Fourier analysis Analysis of functions or signals in frequency space Continuous aperiodic signal: Fourier transform Continuous periodic signal: Fourier series Discrete periodic signal: Discrete Fourier transform Orthonormal basis functions and inner products in each case Complex numbers possible as coefficients even for functions in real space Inner product based on summation/integration of function with basis elements. 3 Fourier transform The set {(/ 2π)e j2πkx k є R} forms an orthonormal set of basis functions for L 2 (R). The resulting decomposition is called the Fourier transform. F(k) = / 2π e -j2πkx f(x) dx f(x) = / 2π e j2πkx F(k) dk Sometimes, 2πk is replaced by ω (angular frequency) and x by t (time). F(ω) = / 2π e -jωx f(t) dt f(t) = / 2π e jωx F(ω) dω forward inverse 32 8
9 Fourier series Orthonormal basis functions (/ 2π), (/ π) cos x, (/ π) sin x, (/ π) cos 2x, (/ π) sin 2x, For any function f(x) є L 2 (0,2π), f(x) = a 0 (/ 2π) + a (/ π) cos x + b (/ π) sin x + a 2 (/ π) cos2x + b 2 (/ π) sin2x + Fourier series f(x) = /2π f(x) dx + /π f(x) cos x dx + /π f(x) sin x dx + /π f(x) cos 2x dx + /π f(x) sin 2x dx + a 0 = <f, / 2π > = / 2π f(x) dx a m = <f, (/ π) cos mx> = / π f(x)cos(mx) dx b m = <f, (/ π) sin mx> = / π f(x)sin(mx) dx Basis elements for Fourier series Fourier approximation
10 Fourier approximation Frequency analysis The independent variable is Time The dependent variable is the Amplitude Most of the information is hidden in the Frequency content 2 Hz 20 Hz Magnitude Magnitude Time Magnitude Magnitude Time 0 Hz 2 Hz + 0 Hz + 20Hz Time Time 38 Discrete Fourier Transform (DFT) Given an input sequence (x 0,x,x 2, x ), transform it into frequency domain so that we have equality at n points. Only need n basis elements Orthonormal basis functions ω is the nth primitive root of unity = e 2πj/n = cos (2π/n) + jsin(2π/n)) (/ n) [,,..] T (/ n) [, ω, ω 2,, ω () ] T (/ n) [, ω 2, ω 4,, ω 2() ] T (/ n) [, ω (), ω 2(),, ω ()2 ] T Proof of orthonormality Remember to take the conjugate Illustrate on unit circle Example: vector (2,4,6,8) / 4[c 0 + c e jx + c 2 e j2x + c 3 e j3x ]= (2,4,6,8) at (0,π/2,π,3π/2) Discrete Fourier Transform Given input x = (x 0,x,x 2, x ), DFT produces X= (X 0,X,X 2, X ) where X f (projection along the f th basis element) is given by X f = / n x t ω -ft = / n x t e -j2πft/n t=0 for f = 0,,, X 0 = / n x t t=0 X = / n x t e -j2πt/n = / n x t (cos (2πt/n) - jsin(2πt/n)) t=0 X 2 = / n x t e -j4πt/n = / n x t (cos (4πt/n) - jsin(4πt/n)) t=
11 Inverse Transform Given X= (X 0,X,X 2, X ), the inverse transform produces x= (x 0,x,x 2, x ) where x t is given by x t = / n X f ω ft = / n X f e j2πft/n, t = 0,,, f=0 x 0 = / n X f f=0 x = / n X f e j2πf/n = / n X f (cos (2πf/n) + jsin(2πf/n)) f=0 x 2 = / n X f e j4πf/n = / n X f (cos (4πf/n) + jsin(4πf/n)) f=0 Fourier matrix and its inverse F jk = (/ n) ω jk ; inner product with this matrix defines the forward transform (analysis) F - jk = (/ n) ω -jk ; inner product with this matrix defines the inverse transform (synthesis) Prove that FF - = I 4 42 Properties Parseval s theorem: energy in time domain = x 2 = x t 2 = energy in frequency domain = X f 2 x-y 2 = X-Y 2 = X f -Y f 2 Convolution in time domain is multiplication in frequency domain. FFT Time complexity O(n log n) How to build an index? Which coefficients? Searching in a reduced dimensional space 43
Mathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More information6 Inner Product Spaces
Lectures 16,17,18 6 Inner Product Spaces 6.1 Basic Definition Parallelogram law, the ability to measure angle between two vectors and in particular, the concept of perpendicularity make the euclidean space
More informationLecture 20: 6.1 Inner Products
Lecture 0: 6.1 Inner Products Wei-Ta Chu 011/1/5 Definition An inner product on a real vector space V is a function that associates a real number u, v with each pair of vectors u and v in V in such a way
More informationLinear Algebra- Final Exam Review
Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.
More informationMath Linear Algebra II. 1. Inner Products and Norms
Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,
More informationProf. M. Saha Professor of Mathematics The University of Burdwan West Bengal, India
CHAPTER 9 BY Prof. M. Saha Professor of Mathematics The University of Burdwan West Bengal, India E-mail : mantusaha.bu@gmail.com Introduction and Objectives In the preceding chapters, we discussed normed
More informationFinite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product
Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )
More informationMath 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces.
Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,...,
More informationLecture 23: 6.1 Inner Products
Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such
More informationRecall that any inner product space V has an associated norm defined by
Hilbert Spaces Recall that any inner product space V has an associated norm defined by v = v v. Thus an inner product space can be viewed as a special kind of normed vector space. In particular every inner
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationThe following definition is fundamental.
1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic
More informationLINEAR ALGEBRA W W L CHEN
LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008. This chapter is available free to all individuals, on the understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationRecall: Dot product on R 2 : u v = (u 1, u 2 ) (v 1, v 2 ) = u 1 v 1 + u 2 v 2, u u = u u 2 2 = u 2. Geometric Meaning:
Recall: Dot product on R 2 : u v = (u 1, u 2 ) (v 1, v 2 ) = u 1 v 1 + u 2 v 2, u u = u 2 1 + u 2 2 = u 2. Geometric Meaning: u v = u v cos θ. u θ v 1 Reason: The opposite side is given by u v. u v 2 =
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More information2. Review of Linear Algebra
2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear
More informationLecture 7: Positive Semidefinite Matrices
Lecture 7: Positive Semidefinite Matrices Rajat Mittal IIT Kanpur The main aim of this lecture note is to prepare your background for semidefinite programming. We have already seen some linear algebra.
More informationCHAPTER VIII HILBERT SPACES
CHAPTER VIII HILBERT SPACES DEFINITION Let X and Y be two complex vector spaces. A map T : X Y is called a conjugate-linear transformation if it is a reallinear transformation from X into Y, and if T (λx)
More informationMath 24 Spring 2012 Sample Homework Solutions Week 8
Math 4 Spring Sample Homework Solutions Week 8 Section 5. (.) Test A M (R) for diagonalizability, and if possible find an invertible matrix Q and a diagonal matrix D such that Q AQ = D. ( ) 4 (c) A =.
More informationContents. Appendix D (Inner Product Spaces) W-51. Index W-63
Contents Appendix D (Inner Product Spaces W-5 Index W-63 Inner city space W-49 W-5 Chapter : Appendix D Inner Product Spaces The inner product, taken of any two vectors in an arbitrary vector space, generalizes
More informationMATH 167: APPLIED LINEAR ALGEBRA Chapter 3
MATH 167: APPLIED LINEAR ALGEBRA Chapter 3 Jesús De Loera, UC Davis February 18, 2012 Orthogonal Vectors and Subspaces (3.1). In real life vector spaces come with additional METRIC properties!! We have
More informationMATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors.
MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. Orthogonal sets Let V be a vector space with an inner product. Definition. Nonzero vectors v 1,v
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More informationThe Hilbert Space of Random Variables
The Hilbert Space of Random Variables Electrical Engineering 126 (UC Berkeley) Spring 2018 1 Outline Fix a probability space and consider the set H := {X : X is a real-valued random variable with E[X 2
More informationINNER PRODUCT SPACE. Definition 1
INNER PRODUCT SPACE Definition 1 Suppose u, v and w are all vectors in vector space V and c is any scalar. An inner product space on the vectors space V is a function that associates with each pair of
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationIntroduction to Signal Spaces
Introduction to Signal Spaces Selin Aviyente Department of Electrical and Computer Engineering Michigan State University January 12, 2010 Motivation Outline 1 Motivation 2 Vector Space 3 Inner Product
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationFourier Series. Spectral Analysis of Periodic Signals
Fourier Series. Spectral Analysis of Periodic Signals he response of continuous-time linear invariant systems to the complex exponential with unitary magnitude response of a continuous-time LI system at
More informationQuantum Computing Lecture 2. Review of Linear Algebra
Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationInner products. Theorem (basic properties): Given vectors u, v, w in an inner product space V, and a scalar k, the following properties hold:
Inner products Definition: An inner product on a real vector space V is an operation (function) that assigns to each pair of vectors ( u, v) in V a scalar u, v satisfying the following axioms: 1. u, v
More informationFunctional Analysis Exercise Class
Functional Analysis Exercise Class Week: December 4 8 Deadline to hand in the homework: your exercise class on week January 5. Exercises with solutions ) Let H, K be Hilbert spaces, and A : H K be a linear
More informationKernel Method: Data Analysis with Positive Definite Kernels
Kernel Method: Data Analysis with Positive Definite Kernels 2. Positive Definite Kernel and Reproducing Kernel Hilbert Space Kenji Fukumizu The Institute of Statistical Mathematics. Graduate University
More informationIntroduction to Linear Algebra, Second Edition, Serge Lange
Introduction to Linear Algebra, Second Edition, Serge Lange Chapter I: Vectors R n defined. Addition and scalar multiplication in R n. Two geometric interpretations for a vector: point and displacement.
More informationElements of Positive Definite Kernel and Reproducing Kernel Hilbert Space
Elements of Positive Definite Kernel and Reproducing Kernel Hilbert Space Statistical Inference with Reproducing Kernel Hilbert Space Kenji Fukumizu Institute of Statistical Mathematics, ROIS Department
More information33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM
33AH, WINTER 2018: STUDY GUIDE FOR FINAL EXAM (UPDATED MARCH 17, 2018) The final exam will be cumulative, with a bit more weight on more recent material. This outline covers the what we ve done since the
More informationVectors in Function Spaces
Jim Lambers MAT 66 Spring Semester 15-16 Lecture 18 Notes These notes correspond to Section 6.3 in the text. Vectors in Function Spaces We begin with some necessary terminology. A vector space V, also
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in
More informationThe Gram-Schmidt Process 1
The Gram-Schmidt Process In this section all vector spaces will be subspaces of some R m. Definition.. Let S = {v...v n } R m. The set S is said to be orthogonal if v v j = whenever i j. If in addition
More information1. Subspaces A subset M of Hilbert space H is a subspace of it is closed under the operation of forming linear combinations;i.e.,
Abstract Hilbert Space Results We have learned a little about the Hilbert spaces L U and and we have at least defined H 1 U and the scale of Hilbert spaces H p U. Now we are going to develop additional
More informationClass notes: Approximation
Class notes: Approximation Introduction Vector spaces, linear independence, subspace The goal of Numerical Analysis is to compute approximations We want to approximate eg numbers in R or C vectors in R
More informationChapter 6 Inner product spaces
Chapter 6 Inner product spaces 6.1 Inner products and norms Definition 1 Let V be a vector space over F. An inner product on V is a function, : V V F such that the following conditions hold. x+z,y = x,y
More informationx 3y 2z = 6 1.2) 2x 4y 3z = 8 3x + 6y + 8z = 5 x + 3y 2z + 5t = 4 1.5) 2x + 8y z + 9t = 9 3x + 5y 12z + 17t = 7
Linear Algebra and its Applications-Lab 1 1) Use Gaussian elimination to solve the following systems x 1 + x 2 2x 3 + 4x 4 = 5 1.1) 2x 1 + 2x 2 3x 3 + x 4 = 3 3x 1 + 3x 2 4x 3 2x 4 = 1 x + y + 2z = 4 1.4)
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More information1. Foundations of Numerics from Advanced Mathematics. Linear Algebra
Foundations of Numerics from Advanced Mathematics Linear Algebra Linear Algebra, October 23, 22 Linear Algebra Mathematical Structures a mathematical structure consists of one or several sets and one or
More informationLinear algebra II Homework #1 solutions A = This means that every eigenvector with eigenvalue λ = 1 must have the form
Linear algebra II Homework # solutions. Find the eigenvalues and the eigenvectors of the matrix 4 6 A =. 5 Since tra = 9 and deta = = 8, the characteristic polynomial is f(λ) = λ (tra)λ+deta = λ 9λ+8 =
More informationCambridge University Press The Mathematics of Signal Processing Steven B. Damelin and Willard Miller Excerpt More information
Introduction Consider a linear system y = Φx where Φ can be taken as an m n matrix acting on Euclidean space or more generally, a linear operator on a Hilbert space. We call the vector x a signal or input,
More informationChapter 2. Vectors and Vector Spaces
2.1. Operations on Vectors 1 Chapter 2. Vectors and Vector Spaces Section 2.1. Operations on Vectors Note. In this section, we define several arithmetic operations on vectors (especially, vector addition
More informationSome notes on Linear Algebra. Mark Schmidt September 10, 2009
Some notes on Linear Algebra Mark Schmidt September 10, 2009 References Linear Algebra and Its Applications. Strang, 1988. Practical Optimization. Gill, Murray, Wright, 1982. Matrix Computations. Golub
More informationMATH 115A: SAMPLE FINAL SOLUTIONS
MATH A: SAMPLE FINAL SOLUTIONS JOE HUGHES. Let V be the set of all functions f : R R such that f( x) = f(x) for all x R. Show that V is a vector space over R under the usual addition and scalar multiplication
More informationGQE ALGEBRA PROBLEMS
GQE ALGEBRA PROBLEMS JAKOB STREIPEL Contents. Eigenthings 2. Norms, Inner Products, Orthogonality, and Such 6 3. Determinants, Inverses, and Linear (In)dependence 4. (Invariant) Subspaces 3 Throughout
More informationBASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x
BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,
More informationApplied Analysis (APPM 5440): Final exam 1:30pm 4:00pm, Dec. 14, Closed books.
Applied Analysis APPM 44: Final exam 1:3pm 4:pm, Dec. 14, 29. Closed books. Problem 1: 2p Set I = [, 1]. Prove that there is a continuous function u on I such that 1 ux 1 x sin ut 2 dt = cosx, x I. Define
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More informationMTH 2032 SemesterII
MTH 202 SemesterII 2010-11 Linear Algebra Worked Examples Dr. Tony Yee Department of Mathematics and Information Technology The Hong Kong Institute of Education December 28, 2011 ii Contents Table of Contents
More informationEigenvalues and Eigenvectors
/88 Chia-Ping Chen Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Eigenvalue Problem /88 Eigenvalue Equation By definition, the eigenvalue equation for matrix
More informationMATH 235: Inner Product Spaces, Assignment 7
MATH 235: Inner Product Spaces, Assignment 7 Hand in questions 3,4,5,6,9, by 9:3 am on Wednesday March 26, 28. Contents Orthogonal Basis for Inner Product Space 2 2 Inner-Product Function Space 2 3 Weighted
More informationInner Product Spaces 5.2 Inner product spaces
Inner Product Spaces 5.2 Inner product spaces November 15 Goals Concept of length, distance, and angle in R 2 or R n is extended to abstract vector spaces V. Sucn a vector space will be called an Inner
More informationBasic Calculus Review
Basic Calculus Review Lorenzo Rosasco ISML Mod. 2 - Machine Learning Vector Spaces Functionals and Operators (Matrices) Vector Space A vector space is a set V with binary operations +: V V V and : R V
More informationDavid Hilbert was old and partly deaf in the nineteen thirties. Yet being a diligent
Chapter 5 ddddd dddddd dddddddd ddddddd dddddddd ddddddd Hilbert Space The Euclidean norm is special among all norms defined in R n for being induced by the Euclidean inner product (the dot product). A
More informationSolutions: Problem Set 3 Math 201B, Winter 2007
Solutions: Problem Set 3 Math 201B, Winter 2007 Problem 1. Prove that an infinite-dimensional Hilbert space is a separable metric space if and only if it has a countable orthonormal basis. Solution. If
More informationDefinition 1. A set V is a vector space over the scalar field F {R, C} iff. there are two operations defined on V, called vector addition
6 Vector Spaces with Inned Product Basis and Dimension Section Objective(s): Vector Spaces and Subspaces Linear (In)dependence Basis and Dimension Inner Product 6 Vector Spaces and Subspaces Definition
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More information5. Orthogonal matrices
L Vandenberghe EE133A (Spring 2017) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal
More informationIr O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )
Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O
More informationNumerical Linear Algebra
University of Alabama at Birmingham Department of Mathematics Numerical Linear Algebra Lecture Notes for MA 660 (1997 2014) Dr Nikolai Chernov April 2014 Chapter 0 Review of Linear Algebra 0.1 Matrices
More informationSolutions for MAS277 Problems
Solutions for MAS77 Problems Solutions for Chapter problems: Inner product spaces 1. No. The only part that fails, however, is the condition that f, f should mean that f. If f is a non-zero function that
More informationorthogonal relations between vectors and subspaces Then we study some applications in vector spaces and linear systems, including Orthonormal Basis,
5 Orthogonality Goals: We use scalar products to find the length of a vector, the angle between 2 vectors, projections, orthogonal relations between vectors and subspaces Then we study some applications
More informationAppendix A Functional Analysis
Appendix A Functional Analysis A.1 Metric Spaces, Banach Spaces, and Hilbert Spaces Definition A.1. Metric space. Let X be a set. A map d : X X R is called metric on X if for all x,y,z X it is i) d(x,y)
More informationEconomics 204 Summer/Fall 2010 Lecture 10 Friday August 6, 2010
Economics 204 Summer/Fall 2010 Lecture 10 Friday August 6, 2010 Diagonalization of Symmetric Real Matrices (from Handout Definition 1 Let δ ij = { 1 if i = j 0 if i j A basis V = {v 1,..., v n } of R n
More informationFunctional Analysis. Franck Sueur Metric spaces Definitions Completeness Compactness Separability...
Functional Analysis Franck Sueur 2018-2019 Contents 1 Metric spaces 1 1.1 Definitions........................................ 1 1.2 Completeness...................................... 3 1.3 Compactness......................................
More information235 Final exam review questions
5 Final exam review questions Paul Hacking December 4, 0 () Let A be an n n matrix and T : R n R n, T (x) = Ax the linear transformation with matrix A. What does it mean to say that a vector v R n is an
More informationVector Spaces, Affine Spaces, and Metric Spaces
Vector Spaces, Affine Spaces, and Metric Spaces 2 This chapter is only meant to give a short overview of the most important concepts in linear algebra, affine spaces, and metric spaces and is not intended
More informationA Review of Linear Algebra
A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose
More informationUniversity of Leeds, School of Mathematics MATH 3181 Inner product and metric spaces, Solutions 1
University of Leeds, School of Mathematics MATH 38 Inner product and metric spaces, Solutions. (i) No. If x = (, ), then x,x =, but x is not the zero vector. So the positive definiteness property fails
More informationChapter 5. Basics of Euclidean Geometry
Chapter 5 Basics of Euclidean Geometry 5.1 Inner Products, Euclidean Spaces In Affine geometry, it is possible to deal with ratios of vectors and barycenters of points, but there is no way to express the
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationMATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators.
MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators. Adjoint operator and adjoint matrix Given a linear operator L on an inner product space V, the adjoint of L is a transformation
More informationPart IA. Vectors and Matrices. Year
Part IA Vectors and Matrices Year 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2018 Paper 1, Section I 1C Vectors and Matrices For z, w C define the principal value of z w. State de Moivre s
More informationHilbert Spaces. Hilbert space is a vector space with some extra structure. We start with formal (axiomatic) definition of a vector space.
Hilbert Spaces Hilbert space is a vector space with some extra structure. We start with formal (axiomatic) definition of a vector space. Vector Space. Vector space, ν, over the field of complex numbers,
More informationOrdinary Differential Equations II
Ordinary Differential Equations II February 23 2017 Separation of variables Wave eq. (PDE) 2 u t (t, x) = 2 u 2 c2 (t, x), x2 c > 0 constant. Describes small vibrations in a homogeneous string. u(t, x)
More informationReal Analysis, 2nd Edition, G.B.Folland Elements of Functional Analysis
Real Analysis, 2nd Edition, G.B.Folland Chapter 5 Elements of Functional Analysis Yung-Hsiang Huang 5.1 Normed Vector Spaces 1. Note for any x, y X and a, b K, x+y x + y and by ax b y x + b a x. 2. It
More informationInner Product Spaces 6.1 Length and Dot Product in R n
Inner Product Spaces 6.1 Length and Dot Product in R n Summer 2017 Goals We imitate the concept of length and angle between two vectors in R 2, R 3 to define the same in the n space R n. Main topics are:
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More informationSection 3.9. Matrix Norm
3.9. Matrix Norm 1 Section 3.9. Matrix Norm Note. We define several matrix norms, some similar to vector norms and some reflecting how multiplication by a matrix affects the norm of a vector. We use matrix
More informationVector spaces. DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis.
Vector spaces DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Vector space Consists of: A set V A scalar
More informationLecture Notes 1: Vector spaces
Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More informationChapter 4 Euclid Space
Chapter 4 Euclid Space Inner Product Spaces Definition.. Let V be a real vector space over IR. A real inner product on V is a real valued function on V V, denoted by (, ), which satisfies () (x, y) = (y,
More informationDuke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014
Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document
More information