Kernel Method: Data Analysis with Positive Definite Kernels


 Gwendolyn Walton
 1 years ago
 Views:
Transcription
1 Kernel Method: Data Analysis with Positive Definite Kernels 2. Positive Definite Kernel and Reproducing Kernel Hilbert Space Kenji Fukumizu The Institute of Statistical Mathematics. Graduate University of Advanced Studies / Tokyo Institute of Technology Nov , 2010 Intensive Course at Tokyo Institute of Technology
2 2 / 24 Outline Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS
3 3 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS
4 4 / 24 Definition of Positive Definite Kernel Definition. Let X be a set. k : X X R is a positive definite kernel if k(x, y) = k(y, x) and for every x 1,..., x n X and c 1,..., c n R n c i c j k(x i, x j ) 0, i,j=1 i.e. the symmetric matrix k(x 1, x 1 ) k(x 1, x n ) (k(x i, x j )) n i,j=1 =..... k(x n, x 1 ) k(x n, x ) is positive semidefinite. The symmetric matrix (k(x i, x j )) n i,j=1 is often called a Gram matrix.
5 5 / 24 Definition: Complexvalued Case Definition. Let X be a set. k : X X C is a positive definite kernel if for every x 1,..., x n X and c 1,..., c n C n c i c j k(x i, x j ) 0. i,j=1 Remark. The Hermitian property k(y, x) = k(x, y) is derived from the positivedefiniteness. [Exercise]
6 6 / 24 Some Basic Properties Facts. Assume k : X X C is positive definite. Then, for any x, y in X, 1. k(x, x) k(x, y) 2 k(x, x)k(y, y). Proof. (1) is obvious. For (2), with the fact k(y, x) = k(x, y), the definition of positive definiteness implies that the eigenvalues of the hermitian matrix ( k(x, x) ) k(x, y) k(x, y) k(y, y) is nonnegative, thus, its determinant k(x, x)k(y, y) k(x, y) 2 is nonnegative.
7 Examples Real valued positive definite kernels on R n :  Linear kernel 1 k 0 (x, y) = x T y  Exponential k E (x, y) = exp(βx T y) (β > 0)  Gaussian RBF (radial basis function) kernel ( k G (x, y) = exp 1 2σ 2 x y 2) (σ > 0)  Laplacian kernel ( k L (x, y) = exp α ) n i=1 x i y i (α > 0)  Polynomial kernel k P (x, y) = (x T y + c) d (c 0, d N) 1 [Exercise] prove that the linear kernel is positive definite. 7 / 24
8 8 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS
9 9 / 24 Reproducing kernel Hilbert space Definition. Let X be a set. A reproducing kernel Hilbert space (RKHS) (over X ) is a Hilbert space H consisting of functions on X such that for each x X there is a function k x H with the property f, k x H = f(x) ( f H) (reproducing property). k(, x) := k x ( ) is called a reproducing kernel of H. Fact 1. A reproducing kernel is Hermitian (or symmetric). Proof. k(y, x) = k(, x), k y = k x, k y = k y, k x = k(, y), k x = k(x, y). Fact 2. The reproducing kernel is unique, if exists. [Exercise]
10 10 / 24 Positive Definite Kernel and RKHS I Proposition 1 (RKHS positive definite kernel) The reproducing kernel of a RKHS is positive definite. Proof. n c i c j k(x i, x j ) = i,j=1 n c i c j k(, x i ), k(, x j ) i,j=1 n = c i k(, x i ), i=1 n j=1 c j k(, x j ) = n c i k(, x i ) 2 0. i=1
11 11 / 24 Positive Definite Kernel and RKHS II Theorem 2 (positive definite kernel RKHS. MooreAronszajn) Let k : X X C (or R) be a positive definite kernel on a set X. Then, there uniquely exists a RKHS H k on X such that 1. k(, x) H k for every x X, 2. Span{k(, x) x X } is dense in H k, 3. k is the reproducing kernel on H k, i.e., f, k(, x) H = f(x) ( x X, f H k ). Proof omitted.
12 12 / 24 Positive Definite Kernel and RKHS III Onetoone correspondence between positive definite kernels and RKHS. k H k Proposition 1: RKHS positive definite kernel k. Theorem 2: k H k (injective).
13 13 / 24 RKHS as Feature Space If we define Φ : X H k, x k(, x), then, Φ(x), Φ(y) = k(, x), k(, y) = k(x, y). RKHS associated with a positive definite kernel k gives a desired feature space!! In kernel methods, the above feature map and feature space are always used.
14 14 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS
15 15 / 24 Operations that Preserve Positive Definiteness I Proposition 3 If k i : X X C (i = 1, 2,...) are positive definite kernels, then so are the following: 1. (positive combination) ak 1 + bk 2 (a, b 0). 2. (product) k 1 k 2 (k 1 (x, y)k 2 (x, y)). 3. (limit) lim i k i (x, y), assuming the limit exists. Remark. From Proposition 3, the set of all positive definite kernels is a closed (w.r.t. pointwise convergence) convex cone stable under multiplication.
16 Operations that Preserve Positive Definiteness II Proof. (1): Obvious. (3): The nonnegativity in the definition holds also for the limit. (2): It suffices to show that two Hermitian matrices A and B are positive semidefinite, so is their componentwise product. This is done by the following lemma. Definition. For two matrices A and B of the same size, the matrix C with C ij = A ij B ij is called the Hadamard product of A and B. The Hadamard product of A and B is denoted by A B. Lemma 4 Let A and B be nonnegative Hermitian matrices of the same size. Then, A B is also nonnegative. 16 / 24
17 17 / 24 Operations that Preserve Positive Definiteness III Proof. Let A = UΛU be the eigendecomposition of A, where U = (u 1,..., u p ): a unitary matrix, i.e., U = U T Λ: diagonal matrix with nonnegative entries (λ 1,..., λ p ). Then, for arbitrary c 1,..., c p C, p p c i c j (A B) ij = λ a c i c j u a i ū a j B ij = λ a ξ at Bξ a, i,j=1 a=1 a=1 where ξ a = (c 1 u a 1,..., c p u a p) T C p. Since ξ at Bξ a and λ a are nonnegative for each a, so is the sum.
18 18 / 24 Feature Map must be Positive Definite Proposition 5 Let V be an vector space with an inner product,. If we have a map Φ : X V, x Φ(x), a positive definite kernel on X is defined by k(x, y) = Φ(x), Φ(y). Proof. Let x 1,..., x n in X and c 1,..., c n C. n i,j=1 c ic j k(x i, x j ) = n i,j=1 c ic j Φ(x i ), Φ(x j ) n = i=1 c iφ(x i ), n j=1 c jφ(x j ) = n i=1 c iφ(x i ) 2 0.
19 Proposition 6 Modification Let k : X X C be a positive definite kernel and f : X C be an arbitrary function. Then, is positive definite. In particular, and k(x, y) = f(x)k(x, y)f(y) f(x)f(y) k(x, y) k(x, x) k(y, y) (normalized kernel) are positive definite. Proof is left as an exercise. 19 / 24
20 20 / 24 Proofs for Positive Definiteness of Examples Linear kernel: Proposition 5 Exponential: exp(βx T y) = 1 + βx T y + β2 2! (xt y) 2 + β3 3! (xt y) 3 + Use Proposition 3. Gaussian RBF kernel: ( exp 1 2σ 2 x y 2) = exp ( x 2 ) ( x T y ) 2σ 2 exp σ 2 exp ( y 2 ) 2σ 2. Apply Proposition 6. Laplacian kernel: The proof is shown later. Polynomial kernel: Just sum and product.
21 21 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS
22 Proposition 7 Definition by Evaluation Map Let H be a Hilbert space consisting of functions on a set X. Then, H is a RKHS if and only if the evaluation map e x : H K, e x (f) = f(x), is a continuous linear functional for each x X. Proof. Assume H is a RKHS. The boundedness of e x is obvious from e x (f) = f, k x k x f. Conversely, assume the evaluation map is continuous. By Riesz lemma, there is k x H such that f, k x = e x (f) = f(x), which means H is a RKHS having k x as a reproducing kernel. 22 / 24
23 Continuity The functions in a RKHS are "nice" functions under some conditions. Proposition 8 Let k be a positive definite kernel on a topological space X, and H k be the associated RKHS. If Re[k(y, x)] is continuous for every x, y X, then all the functions in H k are continuous. Proof. Let f be an arbitrary function in H k. f(x) f(y) = f, k(, x) k(, y) f k(, x) k(, y). The assertion is easy from k(, x) k(, y) 2 = k(x, x) + k(y, y) 2Re[k(x, y)]. Remark. If k(x, y) is differentiable, then all the functions in H k are differentiable. c.f. L 2 space contains noncontinuous functions. 23 / 24
24 24 / 24 Summary of Sections 1 and 2 We would like to use a feature map Φ : X H to incorporate nonlinearity or high order moments. The inner product in the feature space must be computed efficiently. Ideally, Φ(x), Φ(y) = k(x, y). To satisfy the above relation, the kernel k must be positive definite. A positive definite kernel k defines an associated RKHS, where k is the reproducing kernel; k(, x), k(, y) = k(x, y). Use a RKHS as a feature space, and Φ : x k(, x) as the feature map.
25 25 / 24 Appendix: Quick introduction to Hilbert spaces Definition of Hilbert space Basic properties of Hilbert space Appendix: Proofs Proof of Theorem 2
26 26 / 24 Vector space with inner product I Definition. V : vector space over a field K = R or C. V is called an inner product space if it has an inner product (or scalar product, dot product) (, ) : V V K such that for every x, y, z V 1. (Strong positivity) (x, x) 0, and (x, x) = 0 if and only if x = 0, 2. (Addition) (x + y, z) = (x, z) + (y, z), 3. (Scalar multiplication) (αx, y) = α(x, y) ( α K), 4. (Hermitian) (y, x) = (x, y).
27 Vector space with inner product II (V, (, )): inner product space. Norm of x V : x = (x, x) 1/2. Metric between x and y: d(x, y) = x y. Theorem 9 CauchySchwarz inequality (x, y) x y. Remark: CauchySchwarz inequality holds without requiring x = 0 x = / 24
28 Hilbert space I Definition. A vector space with inner product (H, (, )) is called Hilbert space if the induced metric is complete, i.e. every Cauchy sequence 2 converges to an element in H. Remark 1: A Hilbert space may be either finite or infinite dimensional. Example 1. R n and C n are finite dimensional Hilbert space with the ordinary inner product (x, y) R n = n i=1 x iy i or (x, y) C n = n i=1 x iy i. 2 A sequence {x n} n=1 in a metric space (X, d) is called a Cauchy sequence if d(x n, x m) 0 for n, m. 28 / 24
29 29 / 24 Hilbert space II Example 2. L 2 (Ω, µ). Let (Ω, B, µ) is a measure space. { L = f : Ω C } f 2 dµ <. The inner product on L is define by (f, g) = fgdµ. L 2 (Ω, µ) is defined by the equivalent classes identifying f and g if their values differ only on a measurezero set.  L 2 (Ω, µ) is complete.  L 2 (R n, dx) is infinite dimensional.
30 30 / 24 Appendix: Quick introduction to Hilbert spaces Definition of Hilbert space Basic properties of Hilbert space Appendix: Proofs Proof of Theorem 2
31 31 / 24 Orthogonal complement. Orthogonality Let H be a Hilbert space and V be a closed subspace. V := {x H (x, y) = 0 for all y V } is a closed subspace, and called the orthogonal complement. Orthogonal projection. Let H be a Hilbert space and V be a closed subspace. Every x H can be uniquely decomposed x = y + z, y V and z V, that is, H = V V.
32 32 / 24 Complete orthonormal system I ONS and CONS. A subset {u i } i I of H is called an orthonormal system (ONS) if (u i, u j ) = δ ij (δ ij is Kronecker s delta). A subset {u i } i I of H is called a complete orthonormal system (CONS) if it is ONS and if (x, u i ) = 0 ( i I) implies x = 0. Fact: Any ONS in a Hilbert space can be extended to a CONS.
33 33 / 24 Complete orthonormal system II Separability A Hilbert space is separable if it has a countable CONS. Assumption In this course, a Hilbert space is always assumed to be separable.
34 Complete orthonormal system III Theorem 10 (Fourier series expansion) Let {u i } i=1 be a CONS of a separable Hilbert space. For each x H, Proof omitted. x = i=1 (x, u i)u i, (Fourier expansion) x 2 = i=1 (x, u i) 2. (Parseval s equality) Example: CONS of L 2 ([0 2π], dx) u n (t) = 1 2π e 1nt (n = 0, 1, 2,...) Then, f(t) = n=0 a nu n (t) is the (ordinary) Fourier expansion of a periodic function. 34 / 24
35 35 / 24 Bounded operator I Let H 1 and H 2 be Hilbert spaces. A linear transform T : H 1 H 2 is often called operator. Definition. A linear operator H 1 and H 2 is called bounded if sup T x H2 <. x H1 =1 The operator norm of a bounded operator T is defined by T = T x H2 sup T x H2 = sup. x H1 =1 x 0 x H1 (Corresponds to the largest singular value of a matrix.) Fact. If T : H 1 H 2 is bounded, T x H2 T x H1.
36 36 / 24 Proposition 11 Bounded operator II A linear operator is bounded if and only if it is continuous. Proof. Assume T : H 1 H 2 is bounded. Then, means continuity of T. T x T x 0 T x x 0 Assume T is continuous. For any ε > 0, there is δ > 0 such that T x < ε for all x H 1 with x < 2δ. Then, 1 sup T x = sup x =1 x =δ δ T x ε δ.
37 Riesz lemma I Definition. A linear functional is a linear transform from H to C (or R). The vector space of all the bounded (continuous) linear functionals called the dual space of H, and is denoted by H. Theorem 12 (Riesz lemma) For each φ H, there is a unique y φ H such that φ(x) = (x, y φ ) ( x H). Proof. Consider the case of R for simplicity. ) Obvious by CauchySchwartz. 37 / 24
38 38 / 24 Riesz lemma II ) If φ(x) = 0 for all x, take y = 0. Otherwise, let V = {x H φ(x) = 0}. Since φ is a bounded linear functional, V is a closed subspace, and V H. Take z V with z = 1. By orthogonal decomposition, for any x H, x (x, z)z V. Apply φ, then Take y φ = φ(z)z. φ(x) (x, z)φ(z) = 0, i.e., φ(x) = (x, φ(z)z).
39 Riesz lemma III 39 / 24
40 40 / 24 Appendix: Quick introduction to Hilbert spaces Definition of Hilbert space Basic properties of Hilbert space Appendix: Proofs Proof of Theorem 2
41 41 / 24 Proof. (Described in R case.) Proof of Theorem 2 I Construction of an inner product space: H 0 := Span{k(, x) x X }. Define an inner product on H 0 : for f = n i=1 a ik(, x i ) and g = m j=1 b jk(, y j ), f, g := n i=1 m j=1 a ib j k(x i, y j ). This is independent of the way of representing f and g from the expression f, g = m j=1 b jf(y j ) = n i=1 a ig(x i ).
42 42 / 24 Proof of Theorem 2 II Reproducing property on H 0 : f, k(, x) = n i=1 a ik(x i, x) = f(x). Welldefined as an inner product: It is easy to see, is bilinear form, and f 2 = n i,j=1 a ia j k(x i, x j ) 0 by the positive definiteness of f. If f = 0, from CauchySchwarz inequality, 3 f(x) = f, k(, x) f k(, x) = 0 for all x X ; thus f = 0.
43 Proof of Theorem 2 III Completion: Let H be the completion of H 0. H 0 is dense in H by the completion. H is realized by functions: Let {f n } be a Cauchy sequence in H. For each x X, {f n (x)} is a Cauchy sequence, because f n (x) f m (x) = f n f m, k(, x) f n f m k(, x). Define f(x) = lim n f n (x). This value is the same for equivalent sequences, because {f n } {g n } implies f n (x) g n (x) = f n g n, k(, x) f n g n k(, x) 0. Thus, any element [{f n }] in H can be regarded as a function f on X. 3 Note that CauchySchwarz inequality holds without assuming strong positivity of the inner product. 43 / 24
Elements of Positive Definite Kernel and Reproducing Kernel Hilbert Space
Elements of Positive Definite Kernel and Reproducing Kernel Hilbert Space Statistical Inference with Reproducing Kernel Hilbert Space Kenji Fukumizu Institute of Statistical Mathematics, ROIS Department
More informationTheory of Positive Definite Kernel and Reproducing Kernel Hilbert Space
Theory of Positive Definite Kernel and Reproducing Kernel Hilbert Space Statistical Inference with Reproducing Kernel Hilbert Space Kenji Fukumizu Institute of Statistical Mathematics, ROIS Department
More informationFinitedimensional spaces. C n is the space of ntuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product
Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a prehilbert space, or a unitary space) if there is a mapping (, )
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationReproducing Kernel Hilbert Spaces Class 03, 15 February 2006 Andrea Caponnetto
Reproducing Kernel Hilbert Spaces 9.520 Class 03, 15 February 2006 Andrea Caponnetto About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing Kernel Hilbert
More informationOPERATOR THEORY ON HILBERT SPACE. Class notes. John Petrovic
OPERATOR THEORY ON HILBERT SPACE Class notes John Petrovic Contents Chapter 1. Hilbert space 1 1.1. Definition and Properties 1 1.2. Orthogonality 3 1.3. Subspaces 7 1.4. Weak topology 9 Chapter 2. Operators
More informationHilbert spaces. 1. CauchySchwarzBunyakowsky inequality
(October 29, 2016) Hilbert spaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ [This document is http://www.math.umn.edu/ garrett/m/fun/notes 201617/03 hsp.pdf] Hilbert spaces are
More informationCHAPTER VIII HILBERT SPACES
CHAPTER VIII HILBERT SPACES DEFINITION Let X and Y be two complex vector spaces. A map T : X Y is called a conjugatelinear transformation if it is a reallinear transformation from X into Y, and if T (λx)
More informationAN INTRODUCTION TO THE THEORY OF REPRODUCING KERNEL HILBERT SPACES
AN INTRODUCTION TO THE THEORY OF REPRODUCING KERNEL HILBERT SPACES VERN I PAULSEN Abstract These notes give an introduction to the theory of reproducing kernel Hilbert spaces and their multipliers We begin
More informationFall TMA4145 Linear Methods. Solutions to exercise set 9. 1 Let X be a Hilbert space and T a bounded linear operator on X.
TMA445 Linear Methods Fall 26 Norwegian University of Science and Technology Department of Mathematical Sciences Solutions to exercise set 9 Let X be a Hilbert space and T a bounded linear operator on
More informationCIS 520: Machine Learning Oct 09, Kernel Methods
CIS 520: Machine Learning Oct 09, 207 Kernel Methods Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture They may or may not cover all the material discussed
More informationIntroduction to Signal Spaces
Introduction to Signal Spaces Selin Aviyente Department of Electrical and Computer Engineering Michigan State University January 12, 2010 Motivation Outline 1 Motivation 2 Vector Space 3 Inner Product
More informationC.6 Adjoints for Operators on Hilbert Spaces
C.6 Adjoints for Operators on Hilbert Spaces 317 Additional Problems C.11. Let E R be measurable. Given 1 p and a measurable weight function w: E (0, ), the weighted L p space L p s (R) consists of all
More informationOctober 25, 2013 INNER PRODUCT SPACES
October 25, 2013 INNER PRODUCT SPACES RODICA D. COSTIN Contents 1. Inner product 2 1.1. Inner product 2 1.2. Inner product spaces 4 2. Orthogonal bases 5 2.1. Existence of an orthogonal basis 7 2.2. Orthogonal
More informationInfinitedimensional Vector Spaces and Sequences
2 Infinitedimensional Vector Spaces and Sequences After the introduction to frames in finitedimensional vector spaces in Chapter 1, the rest of the book will deal with expansions in infinitedimensional
More informationFunctional Analysis Review
Functional Analysis Review Lorenzo Rosasco slides courtesy of Andre Wibisono 9.520: Statistical Learning Theory and Applications September 9, 2013 1 2 3 4 Vector Space A vector space is a set V with binary
More informationInequalities in Hilbert Spaces
Inequalities in Hilbert Spaces Jan Wigestrand Master of Science in Mathematics Submission date: March 8 Supervisor: Eugenia Malinnikova, MATH Norwegian University of Science and Technology Department of
More informationTHEOREMS, ETC., FOR MATH 515
THEOREMS, ETC., FOR MATH 515 Proposition 1 (=comment on page 17). If A is an algebra, then any finite union or finite intersection of sets in A is also in A. Proposition 2 (=Proposition 1.1). For every
More informationReal Analysis Notes. Thomas Goller
Real Analysis Notes Thomas Goller September 4, 2011 Contents 1 Abstract Measure Spaces 2 1.1 Basic Definitions........................... 2 1.2 Measurable Functions........................ 2 1.3 Integration..............................
More informationWiener Measure and Brownian Motion
Chapter 16 Wiener Measure and Brownian Motion Diffusion of particles is a product of their apparently random motion. The density u(t, x) of diffusing particles satisfies the diffusion equation (16.1) u
More informationPROBLEMS. (b) (Polarization Identity) Show that in any inner product space
1 Professor Carl Cowen Math 54600 Fall 09 PROBLEMS 1. (Geometry in Inner Product Spaces) (a) (Parallelogram Law) Show that in any inner product space x + y 2 + x y 2 = 2( x 2 + y 2 ). (b) (Polarization
More informationReview and problem list for Applied Math I
Review and problem list for Applied Math I (This is a first version of a serious review sheet; it may contain errors and it certainly omits a number of topic which were covered in the course. Let me know
More informationKernel Methods. Machine Learning A W VO
Kernel Methods Machine Learning A 708.063 07W VO Outline 1. Dual representation 2. The kernel concept 3. Properties of kernels 4. Examples of kernel machines Kernel PCA Support vector regression (Relevance
More informationREAL AND COMPLEX ANALYSIS
REAL AND COMPLE ANALYSIS Third Edition Walter Rudin Professor of Mathematics University of Wisconsin, Madison Version 1.1 No rights reserved. Any part of this work can be reproduced or transmitted in any
More informationReproducing Kernel Hilbert Spaces
Reproducing Kernel Hilbert Spaces Lorenzo Rosasco 9.520 Class 03 February 12, 2007 About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing Kernel Hilbert
More informationSupport Vector Machines
Wien, June, 2010 Paul Hofmarcher, Stefan Theussl, WU Wien Hofmarcher/Theussl SVM 1/21 Linear Separable Separating Hyperplanes NonLinear Separable SoftMargin Hyperplanes Hofmarcher/Theussl SVM 2/21 (SVM)
More informationFunctional Analysis II held by Prof. Dr. Moritz Weber in summer 18
Functional Analysis II held by Prof. Dr. Moritz Weber in summer 18 General information on organisation Tutorials and admission for the final exam To take part in the final exam of this course, 50 % of
More informationFunctional Analysis F3/F4/NVP (2005) Homework assignment 2
Functional Analysis F3/F4/NVP (25) Homework assignment 2 All students should solve the following problems:. Section 2.7: Problem 8. 2. Let x (t) = t 2 e t/2, x 2 (t) = te t/2 and x 3 (t) = e t/2. Orthonormalize
More informationAn introduction to some aspects of functional analysis
An introduction to some aspects of functional analysis Stephen Semmes Rice University Abstract These informal notes deal with some very basic objects in functional analysis, including norms and seminorms
More informationReproducing Kernel Hilbert Spaces
9.520: Statistical Learning Theory and Applications February 10th, 2010 Reproducing Kernel Hilbert Spaces Lecturer: Lorenzo Rosasco Scribe: Greg Durrett 1 Introduction In the previous two lectures, we
More informationExercise Solutions to Functional Analysis
Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n
More informationA Brief Introduction to Functional Analysis
A Brief Introduction to Functional Analysis Sungwook Lee Department of Mathematics University of Southern Mississippi sunglee@usm.edu July 5, 2007 Definition 1. An algebra A is a vector space over C with
More informationKernels MIT Course Notes
Kernels MIT 15.097 Course Notes Cynthia Rudin Credits: Bartlett, Schölkopf and Smola, Cristianini and ShaweTaylor The kernel trick that I m going to show you applies much more broadly than SVM, but we
More informationMATHEMATICS. Course Syllabus. Section A: Linear Algebra. Subject Code: MA. Course Structure. Ordinary Differential Equations
MATHEMATICS Subject Code: MA Course Structure Sections/Units Section A Section B Section C Linear Algebra Complex Analysis Real Analysis Topics Section D Section E Section F Section G Section H Section
More informationCHAPTER V DUAL SPACES
CHAPTER V DUAL SPACES DEFINITION Let (X, T ) be a (real) locally convex topological vector space. By the dual space X, or (X, T ), of X we mean the set of all continuous linear functionals on X. By the
More informationMath Linear Algebra II. 1. Inner Products and Norms
Math 342  Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,
More information16 1 Basic Facts from Functional Analysis and Banach Lattices
16 1 Basic Facts from Functional Analysis and Banach Lattices 1.2.3 Banach Steinhaus Theorem Another fundamental theorem of functional analysis is the Banach Steinhaus theorem, or the Uniform Boundedness
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationEncyclopedia of Mathematics, Supplemental Vol. 3, Kluwer Acad. Publishers, Dordrecht,
Encyclopedia of Mathematics, Supplemental Vol. 3, Kluwer Acad. Publishers, Dordrecht, 2001, 328329 1 Reproducing kernel Consider an abstract set E and a linear set F of functions f : E C. Assume that
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,
More informationLecture 3: Review of Linear Algebra
ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,
More informationSince G is a compact Lie group, we can apply Schur orthogonality to see that G χ π (g) 2 dg =
Problem 1 Show that if π is an irreducible representation of a compact lie group G then π is also irreducible. Give an example of a G and π such that π = π, and another for which π π. Is this true for
More informationNotions such as convergent sequence and Cauchy sequence make sense for any metric space. Convergent Sequences are Cauchy
Banach Spaces These notes provide an introduction to Banach spaces, which are complete normed vector spaces. For the purposes of these notes, all vector spaces are assumed to be over the real numbers.
More informationExercises to Applied Functional Analysis
Exercises to Applied Functional Analysis Exercises to Lecture 1 Here are some exercises about metric spaces. Some of the solutions can be found in my own additional lecture notes on Blackboard, as the
More information2. Review of Linear Algebra
2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 26 of the web textbook 1 2.1 Vectors 2.1.1
More informationA Concise Course on Stochastic Partial Differential Equations
A Concise Course on Stochastic Partial Differential Equations Michael Röckner Reference: C. Prevot, M. Röckner: Springer LN in Math. 1905, Berlin (2007) And see the references therein for the original
More informationKernel Methods. Outline
Kernel Methods Quang Nguyen University of Pittsburgh CS 3750, Fall 2011 Outline Motivation Examples Kernels Definitions Kernel trick Basic properties Mercer condition Constructing feature space Hilbert
More informationThe weak topology of locally convex spaces and the weak* topology of their duals
The weak topology of locally convex spaces and the weak* topology of their duals Jordan Bell jordan.bell@gmail.com Department of Mathematics, University of Toronto April 3, 2014 1 Introduction These notes
More informationBERGMAN KERNEL ON COMPACT KÄHLER MANIFOLDS
BERGMAN KERNEL ON COMPACT KÄHLER MANIFOLDS SHOO SETO Abstract. These are the notes to an expository talk I plan to give at MGSC on Kähler Geometry aimed for beginning graduate students in hopes to motivate
More informationNotes on Integrable Functions and the Riesz Representation Theorem Math 8445, Winter 06, Professor J. Segert. f(x) = f + (x) + f (x).
References: Notes on Integrable Functions and the Riesz Representation Theorem Math 8445, Winter 06, Professor J. Segert Evans, Partial Differential Equations, Appendix 3 Reed and Simon, Functional Analysis,
More information9.2 Support Vector Machines 159
9.2 Support Vector Machines 159 9.2.3 Kernel Methods We have all the tools together now to make an exciting step. Let us summarize our findings. We are interested in regularized estimation problems of
More informationTopological vectorspaces
(July 25, 2011) Topological vectorspaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ Natural nonfréchet spaces Topological vector spaces Quotients and linear maps More topological
More informationGeneral Inner Product & Fourier Series
General Inner Products 1 General Inner Product & Fourier Series Advanced Topics in Linear Algebra, Spring 2014 Cameron Braithwaite 1 General Inner Product The inner product is an algebraic operation that
More informationLECTURE 5: THE METHOD OF STATIONARY PHASE
LECTURE 5: THE METHOD OF STATIONARY PHASE Some notions.. A crash course on Fourier transform For j =,, n, j = x j. D j = i j. For any multiindex α = (α,, α n ) N n. α = α + + α n. α! = α! α n!. x α =
More informationThe Hilbert Space of Random Variables
The Hilbert Space of Random Variables Electrical Engineering 126 (UC Berkeley) Spring 2018 1 Outline Fix a probability space and consider the set H := {X : X is a realvalued random variable with E[X 2
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationADJOINTS, ABSOLUTE VALUES AND POLAR DECOMPOSITIONS
J. OPERATOR THEORY 44(2000), 243 254 c Copyright by Theta, 2000 ADJOINTS, ABSOLUTE VALUES AND POLAR DECOMPOSITIONS DOUGLAS BRIDGES, FRED RICHMAN and PETER SCHUSTER Communicated by William B. Arveson Abstract.
More informationKarhunenLoève decomposition of Gaussian measures on Banach spaces
KarhunenLoève decomposition of Gaussian measures on Banach spaces JeanCharles Croix jeancharles.croix@emse.fr Génie Mathématique et Industriel (GMI) First workshop on Gaussian processes at SaintEtienne
More informationLinear Algebra and Dirac Notation, Pt. 2
Linear Algebra and Dirac Notation, Pt. 2 PHYS 500  Southern Illinois University February 1, 2017 PHYS 500  Southern Illinois University Linear Algebra and Dirac Notation, Pt. 2 February 1, 2017 1 / 14
More informationVector Spaces, Affine Spaces, and Metric Spaces
Vector Spaces, Affine Spaces, and Metric Spaces 2 This chapter is only meant to give a short overview of the most important concepts in linear algebra, affine spaces, and metric spaces and is not intended
More informationExtension of positive definite functions
University of Iowa Iowa Research Online Theses and Dissertations Spring 2013 Extension of positive definite functions Robert Niedzialomski University of Iowa Copyright 2013 Robert Niedzialomski This dissertation
More informationElements of Convex Optimization Theory
Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a selfcontained overview of elements of convex optimization
More informationA Précis of Functional Analysis for Engineers DRAFT NOT FOR DISTRIBUTION. JeanFrançois Hiller and KlausJürgen Bathe
A Précis of Functional Analysis for Engineers DRAFT NOT FOR DISTRIBUTION JeanFrançois Hiller and KlausJürgen Bathe August 29, 22 1 Introduction The purpose of this précis is to review some classical
More informationwhere the bar indicates complex conjugation. Note that this implies that, from Property 2, x,αy = α x,y, x,y X, α C.
Lecture 4 Inner product spaces Of course, you are familiar with the idea of inner product spaces at least finitedimensional ones. Let X be an abstract vector space with an inner product, denoted as,,
More informationSome Useful Background for Talk on the Fast JohnsonLindenstrauss Transform
Some Useful Background for Talk on the Fast JohnsonLindenstrauss Transform Nir Ailon May 22, 2007 This writeup includes very basic background material for the talk on the Fast Johnson Lindenstrauss Transform
More informationKarhunenLoève decomposition of Gaussian measures on Banach spaces
KarhunenLoève decomposition of Gaussian measures on Banach spaces JeanCharles Croix GT APSSE  April 2017, the 13th joint work with Xavier Bay. 1 / 29 Sommaire 1 Preliminaries on Gaussian processes 2
More informationContinuity of convex functions in normed spaces
Continuity of convex functions in normed spaces In this chapter, we consider continuity properties of realvalued convex functions defined on open convex sets in normed spaces. Recall that every infinitedimensional
More informationContents. 2 Sequences and Series Approximation by Rational Numbers Sequences Basics on Sequences...
Contents 1 Real Numbers: The Basics... 1 1.1 Notation... 1 1.2 Natural Numbers... 4 1.3 Integers... 5 1.4 Fractions and Rational Numbers... 10 1.4.1 Introduction... 10 1.4.2 Powers and Radicals of Rational
More information1 The Projection Theorem
Several Important Theorems by Francis J. Narcowich November, 14 1 The Projection Theorem Let H be a Hilbert space. When V is a finite dimensional subspace of H and f H, we can always find a unique p V
More informationIr O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )
Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O
More informationMath 5520 Homework 2 Solutions
Math 552 Homework 2 Solutions March, 26. Consider the function fx) = 2x ) 3 if x, 3x ) 2 if < x 2. Determine for which k there holds f H k, 2). Find D α f for α k. Solution. We show that k = 2. The formulas
More informationFUNCTIONAL ANALYSIS: NOTES AND PROBLEMS
FUNCTIONAL ANALYSIS: NOTES AND PROBLEMS Abstract. These are the notes prepared for the course MTH 405 to be offered to graduate students at IIT Kanpur. Contents 1. Basic Inequalities 1 2. Normed Linear
More informationConvex Feasibility Problems
Laureate Prof. Jonathan Borwein with Matthew Tam http://carma.newcastle.edu.au/drmethods/paseky.html Spring School on Variational Analysis VI Paseky nad Jizerou, April 19 25, 2015 Last Revised: May 6,
More informationThe Kernel Trick, Gram Matrices, and Feature Extraction. CS6787 Lecture 4 Fall 2017
The Kernel Trick, Gram Matrices, and Feature Extraction CS6787 Lecture 4 Fall 2017 Momentum for Principle Component Analysis CS6787 Lecture 3.1 Fall 2017 Principle Component Analysis Setting: find the
More informationDefinition and basic properties of heat kernels I, An introduction
Definition and basic properties of heat kernels I, An introduction Zhiqin Lu, Department of Mathematics, UC Irvine, Irvine CA 92697 April 23, 2010 In this lecture, we will answer the following questions:
More informationMath 127: Course Summary
Math 27: Course Summary Rich Schwartz October 27, 2009 General Information: M27 is a course in functional analysis. Functional analysis deals with normed, infinite dimensional vector spaces. Usually, these
More informationGELFAND S THEOREM. Christopher McMurdie. Advisor: Arlo Caine. Spring 2004
GELFAND S THEOREM Christopher McMurdie Advisor: Arlo Caine Super Advisor: Dr Doug Pickrell Spring 004 This paper is a proof of the first part of Gelfand s Theorem, which states the following: Every compact,
More informationA BRIEF INTRODUCTION TO HILBERT SPACE FRAME THEORY AND ITS APPLICATIONS AMS SHORT COURSE: JOINT MATHEMATICS MEETINGS SAN ANTONIO, 2015 PETER G. CASAZZA Abstract. This is a short introduction to Hilbert
More informationOptimization Theory. A Concise Introduction. Jiongmin Yong
October 11, 017 16:5 wsbook9x6 Book Title Optimization Theory 01708Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 01708Lecture Notes page Optimization
More informationMath 209B Homework 2
Math 29B Homework 2 Edward Burkard Note: All vector spaces are over the field F = R or C 4.6. Two Compactness Theorems. 4. Point Set Topology Exercise 6 The product of countably many sequentally compact
More informationLecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set.
Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. Definition A set of k vectors {u, u 2,..., u k }, where each u i R n, is said to be an
More informationFive MiniCourses on Analysis
Christopher Heil Five MiniCourses on Analysis Metrics, Norms, Inner Products, and Topology Lebesgue Measure and Integral Operator Theory and Functional Analysis Borel and Radon Measures Topological Vector
More informationQuick Review of Matrix and Real Linear Algebra
Division of the Humanities and Social Sciences Quick Review of Matrix and Real Linear Algebra KC Border Subject to constant revision Last major revision: December 12, 2016 Contents 1 Scalar fields 1 2
More informationLinear Algebra Review
January 29, 2013 Table of contents Metrics Metric Given a space X, then d : X X R + 0 and z in X if: d(x, y) = 0 is equivalent to x = y d(x, y) = d(y, x) d(x, y) d(x, z) + d(z, y) is a metric is for all
More informationICML  Kernels & RKHS Workshop. Distances and Kernels for Structured Objects
ICML  Kernels & RKHS Workshop Distances and Kernels for Structured Objects Marco Cuturi  Kyoto University Kernel & RKHS Workshop 1 Outline Distances and Positive Definite Kernels are crucial ingredients
More informationShort note on compact operators  Monday 24 th March, Sylvester ErikssonBique
Short note on compact operators  Monday 24 th March, 2014 Sylvester ErikssonBique 1 Introduction In this note I will give a short outline about the structure theory of compact operators. I restrict attention
More informationInner products. Theorem (basic properties): Given vectors u, v, w in an inner product space V, and a scalar k, the following properties hold:
Inner products Definition: An inner product on a real vector space V is an operation (function) that assigns to each pair of vectors ( u, v) in V a scalar u, v satisfying the following axioms: 1. u, v
More informationANALYSIS QUALIFYING EXAM FALL 2017: SOLUTIONS. 1 cos(nx) lim. n 2 x 2. g n (x) = 1 cos(nx) n 2 x 2. x 2.
ANALYSIS QUALIFYING EXAM FALL 27: SOLUTIONS Problem. Determine, with justification, the it cos(nx) n 2 x 2 dx. Solution. For an integer n >, define g n : (, ) R by Also define g : (, ) R by g(x) = g n
More informationFunctional Analysis. James Emery. Edit: 8/7/15
Functional Analysis James Emery Edit: 8/7/15 Contents 0.1 Green s functions in Ordinary Differential Equations...... 2 0.2 Integral Equations........................ 2 0.2.1 Fredholm Equations...................
More informationLecture Notes 1: Vector spaces
Optimizationbased data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More informationLet R be the line parameterized by x. Let f be a complex function on R that is integrable. The Fourier transform ˆf = F f is. e ikx f(x) dx. (1.
Chapter 1 Fourier transforms 1.1 Introduction Let R be the line parameterized by x. Let f be a complex function on R that is integrable. The Fourier transform ˆf = F f is ˆf(k) = e ikx f(x) dx. (1.1) It
More informationThe CalderonVaillancourt Theorem
The CalderonVaillancourt Theorem What follows is a completely self contained proof of the CalderonVaillancourt Theorem on the L 2 boundedness of pseudodifferential operators. 1 The result Definition
More informationEigenvalues and Eigenfunctions of the Laplacian
The Waterloo Mathematics Review 23 Eigenvalues and Eigenfunctions of the Laplacian Mihai Nica University of Waterloo mcnica@uwaterloo.ca Abstract: The problem of determining the eigenvalues and eigenvectors
More informationGeneral Notation. Exercises and Problems
Exercises and Problems The text contains both Exercises and Problems. The exercises are incorporated into the development of the theory in each section. Additional Problems appear at the end of most sections.
More information2 (Bonus). Let A X consist of points (x, y) such that either x or y is a rational number. Is A measurable? What is its Lebesgue measure?
MA 6454A (Real Analysis), Dr. Chernov Homework assignment 1 (Due 9/5). Prove that every countable set A is measurable and µ(a) = 0. 2 (Bonus). Let A consist of points (x, y) such that either x or y is
More information1.2 Fundamental Theorems of Functional Analysis
1.2 Fundamental Theorems of Functional Analysis 15 Indeed, h = ω ψ ωdx is continuous compactly supported with R hdx = 0 R and thus it has a unique compactly supported primitive. Hence fφ dx = f(ω ψ ωdy)dx
More informationThe Dual of a Hilbert Space
The Dual of a Hilbert Space Let V denote a real Hilbert space with inner product, u,v V. Let V denote the space of bounded linear functionals on V, equipped with the norm, F V sup F v : v V 1. Then for
More information1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)
Notes for 20170130 Most of mathematics is best learned by doing. Linear algebra is no exception. You have had a previous class in which you learned the basics of linear algebra, and you will have plenty
More informationBanach Spaces II: Elementary Banach Space Theory
BS II c Gabriel Nagy Banach Spaces II: Elementary Banach Space Theory Notes from the Functional Analysis Course (Fall 07  Spring 08) In this section we introduce Banach spaces and examine some of their
More information