# Kernel Method: Data Analysis with Positive Definite Kernels

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Kernel Method: Data Analysis with Positive Definite Kernels 2. Positive Definite Kernel and Reproducing Kernel Hilbert Space Kenji Fukumizu The Institute of Statistical Mathematics. Graduate University of Advanced Studies / Tokyo Institute of Technology Nov , 2010 Intensive Course at Tokyo Institute of Technology

2 2 / 24 Outline Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS

3 3 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS

4 4 / 24 Definition of Positive Definite Kernel Definition. Let X be a set. k : X X R is a positive definite kernel if k(x, y) = k(y, x) and for every x 1,..., x n X and c 1,..., c n R n c i c j k(x i, x j ) 0, i,j=1 i.e. the symmetric matrix k(x 1, x 1 ) k(x 1, x n ) (k(x i, x j )) n i,j=1 =..... k(x n, x 1 ) k(x n, x ) is positive semidefinite. The symmetric matrix (k(x i, x j )) n i,j=1 is often called a Gram matrix.

5 5 / 24 Definition: Complex-valued Case Definition. Let X be a set. k : X X C is a positive definite kernel if for every x 1,..., x n X and c 1,..., c n C n c i c j k(x i, x j ) 0. i,j=1 Remark. The Hermitian property k(y, x) = k(x, y) is derived from the positive-definiteness. [Exercise]

6 6 / 24 Some Basic Properties Facts. Assume k : X X C is positive definite. Then, for any x, y in X, 1. k(x, x) k(x, y) 2 k(x, x)k(y, y). Proof. (1) is obvious. For (2), with the fact k(y, x) = k(x, y), the definition of positive definiteness implies that the eigenvalues of the hermitian matrix ( k(x, x) ) k(x, y) k(x, y) k(y, y) is non-negative, thus, its determinant k(x, x)k(y, y) k(x, y) 2 is non-negative.

7 Examples Real valued positive definite kernels on R n : - Linear kernel 1 k 0 (x, y) = x T y - Exponential k E (x, y) = exp(βx T y) (β > 0) - Gaussian RBF (radial basis function) kernel ( k G (x, y) = exp 1 2σ 2 x y 2) (σ > 0) - Laplacian kernel ( k L (x, y) = exp α ) n i=1 x i y i (α > 0) - Polynomial kernel k P (x, y) = (x T y + c) d (c 0, d N) 1 [Exercise] prove that the linear kernel is positive definite. 7 / 24

8 8 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS

9 9 / 24 Reproducing kernel Hilbert space Definition. Let X be a set. A reproducing kernel Hilbert space (RKHS) (over X ) is a Hilbert space H consisting of functions on X such that for each x X there is a function k x H with the property f, k x H = f(x) ( f H) (reproducing property). k(, x) := k x ( ) is called a reproducing kernel of H. Fact 1. A reproducing kernel is Hermitian (or symmetric). Proof. k(y, x) = k(, x), k y = k x, k y = k y, k x = k(, y), k x = k(x, y). Fact 2. The reproducing kernel is unique, if exists. [Exercise]

10 10 / 24 Positive Definite Kernel and RKHS I Proposition 1 (RKHS positive definite kernel) The reproducing kernel of a RKHS is positive definite. Proof. n c i c j k(x i, x j ) = i,j=1 n c i c j k(, x i ), k(, x j ) i,j=1 n = c i k(, x i ), i=1 n j=1 c j k(, x j ) = n c i k(, x i ) 2 0. i=1

11 11 / 24 Positive Definite Kernel and RKHS II Theorem 2 (positive definite kernel RKHS. Moore-Aronszajn) Let k : X X C (or R) be a positive definite kernel on a set X. Then, there uniquely exists a RKHS H k on X such that 1. k(, x) H k for every x X, 2. Span{k(, x) x X } is dense in H k, 3. k is the reproducing kernel on H k, i.e., f, k(, x) H = f(x) ( x X, f H k ). Proof omitted.

12 12 / 24 Positive Definite Kernel and RKHS III One-to-one correspondence between positive definite kernels and RKHS. k H k Proposition 1: RKHS positive definite kernel k. Theorem 2: k H k (injective).

13 13 / 24 RKHS as Feature Space If we define Φ : X H k, x k(, x), then, Φ(x), Φ(y) = k(, x), k(, y) = k(x, y). RKHS associated with a positive definite kernel k gives a desired feature space!! In kernel methods, the above feature map and feature space are always used.

14 14 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS

15 15 / 24 Operations that Preserve Positive Definiteness I Proposition 3 If k i : X X C (i = 1, 2,...) are positive definite kernels, then so are the following: 1. (positive combination) ak 1 + bk 2 (a, b 0). 2. (product) k 1 k 2 (k 1 (x, y)k 2 (x, y)). 3. (limit) lim i k i (x, y), assuming the limit exists. Remark. From Proposition 3, the set of all positive definite kernels is a closed (w.r.t. pointwise convergence) convex cone stable under multiplication.

16 Operations that Preserve Positive Definiteness II Proof. (1): Obvious. (3): The non-negativity in the definition holds also for the limit. (2): It suffices to show that two Hermitian matrices A and B are positive semidefinite, so is their component-wise product. This is done by the following lemma. Definition. For two matrices A and B of the same size, the matrix C with C ij = A ij B ij is called the Hadamard product of A and B. The Hadamard product of A and B is denoted by A B. Lemma 4 Let A and B be non-negative Hermitian matrices of the same size. Then, A B is also non-negative. 16 / 24

17 17 / 24 Operations that Preserve Positive Definiteness III Proof. Let A = UΛU be the eigendecomposition of A, where U = (u 1,..., u p ): a unitary matrix, i.e., U = U T Λ: diagonal matrix with non-negative entries (λ 1,..., λ p ). Then, for arbitrary c 1,..., c p C, p p c i c j (A B) ij = λ a c i c j u a i ū a j B ij = λ a ξ at Bξ a, i,j=1 a=1 a=1 where ξ a = (c 1 u a 1,..., c p u a p) T C p. Since ξ at Bξ a and λ a are non-negative for each a, so is the sum.

18 18 / 24 Feature Map must be Positive Definite Proposition 5 Let V be an vector space with an inner product,. If we have a map Φ : X V, x Φ(x), a positive definite kernel on X is defined by k(x, y) = Φ(x), Φ(y). Proof. Let x 1,..., x n in X and c 1,..., c n C. n i,j=1 c ic j k(x i, x j ) = n i,j=1 c ic j Φ(x i ), Φ(x j ) n = i=1 c iφ(x i ), n j=1 c jφ(x j ) = n i=1 c iφ(x i ) 2 0.

19 Proposition 6 Modification Let k : X X C be a positive definite kernel and f : X C be an arbitrary function. Then, is positive definite. In particular, and k(x, y) = f(x)k(x, y)f(y) f(x)f(y) k(x, y) k(x, x) k(y, y) (normalized kernel) are positive definite. Proof is left as an exercise. 19 / 24

20 20 / 24 Proofs for Positive Definiteness of Examples Linear kernel: Proposition 5 Exponential: exp(βx T y) = 1 + βx T y + β2 2! (xt y) 2 + β3 3! (xt y) 3 + Use Proposition 3. Gaussian RBF kernel: ( exp 1 2σ 2 x y 2) = exp ( x 2 ) ( x T y ) 2σ 2 exp σ 2 exp ( y 2 ) 2σ 2. Apply Proposition 6. Laplacian kernel: The proof is shown later. Polynomial kernel: Just sum and product.

21 21 / 24 Positive definite kernel Definition and examples of positive definite kernel Reproducing kernel Hilbert space RKHS and positive definite kernel Some basic properties Properties of positive definite kernels Properties of RKHS

22 Proposition 7 Definition by Evaluation Map Let H be a Hilbert space consisting of functions on a set X. Then, H is a RKHS if and only if the evaluation map e x : H K, e x (f) = f(x), is a continuous linear functional for each x X. Proof. Assume H is a RKHS. The boundedness of e x is obvious from e x (f) = f, k x k x f. Conversely, assume the evaluation map is continuous. By Riesz lemma, there is k x H such that f, k x = e x (f) = f(x), which means H is a RKHS having k x as a reproducing kernel. 22 / 24

23 Continuity The functions in a RKHS are "nice" functions under some conditions. Proposition 8 Let k be a positive definite kernel on a topological space X, and H k be the associated RKHS. If Re[k(y, x)] is continuous for every x, y X, then all the functions in H k are continuous. Proof. Let f be an arbitrary function in H k. f(x) f(y) = f, k(, x) k(, y) f k(, x) k(, y). The assertion is easy from k(, x) k(, y) 2 = k(x, x) + k(y, y) 2Re[k(x, y)]. Remark. If k(x, y) is differentiable, then all the functions in H k are differentiable. c.f. L 2 space contains non-continuous functions. 23 / 24

24 24 / 24 Summary of Sections 1 and 2 We would like to use a feature map Φ : X H to incorporate nonlinearity or high order moments. The inner product in the feature space must be computed efficiently. Ideally, Φ(x), Φ(y) = k(x, y). To satisfy the above relation, the kernel k must be positive definite. A positive definite kernel k defines an associated RKHS, where k is the reproducing kernel; k(, x), k(, y) = k(x, y). Use a RKHS as a feature space, and Φ : x k(, x) as the feature map.

25 25 / 24 Appendix: Quick introduction to Hilbert spaces Definition of Hilbert space Basic properties of Hilbert space Appendix: Proofs Proof of Theorem 2

26 26 / 24 Vector space with inner product I Definition. V : vector space over a field K = R or C. V is called an inner product space if it has an inner product (or scalar product, dot product) (, ) : V V K such that for every x, y, z V 1. (Strong positivity) (x, x) 0, and (x, x) = 0 if and only if x = 0, 2. (Addition) (x + y, z) = (x, z) + (y, z), 3. (Scalar multiplication) (αx, y) = α(x, y) ( α K), 4. (Hermitian) (y, x) = (x, y).

27 Vector space with inner product II (V, (, )): inner product space. Norm of x V : x = (x, x) 1/2. Metric between x and y: d(x, y) = x y. Theorem 9 Cauchy-Schwarz inequality (x, y) x y. Remark: Cauchy-Schwarz inequality holds without requiring x = 0 x = / 24

28 Hilbert space I Definition. A vector space with inner product (H, (, )) is called Hilbert space if the induced metric is complete, i.e. every Cauchy sequence 2 converges to an element in H. Remark 1: A Hilbert space may be either finite or infinite dimensional. Example 1. R n and C n are finite dimensional Hilbert space with the ordinary inner product (x, y) R n = n i=1 x iy i or (x, y) C n = n i=1 x iy i. 2 A sequence {x n} n=1 in a metric space (X, d) is called a Cauchy sequence if d(x n, x m) 0 for n, m. 28 / 24

29 29 / 24 Hilbert space II Example 2. L 2 (Ω, µ). Let (Ω, B, µ) is a measure space. { L = f : Ω C } f 2 dµ <. The inner product on L is define by (f, g) = fgdµ. L 2 (Ω, µ) is defined by the equivalent classes identifying f and g if their values differ only on a measure-zero set. - L 2 (Ω, µ) is complete. - L 2 (R n, dx) is infinite dimensional.

30 30 / 24 Appendix: Quick introduction to Hilbert spaces Definition of Hilbert space Basic properties of Hilbert space Appendix: Proofs Proof of Theorem 2

31 31 / 24 Orthogonal complement. Orthogonality Let H be a Hilbert space and V be a closed subspace. V := {x H (x, y) = 0 for all y V } is a closed subspace, and called the orthogonal complement. Orthogonal projection. Let H be a Hilbert space and V be a closed subspace. Every x H can be uniquely decomposed x = y + z, y V and z V, that is, H = V V.

32 32 / 24 Complete orthonormal system I ONS and CONS. A subset {u i } i I of H is called an orthonormal system (ONS) if (u i, u j ) = δ ij (δ ij is Kronecker s delta). A subset {u i } i I of H is called a complete orthonormal system (CONS) if it is ONS and if (x, u i ) = 0 ( i I) implies x = 0. Fact: Any ONS in a Hilbert space can be extended to a CONS.

33 33 / 24 Complete orthonormal system II Separability A Hilbert space is separable if it has a countable CONS. Assumption In this course, a Hilbert space is always assumed to be separable.

34 Complete orthonormal system III Theorem 10 (Fourier series expansion) Let {u i } i=1 be a CONS of a separable Hilbert space. For each x H, Proof omitted. x = i=1 (x, u i)u i, (Fourier expansion) x 2 = i=1 (x, u i) 2. (Parseval s equality) Example: CONS of L 2 ([0 2π], dx) u n (t) = 1 2π e 1nt (n = 0, 1, 2,...) Then, f(t) = n=0 a nu n (t) is the (ordinary) Fourier expansion of a periodic function. 34 / 24

35 35 / 24 Bounded operator I Let H 1 and H 2 be Hilbert spaces. A linear transform T : H 1 H 2 is often called operator. Definition. A linear operator H 1 and H 2 is called bounded if sup T x H2 <. x H1 =1 The operator norm of a bounded operator T is defined by T = T x H2 sup T x H2 = sup. x H1 =1 x 0 x H1 (Corresponds to the largest singular value of a matrix.) Fact. If T : H 1 H 2 is bounded, T x H2 T x H1.

36 36 / 24 Proposition 11 Bounded operator II A linear operator is bounded if and only if it is continuous. Proof. Assume T : H 1 H 2 is bounded. Then, means continuity of T. T x T x 0 T x x 0 Assume T is continuous. For any ε > 0, there is δ > 0 such that T x < ε for all x H 1 with x < 2δ. Then, 1 sup T x = sup x =1 x =δ δ T x ε δ.

37 Riesz lemma I Definition. A linear functional is a linear transform from H to C (or R). The vector space of all the bounded (continuous) linear functionals called the dual space of H, and is denoted by H. Theorem 12 (Riesz lemma) For each φ H, there is a unique y φ H such that φ(x) = (x, y φ ) ( x H). Proof. Consider the case of R for simplicity. ) Obvious by Cauchy-Schwartz. 37 / 24

38 38 / 24 Riesz lemma II ) If φ(x) = 0 for all x, take y = 0. Otherwise, let V = {x H φ(x) = 0}. Since φ is a bounded linear functional, V is a closed subspace, and V H. Take z V with z = 1. By orthogonal decomposition, for any x H, x (x, z)z V. Apply φ, then Take y φ = φ(z)z. φ(x) (x, z)φ(z) = 0, i.e., φ(x) = (x, φ(z)z).

39 Riesz lemma III 39 / 24

40 40 / 24 Appendix: Quick introduction to Hilbert spaces Definition of Hilbert space Basic properties of Hilbert space Appendix: Proofs Proof of Theorem 2

41 41 / 24 Proof. (Described in R case.) Proof of Theorem 2 I Construction of an inner product space: H 0 := Span{k(, x) x X }. Define an inner product on H 0 : for f = n i=1 a ik(, x i ) and g = m j=1 b jk(, y j ), f, g := n i=1 m j=1 a ib j k(x i, y j ). This is independent of the way of representing f and g from the expression f, g = m j=1 b jf(y j ) = n i=1 a ig(x i ).

42 42 / 24 Proof of Theorem 2 II Reproducing property on H 0 : f, k(, x) = n i=1 a ik(x i, x) = f(x). Well-defined as an inner product: It is easy to see, is bilinear form, and f 2 = n i,j=1 a ia j k(x i, x j ) 0 by the positive definiteness of f. If f = 0, from Cauchy-Schwarz inequality, 3 f(x) = f, k(, x) f k(, x) = 0 for all x X ; thus f = 0.

43 Proof of Theorem 2 III Completion: Let H be the completion of H 0. H 0 is dense in H by the completion. H is realized by functions: Let {f n } be a Cauchy sequence in H. For each x X, {f n (x)} is a Cauchy sequence, because f n (x) f m (x) = f n f m, k(, x) f n f m k(, x). Define f(x) = lim n f n (x). This value is the same for equivalent sequences, because {f n } {g n } implies f n (x) g n (x) = f n g n, k(, x) f n g n k(, x) 0. Thus, any element [{f n }] in H can be regarded as a function f on X. 3 Note that Cauchy-Schwarz inequality holds without assuming strong positivity of the inner product. 43 / 24

### Elements of Positive Definite Kernel and Reproducing Kernel Hilbert Space

Elements of Positive Definite Kernel and Reproducing Kernel Hilbert Space Statistical Inference with Reproducing Kernel Hilbert Space Kenji Fukumizu Institute of Statistical Mathematics, ROIS Department

### Theory of Positive Definite Kernel and Reproducing Kernel Hilbert Space

Theory of Positive Definite Kernel and Reproducing Kernel Hilbert Space Statistical Inference with Reproducing Kernel Hilbert Space Kenji Fukumizu Institute of Statistical Mathematics, ROIS Department

### Finite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product

Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )

### Functional Analysis Review

Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### Reproducing Kernel Hilbert Spaces Class 03, 15 February 2006 Andrea Caponnetto

Reproducing Kernel Hilbert Spaces 9.520 Class 03, 15 February 2006 Andrea Caponnetto About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing Kernel Hilbert

### OPERATOR THEORY ON HILBERT SPACE. Class notes. John Petrovic

OPERATOR THEORY ON HILBERT SPACE Class notes John Petrovic Contents Chapter 1. Hilbert space 1 1.1. Definition and Properties 1 1.2. Orthogonality 3 1.3. Subspaces 7 1.4. Weak topology 9 Chapter 2. Operators

### Hilbert spaces. 1. Cauchy-Schwarz-Bunyakowsky inequality

(October 29, 2016) Hilbert spaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ [This document is http://www.math.umn.edu/ garrett/m/fun/notes 2016-17/03 hsp.pdf] Hilbert spaces are

### CHAPTER VIII HILBERT SPACES

CHAPTER VIII HILBERT SPACES DEFINITION Let X and Y be two complex vector spaces. A map T : X Y is called a conjugate-linear transformation if it is a reallinear transformation from X into Y, and if T (λx)

### AN INTRODUCTION TO THE THEORY OF REPRODUCING KERNEL HILBERT SPACES

AN INTRODUCTION TO THE THEORY OF REPRODUCING KERNEL HILBERT SPACES VERN I PAULSEN Abstract These notes give an introduction to the theory of reproducing kernel Hilbert spaces and their multipliers We begin

### Fall TMA4145 Linear Methods. Solutions to exercise set 9. 1 Let X be a Hilbert space and T a bounded linear operator on X.

TMA445 Linear Methods Fall 26 Norwegian University of Science and Technology Department of Mathematical Sciences Solutions to exercise set 9 Let X be a Hilbert space and T a bounded linear operator on

### CIS 520: Machine Learning Oct 09, Kernel Methods

CIS 520: Machine Learning Oct 09, 207 Kernel Methods Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture They may or may not cover all the material discussed

### Introduction to Signal Spaces

Introduction to Signal Spaces Selin Aviyente Department of Electrical and Computer Engineering Michigan State University January 12, 2010 Motivation Outline 1 Motivation 2 Vector Space 3 Inner Product

### C.6 Adjoints for Operators on Hilbert Spaces

C.6 Adjoints for Operators on Hilbert Spaces 317 Additional Problems C.11. Let E R be measurable. Given 1 p and a measurable weight function w: E (0, ), the weighted L p space L p s (R) consists of all

### October 25, 2013 INNER PRODUCT SPACES

October 25, 2013 INNER PRODUCT SPACES RODICA D. COSTIN Contents 1. Inner product 2 1.1. Inner product 2 1.2. Inner product spaces 4 2. Orthogonal bases 5 2.1. Existence of an orthogonal basis 7 2.2. Orthogonal

### Infinite-dimensional Vector Spaces and Sequences

2 Infinite-dimensional Vector Spaces and Sequences After the introduction to frames in finite-dimensional vector spaces in Chapter 1, the rest of the book will deal with expansions in infinitedimensional

### Functional Analysis Review

Functional Analysis Review Lorenzo Rosasco slides courtesy of Andre Wibisono 9.520: Statistical Learning Theory and Applications September 9, 2013 1 2 3 4 Vector Space A vector space is a set V with binary

### Inequalities in Hilbert Spaces

Inequalities in Hilbert Spaces Jan Wigestrand Master of Science in Mathematics Submission date: March 8 Supervisor: Eugenia Malinnikova, MATH Norwegian University of Science and Technology Department of

### THEOREMS, ETC., FOR MATH 515

THEOREMS, ETC., FOR MATH 515 Proposition 1 (=comment on page 17). If A is an algebra, then any finite union or finite intersection of sets in A is also in A. Proposition 2 (=Proposition 1.1). For every

### Real Analysis Notes. Thomas Goller

Real Analysis Notes Thomas Goller September 4, 2011 Contents 1 Abstract Measure Spaces 2 1.1 Basic Definitions........................... 2 1.2 Measurable Functions........................ 2 1.3 Integration..............................

### Wiener Measure and Brownian Motion

Chapter 16 Wiener Measure and Brownian Motion Diffusion of particles is a product of their apparently random motion. The density u(t, x) of diffusing particles satisfies the diffusion equation (16.1) u

### PROBLEMS. (b) (Polarization Identity) Show that in any inner product space

1 Professor Carl Cowen Math 54600 Fall 09 PROBLEMS 1. (Geometry in Inner Product Spaces) (a) (Parallelogram Law) Show that in any inner product space x + y 2 + x y 2 = 2( x 2 + y 2 ). (b) (Polarization

### Review and problem list for Applied Math I

Review and problem list for Applied Math I (This is a first version of a serious review sheet; it may contain errors and it certainly omits a number of topic which were covered in the course. Let me know

### Kernel Methods. Machine Learning A W VO

Kernel Methods Machine Learning A 708.063 07W VO Outline 1. Dual representation 2. The kernel concept 3. Properties of kernels 4. Examples of kernel machines Kernel PCA Support vector regression (Relevance

### REAL AND COMPLEX ANALYSIS

REAL AND COMPLE ANALYSIS Third Edition Walter Rudin Professor of Mathematics University of Wisconsin, Madison Version 1.1 No rights reserved. Any part of this work can be reproduced or transmitted in any

### Reproducing Kernel Hilbert Spaces

Reproducing Kernel Hilbert Spaces Lorenzo Rosasco 9.520 Class 03 February 12, 2007 About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing Kernel Hilbert

### Support Vector Machines

Wien, June, 2010 Paul Hofmarcher, Stefan Theussl, WU Wien Hofmarcher/Theussl SVM 1/21 Linear Separable Separating Hyperplanes Non-Linear Separable Soft-Margin Hyperplanes Hofmarcher/Theussl SVM 2/21 (SVM)

### Functional Analysis II held by Prof. Dr. Moritz Weber in summer 18

Functional Analysis II held by Prof. Dr. Moritz Weber in summer 18 General information on organisation Tutorials and admission for the final exam To take part in the final exam of this course, 50 % of

### Functional Analysis F3/F4/NVP (2005) Homework assignment 2

Functional Analysis F3/F4/NVP (25) Homework assignment 2 All students should solve the following problems:. Section 2.7: Problem 8. 2. Let x (t) = t 2 e t/2, x 2 (t) = te t/2 and x 3 (t) = e t/2. Orthonormalize

### An introduction to some aspects of functional analysis

An introduction to some aspects of functional analysis Stephen Semmes Rice University Abstract These informal notes deal with some very basic objects in functional analysis, including norms and seminorms

### Reproducing Kernel Hilbert Spaces

9.520: Statistical Learning Theory and Applications February 10th, 2010 Reproducing Kernel Hilbert Spaces Lecturer: Lorenzo Rosasco Scribe: Greg Durrett 1 Introduction In the previous two lectures, we

### Exercise Solutions to Functional Analysis

Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n

### A Brief Introduction to Functional Analysis

A Brief Introduction to Functional Analysis Sungwook Lee Department of Mathematics University of Southern Mississippi sunglee@usm.edu July 5, 2007 Definition 1. An algebra A is a vector space over C with

### Kernels MIT Course Notes

Kernels MIT 15.097 Course Notes Cynthia Rudin Credits: Bartlett, Schölkopf and Smola, Cristianini and Shawe-Taylor The kernel trick that I m going to show you applies much more broadly than SVM, but we

### MATHEMATICS. Course Syllabus. Section A: Linear Algebra. Subject Code: MA. Course Structure. Ordinary Differential Equations

MATHEMATICS Subject Code: MA Course Structure Sections/Units Section A Section B Section C Linear Algebra Complex Analysis Real Analysis Topics Section D Section E Section F Section G Section H Section

### CHAPTER V DUAL SPACES

CHAPTER V DUAL SPACES DEFINITION Let (X, T ) be a (real) locally convex topological vector space. By the dual space X, or (X, T ), of X we mean the set of all continuous linear functionals on X. By the

### Math Linear Algebra II. 1. Inner Products and Norms

Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,

### 16 1 Basic Facts from Functional Analysis and Banach Lattices

16 1 Basic Facts from Functional Analysis and Banach Lattices 1.2.3 Banach Steinhaus Theorem Another fundamental theorem of functional analysis is the Banach Steinhaus theorem, or the Uniform Boundedness

### Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Encyclopedia of Mathematics, Supplemental Vol. 3, Kluwer Acad. Publishers, Dordrecht,

Encyclopedia of Mathematics, Supplemental Vol. 3, Kluwer Acad. Publishers, Dordrecht, 2001, 328-329 1 Reproducing kernel Consider an abstract set E and a linear set F of functions f : E C. Assume that

### Lecture 3: Review of Linear Algebra

ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,

### Lecture 3: Review of Linear Algebra

ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,

### Since G is a compact Lie group, we can apply Schur orthogonality to see that G χ π (g) 2 dg =

Problem 1 Show that if π is an irreducible representation of a compact lie group G then π is also irreducible. Give an example of a G and π such that π = π, and another for which π π. Is this true for

### Notions such as convergent sequence and Cauchy sequence make sense for any metric space. Convergent Sequences are Cauchy

Banach Spaces These notes provide an introduction to Banach spaces, which are complete normed vector spaces. For the purposes of these notes, all vector spaces are assumed to be over the real numbers.

### Exercises to Applied Functional Analysis

Exercises to Applied Functional Analysis Exercises to Lecture 1 Here are some exercises about metric spaces. Some of the solutions can be found in my own additional lecture notes on Blackboard, as the

### 2. Review of Linear Algebra

2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear

### Lecture 2: Linear Algebra Review

EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1

### A Concise Course on Stochastic Partial Differential Equations

A Concise Course on Stochastic Partial Differential Equations Michael Röckner Reference: C. Prevot, M. Röckner: Springer LN in Math. 1905, Berlin (2007) And see the references therein for the original

### Kernel Methods. Outline

Kernel Methods Quang Nguyen University of Pittsburgh CS 3750, Fall 2011 Outline Motivation Examples Kernels Definitions Kernel trick Basic properties Mercer condition Constructing feature space Hilbert

### The weak topology of locally convex spaces and the weak-* topology of their duals

The weak topology of locally convex spaces and the weak-* topology of their duals Jordan Bell jordan.bell@gmail.com Department of Mathematics, University of Toronto April 3, 2014 1 Introduction These notes

### BERGMAN KERNEL ON COMPACT KÄHLER MANIFOLDS

BERGMAN KERNEL ON COMPACT KÄHLER MANIFOLDS SHOO SETO Abstract. These are the notes to an expository talk I plan to give at MGSC on Kähler Geometry aimed for beginning graduate students in hopes to motivate

### Notes on Integrable Functions and the Riesz Representation Theorem Math 8445, Winter 06, Professor J. Segert. f(x) = f + (x) + f (x).

References: Notes on Integrable Functions and the Riesz Representation Theorem Math 8445, Winter 06, Professor J. Segert Evans, Partial Differential Equations, Appendix 3 Reed and Simon, Functional Analysis,

### 9.2 Support Vector Machines 159

9.2 Support Vector Machines 159 9.2.3 Kernel Methods We have all the tools together now to make an exciting step. Let us summarize our findings. We are interested in regularized estimation problems of

### Topological vectorspaces

(July 25, 2011) Topological vectorspaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ Natural non-fréchet spaces Topological vector spaces Quotients and linear maps More topological

### General Inner Product & Fourier Series

General Inner Products 1 General Inner Product & Fourier Series Advanced Topics in Linear Algebra, Spring 2014 Cameron Braithwaite 1 General Inner Product The inner product is an algebraic operation that

### LECTURE 5: THE METHOD OF STATIONARY PHASE

LECTURE 5: THE METHOD OF STATIONARY PHASE Some notions.. A crash course on Fourier transform For j =,, n, j = x j. D j = i j. For any multi-index α = (α,, α n ) N n. α = α + + α n. α! = α! α n!. x α =

### The Hilbert Space of Random Variables

The Hilbert Space of Random Variables Electrical Engineering 126 (UC Berkeley) Spring 2018 1 Outline Fix a probability space and consider the set H := {X : X is a real-valued random variable with E[X 2

### 1. General Vector Spaces

1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### ADJOINTS, ABSOLUTE VALUES AND POLAR DECOMPOSITIONS

J. OPERATOR THEORY 44(2000), 243 254 c Copyright by Theta, 2000 ADJOINTS, ABSOLUTE VALUES AND POLAR DECOMPOSITIONS DOUGLAS BRIDGES, FRED RICHMAN and PETER SCHUSTER Communicated by William B. Arveson Abstract.

### Karhunen-Loève decomposition of Gaussian measures on Banach spaces

Karhunen-Loève decomposition of Gaussian measures on Banach spaces Jean-Charles Croix jean-charles.croix@emse.fr Génie Mathématique et Industriel (GMI) First workshop on Gaussian processes at Saint-Etienne

### Linear Algebra and Dirac Notation, Pt. 2

Linear Algebra and Dirac Notation, Pt. 2 PHYS 500 - Southern Illinois University February 1, 2017 PHYS 500 - Southern Illinois University Linear Algebra and Dirac Notation, Pt. 2 February 1, 2017 1 / 14

### Vector Spaces, Affine Spaces, and Metric Spaces

Vector Spaces, Affine Spaces, and Metric Spaces 2 This chapter is only meant to give a short overview of the most important concepts in linear algebra, affine spaces, and metric spaces and is not intended

### Extension of positive definite functions

University of Iowa Iowa Research Online Theses and Dissertations Spring 2013 Extension of positive definite functions Robert Niedzialomski University of Iowa Copyright 2013 Robert Niedzialomski This dissertation

### Elements of Convex Optimization Theory

Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a self-contained overview of elements of convex optimization

### A Précis of Functional Analysis for Engineers DRAFT NOT FOR DISTRIBUTION. Jean-François Hiller and Klaus-Jürgen Bathe

A Précis of Functional Analysis for Engineers DRAFT NOT FOR DISTRIBUTION Jean-François Hiller and Klaus-Jürgen Bathe August 29, 22 1 Introduction The purpose of this précis is to review some classical

### where the bar indicates complex conjugation. Note that this implies that, from Property 2, x,αy = α x,y, x,y X, α C.

Lecture 4 Inner product spaces Of course, you are familiar with the idea of inner product spaces at least finite-dimensional ones. Let X be an abstract vector space with an inner product, denoted as,,

### Some Useful Background for Talk on the Fast Johnson-Lindenstrauss Transform

Some Useful Background for Talk on the Fast Johnson-Lindenstrauss Transform Nir Ailon May 22, 2007 This writeup includes very basic background material for the talk on the Fast Johnson Lindenstrauss Transform

### Karhunen-Loève decomposition of Gaussian measures on Banach spaces

Karhunen-Loève decomposition of Gaussian measures on Banach spaces Jean-Charles Croix GT APSSE - April 2017, the 13th joint work with Xavier Bay. 1 / 29 Sommaire 1 Preliminaries on Gaussian processes 2

### Continuity of convex functions in normed spaces

Continuity of convex functions in normed spaces In this chapter, we consider continuity properties of real-valued convex functions defined on open convex sets in normed spaces. Recall that every infinitedimensional

### Contents. 2 Sequences and Series Approximation by Rational Numbers Sequences Basics on Sequences...

Contents 1 Real Numbers: The Basics... 1 1.1 Notation... 1 1.2 Natural Numbers... 4 1.3 Integers... 5 1.4 Fractions and Rational Numbers... 10 1.4.1 Introduction... 10 1.4.2 Powers and Radicals of Rational

### 1 The Projection Theorem

Several Important Theorems by Francis J. Narcowich November, 14 1 The Projection Theorem Let H be a Hilbert space. When V is a finite dimensional subspace of H and f H, we can always find a unique p V

### Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )

Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

### Math 5520 Homework 2 Solutions

Math 552 Homework 2 Solutions March, 26. Consider the function fx) = 2x ) 3 if x, 3x ) 2 if < x 2. Determine for which k there holds f H k, 2). Find D α f for α k. Solution. We show that k = 2. The formulas

### FUNCTIONAL ANALYSIS: NOTES AND PROBLEMS

FUNCTIONAL ANALYSIS: NOTES AND PROBLEMS Abstract. These are the notes prepared for the course MTH 405 to be offered to graduate students at IIT Kanpur. Contents 1. Basic Inequalities 1 2. Normed Linear

### Convex Feasibility Problems

Laureate Prof. Jonathan Borwein with Matthew Tam http://carma.newcastle.edu.au/drmethods/paseky.html Spring School on Variational Analysis VI Paseky nad Jizerou, April 19 25, 2015 Last Revised: May 6,

### The Kernel Trick, Gram Matrices, and Feature Extraction. CS6787 Lecture 4 Fall 2017

The Kernel Trick, Gram Matrices, and Feature Extraction CS6787 Lecture 4 Fall 2017 Momentum for Principle Component Analysis CS6787 Lecture 3.1 Fall 2017 Principle Component Analysis Setting: find the

### Definition and basic properties of heat kernels I, An introduction

Definition and basic properties of heat kernels I, An introduction Zhiqin Lu, Department of Mathematics, UC Irvine, Irvine CA 92697 April 23, 2010 In this lecture, we will answer the following questions:

### Math 127: Course Summary

Math 27: Course Summary Rich Schwartz October 27, 2009 General Information: M27 is a course in functional analysis. Functional analysis deals with normed, infinite dimensional vector spaces. Usually, these

### GELFAND S THEOREM. Christopher McMurdie. Advisor: Arlo Caine. Spring 2004

GELFAND S THEOREM Christopher McMurdie Advisor: Arlo Caine Super Advisor: Dr Doug Pickrell Spring 004 This paper is a proof of the first part of Gelfand s Theorem, which states the following: Every compact,

A BRIEF INTRODUCTION TO HILBERT SPACE FRAME THEORY AND ITS APPLICATIONS AMS SHORT COURSE: JOINT MATHEMATICS MEETINGS SAN ANTONIO, 2015 PETER G. CASAZZA Abstract. This is a short introduction to Hilbert

### Optimization Theory. A Concise Introduction. Jiongmin Yong

October 11, 017 16:5 ws-book9x6 Book Title Optimization Theory 017-08-Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 017-08-Lecture Notes page Optimization

### Math 209B Homework 2

Math 29B Homework 2 Edward Burkard Note: All vector spaces are over the field F = R or C 4.6. Two Compactness Theorems. 4. Point Set Topology Exercise 6 The product of countably many sequentally compact

### Lecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set.

Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. Definition A set of k vectors {u, u 2,..., u k }, where each u i R n, is said to be an

### Five Mini-Courses on Analysis

Christopher Heil Five Mini-Courses on Analysis Metrics, Norms, Inner Products, and Topology Lebesgue Measure and Integral Operator Theory and Functional Analysis Borel and Radon Measures Topological Vector

### Quick Review of Matrix and Real Linear Algebra

Division of the Humanities and Social Sciences Quick Review of Matrix and Real Linear Algebra KC Border Subject to constant revision Last major revision: December 12, 2016 Contents 1 Scalar fields 1 2

### Linear Algebra Review

January 29, 2013 Table of contents Metrics Metric Given a space X, then d : X X R + 0 and z in X if: d(x, y) = 0 is equivalent to x = y d(x, y) = d(y, x) d(x, y) d(x, z) + d(z, y) is a metric is for all

### ICML - Kernels & RKHS Workshop. Distances and Kernels for Structured Objects

ICML - Kernels & RKHS Workshop Distances and Kernels for Structured Objects Marco Cuturi - Kyoto University Kernel & RKHS Workshop 1 Outline Distances and Positive Definite Kernels are crucial ingredients

### Short note on compact operators - Monday 24 th March, Sylvester Eriksson-Bique

Short note on compact operators - Monday 24 th March, 2014 Sylvester Eriksson-Bique 1 Introduction In this note I will give a short outline about the structure theory of compact operators. I restrict attention

### Inner products. Theorem (basic properties): Given vectors u, v, w in an inner product space V, and a scalar k, the following properties hold:

Inner products Definition: An inner product on a real vector space V is an operation (function) that assigns to each pair of vectors ( u, v) in V a scalar u, v satisfying the following axioms: 1. u, v

### ANALYSIS QUALIFYING EXAM FALL 2017: SOLUTIONS. 1 cos(nx) lim. n 2 x 2. g n (x) = 1 cos(nx) n 2 x 2. x 2.

ANALYSIS QUALIFYING EXAM FALL 27: SOLUTIONS Problem. Determine, with justification, the it cos(nx) n 2 x 2 dx. Solution. For an integer n >, define g n : (, ) R by Also define g : (, ) R by g(x) = g n

### Functional Analysis. James Emery. Edit: 8/7/15

Functional Analysis James Emery Edit: 8/7/15 Contents 0.1 Green s functions in Ordinary Differential Equations...... 2 0.2 Integral Equations........................ 2 0.2.1 Fredholm Equations...................

### Lecture Notes 1: Vector spaces

Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector

### Let R be the line parameterized by x. Let f be a complex function on R that is integrable. The Fourier transform ˆf = F f is. e ikx f(x) dx. (1.

Chapter 1 Fourier transforms 1.1 Introduction Let R be the line parameterized by x. Let f be a complex function on R that is integrable. The Fourier transform ˆf = F f is ˆf(k) = e ikx f(x) dx. (1.1) It

### The Calderon-Vaillancourt Theorem

The Calderon-Vaillancourt Theorem What follows is a completely self contained proof of the Calderon-Vaillancourt Theorem on the L 2 boundedness of pseudo-differential operators. 1 The result Definition

### Eigenvalues and Eigenfunctions of the Laplacian

The Waterloo Mathematics Review 23 Eigenvalues and Eigenfunctions of the Laplacian Mihai Nica University of Waterloo mcnica@uwaterloo.ca Abstract: The problem of determining the eigenvalues and eigenvectors

### General Notation. Exercises and Problems

Exercises and Problems The text contains both Exercises and Problems. The exercises are incorporated into the development of the theory in each section. Additional Problems appear at the end of most sections.

### 2 (Bonus). Let A X consist of points (x, y) such that either x or y is a rational number. Is A measurable? What is its Lebesgue measure?

MA 645-4A (Real Analysis), Dr. Chernov Homework assignment 1 (Due 9/5). Prove that every countable set A is measurable and µ(a) = 0. 2 (Bonus). Let A consist of points (x, y) such that either x or y is

### 1.2 Fundamental Theorems of Functional Analysis

1.2 Fundamental Theorems of Functional Analysis 15 Indeed, h = ω ψ ωdx is continuous compactly supported with R hdx = 0 R and thus it has a unique compactly supported primitive. Hence fφ dx = f(ω ψ ωdy)dx

### The Dual of a Hilbert Space

The Dual of a Hilbert Space Let V denote a real Hilbert space with inner product, u,v V. Let V denote the space of bounded linear functionals on V, equipped with the norm, F V sup F v : v V 1. Then for