About the primer To review concepts in functional analysis that Goal briey used throughout the course. The following be will concepts will be describe

Transcription

1 Camp 1: Functional analysis Math Mukherjee + Alessandro Verri Sayan

2 About the primer To review concepts in functional analysis that Goal briey used throughout the course. The following be will concepts will be described 1. Function spaces 2. Metric spaces 3. Convergence 4. Measure Dense subsets 5. denitions and concepts come primarily from \Introductory Real The Analysis" by Kolmogorov and Fomin (highly recommended).

3 6. Separable spaces 7. Complete metric spaces 8. Compact metric spaces 9. Linear spaces 10. Linear functionals 11. Norms and semi-norms of linear spaces 12. Convergence revisited 13. Euclidean spaces 14. Orthogonality and bases 15. Hilbert spaces

4 16. Delta functions 17. Fourier transform 18. Functional derivatives 19. Expectations 20. Law of large numbers

5 in the space can be thought of as a point. Examples: function C[a, b], the set of all real-valued continuous functions 1. the interval [a, b] in L1[a, b], the set of all real-valued functions whose absolute 2. value is integrable in the interval [a, b] L2[a, b], the set of all real-valued functions square integrable 3. in the interval [a, b] that the functions in 2 and 3 are not necessarily Note continuous! Function space A function space is a space made of functions. Each

6 Metric space a metric space is meant a pair (X, ρ) consisting of a By X and a distance ρ, a single-valued, nonnegative, space function ρ(x, y) dened for all x, y X which has the real three properties: following 1. ρ(x, y) = 0 i x = y 2. ρ(x, y) = ρ(y, x) 3. Triangle inequality: ρ(x, z) ρ(x, y) + ρ(y, z)

7 Examples The set of all real numbers with distance 1. y) = ρ(x, x y is the metric space IR 1. The set of all ordered n-tuples 2. = x (x 1,..., x n ) real numbers with distance of ρ(x, y) = n (x i y i ) 2 i=1 is the metric space IR n.

8 The set of all functions satisfying the criteria 3. f (x)dx < 2 distance with ρ(f (x)) = 1 (x),f (x) 2 (f 1 (x)) f 2 2 dx is the metric space L 2 (IR). The set of all probability densities with Kullback-Leibler 4. divergence ρ(p 1 (x),p 2 (x)) = p ln 1(x) p p 1(x)dx (x) 2 not a metric space. The divergence is not symmetric is ρ(p 1 (x),p (x)) 2 ρ(p 2 (x),p 1 (x)).

9 Convergence open/closed sphere in a metric space S is the set of An x IR for which points ρ(x,x) 0 <r ρ(x,x) 0 r open closed. open sphere of radius ɛ with center x 0 will be called an An of x 0, denoted O ɛ-neighborhood ɛ (x ). 0 sequence of points {x A n = } x 1,x 2,..., x n, in a metric... S converges to a point x S if every neighborhood space O ɛ of x contains all points x (x) n from a certain starting Given any ɛ>0 there is an integer N integer. ɛ that such O ɛ contains all points x (x) n n>n with ɛ {x. n converges } to x i n ρ(x, x n) = 0. lim

10 Measure the course we will see integrals of the form Throughout V (f(x),y)dν(x) V (f(x),y) p(x)dx ν(x) is the measure. concept of the measure ν(e) of a set E is a natural The of the concept extension The length l() of a line segment 1) The volume V (G) of a space G 2) The integral of a nonnegative function of a region in 3) space.

11 Lebesgue measure f be a ν-measurable function (it has nite measure) Let no more than countably many distinct values taking y 1,y 2,..., y n,... by the Lebesgue integral of f over the set A denoted Then dν, f(x) we mean the quantity A n y n ν(a n ) where A n = {x : x A, f(x) = y n }, the series is absolutely convergent. The measure provided is the Lebesgue measure. ν

12 Lebesgue integral We can compute the integral f(x)dx 2 by adding up the area under the red rectangles f(x) x

13 more tradition form of the integral is the Riemann The The intuition is that of limit of an innite sum of integral. otherwise 0 not exist in the Riemann sense. does Riemann integral innitesimally small rectangles, A f(x)dx = n f(x n )x. in the Riemann sense require continuous or piecewise Integrals continuous functions, the Lebesgue from shown pre- relaxes Thus, the integral viously f(x)dx this. 1 0 with f : [0, 1] IR dened as f = 1 if t is rational

14 Lebesgue-Stieltjes integral F be a nondecreasing function dened on a closed Let [a, b] and suppose F is continuous from the left at interval every point [a, b). F is called the generating function of the Lebesgue-Stieltjes measure ν F. Lebesgue-Stieltjes integral of a function f is denoted The by b a f(x) df(x) is the Lebesgue integral which [a,b] f(x) dν F. example of dν An F a probability density p(x)dx. Then ν is F correspond to the cumulative distribution function. would

15 Dense A and B be subspaces of a metric space IR. A is said Let be dense in B if to A B. A the closure of the subset is In particular A is said to be everywhere dense in IR if A. = R. A point x IR is called a contact point of a set A IR if A neighborhood of x contains at least on point of A. every set of all contact points of a set A denoted by The A is the closure of A. called

16 The set of all polynomials with rational coecients is 2. in C[a, b]. dense Let K be a positive denite Radial Basis Function then 3. functions the A hypothesis space that is dense in L 2 is a desired Note: of any approximation scheme. property Examples 1. The set of all rational points is dense in the real line. f(x) = n c i K(x x i ) i=1 is dense in L 2.

17 Separable metric space is said to be separable if it has a countable A dense subset. everywhere Examples: n spaces IR 1, IR, L 2 [a, b], and C[a, b] are all separable. 1. The The set of real numbers is separable since the set of 2. numbers is a countable subset of the reals and rational the set of rationals is is everywhere dense.

18 Completeness A sequence of functions f n is fundamental if ɛ >0 N ɛ such that n and m>n ɛ, ρ(f n,f m ) <ɛ. metric space is complete if all fundamental sequences A to a point in the space. converge L 1, and L 2 are complete. That C2 is not complete, C, can be seen through a counterexample. instead,

19 Incompleteness of C 2 { if 1 1 t<0 if 0 t 1 1 φ n (t) = if 1 t< 1/n 1 if nt 1/n t<1/n Consider the sequence of functions (n = 1, 2,...) 1 if 1/n t 1 and assume that φ n converges to a continuous function φ in the metric of C2. Let f(t) =

20 Incompleteness of C 2 (cont.) Clearly, ( (f(t) φ(t)) 2 dt) 1/2 ( ( (φ n (t) φ(t)) 2 dt) 1/2. (f(t) φ n (t)) 2 dt) 1/2 + the term is strictly positive, because f(t) is not Now l.h.s. for n we have continuous, while (f(t) φ n dt 0. 2 (t)) contrary to what assumed, φ Therefore, n converge cannot φ in the metric of C2. to

21 The space of real numbers is the completion of the 1. of rational numbers. space Completion of a metric space a space IR with closure IR, a complete metric Given metric is called a completion of IR if IR IR and IR space = IR. IR Examples Let K be a positive denite Radial Basis Function then 2. L is the completion the space of functions 2 f(x) = n c i K(x x i ). i=1

22 Compact spaces metric space is compact i it is totally bounded and A complete. IR be a metric space and ɛ any positive number. Then Let set A IR is said to be an ɛ-net for a set M IR if for a x M, there is at least one point a A such that every a) <ɛ. ρ(x, a metric space IR and a subset M IR suppose M Given a nite ɛ-net for every ɛ>0. Then M is said to be has totally bounded. A compact space has a nite ɛ-net for all ɛ>0.

23 contained in some hypercube Q. We can partition is hypercube into smaller hypercubes with sides of this This is not true for innite-dimensional spaces. The 2. sphere in l 2 with constraint unit Examples n Euclidean n-space, IR, total boundedness is equivalent 1. In boundedness. If M IR is bounded then M to ɛ. The vertices of the little cubes from a nite length of Q. nɛ/2-net x 2 n = 1, n=1 points e 1 = (1, 0, 0,...), e 2 = (0, 1, 0, 0,...),..., is bounded but not totally bounded. Consider the

24 the n-th coordinate of e where n one and all others are is These points lie on but the distance between zero. is 2. So cannot have a nite ɛ-net with any two ɛ< 2/2. Innite-dimensional spaces maybe totally bounded. Let 3. be the set of points x = (x 1,..., x n in l 2 satisfying,..) inequalities the x 1 < 1, x 2 <,..., x n < n 1,... set called the Hilbert cube is an example of The innite-dimensional totally bounded set. Given any an ɛ>0, choose n such that < ɛ, 1 n+1 2 2

25 with each point and = x (x 1,..., x n,..) associate the point is x (x 1,..., x = n, 0, 0, (1)...). Then ρ(x, x ) = k=n+1 x 2 k < k=n 1 k < < ɛ. 1 n of all points in that satisfy (1) is totally The set it is a bounded set in n-space. since bounded The RKHS induced by a kernel K with an innite number 4. of positive eigenvalues that decay exponentially is compact. In this case, our vector x = (x 1,..., x n,..) can

26 written in terms of its basis functions, the eigenvectors be of K. Now for the RKHS norm to be bounded x 1 <µ 1, x 2 <µ 2,..., x n <µ n,... we know that µ and n O(n α ). So we have the case = to the Hilbert cube and we can introduce a analogous point x = (x 1,..., x n, 0, 0,...) (2) a bounded n-space which can be made arbitrarily in to x. close

27 family of functions φ dened on a closed interval [a, b] A said to be uniformly bounded if for K>0 is theorem: A necessary and sucient condition for Arzela's family of continuous functions dened on a closed a [a, b] to be (relatively) compact in C[a, b] is that interval uniformly bounded and equicontinuous. is Compactness and continuity φ(x) <K for all x [a, b] and all φ. family of functions φ is equicontinuous of for any given A there exists δ>0 such that x y <δ implies ɛ>0 φ(x) φ(y) <ɛ for all x, y [a, b] and all φ.

28 Linear space set L of elements x, y, z,... is a linear space if the following A three axioms are satised: Any two elements x, y L uniquely determine a third 1. in x + y L called the sum of x and y such element that x + y = y + x (commutativity) (a) (x + y) + z = x + (y + z) (associativity) (b) An element 0 L exists for which x + 0 = x for all (c) x L For every x L there exists an element x L (d) the property x + ( x) = 0 with

29 Any number α and any element x L uniquely determine 2. an element αx L called the product such that α(βx) = β(αx) (a) 1x = x (b) 3. Addition and multiplication follow two distributive laws (a)(α + β)x = αx + βx (b)α(x + y) = αx + αy

30 Linear functional functional, F, is a function that maps another function A a real-value to F : f IR. linear functional dened on a linear space L, satises the A two properties following 1. Additive: F(f + g) = F(f) + F(g) for all f, g L 2. Homogeneous: F(αf) = αf(f)

31 Examples n IR be a real n-space with elements x = (x 1,..., x 1. Let n ), = a (a 1,..., a n and n a xed element in IR. Then be ) F(x) = n a i x i i=1 is a linear functional b a f(x)p(x)dx 2. The integral F[f(x)] = is a linear functional 3. Evaluation functional: another linear functional is the

32 delta function Dirac δ t = f(t). [f( )] can be written Which δ t = [f( )] b a f(x)δ(x t)dx. Evaluation functional: a positive denite kernel in a 4. RKHS F t [f( )] = (K t,f) = f(t). This is simply the reproducing property of the RKHS.

33 Normed space normed space is a linear (vector) space N in which a A is dened. A nonnegative function is a norm i norm f, g N and α IR 1. f 0 and f = 0 i f = 0 2. f + g f + g 3. αf = α f. if all conditions are satised except f = 0 i f = 0 Note, the space has a seminorm instead of a norm. then

34 Measuring distances in a normed space a normed space N, the distance ρ between f and g, or In metric, can be dened as a ρ(f, g) = g f. Note that f, g, h N 1. ρ(f, g) = 0 i f = g. 2. ρ(f, g) = ρ(g, f). 3. ρ(f, h) ρ(f, g) + ρ(g, h).

35 Example: continuous functions norm in C[a, b] can be established by dening A = max f f(t). a t b distance between two functions is then measured as The g) = max ρ(f, g(t) f(t). a t b With this metric, C[a, b] is denoted as C.

36 Examples (cont.) f = b a f(t) dt. A norm in L 1 [a, b] can be established by dening ρ(f, g) = b a g(t) f(t) dt. The distance between two functions is then measured as With this metric, L 1 [a, b] is denoted as L 1.

37 this metric, C 2 [a, b] and L 2 [a, b] are denoted as C 2 With L 2 respectively. and Examples (cont.) f = ( b a f2 (t)dt ) 1/2. A norm in C 2 [a, b] and L 2 [a, b] can be established by dening ρ(f, g) = ( b a (g(t) f(t))2 dt ) 1/2. The distance between two functions now becomes

38 sequence of functions f A n to a function f almost converge i everywhere sequence of functions f A n to a function f uniformly converge i Convergence revisited f n(x) = f(x) lim n + A sequence of functions f n converge to a function f in measure i ɛ >0 µ{x f lim n(x) f(x) ɛ} = 0. n + : lim n + x (f n(x) f(x)) = 0 sup

39 between dierent types of Relationship convergence the case of bounded intervals: uniform convergence (C) In implies convergence in the quadratic mean (L ) which implies 2 in the mean (L 1 ) which implies conver- convergence gence in measure almost everywhere convergence which implies convergence in measure.

40 between dierent types of Relationship convergence uniform convergence implies all other type of convergence That is clear. L 2 over a bounded interval of width A. Keeping Consider mind that the function g = 1 belongs to L 2 and that in g L 2 have f L = 1 A f dx = A f 1dx f L 2 1 L2 = A f L2 = A, convergence in the quadratic mean implies convergence in the mean because for every function f L 2 we and hence that f L 1.

41 convergence implies convergence in Any measure in measure is obtained by convergence in the Convergence through Chebyshev's inequality: mean any real random variable X and t>0, For t) P( X E[X /t 2 ]. 2 proof that almost everywhere convergence implies The in measure is somewhat more complicated. convergence

42 everywhere convergence does not Almost convergence in the (quadratic) mean imply the [0, let f Over interval 1] n be { n x (0, 1/n] f n = 0 otherwise f Clearly n 0 for all x 1]. Note that each [0, f n is a continuous function and that the convergence is not not (the closer the x to 0, the larger n must be for uniform f n = 0). However, (x) 1 f n(x) dx = 1 for all n, 0 both the Riemann or the Lebesgue sense. in

43 in the quadratic mean does Convergence imply convergence at all! not f n i = { i 1 1 n <x i n Over the interval (0, 1], for every n = 1, 2,..., and i = 1,..., n let 0 otherwise the Clearly sequence 1 f,f1,f 2,..., f n 1,f n 2,...fn 1,f n n n,f n+1 2 1,..., to 0 both in measure and in the quadratic mean. converges the same sequence does not converge for any x! However,

44 almost surely implies convergence in probability. Convergence sequence X 1 A,...X n law of large numbers if for i almost c to satises the strong some constant c, 1 ni=1 n X converges The satises weak law of large numbers i for i in c to surely. sequence the some constant c, 1 ni=1 n X converges in probability and almost Convergence surely event with probability 1 is said to happen almost Any A sequence of real random variables Y surely. n converges almost surely to a random variable Y i P(Y n Y ) = 1. sequence Y A n in probability to Y i for every converges lim ɛ>0, n P( Y n Y >ɛ) = 0. probability.

45 Euclidean space is a linear (vector) space E in which a A product is dened. A real valued function (, ) is a dot dot Euclidean space becomes a normed linear space when A with the norm equipped Euclidean space product i f, g,h E and α IR 1. (f, g) = (g, f) 2. (f + g, h) = (f, h ) + (g, h) and (αf, g) = α(f, g) 3. (f, f) 0 and (f, f) = 0 i f = 0. f = (f, f).

46 set of nonzero vectors {x A α in a Euclidean space } is E to be an orthogonal system if said Orthogonal systems and bases (x α,x β ) = 0 for α = β an orthonormal system if and (x α,x β = 0 for α = β ) (x α,x β ) = 1 for α = β. orthogonal system {x An α is called an orthogonal basis } it is complete (the smallest closed subspace containing if {x α } is the whole space E). A complete orthonormal system is called an orthonormal basis.

47 Examples n is a real n-space, the set of n-tuples x = (x 1,..., x 1. IR n ), y (y 1,..., y = n we dene the dot product as If ). (x, y) = n x i y i i=1 we get Euclidean n-space. The corresponding norms and distances in IR n are x = ρ(x, y) = x y = n i=1 n x 2 i (x i y i ) 2. i=1

48 an innite-dimensional Euclidean space when becomes with the dot product equipped vectors The e = 1 (1, 0, 0, 0)..., e = 2 (0, 1, 0, 0)..., e n = (0, 0, 0,..., 1) form an orthonormal basis in IR n. The space l 2 with elements x = (x 1,x 2,..., x 2. n...),, = y (y 1,y 2,..., y n...),..., where, x 2 i <, y 2 i <,...,..., i=1 i=1 (x, y) = x i y i. i=1

49 simplest orthonormal basis in l 2 consists of vectors The e = 1 (1, 0, 0,...) 0, e = 2 (0, 1, 0,...) 0, e = 3 (0, 0, 1,...) 0, e = 4 (0, 0, 0,...) 1, there are an innite number of these bases. The space C 2 [a, b] consisting of all continuous functions 3. [a, b] equipped with the dot product on (f, g) = b a f(t)g(t)dt is another example of Euclidean space.

50 important example of orthogonal bases in this space An the following set of functions is cos 1, 2πnt 2πnt, sin b a b a (n = 1, 2,...).

51 Hilbert space is a Euclidean space that is complete, A and generally innite-dimensional. separable, H is separable (contains a countable everywhere dense 3. subset) Hilbert space A Hilbert space is a set H of elements f, g,... for which 1. H is a Euclidean space equipped with a scalar product 2. H is complete with respect to metric ρ(f, g) = f g 4. (generally) H is innite-dimensional. l 2 and L 2 are examples of Hilbert spaces.

52 The δ function now consider the functional which returns the value of We f at the location C (an evaluation functional), t [f] = f(t). that this functional is degenerate because it does not Note on the entire function f, but only on the value of depend f at the specic location t. The δ(t) is not a functional but a distribution.

53 The δ function (cont.) The same functional can be written as [f] = f(t) = f(s)δ(s t)ds. ordinary function exists (in L 2 ) that behaves like δ(t), No can think of δ(t) as a function that vanishes for t = 0 one takes innite value at t = 0 in such a way that and δ(t)dt = 1.

54 The δ function (cont.) δ function can be seen as the limit of a sequence of The functions. For example, if ordinary r ɛ (t) = 1 ɛ (U(t) U(t ɛ)) is a rectangular pulse of unit area, consider the limit lim ɛ 0 f(s)r ɛ (s t)ds. 1 ɛ t+ɛ t f(s)ds = f(t) By denition of r ɛ this gives lim ɛ 0 because f is continuous.

55 Fourier Transform Fourier Transform of a real valued function f L1 is The complex valued function ~ f(ω) dened as the F[f(x)] = ~ f(ω) = + f(x) e jωx dx. FT ~ The f be thought of as a representation of the can content of f(x). The original function f can information = 1 f(x) 2π + ~f(ω) e jωx dω. be obtained through the inverse Fourier Transform as

56 Properties 1 f(at) a F f (t) F(t) F (ω) ( ) ω 2πf( ω) f(t t ) 0 f(t)e jω 0t F(ω ω ) 0 a F(ω)e jt 0ω d n f(t) dt n (jω) n F(ω) ( jt) n f(t) dn F(ω) dω n f 1(τ)f 2 (t τ)dτ F 1 (ω)f 2 (ω) f (τ)f(t + τ)dτ F(ω) 2

57 Properties The box and the sinc = 1 if a t a and 0 otherwise f(t) = 2 sin(aω) F(ω). ω

58 Properties The Gaussian f(t) = e at2 F(ω) = π a e ω2 /4a

59 Properties Laplacian and Cauchy distributions The = e a t f(t) F(ω) = 2a a 2 + ω

60 due care, the Fourier Transform can be dened in With distribution sense. For example, we have the Fourier Transform in the distribution sense δ(x) 1 cos(ω0x) π(δ(ω ω0) + δ(ω + ω0)) sin(ω0x) jπ(δ(ω + ω0) δ(ω ω0)) U(x) πδ(ω) j/ω x 2/ω 2

61 Parseval's formula f is also square integrable, the Fourier Transform leaves If norm of f unchanged. Parseval's formula states that the + f(x) dx = 1 2π 2 + ~ f(ω) 2 dω.

62 following are Fourier transforms of some functions and The distributions Transforms of functions and Fourier distributions f(x) = δ(x) ~ f(ω) = 1 f(x) = cos(ω0x) ~ f(ω) = π(δ(ω ω 0) + δ(ω + ω0)) f(x) = sin(ω0x) ~ f(ω) = iπ(δ(ω + ω 0) δ(ω ω0)) f(x) = U(x) ~ f(ω) = πδ(ω) i/ω f(x) = x ~ f(ω) = 2/ω 2.

63 Functional dierentiation analogy with standard calculus, the minimum of a functional In can be obtained by setting equal to zero the derivative of the functional. If the functional depends on the of the unknown function, a further step is required derivatives (as the unknown function has to be found as the solution of a dierential equation).

64 Functional dierentiation derivative a functional [f] is dened The of = lim [f(t) D[f] + hδ(t s)] [f(t)]. h h 0 Df(s) the derivative depends on the location s. For Note that [f] = + f(t)g(t)dt if example, + = D[f] Df(s) g(t)δ(t s)dt = g(s).

65 Intuition f : [a, b] IR, a = x1 and b = x Let N The intuition behind. denition is that the functional [f] can be thought this of as the limit for N of the function of N variables N = N (f1,f2,..., f N ) with f1 = f(x1), f2 = f(x2),... f N = f(x N ). For N, depends on the entire function f. The on the location brought in by the δ function dependence to the partial derivative with respect to the corresponds variable f k.

66 Functional dierentiation (cont.) If [f] = f(t), the derivative is simply D[f] = Df(t) Df(s) Df(s) = δ(t s). to ordinary calculus, the minimum of a functional Similarly is obtained as the function solution to the equation [f] 0. = D[f] Df(s)

67 Random variables are given a random variable ξ F. To dene a random We you need three things: variable a set to draw the values from, we'll call this 1) a σ-algebra of subsets of, we'll call this B 2) 3) a probability measure F on B with F() = 1 (, B,F) is a probability space and a random variable So a masurable function X : IR. is

68 Expectations IEξ ξdf. Given a random variable ξ F the expectation is the of the random variable σ 2 (ξ) is Similarly variance IE(ξ IEξ). 2 var(ξ)

69 Law of large numbers lim l 1 l l I [f(xi ) y i ] IE x,yi [f(x) y]. The law of large numbers tells us: i=1 lσ Central Limit Theorem states: If the 1 l( l I IEI) N(0, 1), vari which implies 1 l I IEI k l. lσ Central Limit Theorem implies If c the I IEI k l. 1 l