Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces

Size: px

Start display at page:

Download "Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces"

Jeremy Morrison
6 years ago
Views:

1 MA 751 Part 3 Infinite Dimensional Vector Spaces 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces Microarray experiment: Question: Gene expression - when is the DNA in a gene 1 transcribed and thus expressed (as RNA) in a cell?

2 One solution: Measure RNA levels (result of transcription) Method: Microarray

3 "Probe": Single-stranded DNA with a defined identity tethered to a solid medium

4 "Target": DNA or RNA (labeled with a phosphorescent color to identify its presence) Two main types of Microarrays cdna arrays: Spotted onto surface oligonucleotide arrays: created on surface

6 population of cdna microarray chip

7 Resulting Microarrays:

8 From: "Data Analysis Tools for DNA microarrays" by S. Draghici, published by Chapman and Hall/CRC Press.

Affymetrix Agilent Amersham Mergen Heart

:Liver Heart:Liver Heart:Liver Heart:Liver

2595/9970 (26%) human liver: 318/9778 (3%)

9 Affymetrix Agilent Amersham Mergen Heart replicates Heart replicates Heart replicates Heart replicates Heart:Liver Heart:Liver Heart:Liver Heart:Liver Human liver vs. human heart: 3904/22,283 (18%) Human liver vs. human liver: 3875/22,283 (17%) Human heart vs. human heart: 4026/22,283 (18%) Human liver vs. human heart: 6963/14,159 (49%) Human liver vs. human liver: 5129/14,159 (36%) Human heart vs. human heart: 1204/18,006 (6%) Human liver vs. human heart: 2595/9970 (26%) Human liver vs. human liver: 318/9778 (3%) Human heart vs. human heart: 454/9772 (5%) Human liver vs. human heart: 8572/11,904 (72%) Human liver vs. human liver: 2811/11,904 (24%) Human heart vs. human heart: 3515/11,904 (30%)

10 Result: for each subject tissue sample =, obtain a feature vector: FÐ=Ñ œ x œ ÐB" ß á ß B$!ß!!! Ñ consisting of expression levels of 30,000 genes. Can we classify tissues this way? Goals: 1. differentiate two different but similar cancers. 2. Understand genetic pathways of cancer

11 Basic difficulties: few samples (e.g., ); high dimension (e.g., 5, ,000). Curse of dimensionality - too few samples and too many parameters (dimensions) to fit them. Tool: Support vector machine (SVM)

12 Procedure: look at feature space J in which FÐ=Ñ lives, and differentiate examples of one and the other cancer with a hyperplane: Methods needed: Machine learning; reproducing kernel Hilbert spaces (RKHS)

13 2. Statistical & machine Learning: More Motivation The role of learning theory has grown a great deal in: Mathematics Statistics Computational Biology Neurosciences, e.g., theory of plasticity, workings of visual cortex

14 Source: University of Washington

15 Computer science, e.g., vision theory, graphics, speech synthesis Source: T. Poggio/M

16 Face identification: MIT

17 People classification or detection: Poggio/MIT

18 We want the theory behind such learning algorithms- 3. The problem: Learning theory Given an unknown function 0ÐxÑ À Ä ß learn 0ÐxÑ. We have already seen the problem of representing a known function 0ÐxÑ using neural networks. Now we wish to find a way to determine an unknown 0Ðx Ñ from knowing its value at several inputs..

19 Example 1: x is retinal activation pattern (i.e., B3 œ activation level of retinal neuron 3), and Cœ0Ðx Ñ! if the retinal pattern is a chair; Cœ0Ðx Ñ! otherwise. [Thus: want concept of a chair] Given: examples of chairs (and non-chairs): x" ßx# ßáßx8, together with proper outputs C" ßáßC8Þ This is the information: R0 œ Ð0Ðx Ñßáß0Ðx ÑÑ " 8

20 Goal: Give best possible estimate of the unknown function 0, i.e., try to learn the concept 0 from the examples R0. But: given pointwise information about 0 not sufficient: which is the "right" 0ÐBÑ given the data R0 below?

22 (a)

23 (b) [How to decide?]

24 4. Infinite dimensional vector spaces: [This material is essentially the content of a course in real analysis; see me if you want more information on this.] Let L be a vector space with inner product. Recall by definition Recall also m@m œ length œ m@m #.

25 Define distance between two vectors " # to be m@ m. Consider an infinite collection of vectors W œö@ ß@ ß@ á Þ " # $ Define infinite linear combinations by: _ " -@ œa 3œ" 3 3 if 8 ma! - 3@ 3m 8Ä_ Ò!Þ 3œ"

26 [Definitions of linear independence and spanning for infinite collections of vectors are the same, except we allow infinite sums. Henceforth we always allow infinite linear combinations.] Def. 1: In an inner product vector space L, a Cauchy sequence is a sequence of " # for which their mutual distances go to 0 (so they behave as if converging to a limit), i.e., ll@ 7ll 8ß 7 Ò Ä _!.

27 Def 2: An inner product vector space L with an inner product is complete if any infinite sequence of 3 which is a Cauchy sequence (i.e., behaves convergently) actually converges to some limiting i.e., for ll@ 8Ä_ Ò! Def 3: An infinite collection of vectors F œö@ " ß@ # ßá spans L if every vector A L can be written as a (possibly infinite) linear combination of vectors in F.

28 The collection F is linearly independent if no vector in F is a linear combination of the others. The collection F is a basis for L if it spans L and is linearly indepenent. [I.E., the notions of span, linear independence, and basis work the same way as before, except we now allow infinite linear combinations] Ex: Not all inner product spaces are Hilbert spaces since not all are complete. As an example, consider the space T œ Öall polynomials on Ò!ß "Ó. Define " inner product Ð0ß 1Ñ œ '! 0ÐBÑ1ÐBÑ.BÞ

29 # ' " #! Then resulting norm is ll0ðbñll œ Ø0ß 0Ù œ 0 ÐBÑ.BÞ This space is not complete: consider the R defined by the partial sums of the Taylor series for B / R R 8 B œ " = a polynomial. 8x 8œ!

30 Note that if R Q then R 8 Q 8 R 8 R 8 B B B B ll@ Qll œ ¾" " ¾ œ ¾ " ¾ Ÿ " ½ 8x 8x 8x 8 But: 8 8œ! 8œ! 8œQ " 8œQ " B " 8 " #8 " " m m œ mb m œ B.B œ Þ 8x 8x 8x ( Œ 8x #8 "! " "Î# "Î#

31 _ So easy to show! m that 8=0 8 B 8x m _. Thus it easily follows ll@ Q ll Ò!ß RßQ Ä _ so that the is a Cauchy sequence in L. R

32 But note that by Taylor series R 8 B R ÐBÑ / œ " B / Ä! 8x RÄ_ 8œ! uniformly on [0,1]. Thus easy to show that B m@ R ÐBÑ / m Ò!Þ RÄ_

33 So: R ÐBÑ Ä / Þ But: can show that a sequence of functions can't converge to 2 different functions. Thus there is no polynomial :ÐBÑ (i.e. something in our space T ) such ÐBÑ Ä :ÐBÑÞ R R do not converge to something in T and thus T is not complete! [Moral: intuitively, complete space is one where any convergent sequence T8 converges to an element T of the original space.]

34 Theorem 4: If F œö@ " ß@ # ß@ $ ßá is a collection of vectors which is orthonormal (i.e, unit lengths and inner product!), then it is automatically linearly independent. If F is a basis for L and is orthonormal, it is called an orthonormal basis.

35 $ Ex 2: Lœ " $ 3 is a Hilbert space (i.e., not hard to show that it's complete). Inner product is the usual one for vectors: This L is a Hilbert space. Orthonormal " " # # $ $ Ô " Ô! Ô! e" œ! à e# œ " à e$ œ! Þ Õ! Ø Õ! Ø Õ" Ø

36 Ex 3: #! " $ # 3 Lœ œ polynomials on Ò!ß "Ó œ Ö+ + B + B À + forms a Hilbert space. Inner product: " # ' "! " # Ð: ÐBÑß : ÐBÑÑ œ : ÐBÑ: ÐBÑ.BÞ Note this is complete since an infinite sum of second order polynomials which converges, converges to another second order polynomial (this can be proved). Thus L is a Hilbert space.

37 Ex 4: Let Hilbert _ L œ œö@œð@ " ß@ # ß@ $ ßáÑl@ 3 space, if we define the inner product Ð@ß AÑ A á œ "@ A _ " " # # 3 3 3œ" is a Length of a is m@m œ 3@ 3 œ 3 # Þ 3œ" 3œ"

38 Thus to have well-defined lengths we add the condition that _ to the definition of L. Then can show that L satisfies all the properties of a Hilbert space (in particular it's complete).

39 Can show that the set @ ã " # $ œ Ð"ß!ß!ßáÑ œ Ð!ß"ß!ßáÑ œð!ß!ß"ßáñ is certainly orthonormal, and it spans orthonormal basis for L. L, so it is an

Gene expression experiments. Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces

Gene expression experiments. Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces MA 751 Part 3 Gene expression experiments Infinite Dimensional Vector Spaces 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces Gene expression experiments Question: Gene