Gene expression experiments. Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces

Size: px

Start display at page:

Download "Gene expression experiments. Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces"

Estella Mosley
6 years ago
Views:

1 MA 751 Part 3 Gene expression experiments Infinite Dimensional Vector Spaces 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces Gene expression experiments Question: Gene expression - when is the DNA in a gene 1 transcribed and thus expressed (as RNA) in a cell?

2 Gene expression experiments One solution: Measure RNA levels (result of transcription) Method: Microarray or RNA Seq array Result: for each subject tissue sample =, obtain a feature vector: FÐ=Ñ œ x œ ÐB" ß á ß B25ß!!! Ñ consisting of expression levels of 25,000 genes. Can we classify tissues this way?

3 Goals: Gene expression experiments 1. Differentiate two different but similar cancers. 2. Understand genetic pathways of cancer Basic difficulties: few samples (e.g., ); high dimension (e.g., 5, ,000). Curse of dimensionality - too few samples and too many parameters (dimensions) to fit them. Tool: Support vector machine (SVM)

4 Gene expression experiments Procedure: look at feature space J in which FÐ=Ñ lives, and differentiate examples of one and the other cancer with a hyperplane:

5 Gene expression experiments Methods needed for full analysis (of SVM and other high dimensional methods): Reproducing kernel Hilbert spaces (RKHS)

6 2. Machine Learning: The role of learning theory The role of learning theory has grown a great deal in: Mathematics Statistics Computational Biology Neurosciences, e.g., theory of plasticity, workings of visual cortex

7 Source: University of Washington

8 Kernel methods Kernel methods are used widely in: Computer science, e.g., vision theory, graphics, speech synthesis

9 Kernel methods Source: T. Poggio/M

10 Kernel methods Face identification: MIT

11 Kernel methods People classification or detection: Poggio/MIT

12 Learning theory We want the theory behind such learning algorithms- 3. The learning theory problem Given an unknown function 0ÐxÑ À Ä ß learn 0ÐxÑ from a few examples, i.e., a few inputs x where 0ÐxÑ is known. Determine unknown points x. 0ÐxÑ. from knowing its value at several

13 Learning theory Example 1: x is retinal activation pattern (i.e., B 3 œ activation level of retinal neuron 3), and Cœ0ÐxÑ! if the retinal pattern is a chair; Cœ0ÐxÑ! otherwise. [Thus: want concept of a chair] Given: examples of chairs (and non-chairs): x" ßx# ßáßx8, together with proper outputs CßáßCÞ " 8 This is the information: R0 œ Ð0Ðx Ñßáß0Ðx ÑÑ " 8

14 Learning theory Goal: Give best possible estimate of the unknown function 0, i.e., try to learn the concept 0 from the examples R0. But: given pointwise information about 0 not sufficient: which is the "right" 0ÐBÑ given the data R0 below?

15 Learning theory

16 (a) Learning theory

17 (b) Learning theory [How to decide?]

18 Infinite dimensional spaces 4. Infinite dimensional vector spaces: [This material is short course in real/functional analysis; see me if you want more sources] [Notation: in infinite dimensions generally remove boldface from vectors] Let L be a vector space with inner product. Recall by definition œ m@m #.

19 Infinite dimensional spaces Recall œ œ Distance between Consider infinite collection " # " # W œö@ ß@ ß@ á LÞ " # $

20 Infinite dimensional spaces Define infinite linear combinations by: if œa 3œ" ½A -@ ½!Þ 3œ" 3 3 8Ä Ò [Definitions of span, linear independence, basis same except for infinite sums. Henceforth always allow infinite linear combinations.]

21 Infinite dimensional spaces Def. 4. All previous linear algebra definitions (e.g. spanning, linear independence, basis) extend directly to the case of infinite numbers of vectors. Example: A collection Ö@ " ß@ # ßá of vectors spans a vector space Z if every Z can be written as a (possibly infinite) linear œ-@ -@ á œ -@Þ " " # # 3 3 3œ" [Henceforth always allow infinite linear combinations.]

22 Hilbert spaces Def 5: An inner product space L is complete if any sequence ÖB L which is Cauchy, i.e., mb B m Ä! (that is, 3 3œ" 3 4 3ß4Ä it should converge) actually converges to some B L, i.e. B3 Ä BÞ [Thus if the sequence bunches up, there is something for it to converge to.] Such an inner product space Hilbert space. L that is complete is called a

23 Example of incomplete space Ex: Not all inner product spaces are Hilbert spaces since not all are complete. As an example, consider the space T œ Öall polynomials on Ò!ß "Ó. Define inner product " Ð0ß 1Ñ œ '! 0ÐBÑ1ÐBÑ.BÞ # ' " #! Then resulting norm is ll0ðbñll œ Ø0ß 0Ù œ 0 ÐBÑ.BÞ This space is not complete: consider the vectors B by the partial sums of the Taylor series for / R defined

24 Example of incomplete R R 8 œ 8œ! B 8x = a polynomial.

25 Example of incomplete space Note that if R Qthen R 8 Q 8 R 8 R 8 B B B B ll@ Qll œ ¾ ¾ œ ¾ ¾ Ÿ ½ ½ 8x 8x 8x 8x But: 8 8œ! 8œ! 8œQ " 8œQ " B " 8 " #8 " " m m œ mb m œ B.B œ Þ 8x 8x 8x ( Œ 8x #8 "! " "Î# "Î#

26 So easy to show m Example of incomplete space 8=0 8 B 8x m. Thus it easily follows that ll@ Q ll Ò!ß RßQ Ä so that the is a Cauchy sequence in L. R

27 Example of incomplete space But note that by Taylor series R 8 B R ÐBÑ / œ B / Ä! 8x RÄ 8œ! uniformly on [0,1]. Thus easy to show that B m@ R ÐBÑ / m Ò!Þ RÄ

28 So: Example of incomplete space R ÐBÑ Ä / Þ But: can show that a sequence of functions can't converge to 2 different functions. Thus there is no polynomial :ÐBÑ (i.e. something in our space T ) such ÐBÑ Ä :ÐBÑÞ R R do not converge to something in T and thus T is not complete! [Moral: intuitively, complete space is one where any convergent sequence T8 converges to an element T of the original space.]

29 Theorem 4: If F œö@ " ß@ # ß@ $ ßá is a collection of vectors that is orthonormal (i.e, unit lengths and inner product!), then it is automatically linearly independent. If F is a basis for L and is orthonormal, it is called an orthonormal basis.

30 $ Examples of Hilbert spaces Ex 2: Lœ " $ 3 is a Hilbert space (i.e., not hard to show that it's complete). Inner product is the usual one for vectors: This L is a Hilbert space. Orthonormal " " # # $ $ Ô " Ô! Ô! e" œ! à e# œ " à e$ œ! Þ Õ! Ø Õ! Ø Õ" Ø

31 Examples of Hilbert spaces Ex 3: L œ # œ second order polynomials on Ò!ß "Ó œ Ö+ + B + B À + #! " $ 3 forms a Hilbert space. Inner product: " # ' "! " # Ð: ÐBÑß : ÐBÑÑ œ : ÐBÑ: ÐBÑ.BÞ

32 Examples of Hilbert spaces Note it is not hard to show that L is complete (in fact any finite dimensional vector space is complete). Thus L is a Hilbert space. Ex 4: Note L œ œö@œð@ " ß@ # ß@ $ ßáÑl@ 3 Hilbert space, if we define the inner product Ð@ß AÑ A á A " " # # 3 3 3œ" is a

33 Examples of Hilbert spaces Length of a is m@m œ 3@ 3 œ 3 # Þ 3œ" 3œ" Thus to have well-defined lengths we add the condition that m@m to the definition of L. Then can show that L satisfies all the properties of a Hilbert space (in particular it's complete).

34 Examples of Hilbert spaces Can show that the set @ " # $ œ Ð"ß!ß!ßáÑ œ Ð!ß"ß!ßáÑ œð!ß!ß"ßáñ ã is certainly orthonormal, and it spans orthonormal basis for L. L, so it is an

Infinite Dimensional Vector Spaces. 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces

MA 751 Part 3 Infinite Dimensional Vector Spaces 1. Motivation: Statistical machine learning and reproducing kernel Hilbert Spaces Microarray experiment: Question: Gene expression - when is the DNA in