DD Advanced Machine Learning
|
|
|
- Dwayne Curtis
- 5 years ago
- Views:
Transcription
1 Modelling Carl Henrik Royal Institute of Technology November 4, 2015
2 Who do I think you are? Mathematically competent linear algebra multivariate calculus Ok programmers Able to extend knowledge beyond lectures Motivated and willing to learn Not expecting cookbook recipies
3 Who do I hope that you will become? Understand the importance of uncertainty See ML as a science not a collection of methods Capable to place methods in context Have the background to learn by yourselves Appreciating the difficulties and challenges to ML
4 Whats the focus of this part of the course My plan My view on Machine Learning 1 Look at each part of a probabilistic model in detail how do they interact what do they provide 2 Different models parametric non-parameteric 3 Inference Really simple data
5 Block Structure 4 Lectures 2 Practical sessions 1 Assignment Deadline November 19th 1 Scheduled Help session
6 Assignment Three parts aligned with lectures Part 1 (Lecture 2 & 3) Task: probabilistic regression Aim: understand probabilistic objects Part 2 (Lecture 3 & 4) Task: probabilistic representation learning Aim: understand probabilistic methodology Part 3 (Self study) Task: probabilistic model selection Aim: show that you can extend your knowledge from 1 and 2
7 Assignment Three parts aligned with lectures Part 1 (Lecture 2 & 3) Task: probabilistic regression Aim: understand probabilistic objects Part 2 (Lecture 3 & 4) Task: probabilistic representation learning Aim: understand probabilistic methodology Part 3 (Self study) Task: probabilistic model selection Aim: show that you can extend your knowledge from 1 and 2
8 Assignment Three parts aligned with lectures Part 1 (Lecture 2 & 3) Task: probabilistic regression Aim: understand probabilistic objects Part 2 (Lecture 3 & 4) Task: probabilistic representation learning Aim: understand probabilistic methodology Part 3 (Self study) Task: probabilistic model selection Aim: show that you can extend your knowledge from 1 and 2
9 Assignment Three parts aligned with lectures Part 1 (Lecture 2 & 3) Task: probabilistic regression Aim: understand probabilistic objects Part 2 (Lecture 3 & 4) Task: probabilistic representation learning Aim: understand probabilistic methodology Part 3 (Self study) Task: probabilistic model selection Aim: show that you can extend your knowledge from 1 and 2
10 Science Model Data Algorithm Prediction
11 Science Model Data Algorithm Prediction
12 Introduction Regression Kernel Methods References Science Model Data Algorithm Prediction
13 Introduction Regression Kernel Methods References Science Model Data Algorithm Prediction
14 Introduction Regression Kernel Methods References Science Model Data Algorithm Prediction
15 Introduction Regression Kernel Methods References Science Model Data Algorithm Prediction
16 My view on Machine Learning
17 My view on Machine Learning 1 An intelligence which at a given instant knew all the forces acting in nature and the position of every object in the universe 1 A philosophical essay on probabilities, Laplace
18 My view on Machine Learning 1 An intelligence which at a given instant knew all the forces acting in nature and the position of every object in the universe - if endowed with a brain sufficiently vast to make all necessary calculations 1 A philosophical essay on probabilities, Laplace
19 My view on Machine Learning 1 An intelligence which at a given instant knew all the forces acting in nature and the position of every object in the universe - if endowed with a brain sufficiently vast to make all necessary calculations - could describe with a single formula the motions of the largest astronomical bodies and those of the smallest atoms. 1 A philosophical essay on probabilities, Laplace
20 My view on Machine Learning 1 An intelligence which at a given instant knew all the forces acting in nature and the position of every object in the universe - if endowed with a brain sufficiently vast to make all necessary calculations - could describe with a single formula the motions of the largest astronomical bodies and those of the smallest atoms. To such an intelligence, nothing would be uncertain; the future, like the past, would be an open book. 1 A philosophical essay on probabilities, Laplace
21 My view on Machine Learning 1 The theory of probabilities is at bottom nothing but common sense reduced to calculus; it enables us to appreciate with exactness that which accurate minds feel with a sort of instinct for which of times they are unable to account. 1 A philosophical essay on probabilities, Laplace
22 My view on Machine Learning 1 It is remarkable that a science which began with the consideration of games of chance should have become the most important object of human knowledge. 1 A philosophical essay on probabilities, Laplace
23 My view on Machine Learning It was our use of probability theory as logic that has enabled us to do so easily what was impossible for those who thought of probability as a physical phenomenon associated with randomness. Quite the opposite; we have thought of probability distributions as carriers of information. 1 1 Probability theory: The logic of science, Jaynes
24 My view on Machine Learning Theme Acknowledge that we do not know everything, i.e. my knowledge is uncertain. Methodology to propagate my uncertainty through all levels of reasoning. Incorporate my uncertain knowledge with observations such that when I see data it reduces my uncertainty according to the evidence provided in the observations.
25 Introduction Regression Kernel Methods
26 Regression Two variates Input data x i R q Output data yi R D Relationship: f : X Y
27 Regression Uncertainty We are uncertain in our data This means we cannot trust our observations the mapping that we learn the predictions that we make under the mapping This part of the course is about making this principled!
28 Regression Uncertainty We are uncertain in our data This means we cannot trust our observations the mapping that we learn the predictions that we make under the mapping This part of the course is about making this principled!
29 Outline Re-cap of Probability basics Re-cap Central Limit Theorem Probabilistic formulation Dual Formulation
30 Probability Basics 1 Expected Value E[x] = µ(x) = xp(x)dx (1) Shows the center of gravity of a distribution Sampled expected value (mean) x = 1 N N x i (2) i 1 Bishop 2006, p
31 Probability Basics 1 Variance σ 2 (x) = var[x] = E[(x E[x]) 2 ] (3) Shows the spread of a distribution Sample variance σ 2 (x) = 1 N 1 N (x i µ(x i )) 2 (4) i 1 Bishop 2006, p
32 Probability Basics Matplotlib3D, /Lecture1/probBasics.py 2 Bishop 2006, p
33 Probability Basics Matplotlib3D, /Lecture1/probBasics.py 2 Bishop 2006, p
34 Probability Basics Matplotlib3D, /Lecture1/probBasics.py 2 Bishop 2006, p
35 Probability Basics 1 Covariance σ(x, y) = E[(x E[x])(y E[y])] (5) [σ(x, Y)] ij = σ(x i, y j ) = k(x i, y j ) (6) Shows how the spread of how to variables vary together Sample co-variance σ(x, y) = 1 N 1 N (x i µ(x i ))(y i µ(y)) (7) i 1 Bishop 2006, p
36 Probability Basics 1 1 Bishop 2006, p
37 Probability Basics 1 1 Bishop 2006, p
38 Probability Basics 1 1 Bishop 2006, p
39 Probability Basics Matplotlib3D, /Lecture1/probBasics.py 2 Bishop 2006, p
40 Linear Regression 3 y i = Wx i (8) Uncertainty Lets assume the relationship is linear Uncertainty in outputs y i Addative noise yi = Wx i + ϵ What form does the noise have ϵ What do we know about the generating process? 3 Bishop 2006, p
41 Linear Regression 3 y i = Wx i + ϵ (9) Uncertainty Lets assume the relationship is linear Uncertainty in outputs y i Addative noise yi = Wx i + ϵ What form does the noise have ϵ What do we know about the generating process? 3 Bishop 2006, p
42 Linear Regression 3 What distribution? Central Limit Theorem a The central limit theorem states that the distribution of the sum (or average) of a large number of independent, identically distributed variables will be approximately normal, regardless of the underlying distribution. a Bishop 2006, p Bishop 2006, p
43 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
44 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
45 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
46 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
47 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
48 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
49 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
50 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
51 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
52 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
53 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
54 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
55 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
56 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
57 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
58 Linear Regression /Lecture1/centralLimit.py 4 Bishop 2006, p
59 Linear Regression 3 p(w Y, X) (10) Uncertainty in Model Posterior conditional distribution after the relevant information has been taken into account What is relevant our belief the observations 3 Bishop 2006, p
60 Linear Regression 3 p(w) (11) Belief about model before seeing data Prior What do I know about the regression parameters 3 Bishop 2006, p
61 Linear Regression 3 p(w) (12) 3 Bishop 2006, p
62 Linear Regression 3 p(w) (13) Belief about model before seeing data Prior What do I know about the regression parameters p(w) = N (0, σ 2 I) (14) 3 Bishop 2006, p
63 Linear Regression 3 p(y i W, x i ) (15) How well does my model predict the data Likelihood Think error function but also how different errors p(y i W, x i ) = N (y i Wx i, τ 2 I) (16) 3 Bishop 2006, p
64 Linear Regression 3 Structure Do the variables co-vary? Are there (in-)dependency structures that I can exploit? p(y W, X) = N p(y i W, x i ) (17) i 3 Bishop 2006, p
65 Linear Regression 3 How do we put everything together? Want to reach the posterior distribution after all relevant information have been taken into account Prediction should reflect my beliefs in the model and the information in the observations We have a gigantic number of possible solutions that are allowed by our data and belief 3 Bishop 2006, p
66 Linear Regression 3 How do we put everything together? Want to reach the posterior distribution after all relevant information have been taken into account Prediction should reflect my beliefs in the model and the information in the observations We have a gigantic number of possible solutions that are allowed by our data and belief 3 Bishop 2006, p
67 Linear Regression 3 How do we put everything together? Want to reach the posterior distribution after all relevant information have been taken into account Prediction should reflect my beliefs in the model and the information in the observations We have a gigantic number of possible solutions that are allowed by our data and belief 3 Bishop 2006, p
68 Linear Regression 3 How do we put everything together? Want to reach the posterior distribution after all relevant information have been taken into account Prediction should reflect my beliefs in the model and the information in the observations We have a gigantic number of possible solutions that are allowed by our data and belief 3 Bishop 2006, p
69 Linear Regression 3 How do we put everything together? Want to reach the posterior distribution after all relevant information have been taken into account Prediction should reflect my beliefs in the model and the information in the observations We have a gigantic number of possible solutions that are allowed by our data and belief 3 Bishop 2006, p
70 Linear Regression 3 p(w D) = p(d W)p(W) p(d) (18) Evidence The denominator shows where the model spreads it probability mass over the data-space (evidence of the model) The denominator does not change with W 3 Bishop 2006, p
71 Linear Regression 3 p(w D) = p(d W)p(W) p(d) (19) p(w Y, X) p(y W, X)p(W) (20) Evidence The denominator shows where the model spreads it probability mass over the data-space (evidence of the model) The denominator does not change with W 3 Bishop 2006, p
72 p(w) 4 4 Bishop 2006, p. 155
73 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
74 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
75 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
76 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
77 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
78 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
79 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
80 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
81 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
82 p(w) p(w X, Y) W samples 4 4 Bishop 2006, p. 155
83 Assignment You should now be able to do the linear part of Task 2.1 and Task 2.2 of the assignment.
84 Evidence 5 5 Bishop 2006, 3.4 p p(d) (21)
85 Toolbox 1. Formulate prediction error by likelihood 2. Formulate belief of model in prior 3. Marginalise irrelevant variables 4. Choose model based on evidence p M (D) (Assignment)
86 Toolbox 1. Formulate prediction error by likelihood 2. Formulate belief of model in prior 3. Marginalise irrelevant variables 4. Choose model based on evidence p M (D) (Assignment)
87 Toolbox 1. Formulate prediction error by likelihood 2. Formulate belief of model in prior 3. Marginalise irrelevant variables 4. Choose model based on evidence p M (D) (Assignment)
88 Toolbox 1. Formulate prediction error by likelihood 2. Formulate belief of model in prior 3. Marginalise irrelevant variables 4. Choose model based on evidence p M (D) (Assignment)
89 Marginalisation p(w) = p(w θ)p(θ)dθ (22) Average according to belief and how well the model fits the observations Pushes belief through model
90 Marginalisation p(w) = p(w θ)p(θ)dθ (23) Average according to belief and how well the model fits the observations Pushes belief through model
91 Marginalisation Nature laughs at the difficulties of integration
92 Choosing Distributions p(x Y) = p(y X)p(X) p(y) (24) Conjugate Distributions The posterior and the prior are in the same family Relationship with all three terms 6 6 Wikipedia
93 Choosing Distributions p(x Y) = p(y X)p(X) p(y) (25) Conjugate Distributions The posterior and the prior are in the same family Relationship with all three terms Carls intuition combining belief in parameters through model should not change the family of the distribution over the parameters 6 6 Wikipedia
94 Choosing Distributions p(x Y) = p(y X)p(X) p(y) (26) Remainder of this part In this part of the course we will only look at Gaussians Gaussians are self-conjugate Gaussian likelihood + Gaussian prior Gaussian posterior On lecture 5 I will show you approximate ways to compute an integral Hedvig will look at non-gaussian priors and likelihoods in her part.
95 Choosing Distributions p(x Y) = p(y X)p(X) p(y) (27) Remainder of this part In this part of the course we will only look at Gaussians Gaussians are self-conjugate Gaussian likelihood + Gaussian prior Gaussian posterior On lecture 5 I will show you approximate ways to compute an integral Hedvig will look at non-gaussian priors and likelihoods in her part.
96 Choosing Distributions p(x Y) = p(y X)p(X) p(y) (28) Remainder of this part In this part of the course we will only look at Gaussians Gaussians are self-conjugate Gaussian likelihood + Gaussian prior Gaussian posterior On lecture 5 I will show you approximate ways to compute an integral Hedvig will look at non-gaussian priors and likelihoods in her part.
97 Reflection That was ALL of Machine Learning Everything else is just details how to choose model what is the right prior how to integrate You will have to approximate and use heuristics but always relate to this
98 Reflection That was ALL of Machine Learning Everything else is just details how to choose model what is the right prior how to integrate You will have to approximate and use heuristics but always relate to this
99 Reflection That was ALL of Machine Learning Everything else is just details how to choose model what is the right prior how to integrate You will have to approximate and use heuristics but always relate to this
100 Reflection That was ALL of Machine Learning Everything else is just details how to choose model what is the right prior how to integrate You will have to approximate and use heuristics but always relate to this
101 Reflection That was ALL of Machine Learning Everything else is just details how to choose model what is the right prior how to integrate You will have to approximate and use heuristics but always relate to this
102 Reflection That was ALL of Machine Learning Everything else is just details how to choose model what is the right prior how to integrate You will have to approximate and use heuristics but always relate to this
103 Example: Image restoration 6 6 Lecture1/imageExample.py
104 Example: Image restoration 6 6 Lecture1/imageExample.py
105 Example: Image restoration 6 p(y X, θ) = N (f(x), σ 2 I) (29) y i = 1 3 (xr i + x g i + xb i) (30) 6 Lecture1/imageExample.py
106 Example: Image restoration 6 6 Lecture1/imageExample.py
107 Introduction Regression Kernel Methods References Example: Image restoration6 p(y X, θ) = N (f (X), σ 2 I) 1 yi = (xri + xgi + xbi ) 3 p(x Y, θ) p(y X, θ)p(x) 6 (31) (32) (33) Lecture1/imageExample.py
108 Introduction Regression Kernel Methods
109 Dual Linear Regression 7 p(y W, X)p(W) p(w Y, X) = p(y) (34) N N p(y W, X) = p(y i W, X) = N (y i, σ 2 I) (35) i i p(w) = N (0, τ 2 I) (36) 7 Bishop 2006, p. 6.1.
110 Dual Linear Regression 7 p(y W, X)p(W) p(w Y, X) = p(y) (37) N N p(y W, X) = p(y i W, X) = N (y i, σ 2 I) (38) i i p(w) = N (0, τ 2 I) (39) p(w Y, X) p(y W, X)p(W) (40) 7 Bishop 2006, p. 6.1.
111 Dual Linear Regression 7 Lets look at a simple 1D problem y R 1 N (41) x R 1 N (42) 7 Bishop 2006, p. 6.1.
112 Dual Linear Regression 7 p(w y, X) = N i 1 2πσ 2 e 1 2σ 2 (wt x i y i ) T (w T x i y i ) 1 2πτ 2 e 1 2τ 2 (wt w) 1 ( 2πσ 2 ) N e 1 2σ 2 (wt X y) T (w T X y) 1 ( 2πτ 2 ) N e (43) 1 2τ 2 (wt w) (44) Objective Want to find the parameters that maximises the above Logarithm is monotonic Minimise negative logarithm of p(w y, X) 7 Bishop 2006, p. 6.1.
113 Dual Linear Regression 7 p(w y, X) = N i 1 2πσ 2 e 1 2σ 2 (wt x i y i ) T (w T x i y i ) 1 2πτ 2 e 1 2τ 2 (wt w) 1 ( 2πσ 2 ) N e 1 2σ 2 (wt X y) T (w T X y) 1 ( 2πτ 2 ) N e (45) 1 2τ 2 (wt w) (46) Objective Want to find the parameters that maximises the above Logarithm is monotonic Minimise negative logarithm of p(w y, X) 7 Bishop 2006, p. 6.1.
114 Dual Linear Regression 7 J(w) = 1 2 (wt X y) T (w T X y) + λ 2 wt w (47) Objective Want to find the parameters that maximises the above Logarithm is monotonic Minimise negative logarithm of p(w y, X) 7 Bishop 2006, p. 6.1.
115 Dual Linear Regression 7 J(w) = 1 2 (wt X y) T (w T X y) + λ 2 wt w (48) δ δw J(w) = 1 2 2XT (w T X y) + λ 2w (49) 2 Optimisation Lets make a point-estimate Pick w that minimises J(w) 7 Bishop 2006, p. 6.1.
116 Dual Linear Regression 7 J(w) = 1 2 (wt X y) T (w T X y) + λ 2 wt w (50) δ δw J(w) = 1 2 2XT (w T X y) + λ 2w (51) 2 w = 1 λ XT (w T X y) = (52) Optimisation Lets make a point-estimate Pick w that minimises J(w) 7 Bishop 2006, p. 6.1.
117 Dual Linear Regression 7 Optimisation J(w) = 1 2 (wt X y) T (w T X y) + λ 2 wt w (53) δ δw J(w) = 1 2 2XT (w T X y) + λ 2w 2 (54) w = 1 N λ XT (w T X y) = X T a = α n x n (55) Lets make a point-estimate Pick w that minimises J(w) 7 Bishop 2006, p n
118 Dual Linear Regression 7 J(w) = 1 2 (wt X y) T (w T X y) + λ 2 wt w (56) w = X T a (57) Formulate Dual J(a) = 1 2 at XX T XX T a a T XX T y yt y + λ 2 at XX T a (58) 7 Bishop 2006, p. 6.1.
119 Dual Linear Regression 7 [K] ij = x T i x j (59) J(a) = 1 2 at KKa aky yt y + λ 2 at Ka (60) 7 Bishop 2006, p. 6.1.
120 Dual Linear Regression 7 [K] ij = x T i x j (61) J(a) = 1 2 at KKa aky yt y + λ 2 at Ka (62) α i = 1 λ (wt x i y i ) (63) N w = α i x i = X T a (64) i a = (K + λi) 1 y (65) 7 Bishop 2006, p. 6.1.
121 Dual Linear Regression 7 [K] ij = x T i x j (66) J(a) = 1 2 at KKa aky yt y + λ 2 at Ka (67) a = (K + λi) 1 y (68) y(x ) = w T x = a T Xx = a T k(x, x ) = (69) = ((K + λi) 1 y) T k(x, x ) = k(x, X)(K + λi) 1 y (70) 7 Bishop 2006, p. 6.1.
122 Dual Linear Regression 7 Linear Regression 1. See data (x i, y) N i 2. Encode relationship in parameter W 3. Throw training away data 4. Make predictions using W 7 Bishop 2006, p. 6.1.
123 Dual Linear Regression 7 Linear Regression 1. See data (x i, y) N i 2. Encode relationship in parameter W 3. Throw training away data 4. Make predictions using W Dual Do NOT throw away data Make predictions using relationship to training data Model complexity depends on data (i.e. it adapts) Non parametric regression 7 Bishop 2006, p. 6.1.
124 Dual Linear Regression 7 Linear Regression 1. See data (x i, y) N i 2. Encode relationship in parameter W 3. Throw training away data 4. Make predictions using W Dual Do NOT throw away data Make predictions using relationship to training data Model complexity depends on data (i.e. it adapts) Non parametric regression 7 Bishop 2006, p. 6.1.
125 Kernels Dual linear regression allows us to write everything in terms of inner products we do not need representation x i What if we map data prior to regression? ϕ : x i f i y(x ) = w T ϕ(x ) = a T ϕ(x)ϕ(x ) = k(x, X)(K + λi) 1 y k(x, x ) = ϕ(x) T ϕ(x ) (71) In dual case we do not need to know ϕ( ) only ϕ( ) T ϕ( )
126 Kernels Dual linear regression allows us to write everything in terms of inner products we do not need representation x i What if we map data prior to regression? ϕ : x i f i y(x ) = w T ϕ(x ) = a T ϕ(x)ϕ(x ) = k(x, X)(K + λi) 1 y k(x, x ) = ϕ(x) T ϕ(x ) (72) In dual case we do not need to know ϕ( ) only ϕ( ) T ϕ( )
127 Kernels k(x i, x j ) = ϕ(x i ) T ϕ(x j ) = ϕ(x i ) ϕ(x j ) cos(θ) (73) Kernel Functions A function that describes an inner product Sub-class of functions think triangle in-equality If we have k(, ) we never have to know the mapping
128 Kernels 8 x R 2 (74) (x T i x j ) 2 = (x i1 x j1 + x i2 x j2 ) 2 = (75) = x 2 i1x 2 j1 + 2x i1 x j1 x i2 x j2 + x 2 i2x 2 j2 = (76) = (x 2 i1, 2x i1 x i2, x 2 i2)(x 2 j1, 2x j1 x j2, x 2 j2) T = (77) = ϕ(x i ) T ϕ(x j ) (78) 8 Bishop 2006, p. 6.2
129 Kernels x R 2 (79) (x T i x j ) 2 = (x i1 x j1 + x i2 x j2 ) 2 = (80) = x 2 i1x 2 j1 + 2x i1 x j1 x i2 x j2 + x 2 i2x 2 j2 = (81) = (x 2 i1, 2x i1 x i2, x 2 i2)(x 2 j1, 2x j1 x j2, x 2 j2) T = (82) = ϕ(x i ) T ϕ(x j ) (83) So k(x i, x j ) = (x T i x j) 2 is a kernel of the mapping ϕ(x) = ((e T 1 x)2, 2e T 1 xet 2 x, (et 2 x)2 ) 8 8 Bishop 2006, p. 6.2
130 The benefits of Kernels Kernels allows for implicit feature mappings We do NOT need to know the feature space The space can have infinite dimensionality The mapping can be non-linear but the problem is still linear! Allows for putting weird things like, strings (DNA) in a vector space More next lecture, these things are very powerful
131 The benefits of Kernels Kernels allows for implicit feature mappings We do NOT need to know the feature space The space can have infinite dimensionality The mapping can be non-linear but the problem is still linear! Allows for putting weird things like, strings (DNA) in a vector space More next lecture, these things are very powerful
132 The benefits of Kernels Kernels allows for implicit feature mappings We do NOT need to know the feature space The space can have infinite dimensionality The mapping can be non-linear but the problem is still linear! Allows for putting weird things like, strings (DNA) in a vector space More next lecture, these things are very powerful
133 The benefits of Kernels Kernels allows for implicit feature mappings We do NOT need to know the feature space The space can have infinite dimensionality The mapping can be non-linear but the problem is still linear! Allows for putting weird things like, strings (DNA) in a vector space More next lecture, these things are very powerful
134 The benefits of Kernels Kernels allows for implicit feature mappings We do NOT need to know the feature space The space can have infinite dimensionality The mapping can be non-linear but the problem is still linear! Allows for putting weird things like, strings (DNA) in a vector space More next lecture, these things are very powerful
135 Next Time Lecture 2 November 5th M2 Continue with Kernels relation to co-variance Non-parametric Regression Gaussian Processes Start Assignment
136 Next Time Lecture 2 November 5th M2 Continue with Kernels relation to co-variance Non-parametric Regression Gaussian Processes Start Assignment
137 e.o.f.
138 References I Pierre Simon Laplace. A philosophical essay on probabilities E T Jaynes. Probability theory: The logic of science. Ed. by G Larry Bretthorst. Cambridge university press, June Christopher M Bishop. Pattern recognition and machine learning
GWAS V: Gaussian processes
GWAS V: Gaussian processes Dr. Oliver Stegle Christoh Lippert Prof. Dr. Karsten Borgwardt Max-Planck-Institutes Tübingen, Germany Tübingen Summer 2011 Oliver Stegle GWAS V: Gaussian processes Summer 2011
Computer Vision Group Prof. Daniel Cremers. 2. Regression (cont.)
Prof. Daniel Cremers 2. Regression (cont.) Regression with MLE (Rep.) Assume that y is affected by Gaussian noise : t = f(x, w)+ where Thus, we have p(t x, w, )=N (t; f(x, w), 2 ) 2 Maximum A-Posteriori
Machine learning - HT Maximum Likelihood
Machine learning - HT 2016 3. Maximum Likelihood Varun Kanade University of Oxford January 27, 2016 Outline Probabilistic Framework Formulate linear regression in the language of probability Introduce
Gaussian Processes for Machine Learning
Gaussian Processes for Machine Learning Carl Edward Rasmussen Max Planck Institute for Biological Cybernetics Tübingen, Germany carl@tuebingen.mpg.de Carlos III, Madrid, May 2006 The actual science of
Machine Learning. Bayesian Regression & Classification. Marc Toussaint U Stuttgart
Machine Learning Bayesian Regression & Classification learning as inference, Bayesian Kernel Ridge regression & Gaussian Processes, Bayesian Kernel Logistic Regression & GP classification, Bayesian Neural
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters
Gaussian processes. Chuong B. Do (updated by Honglak Lee) November 22, 2008
Gaussian processes Chuong B Do (updated by Honglak Lee) November 22, 2008 Many of the classical machine learning algorithms that we talked about during the first half of this course fit the following pattern:
Statistical Techniques in Robotics (16-831, F12) Lecture#21 (Monday November 12) Gaussian Processes
Statistical Techniques in Robotics (16-831, F12) Lecture#21 (Monday November 12) Gaussian Processes Lecturer: Drew Bagnell Scribe: Venkatraman Narayanan 1, M. Koval and P. Parashar 1 Applications of Gaussian
Lecture : Probabilistic Machine Learning
Lecture : Probabilistic Machine Learning Riashat Islam Reasoning and Learning Lab McGill University September 11, 2018 ML : Many Methods with Many Links Modelling Views of Machine Learning Machine Learning
COMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017
COMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University FEATURE EXPANSIONS FEATURE EXPANSIONS
Gaussian processes and bayesian optimization Stanisław Jastrzębski. kudkudak.github.io kudkudak
Gaussian processes and bayesian optimization Stanisław Jastrzębski kudkudak.github.io kudkudak Plan Goal: talk about modern hyperparameter optimization algorithms Bayes reminder: equivalent linear regression
Machine Learning - MT & 5. Basis Expansion, Regularization, Validation
Machine Learning - MT 2016 4 & 5. Basis Expansion, Regularization, Validation Varun Kanade University of Oxford October 19 & 24, 2016 Outline Basis function expansion to capture non-linear relationships
Bayesian Linear Regression [DRAFT - In Progress]
![Bayesian Linear Regression [DRAFT - In Progress]](/thumbs/92/109828713.jpg "Bayesian Linear Regression [DRAFT - In Progress]")
Bayesian Linear Regression [DRAFT - In Progress] David S. Rosenberg Abstract Here we develop some basics of Bayesian linear regression. Most of the calculations for this document come from the basic theory
Intelligent Systems I
Intelligent Systems I 00 INTRODUCTION Stefan Harmeling & Philipp Hennig 24. October 2013 Max Planck Institute for Intelligent Systems Dptmt. of Empirical Inference Which Card? Opening Experiment Which
GAUSSIAN PROCESS REGRESSION
GAUSSIAN PROCESS REGRESSION CSE 515T Spring 2015 1. BACKGROUND The kernel trick again... The Kernel Trick Consider again the linear regression model: y(x) = φ(x) w + ε, with prior p(w) = N (w; 0, Σ). The
Gaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
PATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Chapter 1. Introduction Shuai Huang April 21, 2014 Outline 1 What is Machine Learning? 2 Curve Fitting 3 Probability Theory 4 Model Selection 5 The curse of dimensionality
Tutorial on Gaussian Processes and the Gaussian Process Latent Variable Model
Tutorial on Gaussian Processes and the Gaussian Process Latent Variable Model (& discussion on the GPLVM tech. report by Prof. N. Lawrence, 06) Andreas Damianou Department of Neuro- and Computer Science,
Nonparameteric Regression:
Nonparameteric Regression: Nadaraya-Watson Kernel Regression & Gaussian Process Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
Gaussian Process Regression
Gaussian Process Regression 4F1 Pattern Recognition, 21 Carl Edward Rasmussen Department of Engineering, University of Cambridge November 11th - 16th, 21 Rasmussen (Engineering, Cambridge) Gaussian Process
Statistical Techniques in Robotics (16-831, F12) Lecture#20 (Monday November 12) Gaussian Processes
Statistical Techniques in Robotics (6-83, F) Lecture# (Monday November ) Gaussian Processes Lecturer: Drew Bagnell Scribe: Venkatraman Narayanan Applications of Gaussian Processes (a) Inverse Kinematics
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian
Density Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
Introduction to Machine Learning
Introduction to Machine Learning Kernel Methods Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574 1 / 21
Bayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 2: Bayesian Basics https://people.orie.cornell.edu/andrew/orie6741 Cornell University August 25, 2016 1 / 17 Canonical Machine Learning
Introduction to Gaussian Processes
Introduction to Gaussian Processes Neil D. Lawrence GPSS 10th June 2013 Book Rasmussen and Williams (2006) Outline The Gaussian Density Covariance from Basis Functions Basis Function Representations Constructing
Cheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 89 Part II
Gaussian Processes. 1 What problems can be solved by Gaussian Processes?
Statistical Techniques in Robotics (16-831, F1) Lecture#19 (Wednesday November 16) Gaussian Processes Lecturer: Drew Bagnell Scribe:Yamuna Krishnamurthy 1 1 What problems can be solved by Gaussian Processes?
Multivariate Bayesian Linear Regression MLAI Lecture 11
Multivariate Bayesian Linear Regression MLAI Lecture 11 Neil D. Lawrence Department of Computer Science Sheffield University 21st October 2012 Outline Univariate Bayesian Linear Regression Multivariate
CS-E3210 Machine Learning: Basic Principles
CS-E3210 Machine Learning: Basic Principles Lecture 4: Regression II slides by Markus Heinonen Department of Computer Science Aalto University, School of Science Autumn (Period I) 2017 1 / 61 Today s introduction
STA 4273H: Sta-s-cal Machine Learning
STA 4273H: Sta-s-cal Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistical Sciences! rsalakhu@cs.toronto.edu! h0p://www.cs.utoronto.ca/~rsalakhu/ Lecture 2 In our
Introduction to Bayesian Statistics
School of Computing & Communication, UTS January, 207 Random variables Pre-university: A number is just a fixed value. When we talk about probabilities: When X is a continuous random variable, it has a
Introduction Dual Representations Kernel Design RBF Linear Reg. GP Regression GP Classification Summary. Kernel Methods. Henrik I Christensen
Kernel Methods Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I Christensen (RIM@GT) Kernel Methods 1 / 37 Outline
Probabilistic & Unsupervised Learning
Probabilistic & Unsupervised Learning Gaussian Processes Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London
GWAS IV: Bayesian linear (variance component) models
GWAS IV: Bayesian linear (variance component) models Dr. Oliver Stegle Christoh Lippert Prof. Dr. Karsten Borgwardt Max-Planck-Institutes Tübingen, Germany Tübingen Summer 2011 Oliver Stegle GWAS IV: Bayesian
Bayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 4 Occam s Razor, Model Construction, and Directed Graphical Models https://people.orie.cornell.edu/andrew/orie6741 Cornell University September
Probabilistic Graphical Models
Probabilistic Graphical Models Introduction. Basic Probability and Bayes Volkan Cevher, Matthias Seeger Ecole Polytechnique Fédérale de Lausanne 26/9/2011 (EPFL) Graphical Models 26/9/2011 1 / 28 Outline
Bayesian RL Seminar. Chris Mansley September 9, 2008
Bayesian RL Seminar Chris Mansley September 9, 2008 Bayes Basic Probability One of the basic principles of probability theory, the chain rule, will allow us to derive most of the background material in
COMP 551 Applied Machine Learning Lecture 20: Gaussian processes
COMP 55 Applied Machine Learning Lecture 2: Gaussian processes Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: (herke.vanhoof@mcgill.ca) Class web page: www.cs.mcgill.ca/~hvanho2/comp55
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
PILCO: A Model-Based and Data-Efficient Approach to Policy Search (M.P. Deisenroth and C.E. Rasmussen) CSC2541 November 4, 2016 PILCO Graphical Model PILCO Probabilistic Inference for Learning COntrol
Gaussian Processes (10/16/13)
STA561: Probabilistic machine learning Gaussian Processes (10/16/13) Lecturer: Barbara Engelhardt Scribes: Changwei Hu, Di Jin, Mengdi Wang 1 Introduction In supervised learning, we observe some inputs
COMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 17. Bayesian inference; Bayesian regression Training == optimisation (?) Stages of learning & inference: Formulate model Regression
Model Selection for Gaussian Processes
Institute for Adaptive and Neural Computation School of Informatics,, UK December 26 Outline GP basics Model selection: covariance functions and parameterizations Criteria for model selection Marginal
COMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 2. Statistical Schools Adapted from slides by Ben Rubinstein Statistical Schools of Thought Remainder of lecture is to provide
Parameter estimation Conditional risk
Parameter estimation Conditional risk Formalizing the problem Specify random variables we care about e.g., Commute Time e.g., Heights of buildings in a city We might then pick a particular distribution
Pattern Recognition and Machine Learning. Bishop Chapter 6: Kernel Methods
Pattern Recognition and Machine Learning Chapter 6: Kernel Methods Vasil Khalidov Alex Kläser December 13, 2007 Training Data: Keep or Discard? Parametric methods (linear/nonlinear) so far: learn parameter
Bayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
John Ashburner. Wellcome Trust Centre for Neuroimaging, UCL Institute of Neurology, 12 Queen Square, London WC1N 3BG, UK.
FOR MEDICAL IMAGING John Ashburner Wellcome Trust Centre for Neuroimaging, UCL Institute of Neurology, 12 Queen Square, London WC1N 3BG, UK. PIPELINES V MODELS PROBABILITY THEORY MEDICAL IMAGE COMPUTING
Cheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
NPFL108 Bayesian inference. Introduction. Filip Jurčíček. Institute of Formal and Applied Linguistics Charles University in Prague Czech Republic
NPFL108 Bayesian inference Introduction Filip Jurčíček Institute of Formal and Applied Linguistics Charles University in Prague Czech Republic Home page: http://ufal.mff.cuni.cz/~jurcicek Version: 21/02/2014
Bayesian Models in Machine Learning
Bayesian Models in Machine Learning Lukáš Burget Escuela de Ciencias Informáticas 2017 Buenos Aires, July 24-29 2017 Frequentist vs. Bayesian Frequentist point of view: Probability is the frequency of
Cheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 143 Part IV
Expectation Propagation for Approximate Bayesian Inference
Expectation Propagation for Approximate Bayesian Inference José Miguel Hernández Lobato Universidad Autónoma de Madrid, Computer Science Department February 5, 2007 1/ 24 Bayesian Inference Inference Given
DEPARTMENT OF COMPUTER SCIENCE AUTUMN SEMESTER MACHINE LEARNING AND ADAPTIVE INTELLIGENCE
Data Provided: None DEPARTMENT OF COMPUTER SCIENCE AUTUMN SEMESTER 204 205 MACHINE LEARNING AND ADAPTIVE INTELLIGENCE hour Please note that the rubric of this paper is made different from many other papers.
CSC2541 Lecture 2 Bayesian Occam s Razor and Gaussian Processes
CSC2541 Lecture 2 Bayesian Occam s Razor and Gaussian Processes Roger Grosse Roger Grosse CSC2541 Lecture 2 Bayesian Occam s Razor and Gaussian Processes 1 / 55 Adminis-Trivia Did everyone get my e-mail
Learning Gaussian Process Models from Uncertain Data
Learning Gaussian Process Models from Uncertain Data Patrick Dallaire, Camille Besse, and Brahim Chaib-draa DAMAS Laboratory, Computer Science & Software Engineering Department, Laval University, Canada
Cheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 254 Part V
DEPARTMENT OF COMPUTER SCIENCE Autumn Semester MACHINE LEARNING AND ADAPTIVE INTELLIGENCE
Data Provided: None DEPARTMENT OF COMPUTER SCIENCE Autumn Semester 203 204 MACHINE LEARNING AND ADAPTIVE INTELLIGENCE 2 hours Answer THREE of the four questions. All questions carry equal weight. Figures
Discrete Mathematics and Probability Theory Fall 2015 Lecture 21
CS 70 Discrete Mathematics and Probability Theory Fall 205 Lecture 2 Inference In this note we revisit the problem of inference: Given some data or observations from the world, what can we infer about
Lecture 3: Pattern Classification. Pattern classification
EE E68: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mitures and
Linear Regression and Discrimination
Linear Regression and Discrimination Kernel-based Learning Methods Christian Igel Institut für Neuroinformatik Ruhr-Universität Bochum, Germany http://www.neuroinformatik.rub.de July 16, 2009 Christian
CPSC 340: Machine Learning and Data Mining. MLE and MAP Fall 2017
CPSC 340: Machine Learning and Data Mining MLE and MAP Fall 2017 Assignment 3: Admin 1 late day to hand in tonight, 2 late days for Wednesday. Assignment 4: Due Friday of next week. Last Time: Multi-Class
Outline Lecture 2 2(32)
Outline Lecture (3), Lecture Linear Regression and Classification it is our firm belief that an understanding of linear models is essential for understanding nonlinear ones Thomas Schön Division of Automatic
Machine Learning and Bayesian Inference
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone extension 63725 Email: sbh11@cl.cam.ac.uk www.cl.cam.ac.uk/ sbh11/ Copyright c Sean Holden 22-17. 1 Artificial
ADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING. Non-linear regression techniques Part - II
1 Non-linear regression techniques Part - II Regression Algorithms in this Course Support Vector Machine Relevance Vector Machine Support vector regression Boosting random projections Relevance vector
σ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =
Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,
Lecture 6: April 19, 2002
EE596 Pat. Recog. II: Introduction to Graphical Models Spring 2002 Lecturer: Jeff Bilmes Lecture 6: April 19, 2002 University of Washington Dept. of Electrical Engineering Scribe: Huaning Niu,Özgür Çetin
How to build an automatic statistician
How to build an automatic statistician James Robert Lloyd 1, David Duvenaud 1, Roger Grosse 2, Joshua Tenenbaum 2, Zoubin Ghahramani 1 1: Department of Engineering, University of Cambridge, UK 2: Massachusetts
CSCI567 Machine Learning (Fall 2014)
CSCI567 Machine Learning (Fall 24) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu October 2, 24 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 24) October 2, 24 / 24 Outline Review
Parametric Techniques
Parametric Techniques Jason J. Corso SUNY at Buffalo J. Corso (SUNY at Buffalo) Parametric Techniques 1 / 39 Introduction When covering Bayesian Decision Theory, we assumed the full probabilistic structure
Least Squares Regression
CIS 50: Machine Learning Spring 08: Lecture 4 Least Squares Regression Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may not cover all the
y Xw 2 2 y Xw λ w 2 2
CS 189 Introduction to Machine Learning Spring 2018 Note 4 1 MLE and MAP for Regression (Part I) So far, we ve explored two approaches of the regression framework, Ordinary Least Squares and Ridge Regression:
Probabilistic Reasoning in Deep Learning
Probabilistic Reasoning in Deep Learning Dr Konstantina Palla, PhD palla@stats.ox.ac.uk September 2017 Deep Learning Indaba, Johannesburgh Konstantina Palla 1 / 39 OVERVIEW OF THE TALK Basics of Bayesian
Kernel methods, kernel SVM and ridge regression
Kernel methods, kernel SVM and ridge regression Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Collaborative Filtering 2 Collaborative Filtering R: rating matrix; U: user factor;
Lecture 5: GPs and Streaming regression
Lecture 5: GPs and Streaming regression Gaussian Processes Information gain Confidence intervals COMP-652 and ECSE-608, Lecture 5 - September 19, 2017 1 Recall: Non-parametric regression Input space X
Gaussian with mean ( µ ) and standard deviation ( σ)
Slide from Pieter Abbeel Gaussian with mean ( µ ) and standard deviation ( σ) 10/6/16 CSE-571: Robotics X ~ N( µ, σ ) Y ~ N( aµ + b, a σ ) Y = ax + b + + + + 1 1 1 1 1 1 1 1 1 1, ~ ) ( ) ( ), ( ~ ), (
Nonparametric Regression With Gaussian Processes
Nonparametric Regression With Gaussian Processes From Chap. 45, Information Theory, Inference and Learning Algorithms, D. J. C. McKay Presented by Micha Elsner Nonparametric Regression With Gaussian Processes
These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher Bishop
Music and Machine Learning (IFT68 Winter 8) Prof. Douglas Eck, Université de Montréal These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher Bishop
PMR Learning as Inference
Outline PMR Learning as Inference Probabilistic Modelling and Reasoning Amos Storkey Modelling 2 The Exponential Family 3 Bayesian Sets School of Informatics, University of Edinburgh Amos Storkey PMR Learning
Machine Learning. Lecture 4: Regularization and Bayesian Statistics. Feng Li. https://funglee.github.io
Machine Learning Lecture 4: Regularization and Bayesian Statistics Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 207 Overfitting Problem
Advanced Machine Learning Practical 4b Solution: Regression (BLR, GPR & Gradient Boosting)
Advanced Machine Learning Practical 4b Solution: Regression (BLR, GPR & Gradient Boosting) Professor: Aude Billard Assistants: Nadia Figueroa, Ilaria Lauzana and Brice Platerrier E-mails: aude.billard@epfl.ch,
Overfitting, Bias / Variance Analysis
Overfitting, Bias / Variance Analysis Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 8, 207 / 40 Outline Administration 2 Review of last lecture 3 Basic
Lecture 4: Types of errors. Bayesian regression models. Logistic regression
Lecture 4: Types of errors. Bayesian regression models. Logistic regression A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting more generally COMP-652 and ECSE-68, Lecture
Linear regression example Simple linear regression: f(x) = ϕ(x)t w w ~ N(0, ) The mean and covariance are given by E[f(x)] = ϕ(x)e[w] = 0.
![Linear regression example Simple linear regression: f(x) = ϕ(x)t w w ~ N(0, ) The mean and covariance are given by E[f(x)] = ϕ(x)e[w] = 0.](/thumbs/95/123587665.jpg "Linear regression example Simple linear regression: f(x) = ϕ(x)t w w ~ N(0, ) The mean and covariance are given by E[f(x)] = ϕ(x)e[w] = 0.")
Gaussian Processes Gaussian Process Stochastic process: basically, a set of random variables. may be infinite. usually related in some way. Gaussian process: each variable has a Gaussian distribution every
Parametric Techniques Lecture 3
Parametric Techniques Lecture 3 Jason Corso SUNY at Buffalo 22 January 2009 J. Corso (SUNY at Buffalo) Parametric Techniques Lecture 3 22 January 2009 1 / 39 Introduction In Lecture 2, we learned how to
CPSC 340: Machine Learning and Data Mining
CPSC 340: Machine Learning and Data Mining MLE and MAP Original version of these slides by Mark Schmidt, with modifications by Mike Gelbart. 1 Admin Assignment 4: Due tonight. Assignment 5: Will be released
MA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems
MA/ST 810 Mathematical-Statistical Modeling and Analysis of Complex Systems Principles of Statistical Inference Recap of statistical models Statistical inference (frequentist) Parametric vs. semiparametric
ECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
Probabilistic Graphical Models Lecture 20: Gaussian Processes
Probabilistic Graphical Models Lecture 20: Gaussian Processes Andrew Gordon Wilson www.cs.cmu.edu/~andrewgw Carnegie Mellon University March 30, 2015 1 / 53 What is Machine Learning? Machine learning algorithms
Machine Learning - MT Linear Regression
Machine Learning - MT 2016 2. Linear Regression Varun Kanade University of Oxford October 12, 2016 Announcements All students eligible to take the course for credit can sign-up for classes and practicals
Bayesian Linear Regression. Sargur Srihari
Bayesian Linear Regression Sargur srihari@cedar.buffalo.edu Topics in Bayesian Regression Recall Max Likelihood Linear Regression Parameter Distribution Predictive Distribution Equivalent Kernel 2 Linear
Statistical learning. Chapter 20, Sections 1 3 1
Statistical learning Chapter 20, Sections 1 3 Chapter 20, Sections 1 3 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete
Bristol Machine Learning Reading Group
Bristol Machine Learning Reading Group Introduction to Variational Inference Carl Henrik Ek - carlhenrik.ek@bristol.ac.uk November 25, 2016 http://www.carlhenrik.com Introduction Ronald Aylmer Fisher 1
Machine Learning Srihari. Probability Theory. Sargur N. Srihari
Probability Theory Sargur N. Srihari srihari@cedar.buffalo.edu 1 Probability Theory with Several Variables Key concept is dealing with uncertainty Due to noise and finite data sets Framework for quantification
Lecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods.
Lecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods. Linear models for classification Logistic regression Gradient descent and second-order methods
Lecture 4: Probabilistic Learning
DD2431 Autumn, 2015 1 Maximum Likelihood Methods Maximum A Posteriori Methods Bayesian methods 2 Classification vs Clustering Heuristic Example: K-means Expectation Maximization 3 Maximum Likelihood Methods
Artificial Intelligence: what have we seen so far? Machine Learning and Bayesian Inference
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone extension 6375 Email: sbh@cl.cam.ac.uk www.cl.cam.ac.uk/ sbh/ Artificial Intelligence: what have we seen so
Introduction to Machine Learning
Introduction to Machine Learning Introduction to Probabilistic Methods Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB