SCUOLA DI SPECIALIZZAZIONE IN FISICA MEDICA. Sistemi di Elaborazione dell Informazione. Regressione. Ruggero Donida Labati

Transcription

1 SCUOLA DI SPECIALIZZAZIONE IN FISICA MEDICA Sistemi di Elaborazione dell Informazione Regressione Ruggero Donida Labati Dipartimento di Informatica via Bramante 65, Crema (CR), Italy

2 Regression The following slides are from: C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006,.

3 Polynomial Curve Fitting

4 Sum-of-Squares Error Function

5 0 th Order Polynomial

6 1 st Order Polynomial

7 3 rd Order Polynomial

8 9 th Order Polynomial

9 Over-fitting Root-Mean-Square (RMS) Error:

10 Polynomial Coefficients

11 9 th Order Polynomial Data Set Size:

12 9 th Order Polynomial Data Set Size:

13 Regularization Penalize large coefficient values

14 Regularization:

15 Regularization:

16 Regularization: vs.

17 Polynomial Coefficients

18 How Bayesian Inference Works blogs/how-bayesian-inference-works

19 Apples and Oranges Probability Theory

20 Probability Theory Marginal Probability Joint Probability Conditional Probability

21 Probability Theory Sum Rule Product Rule

22 The Rules of Probability Sum Rule Product Rule

23 Bayes Theorem Data posterior likelihood prior Model parameters Alternatively posterior = likelihood prior evidence

24 The example p(b = r) = 4/10 p(b = b) = 6/10 p(f = a B = r) = 1/4 p(f = o B = r) = 3/4 p(f = a B = b) = 3/4 p(f = o B = b) = 1/4

25 Probability Densities

26 Expectations Conditional Expectation (discrete) Approximate Expectation (discrete and continuous)

27 Variances and Covariances

28 The Gaussian Distribution

29 Gaussian Mean and Variance

30 The Multivariate Gaussian

31 Gaussian Parameter Estimation Likelihood function

32 Maximum (Log) Likelihood

33 Maximum Likelihood Determine by minimizing sum-of-squares error,.

34 Curve Fitting Re-visited

35 Linear models for regression

36 Linear Basis Function Models (1) Example: Polynomial Curve Fitting

37 Linear Basis Function Models (2)

38 Linear Basis Function Models (3) Polynomial basis functions: These are global; a small change in x affect all basis functions.

39 Linear Basis Function Models (4) Gaussian basis functions: These are local; a small change in x only affect nearby basis functions. μ j and s control location and scale (width).

40 Linear Basis Function Models (5) Sigmoidal basis functions: where Also these are local; a small change in x only affect nearby basis functions. μ j and s control location and scale (slope).

41 Maximum Likelihood and Least Squares (1) Assume observations from a deterministic function with added Gaussian noise: where which is the same as saying, Given observed inputs,, and targets,, we obtain the likelihood function

42 Maximum Likelihood and Least Squares (2) Taking the logarithm, we get where is the sum-of-squares error.

43 Maximum Likelihood and Least Squares (3) Computing the gradient and setting it to zero yields Solving for w, we get The Moore-Penrose pseudo-inverse,. where

44 Sequential Learning Data items considered one at a time (a.k.a. online learning); use stochastic (sequential) gradient descent: This is known as the least-mean-squares (LMS) algorithm. Issue: how to choose?

45 Regularized Least Squares Consider the error function: Data term + Regularization term With the sum-of-squares error function and a quadratic regularizer, we get which is minimized by λ is called the regularization coefficient.

46 The Bias-Variance Decomposition (1) Recall the expected squared loss, where The second term of E[L] corresponds to the noise inherent in the random variable t. What about the first term?

47 The Bias-Variance Decomposition (2) Suppose we were given multiple data sets, each of size N. Any particular data set, D, will give a particular function y(x;d). We then have

48 The Bias-Variance Decomposition (3) Taking the expectation over D yields

49 The Bias-Variance Decomposition (4) Thus we can write where

50 The Bias-Variance Decomposition (5) Example: 25 data sets from the sinusoidal, varying the degree of regularization, λ.

51 The Bias-Variance Decomposition (6) Example: 25 data sets from the sinusoidal, varying the degree of regularization, λ.

52 The Bias-Variance Decomposition (7) Example: 25 data sets from the sinusoidal, varying the degree of regularization, λ.

53 The Bias-Variance Trade-off From these plots, we note that an over-regularized model (large λ) will have a high bias, while an underregularized model (small λ) will have a high variance.

54 Bayesian Linear Regression (1) Define a conjugate prior over w Combining this with the likelihood function and using results for marginal and conditional Gaussian distributions, gives the posterior where

55 Bayesian Linear Regression (2) A common choice for the prior is for which Next we consider an example

56 Bayesian Linear Regression (3) 0 data points observed Prior Data Space

57 Bayesian Linear Regression (4) 1 data point observed Likelihood Posterior Data Space

58 Bayesian Linear Regression (5) 2 data points observed Likelihood Posterior Data Space

59 Bayesian Linear Regression (6) 20 data points observed Likelihood Posterior Data Space