Non-parametric Methods

Transcription

1 Non-parametric Methods Machine Learning Alireza Ghane Non-Parametric Methods Alireza Ghane / Torsten Möller 1

2 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 2

3 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 3

4 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 4

5 Hand-written Digit Recognition 518 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 22 Fig. 8. All of the misclassified MNIST test digits using our method 63 out of 1,). The text above each digit indicates the example number followed by the true label and the assigned label. Belongie et al. PAMI 22 straightforward sum of squared differences SSD). SSD error rate with an average of only four two-dimensional Difficult performs very well toonhand-craft this easy database duerules to the lackabout of views for digits each three-dimensional object, thanks to the variation in lighting [24] PCA just makes it faster). flexibility provided by the matching algorithm. The prototype selection algorithm is illustrated in Fig. 1. As seen, views are allocated mainly for more complex 6.3 MPEG-7 Shape Silhouette Database categories with high within class variability. The curve Our next experiment involves the MPEG-7 shape silhouette Non-Parametric Methods Alireza marked SC-proto in Fig. 9 shows the improved classification database, Ghane specifically / TorstenCore Möller Experiment CE-Shape-1 part B, 5

6 Hand-written Digit Recognition CHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 22 x i = t i = (,,, 1,,,,,, ) Represent input image as a vector x i R 784. Suppose we have a target vector t i This is supervised learning Discrete, finite label set: perhaps t i {, 1} 1, a classification problem Given a training set {(x 1, t 1 ),..., (x N, t N )}, learning problem is to construct a good function y(x) from these. y : R 784 R 1 Non-Parametric Methods Alireza Ghane / Torsten Möller 6

7 Face Detection Classification problem Schneiderman and Kanade, IJCV 22 t i {, 1, 2}, non-face, frontal face, profile face. Non-Parametric Methods Alireza Ghane / Torsten Möller 7

8 Spam Detection Classification problem t i {, 1}, non-spam, spam x i counts of words, e.g. Viagra, stock, outperform, multi-bagger Non-Parametric Methods Alireza Ghane / Torsten Möller 8

9 Stock Price Prediction Problems in which t i is continuous are called regression E.g. t i is stock price, x i contains company profit, debt, cash flow, gross sales, number of spam s sent,... Non-Parametric Methods Alireza Ghane / Torsten Möller 9

10 Clustering Images Wang et al., CVPR 26 Only x i is defined: unsupervised learning E.g. x i describes image, find groups of similar images Non-Parametric Methods Alireza Ghane / Torsten Möller 1

11 Types of Learning Problems Supervised Learning Classification Regression Unsupervised Learning Density estimation Clustering: k-means, mixture models, hierarchical clustering Hidden Markov models Reinforcement Learning Non-Parametric Methods Alireza Ghane / Torsten Möller 11

12 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 12

13 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 13

14 An Example - Polynomial Curve Fitting Suppose we are given training set of N observations (x 1,..., x N ) and (t 1,..., t N ), x i, t i R Regression problem, estimate y(x) from these data Non-Parametric Methods Alireza Ghane / Torsten Möller 14

15 Polynomial Curve Fitting What form is y(x)? Let s try polynomials of degree M: y(x, w) = w +w 1 x+w 2 x w M x M 1 This is the hypothesis space. How do we measure success? Sum of squared errors: E(w) = 1 2 N {y(x n, w) t n } 2 n=1 t 1 1 tn y(xn, w) Among functions in the class, choose that which minimizes this error xn x Non-Parametric Methods Alireza Ghane / Torsten Möller 15

16 Which Degree of Polynomial? A model selection problem M = 9 E(w ) = : This is over-fitting Non-Parametric Methods Alireza Ghane / Torsten Möller 16

17 Generalization 1 Training Test Generalization is the holy grail of ML Want good performance for new data Measure generalization using a separate set Use root-mean-squared (RMS) error: E RMS = 2E(w )/N Non-Parametric Methods Alireza Ghane / Torsten Möller 17

18 Controlling Over-fitting: Regularization As order of polynomial M increases, so do coefficient magnitudes Penalize large coefficients in error function: Ẽ(w) = 1 2 N {y(x n, w) t n } 2 + λ 2 w 2 n=1 Non-Parametric Methods Alireza Ghane / Torsten Möller 18

19 Controlling Over-fitting: Regularization As order of polynomial M increases, so do coefficient magnitudes Penalize large coefficients in error function: Ẽ(w) = 1 2 N {y(x n, w) t n } 2 + λ 2 w 2 n=1 Non-Parametric Methods Alireza Ghane / Torsten Möller 19

20 Controlling Over-fitting: Regularization Non-Parametric Methods Alireza Ghane / Torsten Möller 2

21 Controlling Over-fitting: Regularization 1 Training Test Note the E RMS for the training set. Perfect match of training set with the model is a result of over-fitting Training and test error show similar trend Non-Parametric Methods Alireza Ghane / Torsten Möller 21

22 Over-fitting: Dataset size With more data, more complex model (M = 9) can be fit Rule of thumb: 1 datapoints for each parameter Non-Parametric Methods Alireza Ghane / Torsten Möller 22

23 Validation Set Split training data into training set and validation set Train different models (e.g. diff. order polynomials) on training set Choose model (e.g. order of polynomial) with minimum error on validation set Non-Parametric Methods Alireza Ghane / Torsten Möller 23

24 Cross-validation run 1 run 2 run 3 run 4 Data are often limited Cross-validation creates S groups of data, use S 1 to train, other to validate Extreme case leave-one-out cross-validation (LOO-CV): S is number of training data points Cross-validation is an effective method for model selection, but can be slow Models with multiple complexity parameters: exponential number of runs Non-Parametric Methods Alireza Ghane / Torsten Möller 24

25 Summary Want models that generalize to new data Train model on training set Measure performance on held-out test set Performance on test set is good estimate of performance on new data Non-Parametric Methods Alireza Ghane / Torsten Möller 25

26 Summary - Model Selection Which model to use? E.g. which degree polynomial? Training set error is lower with more complex model Can t just choose the model with lowest training error Peeking at test error is unfair. E.g. picking polynomial with lowest test error Performance on test set is no longer good estimate of performance on new data Non-Parametric Methods Alireza Ghane / Torsten Möller 26

27 Summary - Model Selection Which model to use? E.g. which degree polynomial? Training set error is lower with more complex model Can t just choose the model with lowest training error Peeking at test error is unfair. E.g. picking polynomial with lowest test error Performance on test set is no longer good estimate of performance on new data Non-Parametric Methods Alireza Ghane / Torsten Möller 27

28 Summary - Solutions I Use a validation set Train models on training set. E.g. different degree polynomials Measure performance on held-out validation set Measure performance of that model on held-out test set Can use cross-validation on training set instead of a separate validation set if little data and lots of time Choose model with lowest error over all cross-validation folds (e.g. polynomial degree) Retrain that model using all training data (e.g. polynomial coefficients) Non-Parametric Methods Alireza Ghane / Torsten Möller 28

29 Summary - Solutions I Use a validation set Train models on training set. E.g. different degree polynomials Measure performance on held-out validation set Measure performance of that model on held-out test set Can use cross-validation on training set instead of a separate validation set if little data and lots of time Choose model with lowest error over all cross-validation folds (e.g. polynomial degree) Retrain that model using all training data (e.g. polynomial coefficients) Non-Parametric Methods Alireza Ghane / Torsten Möller 29

30 Summary - Solutions II Use regularization Train complex model (e.g high order polynomial) but penalize being too complex (e.g. large weight magnitudes) Need to balance error vs. regularization (λ) Choose λ using cross-validation Get more data Non-Parametric Methods Alireza Ghane / Torsten Möller 3

31 Summary - Solutions II Use regularization Train complex model (e.g high order polynomial) but penalize being too complex (e.g. large weight magnitudes) Need to balance error vs. regularization (λ) Choose λ using cross-validation Get more data Non-Parametric Methods Alireza Ghane / Torsten Möller 31

32 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 32

33 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion

34 Decision Theory For a sample x, decide which class(c k ) it is from. Ideas: Maximum Likelihood Minimum Loss/Cost (e.g. misclassification rate) Maximum Aposteriori (MAP) Intro. to Machine Learning Alireza Ghane 34

35 Decision: Maximum Likelihood Inference step: Determine statistics from training data. p(x, t) OR p(x C k ) Decision step: Determine optimal t for test input x: t = arg max{ p (x C k ) k }{{} Likelihood } Intro. to Machine Learning Alireza Ghane 35

36 Decision: Maximum Likelihood Inference step: Determine statistics from training data. p(x, t) OR p(x C k ) Decision step: Determine optimal t for test input x: t = arg max{ p (x C k ) k }{{} Likelihood } Intro. to Machine Learning Alireza Ghane 36

37 Decision: Maximum Likelihood Inference step: Determine statistics from training data. p(x, t) OR p(x C k ) Decision step: Determine optimal t for test input x: t = arg max{ p (x C k ) k }{{} Likelihood } Intro. to Machine Learning Alireza Ghane 37

38 Decision: Minimum Misclassification Rate q(mistake) = p (x R 1, C 2 ) + p (x R 2, C 1 ) = R 1 p (x, C 2 ) dx + R 2 p (x, C 1 ) dx q(mistake) = k R j p (x, C k ) dx j p(x, C1) x x p(x, C2) ˆx: decision boundary. x : optimal decision boundary x : arg min{p (mistake)} R 1 x R1 R2 Intro. to Machine Learning Alireza Ghane 38

39 Decision: Minimum Misclassification Rate q(mistake) = p (x R 1, C 2 ) + p (x R 2, C 1 ) = R 1 p (x, C 2 ) dx + R 2 p (x, C 1 ) dx q(mistake) = k R j p (x, C k ) dx j p(x, C1) x x p(x, C2) ˆx: decision boundary. x : optimal decision boundary x : arg min{p (mistake)} R 1 x R1 R2 Intro. to Machine Learning Alireza Ghane 39

40 Decision: Minimum Loss/Cost Misclassification rate: R : arg min {R i i {1,,K}} L (R j, C k ) Weighted loss/cost function: R : arg min W j,k L (R j, C k ) {R i i {1,,K}} Is useful when: The population of the classes are different The failure cost is non-symmetric k k j j Intro. to Machine Learning Alireza Ghane 4

41 Decision: Maximum Aposteriori (MAP) Bayes Theorem: P {A B} = P {B A}P {A} P {B} p(c k x) }{{} P osterior p(x C k ) }{{} Likelihood p(c k ) }{{} P rior Provides an Aposteriori Belief for the estimation, rather than a single point estimate. Can utilize Apriori Information in the decision. Intro. to Machine Learning Alireza Ghane 41

42 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Non-Parametric Methods Alireza Ghane / Torsten Möller 42

43 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion

44 Coin Tossing Let s say you re given a coin, and you want to find out P (heads), the probability that if you flip it it lands as heads. Flip it a few times: H H T P (heads) = 2/3 Hmm... is this rigorous? Does this make sense? Non-Parametric Methods Alireza Ghane / Torsten Möller 44

45 Coin Tossing Let s say you re given a coin, and you want to find out P (heads), the probability that if you flip it it lands as heads. Flip it a few times: H H T P (heads) = 2/3 Hmm... is this rigorous? Does this make sense? Non-Parametric Methods Alireza Ghane / Torsten Möller 45

46 Coin Tossing Let s say you re given a coin, and you want to find out P (heads), the probability that if you flip it it lands as heads. Flip it a few times: H H T P (heads) = 2/3 Hmm... is this rigorous? Does this make sense? Non-Parametric Methods Alireza Ghane / Torsten Möller 46

47 Coin Tossing - Model Bernoulli distribution P (heads) = µ, P (tails) = 1 µ Assume coin flips are independent and identically distributed (i.i.d.) i.e. All are separate samples from the Bernoulli distribution Given data D = {x 1,..., x N }, heads: x i = 1, tails: x i =, the likelihood of the data is: p(d µ) = N p(x n µ) = n=1 N µ xn (1 µ) 1 xn n=1 Non-Parametric Methods Alireza Ghane / Torsten Möller 47

48 Maximum Likelihood Estimation Given D with h heads and t tails What should µ be? Maximum Likelihood Estimation (MLE): choose µ which maximizes the likelihood of the data µ ML = arg max µ p(d µ) Since ln( ) is monotone increasing: µ ML = arg max ln p(d µ) µ Non-Parametric Methods Alireza Ghane / Torsten Möller 48

49 Likelihood: Log-likelihood: Maximum Likelihood Estimation ln p(d µ) = p(d µ) = N µ xn (1 µ) 1 xn n=1 N x n ln µ + (1 x n ) ln(1 µ) n=1 Take derivative, set to : d N dµ ln p(d µ) = 1 x n µ (1 x 1 n) 1 µ = 1 µ h 1 1 µ t n=1 µ = h t + h Non-Parametric Methods Alireza Ghane / Torsten Möller 49

50 Likelihood: Log-likelihood: Maximum Likelihood Estimation ln p(d µ) = p(d µ) = N µ xn (1 µ) 1 xn n=1 N x n ln µ + (1 x n ) ln(1 µ) n=1 Take derivative, set to : d N dµ ln p(d µ) = 1 x n µ (1 x 1 n) 1 µ = 1 µ h 1 1 µ t n=1 µ = h t + h Non-Parametric Methods Alireza Ghane / Torsten Möller 5

51 Likelihood: Log-likelihood: Maximum Likelihood Estimation ln p(d µ) = p(d µ) = N µ xn (1 µ) 1 xn n=1 N x n ln µ + (1 x n ) ln(1 µ) n=1 Take derivative, set to : d N dµ ln p(d µ) = 1 x n µ (1 x 1 n) 1 µ = 1 µ h 1 1 µ t n=1 µ = h t + h Non-Parametric Methods Alireza Ghane / Torsten Möller 51

52 Likelihood: Log-likelihood: Maximum Likelihood Estimation ln p(d µ) = p(d µ) = N µ xn (1 µ) 1 xn n=1 N x n ln µ + (1 x n ) ln(1 µ) n=1 Take derivative, set to : d N dµ ln p(d µ) = 1 x n µ (1 x 1 n) 1 µ = 1 µ h 1 1 µ t n=1 µ = h t + h Non-Parametric Methods Alireza Ghane / Torsten Möller 52

53 Likelihood: Log-likelihood: Maximum Likelihood Estimation ln p(d µ) = p(d µ) = N µ xn (1 µ) 1 xn n=1 N x n ln µ + (1 x n ) ln(1 µ) n=1 Take derivative, set to : d N dµ ln p(d µ) = 1 x n µ (1 x 1 n) 1 µ = 1 µ h 1 1 µ t n=1 µ = h t + h Non-Parametric Methods Alireza Ghane / Torsten Möller 53

54 Bayesian Learning Wait, does this make sense? What if I flip 1 time, heads? Do I believe µ=1? Learn µ the Bayesian way: P (µ D) = P (µ D) }{{} posterior P (D µ)p (µ) P (D) P (D µ) P (µ) }{{}}{{} prior likelihood Prior encodes knowledge that most coins are 5-5 Conjugate prior makes math simpler, easy interpretation For Bernoulli, the beta distribution is its conjugate Non-Parametric Methods Alireza Ghane / Torsten Möller 54

55 Bayesian Learning Wait, does this make sense? What if I flip 1 time, heads? Do I believe µ=1? Learn µ the Bayesian way: P (µ D) = P (µ D) }{{} posterior P (D µ)p (µ) P (D) P (D µ) P (µ) }{{}}{{} prior likelihood Prior encodes knowledge that most coins are 5-5 Conjugate prior makes math simpler, easy interpretation For Bernoulli, the beta distribution is its conjugate Non-Parametric Methods Alireza Ghane / Torsten Möller 55

56 Bayesian Learning Wait, does this make sense? What if I flip 1 time, heads? Do I believe µ=1? Learn µ the Bayesian way: P (µ D) = P (µ D) }{{} posterior P (D µ)p (µ) P (D) P (D µ) P (µ) }{{}}{{} prior likelihood Prior encodes knowledge that most coins are 5-5 Conjugate prior makes math simpler, easy interpretation For Bernoulli, the beta distribution is its conjugate Non-Parametric Methods Alireza Ghane / Torsten Möller 56

57 Beta Distribution We will use the Beta distribution to express our prior knowledge about coins: Beta(µ a, b) = Γ(a + b) µ a 1 (1 µ) b 1 Γ(a)Γ(b) }{{} normalization Parameters a and b control the shape of this distribution Non-Parametric Methods Alireza Ghane / Torsten Möller 57

58 Posterior P (µ D) P (D µ)p (µ) N µ xn (1 µ) 1 xn µ a 1 (1 µ) b 1 }{{} n=1 }{{} prior likelihood µ h (1 µ) t µ a 1 (1 µ) b 1 µ h+a 1 (1 µ) t+b 1 Simple form for posterior is due to use of conjugate prior Parameters a and b act as extra observations Note that as N = h + t, prior is ignored Non-Parametric Methods Alireza Ghane / Torsten Möller 58

59 Posterior P (µ D) P (D µ)p (µ) N µ xn (1 µ) 1 xn µ a 1 (1 µ) b 1 }{{} n=1 }{{} prior likelihood µ h (1 µ) t µ a 1 (1 µ) b 1 µ h+a 1 (1 µ) t+b 1 Simple form for posterior is due to use of conjugate prior Parameters a and b act as extra observations Note that as N = h + t, prior is ignored Non-Parametric Methods Alireza Ghane / Torsten Möller 59

60 Posterior P (µ D) P (D µ)p (µ) N µ xn (1 µ) 1 xn µ a 1 (1 µ) b 1 }{{} n=1 }{{} prior likelihood µ h (1 µ) t µ a 1 (1 µ) b 1 µ h+a 1 (1 µ) t+b 1 Simple form for posterior is due to use of conjugate prior Parameters a and b act as extra observations Note that as N = h + t, prior is ignored Non-Parametric Methods Alireza Ghane / Torsten Möller 6

61 Maximum A Posteriori Given posterior P (µ D) we could compute a single value, known as the Maximum a Posteriori (MAP) estimate for µ: µ MAP = arg max µ P (µ D) Known as point estimation However, correct Bayesian thing to do is to use the full distribution over µ i.e. Compute E µ [f] = p(µ D)f(µ)dµ This integral is usually hard to compute Non-Parametric Methods Alireza Ghane / Torsten Möller 61

62 Maximum A Posteriori Given posterior P (µ D) we could compute a single value, known as the Maximum a Posteriori (MAP) estimate for µ: µ MAP = arg max µ P (µ D) Known as point estimation However, correct Bayesian thing to do is to use the full distribution over µ i.e. Compute E µ [f] = p(µ D)f(µ)dµ This integral is usually hard to compute Non-Parametric Methods Alireza Ghane / Torsten Möller 62

63 Maximum A Posteriori Given posterior P (µ D) we could compute a single value, known as the Maximum a Posteriori (MAP) estimate for µ: µ MAP = arg max µ P (µ D) Known as point estimation However, correct Bayesian thing to do is to use the full distribution over µ i.e. Compute E µ [f] = p(µ D)f(µ)dµ This integral is usually hard to compute Non-Parametric Methods Alireza Ghane / Torsten Möller 63

64 Polynomial Curve Fitting: What We Did What form is y(x)? Let s try polynomials of degree M: y(x, w) = w +w 1 x+w 2 x w M x M 1 This is the hypothesis space. How do we measure success? Sum of squared errors: E(w) = 1 2 N {y(x n, w) t n } 2 n=1 t 1 1 tn y(xn, w) Among functions in the class, choose that which minimizes this error xn x Intro. to Machine Learning Alireza Ghane 64

65 Curve Fitting: Probabilistic Approach t y(x, w) y(x, w) p(t x, w, β) 2σ x x N p(t x, w, β) = N ( t n y(x n, w), β 1) n=1 Intro. to Machine Learning Alireza Ghane 65

66 Curve Fitting: Probabilistic Approach t y(x, w) y(x, w) p(t x, w, β) 2σ x x p(t x, w, β) = N N ( t n y(x n, w), β 1) n=1 ln (p(t x, w, β)) = β N {y(x n, w) t n } 2 + N 2 2 ln β N ln (2π) n=1 }{{}} 2 {{}}{{} const. const. βe(w) Intro. to Machine Learning Alireza Ghane 66

67 Curve Fitting: Probabilistic Approach t y(x, w) y(x, w) p(t x, w, β) 2σ x x p(t x, w, β) = N N ( t n y(x n, w), β 1) n=1 ln (p(t x, w, β)) = β N {y(x n, w) t n } 2 + N 2 2 ln β N ln (2π) n=1 }{{}} 2 {{}}{{} const. const. βe(w) Maximize log-likelihood Minimize E(w). Can optimize for β as well. Intro. to Machine Learning Alireza Ghane 67

68 Curve Fitting: Bayesian Approach t y(x, w) y(x, w) p(t x, w, β) 2σ x x N p(t x, w, β) = N ( t n y(x n, w), β 1) n=1 Intro. to Machine Learning Alireza Ghane 68

69 Curve Fitting: Bayesian Approach t y(x, w) y(x, w) p(t x, w, β) 2σ x x N p(t x, w, β) = N ( t n y(x n, w), β 1) n=1 Posterior Dist.:p (w x, t, α, β) p (t x, w, β) p (w α) Intro. to Machine Learning Alireza Ghane 69

70 Curve Fitting: Bayesian Approach t y(x, w) y(x, w) p(t x, w, β) 2σ x x N p(t x, w, β) = N ( t n y(x n, w), β 1) n=1 Posterior Dist.:p (w x, t, α, β) p (t x, w, β) p (w α) Minimize: β N {y(x n, w) t n } 2 + α 2 2 wt w n=1 }{{}}{{} regularization. βe(w) Intro. to Machine Learning Alireza Ghane 7

71 Curve Fitting: Bayesian p (t x, w, β, α) = N (t P w (x ), Q w,β,α (x )) t y(x, w) y(x, w) p(t x, w, β) 2σ x x Intro. to Machine Learning Alireza Ghane 71

72 Curve Fitting: Bayesian p (t x, w, β, α) = N (t P w (x ), Q w,β,α (x )) t y(x, w) p (t x, x, t) = N ( t m(x), s 2 (x) ) y(x, w) p(t x, w, β) 2σ x x Intro. to Machine Learning Alireza Ghane 72

73 Curve Fitting: Bayesian p (t x, w, β, α) = N (t P w (x ), Q w,β,α (x )) t y(x, w) p (t x, x, t) = N ( t m(x), s 2 (x) ) y(x, w) p(t x, w, β) 2σ N m(x) = φ(x) T S φ(x n )t n n=1 s 2 (x) = β 1 ( 1 + φ(x) T Sφ(x) ) 1 x x S 1 = α N β I + φ(x n )φ(x n ) T n=1 1 1 Intro. to Machine Learning Alireza Ghane 73

74 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Intro. to Machine Learning Alireza Ghane 74

75 Histograms Consider the problem of modelling the distribution of brightness values in pictures taken on sunny days versus cloudy days We could build histograms of pixel values for each class Intro. to Machine Learning Alireza Ghane 75

76 Histograms E.g. for sunny days Count n i number of datapoints (pixels) with brightness value falling into each bin: p i = n i N i Sensitive to bin width i Discontinuous due to bin edges In D-dim space with M bins per dimension, M D bins Intro. to Machine Learning Alireza Ghane 76

77 Histograms E.g. for sunny days Count n i number of datapoints (pixels) with brightness value falling into each bin: p i = n i N i Sensitive to bin width i Discontinuous due to bin edges In D-dim space with M bins per dimension, M D bins Intro. to Machine Learning Alireza Ghane 77

78 Histograms E.g. for sunny days Count n i number of datapoints (pixels) with brightness value falling into each bin: p i = n i N i Sensitive to bin width i Discontinuous due to bin edges In D-dim space with M bins per dimension, M D bins Intro. to Machine Learning Alireza Ghane 78

79 Histograms E.g. for sunny days Count n i number of datapoints (pixels) with brightness value falling into each bin: p i = n i N i Sensitive to bin width i Discontinuous due to bin edges In D-dim space with M bins per dimension, M D bins Intro. to Machine Learning Alireza Ghane 79

80 Local Density Estimation In a histogram we use nearby points to estimate density For a small region around x, estimate density as: p(x) = K NV K is number of points in region, V is volume of region, N is total number of datapoints Intro. to Machine Learning Alireza Ghane 8

81 Kernel Density Estimation Try to keep idea of using nearby points to estimate density, but obtain smoother estimate Estimate density by placing a small bump at each datapoint Kernel function k( ) determines shape of these bumps Density estimate is p(x) 1 N N ( ) x xn k h n=1 Intro. to Machine Learning Alireza Ghane 81

82 Kernel Density Estimation Example using Gaussian kernel: p(x) = 1 N N 1 (2πh 2 exp { x x n 2 } ) 1/2 2h 2 n=1 Intro. to Machine Learning Alireza Ghane 82

83 Kernel Density Estimation !3!2! Other kernels: Rectangle, Triangle, Epanechnikov Intro. to Machine Learning Alireza Ghane 83

84 Kernel Density Estimation !3!2! ! Other kernels: Rectangle, Triangle, Epanechnikov Intro. to Machine Learning Alireza Ghane 84

85 Kernel Density Estimation !3!2! ! Other kernels: Rectangle, Triangle, Epanechnikov Fast at training time, slow at test time keep all datapoints Intro. to Machine Learning Alireza Ghane 85

86 Kernel Density Estimation !3!2! ! Other kernels: Rectangle, Triangle, Epanechnikov Fast at training time, slow at test time keep all datapoints Sensitive to kernel bandwidth h Intro. to Machine Learning Alireza Ghane 86

87 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Intro. to Machine Learning Alireza Ghane 87

88 5 Nearest-neighbour Instead of relying on kernel bandwidth to get proper density estimate, fix number of nearby points K: p(x) = K NV Note: diverges, not proper density estimate Intro. to Machine Learning Alireza Ghane 88

89 Nearest-neighbour for Classification K Nearest neighbour is often used for classification Classification: predict labels t i from x i Intro. to Machine Learning Alireza Ghane 89

90 x 2 Nearest-neighbour for Classification (a) x 1 K Nearest neighbour is often used for classification Classification: predict labels t i from x i e.g. x i R 2 and t i {, 1}, 3-nearest neighbour Intro. to Machine Learning Alireza Ghane 9

91 Nearest-neighbour for Classification x 2 x 2 (a) x 1 (b) x 1 K Nearest neighbour is often used for classification Classification: predict labels t i from x i e.g. x i R 2 and t i {, 1}, 3-nearest neighbour K = 1 referred to as nearest-neighbour Intro. to Machine Learning Alireza Ghane 91

92 Nearest-neighbour for Classification Good baseline method Slow, but can use fancy data structures for efficiency (KD-trees, Locality Sensitive Hashing) Nice theoretical properties As we obtain more training data points, space becomes more filled with labelled data As N error no more than twice Bayes error Intro. to Machine Learning Alireza Ghane 92

93 Bayes Error p(x, C 1 ) x x p(x, C 2 ) x R 1 R 2 Best classification possible given features Two classes, PDFs shown Decision rule: C 1 if x ˆx; makes errors on red, green, and blue regions Optimal decision rule: C 1 if x x, Bayes error is area of green and blue regions Intro. to Machine Learning Alireza Ghane 93

94 Bayes Error p(x, C 1 ) x x p(x, C 2 ) x R 1 R 2 Best classification possible given features Two classes, PDFs shown Decision rule: C 1 if x ˆx; makes errors on red, green, and blue regions Optimal decision rule: C 1 if x x, Bayes error is area of green and blue regions Intro. to Machine Learning Alireza Ghane 94

95 Bayes Error p(x, C 1 ) x x p(x, C 2 ) x R 1 R 2 Best classification possible given features Two classes, PDFs shown Decision rule: C 1 if x ˆx; makes errors on red, green, and blue regions Optimal decision rule: C 1 if x x, Bayes error is area of green and blue regions Intro. to Machine Learning Alireza Ghane 95

96 Outline Machine Learning: What, Why, and How? Curve Fitting: (e.g.) Regression and Model Selection Decision Theory: ML, Loss Function, MAP Probability Theory: (e.g.) Probabilities and Parameter Estimation Kernel Density Estimation Nearest-neighbour Conclusion Intro. to Machine Learning Alireza Ghane 96

97 Conclusion Readings: Chapter 1.1, 1.3, 1.5, 2.1 Types of learning problems Supervised: regression, classification Unsupervised Learning as optimization Squared error loss function Maximum likelihood (ML) Maximum a posteriori (MAP) Want generalization, avoid over-fitting Cross-validation Regularization Bayesian prior on model parameters Intro. to Machine Learning Alireza Ghane 97

98 Conclusion Readings: Ch. 2.5 Kernel density estimation Model density p(x) using kernels around training datapoints Nearest neighbour Model density or perform classification using nearest training datapoints Multivariate Gaussian Needed for next week s lectures, if you need a refresher read pp Intro. to Machine Learning Alireza Ghane 98