Outline. Intro. to Machine Learning. Outline. Course Info. Course Info.: People, References, Resources

Transcription

1 Course Info. Machine Learning Curve Fiing Decision Theory Probabiliy Theory Course Info. Machine Learning Curve Fiing Decision Theory Probabiliy Theory Ouline Course Info.: People, References, Resources Inroducion o Machine Learning Machine Learning: Wha, Why, and How? Bishop PRML Ch. Curve Fiing: (e.g.) Regression and Model Selecion Alireza Ghane Decision Theory: ML, Loss Funcion, MAP Probabiliy Theory: (e.g.) Probabiliies and Parameer Esimaion Inro. o Machine Learning Course Info. Machine Learning Curve Fiing Decision Theory Probabiliy Theory Course Info. Machine Learning Ouline Alireza Ghane / Torsen Mo ller Curve Fiing Decision Theory Probabiliy Theory Course Info. Course Info.: People, References, Resources Machine Learning: Wha, Why, and How? Curve Fiing: (e.g.) Regression and Model Selecion Dr. Torsen Mo ller Decision Theory: ML, Loss Funcion, MAP Probabiliy Theory: (e.g.) Probabiliies and Parameer Esimaion Home: hp://vda.univie.ac.a/teaching/ml/5s Inro. o Machine Learning Discussions: hps://moodle.univie.ac.a/ Alireza Ghane / Torsen Mo ller 2 Inro. o Machine Learning Alireza Ghane / Torsen Mo ller Alireza Ghane 3

2 Regisraion Course ma paricipans: 25 Course Regisered: 6 Number of Seas: 48 Ecess: 3 Sign your name on he shee If you miss he firs wo sessions, you will be auomaically SIGNED OFF he course! References Main Tebook: Paern Recogniion and Machine Learning, Chrisopher M. Bishop, Springer 26. Oher Useful Resources: The Elemens of Saisical Learning, Trevor Hasie, Rober Tibshirani, and Jerome Friedman. Machine Learning, Tom Michel. Paern Classificaion (2nd ed.), Richard O. Duda, Peer E. Har, and David G. Sork. Machine Learning, An Algorihmic Perspecive, Sephen Marsland. The Top Ten Algorihms in Daa Mining, X. Wu, V. Kumar. Learning from Daa, Cherkassky-Mulier. Online Courses: Andrew Ng: hp://ml-class.org/ Inro. o Machine Learning Alireza Ghane / Torsen Möller 4 Inro. o Machine Learning Alireza Ghane / Torsen Möller 5 Grading: Assignmens / Labs ( 5% ) 5 assignmens, % each Final Eam ( 4 % ) Class Feedback ( % ) Assignmen lae policy 5 grace days for all assignmens ogeher afer he grace days, 25% penaly for each day Grading Course Topics We will cover echniques in he sandard ML oolki maimum likelihood, regularizaion, suppor vecor machines (SVM), Fisher s linear discriminan (LDA), boosing, principal componens analysis (PCA), Markov random fields (MRF), neural neworks, graphical models, belief propagaion, epecaion-maimizaion (EM), miure models, miures of epers (MoE), hidden Markov models (HMM), paricle filers, Markov Chain Mone Carlo (MCMC), Gibbs sampling,... Programming wih: MATLAB (licensed): hp://de.mahworks.com/ Ocave (free): hps:// Inro. o Machine Learning Alireza Ghane / Torsen Möller 6 Inro. o Machine Learning Alireza Ghane / Torsen Möller 7

3 Calculus: Background E = mc 2 E c = 2mc Linear algebra (PRML Appendi C): Au i = λ i u i ; Probabiliy (PRML Ch..2): p(x) = p(x, Y ); p() = Y (T a) = a p(, y)dy; E [f] = I will be possible o refresh, bu if you ve never seen hese before his course will be very difficul. p()f()d Inro. o Machine Learning Alireza Ghane / Torsen Möller 8 Ouline Course Info.: People, References, Resources Machine Learning: Wha, Why, and How? Curve Fiing: (e.g.) Regression and Model Selecion Decision Theory: ML, Loss Funcion, MAP Probabiliy Theory: (e.g.) Probabiliies and Parameer Esimaion Inro. o Machine Learning Alireza Ghane / Greg Mori 9 Wha is Machine Learning (ML)? Why ML? Algorihms ha auomaically improve performance hrough eperience Ofen his means define a model by hand, and use daa o fi is parameers The real world is comple difficul o hand-craf soluions. ML is he preferred framework for applicaions in many fields: Compuer Vision Naural Language Processing, Speech Recogniion Roboics... Inro. o Machine Learning Alireza Ghane / Greg Mori Inro. o Machine Learning Alireza Ghane / Greg Mori

4 Hand-wrien Digi Recogniion 58 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 22 Hand-wrien Digi Recogniion 58 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 22 i = i = (,,,,,,,,, ) Represen inpu image as a vecor i R 784. Fig. 8. All of he misclassified MNIST es digis using our mehod 63 ou of,). The e above each digi indicaes he eample number followed by he rue label and he assigned label. Belongie e al. PAMI 22 sraighforward sum of squared differences SSD). SSD error rae wih an average of only four wo-dimensional performs very well on his easy daabase due o he lack of views for each hree-dimensional objec, hanks o he Difficul o hand-craf rules abou digis variaion in lighing [24] PCA jus makes i faser). fleibiliy provided by he maching algorihm. The prooype selecion algorihm is illusraed in Fig MPEG-7 Shape Silhouee Daabase As seen, views are allocaed mainly for more comple caegories wih high wihin class variabiliy. The curve Our ne eperimen involves he MPEG-7 shape silhouee Inro. o Machine Learning Alireza Ghane / Greg Mori 2 marked SC-proo in Fig. 9 shows he improved classificaion daabase, specifically Core Eperimen CE-Shape- par B, performance using his prooype selecion sraegy insead which measures performance of similariy-based rerieval of equally-spaced views. Noe ha we obain a 2.4 percen [25]. The daabase consiss of,4 images: 7 shape caegories, 2 images per caegory. The performance is measured using he so-called ªbullseye es,º in which each Suppose we have a arge vecor i This is supervised learning Discree, finie label se: perhaps i {, }, a classificaion problem Given a raining se {(, ),..., ( N, N )}, learning problem is o consruc a good funcion y() from hese. y : R 784 R Inro. o Machine Learning Alireza Ghane / Greg Mori 3 Fig. 9. 3D objec recogniion using he COIL-2 daa se. Comparison of es se error for SSD, Shape Disance SD), and Shape Disance wih k-medoids prooypes SD-proo) versus number of prooype views. For SSD and SD, we varied he number of prooypes uniformly for all objecs. For SD-proo, he number of prooypes per objec depended on he wihin-objec variaion as well as he beween-objec similariy. Fig.. Prooype views seleced for wo differen 3D objecs from he COIL daa se using he algorihm described in Secion 5.2. Wih his approach, views are allocaed adapively depending on he visual compleiy of an objec wih respec o viewing angle. Fig. 8. All of he misclassified MNIST es digis using our mehod 63 ou of,). The e above each digi indicaes he eample number followed by he rue label and he assigned label. sraighforward sum of squared differences SSD). SSD performs very well on his easy daabase due o he lack of variaion in lighing [24] PCA jus makes i faser). The prooype selecion algorihm is illusraed in Fig.. As seen, views are allocaed mainly for more comple caegories wih high wihin class variabiliy. The curve marked SC-proo in Face Fig. 9Deecion shows he improved classificaion performance using his prooype selecion sraegy insead of equally-spaced views. Noe ha we obain a 2.4 percen error rae wih an average of only four wo-dimensional views for each hree-dimensional objec, hanks o he fleibiliy provided by he maching algorihm. 6.3 MPEG-7 Shape Silhouee Daabase Our ne eperimen involves he MPEG-7 shape silhouee daabase, specifically Core Eperimen CE-Shape- par B, Spam Deecion which measures performance of similariy-based rerieval [25]. The daabase consiss of,4 images: 7 shape caegories, 2 images per caegory. The performance is measured using he so-called ªbullseye es,º in which each Classificaion problem Schneiderman and Kanade, IJCV 22 i {, }, non-spam, spam Classificaion problem i couns of words, e.g. Viagra, sock, ouperform, Fig. 9. 3D objec recogniion using he COIL-2 daa se. Comparison of i {,, 2}, non-face, fronal face, profile face. es se error for SSD, Shape Disance SD), and Shape Disance wih k-medoids prooypes SD-proo) versus number of prooype views. For SSD and SD, we varied he number of prooypes uniformly for all muli-bagger Fig.. Prooype views seleced for wo differen 3D objecs from he COIL daa se using he algorihm described in Secion 5.2. Wih his Inro. o Machine Learning Alireza Ghane / Greg Mori 4 Inro. o Machine Learning Alireza Ghane / Greg Mori objecs. For SD-proo, he number of prooypes per objec depended on approach, views are allocaed adapively depending on he visual 5 he wihin-objec variaion as well as he beween-objec similariy. compleiy of an objec wih respec o viewing angle.

5 Cavea - Horses (source?) Once upon a ime here were wo neighboring farmers, Jed and Ned. Each owned a horse, and he horses boh liked o jump he fence beween he wo farms. Clearly he farmers needed some means o ell whose horse was whose. So Jed and Ned go ogeher and agreed on a scheme for discriminaing beween horses. Jed would cu a small noch in one ear of his horse. No a big, painful noch, bu jus big enough o be seen. Well, wouldn you know i, he day afer Jed cu he noch in horse s ear, Ned s horse caugh on he barbed wire fence and ore his ear he eac same way! Somehing else had o be devised, so Jed ied a big blue bow on he ail of his horse. Bu he ne day, Jed s horse jumped he fence, ran ino he field where Ned s horse was grazing, and chewed he bow righ off he oher horse s ail. Ae he whole bow! Inro. o Machine Learning Alireza Ghane / Greg Mori 6 Cavea - Horses (source?) Finally, Jed suggesed, and Ned concurred, ha hey should pick a feaure ha was less ap o change. Heigh seemed like a good feaure o use. Bu were he heighs differen? Well, each farmer wen and measured his horse, and do you know wha? The brown horse was a full inch aller han he whie one! Moral of he sory: ML provides heory and ools for seing parameers. Make sure you have he righ model and feaures. Think abou your feaure vecor. Inro. o Machine Learning Alireza Ghane / Greg Mori 7 Sock Price Predicion Clusering Images Problems in which i is coninuous are called regression E.g. i is sock price, i conains company profi, deb, cash flow, gross sales, number of spam s sen,... Inro. o Machine Learning Alireza Ghane / Greg Mori 8 Wang e al., CVPR 26 Only i is defined: unsupervised learning E.g. i describes image, find groups of similar images Inro. o Machine Learning Alireza Ghane / Greg Mori 9

6 Types of Learning Problems Ouline Course Info.: People, References, Resources Supervised Learning Classificaion Regression Unsupervised Learning Densiy esimaion Clusering: k-means, miure models, hierarchical clusering Hidden Markov models Reinforcemen Learning Machine Learning: Wha, Why, and How? Curve Fiing: (e.g.) Regression and Model Selecion Decision Theory: ML, Loss Funcion, MAP Probabiliy Theory: (e.g.) Probabiliies and Parameer Esimaion Inro. o Machine Learning Alireza Ghane / Greg Mori 2 Inro. o Machine Learning Alireza Ghane / Greg Mori 2 An Eample - Polynomial Curve Fiing Polynomial Curve Fiing Suppose we are given raining se of N observaions (,..., N ) and (,..., N ), i, i R Regression problem, esimae y() from hese daa Wha form is y()? Le s ry polynomials of degree M: y(, w) = w +w +w w M M This is he hypohesis space. How do we measure success? Sum of squared errors: E(w) = 2 N {y( n, w) n } 2 n= Among funcions in he class, choose ha which minimizes his error n y(n, w) n Inro. o Machine Learning Alireza Ghane / Greg Mori 22 Inro. o Machine Learning Alireza Ghane / Greg Mori 23

7 Polynomial Curve Fiing Which Degree of Polynomial? Error funcion Bes coefficiens E(w) = 2 N {y( n, w) n } 2 n= w = arg min E(w) w Found using pseudo-inverse (more laer) Inro. o Machine Learning Alireza Ghane / Greg Mori 24 A model selecion problem M = 9 E(w ) = : This is over-fiing Inro. o Machine Learning Alireza Ghane / Greg Mori 25 Generalizaion Conrolling Over-fiing: Regularizaion Training Tes Generalizaion is he holy grail of ML Wan good performance for new daa Measure generalizaion using a separae se Use roo-mean-squared (RMS) error: E RMS = 2E(w )/N As order of polynomial M increases, so do coefficien magniudes Penalize large coefficiens in error funcion: Ẽ(w) = 2 N {y( n, w) n } 2 + λ 2 w 2 n= Inro. o Machine Learning Alireza Ghane / Greg Mori 26 Inro. o Machine Learning Alireza Ghane / Greg Mori 27

8 Conrolling Over-fiing: Regularizaion Conrolling Over-fiing: Regularizaion Training Tes Noe he E RMS for he raining se. Perfec mach of raining se wih he model is a resul of over-fiing Training and es error show similar rend Inro. o Machine Learning Alireza Ghane / Greg Mori 28 Inro. o Machine Learning Alireza Ghane / Greg Mori 29 Over-fiing: Daase size Validaion Se Spli raining daa ino raining se and validaion se Train differen models (e.g. diff. order polynomials) on raining se Choose model (e.g. order of polynomial) wih minimum error on validaion se Wih more daa, more comple model (M = 9) can be fi Rule of humb: daapoins for each parameer Inro. o Machine Learning Alireza Ghane / Greg Mori 3 Inro. o Machine Learning Alireza Ghane / Greg Mori 3

9 Cross-validaion Summary run run 2 run 3 run 4 Daa are ofen limied Cross-validaion creaes S groups of daa, use S o rain, oher o validae Ereme case leave-one-ou cross-validaion (LOO-CV): S is number of raining daa poins Cross-validaion is an effecive mehod for model selecion, bu can be slow Models wih muliple compleiy parameers: eponenial number of runs Wan models ha generalize o new daa Train model on raining se Measure performance on held-ou es se Performance on es se is good esimae of performance on new daa Inro. o Machine Learning Alireza Ghane / Greg Mori 32 Inro. o Machine Learning Alireza Ghane / Greg Mori 33 Summary - Model Selecion Summary - Soluions I Which model o use? E.g. which degree polynomial? Training se error is lower wih more comple model Can jus choose he model wih lowes raining error Peeking a es error is unfair. E.g. picking polynomial wih lowes es error Performance on es se is no longer good esimae of performance on new daa Use a validaion se Train models on raining se. E.g. differen degree polynomials Measure performance on held-ou validaion se Measure performance of ha model on held-ou es se Can use cross-validaion on raining se insead of a separae validaion se if lile daa and los of ime Choose model wih lowes error over all cross-validaion folds (e.g. polynomial degree) Rerain ha model using all raining daa (e.g. polynomial coefficiens) Inro. o Machine Learning Alireza Ghane / Greg Mori 34 Inro. o Machine Learning Alireza Ghane / Greg Mori 35

10 Summary - Soluions II Ouline Course Info.: People, References, Resources Use regularizaion Train comple model (e.g high order polynomial) bu penalize being oo comple (e.g. large weigh magniudes) Need o balance error vs. regularizaion (λ) Choose λ using cross-validaion Ge more daa Machine Learning: Wha, Why, and How? Curve Fiing: (e.g.) Regression and Model Selecion Decision Theory: ML, Loss Funcion, MAP Probabiliy Theory: (e.g.) Probabiliies and Parameer Esimaion Inro. o Machine Learning Alireza Ghane / Greg Mori 36 Decision Theory Decision: Maimum Likelihood For a sample, decide which class(c k ) i is from. Ideas: Maimum Likelihood Minimum Loss/Cos (e.g. misclassificaion rae) Maimum Aposeriori (MAP) Inference sep: Deermine saisics from raining daa. p(, ) OR p( C k ) Decision sep: Deermine opimal for es inpu : = arg ma{ p ( C k ) k }{{} Likelihood } Inro. o Machine Learning Alireza Ghane 38 Inro. o Machine Learning Alireza Ghane 39

11 Decision: Minimum Misclassificaion Rae p(misake) = p ( R, C 2 ) + p ( R 2, C ) = R p (, C 2 ) d + R 2 p (, C ) d p(, C) R p(misake) = k p(, C2) R2 j R j p (, C k ) d ˆ: decision boundary. : opimal decision boundary : arg min{p (misake)} R Decision: Minimum Loss/Cos Misclassificaion rae: R : arg min {R i i {,,K}} L (R j, C k ) Weighed loss/cos funcion: R : arg min W j,k L (R j, C k ) {R i i {,,K}} Is useful when: The populaion of he classes are differen The failure cos is non-symmeric k k j j Inro. o Machine Learning Alireza Ghane 4 Inro. o Machine Learning Alireza Ghane 4 Decision: Maimum Aposeriori (MAP) Ouline Bayes Theorem: P {A B} = P {B A}P {A} P {B} Course Info.: People, References, Resources Machine Learning: Wha, Why, and How? p(c k ) }{{} P oserior p( C k ) }{{} p(c k ) }{{} Likelihood P rior Provides an Aposeriori Belief for he esimaion, raher han a single poin esimae. Can uilize Apriori Informaion in he decision. Curve Fiing: (e.g.) Regression and Model Selecion Decision Theory: ML, Loss Funcion, MAP Probabiliy Theory: (e.g.) Probabiliies and Parameer Esimaion Inro. o Machine Learning Alireza Ghane 42

12 Coin Tossing Coin Tossing - Model Le s say you re given a coin, and you wan o find ou P (heads), he probabiliy ha if you flip i i lands as heads. Flip i a few imes: H H T P (heads) = 2/3 Hmm... is his rigorous? Does his make sense? Bernoulli disribuion P (heads) = µ, P (ails) = µ Assume coin flips are independen and idenically disribued (i.i.d.) i.e. All are separae samples from he Bernoulli disribuion Given daa D = {,..., N }, heads: i =, ails: i =, he likelihood of he daa is: N N p(d µ) = p( n µ) = µ n ( µ) n n= n= Inro. o Machine Learning Alireza Ghane / Greg Mori 44 Inro. o Machine Learning Alireza Ghane / Greg Mori 45 Maimum Likelihood Esimaion Given D wih h heads and ails Wha should µ be? Maimum Likelihood Esimaion (MLE): choose µ which maimizes he likelihood of he daa µ ML = arg ma µ p(d µ) Since ln( ) is monoone increasing: µ ML = arg ma ln p(d µ) µ Inro. o Machine Learning Alireza Ghane / Greg Mori 46 Likelihood: Maimum Likelihood Esimaion Log-likelihood: ln p(d µ) = p(d µ) = N µ n ( µ) n n= N n ln µ + ( n ) ln( µ) n= Take derivaive, se o : d N dµ ln p(d µ) = n µ ( n) µ = µ h µ n= µ = h + h Inro. o Machine Learning Alireza Ghane / Greg Mori 47

13 Bayesian Learning Wai, does his make sense? Wha if I flip ime, heads? Do I believe µ=? Learn µ he Bayesian way: P (µ D) = P (µ D) }{{} poserior P (D µ)p (µ) P (D) P (D µ) P (µ) }{{}}{{} prior likelihood Bea Disribuion We will use he Bea disribuion o epress our prior knowledge abou coins: Bea(µ a, b) = Γ(a + b) µ a ( µ) b Γ(a)Γ(b) }{{} normalizaion Parameers a and b conrol he shape of his disribuion Prior encodes knowledge ha mos coins are 5-5 Conjugae prior makes mah simpler, easy inerpreaion For Bernoulli, he bea disribuion is is conjugae Inro. o Machine Learning Alireza Ghane / Greg Mori 48 Inro. o Machine Learning Alireza Ghane / Greg Mori 49 Poserior Maimum A Poseriori P (µ D) P (D µ)p (µ) N µ n ( µ) n µ a ( µ) b }{{} n= }{{} prior likelihood µ h ( µ) µ a ( µ) b µ h+a ( µ) +b Simple form for poserior is due o use of conjugae prior Parameers a and b ac as era observaions Noe ha as N = h +, prior is ignored Given poserior P (µ D) we could compue a single value, known as he Maimum a Poseriori (MAP) esimae for µ: µ MAP = arg ma µ P (µ D) Known as poin esimaion However, correc Bayesian hing o do is o use he full disribuion over µ i.e. Compue E µ [f] = p(µ D)f(µ)dµ This inegral is usually hard o compue Inro. o Machine Learning Alireza Ghane / Greg Mori 5 Inro. o Machine Learning Alireza Ghane / Greg Mori 5

14 Polynomial Curve Fiing: Wha We Did Curve Fiing: Probabilisic Approach Wha form is y()? Le s ry polynomials of degree M: y(, w) y(, w) = w +w +w w M M This is he hypohesis space. How do we measure success? Sum of squared errors: n y(, w) p(, w, β) 2σ y(, w) p(, w, β) y(, w) 2σ E(w) = 2 N {y( n, w) n } 2 n= Among funcions in he class, choose ha which minimizes his error n y(n, w) Inro. o Machine Learning Alireza Ghane 52 p(, w, β) = N N ( n y( n, w), β ) n= ln (p(, w, β)) = β N {y( n, w) n } 2 + N 2 2 ln β N ln (2π) Inro. o Machine Learning Alireza Ghane n= }{{}} 2 {{ 53}}{{} cons. cons. βe(w) Maimize log-likelihood Minimize E(w). Can opimize for β as well. Curve Fiing: Bayesian Approach Curve Fiing: Bayesian y(, w) p (, w, β, α) = N ( P w ( ), Q w,β,α ( )) y(, w) y(, w) p(, w, β) = p(, w, β) 2σ y(, w) N N ( n y( n, w), β ) n= Poserior Dis.:p (w,, α, β) p (, w, β) p ( α) Minimize: β N {y( n, w) n } 2 + α 2 2 wt w n= }{{}}{{} regularizaion. βe(w) Inro. o Machine Learning Alireza Ghane 54 p(, w, β) y(, w) 2σ y(, w) p (,, ) = N ( m(), s 2 () ) y(, w) 2σ m() p(, = w, φ() T β) S N φ( n ) n n= s 2 () = β ( + φ() T Sφ() ) S = α N β I + φ( n )φ( n ) T y(, w) 2σ p(, w, β) n= Inro. o Machine Learning Alireza Ghane 55

15 Readings: Chaper.,.3,.5, 2. Types of learning problems Supervised: regression, classificaion Unsupervised Learning as opimizaion Squared error loss funcion Maimum likelihood (ML) Maimum a poseriori (MAP) Wan generalizaion, avoid over-fiing Cross-validaion Regularizaion Bayesian prior on model parameers Inro. o Machine Learning Alireza Ghane 56