LECTURE 21: Support Vector Machines
|
|
- Claud Golden
- 5 years ago
- Views:
Transcription
1 LECURE 2: Support Vector Maches Emprcal Rsk Mmzato he VC dmeso Structural Rsk Mmzato Maxmum mar hyperplae he Laraa dual problem Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty
2 Itroducto () Cosder the famlar problem of lear a bary classfcato problem from data Assume a ve a dataset (X,Y){(x,y ),(x 2,y 2 ), (x,y )}, here the oal s to lear a fucto yf(x) that ll correctly classfy usee examples Ho do e fd such fucto? By optmz some measure of performace of the leared model What s a ood measure of performace? As e sa Lecture 4, a ood measure s the expected rsk [ ] C( f( x),y) dp( x, y) R f here C(f,y) s a sutable cost fucto, such as the squared error C(f,y)(f(x)-y) 2 Ufortuately, the rsk caot be measured drectly sce the uderly pdf s uko. Istead, e typcally use the rsk over the tra set, also ko as the emprcal rsk R emp [] f C( f( x ), y ) Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 2
3 Itroducto (2) Emprcal Rsk Mmzato A formal term for a smple cocept: fd the fucto f(x) that mmzes the averae rsk o the tra set Mmz the emprcal rsk s ot a bad th to do, provded that suffcet tra data s avalable, sce the la of lare umbers esures that the emprcal rsk ll asymptotcally covere to the expected rsk for Hoever, for small samples, oe caot uaratee that ERM ll also mmze the expected rsk. hs s the all too famlar ssue of eeralzato Ho do e avod overftt? By cotroll model complexty. Itutvely, e should prefer the smplest model that explas the data (Occam s razor) Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Müller et al., 200] 3
4 he VC dmeso () he Vapk-Chervoeks dmeso s a measure of the complexty (or capacty) of a class of fuctos f(α) he VC dmeso measures the larest umber of examples that ca be explaed by the famly f(α) he basc arumet s that hh capacty ad eeralzato propertes are at odds If the famly f(α) has eouh capacty to expla every possble dataset, e should ot expect these fuctos to eeralze very ell O the other had, f fuctos f(α) have small capacty but they are able to expla our partcular dataset, e have stroer reasos to beleve that they ll also ork ell o usee data Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Crsta ad Schölkopf, 2002] 4
5 he VC dmeso (2) Shatter a set of examples Assume a bary classfcato problem th examples R D ad cosder the set of 2 possble dchotomes For stace, th 3 examples, the set of all possble dchotomes s {(000), (00), (00), (0), (00), (0), (0), ()} A class of fuctos f(α) s sad to shatter the dataset f, for every possble dchotomy, there s a fucto f(α) that models t he VC dmeso he VC dmeso VC(f) s the sze of the larest dataset that ca be shattered by the set of fuctos f(α) If the VC dmeso of f(α) s h, the there exsts at least oe set of h pots that ca be shattered by f(α), but eeral t ll ot be true that every set of h pots ca be shattered Oe may eve fd a set of <h pots that caot be shattered by ths set of fuctos Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Mtchel, 997] 5
6 he VC dmeso (3) Cosder a bary classfcato problem R 2, ad let f(α) be the famly of oreted hyperplaes (e.., perceptros) For 3, oe ca perform a lear separato of all pots for every possble class assmet (see examples belo) For 4, a hyperplae caot separate all possble class assmets (e.., cosder the XOR problem) herefore, the VC dmeso of the set of oreted les R 2 s three It ca be sho that the VC dmeso of the famly of oreted separat hyperplaes R D s at least D+ Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 6
7 he VC dmeso (4) he VC dmeso ad the umber of free model parameters Oe may tutvely expect that models th a lare umber of free parameters ould have hh VC dmeso, hereas models th fe parameters ould have lo VC dmesos Couter example Cosder the oe-parameter fucto f(x,α)s(s(αx)), x,α R You choose a arbtrary umber h (as lare as you at) I choose the set of examples x 0 -, h You choose ay labels you lke y, y 2, y h ; x {-,+} I choose α to be h y 0 α π + 2 Despte hav oly oe parameter, the fucto f(x,α) shatters a arbtrarly lare umber of pots chose accord to the outled procedure Ad, at the same tme, oe ca fd four pots that caot be shattered by ths fucto! So hat do e make of ths? ( ) he VC dmeso s a more sophstcated measure of model complexty tha dmesoalty or umber of free parameters [Pardo,2000] Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 7
8 Structural Rsk Mmzato () Why s the VC dmeso relevat? Because the VC dmeso provdes bouds o the expected rsk as a fucto of the emprcal rsk ad the umber of avalable examples It ca be sho that, th probablty -η, the follo boud holds () R () f R f emp ( ( 2/h) + ) l( η/4) h l VC cofdece Eq. () here h s the VC dmeso of f(α), s the umber of tra examples, ad >h As the rato /h ets larer, the VC cofdece becomes smaller ad the actual rsk becomes closer to the emprcal rsk herefore, ths expresso s cosstet th the tuto that ERM s oly sutable he suffcet data s avalable hs ad other results are part of the feld ko as Statstcal Lear heory or Vapk-Chervoeks heory, from hch Support Vector Maches orated Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 8
9 Structural Rsk Mmzato (2) Structural Rsk Mmzato Aother formal term for a tutve cocept: the optmal model s foud by strk a balace betee the emprcal rsk ad the VC dmeso he SRM prcple proceeds as follos Costruct a ested structure for famly of fucto classes F F 2 F k th o-decreas VC dmesos (h h 2 h k ) For each class F, compute the soluto f that mmzes the emprcal rsk Choose the fucto class F, ad the correspod soluto f, that mmzes the rsk boud o the RHS of equato () I other ords ra a set of maches, oe for each subset For a ve subset, tra to mmze the emprcal rsk Choose the mache hose sum of emprcal rsk ad VC cofdece s mmum Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 9
10 Structural Rsk Mmzato (3) Uderftt Overftt Expected rsk VC cofdece Emprcal Rsk VC dmeso S S 2 S k Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Cherkassky ad Muler, 998] 0
11 he VC dmeso practce Ufortuately, comput a upper boud o the expected rsk s ot practcal varous stuatos he VC dmeso caot be accurately estmated for o-lear models such as eural etorks Implemetato of Structural Rsk Mmzato may lead to a o-lear optmzato problem he VC dmeso may be fte (e.., k earest ehbor), requr fte amout of data or he upper boud may sometmes be trval (e.., larer tha oe) Fortuately, Statstcal Lear heory ca be rorously appled the realm of lear models Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Cherkassky ad Muler, 998; Müller et al., 200]
12 Optmal separat hyperplaes () Cosder the problem of fd a separat hyperplae for a learly separable dataset {(x,y ),(x 2,y 2 ),,(x,y )}, x R D, y {-,+} Whch of the fte hyperplaes should e choose? Itutvely, a hyperplae that passes too close to the tra examples ll be sestve to ose ad, therefore, less lkely to eeralze ell for data outsde the tra set Istead, t seems reasoable to expect that a hyperplae that s farthest from all tra examples ll have better eeralzato capabltes herefore, the optmal separat hyperplae ll be the oe th the larest mar, hch s defed as the mmum dstace of a example to the decso surface x 2 x 2 Optmal hyperplae Maxmum mar x x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Cherkassky ad Muler, 998] 2
13 Optmal separat hyperplaes (2) Ho does ths tutve result relate to the VC dmeso? It ca be sho [Vapk, 998] that the VC dmeso of a separat hyperplae th a mar m s bouded as follos R m m h 2,D + here D s the dmesoalty of the put space, ad R s the radus of the smallest sphere cota all the put vectors herefore, by maxmz the mar e are fact mmz the VC dmeso Ad, sce the separat hyperplae has zero emprcal error (t correctly separates all the tra examples), maxmz the mar ll also mmze the upper boud o the expected rsk Cocluso he separat hyperplae th maxmum mar ll also mmze the structural rsk 2 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 3
14 Optmal separat hyperplaes (3) o further uderstad the relatoshp betee mar ad capacty, cosder the to separat hyperplaes depcted belo A sky oe (small mar), hch ll be able to adopt may oretatos A fat oe (lare mar), hch ll have lmted flexblty A larer mar ecessarly results loer capacty We ormally thk of complexty as be a fucto of the umber of parameters Istead, Statstcal Lear heory tells us that f the mar s suffcetly lare, the complexty of the fucto ll be lo eve f the dmesoalty s very hh! x 2 Small mar Lare mar x 2 x x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Beett ad Campbell, 2000] 4
15 Optmal separat hyperplaes (4) Sce e at to maxmze the mar, let s express t as a fucto of the eht vector ad bas of the separat hyperplae From basc troometry, the dstace betee a pot x ad a plae (,b) s x + b otc that the optmal hyperplae has fte solutos by smply scal the eht vector ad bas, e choose the soluto for hch the dscrmat fucto becomes oe for the tra examples closest to the boudary x + b hs s ko as the caocal hyperplae herefore, the dstace from the closest example to the boudary s x 2 2 Ad the mar becomes x + b b x + b m 2 x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 5
16 (Dstace betee a plae ad a pot) x 2 A r θ AB B x A r r r r AB AB AB cosθ AB AB ( x x,x x ) (, ) A x A B b 67 x 2A 8 B 2B x A + b 2 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 6
17 Optmal separat hyperplaes (5) herefore, the problem of maxmz the mar s equvalet to mmze subject to J y ( ) 2 ( x + b) 2 otce that J() s a quadratc fucto, hch meas that there exsts a sle lobal mmum ad o local mma o solve ths problem, e ll use classcal Laraa optmzato techques We frst preset the Kuh-ucker heorem, hch provdes a essetal result for the terpretato of Support Vector Maches Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 7
18 (Kuh-ucker heorem) Gve a optmzato problem th covex doma Ω R mmze subject to f h ( z) ( z) z Ω 0,...,k ( z) 0,..., m th f C covex ad, h affe, ecessary ad suffcet codtos for a ormal pot z* to be a optmum are the exstece of α*, β* such that L L α ( z*,α*,β *) z ( z*,α*,β *) β * ( z *) ( z *) 0 α * ,...,k,...,k,...,k here L k ( z,α,β) f( z) + α ( z) + β h ( z) m L(z,α,β) s ko as a eeralzed Laraa fucto he thrd codto s ko as the Karush-Kuh-ucker (KK) complemetary codto. It mples that for actve costrats α 0; ad for actve costrats α 0 As e ll see a mute, the KK codto allos us to detfy the tra examples that defe the larest mar hyperplae. hese examples ll be ko as Support Vectors. Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Crsta ad Shae-aylor, 2000] 8
19 he Laraa dual problem () Costraed mmzato of J()/2 2 s solved by troduc the Laraa L P 2 (,b,α) - α [ y ( x + b) ] 2 hch yelds a ucostraed optmzato problem that s solved by: mmz L P th respect to the prmal varables ad b, ad maxmz L P th respect to the dual varables α 0 (the Larae multplers) hus, the optmum s defed by a saddle pot (see belo for llustrato) hs s ko as the Laraa prmal problem A saddle pot Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Hayk, 999] 9
20 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 20 he Laraa dual problem (2) o smplfy the prmal problem, e elmate the prmal varables (,b) us the frst Kuh-ucker codto J/ z0 Dfferetat L P (,b,α) th respect to ad b, ad sett to zero yelds Expaso of L P yelds Us the optmalty codto J/ 0, the frst term L P ca be expressed as he secod term L P ca be expressed the same ay he thrd term L P s zero by vrtue of the optmalty codto J/ b0 ( ) ( ) P P 0 α y 0 b,b,α L α y x 0,b,α L ( ) + P α α y b x α y - 2,b,α L j j j j j j j j x x α α y y x x α y α y x α y α y x From [Hayk, 999]
21 he Laraa dual problem (3) Mer these expressos toether e obta L ( α) α j Subject to the (smpler) costrats α 0 ad D 2 α α y y x j j x αy 0 j hs s ko as the Laraa dual problem Commets We have trasformed the problem of fd a saddle pot for L P (,b) to the easer oe of maxmz L D (α) otce that L D (α) depeds o the Larae multplers α, ot o (,b) he prmal problem scales th dmesoalty ( has oe coeffcet for each dmeso), hereas the dual problem scales th the amout of tra data (there s oe Larae multpler per example) Moreover, L D (α) the tra data appears oly as dot products x x j As e ll see the ext lecture, ths property ca be cleverly exploted to perform the classfcato a hher (e.., fte) dmesoal space Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Hayk, 999] 2
22 Support Vectors he K complemetary codto states that, for every pot the tra set, the follo equalty must hold α [ y ( x + b) ] 0... herefore, for each example, ether α 0 or y ( x +b-)0 must hold hose pots for hch α >0 must the le o oe of the to hyperplaes that defe the larest mar (oly at these hyperplaes the term y ( x +b-) becomes zero) hese pots are ko as the Support Vectors All the other pots must have α 0 ote that oly the support vectors cotrbute to def the optmal hyperplae J (,b,α) 0 OE: the bas term b s foud from the KK complemetary codto o the support vectors herefore, the complete dataset could be replaced by oly the support vectors, ad the separat hyperplae ould be the same α y x x 2 Support Vectors (α>0) x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 22
Lecture 7: Linear and quadratic classifiers
Lecture 7: Lear ad quadratc classfers Bayes classfers for ormally dstrbuted classes Case : Σ σ I Case : Σ Σ (Σ daoal Case : Σ Σ (Σ o-daoal Case 4: Σ σ I Case 5: Σ Σ j eeral case Lear ad quadratc classfers:
More informationSupport vector machines
CS 75 Mache Learg Lecture Support vector maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 75 Mache Learg Outle Outle: Algorthms for lear decso boudary Support vector maches Mamum marg hyperplae.
More informationLecture 12: Multilayer perceptrons II
Lecture : Multlayer perceptros II Bayes dscrmats ad MLPs he role of hdde uts A eample Itroducto to Patter Recoto Rcardo Guterrez-Osua Wrht State Uversty Bayes dscrmats ad MLPs ( As we have see throuhout
More informationIntroduction to local (nonparametric) density estimation. methods
Itroducto to local (oparametrc) desty estmato methods A slecture by Yu Lu for ECE 66 Sprg 014 1. Itroducto Ths slecture troduces two local desty estmato methods whch are Parze desty estmato ad k-earest
More informationAn Introduction to. Support Vector Machine
A Itroducto to Support Vector Mache Support Vector Mache (SVM) A classfer derved from statstcal learg theory by Vapk, et al. 99 SVM became famous whe, usg mages as put, t gave accuracy comparable to eural-etwork
More informationSupport vector machines II
CS 75 Mache Learg Lecture Support vector maches II Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Learl separable classes Learl separable classes: here s a hperplae that separates trag staces th o error
More informationCS 1675 Introduction to Machine Learning Lecture 12 Support vector machines
CS 675 Itroducto to Mache Learg Lecture Support vector maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Mdterm eam October 9, 7 I-class eam Closed book Stud materal: Lecture otes Correspodg chapters
More informationBinary classification: Support Vector Machines
CS 57 Itroducto to AI Lecture 6 Bar classfcato: Support Vector Maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 57 Itro to AI Supervsed learg Data: D { D, D,.., D} a set of eamples D, (,,,,,
More informationFeature Selection: Part 2. 1 Greedy Algorithms (continued from the last lecture)
CSE 546: Mache Learg Lecture 6 Feature Selecto: Part 2 Istructor: Sham Kakade Greedy Algorthms (cotued from the last lecture) There are varety of greedy algorthms ad umerous amg covetos for these algorthms.
More informationChapter 4 (Part 1): Non-Parametric Classification (Sections ) Pattern Classification 4.3) Announcements
Aoucemets No-Parametrc Desty Estmato Techques HW assged Most of ths lecture was o the blacboard. These sldes cover the same materal as preseted DHS Bometrcs CSE 90-a Lecture 7 CSE90a Fall 06 CSE90a Fall
More informationTo use adaptive cluster sampling we must first make some definitions of the sampling universe:
8.3 ADAPTIVE SAMPLING Most of the methods dscussed samplg theory are lmted to samplg desgs hch the selecto of the samples ca be doe before the survey, so that oe of the decsos about samplg deped ay ay
More informationTESTS BASED ON MAXIMUM LIKELIHOOD
ESE 5 Toy E. Smth. The Basc Example. TESTS BASED ON MAXIMUM LIKELIHOOD To llustrate the propertes of maxmum lkelhood estmates ad tests, we cosder the smplest possble case of estmatg the mea of the ormal
More informationChapter 14 Logistic Regression Models
Chapter 4 Logstc Regresso Models I the lear regresso model X β + ε, there are two types of varables explaatory varables X, X,, X k ad study varable y These varables ca be measured o a cotuous scale as
More information15-381: Artificial Intelligence. Regression and neural networks (NN)
5-38: Artfcal Itellece Reresso ad eural etorks NN) Mmck the bra I the earl das of AI there as a lot of terest develop models that ca mmc huma thk. Whle o oe ke eactl ho the bra orks ad, eve thouh there
More informationKernel-based Methods and Support Vector Machines
Kerel-based Methods ad Support Vector Maches Larr Holder CptS 570 Mache Learg School of Electrcal Egeerg ad Computer Scece Washgto State Uverst Refereces Muller et al. A Itroducto to Kerel-Based Learg
More informationEconometric Methods. Review of Estimation
Ecoometrc Methods Revew of Estmato Estmatg the populato mea Radom samplg Pot ad terval estmators Lear estmators Ubased estmators Lear Ubased Estmators (LUEs) Effcecy (mmum varace) ad Best Lear Ubased Estmators
More informationNew Schedule. Dec. 8 same same same Oct. 21. ^2 weeks ^1 week ^1 week. Pattern Recognition for Vision
ew Schedule Dec. 8 same same same Oct. ^ weeks ^ week ^ week Fall 004 Patter Recogto for Vso 9.93 Patter Recogto for Vso Classfcato Berd Hesele Fall 004 Overvew Itroducto Lear Dscrmat Aalyss Support Vector
More information{ }{ ( )} (, ) = ( ) ( ) ( ) Chapter 14 Exercises in Sampling Theory. Exercise 1 (Simple random sampling): Solution:
Chapter 4 Exercses Samplg Theory Exercse (Smple radom samplg: Let there be two correlated radom varables X ad A sample of sze s draw from a populato by smple radom samplg wthout replacemet The observed
More informationThe Mathematical Appendix
The Mathematcal Appedx Defto A: If ( Λ, Ω, where ( λ λ λ whch the probablty dstrbutos,,..., Defto A. uppose that ( Λ,,..., s a expermet type, the σ-algebra o λ λ λ are defed s deoted by ( (,,...,, σ Ω.
More informationInvestigating Cellular Automata
Researcher: Taylor Dupuy Advsor: Aaro Wootto Semester: Fall 4 Ivestgatg Cellular Automata A Overvew of Cellular Automata: Cellular Automata are smple computer programs that geerate rows of black ad whte
More informationLECTURE 9: Principal Components Analysis
LECURE 9: Prcpal Compoets Aalss he curse of dmesoalt Dmesoalt reducto Feature selecto vs. feature etracto Sal represetato vs. sal classfcato Prcpal Compoets Aalss Itroducto to Patter Aalss Rcardo Guterrez-Osua
More informationPoint Estimation: definition of estimators
Pot Estmato: defto of estmators Pot estmator: ay fucto W (X,..., X ) of a data sample. The exercse of pot estmato s to use partcular fuctos of the data order to estmate certa ukow populato parameters.
More informationChapter 9 Jordan Block Matrices
Chapter 9 Jorda Block atrces I ths chapter we wll solve the followg problem. Gve a lear operator T fd a bass R of F such that the matrx R (T) s as smple as possble. f course smple s a matter of taste.
More informationSupervised learning: Linear regression Logistic regression
CS 57 Itroducto to AI Lecture 4 Supervsed learg: Lear regresso Logstc regresso Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 57 Itro to AI Data: D { D D.. D D Supervsed learg d a set of eamples s
More informationDimensionality Reduction and Learning
CMSC 35900 (Sprg 009) Large Scale Learg Lecture: 3 Dmesoalty Reducto ad Learg Istructors: Sham Kakade ad Greg Shakharovch L Supervsed Methods ad Dmesoalty Reducto The theme of these two lectures s that
More informationRademacher Complexity. Examples
Algorthmc Foudatos of Learg Lecture 3 Rademacher Complexty. Examples Lecturer: Patrck Rebesch Verso: October 16th 018 3.1 Itroducto I the last lecture we troduced the oto of Rademacher complexty ad showed
More informationA conic cutting surface method for linear-quadraticsemidefinite
A coc cuttg surface method for lear-quadratcsemdefte programmg Mohammad R. Osoorouch Calfora State Uversty Sa Marcos Sa Marcos, CA Jot wor wth Joh E. Mtchell RPI July 3, 2008 Outle: Secod-order coe: defto
More informationUnsupervised Learning and Other Neural Networks
CSE 53 Soft Computg NOT PART OF THE FINAL Usupervsed Learg ad Other Neural Networs Itroducto Mture Destes ad Idetfablty ML Estmates Applcato to Normal Mtures Other Neural Networs Itroducto Prevously, all
More informationLECTURE 2: Linear and quadratic classifiers
LECURE : Lear ad quadratc classfers g Part : Bayesa Decso heory he Lkelhood Rato est Maxmum A Posteror ad Maxmum Lkelhood Dscrmat fuctos g Part : Quadratc classfers Bayes classfers for ormally dstrbuted
More informationUNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Exam: ECON430 Statstcs Date of exam: Frday, December 8, 07 Grades are gve: Jauary 4, 08 Tme for exam: 0900 am 00 oo The problem set covers 5 pages Resources allowed:
More informationSummary of the lecture in Biostatistics
Summary of the lecture Bostatstcs Probablty Desty Fucto For a cotuos radom varable, a probablty desty fucto s a fucto such that: 0 dx a b) b a dx A probablty desty fucto provdes a smple descrpto of the
More informationFunctions of Random Variables
Fuctos of Radom Varables Chapter Fve Fuctos of Radom Varables 5. Itroducto A geeral egeerg aalyss model s show Fg. 5.. The model output (respose) cotas the performaces of a system or product, such as weght,
More information1 Lyapunov Stability Theory
Lyapuov Stablty heory I ths secto we cosder proofs of stablty of equlbra of autoomous systems. hs s stadard theory for olear systems, ad oe of the most mportat tools the aalyss of olear systems. It may
More informationPTAS for Bin-Packing
CS 663: Patter Matchg Algorthms Scrbe: Che Jag /9/00. Itroducto PTAS for B-Packg The B-Packg problem s NP-hard. If we use approxmato algorthms, the B-Packg problem could be solved polyomal tme. For example,
More informationESS Line Fitting
ESS 5 014 17. Le Fttg A very commo problem data aalyss s lookg for relatoshpetwee dfferet parameters ad fttg les or surfaces to data. The smplest example s fttg a straght le ad we wll dscuss that here
More informationLecture 3. Sampling, sampling distributions, and parameter estimation
Lecture 3 Samplg, samplg dstrbutos, ad parameter estmato Samplg Defto Populato s defed as the collecto of all the possble observatos of terest. The collecto of observatos we take from the populato s called
More informationCSE 5526: Introduction to Neural Networks Linear Regression
CSE 556: Itroducto to Neural Netorks Lear Regresso Part II 1 Problem statemet Part II Problem statemet Part II 3 Lear regresso th oe varable Gve a set of N pars of data , appromate d by a lear fucto
More informationECON 5360 Class Notes GMM
ECON 560 Class Notes GMM Geeralzed Method of Momets (GMM) I beg by outlg the classcal method of momets techque (Fsher, 95) ad the proceed to geeralzed method of momets (Hase, 98).. radtoal Method of Momets
More informationbest estimate (mean) for X uncertainty or error in the measurement (systematic, random or statistical) best
Error Aalyss Preamble Wheever a measuremet s made, the result followg from that measuremet s always subject to ucertaty The ucertaty ca be reduced by makg several measuremets of the same quatty or by mprovg
More informationMATH 247/Winter Notes on the adjoint and on normal operators.
MATH 47/Wter 00 Notes o the adjot ad o ormal operators I these otes, V s a fte dmesoal er product space over, wth gve er * product uv, T, S, T, are lear operators o V U, W are subspaces of V Whe we say
More informationCS 2750 Machine Learning. Lecture 8. Linear regression. CS 2750 Machine Learning. Linear regression. is a linear combination of input components x
CS 75 Mache Learg Lecture 8 Lear regresso Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 75 Mache Learg Lear regresso Fucto f : X Y s a lear combato of put compoets f + + + K d d K k - parameters
More informationSolving Constrained Flow-Shop Scheduling. Problems with Three Machines
It J Cotemp Math Sceces, Vol 5, 2010, o 19, 921-929 Solvg Costraed Flow-Shop Schedulg Problems wth Three Maches P Pada ad P Rajedra Departmet of Mathematcs, School of Advaced Sceces, VIT Uversty, Vellore-632
More informationLinear Regression Linear Regression with Shrinkage. Some slides are due to Tommi Jaakkola, MIT AI Lab
Lear Regresso Lear Regresso th Shrkage Some sldes are due to Tomm Jaakkola, MIT AI Lab Itroducto The goal of regresso s to make quattatve real valued predctos o the bass of a vector of features or attrbutes.
More informationCHAPTER 4 RADICAL EXPRESSIONS
6 CHAPTER RADICAL EXPRESSIONS. The th Root of a Real Number A real umber a s called the th root of a real umber b f Thus, for example: s a square root of sce. s also a square root of sce ( ). s a cube
More informationStatistics MINITAB - Lab 5
Statstcs 10010 MINITAB - Lab 5 PART I: The Correlato Coeffcet Qute ofte statstcs we are preseted wth data that suggests that a lear relatoshp exsts betwee two varables. For example the plot below s of
More informationC.11 Bang-bang Control
Itroucto to Cotrol heory Iclug Optmal Cotrol Nguye a e -.5 C. Bag-bag Cotrol. Itroucto hs chapter eals wth the cotrol wth restrctos: s boue a mght well be possble to have scotutes. o llustrate some of
More informationENGI 3423 Simple Linear Regression Page 12-01
ENGI 343 mple Lear Regresso Page - mple Lear Regresso ometmes a expermet s set up where the expermeter has cotrol over the values of oe or more varables X ad measures the resultg values of aother varable
More informationSimple Linear Regression
Statstcal Methods I (EST 75) Page 139 Smple Lear Regresso Smple regresso applcatos are used to ft a model descrbg a lear relatoshp betwee two varables. The aspects of least squares regresso ad correlato
More informationPart 4b Asymptotic Results for MRR2 using PRESS. Recall that the PRESS statistic is a special type of cross validation procedure (see Allen (1971))
art 4b Asymptotc Results for MRR usg RESS Recall that the RESS statstc s a specal type of cross valdato procedure (see Alle (97)) partcular to the regresso problem ad volves fdg Y $,, the estmate at the
More informationSpecial Instructions / Useful Data
JAM 6 Set of all real umbers P A..d. B, p Posso Specal Istructos / Useful Data x,, :,,, x x Probablty of a evet A Idepedetly ad detcally dstrbuted Bomal dstrbuto wth parameters ad p Posso dstrbuto wth
More information7.0 Equality Contraints: Lagrange Multipliers
Systes Optzato 7.0 Equalty Cotrats: Lagrage Multplers Cosder the zato of a o-lear fucto subject to equalty costrats: g f() R ( ) 0 ( ) (7.) where the g ( ) are possbly also olear fuctos, ad < otherwse
More informationX ε ) = 0, or equivalently, lim
Revew for the prevous lecture Cocepts: order statstcs Theorems: Dstrbutos of order statstcs Examples: How to get the dstrbuto of order statstcs Chapter 5 Propertes of a Radom Sample Secto 55 Covergece
More informationOrdinary Least Squares Regression. Simple Regression. Algebra and Assumptions.
Ordary Least Squares egresso. Smple egresso. Algebra ad Assumptos. I ths part of the course we are gog to study a techque for aalysg the lear relatoshp betwee two varables Y ad X. We have pars of observatos
More informationENGI 4421 Joint Probability Distributions Page Joint Probability Distributions [Navidi sections 2.5 and 2.6; Devore sections
ENGI 441 Jot Probablty Dstrbutos Page 7-01 Jot Probablty Dstrbutos [Navd sectos.5 ad.6; Devore sectos 5.1-5.] The jot probablty mass fucto of two dscrete radom quattes, s, P ad p x y x y The margal probablty
More informationLecture 9: Tolerant Testing
Lecture 9: Tolerat Testg Dael Kae Scrbe: Sakeerth Rao Aprl 4, 07 Abstract I ths lecture we prove a quas lear lower boud o the umber of samples eeded to do tolerat testg for L dstace. Tolerat Testg We have
More informationDiscrete Mathematics and Probability Theory Fall 2016 Seshia and Walrand DIS 10b
CS 70 Dscrete Mathematcs ad Probablty Theory Fall 206 Sesha ad Walrad DIS 0b. Wll I Get My Package? Seaky delvery guy of some compay s out delverg packages to customers. Not oly does he had a radom package
More informationBounds on the expected entropy and KL-divergence of sampled multinomial distributions. Brandon C. Roy
Bouds o the expected etropy ad KL-dvergece of sampled multomal dstrbutos Brado C. Roy bcroy@meda.mt.edu Orgal: May 18, 2011 Revsed: Jue 6, 2011 Abstract Iformato theoretc quattes calculated from a sampled
More informationECE 421/599 Electric Energy Systems 7 Optimal Dispatch of Generation. Instructor: Kai Sun Fall 2014
ECE 4/599 Electrc Eergy Systems 7 Optmal Dspatch of Geerato Istructor: Ka Su Fall 04 Backgroud I a practcal power system, the costs of geeratg ad delverg electrcty from power plats are dfferet (due to
More informationMultivariate Transformation of Variables and Maximum Likelihood Estimation
Marquette Uversty Multvarate Trasformato of Varables ad Maxmum Lkelhood Estmato Dael B. Rowe, Ph.D. Assocate Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Copyrght 03 by Marquette Uversty
More information1 Solution to Problem 6.40
1 Soluto to Problem 6.40 (a We wll wrte T τ (X 1,...,X where the X s are..d. wth PDF f(x µ, σ 1 ( x µ σ g, σ where the locato parameter µ s ay real umber ad the scale parameter σ s > 0. Lettg Z X µ σ we
More informationLecture 02: Bounding tail distributions of a random variable
CSCI-B609: A Theorst s Toolkt, Fall 206 Aug 25 Lecture 02: Boudg tal dstrbutos of a radom varable Lecturer: Yua Zhou Scrbe: Yua Xe & Yua Zhou Let us cosder the ubased co flps aga. I.e. let the outcome
More informationCubic Nonpolynomial Spline Approach to the Solution of a Second Order Two-Point Boundary Value Problem
Joural of Amerca Scece ;6( Cubc Nopolyomal Sple Approach to the Soluto of a Secod Order Two-Pot Boudary Value Problem W.K. Zahra, F.A. Abd El-Salam, A.A. El-Sabbagh ad Z.A. ZAk * Departmet of Egeerg athematcs
More informationChapter 5 Properties of a Random Sample
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Revew for the prevous lecture Cocepts: t-dstrbuto, F-dstrbuto Theorems: Dstrbutos of sample mea ad sample varace, relatoshp betwee sample mea ad sample
More informationC-1: Aerodynamics of Airfoils 1 C-2: Aerodynamics of Airfoils 2 C-3: Panel Methods C-4: Thin Airfoil Theory
ROAD MAP... AE301 Aerodyamcs I UNIT C: 2-D Arfols C-1: Aerodyamcs of Arfols 1 C-2: Aerodyamcs of Arfols 2 C-3: Pael Methods C-4: Th Arfol Theory AE301 Aerodyamcs I Ut C-3: Lst of Subects Problem Solutos?
More informationBayes (Naïve or not) Classifiers: Generative Approach
Logstc regresso Bayes (Naïve or ot) Classfers: Geeratve Approach What do we mea by Geeratve approach: Lear p(y), p(x y) ad the apply bayes rule to compute p(y x) for makg predctos Ths s essetally makg
More information3. Basic Concepts: Consequences and Properties
: 3. Basc Cocepts: Cosequeces ad Propertes Markku Jutt Overvew More advaced cosequeces ad propertes of the basc cocepts troduced the prevous lecture are derved. Source The materal s maly based o Sectos.6.8
More informationCIS 800/002 The Algorithmic Foundations of Data Privacy October 13, Lecture 9. Database Update Algorithms: Multiplicative Weights
CIS 800/002 The Algorthmc Foudatos of Data Prvacy October 13, 2011 Lecturer: Aaro Roth Lecture 9 Scrbe: Aaro Roth Database Update Algorthms: Multplcatve Weghts We ll recall aga) some deftos from last tme:
More information( ) 2 2. Multi-Layer Refraction Problem Rafael Espericueta, Bakersfield College, November, 2006
Mult-Layer Refracto Problem Rafael Espercueta, Bakersfeld College, November, 006 Lght travels at dfferet speeds through dfferet meda, but refracts at layer boudares order to traverse the least-tme path.
More informationAssignment 5/MATH 247/Winter Due: Friday, February 19 in class (!) (answers will be posted right after class)
Assgmet 5/MATH 7/Wter 00 Due: Frday, February 9 class (!) (aswers wll be posted rght after class) As usual, there are peces of text, before the questos [], [], themselves. Recall: For the quadratc form
More informationLecture 5: Interpolation. Polynomial interpolation Rational approximation
Lecture 5: Iterpolato olyomal terpolato Ratoal appromato Coeffcets of the polyomal Iterpolato: Sometme we kow the values of a fucto f for a fte set of pots. Yet we wat to evaluate f for other values perhaps
More information6. Nonparametric techniques
6. Noparametrc techques Motvato Problem: how to decde o a sutable model (e.g. whch type of Gaussa) Idea: just use the orgal data (lazy learg) 2 Idea 1: each data pot represets a pece of probablty P(x)
More informationLecture 07: Poles and Zeros
Lecture 07: Poles ad Zeros Defto of poles ad zeros The trasfer fucto provdes a bass for determg mportat system respose characterstcs wthout solvg the complete dfferetal equato. As defed, the trasfer fucto
More information8.1 Hashing Algorithms
CS787: Advaced Algorthms Scrbe: Mayak Maheshwar, Chrs Hrchs Lecturer: Shuch Chawla Topc: Hashg ad NP-Completeess Date: September 21 2007 Prevously we looked at applcatos of radomzed algorthms, ad bega
More informationUNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Postpoed exam: ECON430 Statstcs Date of exam: Jauary 0, 0 Tme for exam: 09:00 a.m. :00 oo The problem set covers 5 pages Resources allowed: All wrtte ad prted
More informationRegression and the LMS Algorithm
CSE 556: Itroducto to Neural Netorks Regresso ad the LMS Algorthm CSE 556: Regresso 1 Problem statemet CSE 556: Regresso Lear regresso th oe varable Gve a set of N pars of data {, d }, appromate d b a
More informationArithmetic Mean and Geometric Mean
Acta Mathematca Ntresa Vol, No, p 43 48 ISSN 453-6083 Arthmetc Mea ad Geometrc Mea Mare Varga a * Peter Mchalča b a Departmet of Mathematcs, Faculty of Natural Sceces, Costate the Phlosopher Uversty Ntra,
More information. The set of these sums. be a partition of [ ab, ]. Consider the sum f( x) f( x 1)
Chapter 7 Fuctos o Bouded Varato. Subject: Real Aalyss Level: M.Sc. Source: Syed Gul Shah (Charma, Departmet o Mathematcs, US Sargodha Collected & Composed by: Atq ur Rehma (atq@mathcty.org, http://www.mathcty.org
More informationBasics of Information Theory: Markku Juntti. Basic concepts and tools 1 Introduction 2 Entropy, relative entropy and mutual information
: Markku Jutt Overvew Te basc cocepts o ormato teory lke etropy mutual ormato ad EP are eeralzed or cotuous-valued radom varables by troduc deretal etropy ource Te materal s maly based o Capter 9 o te
More informationLecture 12 APPROXIMATION OF FIRST ORDER DERIVATIVES
FDM: Appromato of Frst Order Dervatves Lecture APPROXIMATION OF FIRST ORDER DERIVATIVES. INTRODUCTION Covectve term coservato equatos volve frst order dervatves. The smplest possble approach for dscretzato
More informationA tighter lower bound on the circuit size of the hardest Boolean functions
Electroc Colloquum o Computatoal Complexty, Report No. 86 2011) A tghter lower boud o the crcut sze of the hardest Boolea fuctos Masak Yamamoto Abstract I [IPL2005], Fradse ad Mlterse mproved bouds o the
More informationBayes Decision Theory - II
Bayes Decso Theory - II Ke Kreutz-Delgado (Nuo Vascocelos) ECE 175 Wter 2012 - UCSD Nearest Neghbor Classfer We are cosderg supervsed classfcato Nearest Neghbor (NN) Classfer A trag set D = {(x 1,y 1 ),,
More information4 Inner Product Spaces
11.MH1 LINEAR ALGEBRA Summary Notes 4 Ier Product Spaces Ier product s the abstracto to geeral vector spaces of the famlar dea of the scalar product of two vectors or 3. I what follows, keep these key
More informationTHE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA
THE ROYAL STATISTICAL SOCIETY EXAMINATIONS SOLUTIONS GRADUATE DIPLOMA PAPER II STATISTICAL THEORY & METHODS The Socety provdes these solutos to assst caddates preparg for the examatos future years ad for
More informationDecomposition of Hadamard Matrices
Chapter 7 Decomposto of Hadamard Matrces We hae see Chapter that Hadamard s orgal costructo of Hadamard matrces states that the Kroecer product of Hadamard matrces of orders m ad s a Hadamard matrx of
More informationSTATISTICAL INFERENCE
(STATISTICS) STATISTICAL INFERENCE COMPLEMENTARY COURSE B.Sc. MATHEMATICS III SEMESTER ( Admsso) UNIVERSITY OF CALICUT SCHOOL OF DISTANCE EDUCATION CALICUT UNIVERSITY P.O., MALAPPURAM, KERALA, INDIA -
More informationTaylor s Series and Interpolation. Interpolation & Curve-fitting. CIS Interpolation. Basic Scenario. Taylor Series interpolates at a specific
CIS 54 - Iterpolato Roger Crawfs Basc Scearo We are able to prod some fucto, but do ot kow what t really s. Ths gves us a lst of data pots: [x,f ] f(x) f f + x x + August 2, 25 OSU/CIS 54 3 Taylor s Seres
More informationDepartment of Statistics, Banaras Hindu University Varanasi , India 2 Chair of Department of Mathematics, University of New Mexico, Gallup, USA
A Famly of eda Based Estmators Smple Radom Samplg Hemat K.Verma, Rajesh Sgh ad Floret Smaradache Departmet of Statstcs, Baaras Hdu Uversty Varaas-5, Ida har of Departmet of athematcs, Uversty of e exco,
More informationLattices. Mathematical background
Lattces Mathematcal backgroud Lattces : -dmesoal Eucldea space. That s, { T x } x x = (,, ) :,. T T If x= ( x,, x), y = ( y,, y), the xy, = xy (er product of xad y) x = /2 xx, (Eucldea legth or orm of
More information( ) = ( ) ( ) Chapter 13 Asymptotic Theory and Stochastic Regressors. Stochastic regressors model
Chapter 3 Asmptotc Theor ad Stochastc Regressors The ature of eplaator varable s assumed to be o-stochastc or fed repeated samples a regresso aalss Such a assumpto s approprate for those epermets whch
More informationBeam Warming Second-Order Upwind Method
Beam Warmg Secod-Order Upwd Method Petr Valeta Jauary 6, 015 Ths documet s a part of the assessmet work for the subject 1DRP Dfferetal Equatos o Computer lectured o FNSPE CTU Prague. Abstract Ths documet
More informationCHAPTER VI Statistical Analysis of Experimental Data
Chapter VI Statstcal Aalyss of Expermetal Data CHAPTER VI Statstcal Aalyss of Expermetal Data Measuremets do ot lead to a uque value. Ths s a result of the multtude of errors (maly radom errors) that ca
More informationLecture Notes Types of economic variables
Lecture Notes 3 1. Types of ecoomc varables () Cotuous varable takes o a cotuum the sample space, such as all pots o a le or all real umbers Example: GDP, Polluto cocetrato, etc. () Dscrete varables fte
More informationLecture 16: Backpropogation Algorithm Neural Networks with smooth activation functions
CO-511: Learg Theory prg 2017 Lecturer: Ro Lv Lecture 16: Bacpropogato Algorthm Dsclamer: These otes have ot bee subected to the usual scruty reserved for formal publcatos. They may be dstrbuted outsde
More information3D Geometry for Computer Graphics. Lesson 2: PCA & SVD
3D Geometry for Computer Graphcs Lesso 2: PCA & SVD Last week - egedecomposto We wat to lear how the matrx A works: A 2 Last week - egedecomposto If we look at arbtrary vectors, t does t tell us much.
More informationMulti Objective Fuzzy Inventory Model with. Demand Dependent Unit Cost and Lead Time. Constraints A Karush Kuhn Tucker Conditions.
It. Joural of Math. Aalyss, Vol. 8, 204, o. 4, 87-93 HIKARI Ltd, www.m-hkar.com http://dx.do.org/0.2988/jma.204.30252 Mult Objectve Fuzzy Ivetory Model wth Demad Depedet Ut Cost ad Lead Tme Costrats A
More informationA Remark on the Uniform Convergence of Some Sequences of Functions
Advaces Pure Mathematcs 05 5 57-533 Publshed Ole July 05 ScRes. http://www.scrp.org/joural/apm http://dx.do.org/0.436/apm.05.59048 A Remark o the Uform Covergece of Some Sequeces of Fuctos Guy Degla Isttut
More informationMedian as a Weighted Arithmetic Mean of All Sample Observations
Meda as a Weghted Arthmetc Mea of All Sample Observatos SK Mshra Dept. of Ecoomcs NEHU, Shllog (Ida). Itroducto: Iumerably may textbooks Statstcs explctly meto that oe of the weakesses (or propertes) of
More informationAnalysis of Variance with Weibull Data
Aalyss of Varace wth Webull Data Lahaa Watthaacheewaul Abstract I statstcal data aalyss by aalyss of varace, the usual basc assumptos are that the model s addtve ad the errors are radomly, depedetly, ad
More informationPROJECTION PROBLEM FOR REGULAR POLYGONS
Joural of Mathematcal Sceces: Advaces ad Applcatos Volume, Number, 008, Pages 95-50 PROJECTION PROBLEM FOR REGULAR POLYGONS College of Scece Bejg Forestry Uversty Bejg 0008 P. R. Cha e-mal: sl@bjfu.edu.c
More informationLinear regression (cont.) Linear methods for classification
CS 75 Mache Lear Lecture 7 Lear reresso cot. Lear methods for classfcato Mlos Hausrecht mlos@cs.ptt.edu 539 Seott Square CS 75 Mache Lear Coeffcet shrae he least squares estmates ofte have lo bas but hh
More information