CS 2750 Machine Learning. Lecture 23. Concept learning. CS 2750 Machine Learning. Concept Learning
|
|
|
- Bernice Crawford
- 6 years ago
- Views:
Transcription
1 Lecture 3 Cocept learig Milos Hauskrecht milos@cs.pitt.edu Cocept Learig Outlie: Learig boolea fuctios Most geeral ad most specific cosistet hypothesis. Mitchell s versio space algorithm Probably approximately correct (PAC) learig. Sample complexity for PAC. VapikChervoekis (VC) dimesio. Improved sample complexity bouds. 1
2 Learig cocepts Assume objects (examples) described i terms of attributes: Sky AirTemp Humidity Wid Water Forecast EjoySport Suy Warm Normal Strog Warm Same yes Raiy Cold Normal Strog Warm Chage o Cocept = a set of objects Cocept learig: Give a sample of labeled objects we wat to lear a boolea mappig from objects to T/F idetifyig a uderlyig cocept E.g. EjoySport cocept Cocept (hypothesis) space H Restrictio o the boolea descriptio of cocepts Learig cocepts Object (istace) space X Cocept (hypothesis) spaces H H X!!!! Assume biary attributes (e.g. true/false, warm/cold) Istace space X: differet objects Cocept space H: possible cocepts = all possible subsets of objects
3 Learig cocepts Problem: Cocept space too large Solutio: restricted hypothesis space H Example: cojuctive cocepts ( Sky Suy ) (Weather Cold ) 3 possible cocepts Why? Other restricted spaces: 3CNF (or kcnf) 3DNF (or kdnf) ( 7 a1 a3 a ) (...) ( 9 a1 a5 a ) (...) Learig cocepts After seeig k examples the hypothesis space (eve if restricted) ca have may cosistet cocept hypotheses Cosistet hypothesis: a cocept c that evaluates to T o all positive examples ad to F o all egatives. What to lear? Geeral to specific learig. Start from all true ad refie with the maximal (cosistet) geeralizatio. Specific to geeral learig. Start from all false ad refie with the most restrictive specializatio. Versio space learig. Keep all cosistet hypothesis aroud the combiatio of the above two cases. 3
4 Specific to geeral learig (for cojuctive cocepts) Assume two hypotheses: h1 ( Suy,?,? Strog h ( Suy,?,?,?,?,?) The we say that:,?,?) h is more geeral tha h1, h1 is a special case (specializatio of) h arbitrary Specific to geeral learig: start from the allfalse hypothesis h0 (,,,,, ) by scaig samples, gradually refie the hypothesis (make it more geeral) wheever it does ot satisfy the ew sample see (keep the most restrictive specializatio of positives) Specific to geeral learig. Example Cojuctive cocepts, target is a cojuctive cocept h (,,,,, ) All false (Suy,Warm, Normal, Strog, Warm, Same) T h ( Suy, Warm, Normal, Strog, Warm, Same ) (Raiy, Cold, Normal, Strog, Warm, Chage) F h ( Suy, Warm, Normal, Strog, Warm, Same ) (Suy,Warm, High, Strog, Warm, Same) T h ( Suy, Warm,?, Strog, Warm, Same ) (Suy,Warm, High, Strog, Cool, Same) T h ( Suy, Warm,?, Strog,?, Same ) 4
5 Geeral to specific learig Dual problem to the specific to geeral learig Start from the all true hypothesis h0 (?,?,?,?,?,?) Refie the cocept descriptio such that all samples are cosistet (keep maximal possible geeralizatio) h (?,?,?,?,?,?) (Suy,Warm, Normal, Strog, Warm, Same) T h (?,?,?,?,?,?) (Suy,Warm, High, Strog, Warm, Same) T h (?,?,?,?,?,?) (Raiy, Cold, Normal, Strog, Warm, Chage) F h ( Suy,?,?,?,?,?), (?,?,?,?,?, (?, Warm Same )?,?,?,?), Mitchell s versio space algorithm Keeps the space of cosistet hypotheses Most geeral rule Upper boud (frige) Pushed dow by examples Versio space Lower boud (frige) Pushed up by examples Most specific rule 5
6 Mitchell s versio space algorithm Keeps ad refies the friges of the versio space Coverges to the target cocept wheever the target is a member of the hypotheses space H Assumptio: No oise i the data samples (the same example has always the same label) The hope is that the frige is always small Is this correct? Expoetial frige set example Cojuctive cocepts, upper frige (geeral to specific) Samples: ( true, true, true, true,..., true ) T 1 ( false, false, true, true,..., true ) ( true, true, false, false,..., true )... ( true, true, true,..., false, false ) Maximal geeralizatios differet hypotheses we eed to remember ( true,?, true,?,..., true,?) (?, true, true,?,..., true,?) ( true,?,?, true,..., true,?)... (?, true,?, true,...,?, true ) F F F 6
7 Learig cocepts Versio space algorithm may require large umber of samples to coverge to the target cocept I the worst case we must see all cocepts before covergig to it. The samples may come from differet distributios it may take a very log time to see all examples The frige ca go expoetial i the umber of attributes Alterative solutio: Select a hypothesis that is cosistet after some umber of (, ) samples is see by our algorithm Ca we tell how far are we from the solutio? Yes!!! PAC framework develops the criteria for measurig the accuracy of our choice i probabilistic terms Valiat s framework Probability distributio from which samples are draw There is a error permitted i assigig the labels to examples The cocept leared does ot have to be perfect but it should ot be very far from the target cocept ct target cocept c leared cocept x ext sample from the distributio Error ( ct, c) P( x c x ct ) P( x c x ct ) accuracy parameter We would like to have cocept such that Error ( c, c) T 7
8 PAC learig To get the error to be smaller tha the accuracy parameter i all cases may be hard: Some examples may be very rare ad to see them may require large umber of samples Istead we choose: where P( Error ( c, c) ) 1 T is a cofidece factor Probably approximately correct (PAC) learig With probability 1 a cocept with a error ot more tha is foud Sample complexity of PAC learig How may samples we eed to see to satisfy PAC criterio? Assume: we saw m idepedet samples draw from the distributio, ad h is a hypothesis that is cosistet with all m examples ad its error is larger tha epsilo Error ( c, h) P(a sample is cosistet with a give h) (1 ) P ( m samples are cosistet with a give h) (1 ) There are at most H hypotheses i the space P ( ay bad hypothesis survives m samples) H (1 ) T m m 8
9 Sample complexity of PAC learig P ( ay bad hypothesis survives m samples) H (1 ) H e m I the PAC framework we wat to boud this probability with the cofidece factor H Expressig for m e m (l( 1 / ) l H ) m After m samples satisfyig the above iequality ay cosistet hypothesis satisfies the PAC criterio m Efficiet PAC learability The cocept is efficietly PAC learable if the time it takes to output the cocept is polyomial i, 1 /, 1 / Two aspects: Sample complexity a umber of examples eeded to lear the cocept satisfyig PAC criterio A prerequisite to efficiet PAC learability Time complexity the time it takes to fid the cocept Eve if the sample complexity is OK, the learig procedure may ot be efficiet (e.g. expoetial frige) 9
10 Efficiet PAC learability Sample complexities depeds o the hypothesis space we use Cojuctive cocepts 3 possible cocepts m (l( 1 / ) l 3 ) (l(1 / ) l 3) efficiet All possible cocepts (ubiased hypothesis space) m (l(1 / ) l ) (l(1 / ) l ) iefficiet Efficiet PAC learability Polyomial sample complexity is ecessary but ot sufficiet Algorithm should work i polyomial time Some types of cocept (hypothesis) ca be leared efficietly. Example: cojuctive cocepts Specific to geeral learig. Keeps oe hypothesis aroud. The most specific descriptio of all positive examples. Ca be doe i poly time. Geeral to specific learig. We eed to keep the complete upper frige which ca be expoetial. Caot be doe i poly time. Other cocept (hypothesis) spaces with poly sample complexity: kdnf caot be PAC leared i poly time. kcnf polyomial time solutio 10
11 Learig cojuctive cocepts Learig cojuctive cocepts specific to geeral learig It is sufficiet to keep oe hypothesis aroud which is the most specific descriptio of all positive examples. Ca be doe i poly time. How? Iitial hypothesis: all false a1 a1 a a... a k a k Whe positive imstace is see we remove icosistet terms from the cojuctio: Positive istace: a, 1 a,... a k Hypothesis: a1 a1 a a... a k a k We keep doig this for m steps Learig 3CNF Sample complexity for the kcnf ad kdnf kdnf caot be leared efficietly kcnf ca be leared efficietly. How? Assume 3CNF ( a1 a3 a7 ) ( a a 4 a5 )... Oly a polyomial umber of clauses with at most 3 variables!! 3 ( 1) ( 1)( ) O ( ) Algorithm (specific to geeral learig): Start with the cojuctio of all possible clauses (always false) O positive example ay clause that is ot true is deleted O egative examples do othig Iterestig Ay kdnf ca be coverted ito kcnf 11
12 Quatifyig iductive bias Durig learig oly small fractio of samples see We eed to geeralize to usee examples Choice of the hypotheses space restrict our learig optios biases our learig Other biases: preferece towards simpler hypothesis, smaller degrees of freedom Questios: How to measure the bias? To what extet our biases affect our learig capabilities? Ca we lear eve if the hypotheses space is ifiite? (l( 1 / ) l H ) m VapikChervoekis dimesio Measures the biases of the cocept space Allows us to: Obtai better sample complexity boud Ca be exteded to attributes with ifiite value spaces. VC idea: do ot measure the size of the space, but the umber of distict istaces that ca be completely discrimiated usig H Example: H is a set of space of rectagles Discrimiatio of labeligs of 3 poits with rectagles 1
13 Shatterig of a set of istaces A set of istaces S X H shatters S if for every dichotomy (combiatio of labels) there is a hypothesis h cosistet with the dichotomy Example: H is a set of space of rectagles A set of 3 istaces (most flexible choice) Dichotomy 1 Dichotomy Dichotomy k 3 differet dichotomies, hypothesis for each of them VapikChervoekis dimesio VC dimesio of a hypothesis space H is the size of the largest subset of istaces that is shattered by H. Example: rectagles (VC at least 3) Try 4: Ca be shattered (for the most flexible 4), VC dimesio at least 4 Try 5: No set of 5 poits that ca be shattered, thus VC dimesio is 4 13
14 VC dimesio ad sample complexity Oe ca derive the sample complexity boud for PAC learig usig VC dimesio istead of hypothesis space size (we wo t do it here) m ( 4 l( / ) 8VC dim( H ) l(13 / )) Addig oise We have a target cocept but there is a chace of mislabelig the examples see Ca we PAClear also i this case? Blumer (1986). If h is a hypothesis that agrees with at least 1 m l( ) samples draw from the distributio the P( error ( h, ct ) ) Mitchell gives the sample complexity boud for the choice of the hypothesis with the best traiig error 14
15 Summary Learig boolea fuctios Most geeral ad most specific cosistet hypothesis. Mitchell s versio space algorithm Probably approximately correct (PAC) learig. Sample complexity for PAC. VapikChervoekis (VC) dimesio. Improved sample complexity bouds. Addig oise. 15
CS 2750 Machine Learning. Lecture 22. Concept learning. CS 2750 Machine Learning. Concept Learning
Lecture 22 Cocept learig Milos Hauskrecht milos@cs.pitt.edu 5329 Seott Square Cocept Learig Outlie: Learig boolea fuctios Most geeral ad most specific cosistet hypothesis. Mitchell s versio space algorithm
10-701/ Machine Learning Mid-term Exam Solution
0-70/5-78 Machie Learig Mid-term Exam Solutio Your Name: Your Adrew ID: True or False (Give oe setece explaatio) (20%). (F) For a cotiuous radom variable x ad its probability distributio fuctio p(x), it
Intro to Learning Theory
Lecture 1, October 18, 2016 Itro to Learig Theory Ruth Urer 1 Machie Learig ad Learig Theory Comig soo 2 Formal Framework 21 Basic otios I our formal model for machie learig, the istaces to be classified
Lecture 9: Boosting. Akshay Krishnamurthy October 3, 2017
Lecture 9: Boostig Akshay Krishamurthy akshay@csumassedu October 3, 07 Recap Last week we discussed some algorithmic aspects of machie learig We saw oe very powerful family of learig algorithms, amely
Empirical Process Theory and Oracle Inequalities
Stat 928: Statistical Learig Theory Lecture: 10 Empirical Process Theory ad Oracle Iequalities Istructor: Sham Kakade 1 Risk vs Risk See Lecture 0 for a discussio o termiology. 2 The Uio Boud / Boferoi
Frequentist Inference
Frequetist Iferece The topics of the ext three sectios are useful applicatios of the Cetral Limit Theorem. Without kowig aythig about the uderlyig distributio of a sequece of radom variables {X i }, for
Sequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
Statistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.
Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized
Hypothesis Testing. Evaluation of Performance of Learned h. Issues. Trade-off Between Bias and Variance
Hypothesis Testig Empirically evaluatig accuracy of hypotheses: importat activity i ML. Three questios: Give observed accuracy over a sample set, how well does this estimate apply over additioal samples?
Roberto s Notes on Series Chapter 2: Convergence tests Section 7. Alternating series
Roberto s Notes o Series Chapter 2: Covergece tests Sectio 7 Alteratig series What you eed to kow already: All basic covergece tests for evetually positive series. What you ca lear here: A test for series
Properties and Tests of Zeros of Polynomial Functions
Properties ad Tests of Zeros of Polyomial Fuctios The Remaider ad Factor Theorems: Sythetic divisio ca be used to fid the values of polyomials i a sometimes easier way tha substitutio. This is show by
Uncertainty. Variables. assigns to each sentence numerical degree of belief between 0 and 1. uncertainty
Bayes Classificatio Ucertaity & robability Baye's rule Choosig Hypotheses- Maximum a posteriori Maximum Likelihood - Baye's cocept learig Maximum Likelihood of real valued fuctio Bayes optimal Classifier
Definitions and Theorems. where x are the decision variables. c, b, and a are constant coefficients.
Defiitios ad Theorems Remember the scalar form of the liear programmig problem, Miimize, Subject to, f(x) = c i x i a 1i x i = b 1 a mi x i = b m x i 0 i = 1,2,, where x are the decisio variables. c, b,
Optimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
Fall 2013 MTH431/531 Real analysis Section Notes
Fall 013 MTH431/531 Real aalysis Sectio 8.1-8. Notes Yi Su 013.11.1 1. Defiitio of uiform covergece. We look at a sequece of fuctios f (x) ad study the coverget property. Notice we have two parameters
Stat 421-SP2012 Interval Estimation Section
Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible
Part I: Covers Sequence through Series Comparison Tests
Part I: Covers Sequece through Series Compariso Tests. Give a example of each of the followig: (a) A geometric sequece: (b) A alteratig sequece: (c) A sequece that is bouded, but ot coverget: (d) A sequece
Machine Learning Theory Tübingen University, WS 2016/2017 Lecture 12
Machie Learig Theory Tübige Uiversity, WS 06/07 Lecture Tolstikhi Ilya Abstract I this lecture we derive risk bouds for kerel methods. We will start by showig that Soft Margi kerel SVM correspods to miimizig
Statistical Machine Learning II Spring 2017, Learning Theory, Lecture 7
Statistical Machie Learig II Sprig 2017, Learig Theory, Lecture 7 1 Itroductio Jea Hoorio jhoorio@purdue.edu So far we have see some techiques for provig geeralizatio for coutably fiite hypothesis classes
sin(n) + 2 cos(2n) n 3/2 3 sin(n) 2cos(2n) n 3/2 a n =
60. Ratio ad root tests 60.1. Absolutely coverget series. Defiitio 13. (Absolute covergece) A series a is called absolutely coverget if the series of absolute values a is coverget. The absolute covergece
Lecture 15: Learning Theory: Concentration Inequalities
STAT 425: Itroductio to Noparametric Statistics Witer 208 Lecture 5: Learig Theory: Cocetratio Iequalities Istructor: Ye-Chi Che 5. Itroductio Recall that i the lecture o classificatio, we have see that
6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
Lecture Notes for CS 313H, Fall 2011
Lecture Notes for CS 313H, Fall 011 August 5. We start by examiig triagular umbers: T () = 1 + + + ( = 0, 1,,...). Triagular umbers ca be also defied recursively: T (0) = 0, T ( + 1) = T () + + 1, or usig
Lecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
Statistical Pattern Recognition
Statistical Patter Recogitio Classificatio: No-Parametric Modelig Hamid R. Rabiee Jafar Muhammadi Sprig 2014 http://ce.sharif.edu/courses/92-93/2/ce725-2/ Ageda Parametric Modelig No-Parametric Modelig
Lecture 8: Convergence of transformations and law of large numbers
Lecture 8: Covergece of trasformatios ad law of large umbers Trasformatio ad covergece Trasformatio is a importat tool i statistics. If X coverges to X i some sese, we ofte eed to check whether g(x ) coverges
Topics Machine learning: lecture 2. Review: the learning problem. Hypotheses and estimation. Estimation criterion cont d. Estimation criterion
.87 Machie learig: lecture Tommi S. Jaakkola MIT CSAIL tommi@csail.mit.edu Topics The learig problem hypothesis class, estimatio algorithm loss ad estimatio criterio samplig, empirical ad epected losses
Section 11.8: Power Series
Sectio 11.8: Power Series 1. Power Series I this sectio, we cosider geeralizig the cocept of a series. Recall that a series is a ifiite sum of umbers a. We ca talk about whether or ot it coverges ad i
62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
Essential Question How can you recognize an arithmetic sequence from its graph?
. Aalyzig Arithmetic Sequeces ad Series COMMON CORE Learig Stadards HSF-IF.A.3 HSF-BF.A. HSF-LE.A. Essetial Questio How ca you recogize a arithmetic sequece from its graph? I a arithmetic sequece, the
CHAPTER 10 INFINITE SEQUENCES AND SERIES
CHAPTER 10 INFINITE SEQUENCES AND SERIES 10.1 Sequeces 10.2 Ifiite Series 10.3 The Itegral Tests 10.4 Compariso Tests 10.5 The Ratio ad Root Tests 10.6 Alteratig Series: Absolute ad Coditioal Covergece
1 Review of Probability & Statistics
1 Review of Probability & Statistics a. I a group of 000 people, it has bee reported that there are: 61 smokers 670 over 5 960 people who imbibe (drik alcohol) 86 smokers who imbibe 90 imbibers over 5
ARIMA Models. Dan Saunders. y t = φy t 1 + ɛ t
ARIMA Models Da Sauders I will discuss models with a depedet variable y t, a potetially edogeous error term ɛ t, ad a exogeous error term η t, each with a subscript t deotig time. With just these three
Output Analysis and Run-Length Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
Solution of Final Exam : / Machine Learning
Solutio of Fial Exam : 10-701/15-781 Machie Learig Fall 2004 Dec. 12th 2004 Your Adrew ID i capital letters: Your full ame: There are 9 questios. Some of them are easy ad some are more difficult. So, if
Efficient GMM LECTURE 12 GMM II
DECEMBER 1 010 LECTURE 1 II Efficiet The estimator depeds o the choice of the weight matrix A. The efficiet estimator is the oe that has the smallest asymptotic variace amog all estimators defied by differet
MA131 - Analysis 1. Workbook 3 Sequences II
MA3 - Aalysis Workbook 3 Sequeces II Autum 2004 Cotets 2.8 Coverget Sequeces........................ 2.9 Algebra of Limits......................... 2 2.0 Further Useful Results........................
Machine Learning Brett Bernstein
Machie Learig Brett Berstei Week 2 Lecture: Cocept Check Exercises Starred problems are optioal. Excess Risk Decompositio 1. Let X = Y = {1, 2,..., 10}, A = {1,..., 10, 11} ad suppose the data distributio
Carleton College, Winter 2017 Math 121, Practice Final Prof. Jones. Note: the exam will have a section of true-false questions, like the one below.
Carleto College, Witer 207 Math 2, Practice Fial Prof. Joes Note: the exam will have a sectio of true-false questios, like the oe below.. True or False. Briefly explai your aswer. A icorrectly justified
Math 475, Problem Set #12: Answers
Math 475, Problem Set #12: Aswers A. Chapter 8, problem 12, parts (b) ad (d). (b) S # (, 2) = 2 2, sice, from amog the 2 ways of puttig elemets ito 2 distiguishable boxes, exactly 2 of them result i oe
MATH 324 Summer 2006 Elementary Number Theory Solutions to Assignment 2 Due: Thursday July 27, 2006
MATH 34 Summer 006 Elemetary Number Theory Solutios to Assigmet Due: Thursday July 7, 006 Departmet of Mathematical ad Statistical Scieces Uiversity of Alberta Questio [p 74 #6] Show that o iteger of the
Binary classification, Part 1
Biary classificatio, Part 1 Maxim Ragisky September 25, 2014 The problem of biary classificatio ca be stated as follows. We have a radom couple Z = (X,Y ), where X R d is called the feature vector ad Y
An alternating series is a series where the signs alternate. Generally (but not always) there is a factor of the form ( 1) n + 1
Calculus II - Problem Solvig Drill 20: Alteratig Series, Ratio ad Root Tests Questio No. of 0 Istructios: () Read the problem ad aswer choices carefully (2) Work the problems o paper as eeded (3) Pick
Math F215: Induction April 7, 2013
Math F25: Iductio April 7, 203 Iductio is used to prove that a collectio of statemets P(k) depedig o k N are all true. A statemet is simply a mathematical phrase that must be either true or false. Here
3.2 Properties of Division 3.3 Zeros of Polynomials 3.4 Complex and Rational Zeros of Polynomials
Math 60 www.timetodare.com 3. Properties of Divisio 3.3 Zeros of Polyomials 3.4 Complex ad Ratioal Zeros of Polyomials I these sectios we will study polyomials algebraically. Most of our work will be cocered
1 Generating functions for balls in boxes
Math 566 Fall 05 Some otes o geeratig fuctios Give a sequece a 0, a, a,..., a,..., a geeratig fuctio some way of represetig the sequece as a fuctio. There are may ways to do this, with the most commo ways
Induction: Solutions
Writig Proofs Misha Lavrov Iductio: Solutios Wester PA ARML Practice March 6, 206. Prove that a 2 2 chessboard with ay oe square removed ca always be covered by shaped tiles. Solutio : We iduct o. For
6.867 Machine learning
6.867 Machie learig Mid-term exam October, ( poits) Your ame ad MIT ID: Problem We are iterested here i a particular -dimesioal liear regressio problem. The dataset correspodig to this problem has examples
Machine Learning. Ilya Narsky, Caltech
Machie Learig Ilya Narsky, Caltech Lecture 4 Multi-class problems. Multi-class versios of Neural Networks, Decisio Trees, Support Vector Machies ad AdaBoost. Reductio of a multi-class problem to a set
SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES
SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES Read Sectio 1.5 (pages 5 9) Overview I Sectio 1.5 we lear to work with summatio otatio ad formulas. We will also itroduce a brief overview of sequeces,
Problem Set 4 Due Oct, 12
EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios
Chapter 6: Numerical Series
Chapter 6: Numerical Series 327 Chapter 6 Overview: Sequeces ad Numerical Series I most texts, the topic of sequeces ad series appears, at first, to be a side topic. There are almost o derivatives or itegrals
NUMERICAL METHODS FOR SOLVING EQUATIONS
Mathematics Revisio Guides Numerical Methods for Solvig Equatios Page 1 of 11 M.K. HOME TUITION Mathematics Revisio Guides Level: GCSE Higher Tier NUMERICAL METHODS FOR SOLVING EQUATIONS Versio:. Date:
b i u x i U a i j u x i u x j
M ath 5 2 7 Fall 2 0 0 9 L ecture 1 9 N ov. 1 6, 2 0 0 9 ) S ecod- Order Elliptic Equatios: Weak S olutios 1. Defiitios. I this ad the followig two lectures we will study the boudary value problem Here
Lecture 10 October Minimaxity and least favorable prior sequences
STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least
Lecture 3. Properties of Summary Statistics: Sampling Distribution
Lecture 3 Properties of Summary Statistics: Samplig Distributio Mai Theme How ca we use math to justify that our umerical summaries from the sample are good summaries of the populatio? Lecture Summary
Support vector machine revisited
6.867 Machie learig, lecture 8 (Jaakkola) 1 Lecture topics: Support vector machie ad kerels Kerel optimizatio, selectio Support vector machie revisited Our task here is to first tur the support vector
18.440, March 9, Stirling s formula
Stirlig s formula 8.44, March 9, 9 The factorial fuctio! is importat i evaluatig biomial, hypergeometric, ad other probabilities. If is ot too large,! ca be computed directly, by calculators or computers.
1 Review and Overview
DRAFT a fial versio will be posted shortly CS229T/STATS231: Statistical Learig Theory Lecturer: Tegyu Ma Lecture #3 Scribe: Migda Qiao October 1, 2013 1 Review ad Overview I the first half of this course,
Recurrence Relations
Recurrece Relatios Aalysis of recursive algorithms, such as: it factorial (it ) { if (==0) retur ; else retur ( * factorial(-)); } Let t be the umber of multiplicatios eeded to calculate factorial(). The
The Quark Puzzle A 3D printable model and/or paper printable puzzle that allows students to learn the laws of colour charge through inquiry.
The Quark Puzzle A 3D pritable model ad/or paper pritable puzzle that allows studets to lear the laws of colour charge through iquiry. It is available at this lik: https://zeodo.org/record/1252868#.w3ft-gzauk
Lecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
MAT1026 Calculus II Basic Convergence Tests for Series
MAT026 Calculus II Basic Covergece Tests for Series Egi MERMUT 202.03.08 Dokuz Eylül Uiversity Faculty of Sciece Departmet of Mathematics İzmir/TURKEY Cotets Mootoe Covergece Theorem 2 2 Series of Real
MATH/STAT 352: Lecture 15
MATH/STAT 352: Lecture 15 Sectios 5.2 ad 5.3. Large sample CI for a proportio ad small sample CI for a mea. 1 5.2: Cofidece Iterval for a Proportio Estimatig proportio of successes i a biomial experimet
Zeros of Polynomials
Math 160 www.timetodare.com 4.5 4.6 Zeros of Polyomials I these sectios we will study polyomials algebraically. Most of our work will be cocered with fidig the solutios of polyomial equatios of ay degree
Big Picture. 5. Data, Estimates, and Models: quantifying the accuracy of estimates.
5. Data, Estimates, ad Models: quatifyig the accuracy of estimates. 5. Estimatig a Normal Mea 5.2 The Distributio of the Normal Sample Mea 5.3 Normal data, cofidece iterval for, kow 5.4 Normal data, cofidece
Math 61CM - Solutions to homework 3
Math 6CM - Solutios to homework 3 Cédric De Groote October 2 th, 208 Problem : Let F be a field, m 0 a fixed oegative iteger ad let V = {a 0 + a x + + a m x m a 0,, a m F} be the vector space cosistig
Grouping 2: Spectral and Agglomerative Clustering. CS 510 Lecture #16 April 2 nd, 2014
Groupig 2: Spectral ad Agglomerative Clusterig CS 510 Lecture #16 April 2 d, 2014 Groupig (review) Goal: Detect local image features (SIFT) Describe image patches aroud features SIFT, SURF, HoG, LBP, Group
A quick activity - Central Limit Theorem and Proportions. Lecture 21: Testing Proportions. Results from the GSS. Statistics and the General Population
A quick activity - Cetral Limit Theorem ad Proportios Lecture 21: Testig Proportios Statistics 10 Coli Rudel Flip a coi 30 times this is goig to get loud! Record the umber of heads you obtaied ad calculate
Bertrand s Postulate
Bertrad s Postulate Lola Thompso Ross Program July 3, 2009 Lola Thompso (Ross Program Bertrad s Postulate July 3, 2009 1 / 33 Bertrad s Postulate I ve said it oce ad I ll say it agai: There s always a
Infinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
Machine Learning for Data Science (CS 4786)
Machie Learig for Data Sciece CS 4786) Lecture & 3: Pricipal Compoet Aalysis The text i black outlies high level ideas. The text i blue provides simple mathematical details to derive or get to the algorithm
and each factor on the right is clearly greater than 1. which is a contradiction, so n must be prime.
MATH 324 Summer 200 Elemetary Number Theory Solutios to Assigmet 2 Due: Wedesday July 2, 200 Questio [p 74 #6] Show that o iteger of the form 3 + is a prime, other tha 2 = 3 + Solutio: If 3 + is a prime,
An Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
Series: Infinite Sums
Series: Ifiite Sums Series are a way to mae sese of certai types of ifiitely log sums. We will eed to be able to do this if we are to attai our goal of approximatig trascedetal fuctios by usig ifiite degree
A Question. Output Analysis. Example. What Are We Doing Wrong? Result from throwing a die. Let X be the random variable
A Questio Output Aalysis Let X be the radom variable Result from throwig a die 5.. Questio: What is E (X? Would you throw just oce ad take the result as your aswer? Itroductio to Simulatio WS/ - L 7 /
Math 25 Solutions to practice problems
Math 5: Advaced Calculus UC Davis, Sprig 0 Math 5 Solutios to practice problems Questio For = 0,,, 3,... ad 0 k defie umbers C k C k =! k!( k)! (for k = 0 ad k = we defie C 0 = C = ). by = ( )... ( k +
Practice Problems: Taylor and Maclaurin Series
Practice Problems: Taylor ad Maclauri Series Aswers. a) Start by takig derivatives util a patter develops that lets you to write a geeral formula for the -th derivative. Do t simplify as you go, because
Resolution Proofs of Generalized Pigeonhole Principles
Resolutio Proofs of Geeralized Pigeohole Priciples Samuel R. Buss Departmet of Mathematics Uiversity of Califoria, Berkeley Győrgy Turá Departmet of Mathematics, Statistics, ad Computer Sciece Uiversity
HOMEWORK 2 SOLUTIONS
HOMEWORK SOLUTIONS CSE 55 RANDOMIZED AND APPROXIMATION ALGORITHMS 1. Questio 1. a) The larger the value of k is, the smaller the expected umber of days util we get all the coupos we eed. I fact if = k
CSE 1400 Applied Discrete Mathematics Number Theory and Proofs
CSE 1400 Applied Discrete Mathematics Number Theory ad Proofs Departmet of Computer Scieces College of Egieerig Florida Tech Sprig 01 Problems for Number Theory Backgroud Number theory is the brach of
Fourier Series and the Wave Equation
Fourier Series ad the Wave Equatio We start with the oe-dimesioal wave equatio u u =, x u(, t) = u(, t) =, ux (,) = f( x), u ( x,) = This represets a vibratig strig, where u is the displacemet of the strig
Advanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
Sigma notation. 2.1 Introduction
Sigma otatio. Itroductio We use sigma otatio to idicate the summatio process whe we have several (or ifiitely may) terms to add up. You may have see sigma otatio i earlier courses. It is used to idicate
1 Hash tables. 1.1 Implementation
Lecture 8 Hash Tables, Uiversal Hash Fuctios, Balls ad Bis Scribes: Luke Johsto, Moses Charikar, G. Valiat Date: Oct 18, 2017 Adapted From Virgiia Williams lecture otes 1 Hash tables A hash table is a
Ma 530 Introduction to Power Series
Ma 530 Itroductio to Power Series Please ote that there is material o power series at Visual Calculus. Some of this material was used as part of the presetatio of the topics that follow. What is a Power
w (1) ˆx w (1) x (1) /ρ and w (2) ˆx w (2) x (2) /ρ.
2 5. Weighted umber of late jobs 5.1. Release dates ad due dates: maximimizig the weight of o-time jobs Oce we add release dates, miimizig the umber of late jobs becomes a sigificatly harder problem. For
Lesson 10: Limits and Continuity
www.scimsacademy.com Lesso 10: Limits ad Cotiuity SCIMS Academy 1 Limit of a fuctio The cocept of limit of a fuctio is cetral to all other cocepts i calculus (like cotiuity, derivative, defiite itegrals
Math 113 Exam 3 Practice
Math Exam Practice Exam will cover.-.9. This sheet has three sectios. The first sectio will remid you about techiques ad formulas that you should kow. The secod gives a umber of practice questios for you
Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.
Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator
Complex Analysis Spring 2001 Homework I Solution
Complex Aalysis Sprig 2001 Homework I Solutio 1. Coway, Chapter 1, sectio 3, problem 3. Describe the set of poits satisfyig the equatio z a z + a = 2c, where c > 0 ad a R. To begi, we see from the triagle
ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015
ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],
ENGI 4421 Confidence Intervals (Two Samples) Page 12-01
ENGI 44 Cofidece Itervals (Two Samples) Page -0 Two Sample Cofidece Iterval for a Differece i Populatio Meas [Navidi sectios 5.4-5.7; Devore chapter 9] From the cetral limit theorem, we kow that, for sufficietly
10.6 ALTERNATING SERIES
0.6 Alteratig Series Cotemporary Calculus 0.6 ALTERNATING SERIES I the last two sectios we cosidered tests for the covergece of series whose terms were all positive. I this sectio we examie series whose
PAPER : IIT-JAM 2010
MATHEMATICS-MA (CODE A) Q.-Q.5: Oly oe optio is correct for each questio. Each questio carries (+6) marks for correct aswer ad ( ) marks for icorrect aswer.. Which of the followig coditios does NOT esure
6.867 Machine learning, lecture 7 (Jaakkola) 1
6.867 Machie learig, lecture 7 (Jaakkola) 1 Lecture topics: Kerel form of liear regressio Kerels, examples, costructio, properties Liear regressio ad kerels Cosider a slightly simpler model where we omit
CHAPTER 1 SEQUENCES AND INFINITE SERIES
CHAPTER SEQUENCES AND INFINITE SERIES SEQUENCES AND INFINITE SERIES (0 meetigs) Sequeces ad limit of a sequece Mootoic ad bouded sequece Ifiite series of costat terms Ifiite series of positive terms Alteratig
Lecture 4. We also define the set of possible values for the random walk as the set of all x R d such that P(S n = x) > 0 for some n.
Radom Walks ad Browia Motio Tel Aviv Uiversity Sprig 20 Lecture date: Mar 2, 20 Lecture 4 Istructor: Ro Peled Scribe: Lira Rotem This lecture deals primarily with recurrece for geeral radom walks. We preset
Axis Aligned Ellipsoid
Machie Learig for Data Sciece CS 4786) Lecture 6,7 & 8: Ellipsoidal Clusterig, Gaussia Mixture Models ad Geeral Mixture Models The text i black outlies high level ideas. The text i blue provides simple