1 Definition of Rademacher Complexity

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "1 Definition of Rademacher Complexity"

Transcription

1 COS 511: Theoretcal Machne Learnng Lecturer: Rob Schapre Lecture #9 Scrbe: Josh Chen March 5, 2013 We ve spent the past few classes provng bounds on the generalzaton error of PAClearnng algorths for the cases of consstent and nconsstent hypotheses selected fro fnte and nfnte hypothess spaces. In partcular, last te, we proved bounds for the case of nconsstent hypotheses selected fro nfnte hypothess spaces. However, recall that each te we encountered the proble of an nfnte hypothess space, we had to resort to technques lke usng ghost saples or the VC-denson of a concept class. In ths lecture, we ntroduce a ore odern and elegant approach, usng a concept called Radeacher coplexty. Ths approach turns out to nclude each of the bounds we ve proved n the past few lectures as specal cases. 1 Defnton of Radeacher Coplexty 1.1 Soe usual defntons Before gettng nto the defnton of Radeacher coplexty, we rend ourselves of the usual setup: Let the saple S = ((x 1, y 1 ),..., (x, y )) where, unlke before, y = 1, +1} Let the hypothess h : X 1, +1} To easure how well h fts S, let the tranng error err(h) ˆ = 1 =1 1 h(x ) y Note that, snce we are usng y = 1, +1} nstead of y = 0, 1} as n prevous lectures (for splcty), we can provde an alternatve defnton of tranng error: err(h) ˆ = 1 1h(x ) y } (1) =1 1 f (h(x ), y ) = (1, 1) or ( 1, 1) = 1 0 f (h(x =1 ), y ) = (1, 1) or ( 1, 1) (2) = 1 1 y h(x ) 2 =1 (3) = y h(x ) 2 (4) =1 The ter 1 =1 y h(x ) can be nterpreted as the correlaton of the predctons h(x ) wth the labels y. We see that correlaton s related to tranng error as correlaton = 1 2err(h). ˆ To fnd a hypothess h that nzes tranng error, we can thus equvalently seek to fnd the h satsfyng: 1 arg ax y h(x ) (5) h H

2 1.2 Playng wth correlaton Iagne, now, an experent where we replace a saple s true labels y wth the Radeacher rando varables σ : +1 wth prob. 1/2 σ = (6) 1 wth prob. 1/2 Ths gves a odfed expresson for correlaton: arg ax h H 1 σ h(x ) (7) Instead of selectng the hypothess n H that correlates best wth the labels, ths now selects the hypothess h n H that correlates best wth the rando nose varables σ. Snce h s dependent on the rando varables σ, however, to easure how well H can correlate wth rando nose, we take the expectaton of ths correlaton over the rando varables σ and fnd: E σ [ax h H 1 σ h(x )] (8) Ths ntutvely easures the expressveness of H. We can bound ths expresson usng two extree cases: H = 1 where we only have one choce for a hypothess, and H = 2 where H shatters S. In the frst case, our expectaton equals 0 snce the ax ter dsappears; n the second case our expectaton equals 1 snce there always exsts a hypothess atchng any set of σ s. Thus our easure, as defned above, ust fall between 0 and Generalzng correlaton Instead of workng wth hypotheses h : X 1, +1}, let s generalze our class of functons to the set of all real-valued functons. Replace H wth F, whch we defne to be any faly of functons f : Z R. Now, gven saple S = (z 1,..., z ) wth z Z, f we apply our expresson fro above to F, we arrve at the eprcal Radeacher coplexty of a faly of functons F wth respect to a saple S: 1 1 ˆR S (F) := E σ [sup σ f(z )] (9) Agan, ths expresson easures how well, on average, the functon class F correlates wth rando nose over the saple S. However, often we want to easure the correlaton of F wth respect to a dstrbuton D over X, rather than wth respect to a saple S over X. To fnd ths, we take the expectaton of ˆR S (F) over all saples of sze drawn accordng to D: R (F) := E[ ˆR S (F)] (10) Ths s the Radeacher coplexty, or for clarty, the expected Radeacher coplexty, of F. We now have the defntons we need, and are fnally ready to present our frst generalzaton bounds based on Radeacher coplexty. 1 Note: Snce F can be the faly of all real-valued functons, ax ay not exst. Thus we use sup nstead, whch s defned as the least upper bound on the eleents n a set. For exaple, the sup of the set.9,.99,.999,...} s 1. 2

3 2 Generalzaton bounds based on Radeacher coplexty 2.1 Bounds for general functon classes F The followng theore wll serve as a very general tool for provng unfor convergence bounds va the concept of Radeacher coplexty: Theore 1. Let F be a faly of functons appng fro Z to [0, 1], and let saple S = (z 1,..., z ) where z D for soe dstrbuton D over Z. Defne E[f] := E Z D [f(z)], and defne ÊS[f] := 1 =1 f(z ). Wth probablty 1 δ, for all f F: 2 ( ) E[f] ÊS[f] + 2R (F) + O ( ) E[f] ÊS[f] + 2 ˆR S (F) + O Proof. We derve a bound for E[f] ÊS[f] for all f F, or equvalently, bound sup (E[f] Ê S [f]). Note that ths expresson s a rando varable that depends on S. So we want to bound the followng rando varable: (11) (12) Φ(S) = sup(e[f] ÊS[f]) (13) Step 1: We show, wth probablty 1 δ, Φ(S) E S [Φ(S)] + 2. Ths step allows us to go fro workng wth Φ(S) to workng wth E S [Φ(S)]. then: Recall that McDard s nequalty states that, f: f(x 1,..., x,..., x ) f(x 1,..., x,..., x ) c (14) P r[f(x 1,..., x ) E[f(X 1,..., X )] + ɛ] exp( 2ɛ 2 / Fro the defnton of Φ(S), we have: c 2 ) (15) Φ(S) = sup(e[f] ÊS[f]) (16) = sup (E[f] 1 f(z )) (17) Snce f(z ) [0, 1] for all z, changng any one exaple z to z n the tranng set S wll change 1 f(z ) by at ost 1. Thus ths changng of any one exaple affects Φ(S) by at ost ths aount, plyng that Φ((z 1,..., z,..., z )) Φ((z 1,..., z,..., z )) 1. Ths fts the condton of McDard s nequalty (see (14)) wth c = 1, so we can apply McDard s nequalty and arrve at the bound shown. 2 Note that the Bg-Oh ters n the two expressons have dfferent constants. =1 3

4 Step 2: Defne a ghost saple S = (z 1,..., z ), z D. We show that E S [Φ(S)] E S,S [sup (ÊS [f] ÊS[f])]: E S [Φ(S)] = E S [sup(e[f] ÊS[f])] (18) = E S [sup(e S [ÊS [f]] ÊS[f])] (19) = E S [sup(e S [ÊS [f] ÊS[f]])] (20) E S,S [sup(ês [f] ÊS[f])] (21) Note that we arrve at (19) snce the expected Radeacher coplexty E[f] s equal to the expectaton over all saples S of the eprcal Radeacher coplexty over those S, or E S [ÊS [f]]. We also arrve at (21) by ovng the expectaton over S n (20) outsde of the sup; ths can be done snce the expectaton of a ax over soe functon s at least the ax of that expectaton over that functon. Step 3: We show E S,S [sup (ÊS [f] ÊS[f])] = E S,S,σ[sup σ (f(z ) f(z ))] We use the ghost saplng technque for ths step. In partcular, for each par of eleents z, z n S, S respectvely, swap the two wth probablty 1/2. Let the resultng two sets of exaples be T, T. Snce S, S each ntally represented d saples fro D, we have that T, T S, S. Ths ples: Ê S [f] ÊS[f] ÊT [f] ÊT [f] (22) = 1 f(z ) f(z ) wth prob. 1/2 f(z ) f(z (23) ) wth prob. 1/2 = 1 σ (f(z ) f(z )) (24) Thus the expressons sup (ÊS [f] ÊS[f]) and sup σ (f(z ) f(z )) are equally dstrbuted. The latter depends on an addtonal set of rando varables σ, however, so we ust take the expectaton of the latter over σ as well as S, S. Takng the expectaton of the forer over S, S, as well, we arrve at the expresson shown. Step 4: We show E S,S,σ[sup σ (f(z ) f(z ))] 2R (F) E S,S,σ[sup σ (f(z ) f(z ))] E S,S,σ[sup σ f(z ) + sup ( σ )f(z ))] (25) E S,σ[sup σ f(z )] + E S,σ [sup ( σ )f(z ))] (26) = R (F) + R (F) (27) where we arrve at (27) because σ has the sae dstrbuton as σ. Concluson: Cobnng all the peces together, we fnally have that, wth probablty 1 δ, for all f F: E[f] ÊS[f] 2R (F) + (28) 2 4

5 To derve the bound nvolvng ˆR S (F), we use McDard s nequalty agan. Recall the defnton of ˆR 1 S (F) := E σ [sup σ f(z )]. Snce f [0, 1], changng one eleent n S changes ˆR S (F) by at ost 1. We can apply McDard s nequalty agan, fndng, wth probablty 1 δ: ˆR S (F) R (F) + (29) 2 Usng a δ = δ/2 and applyng the unon bound to (28) and (29), we have our result. Wth probablty 1 δ, for all f F: E[f] ÊS[f] + 2 ˆR S (F) + O( ) (30) 2.2 Bounds for hypothess spaces H To get fro ths generalzaton bound on classes of all real-valued functons to classes of hypotheses, defne the followng: Z = X 1, +1} (31) f h (x, y) = 1h(x) y} (32) F H = f h : h H} (33) Note that, due to (33), each f h F H corresponds to soe h H. Also note that, by these defntons, we have: err(h) = E (x,y) D [1h(x) y}] = E[f h ] (34) err(h) ˆ = 1 1h(x ) y } = h ] (35) Evdently we can use our bound fro Theore 1 to bound err(h) err(h): ˆ 1 ˆR S (F H ) = E σ [ sup σ f h (x, y )] (36) f h F H 1 = E σ [sup σ ( 1 y h(x ) )] (37) h H 2 = E σ [ 1 1 σ + sup ( y σ )h(x )] (38) 2 h H 2 = 1 2 E 1 σ[sup h H = 1 2 E 1 σ[sup h H ( y σ )h(x )] (39) σ h(x )] (40) = 1 2 ˆR S (H) (41) Note that we arrve at (40) snce ( y σ ) has the sae dstrbuton as σ. Now, cobnng (30), (34), (35), and (41), we have: err(h) err(h) ˆ + ˆR S (H) + O( ) (42) 5

Excess Error, Approximation Error, and Estimation Error

Excess Error, Approximation Error, and Estimation Error E0 370 Statstcal Learnng Theory Lecture 10 Sep 15, 011 Excess Error, Approxaton Error, and Estaton Error Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton So far, we have consdered the fnte saple

More information

Learning Theory: Lecture Notes

Learning Theory: Lecture Notes Learnng Theory: Lecture Notes Lecturer: Kamalka Chaudhur Scrbe: Qush Wang October 27, 2012 1 The Agnostc PAC Model Recall that one of the constrants of the PAC model s that the data dstrbuton has to be

More information

Polynomials. 1 More properties of polynomials

Polynomials. 1 More properties of polynomials Polynomals 1 More propertes of polynomals Recall that, for R a commutatve rng wth unty (as wth all rngs n ths course unless otherwse noted), we defne R[x] to be the set of expressons n =0 a x, where a

More information

Lecture 10 Support Vector Machines II

Lecture 10 Support Vector Machines II Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed

More information

10-701/ Machine Learning, Fall 2005 Homework 3

10-701/ Machine Learning, Fall 2005 Homework 3 10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40

More information

P exp(tx) = 1 + t 2k M 2k. k N

P exp(tx) = 1 + t 2k M 2k. k N 1. Subgaussan tals Defnton. Say that a random varable X has a subgaussan dstrbuton wth scale factor σ< f P exp(tx) exp(σ 2 t 2 /2) for all real t. For example, f X s dstrbuted N(,σ 2 ) then t s subgaussan.

More information

1 The Mistake Bound Model

1 The Mistake Bound Model 5-850: Advanced Algorthms CMU, Sprng 07 Lecture #: Onlne Learnng and Multplcatve Weghts February 7, 07 Lecturer: Anupam Gupta Scrbe: Bryan Lee,Albert Gu, Eugene Cho he Mstake Bound Model Suppose there

More information

XII.3 The EM (Expectation-Maximization) Algorithm

XII.3 The EM (Expectation-Maximization) Algorithm XII.3 The EM (Expectaton-Maxzaton) Algorth Toshnor Munaata 3/7/06 The EM algorth s a technque to deal wth varous types of ncoplete data or hdden varables. It can be appled to a wde range of learnng probles

More information

Week 2. This week, we covered operations on sets and cardinality.

Week 2. This week, we covered operations on sets and cardinality. Week 2 Ths week, we covered operatons on sets and cardnalty. Defnton 0.1 (Correspondence). A correspondence between two sets A and B s a set S contaned n A B = {(a, b) a A, b B}. A correspondence from

More information

Lecture 12: Discrete Laplacian

Lecture 12: Discrete Laplacian Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly

More information

arxiv: v2 [math.co] 3 Sep 2017

arxiv: v2 [math.co] 3 Sep 2017 On the Approxate Asyptotc Statstcal Independence of the Peranents of 0- Matrces arxv:705.0868v2 ath.co 3 Sep 207 Paul Federbush Departent of Matheatcs Unversty of Mchgan Ann Arbor, MI, 4809-043 Septeber

More information

ITERATIVE ESTIMATION PROCEDURE FOR GEOSTATISTICAL REGRESSION AND GEOSTATISTICAL KRIGING

ITERATIVE ESTIMATION PROCEDURE FOR GEOSTATISTICAL REGRESSION AND GEOSTATISTICAL KRIGING ESE 5 ITERATIVE ESTIMATION PROCEDURE FOR GEOSTATISTICAL REGRESSION AND GEOSTATISTICAL KRIGING Gven a geostatstcal regresson odel: k Y () s x () s () s x () s () s, s R wth () unknown () E[ ( s)], s R ()

More information

3.1 ML and Empirical Distribution

3.1 ML and Empirical Distribution 67577 Intro. to Machne Learnng Fall semester, 2008/9 Lecture 3: Maxmum Lkelhood/ Maxmum Entropy Dualty Lecturer: Amnon Shashua Scrbe: Amnon Shashua 1 In the prevous lecture we defned the prncple of Maxmum

More information

MATH 5707 HOMEWORK 4 SOLUTIONS 2. 2 i 2p i E(X i ) + E(Xi 2 ) ä i=1. i=1

MATH 5707 HOMEWORK 4 SOLUTIONS 2. 2 i 2p i E(X i ) + E(Xi 2 ) ä i=1. i=1 MATH 5707 HOMEWORK 4 SOLUTIONS CİHAN BAHRAN 1. Let v 1,..., v n R m, all lengths v are not larger than 1. Let p 1,..., p n [0, 1] be arbtrary and set w = p 1 v 1 + + p n v n. Then there exst ε 1,..., ε

More information

Module 2. Random Processes. Version 2 ECE IIT, Kharagpur

Module 2. Random Processes. Version 2 ECE IIT, Kharagpur Module Random Processes Lesson 6 Functons of Random Varables After readng ths lesson, ou wll learn about cdf of functon of a random varable. Formula for determnng the pdf of a random varable. Let, X be

More information

The Second Anti-Mathima on Game Theory

The Second Anti-Mathima on Game Theory The Second Ant-Mathma on Game Theory Ath. Kehagas December 1 2006 1 Introducton In ths note we wll examne the noton of game equlbrum for three types of games 1. 2-player 2-acton zero-sum games 2. 2-player

More information

Least Squares Fitting of Data

Least Squares Fitting of Data Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC http://www.geoetrctools.co/ Copyrght c 1998-2014. All Rghts Reserved. Created: July 15, 1999 Last Modfed: February 9, 2008 Contents 1 Lnear Fttng

More information

Econ107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4)

Econ107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4) I. Classcal Assumptons Econ7 Appled Econometrcs Topc 3: Classcal Model (Studenmund, Chapter 4) We have defned OLS and studed some algebrac propertes of OLS. In ths topc we wll study statstcal propertes

More information

Symmetrization and Rademacher Averages

Symmetrization and Rademacher Averages Stat 928: Statistical Learning Theory Lecture: Syetrization and Radeacher Averages Instructor: Sha Kakade Radeacher Averages Recall that we are interested in bounding the difference between epirical and

More information

COS 521: Advanced Algorithms Game Theory and Linear Programming

COS 521: Advanced Algorithms Game Theory and Linear Programming COS 521: Advanced Algorthms Game Theory and Lnear Programmng Moses Charkar February 27, 2013 In these notes, we ntroduce some basc concepts n game theory and lnear programmng (LP). We show a connecton

More information

Least Squares Fitting of Data

Least Squares Fitting of Data Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC http://www.geoetrctools.co/ Copyrght c 1998-2015. All Rghts Reserved. Created: July 15, 1999 Last Modfed: January 5, 2015 Contents 1 Lnear Fttng

More information

Case A. P k = Ni ( 2L i k 1 ) + (# big cells) 10d 2 P k.

Case A. P k = Ni ( 2L i k 1 ) + (# big cells) 10d 2 P k. THE CELLULAR METHOD In ths lecture, we ntroduce the cellular method as an approach to ncdence geometry theorems lke the Szemeréd-Trotter theorem. The method was ntroduced n the paper Combnatoral complexty

More information

On the number of regions in an m-dimensional space cut by n hyperplanes

On the number of regions in an m-dimensional space cut by n hyperplanes 6 On the nuber of regons n an -densonal space cut by n hyperplanes Chungwu Ho and Seth Zeran Abstract In ths note we provde a unfor approach for the nuber of bounded regons cut by n hyperplanes n general

More information

CS 2750 Machine Learning. Lecture 5. Density estimation. CS 2750 Machine Learning. Announcements

CS 2750 Machine Learning. Lecture 5. Density estimation. CS 2750 Machine Learning. Announcements CS 750 Machne Learnng Lecture 5 Densty estmaton Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square CS 750 Machne Learnng Announcements Homework Due on Wednesday before the class Reports: hand n before

More information

THE CHINESE REMAINDER THEOREM. We should thank the Chinese for their wonderful remainder theorem. Glenn Stevens

THE CHINESE REMAINDER THEOREM. We should thank the Chinese for their wonderful remainder theorem. Glenn Stevens THE CHINESE REMAINDER THEOREM KEITH CONRAD We should thank the Chnese for ther wonderful remander theorem. Glenn Stevens 1. Introducton The Chnese remander theorem says we can unquely solve any par of

More information

On the Calderón-Zygmund lemma for Sobolev functions

On the Calderón-Zygmund lemma for Sobolev functions arxv:0810.5029v1 [ath.ca] 28 Oct 2008 On the Calderón-Zygund lea for Sobolev functons Pascal Auscher october 16, 2008 Abstract We correct an naccuracy n the proof of a result n [Aus1]. 2000 MSC: 42B20,

More information

4 Column generation (CG) 4.1 Basics of column generation. 4.2 Applying CG to the Cutting-Stock Problem. Basic Idea of column generation

4 Column generation (CG) 4.1 Basics of column generation. 4.2 Applying CG to the Cutting-Stock Problem. Basic Idea of column generation 4 Colun generaton (CG) here are a lot of probles n nteger prograng where even the proble defnton cannot be effcently bounded Specfcally, the nuber of coluns becoes very large herefore, these probles are

More information

Integral Transforms and Dual Integral Equations to Solve Heat Equation with Mixed Conditions

Integral Transforms and Dual Integral Equations to Solve Heat Equation with Mixed Conditions Int J Open Probles Copt Math, Vol 7, No 4, Deceber 214 ISSN 1998-6262; Copyrght ICSS Publcaton, 214 www-csrsorg Integral Transfors and Dual Integral Equatons to Solve Heat Equaton wth Mxed Condtons Naser

More information

BAYESIAN NETWORK REASONING WITH UNCERTAIN EVIDENCES

BAYESIAN NETWORK REASONING WITH UNCERTAIN EVIDENCES Internatonal Journal of Uncertanty, Fuzzness and Knowledge-Based Systes Vol. 8, No. 5 (200) 539 564 World Scentfc Publshng Copany DOI: 0.42/S02848850006696 BAYESIAN NETWORK REASONING WITH UNCERTAIN EVIDENCES

More information

STAT 3340 Assignment 1 solutions. 1. Find the equation of the line which passes through the points (1,1) and (4,5).

STAT 3340 Assignment 1 solutions. 1. Find the equation of the line which passes through the points (1,1) and (4,5). (out of 15 ponts) STAT 3340 Assgnment 1 solutons (10) (10) 1. Fnd the equaton of the lne whch passes through the ponts (1,1) and (4,5). β 1 = (5 1)/(4 1) = 4/3 equaton for the lne s y y 0 = β 1 (x x 0

More information

Smarandache-Zero Divisors in Group Rings

Smarandache-Zero Divisors in Group Rings Smarandache-Zero Dvsors n Group Rngs W.B. Vasantha and Moon K. Chetry Department of Mathematcs I.I.T Madras, Chenna The study of zero-dvsors n group rngs had become nterestng problem snce 1940 wth the

More information

Three Algorithms for Flexible Flow-shop Scheduling

Three Algorithms for Flexible Flow-shop Scheduling Aercan Journal of Appled Scences 4 (): 887-895 2007 ISSN 546-9239 2007 Scence Publcatons Three Algorths for Flexble Flow-shop Schedulng Tzung-Pe Hong, 2 Pe-Yng Huang, 3 Gwoboa Horng and 3 Chan-Lon Wang

More information

NUMERICAL DIFFERENTIATION

NUMERICAL DIFFERENTIATION NUMERICAL DIFFERENTIATION 1 Introducton Dfferentaton s a method to compute the rate at whch a dependent output y changes wth respect to the change n the ndependent nput x. Ths rate of change s called the

More information

halftoning Journal of Electronic Imaging, vol. 11, no. 4, Oct Je-Ho Lee and Jan P. Allebach

halftoning Journal of Electronic Imaging, vol. 11, no. 4, Oct Je-Ho Lee and Jan P. Allebach olorant-based drect bnary search» halftonng Journal of Electronc Iagng, vol., no. 4, Oct. 22 Je-Ho Lee and Jan P. Allebach School of Electrcal Engneerng & oputer Scence Kyungpook Natonal Unversty Abstract

More information

The Geometry of Logit and Probit

The Geometry of Logit and Probit The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.

More information

1 Proof of learning bounds

1 Proof of learning bounds COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #4 Scribe: Akshay Mittal February 13, 2013 1 Proof of learning bounds For intuition of the following theore, suppose there exists a

More information

Online Classification: Perceptron and Winnow

Online Classification: Perceptron and Winnow E0 370 Statstcal Learnng Theory Lecture 18 Nov 8, 011 Onlne Classfcaton: Perceptron and Wnnow Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton In ths lecture we wll start to study the onlne learnng

More information

What is LP? LP is an optimization technique that allocates limited resources among competing activities in the best possible manner.

What is LP? LP is an optimization technique that allocates limited resources among competing activities in the best possible manner. (C) 998 Gerald B Sheblé, all rghts reserved Lnear Prograng Introducton Contents I. What s LP? II. LP Theor III. The Splex Method IV. Refneents to the Splex Method What s LP? LP s an optzaton technque that

More information

20. Mon, Oct. 13 What we have done so far corresponds roughly to Chapters 2 & 3 of Lee. Now we turn to Chapter 4. The first idea is connectedness.

20. Mon, Oct. 13 What we have done so far corresponds roughly to Chapters 2 & 3 of Lee. Now we turn to Chapter 4. The first idea is connectedness. 20. Mon, Oct. 13 What we have done so far corresponds roughly to Chapters 2 & 3 of Lee. Now we turn to Chapter 4. The frst dea s connectedness. Essentally, we want to say that a space cannot be decomposed

More information

since [1-( 0+ 1x1i+ 2x2 i)] [ 0+ 1x1i+ assumed to be a reasonable approximation

since [1-( 0+ 1x1i+ 2x2 i)] [ 0+ 1x1i+ assumed to be a reasonable approximation Econ 388 R. Butler 204 revsons Lecture 4 Dummy Dependent Varables I. Lnear Probablty Model: the Regresson model wth a dummy varables as the dependent varable assumpton, mplcaton regular multple regresson

More information

Differentiating Gaussian Processes

Differentiating Gaussian Processes Dfferentatng Gaussan Processes Andrew McHutchon Aprl 17, 013 1 Frst Order Dervatve of the Posteror Mean The posteror mean of a GP s gven by, f = x, X KX, X 1 y x, X α 1 Only the x, X term depends on the

More information

FACTORIZATION IN KRULL MONOIDS WITH INFINITE CLASS GROUP

FACTORIZATION IN KRULL MONOIDS WITH INFINITE CLASS GROUP C O L L O Q U I U M M A T H E M A T I C U M VOL. 80 1999 NO. 1 FACTORIZATION IN KRULL MONOIDS WITH INFINITE CLASS GROUP BY FLORIAN K A I N R A T H (GRAZ) Abstract. Let H be a Krull monod wth nfnte class

More information

Econ Statistical Properties of the OLS estimator. Sanjaya DeSilva

Econ Statistical Properties of the OLS estimator. Sanjaya DeSilva Econ 39 - Statstcal Propertes of the OLS estmator Sanjaya DeSlva September, 008 1 Overvew Recall that the true regresson model s Y = β 0 + β 1 X + u (1) Applyng the OLS method to a sample of data, we estmate

More information

Finite Vector Space Representations Ross Bannister Data Assimilation Research Centre, Reading, UK Last updated: 2nd August 2003

Finite Vector Space Representations Ross Bannister Data Assimilation Research Centre, Reading, UK Last updated: 2nd August 2003 Fnte Vector Space epresentatons oss Bannster Data Asslaton esearch Centre, eadng, UK ast updated: 2nd August 2003 Contents What s a lnear vector space?......... 1 About ths docuent............ 2 1. Orthogonal

More information

The Impact of the Earth s Movement through the Space on Measuring the Velocity of Light

The Impact of the Earth s Movement through the Space on Measuring the Velocity of Light Journal of Appled Matheatcs and Physcs, 6, 4, 68-78 Publshed Onlne June 6 n ScRes http://wwwscrporg/journal/jap http://dxdoorg/436/jap646 The Ipact of the Earth s Moeent through the Space on Measurng the

More information

Expectation propagation

Expectation propagation Expectaton propagaton Lloyd Ellott May 17, 2011 Suppose p(x) s a pdf and we have a factorzaton p(x) = 1 Z n f (x). (1) =1 Expectaton propagaton s an nference algorthm desgned to approxmate the factors

More information

Markov Chain Monte Carlo Lecture 6

Markov Chain Monte Carlo Lecture 6 where (x 1,..., x N ) X N, N s called the populaton sze, f(x) f (x) for at least one {1, 2,..., N}, and those dfferent from f(x) are called the tral dstrbutons n terms of mportance samplng. Dfferent ways

More information

arxiv: v1 [math.co] 1 Mar 2014

arxiv: v1 [math.co] 1 Mar 2014 Unon-ntersectng set systems Gyula O.H. Katona and Dánel T. Nagy March 4, 014 arxv:1403.0088v1 [math.co] 1 Mar 014 Abstract Three ntersecton theorems are proved. Frst, we determne the sze of the largest

More information

Anti-van der Waerden numbers of 3-term arithmetic progressions.

Anti-van der Waerden numbers of 3-term arithmetic progressions. Ant-van der Waerden numbers of 3-term arthmetc progressons. Zhanar Berkkyzy, Alex Schulte, and Mchael Young Aprl 24, 2016 Abstract The ant-van der Waerden number, denoted by aw([n], k), s the smallest

More information

Properties of Least Squares

Properties of Least Squares Week 3 3.1 Smple Lnear Regresson Model 3. Propertes of Least Squares Estmators Y Y β 1 + β X + u weekly famly expendtures X weekly famly ncome For a gven level of x, the expected level of food expendtures

More information

Statistics for Managers Using Microsoft Excel/SPSS Chapter 13 The Simple Linear Regression Model and Correlation

Statistics for Managers Using Microsoft Excel/SPSS Chapter 13 The Simple Linear Regression Model and Correlation Statstcs for Managers Usng Mcrosoft Excel/SPSS Chapter 13 The Smple Lnear Regresson Model and Correlaton 1999 Prentce-Hall, Inc. Chap. 13-1 Chapter Topcs Types of Regresson Models Determnng the Smple Lnear

More information

Near Optimal Online Algorithms and Fast Approximation Algorithms for Resource Allocation Problems

Near Optimal Online Algorithms and Fast Approximation Algorithms for Resource Allocation Problems Near Optal Onlne Algorths and Fast Approxaton Algorths for Resource Allocaton Probles Nkhl R Devanur Kaal Jan Balasubraanan Svan Chrstopher A Wlkens Abstract We present algorths for a class of resource

More information

The Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD

The Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s

More information

2 STATISTICALLY OPTIMAL TRAINING DATA 2.1 A CRITERION OF OPTIMALITY We revew the crteron of statstcally optmal tranng data (Fukumzu et al., 1994). We

2 STATISTICALLY OPTIMAL TRAINING DATA 2.1 A CRITERION OF OPTIMALITY We revew the crteron of statstcally optmal tranng data (Fukumzu et al., 1994). We Advances n Neural Informaton Processng Systems 8 Actve Learnng n Multlayer Perceptrons Kenj Fukumzu Informaton and Communcaton R&D Center, Rcoh Co., Ltd. 3-2-3, Shn-yokohama, Yokohama, 222 Japan E-mal:

More information

SELECTED SOLUTIONS, SECTION (Weak duality) Prove that the primal and dual values p and d defined by equations (4.3.2) and (4.3.3) satisfy p d.

SELECTED SOLUTIONS, SECTION (Weak duality) Prove that the primal and dual values p and d defined by equations (4.3.2) and (4.3.3) satisfy p d. SELECTED SOLUTIONS, SECTION 4.3 1. Weak dualty Prove that the prmal and dual values p and d defned by equatons 4.3. and 4.3.3 satsfy p d. We consder an optmzaton problem of the form The Lagrangan for ths

More information

Random Partitions of Samples

Random Partitions of Samples Random Parttons of Samples Klaus Th. Hess Insttut für Mathematsche Stochastk Technsche Unverstät Dresden Abstract In the present paper we construct a decomposton of a sample nto a fnte number of subsamples

More information

Analytical Chemistry Calibration Curve Handout

Analytical Chemistry Calibration Curve Handout I. Quck-and Drty Excel Tutoral Analytcal Chemstry Calbraton Curve Handout For those of you wth lttle experence wth Excel, I ve provded some key technques that should help you use the program both for problem

More information

Pop-Click Noise Detection Using Inter-Frame Correlation for Improved Portable Auditory Sensing

Pop-Click Noise Detection Using Inter-Frame Correlation for Improved Portable Auditory Sensing Advanced Scence and Technology Letters, pp.164-168 http://dx.do.org/10.14257/astl.2013 Pop-Clc Nose Detecton Usng Inter-Frame Correlaton for Improved Portable Audtory Sensng Dong Yun Lee, Kwang Myung Jeon,

More information

Goodness of fit and Wilks theorem

Goodness of fit and Wilks theorem DRAFT 0.0 Glen Cowan 3 June, 2013 Goodness of ft and Wlks theorem Suppose we model data y wth a lkelhood L(µ) that depends on a set of N parameters µ = (µ 1,...,µ N ). Defne the statstc t µ ln L(µ) L(ˆµ),

More information

Algorithm for reduction of Element Calculus to Element Algebra

Algorithm for reduction of Element Calculus to Element Algebra Algorth for reducton of Eleent Calculus to Eleent Algebra Introducton M. Manukyan, V. Harutunyan The XML databases currently act defnte nterest aong researchers of databases for the followng reasons: 1.

More information

Fundamental loop-current method using virtual voltage sources technique for special cases

Fundamental loop-current method using virtual voltage sources technique for special cases Fundamental loop-current method usng vrtual voltage sources technque for specal cases George E. Chatzaraks, 1 Marna D. Tortorel 1 and Anastasos D. Tzolas 1 Electrcal and Electroncs Engneerng Departments,

More information

CS 468 Lecture 16: Isometry Invariance and Spectral Techniques

CS 468 Lecture 16: Isometry Invariance and Spectral Techniques CS 468 Lecture 16: Isometry Invarance and Spectral Technques Justn Solomon Scrbe: Evan Gawlk Introducton. In geometry processng, t s often desrable to characterze the shape of an object n a manner that

More information

Problem Set 9 Solutions

Problem Set 9 Solutions Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem

More information

Lossy Compression. Compromise accuracy of reconstruction for increased compression.

Lossy Compression. Compromise accuracy of reconstruction for increased compression. Lossy Compresson Compromse accuracy of reconstructon for ncreased compresson. The reconstructon s usually vsbly ndstngushable from the orgnal mage. Typcally, one can get up to 0:1 compresson wth almost

More information

Supplement to Clustering with Statistical Error Control

Supplement to Clustering with Statistical Error Control Supplement to Clusterng wth Statstcal Error Control Mchael Vogt Unversty of Bonn Matthas Schmd Unversty of Bonn In ths supplement, we provde the proofs that are omtted n the paper. In partcular, we derve

More information

Lecture 9: Linear regression: centering, hypothesis testing, multiple covariates, and confounding

Lecture 9: Linear regression: centering, hypothesis testing, multiple covariates, and confounding Recall: man dea of lnear regresson Lecture 9: Lnear regresson: centerng, hypothess testng, multple covarates, and confoundng Sandy Eckel seckel@jhsph.edu 6 May 8 Lnear regresson can be used to study an

More information

Integrals and Invariants of Euler-Lagrange Equations

Integrals and Invariants of Euler-Lagrange Equations Lecture 16 Integrals and Invarants of Euler-Lagrange Equatons ME 256 at the Indan Insttute of Scence, Bengaluru Varatonal Methods and Structural Optmzaton G. K. Ananthasuresh Professor, Mechancal Engneerng,

More information

Denote the function derivatives f(x) in given points. x a b. Using relationships (1.2), polynomials (1.1) are written in the form

Denote the function derivatives f(x) in given points. x a b. Using relationships (1.2), polynomials (1.1) are written in the form SET OF METHODS FO SOUTION THE AUHY POBEM FO STIFF SYSTEMS OF ODINAY DIFFEENTIA EUATIONS AF atypov and YuV Nulchev Insttute of Theoretcal and Appled Mechancs SB AS 639 Novosbrs ussa Introducton A constructon

More information

E Tail Inequalities. E.1 Markov s Inequality. Non-Lecture E: Tail Inequalities

E Tail Inequalities. E.1 Markov s Inequality. Non-Lecture E: Tail Inequalities Algorthms Non-Lecture E: Tal Inequaltes If you hold a cat by the tal you learn thngs you cannot learn any other way. Mar Twan E Tal Inequaltes The smple recursve structure of sp lsts made t relatvely easy

More information

FUZZY MODEL FOR FORECASTING INTEREST RATE OF BANK INDONESIA CERTIFICATE

FUZZY MODEL FOR FORECASTING INTEREST RATE OF BANK INDONESIA CERTIFICATE he 3 rd Internatonal Conference on Quanttatve ethods ISBN 979-989 Used n Econoc and Busness. June 6-8, 00 FUZZY ODEL FOR FORECASING INERES RAE OF BANK INDONESIA CERIFICAE Agus aan Abad, Subanar, Wdodo

More information

Lecture 10: Euler s Equations for Multivariable

Lecture 10: Euler s Equations for Multivariable Lecture 0: Euler s Equatons for Multvarable Problems Let s say we re tryng to mnmze an ntegral of the form: {,,,,,, ; } J f y y y y y y d We can start by wrtng each of the y s as we dd before: y (, ) (

More information

Chapter 2 - The Simple Linear Regression Model S =0. e i is a random error. S β2 β. This is a minimization problem. Solution is a calculus exercise.

Chapter 2 - The Simple Linear Regression Model S =0. e i is a random error. S β2 β. This is a minimization problem. Solution is a calculus exercise. Chapter - The Smple Lnear Regresson Model The lnear regresson equaton s: where y + = β + β e for =,..., y and are observable varables e s a random error How can an estmaton rule be constructed for the

More information

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data Condtonal Random Felds: Probablstc Models for Segmentng and Labelng Sequence Data Paper by John Lafferty, Andrew McCallum, and Fernando Perera ICML 2001 Presentaton by Joe Drsh May 9, 2002 Man Goals Present

More information

e i is a random error

e i is a random error Chapter - The Smple Lnear Regresson Model The lnear regresson equaton s: where + β + β e for,..., and are observable varables e s a random error How can an estmaton rule be constructed for the unknown

More information

Which Separator? Spring 1

Which Separator? Spring 1 Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal

More information

Pulse Coded Modulation

Pulse Coded Modulation Pulse Coded Modulaton PCM (Pulse Coded Modulaton) s a voce codng technque defned by the ITU-T G.711 standard and t s used n dgtal telephony to encode the voce sgnal. The frst step n the analog to dgtal

More information

Lecture 5 Decoding Binary BCH Codes

Lecture 5 Decoding Binary BCH Codes Lecture 5 Decodng Bnary BCH Codes In ths class, we wll ntroduce dfferent methods for decodng BCH codes 51 Decodng the [15, 7, 5] 2 -BCH Code Consder the [15, 7, 5] 2 -code C we ntroduced n the last lecture

More information

Support Vector Machines. Vibhav Gogate The University of Texas at dallas

Support Vector Machines. Vibhav Gogate The University of Texas at dallas Support Vector Machnes Vbhav Gogate he Unversty of exas at dallas What We have Learned So Far? 1. Decson rees. Naïve Bayes 3. Lnear Regresson 4. Logstc Regresson 5. Perceptron 6. Neural networks 7. K-Nearest

More information

Dirichlet s Theorem In Arithmetic Progressions

Dirichlet s Theorem In Arithmetic Progressions Drchlet s Theorem In Arthmetc Progressons Parsa Kavkan Hang Wang The Unversty of Adelade February 26, 205 Abstract The am of ths paper s to ntroduce and prove Drchlet s theorem n arthmetc progressons,

More information

Linear Momentum. Center of Mass.

Linear Momentum. Center of Mass. Lecture 16 Chapter 9 Physcs I 11.06.2013 Lnear oentu. Center of ass. Course webste: http://faculty.ul.edu/ndry_danylov/teachng/physcsi Lecture Capture: http://echo360.ul.edu/danylov2013/physcs1fall.htl

More information

Color Rendering Uncertainty

Color Rendering Uncertainty Australan Journal of Basc and Appled Scences 4(10): 4601-4608 010 ISSN 1991-8178 Color Renderng Uncertanty 1 A.el Bally M.M. El-Ganany 3 A. Al-amel 1 Physcs Department Photometry department- NIS Abstract:

More information

DISCRIMINANTS AND RAMIFIED PRIMES. 1. Introduction A prime number p is said to be ramified in a number field K if the prime ideal factorization

DISCRIMINANTS AND RAMIFIED PRIMES. 1. Introduction A prime number p is said to be ramified in a number field K if the prime ideal factorization DISCRIMINANTS AND RAMIFIED PRIMES KEITH CONRAD 1. Introducton A prme number p s sad to be ramfed n a number feld K f the prme deal factorzaton (1.1) (p) = po K = p e 1 1 peg g has some e greater than 1.

More information

Simulation and Random Number Generation

Simulation and Random Number Generation Smulaton and Random Number Generaton Summary Dscrete Tme vs Dscrete Event Smulaton Random number generaton Generatng a random sequence Generatng random varates from a Unform dstrbuton Testng the qualty

More information

Lecture 12: Classification

Lecture 12: Classification Lecture : Classfcaton g Dscrmnant functons g The optmal Bayes classfer g Quadratc classfers g Eucldean and Mahalanobs metrcs g K Nearest Neghbor Classfers Intellgent Sensor Systems Rcardo Guterrez-Osuna

More information

6.842 Randomness and Computation February 18, Lecture 4

6.842 Randomness and Computation February 18, Lecture 4 6.842 Randomness and Computaton February 18, 2014 Lecture 4 Lecturer: Rontt Rubnfeld Scrbe: Amartya Shankha Bswas Topcs 2-Pont Samplng Interactve Proofs Publc cons vs Prvate cons 1 Two Pont Samplng 1.1

More information

04 - Treaps. Dr. Alexander Souza

04 - Treaps. Dr. Alexander Souza Algorths Theory 04 - Treaps Dr. Alexander Souza The dctonary proble Gven: Unverse (U,

More information

On Markov chain-gambler's ruin problem with ties allowed

On Markov chain-gambler's ruin problem with ties allowed The 48rd Annual Conference on Statstcs Coputer Scences and Operatons Research Dec 23 On Markov chan-gabler's run proble wth tes allowed M. A. El-Shehawey * M. E. El-Tantawey ** and Gh. A. Al-Shreef ***

More information

Other NN Models. Reinforcement learning (RL) Probabilistic neural networks

Other NN Models. Reinforcement learning (RL) Probabilistic neural networks Other NN Models Renforcement learnng (RL) Probablstc neural networks Support vector machne (SVM) Renforcement learnng g( (RL) Basc deas: Supervsed dlearnng: (delta rule, BP) Samples (x, f(x)) to learn

More information

On the size of quotient of two subsets of positive integers.

On the size of quotient of two subsets of positive integers. arxv:1706.04101v1 [math.nt] 13 Jun 2017 On the sze of quotent of two subsets of postve ntegers. Yur Shtenkov Abstract We obtan non-trval lower bound for the set A/A, where A s a subset of the nterval [1,

More information

4DVAR, according to the name, is a four-dimensional variational method.

4DVAR, according to the name, is a four-dimensional variational method. 4D-Varatonal Data Assmlaton (4D-Var) 4DVAR, accordng to the name, s a four-dmensonal varatonal method. 4D-Var s actually a drect generalzaton of 3D-Var to handle observatons that are dstrbuted n tme. The

More information

Games of Threats. Elon Kohlberg Abraham Neyman. Working Paper

Games of Threats. Elon Kohlberg Abraham Neyman. Working Paper Games of Threats Elon Kohlberg Abraham Neyman Workng Paper 18-023 Games of Threats Elon Kohlberg Harvard Busness School Abraham Neyman The Hebrew Unversty of Jerusalem Workng Paper 18-023 Copyrght 2017

More information

Machine Learning. What is a good Decision Boundary? Support Vector Machines

Machine Learning. What is a good Decision Boundary? Support Vector Machines Machne Learnng 0-70/5 70/5-78 78 Sprng 200 Support Vector Machnes Erc Xng Lecture 7 March 5 200 Readng: Chap. 6&7 C.B book and lsted papers Erc Xng @ CMU 2006-200 What s a good Decson Boundar? Consder

More information

Statistical Mechanics and Combinatorics : Lecture III

Statistical Mechanics and Combinatorics : Lecture III Statstcal Mechancs and Combnatorcs : Lecture III Dmer Model Dmer defntons Defnton A dmer coverng (perfect matchng) of a fnte graph s a set of edges whch covers every vertex exactly once, e every vertex

More information

Lecture 6/7 (February 10/12, 2014) DIRAC EQUATION. The non-relativistic Schrödinger equation was obtained by noting that the Hamiltonian 2

Lecture 6/7 (February 10/12, 2014) DIRAC EQUATION. The non-relativistic Schrödinger equation was obtained by noting that the Hamiltonian 2 P470 Lecture 6/7 (February 10/1, 014) DIRAC EQUATION The non-relatvstc Schrödnger equaton was obtaned by notng that the Hamltonan H = P (1) m can be transformed nto an operator form wth the substtutons

More information

12. The Hamilton-Jacobi Equation Michael Fowler

12. The Hamilton-Jacobi Equation Michael Fowler 1. The Hamlton-Jacob Equaton Mchael Fowler Back to Confguraton Space We ve establshed that the acton, regarded as a functon of ts coordnate endponts and tme, satsfes ( ) ( ) S q, t / t+ H qpt,, = 0, and

More information

Topic 23 - Randomized Complete Block Designs (RCBD)

Topic 23 - Randomized Complete Block Designs (RCBD) Topc 3 ANOVA (III) 3-1 Topc 3 - Randomzed Complete Block Desgns (RCBD) Defn: A Randomzed Complete Block Desgn s a varant of the completely randomzed desgn (CRD) that we recently learned. In ths desgn,

More information

Chapter 13. Gas Mixtures. Study Guide in PowerPoint. Thermodynamics: An Engineering Approach, 5th edition by Yunus A. Çengel and Michael A.

Chapter 13. Gas Mixtures. Study Guide in PowerPoint. Thermodynamics: An Engineering Approach, 5th edition by Yunus A. Çengel and Michael A. Chapter 3 Gas Mxtures Study Gude n PowerPont to accopany Therodynacs: An Engneerng Approach, 5th edton by Yunus A. Çengel and Mchael A. Boles The dscussons n ths chapter are restrcted to nonreactve deal-gas

More information

Robust Algorithms for Preemptive Scheduling

Robust Algorithms for Preemptive Scheduling DOI 0.007/s00453-0-978-3 Robust Algorths for Preeptve Schedulng Leah Epsten Asaf Levn Receved: 4 March 0 / Accepted: 9 Noveber 0 Sprnger Scence+Busness Meda New York 0 Abstract Preeptve schedulng probles

More information

Lecture 19. Endogenous Regressors and Instrumental Variables

Lecture 19. Endogenous Regressors and Instrumental Variables Lecture 19. Endogenous Regressors and Instrumental Varables In the prevous lecture we consder a regresson model (I omt the subscrpts (1) Y β + D + u = 1 β The problem s that the dummy varable D s endogenous,.e.

More information

What would be a reasonable choice of the quantization step Δ?

What would be a reasonable choice of the quantization step Δ? CE 108 HOMEWORK 4 EXERCISE 1. Suppose you are samplng the output of a sensor at 10 KHz and quantze t wth a unform quantzer at 10 ts per sample. Assume that the margnal pdf of the sgnal s Gaussan wth mean

More information