Supplemental Material: Proofs
|
|
- Cecilia Brown
- 5 years ago
- Views:
Transcription
1 Proof to Theorem Supplemetal Material: Proofs Proof. Let be the miimal umber of traiig items to esure a uique solutio θ. First cosider the case. It happes if ad oly if θ ad Rak(A) d, which is a special case of Aθ. Clearly, this case is cosistet with LB. Next cosider the case. Sice θ solves (), the KKT coditio holds: We seek all δ such that θ δ satisfies λaθ l(x i θ, y i )x i. (29) A(θ δ) Aθ ad x i (θ δ) x i θ i,,, (3) For ay such δ, simple algebra verifies that θ tδ satisfies the KKT coditio (29) for ay t [, ]. Cosequetly, θ δ also solves the problem i (). To see this, we cosider two situatios: If the loss fuctio l(, ) is covex i the first argumet, the KKT coditio is a sufficiet optimality coditio, which meas that θ δ solves (). If the loss fuctio l(, ) is smooth (ot ecessary covex) i the first argumet, we have f(θ ) f(θ δ) by usig the Taylor expasio (recall f is defied i equatio ): f(θ δ) f(θ ) f(θ tδ), δ (for some t [, ]) f(θ ) l(x i (θ tδ), y i )x i λa(θ tδ)), δ f(θ ) f(θ ). l(x i θ, y i )x i λaθ, δ } {{ } due to the KKT coditio (29) Therefore, θ δ also solves (). However, the uiqueess of θ requires δ to be the oly value satisfyig (3). This is equivalet to say It idicates that Null(A) Null(Spa{x,, x }) {}. (3) Rak(A) Dim(Spa{x,, x }) d. From Dim(spa{x,, x }), we have d Rak(A). We proved the geeral case for LB. If we have Aθ, we ca further improve LB. Let g (g,..., g ) be the vector satisfyig λaθ gi x i ad gi l(x i θ, y i ) i, 2,,. (32) Sice θ satisfies the KKT coditio, such vector g must exist. Applyig Aθ to (32), we have g ad Dim (Spa{A., A.2,, A.d } Spa{x,, x }). (33) To satisfy (3), we must have d Dim (Spa{A., A.2,, A.d, x,, x }).
2 Usig the fact i liear algebra Dim (Spa{A., A.2,, A.d, x,, x }) Dim (Spa{A., A.2,, A.d }) Rak(A) Dim (Spa{x,, x }) Dim (Spa{A., A.2,, A.d } Spa{x,, x }) (from (33)) We coclude that d Rak(A). We completed the proof for LB. Proof to Theorem 2 Proof. Whe A has full rak we have a equivalet expressio for the KKT coditio (29): λa 2 θ A 2 xi l(x i θ, y i ) i,,. (34) Let us decompose A 2 x i for all i,, ito A 2 x i α i A 2 θ u i, where u i is orthogoal to A 2 θ : u i A 2 θ. Equivaletly x i α i Aθ A 2 u i. Applyig this decompositio, we have Puttig it back i (34) we obtai x i θ α i θ 2 A u i A 2 θ α i θ 2 A. ) λa 2 θ (α i A 2 θ u i l(α i θ 2 A, y i ) i,,. (35) Sice u i is orthogoal to A 2 θ, (35) ca be rewritte as satisfyig α i R, y i Y, g i l(α i θ 2 A, y i ) i,, (36) g i u i λa 2 θ A 2 θ α i g i (37) Sice Aθ, we have A 2 θ ad (37) is equivalet to λ α ig i. It follows that λ α i g i sup αg α R,y Y,g l(α θ 2 A,y) sup αg α R,y Y,g l(α θ 2 A,y) It idicates the lower boud for λ. sup α R,y Y,g l(α θ 2 A,y) αg Proof to Theorem 3
3 Proof. Let D {x i, y i },, be a teachig set for [w ; b ]. The followig KKT coditio eeds to be satisfied: [ ] [ ] l(y i (x i w b xi λaw ))y i. (38) If we costruct a ew traiig set } ˆD {ˆx i x i b w 2 Aw, ŷ i y i A,, the [w ; ] satisfies the KKT coditio defied o ˆD. This ca be verified as follows: ] [ ] [ˆxi l(ŷ i (ˆx i w λaw ))ŷ i [ ] l(y i (x i w b x i b Aw [ ] ))y w λaw 2 i A [ ] [ ] [ ] l(y i (x i w b xi λaw b Aw ))y i w 2 A l(y i (x i w b ))y i from (38) from (38) (39) where l(y i(x i w b ))y i is from the bias dimesio i (38). It follows that which is equivalet to l(ŷ iˆx i w )ŷ iˆx i λaw l(ŷ iˆx i w )A 2 ŷiˆx i λa 2 w :z i l(z i w )A 2 zi λa 2 w. (4) We decompose A 2 z i α i A 2 w u i where u i satisfies u i A 2 w. Applyig this decompositio to (4), we have λa 2 w l(α i w 2 A)(α i A 2 w u i ). (4) Sice u i is orthogoal to A 2 w, (4) implies that λa 2 w Sice w we have λ Together with we obtai LB3. l(α i w 2 A)α i A 2 w. l(α i w 2 A)α i. l(α i w 2 A)α i sup αg, α R,g l(α w 2 A )
4 Proof to Propositio Proof. We simply verify the KKT coditio to see that θ is a solutio to () by applyig the costructio i (). The uiqueess of θ is guarateed by the strog covexity of (). Proof to Propositio 2 Proof. We oly eed to verify that the KKT coditio holds for θ. Due to the strog covexity of (2) uiqueess is guarateed automatically. We deote the subgradiet a max( a, ) max( a, ) I(a), where The KKT coditio is, if a < I(a) [, ], if a. (42), otherwise y i x i max( y i x i θ, ) λθ y i x i I(y i x i θ ) λθ λθ ( λ θ λ θ 2 I 2 ) λ θ 2 λθ ( λ θ λθ 2 ) I λ θ 2 λθ ( ) where the last lie is due to I λ θ 2 λ θ 2 givig either the set [, ] or the value. Proof to Corollary 2 Proof. We show this umber matches LB2. Let A I, l(a, b) max( ab, ), ad cosider the deomiator of (7): sup αg sup αg α R,y Y,g l(α θ 2,y) α,y {,},g yi(yα θ 2 ) sup α,g I(α θ 2 ) αg θ 2 where the first equality is due to l(a, b) bi(ab). Therefore, LB2 λ θ 2 which matches the costructio i (3). Proof to Propositio 3 Proof. We first verify that θ is a solutio to (4) based o the teachig set costructio i (6). We oly eed to verify
5 the gradiet of (4) is zero. Computig the gradiet of (4), we have y i x i exp{y i x i θ } λθ ( exp {τ ( τ exp λ θ 2 λ θ 2, x i λ θ 2 λ θ 2 ) λ θ 2 λ θ 2 ( {τ λ θ 2 λ θ 2 )} λθ θ )} θ 2 λθ θ θ 2 λθ where the third equality uses the fact λ θ 2 λ θ 2 τmax ad the property a τ (a). The strog covexity e τ (a) of (4) automatically implies uiqueess. Proof to Corollary 3 Proof. We show that the umber matches LB2. I (7) let A I ad l(a, b) log( exp{ ab}). The deomiator of LB2 is: which implies LB2 Proof to Propositio 4 sup α R,y Y,g l(α θ 2,y) λ θ 2. αg sup αg α,y {,},gy(exp{yα θ 2 }) sup α,g(exp{α θ 2 }) αg sup α α exp{α θ 2 } t exp{t} θ 2 sup t θ 2, Proof. We first prove the case for w. We ca verify that the KKT coditio is satisfied by desigig x ad y as i (8): (x w b y )x λw x w b y. The uiqueess of [w ; b ] is idicated by the strog covexity of (7) whe. We the prove the case for w. With simple algebra, we ca verify the KKT coditio holds via the costructio i (9): (x w b y )x (x 2 w b y 2 )x 2 λw (x w b y ) (x 2 w b y 2 ). Similarly, the uiqueess is implied by the strog covexity of (7) whe 2.
6 Proof to Corollary 4 Proof. We match the lower boud LB i (6). Note θ [w ; b ] R d, ad A i this case is a (d ) (d ) matrix with the d d idetity matrix I d padded with oe additioal row ad colum of zeros for the offset. Therefore Rak(A) Rak(I d ) d. Whe w, Aθ ad LB (d ) Rak(A). Whe w, Aθ ad LB (d ) Rak(A) 2. These lower bouds match the teachig set sizes i (8) ad (9), respectively. Proof to Propositio 5 Proof. Ulike i previous learers (icludig homogeeous SVM), we o loger have strog covexity w.r.t. b. I order to prove that (2) is a teachig set, we eed to verify the KKT coditio ad verify solutio uiqueess. We first verify the KKT coditio to show that the solutio uder (2) icludes the target model [w ; b ]. From (2), we have x w b, x w b. (43) Applyig them to the KKT coditio ad usig the otatio i (42) we obtai [ ] 2 I(x w b x ) [ ] 2 I() x [ 2 I() x [ ] [ ] 2 I() x x λw [ I() w w ] [ ] λw 2 [ [ ] λw λw I(). ] [ ] 2 I( x w b x ) [ ] λw ] It proves that [w ; b ] solves (2) by our teachig set costructio. [ ] λw settig the last dimesio to applyig (2) observig λ w 2 Next we prove uiqueess by cotradictio. We use f(w, b) to deote the objective fuctio i (2) uder the teachig set. It is easy to verify that f(w, b ) λ 2 w 2. Assume that there exists aother solutio [ w; b] differet from [w ; b ]. We ca obtai w 2 w 2 due to λ 2 w 2 f(w, b ) f( w, b) λ 2 w 2. The secod equality is due to [ w; b] beig a solutio; the iequality is due to whole-part relatioship. Therefore, there are oly two possibilities for the orm of w: w w or w t w for some t <. Next we will show that both cases are impossible. (Case ) For the case w w, we have f( w, b) 2 max ( (x w b), ) 2 max ( (x w b), ) λ 2 w 2 2 max x (w w) (b b), 2 max x (w w) (b b), :Δ :Δ λ 2 w 2 2 max (Δ, ) 2 max (Δ, ) f(w, b ).
7 From f( w, b) f(w, b ), it follows Δ ad Δ. Sice Δ Δ (x x ) (w w) 2(w ) (w w) w w w w 2, we have w w w 2. But because w w, we must have w w. Applyig this ew observatio to Δ ad Δ, we obtai b b. It meas that [w ; b ] [ w; b], cotradictig our assumptio [w ; b ] [ w; b]. (Case 2) Next we tur to the case w t w for some t [, ). Recall our assumptio that [ w; b] solves (2). The it follows that the followig specific costructio [ŵ, ˆb] solves (2) as well: ŵ tw, ˆb tb. (44) To see this, we cosider the followig optimizatio problem: mi w,b s.t. L(w, b) : 2 max( (x w b), ) 2 max( (x w b), ) w t w. (45) Sice [ w; b] solves (2), it is easy to see that [ w; b] solves (45) too, otherwise there exists a solutio for (45) which gives a lower fuctio value o (2). The we ca verify that [ŵ; ˆb] solves (45) as well by showig the followig optimality coditio holds: [ ] L(w,b) w L(w,b) b [ŵ;ˆb] N w t w (ŵ, ˆb) Normal coe to the set {[w; b] : w t w } at [ŵ; ˆb] Give a covex closed set Ω ad a poit θ Ω, the ormal coe at poit θ is defied to be a set N Ω (θ) {φ : φ, ψ θ ψ Ω}. (46) The optimality coditio basically suggests that at the optimal poit, the egative (sub)gradiet directio overlaps with the ormal coe. I other words, there is ot ay directio to decrease the objective at the optimal poit. Readers ca refer to Bertsekas & Nedic (23) for more details about the geometric optimality coditio. Because of (43) ad (44), we have x ŵ ˆb t <. Thus at [ŵ; ˆb] the subgradiet is [ ] L(w,b) w L(w,b) 2 b [ŵ;ˆb] [ ] [ x x w ] w 2 (47) Ad the ormal coe is { [ ] } N w t w (ŵ, ˆb) w s s. (48) The itersectio is o-empty by choosig s w. Sice both [ŵ; ˆb] ad [ w; b] solve (45), we have L(ŵ, ˆb) L( w, b). 2 Together with ŵ w, we have f(ŵ, ˆb) L(ŵ, ˆb) λ 2 ŵ 2 f( w, b) f(w, b ). Therefore, we proved that [ŵ; ˆb] solves (2) as well. To see the cotradictio, let us check the fuctio value of f(ŵ, ˆb)
8 via a differet route: f(ŵ, ˆb) f(tw, tb ) 2 max ( t(x w b ), ) 2 max ( t, ) 2 max ( t(x w b ), ) λ 2 w 2 t 2 2 max ( t, ) λ 2 w 2 t 2 ( t) λ 2 w 2 ( t 2 ) λ 2 w 2 ( t) 2 ( t2 ) λ 2 w 2 2 ( t)2 f(w, b ) >f(w, b ), where the first iequality uses the fact that λ w 2. It cotradicts our early assertio f(ŵ, ˆb) f(w, b ). Puttig cases ad 2 together we prove uiqueess. Proof to Corollary 5 Proof. The upper boud directly follows Propositio 5. We oly eed to show the lower boud LB3 λ w 2 i Theorem 3. Let A I, l(a) max( a, ), ad cosider the deomiator of (9): sup αg sup αg α R,g l(α w 2 ) α,g I(α w 2 ) w 2 where the first equality is due to l(a) I(a). Therefore, LB3 λ w 2 which proves the lower boud. Proof to Propositio 6 Proof. We first poit out that for t to be well-defied the argumet to τ () has to be bouded λ w 2. This implies λ w 2. The size of our proposed teachig set is the smallest amog all such symmetric costructio that satisfy this costrait. We verify that the KKT coditio to show the costructio i (23) icludes the solutio [w ; b ]. From (23), we have x w b t x w b t. We apply them ad the teachig set costructio to compute the gradiet of (22): [ ] x 2 exp{x [ x w b } 2 exp{ x w b } [ ] x [ ] [ ] x λw 2 exp{t} 2 exp{t} [ ] [ ] t w λw w 2 exp{t} λ w 2 [ ] [ ] w λw w 2. This verifies the KKT coditio. ] [ ] λw
9 Fially we show uiqueess. The Hessia matrix of the objective fuctio (22) uder our traiig set (23) is: [ ] [ ] exp{t} x x x x x x I 2 ( exp{t}) }{{ 2 x x λ 2. } :a :A :B [ ] x [x Note a > ad A ] [ ] x [x ] is positive semi-defiite. We show that aaλb is positive defiite. Suppose ot. The there exists [u; v] such that [u; v] (aaλb)[u; v]. This implies [u; v] (aa)[u; v]λu u. Sice the first term is o-egative due to A beig positive semi-defiite, u. But the we have 2av 2 which implies [u; v], a cotradictio. Therefore uiqueess is guarateed. Proof to Corollary 6 Proof. The upper boud directly follows Propositio 6. We oly eed to show the lower boud LB3 i Theorem 3. Let A I ad l(a) log( exp{ a}) ad cosider the deomiator of (9): which implies LB3 λ w 2. sup αg sup αg α R,g l( α w 2 ) α,g(exp{α w 2 }) α sup α exp{α w 2 } w 2 t sup t exp{t} w 2, λ w 2 by applyig
b i u x i U a i j u x i u x j
M ath 5 2 7 Fall 2 0 0 9 L ecture 1 9 N ov. 1 6, 2 0 0 9 ) S ecod- Order Elliptic Equatios: Weak S olutios 1. Defiitios. I this ad the followig two lectures we will study the boudary value problem Here
More informationAlgebra of Least Squares
October 19, 2018 Algebra of Least Squares Geometry of Least Squares Recall that out data is like a table [Y X] where Y collects observatios o the depedet variable Y ad X collects observatios o the k-dimesioal
More informationEfficient GMM LECTURE 12 GMM II
DECEMBER 1 010 LECTURE 1 II Efficiet The estimator depeds o the choice of the weight matrix A. The efficiet estimator is the oe that has the smallest asymptotic variace amog all estimators defied by differet
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More informationLECTURE 8: ORTHOGONALITY (CHAPTER 5 IN THE BOOK)
LECTURE 8: ORTHOGONALITY (CHAPTER 5 IN THE BOOK) Everythig marked by is ot required by the course syllabus I this lecture, all vector spaces is over the real umber R. All vectors i R is viewed as a colum
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationGeometry of LS. LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT
OCTOBER 7, 2016 LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT Geometry of LS We ca thik of y ad the colums of X as members of the -dimesioal Euclidea space R Oe ca
More informationBrief Review of Functions of Several Variables
Brief Review of Fuctios of Several Variables Differetiatio Differetiatio Recall, a fuctio f : R R is differetiable at x R if ( ) ( ) lim f x f x 0 exists df ( x) Whe this limit exists we call it or f(
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationProduct measures, Tonelli s and Fubini s theorems For use in MAT3400/4400, autumn 2014 Nadia S. Larsen. Version of 13 October 2014.
Product measures, Toelli s ad Fubii s theorems For use i MAT3400/4400, autum 2014 Nadia S. Larse Versio of 13 October 2014. 1. Costructio of the product measure The purpose of these otes is to preset the
More informationDifferentiable Convex Functions
Differetiable Covex Fuctios The followig picture motivates Theorem 11. f ( x) f ( x) f '( x)( x x) ˆx x 1 Theorem 11 : Let f : R R be differetiable. The, f is covex o the covex set C R if, ad oly if for
More informationDefinition 4.2. (a) A sequence {x n } in a Banach space X is a basis for X if. unique scalars a n (x) such that x = n. a n (x) x n. (4.
4. BASES I BAACH SPACES 39 4. BASES I BAACH SPACES Sice a Baach space X is a vector space, it must possess a Hamel, or vector space, basis, i.e., a subset {x γ } γ Γ whose fiite liear spa is all of X ad
More information2.1. The Algebraic and Order Properties of R Definition. A binary operation on a set F is a function B : F F! F.
CHAPTER 2 The Real Numbers 2.. The Algebraic ad Order Properties of R Defiitio. A biary operatio o a set F is a fuctio B : F F! F. For the biary operatios of + ad, we replace B(a, b) by a + b ad a b, respectively.
More information1. By using truth tables prove that, for all statements P and Q, the statement
Author: Satiago Salazar Problems I: Mathematical Statemets ad Proofs. By usig truth tables prove that, for all statemets P ad Q, the statemet P Q ad its cotrapositive ot Q (ot P) are equivalet. I example.2.3
More informationOptimization Methods MIT 2.098/6.255/ Final exam
Optimizatio Methods MIT 2.098/6.255/15.093 Fial exam Date Give: December 19th, 2006 P1. [30 pts] Classify the followig statemets as true or false. All aswers must be well-justified, either through a short
More informationLinear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d
Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y
More informationAssignment 2 Solutions SOLUTION. ϕ 1 Â = 3 ϕ 1 4i ϕ 2. The other case can be dealt with in a similar way. { ϕ 2 Â} χ = { 4i ϕ 1 3 ϕ 2 } χ.
PHYSICS 34 QUANTUM PHYSICS II (25) Assigmet 2 Solutios 1. With respect to a pair of orthoormal vectors ϕ 1 ad ϕ 2 that spa the Hilbert space H of a certai system, the operator  is defied by its actio
More informationMachine Learning Theory Tübingen University, WS 2016/2017 Lecture 12
Machie Learig Theory Tübige Uiversity, WS 06/07 Lecture Tolstikhi Ilya Abstract I this lecture we derive risk bouds for kerel methods. We will start by showig that Soft Margi kerel SVM correspods to miimizig
More informationLecture 10 October Minimaxity and least favorable prior sequences
STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least
More informationHomework Set #3 - Solutions
EE 15 - Applicatios of Covex Optimizatio i Sigal Processig ad Commuicatios Dr. Adre Tkaceko JPL Third Term 11-1 Homework Set #3 - Solutios 1. a) Note that x is closer to x tha to x l i the Euclidea orm
More informationLinear Support Vector Machines
Liear Support Vector Machies David S. Roseberg The Support Vector Machie For a liear support vector machie (SVM), we use the hypothesis space of affie fuctios F = { f(x) = w T x + b w R d, b R } ad evaluate
More informationSummary and Discussion on Simultaneous Analysis of Lasso and Dantzig Selector
Summary ad Discussio o Simultaeous Aalysis of Lasso ad Datzig Selector STAT732, Sprig 28 Duzhe Wag May 4, 28 Abstract This is a discussio o the work i Bickel, Ritov ad Tsybakov (29). We begi with a short
More informationS1 Notation and Assumptions
Statistica Siica: Supplemet Robust-BD Estimatio ad Iferece for Varyig-Dimesioal Geeral Liear Models Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag Uiversity of Wiscosi-Madiso Supplemetary Material S Notatio
More information( ) (( ) ) ANSWERS TO EXERCISES IN APPENDIX B. Section B.1 VECTORS AND SETS. Exercise B.1-1: Convex sets. are convex, , hence. and. (a) Let.
Joh Riley 8 Jue 03 ANSWERS TO EXERCISES IN APPENDIX B Sectio B VECTORS AND SETS Exercise B-: Covex sets (a) Let 0 x, x X, X, hece 0 x, x X ad 0 x, x X Sice X ad X are covex, x X ad x X The x X X, which
More informationTechnical Proofs for Homogeneity Pursuit
Techical Proofs for Homogeeity Pursuit bstract This is the supplemetal material for the article Homogeeity Pursuit, submitted for publicatio i Joural of the merica Statistical ssociatio. B Proofs B. Proof
More informationFundamental Theorem of Algebra. Yvonne Lai March 2010
Fudametal Theorem of Algebra Yvoe Lai March 010 We prove the Fudametal Theorem of Algebra: Fudametal Theorem of Algebra. Let f be a o-costat polyomial with real coefficiets. The f has at least oe complex
More informationMATH 324 Summer 2006 Elementary Number Theory Solutions to Assignment 2 Due: Thursday July 27, 2006
MATH 34 Summer 006 Elemetary Number Theory Solutios to Assigmet Due: Thursday July 7, 006 Departmet of Mathematical ad Statistical Scieces Uiversity of Alberta Questio [p 74 #6] Show that o iteger of the
More informationAPPENDIX A SMO ALGORITHM
AENDIX A SMO ALGORITHM Sequetial Miimal Optimizatio SMO) is a simple algorithm that ca quickly solve the SVM Q problem without ay extra matrix storage ad without usig time-cosumig umerical Q optimizatio
More informationMath 451: Euclidean and Non-Euclidean Geometry MWF 3pm, Gasson 204 Homework 3 Solutions
Math 451: Euclidea ad No-Euclidea Geometry MWF 3pm, Gasso 204 Homework 3 Solutios Exercises from 1.4 ad 1.5 of the otes: 4.3, 4.10, 4.12, 4.14, 4.15, 5.3, 5.4, 5.5 Exercise 4.3. Explai why Hp, q) = {x
More information10-701/ Machine Learning Mid-term Exam Solution
0-70/5-78 Machie Learig Mid-term Exam Solutio Your Name: Your Adrew ID: True or False (Give oe setece explaatio) (20%). (F) For a cotiuous radom variable x ad its probability distributio fuctio p(x), it
More informationOn Nonsingularity of Saddle Point Matrices. with Vectors of Ones
Iteratioal Joural of Algebra, Vol. 2, 2008, o. 4, 197-204 O Nosigularity of Saddle Poit Matrices with Vectors of Oes Tadeusz Ostrowski Istitute of Maagemet The State Vocatioal Uiversity -400 Gorzów, Polad
More information1 Duality revisited. AM 221: Advanced Optimization Spring 2016
AM 22: Advaced Optimizatio Sprig 206 Prof. Yaro Siger Sectio 7 Wedesday, Mar. 9th Duality revisited I this sectio, we will give a slightly differet perspective o duality. optimizatio program: f(x) x R
More informationAssignment 5: Solutions
McGill Uiversity Departmet of Mathematics ad Statistics MATH 54 Aalysis, Fall 05 Assigmet 5: Solutios. Let y be a ubouded sequece of positive umbers satisfyig y + > y for all N. Let x be aother sequece
More informationWeek 5-6: The Binomial Coefficients
Wee 5-6: The Biomial Coefficiets March 6, 2018 1 Pascal Formula Theorem 11 (Pascal s Formula For itegers ad such that 1, ( ( ( 1 1 + 1 The umbers ( 2 ( 1 2 ( 2 are triagle umbers, that is, The petago umbers
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationLinear Elliptic PDE s Elliptic partial differential equations frequently arise out of conservation statements of the form
Liear Elliptic PDE s Elliptic partial differetial equatios frequetly arise out of coservatio statemets of the form B F d B Sdx B cotaied i bouded ope set U R. Here F, S deote respectively, the flux desity
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationCHAPTER 5 SOME MINIMAX AND SADDLE POINT THEOREMS
CHAPTR 5 SOM MINIMA AND SADDL POINT THORMS 5. INTRODUCTION Fied poit theorems provide importat tools i game theory which are used to prove the equilibrium ad eistece theorems. For istace, the fied poit
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationMathematical Induction
Mathematical Iductio Itroductio Mathematical iductio, or just iductio, is a proof techique. Suppose that for every atural umber, P() is a statemet. We wish to show that all statemets P() are true. I a
More informationw (1) ˆx w (1) x (1) /ρ and w (2) ˆx w (2) x (2) /ρ.
2 5. Weighted umber of late jobs 5.1. Release dates ad due dates: maximimizig the weight of o-time jobs Oce we add release dates, miimizig the umber of late jobs becomes a sigificatly harder problem. For
More informationDefinitions and Theorems. where x are the decision variables. c, b, and a are constant coefficients.
Defiitios ad Theorems Remember the scalar form of the liear programmig problem, Miimize, Subject to, f(x) = c i x i a 1i x i = b 1 a mi x i = b m x i 0 i = 1,2,, where x are the decisio variables. c, b,
More informationn=1 a n is the sequence (s n ) n 1 n=1 a n converges to s. We write a n = s, n=1 n=1 a n
Series. Defiitios ad first properties A series is a ifiite sum a + a + a +..., deoted i short by a. The sequece of partial sums of the series a is the sequece s ) defied by s = a k = a +... + a,. k= Defiitio
More informationMATH 472 / SPRING 2013 ASSIGNMENT 2: DUE FEBRUARY 4 FINALIZED
MATH 47 / SPRING 013 ASSIGNMENT : DUE FEBRUARY 4 FINALIZED Please iclude a cover sheet that provides a complete setece aswer to each the followig three questios: (a) I your opiio, what were the mai ideas
More informationFirst, note that the LS residuals are orthogonal to the regressors. X Xb X y = 0 ( normal equations ; (k 1) ) So,
0 2. OLS Part II The OLS residuals are orthogoal to the regressors. If the model icludes a itercept, the orthogoality of the residuals ad regressors gives rise to three results, which have limited practical
More informationMAS111 Convergence and Continuity
MAS Covergece ad Cotiuity Key Objectives At the ed of the course, studets should kow the followig topics ad be able to apply the basic priciples ad theorems therei to solvig various problems cocerig covergece
More information1+x 1 + α+x. x = 2(α x2 ) 1+x
Math 2030 Homework 6 Solutios # [Problem 5] For coveiece we let α lim sup a ad β lim sup b. Without loss of geerality let us assume that α β. If α the by assumptio β < so i this case α + β. By Theorem
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationSingular Continuous Measures by Michael Pejic 5/14/10
Sigular Cotiuous Measures by Michael Peic 5/4/0 Prelimiaries Give a set X, a σ-algebra o X is a collectio of subsets of X that cotais X ad ad is closed uder complemetatio ad coutable uios hece, coutable
More informationMcGill University Math 354: Honors Analysis 3 Fall 2012 Solutions to selected problems
McGill Uiversity Math 354: Hoors Aalysis 3 Fall 212 Assigmet 3 Solutios to selected problems Problem 1. Lipschitz fuctios. Let Lip K be the set of all fuctios cotiuous fuctios o [, 1] satisfyig a Lipschitz
More informationSolutions to home assignments (sketches)
Matematiska Istitutioe Peter Kumli 26th May 2004 TMA401 Fuctioal Aalysis MAN670 Applied Fuctioal Aalysis 4th quarter 2003/2004 All documet cocerig the course ca be foud o the course home page: http://www.math.chalmers.se/math/grudutb/cth/tma401/
More informationSupplementary Materials for Statistical-Computational Phase Transitions in Planted Models: The High-Dimensional Setting
Supplemetary Materials for Statistical-Computatioal Phase Trasitios i Plated Models: The High-Dimesioal Settig Yudog Che The Uiversity of Califoria, Berkeley yudog.che@eecs.berkeley.edu Jiamig Xu Uiversity
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 3 9/11/2013. Large deviations Theory. Cramér s Theorem
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 3 9//203 Large deviatios Theory. Cramér s Theorem Cotet.. Cramér s Theorem. 2. Rate fuctio ad properties. 3. Chage of measure techique.
More informationLecture 3 The Lebesgue Integral
Lecture 3: The Lebesgue Itegral 1 of 14 Course: Theory of Probability I Term: Fall 2013 Istructor: Gorda Zitkovic Lecture 3 The Lebesgue Itegral The costructio of the itegral Uless expressly specified
More informationMath 61CM - Solutions to homework 3
Math 6CM - Solutios to homework 3 Cédric De Groote October 2 th, 208 Problem : Let F be a field, m 0 a fixed oegative iteger ad let V = {a 0 + a x + + a m x m a 0,, a m F} be the vector space cosistig
More informationEconomics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator
Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters
More informationMath 299 Supplement: Real Analysis Nov 2013
Math 299 Supplemet: Real Aalysis Nov 203 Algebra Axioms. I Real Aalysis, we work withi the axiomatic system of real umbers: the set R alog with the additio ad multiplicatio operatios +,, ad the iequality
More informationTENSOR PRODUCTS AND PARTIAL TRACES
Lecture 2 TENSOR PRODUCTS AND PARTIAL TRACES Stéphae ATTAL Abstract This lecture cocers special aspects of Operator Theory which are of much use i Quatum Mechaics, i particular i the theory of Quatum Ope
More informationTEACHER CERTIFICATION STUDY GUIDE
COMPETENCY 1. ALGEBRA SKILL 1.1 1.1a. ALGEBRAIC STRUCTURES Kow why the real ad complex umbers are each a field, ad that particular rigs are ot fields (e.g., itegers, polyomial rigs, matrix rigs) Algebra
More informationDimension-free PAC-Bayesian bounds for the estimation of the mean of a random vector
Dimesio-free PAC-Bayesia bouds for the estimatio of the mea of a radom vector Olivier Catoi CREST CNRS UMR 9194 Uiversité Paris Saclay olivier.catoi@esae.fr Ilaria Giulii Laboratoire de Probabilités et
More informationCHAPTER 5. Theory and Solution Using Matrix Techniques
A SERIES OF CLASS NOTES FOR 2005-2006 TO INTRODUCE LINEAR AND NONLINEAR PROBLEMS TO ENGINEERS, SCIENTISTS, AND APPLIED MATHEMATICIANS DE CLASS NOTES 3 A COLLECTION OF HANDOUTS ON SYSTEMS OF ORDINARY DIFFERENTIAL
More informationEmpirical Processes: Glivenko Cantelli Theorems
Empirical Processes: Gliveko Catelli Theorems Mouliath Baerjee Jue 6, 200 Gliveko Catelli classes of fuctios The reader is referred to Chapter.6 of Weller s Torgo otes, Chapter??? of VDVW ad Chapter 8.3
More informationChapter IV Integration Theory
Chapter IV Itegratio Theory Lectures 32-33 1. Costructio of the itegral I this sectio we costruct the abstract itegral. As a matter of termiology, we defie a measure space as beig a triple (, A, µ), where
More informationMATH10212 Linear Algebra B Proof Problems
MATH22 Liear Algebra Proof Problems 5 Jue 26 Each problem requests a proof of a simple statemet Problems placed lower i the list may use the results of previous oes Matrices ermiats If a b R the matrix
More informationSlide Set 13 Linear Model with Endogenous Regressors and the GMM estimator
Slide Set 13 Liear Model with Edogeous Regressors ad the GMM estimator Pietro Coretto pcoretto@uisa.it Ecoometrics Master i Ecoomics ad Fiace (MEF) Uiversità degli Studi di Napoli Federico II Versio: Friday
More informationQuestion 1: The magnetic case
September 6, 018 Corell Uiversity, Departmet of Physics PHYS 337, Advace E&M, HW # 4, due: 9/19/018, 11:15 AM Questio 1: The magetic case I class, we skipped over some details, so here you are asked to
More informationACO Comprehensive Exam 9 October 2007 Student code A. 1. Graph Theory
1. Graph Theory Prove that there exist o simple plaar triagulatio T ad two distict adjacet vertices x, y V (T ) such that x ad y are the oly vertices of T of odd degree. Do ot use the Four-Color Theorem.
More informationON WELLPOSEDNESS QUADRATIC FUNCTION MINIMIZATION PROBLEM ON INTERSECTION OF TWO ELLIPSOIDS * M. JA]IMOVI], I. KRNI] 1.
Yugoslav Joural of Operatios Research 1 (00), Number 1, 49-60 ON WELLPOSEDNESS QUADRATIC FUNCTION MINIMIZATION PROBLEM ON INTERSECTION OF TWO ELLIPSOIDS M. JA]IMOVI], I. KRNI] Departmet of Mathematics
More information7 Sequences of real numbers
40 7 Sequeces of real umbers 7. Defiitios ad examples Defiitio 7... A sequece of real umbers is a real fuctio whose domai is the set N of atural umbers. Let s : N R be a sequece. The the values of s are
More informationChapter 7 Isoperimetric problem
Chapter 7 Isoperimetric problem Recall that the isoperimetric problem (see the itroductio its coectio with ido s proble) is oe of the most classical problem of a shape optimizatio. It ca be formulated
More informationBoosting. Professor Ameet Talwalkar. Professor Ameet Talwalkar CS260 Machine Learning Algorithms March 1, / 32
Boostig Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machie Learig Algorithms March 1, 2017 1 / 32 Outlie 1 Admiistratio 2 Review of last lecture 3 Boostig Professor Ameet Talwalkar CS260
More informationChapter 2. Periodic points of toral. automorphisms. 2.1 General introduction
Chapter 2 Periodic poits of toral automorphisms 2.1 Geeral itroductio The automorphisms of the two-dimesioal torus are rich mathematical objects possessig iterestig geometric, algebraic, topological ad
More informationComplex Analysis Spring 2001 Homework I Solution
Complex Aalysis Sprig 2001 Homework I Solutio 1. Coway, Chapter 1, sectio 3, problem 3. Describe the set of poits satisfyig the equatio z a z + a = 2c, where c > 0 ad a R. To begi, we see from the triagle
More information4x 2. (n+1) x 3 n+1. = lim. 4x 2 n+1 n3 n. n 4x 2 = lim = 3
Exam Problems (x. Give the series (, fid the values of x for which this power series coverges. Also =0 state clearly what the radius of covergece is. We start by settig up the Ratio Test: x ( x x ( x x
More informationBertrand s Postulate
Bertrad s Postulate Lola Thompso Ross Program July 3, 2009 Lola Thompso (Ross Program Bertrad s Postulate July 3, 2009 1 / 33 Bertrad s Postulate I ve said it oce ad I ll say it agai: There s always a
More informationIntroduction to Machine Learning DIS10
CS 189 Fall 017 Itroductio to Machie Learig DIS10 1 Fu with Lagrage Multipliers (a) Miimize the fuctio such that f (x,y) = x + y x + y = 3. Solutio: The Lagragia is: L(x,y,λ) = x + y + λ(x + y 3) Takig
More informationDeterminants of order 2 and 3 were defined in Chapter 2 by the formulae (5.1)
5. Determiats 5.. Itroductio 5.2. Motivatio for the Choice of Axioms for a Determiat Fuctios 5.3. A Set of Axioms for a Determiat Fuctio 5.4. The Determiat of a Diagoal Matrix 5.5. The Determiat of a Upper
More informationSequences and Limits
Chapter Sequeces ad Limits Let { a } be a sequece of real or complex umbers A ecessary ad sufficiet coditio for the sequece to coverge is that for ay ɛ > 0 there exists a iteger N > 0 such that a p a q
More informationMachine Learning Theory Tübingen University, WS 2016/2017 Lecture 11
Machie Learig Theory Tübige Uiversity, WS 06/07 Lecture Tolstikhi Ilya Abstract We will itroduce the otio of reproducig kerels ad associated Reproducig Kerel Hilbert Spaces (RKHS). We will cosider couple
More informationOn forward improvement iteration for stopping problems
O forward improvemet iteratio for stoppig problems Mathematical Istitute, Uiversity of Kiel, Ludewig-Mey-Str. 4, D-24098 Kiel, Germay irle@math.ui-iel.de Albrecht Irle Abstract. We cosider the optimal
More information1 Introduction. 1.1 Notation and Terminology
1 Itroductio You have already leared some cocepts of calculus such as limit of a sequece, limit, cotiuity, derivative, ad itegral of a fuctio etc. Real Aalysis studies them more rigorously usig a laguage
More informationLecture 10: Bounded Linear Operators and Orthogonality in Hilbert Spaces
Lecture : Bouded Liear Operators ad Orthogoality i Hilbert Spaces 34 Bouded Liear Operator Let ( X, ), ( Y, ) i i be ored liear vector spaces ad { } X Y The, T is said to be bouded if a real uber c such
More informationAnalytic Continuation
Aalytic Cotiuatio The stadard example of this is give by Example Let h (z) = 1 + z + z 2 + z 3 +... kow to coverge oly for z < 1. I fact h (z) = 1/ (1 z) for such z. Yet H (z) = 1/ (1 z) is defied for
More informationArkansas Tech University MATH 2924: Calculus II Dr. Marcel B. Finan
Arkasas Tech Uiversity MATH 94: Calculus II Dr Marcel B Fia 85 Power Series Let {a } =0 be a sequece of umbers The a power series about x = a is a series of the form a (x a) = a 0 + a (x a) + a (x a) +
More informationSOLUTIONS TO EXAM 3. Solution: Note that this defines two convergent geometric series with respective radii r 1 = 2/5 < 1 and r 2 = 1/5 < 1.
SOLUTIONS TO EXAM 3 Problem Fid the sum of the followig series 2 + ( ) 5 5 2 5 3 25 2 2 This series diverges Solutio: Note that this defies two coverget geometric series with respective radii r 2/5 < ad
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationBeurling Integers: Part 2
Beurlig Itegers: Part 2 Isomorphisms Devi Platt July 11, 2015 1 Prime Factorizatio Sequeces I the last article we itroduced the Beurlig geeralized itegers, which ca be represeted as a sequece of real umbers
More informationUSA Mathematical Talent Search Round 3 Solutions Year 27 Academic Year
/3/27. Fill i each space of the grid with either a or a so that all sixtee strigs of four cosecutive umbers across ad dow are distict. You do ot eed to prove that your aswer is the oly oe possible; you
More informationCSE 1400 Applied Discrete Mathematics Number Theory and Proofs
CSE 1400 Applied Discrete Mathematics Number Theory ad Proofs Departmet of Computer Scieces College of Egieerig Florida Tech Sprig 01 Problems for Number Theory Backgroud Number theory is the brach of
More information5.1 Review of Singular Value Decomposition (SVD)
MGMT 69000: Topics i High-dimesioal Data Aalysis Falll 06 Lecture 5: Spectral Clusterig: Overview (cotd) ad Aalysis Lecturer: Jiamig Xu Scribe: Adarsh Barik, Taotao He, September 3, 06 Outlie Review of
More informationMa 4121: Introduction to Lebesgue Integration Solutions to Homework Assignment 5
Ma 42: Itroductio to Lebesgue Itegratio Solutios to Homework Assigmet 5 Prof. Wickerhauser Due Thursday, April th, 23 Please retur your solutios to the istructor by the ed of class o the due date. You
More informationStatistical and Mathematical Methods DS-GA 1002 December 8, Sample Final Problems Solutions
Statistical ad Mathematical Methods DS-GA 00 December 8, 05. Short questios Sample Fial Problems Solutios a. Ax b has a solutio if b is i the rage of A. The dimesio of the rage of A is because A has liearly-idepedet
More informationpage Suppose that S 0, 1 1, 2.
page 10 1. Suppose that S 0, 1 1,. a. What is the set of iterior poits of S? The set of iterior poits of S is 0, 1 1,. b. Give that U is the set of iterior poits of S, evaluate U. 0, 1 1, 0, 1 1, S. The
More informationIntroduction to Optimization Techniques. How to Solve Equations
Itroductio to Optimizatio Techiques How to Solve Equatios Iterative Methods of Optimizatio Iterative methods of optimizatio Solutio of the oliear equatios resultig form a optimizatio problem is usually
More informationMachine Learning Brett Bernstein
Machie Learig Brett Berstei Week 2 Lecture: Cocept Check Exercises Starred problems are optioal. Excess Risk Decompositio 1. Let X = Y = {1, 2,..., 10}, A = {1,..., 10, 11} ad suppose the data distributio
More informationOutput Analysis and Run-Length Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationReview Problems 1. ICME and MS&E Refresher Course September 19, 2011 B = C = AB = A = A 2 = A 3... C 2 = C 3 = =
Review Problems ICME ad MS&E Refresher Course September 9, 0 Warm-up problems. For the followig matrices A = 0 B = C = AB = 0 fid all powers A,A 3,(which is A times A),... ad B,B 3,... ad C,C 3,... Solutio:
More information