Solutions HW #2. minimize. Ax = b. Give the dual problem, and make the implicit equality constraints explicit. Solution.
|
|
- Brook Curtis
- 5 years ago
- Views:
Transcription
1 Solutons HW #2 Dual of general LP. Fnd the dual functon of the LP mnmze subject to c T x Gx h Ax = b. Gve the dual problem, and make the mplct equalty constrants explct. Soluton. 1. The Lagrangan s L(x, λ, ν) = c T x + λ T (Gx h) + ν T (Ax b) = (c T + λ T G + ν T A)x hλ T ν T b, whch s an affne functon of x. It follows that the dual functon s gven by { λ g(λ, ν) = nf L(x, λ, ν) = T h ν T b c + G T λ + A T ν = 0 x otherwse. 2. The dual problem s maxmze g(λ, ν) subject to λ 0. After makng the mplct constrants explct, we obtan maxmze λ T h ν T b subject to c + G T λ + A T ν = 0 λ 0. Pecewse-lnear mnmzaton. We consder the convex pecewse-lnear mnmzaton problem wth varable x R n. mnmze max,...,m (a T x + b ) (1) 1. Derve a dual problem, based on the Lagrange dual of the equvalent problem wth varables x R n, y R m. mnmze max,...,m y subject to a T x + b = y, = 1,..., m, 2. Formulate the pecewse-lnear mnmzaton problem (1) as an LP, and form the dual of the LP. Relate the LP dual to the dual obtaned n part (a). 1
2 3. Suppose we approxmate the objectve functon n (1) by the smooth functon ) f 0 (x) = log exp(a T x + b ), and solve the unconstraned geometrc program mnmze log exp(at x + b ) ). (2) A dual of ths problem s gven by (5.62). Let p pwl and p gp be the optmal values of (1) and (2), respectvely. Show that 0 p gp p pwl log m. 4. Derve smlar bounds for the dfference between p pwl and the optmal value of mnmze (1/γ) log exp(γ(at x + b )) ), where γ > 0 s a parameter. What happens as we ncrease γ? Soluton. 1. The dual functon s g(λ) = nf x,y ( max y +,...,m ) λ (a T x + b y ). The nfmum over x s fnte only f λ a = 0. To mnmze over y we note that { nf (max y λ T 0 λ 0, 1 y) = T λ = 1 y otherwse. To prove ths, we frst note that f λ 0, 1 T λ = 1, then λ T y = j λ j y j j λ j max y = max y, wth equalty f y = 0, so n that case nf (max y λ T y) = 0. y If λ 0, say λ j < 0, then choosng y = 0, j, and y j = t, wth t 0, and lettng t go to nfnty, gves max y λ T y = 0 + tλ k. Fnally, f 1 T λ 1, choosng y = t1, gves max y λ T y = t(1 1 T λ), f t and 1 < 1 T λ, or f t and 1 > 1 T λ. 2
3 Summng up, we have { b g(λ) = λ λ a = 0, otherwse. λ 0, 1 T λ = 1 The resultng dual problem s 2. The problem s equvalent to the LP maxmze b T λ subject to A T λ = 0 1 T λ = 1 λ 0. The dual problem s mnmze t subject to Ax + b t1. maxmze b T z subject to A T z = 0, 1 T z = 1, z 0, whch s dentcal to the dual derved n (a). 3. Frst both prmal problems PWL and GP are convex so strong dualty holds. Suppose z s dual optmal for the dual GP (5.62) (wth the conventon 0 log 0 = 0) maxmze b T z m z log z subject to 1 T z = 1 A T z = 0 z 0 Then z s also feasble for the dual of the pecewse-lnear formulaton, wth objectve value Ths provdes a lower bound on p pwl : p pwl p gp + b T z = p gp + z log z. z log z p gp log m. The bound follows from concavty of log usng Jensens s nequalty On the other hand, we also have z log(1/z ) log 1 = log m. max(a T x + b ) log exp(a T x + b ) for all x, and therefore p pwl p gp. In concluson, p gp log m p pwl p gp. 3
4 4. We frst reformulate the problem as The Lagrangan s mnmze (1/γ) log m exp(γy ) subject to Ax + b = y. L(x, y, z) = 1 m γ log exp(γy ) + z T (Ax + b y). L s bounded below as a functon of x only f A T z = 0. To fnd the optmum over y, we compute the conjugate functon for z R m, f (z) = sup γz T y log y exp(γy ) = sup y If there exsts z < 0, then, takng y = te, we get z T y log exp(y ). z T y log exp(y ) = tz log((m 1) + e t ), whch goes to nfnty for t. If z 0 and 1 T z 1, then, takng y = t1, we get z T y log exp(y ) = t1 T z log(m) t, whch goes to nfnty for t + or dependng on the sgn of 1 T z 1. If z 0 and 1 T z = z 0 z = 1, we have usng concavty of log, z T y z log z = z (y log z ) log z exp(y log(z )) log exp(y ), z 0 z 0 wth equalty by takng y = log z f z > 0 and y for z = 0. Therefore we have f (z) = z log z. The Lagrange dual functon s then gven for z 0, 1 T z = 1, A T z = 0 by and the dual problem s g(z) = b T z 1 γ z log z, maxmze b T z (1/γ) m z log z subject to A T z = 0 1 T z = 1. 4
5 Let p gp(γ) be the optmal value of the GP. Followng the same argument as above, we can conclude that p gp(γ) 1 γ log m p pwl p gp(γ). In other words, p gp(γ) approaches p pwl as γ ncreases. Suboptmalty of a smple coverng ellpsod. Recall the problem of determnng the mnmum volume ellpsod, centered at the orgn, that contans the ponts a 1,..., a m R n (problem (5.14), page 222): mnmze f 0 (X) = log det(x 1 ) subject to a T Xa 1, = 1,..., m, wth dom f 0 = S n ++. We assume that the vectors a 1,..., a m span R n (whch mples that the problem s bounded below). 1. Show that the matrx ) 1 X sm = a k a T k, k=1 s feasble. Hnt. Show that [ m k=1 a ka T k a a T 1 ] 0, and use Schur complements ( A.5.5) to prove that a T Xa 1 for = 1,..., m. Soluton. [ m k=1 a ka T k a k a T 1 ] = [ k a ka T k ] [ a + 1 ] [ a 1 s the sum of two postve semdefnte matrces, hence postve semdefnte. The Schur complement of the 1, 1 block of ths matrx, m k=1 a ka T k whch s nvertble by hypothess, s therefore also postve semdefnte: ] T 1 a T ( m ) 1 a k a T k a 0, k=1 whch s the desred concluson. 2. Now we establsh a bound on how suboptmal the feasble pont X sm s, va the dual problem, maxmze log det λ a a T ) 1 T λ + n subject to λ 0, wth the mplct constrant m λ a a T 0. (Ths dual s derved on page 222.) To derve a bound, we restrct our attenton to dual varables of the form λ = t1, where t > 0. Fnd (analytcally) the optmal value of t, and evaluate the dual objectve at ths λ. Use ths to prove that the volume of the ellpsod {u u T X sm u 1} s no more than a factor (m/n) n/2 more than the volume of the mnmum volume ellpsod. 5
6 Soluton. The dual functon evaluated at λ = t1 s ) g(λ) = log det a a T + n log t mt + n. Now we ll maxmze over t > 0 to get the best lower bound. Settng the dervatve wth respect to t equal to zero yelds the optmal value t = n/m. Usng ths λ we get the dual objectve value ) g(λ) = log det a a T + n log(n/m). The prmal objectve value for X = X sm s gven by ) 1 log det a a T, so the dualty gap assocated wth X sm and λ s n log(m/n). (Recall that m n, by our assumpton that a 1,..., a m span R n.) It follows that, n terms of the objectve functon, X sm s no more than n log(m/n) suboptmal. The volume V of the ellpsod E assocated wth the matrx X s gven by V = exp( O/2), where O s the assocated objectve functon, O = log det X. The bound follows. Dual problem. Derve a dual problem for mnmze N A x + b 2 + (1/2) x x The problem data are A R m n, b R m, and x 0 R n. Frst ntroduce new varables y R m and equalty constrants y = A x + b. Soluton. The Lagrangan s L(x, z 1,..., z N ) = We frst mnmze over y. We have N y x x nf y ( y 2 + z T y ) = N z T (y A x b ). { 0 z 2 1 otherwse. (If z 2 > 1, choose y = tz and let t, to show that the functon s unbounded below. If z 2 1, t follows from the Cauchy-Schwarz nequalty that y 2 + z T y 0, so the mnmum s reached when y = 0.) We can mnmze over x by settng the gradent wth respect to x equal to zero. Ths yelds x = x 0 + N A T z. Substtutng n the Lagrangan gves the dual functon { N g(z 1,..., z N ) = (A x 0 + b ) T z 1 2 N AT z 2 2 z 2 1, = 1,..., N otherwse. 6
7 The dual problem s maxmze N (A x 0 + b ) T z 1 2 N AT z 2 subject to z 2 1, = 1,..., N. Lagrangan relaxaton of Boolean LP. A Boolean lnear program s an optmzaton problem of the form mnmze subject to c T x Ax b x {0, 1}, = 1,..., n, and s, n general, very dffcult to solve. In exercse (4.15) we studed the LP relaxaton of ths problem, mnmze c T x subject to Ax b (3) 0 x 1, = 1,..., n, whch s far easer to solve, and gves a lower bound on the optmal value of the Boolean LP. In ths problem we derve another lower bound for the Boolean LP, and work out the relaton between the two lower bounds. 1. Lagrangan relaxaton. The Boolean LP can be reformulated as the problem mnmze subject to c T x Ax b x (1 x ) = 0, = 1,..., n, whch has quadratc equalty constrants. Fnd the Lagrange dual of ths problem. The optmal value of the dual problem (whch s convex) gves a lower bound on the optmal value of the Boolean LP. Ths method of fndng a lower bound on the optmal value s called Lagrangan relaxaton. 2. Show that the lower bound obtaned va Lagrangan relaxaton, and va the LP relaxaton (3), are the same. Hnt. Derve the dual of the LP relaxaton (3). Soluton. 1. The Lagrangan s L(x, µ, ν) = c T x + µ T (Ax b) ν T x + x T dag(ν)x = x T dag(ν)x + (c + A T µ ν) T x b T µ. Mnmzng over x gves the dual functon { b g(µ, ν) = T µ (1/4) n (c + a T µ ν ) 2 /ν ν 0 otherwse where a s the th column of A, and we adopt the conventon that a 2 /0 = f a 0, and a 2 /0 = 0 f a = 0. 7
8 The resultng dual problem s maxmze b T µ (1/4) n (c + a T µ ν ) 2 /ν subject to ν 0. In order to smplfy ths dual, we optmze analytcally over ν, by notng that sup ( (c + a T µ ν ) 2 ) { 4(c + a = T µ) c + a T µ 0 ν 0 ν 0 c + a T µ 0 = 4 mn{0, (c + a T µ)} Ths allows us to elmnate ν from the dual problem, and smplfy t as maxmze b T µ + n mn{0, c + a T µ} subject to µ We follow the hnt. The Lagrangan and dual functon of the LP relaxaton are The dual problem s L(x, u, v, w) = c T x + u T (Ax b) v T x + w T (x 1) = (c + A T u v + w) T x b T u 1 T w { b g(u, v, w) = T u 1 T w A T u v + w + c = 0 otherwse. maxmze b T u 1 T w subject to A T u v + w + c = 0 u 0, v 0, w 0, whch s equvalent to the Lagrange relaxaton problem derved above. We conclude that the two relaxatons gve the same value. 8
princeton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationLagrange Multipliers Kernel Trick
Lagrange Multplers Kernel Trck Ncholas Ruozz Unversty of Texas at Dallas Based roughly on the sldes of Davd Sontag General Optmzaton A mathematcal detour, we ll come back to SVMs soon! subject to: f x
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationConvex Optimization. Optimality conditions. (EE227BT: UC Berkeley) Lecture 9 (Optimality; Conic duality) 9/25/14. Laurent El Ghaoui.
Convex Optmzaton (EE227BT: UC Berkeley) Lecture 9 (Optmalty; Conc dualty) 9/25/14 Laurent El Ghaou Organsatonal Mdterm: 10/7/14 (1.5 hours, n class, double-sded cheat sheet allowed) Project: Intal proposal
More informationSELECTED SOLUTIONS, SECTION (Weak duality) Prove that the primal and dual values p and d defined by equations (4.3.2) and (4.3.3) satisfy p d.
SELECTED SOLUTIONS, SECTION 4.3 1. Weak dualty Prove that the prmal and dual values p and d defned by equatons 4.3. and 4.3.3 satsfy p d. We consder an optmzaton problem of the form The Lagrangan for ths
More informationSolutions to exam in SF1811 Optimization, Jan 14, 2015
Solutons to exam n SF8 Optmzaton, Jan 4, 25 3 3 O------O -4 \ / \ / The network: \/ where all lnks go from left to rght. /\ / \ / \ 6 O------O -5 2 4.(a) Let x = ( x 3, x 4, x 23, x 24 ) T, where the varable
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationCOS 521: Advanced Algorithms Game Theory and Linear Programming
COS 521: Advanced Algorthms Game Theory and Lnear Programmng Moses Charkar February 27, 2013 In these notes, we ntroduce some basc concepts n game theory and lnear programmng (LP). We show a connecton
More informationAPPENDIX A Some Linear Algebra
APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly,
More informationMaximal Margin Classifier
CS81B/Stat41B: Advanced Topcs n Learnng & Decson Makng Mamal Margn Classfer Lecturer: Mchael Jordan Scrbes: Jana van Greunen Corrected verson - /1/004 1 References/Recommended Readng 1.1 Webstes www.kernel-machnes.org
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationAssortment Optimization under MNL
Assortment Optmzaton under MNL Haotan Song Aprl 30, 2017 1 Introducton The assortment optmzaton problem ams to fnd the revenue-maxmzng assortment of products to offer when the prces of products are fxed.
More information6.854J / J Advanced Algorithms Fall 2008
MIT OpenCourseWare http://ocw.mt.edu 6.854J / 18.415J Advanced Algorthms Fall 2008 For nformaton about ctng these materals or our Terms of Use, vst: http://ocw.mt.edu/terms. 18.415/6.854 Advanced Algorthms
More informationp(z) = 1 a e z/a 1(z 0) yi a i x (1/a) exp y i a i x a i=1 n i=1 (y i a i x) inf 1 (y Ax) inf Ax y (1 ν) y if A (1 ν) = 0 otherwise
Dustn Lennon Math 582 Convex Optmzaton Problems from Boy, Chapter 7 Problem 7.1 Solve the MLE problem when the nose s exponentally strbute wth ensty p(z = 1 a e z/a 1(z 0 The MLE s gven by the followng:
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More information10-801: Advanced Optimization and Randomized Methods Lecture 2: Convex functions (Jan 15, 2014)
0-80: Advanced Optmzaton and Randomzed Methods Lecture : Convex functons (Jan 5, 04) Lecturer: Suvrt Sra Addr: Carnege Mellon Unversty, Sprng 04 Scrbes: Avnava Dubey, Ahmed Hefny Dsclamer: These notes
More informationYong Joon Ryang. 1. Introduction Consider the multicommodity transportation problem with convex quadratic cost function. 1 2 (x x0 ) T Q(x x 0 )
Kangweon-Kyungk Math. Jour. 4 1996), No. 1, pp. 7 16 AN ITERATIVE ROW-ACTION METHOD FOR MULTICOMMODITY TRANSPORTATION PROBLEMS Yong Joon Ryang Abstract. The optmzaton problems wth quadratc constrants often
More informationLecture 20: Lift and Project, SDP Duality. Today we will study the Lift and Project method. Then we will prove the SDP duality theorem.
prnceton u. sp 02 cos 598B: algorthms and complexty Lecture 20: Lft and Project, SDP Dualty Lecturer: Sanjeev Arora Scrbe:Yury Makarychev Today we wll study the Lft and Project method. Then we wll prove
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More information1 Matrix representations of canonical matrices
1 Matrx representatons of canoncal matrces 2-d rotaton around the orgn: ( ) cos θ sn θ R 0 = sn θ cos θ 3-d rotaton around the x-axs: R x = 1 0 0 0 cos θ sn θ 0 sn θ cos θ 3-d rotaton around the y-axs:
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationMMA and GCMMA two methods for nonlinear optimization
MMA and GCMMA two methods for nonlnear optmzaton Krster Svanberg Optmzaton and Systems Theory, KTH, Stockholm, Sweden. krlle@math.kth.se Ths note descrbes the algorthms used n the author s 2007 mplementatons
More informationAPPROXIMATE PRICES OF BASKET AND ASIAN OPTIONS DUPONT OLIVIER. Premia 14
APPROXIMAE PRICES OF BASKE AND ASIAN OPIONS DUPON OLIVIER Prema 14 Contents Introducton 1 1. Framewor 1 1.1. Baset optons 1.. Asan optons. Computng the prce 3. Lower bound 3.1. Closed formula for the prce
More informationSupplement: Proofs and Technical Details for The Solution Path of the Generalized Lasso
Supplement: Proofs and Techncal Detals for The Soluton Path of the Generalzed Lasso Ryan J. Tbshran Jonathan Taylor In ths document we gve supplementary detals to the paper The Soluton Path of the Generalzed
More informationSome basic inequalities. Definition. Let V be a vector space over the complex numbers. An inner product is given by a function, V V C
Some basc nequaltes Defnton. Let V be a vector space over the complex numbers. An nner product s gven by a functon, V V C (x, y) x, y satsfyng the followng propertes (for all x V, y V and c C) (1) x +
More information14 Lagrange Multipliers
Lagrange Multplers 14 Lagrange Multplers The Method of Lagrange Multplers s a powerful technque for constraned optmzaton. Whle t has applcatons far beyond machne learnng t was orgnally developed to solve
More informationSupport Vector Machines. Jie Tang Knowledge Engineering Group Department of Computer Science and Technology Tsinghua University 2012
Support Vector Machnes Je Tang Knowledge Engneerng Group Department of Computer Scence and Technology Tsnghua Unversty 2012 1 Outlne What s a Support Vector Machne? Solvng SVMs Kernel Trcks 2 What s a
More informationModule 9. Lecture 6. Duality in Assignment Problems
Module 9 1 Lecture 6 Dualty n Assgnment Problems In ths lecture we attempt to answer few other mportant questons posed n earler lecture for (AP) and see how some of them can be explaned through the concept
More informationProseminar Optimierung II. Victor A. Kovtunenko SS 2012/2013: LV
Prosemnar Optmerung II Vctor A. Kovtunenko Insttute for Mathematcs and Scentfc Computng, Karl-Franzens Unversty of Graz, Henrchstr. 36, 8010 Graz, Austra; Lavrent ev Insttute of Hydrodynamcs, Sberan Dvson
More information8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS
SECTION 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS 493 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS All the vector spaces you have studed thus far n the text are real vector spaces because the scalars
More informationSome modelling aspects for the Matlab implementation of MMA
Some modellng aspects for the Matlab mplementaton of MMA Krster Svanberg krlle@math.kth.se Optmzaton and Systems Theory Department of Mathematcs KTH, SE 10044 Stockholm September 2004 1. Consdered optmzaton
More informationCSCE 790S Background Results
CSCE 790S Background Results Stephen A. Fenner September 8, 011 Abstract These results are background to the course CSCE 790S/CSCE 790B, Quantum Computaton and Informaton (Sprng 007 and Fall 011). Each
More informationThe Second Anti-Mathima on Game Theory
The Second Ant-Mathma on Game Theory Ath. Kehagas December 1 2006 1 Introducton In ths note we wll examne the noton of game equlbrum for three types of games 1. 2-player 2-acton zero-sum games 2. 2-player
More informationThe Minimum Universal Cost Flow in an Infeasible Flow Network
Journal of Scences, Islamc Republc of Iran 17(2): 175-180 (2006) Unversty of Tehran, ISSN 1016-1104 http://jscencesutacr The Mnmum Unversal Cost Flow n an Infeasble Flow Network H Saleh Fathabad * M Bagheran
More informationExercise Solutions to Real Analysis
xercse Solutons to Real Analyss Note: References refer to H. L. Royden, Real Analyss xersze 1. Gven any set A any ɛ > 0, there s an open set O such that A O m O m A + ɛ. Soluton 1. If m A =, then there
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationCS 229, Public Course Problem Set #3 Solutions: Learning Theory and Unsupervised Learning
CS9 Problem Set #3 Solutons CS 9, Publc Course Problem Set #3 Solutons: Learnng Theory and Unsupervsed Learnng. Unform convergence and Model Selecton In ths problem, we wll prove a bound on the error of
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationChapter 5. Solution of System of Linear Equations. Module No. 6. Solution of Inconsistent and Ill Conditioned Systems
Numercal Analyss by Dr. Anta Pal Assstant Professor Department of Mathematcs Natonal Insttute of Technology Durgapur Durgapur-713209 emal: anta.bue@gmal.com 1 . Chapter 5 Soluton of System of Lnear Equatons
More informationControl of Uncertain Bilinear Systems using Linear Controllers: Stability Region Estimation and Controller Design
Control of Uncertan Blnear Systems usng Lnear Controllers: Stablty Regon Estmaton Controller Desgn Shoudong Huang Department of Engneerng Australan Natonal Unversty Canberra, ACT 2, Australa shoudong.huang@anu.edu.au
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More informationCollege of Computer & Information Science Fall 2009 Northeastern University 20 October 2009
College of Computer & Informaton Scence Fall 2009 Northeastern Unversty 20 October 2009 CS7880: Algorthmc Power Tools Scrbe: Jan Wen and Laura Poplawsk Lecture Outlne: Prmal-dual schema Network Desgn:
More informationAdditional Codes using Finite Difference Method. 1 HJB Equation for Consumption-Saving Problem Without Uncertainty
Addtonal Codes usng Fnte Dfference Method Benamn Moll 1 HJB Equaton for Consumpton-Savng Problem Wthout Uncertanty Before consderng the case wth stochastc ncome n http://www.prnceton.edu/~moll/ HACTproect/HACT_Numercal_Appendx.pdf,
More informationStanford University CS359G: Graph Partitioning and Expanders Handout 4 Luca Trevisan January 13, 2011
Stanford Unversty CS359G: Graph Parttonng and Expanders Handout 4 Luca Trevsan January 3, 0 Lecture 4 In whch we prove the dffcult drecton of Cheeger s nequalty. As n the past lectures, consder an undrected
More informationPHYS 705: Classical Mechanics. Calculus of Variations II
1 PHYS 705: Classcal Mechancs Calculus of Varatons II 2 Calculus of Varatons: Generalzaton (no constrant yet) Suppose now that F depends on several dependent varables : We need to fnd such that has a statonary
More informationSL n (F ) Equals its Own Derived Group
Internatonal Journal of Algebra, Vol. 2, 2008, no. 12, 585-594 SL n (F ) Equals ts Own Derved Group Jorge Macel BMCC-The Cty Unversty of New York, CUNY 199 Chambers street, New York, NY 10007, USA macel@cms.nyu.edu
More informationLecture 17: Lee-Sidford Barrier
CSE 599: Interplay between Convex Optmzaton and Geometry Wnter 2018 Lecturer: Yn Tat Lee Lecture 17: Lee-Sdford Barrer Dsclamer: Please tell me any mstake you notced. In ths lecture, we talk about the
More informationMEM Chapter 4b. LMI Lab commands
1 MEM8-7 Chapter 4b LMI Lab commands setlms lmvar lmterm getlms lmedt lmnbr matnbr lmnfo feasp dec2mat evallm showlm setmvar mat2dec mncx dellm delmvar gevp 2 Intalzng the LMI System he descrpton of an
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationw ). Then use the Cauchy-Schwartz inequality ( v w v w ).] = in R 4. Can you find a vector u 4 in R 4 such that the
Math S-b Summer 8 Homework #5 Problems due Wed, July 8: Secton 5: Gve an algebrac proof for the trangle nequalty v+ w v + w Draw a sketch [Hnt: Expand v+ w ( v+ w) ( v+ w ) hen use the Cauchy-Schwartz
More informationON A DETERMINATION OF THE INITIAL FUNCTIONS FROM THE OBSERVED VALUES OF THE BOUNDARY FUNCTIONS FOR THE SECOND-ORDER HYPERBOLIC EQUATION
Advanced Mathematcal Models & Applcatons Vol.3, No.3, 2018, pp.215-222 ON A DETERMINATION OF THE INITIAL FUNCTIONS FROM THE OBSERVED VALUES OF THE BOUNDARY FUNCTIONS FOR THE SECOND-ORDER HYPERBOLIC EUATION
More informationLinear Feature Engineering 11
Lnear Feature Engneerng 11 2 Least-Squares 2.1 Smple least-squares Consder the followng dataset. We have a bunch of nputs x and correspondng outputs y. The partcular values n ths dataset are x y 0.23 0.19
More informationTopic 5: Non-Linear Regression
Topc 5: Non-Lnear Regresson The models we ve worked wth so far have been lnear n the parameters. They ve been of the form: y = Xβ + ε Many models based on economc theory are actually non-lnear n the parameters.
More informationu i log v i + 1 T (u v).
where f(v) = n v log v s the negatve entropy of v. Soluton. The negatve entropy s strctly convex and dfferentable on R n ++, hence f(u) > f(v) + f(v) T (u v) for all u, v R n ++ wth u v. Evaluatng both
More informationPROBLEM SET 7 GENERAL EQUILIBRIUM
PROBLEM SET 7 GENERAL EQUILIBRIUM Queston a Defnton: An Arrow-Debreu Compettve Equlbrum s a vector of prces {p t } and allocatons {c t, c 2 t } whch satsfes ( Gven {p t }, c t maxmzes βt ln c t subject
More informationREAL ANALYSIS I HOMEWORK 1
REAL ANALYSIS I HOMEWORK CİHAN BAHRAN The questons are from Tao s text. Exercse 0.0.. If (x α ) α A s a collecton of numbers x α [0, + ] such that x α
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationf(x,y) = (4(x 2 4)x,2y) = 0 H(x,y) =
Problem Set 3: Unconstraned mzaton n R N. () Fnd all crtcal ponts of f(x,y) (x 4) +y and show whch are ma and whch are mnma. () Fnd all crtcal ponts of f(x,y) (y x ) x and show whch are ma and whch are
More information15 Lagrange Multipliers
15 The Method of s a powerful technque for constraned optmzaton. Whle t has applcatons far beyond machne learnng t was orgnally developed to solve physcs equatons), t s used for several ey dervatons n
More informationApproximate Smallest Enclosing Balls
Chapter 5 Approxmate Smallest Enclosng Balls 5. Boundng Volumes A boundng volume for a set S R d s a superset of S wth a smple shape, for example a box, a ball, or an ellpsod. Fgure 5.: Boundng boxes Q(P
More informationFormulas for the Determinant
page 224 224 CHAPTER 3 Determnants e t te t e 2t 38 A = e t 2te t e 2t e t te t 2e 2t 39 If 123 A = 345, 456 compute the matrx product A adj(a) What can you conclude about det(a)? For Problems 40 43, use
More informationAnother converse of Jensen s inequality
Another converse of Jensen s nequalty Slavko Smc Abstract. We gve the best possble global bounds for a form of dscrete Jensen s nequalty. By some examples ts frutfulness s shown. 1. Introducton Throughout
More informationOPTIMISATION. Introduction Single Variable Unconstrained Optimisation Multivariable Unconstrained Optimisation Linear Programming
OPTIMIATION Introducton ngle Varable Unconstraned Optmsaton Multvarable Unconstraned Optmsaton Lnear Programmng Chapter Optmsaton /. Introducton In an engneerng analss, sometmes etremtes, ether mnmum or
More informationEdge Isoperimetric Inequalities
November 7, 2005 Ross M. Rchardson Edge Isopermetrc Inequaltes 1 Four Questons Recall that n the last lecture we looked at the problem of sopermetrc nequaltes n the hypercube, Q n. Our noton of boundary
More informationTHE GUARANTEED COST CONTROL FOR UNCERTAIN LARGE SCALE INTERCONNECTED SYSTEMS
Copyrght 22 IFAC 5th rennal World Congress, Barcelona, Span HE GUARANEED COS CONROL FOR UNCERAIN LARGE SCALE INERCONNECED SYSEMS Hroak Mukadan Yasuyuk akato Yoshyuk anaka Koch Mzukam Faculty of Informaton
More informationInexact Alternating Minimization Algorithm for Distributed Optimization with an Application to Distributed MPC
Inexact Alternatng Mnmzaton Algorthm for Dstrbuted Optmzaton wth an Applcaton to Dstrbuted MPC Ye Pu, Coln N. Jones and Melane N. Zelnger arxv:608.0043v [math.oc] Aug 206 Abstract In ths paper, we propose
More informationBOUNDEDNESS OF THE RIESZ TRANSFORM WITH MATRIX A 2 WEIGHTS
BOUNDEDNESS OF THE IESZ TANSFOM WITH MATIX A WEIGHTS Introducton Let L = L ( n, be the functon space wth norm (ˆ f L = f(x C dx d < For a d d matrx valued functon W : wth W (x postve sem-defnte for all
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far Supervsed machne learnng Lnear models Least squares regresson Fsher s dscrmnant, Perceptron, Logstc model Non-lnear
More informationMATHEMATICAL ENGINEERING TECHNICAL REPORTS. Successive Lagrangian Relaxation Algorithm for Nonconvex Quadratic Optimization
MATHEMATICAL ENGINEERING TECHNICAL REPORTS Successve Lagrangan Relaxaton Algorthm for Nonconvex Quadratc Optmzaton Shnj YAMADA and Akko TAKEDA METR 2017 08 March 2017 DEPARTMENT OF MATHEMATICAL INFORMATICS
More informationMath 217 Fall 2013 Homework 2 Solutions
Math 17 Fall 013 Homework Solutons Due Thursday Sept. 6, 013 5pm Ths homework conssts of 6 problems of 5 ponts each. The total s 30. You need to fully justfy your answer prove that your functon ndeed has
More informationLecture 6: Support Vector Machines
Lecture 6: Support Vector Machnes Marna Melă mmp@stat.washngton.edu Department of Statstcs Unversty of Washngton November, 2018 Lnear SVM s The margn and the expected classfcaton error Maxmum Margn Lnear
More informationGame Theory. Lecture Notes By Y. Narahari. Department of Computer Science and Automation Indian Institute of Science Bangalore, India February 2008
Game Theory Lecture Notes By Y. Narahar Department of Computer Scence and Automaton Indan Insttute of Scence Bangalore, Inda February 2008 Chapter 10: Two Person Zero Sum Games Note: Ths s a only a draft
More informationA Local Variational Problem of Second Order for a Class of Optimal Control Problems with Nonsmooth Objective Function
A Local Varatonal Problem of Second Order for a Class of Optmal Control Problems wth Nonsmooth Objectve Functon Alexander P. Afanasev Insttute for Informaton Transmsson Problems, Russan Academy of Scences,
More informationSupport Vector Machines CS434
Support Vector Machnes CS434 Lnear Separators Many lnear separators exst that perfectly classfy all tranng examples Whch of the lnear separators s the best? + + + + + + + + + Intuton of Margn Consder ponts
More information( ) 2 ( ) ( ) Problem Set 4 Suggested Solutions. Problem 1
Problem Set 4 Suggested Solutons Problem (A) The market demand functon s the soluton to the followng utlty-maxmzaton roblem (UMP): The Lagrangean: ( x, x, x ) = + max U x, x, x x x x st.. x + x + x y x,
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far So far Supervsed machne learnng Lnear models Non-lnear models Unsupervsed machne learnng Generc scaffoldng So far
More informationThe Finite Element Method: A Short Introduction
Te Fnte Element Metod: A Sort ntroducton Wat s FEM? Te Fnte Element Metod (FEM) ntroduced by engneers n late 50 s and 60 s s a numercal tecnque for solvng problems wc are descrbed by Ordnary Dfferental
More informationMin Cut, Fast Cut, Polynomial Identities
Randomzed Algorthms, Summer 016 Mn Cut, Fast Cut, Polynomal Identtes Instructor: Thomas Kesselhem and Kurt Mehlhorn 1 Mn Cuts n Graphs Lecture (5 pages) Throughout ths secton, G = (V, E) s a mult-graph.
More informationLecture 21: Numerical methods for pricing American type derivatives
Lecture 21: Numercal methods for prcng Amercan type dervatves Xaoguang Wang STAT 598W Aprl 10th, 2014 (STAT 598W) Lecture 21 1 / 26 Outlne 1 Fnte Dfference Method Explct Method Penalty Method (STAT 598W)
More information8/25/17. Data Modeling. Data Modeling. Data Modeling. Patrice Koehl Department of Biological Sciences National University of Singapore
8/5/17 Data Modelng Patrce Koehl Department of Bologcal Scences atonal Unversty of Sngapore http://www.cs.ucdavs.edu/~koehl/teachng/bl59 koehl@cs.ucdavs.edu Data Modelng Ø Data Modelng: least squares Ø
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More informationFirst day August 1, Problems and Solutions
FOURTH INTERNATIONAL COMPETITION FOR UNIVERSITY STUDENTS IN MATHEMATICS July 30 August 4, 997, Plovdv, BULGARIA Frst day August, 997 Problems and Solutons Problem. Let {ε n } n= be a sequence of postve
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationLecture 10 Support Vector Machines. Oct
Lecture 10 Support Vector Machnes Oct - 20-2008 Lnear Separators Whch of the lnear separators s optmal? Concept of Margn Recall that n Perceptron, we learned that the convergence rate of the Perceptron
More informationLinear, affine, and convex sets and hulls In the sequel, unless otherwise specified, X will denote a real vector space.
Lnear, affne, and convex sets and hulls In the sequel, unless otherwse specfed, X wll denote a real vector space. Lnes and segments. Gven two ponts x, y X, we defne xy = {x + t(y x) : t R} = {(1 t)x +
More informationOn the Global Linear Convergence of the ADMM with Multi-Block Variables
On the Global Lnear Convergence of the ADMM wth Mult-Block Varables Tany Ln Shqan Ma Shuzhong Zhang May 31, 01 Abstract The alternatng drecton method of multplers ADMM has been wdely used for solvng structured
More informationLaboratory 3: Method of Least Squares
Laboratory 3: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly they are correlated wth
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationNatural Language Processing and Information Retrieval
Natural Language Processng and Informaton Retreval Support Vector Machnes Alessandro Moschtt Department of nformaton and communcaton technology Unversty of Trento Emal: moschtt@ds.untn.t Summary Support
More informationIV. Performance Optimization
IV. Performance Optmzaton A. Steepest descent algorthm defnton how to set up bounds on learnng rate mnmzaton n a lne (varyng learnng rate) momentum learnng examples B. Newton s method defnton Gauss-Newton
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationChapter 7 Generalized and Weighted Least Squares Estimation. In this method, the deviation between the observed and expected values of
Chapter 7 Generalzed and Weghted Least Squares Estmaton The usual lnear regresson model assumes that all the random error components are dentcally and ndependently dstrbuted wth constant varance. When
More informationIJRSS Volume 2, Issue 2 ISSN:
IJRSS Volume, Issue ISSN: 49-496 An Algorthm To Fnd Optmum Cost Tme Trade Off Pars In A Fractonal Capactated Transportaton Problem Wth Restrcted Flow KAVITA GUPTA* S.R. ARORA** _ Abstract: Ths paper presents
More informationNP-Completeness : Proofs
NP-Completeness : Proofs Proof Methods A method to show a decson problem Π NP-complete s as follows. (1) Show Π NP. (2) Choose an NP-complete problem Π. (3) Show Π Π. A method to show an optmzaton problem
More informationConvexity preserving interpolation by splines of arbitrary degree
Computer Scence Journal of Moldova, vol.18, no.1(52), 2010 Convexty preservng nterpolaton by splnes of arbtrary degree Igor Verlan Abstract In the present paper an algorthm of C 2 nterpolaton of dscrete
More informationComputing Correlated Equilibria in Multi-Player Games
Computng Correlated Equlbra n Mult-Player Games Chrstos H. Papadmtrou Presented by Zhanxang Huang December 7th, 2005 1 The Author Dr. Chrstos H. Papadmtrou CS professor at UC Berkley (taught at Harvard,
More informationLaboratory 1c: Method of Least Squares
Lab 1c, Least Squares Laboratory 1c: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly
More informationPhysics 4B. A positive value is obtained, so the current is counterclockwise around the circuit.
Physcs 4B Solutons to Chapter 7 HW Chapter 7: Questons:, 8, 0 Problems:,,, 45, 48,,, 7, 9 Queston 7- (a) no (b) yes (c) all te Queston 7-8 0 μc Queston 7-0, c;, a;, d; 4, b Problem 7- (a) Let be the current
More informationGeometric Registration for Deformable Shapes. 2.1 ICP + Tangent Space optimization for Rigid Motions
Geometrc Regstraton for Deformable Shapes 2.1 ICP + Tangent Space optmzaton for Rgd Motons Regstraton Problem Gven Two pont cloud data sets P (model) and Q (data) sampled from surfaces Φ P and Φ Q respectvely.
More informationSupporting Information
Supportng Informaton The neural network f n Eq. 1 s gven by: f x l = ReLU W atom x l + b atom, 2 where ReLU s the element-wse rectfed lnear unt, 21.e., ReLUx = max0, x, W atom R d d s the weght matrx to
More information