CHAPTER 6 CONSTRAINED OPTIMIZATION 1: K-T CONDITIONS
|
|
- Reynold Carter
- 6 years ago
- Views:
Transcription
1 Chapter 6: Constraned Optzaton CHAPER 6 CONSRAINED OPIMIZAION : K- CONDIIONS Introducton We now begn our dscusson of gradent-based constraned optzaton. Recall that n Chapter 3 we looked at gradent-based unconstraned optzaton and learned about the necessary and suffcent condtons for an unconstraned optu, varous search drectons, conductng a lne search, and quas-newton ethods. We wll buld on that foundaton as we extend the theory to probles wth constrants. Necessary Condtons for Constraned Optu At an unconstraned local optu, there s no drecton n whch we can ove to prove the objectve functon. We can state the necessary condtons atheatcally as f 0. At a constraned local optu, there s no feasble drecton n whch we can ove to prove the objectve. hat s, there ay be drectons fro the current pont that wll prove the objectve, but these drectons pont nto nfeasble space. he necessary condtons for a constraned local optu are called the Kuhn-ucker Condtons, and these condtons play a very portant role n constraned optzaton theory and algorth developent.. Proble For It wll be convenent to cast our optzaton proble nto one of two partcular fors. hs s no restrcton snce any proble can be cast nto ether of these fors. or Max s.t.: Mn s.t.: f x x 0,, x 0,, g b k g b k f x x 0,, x 0,, g b k g b k. Graphcal Exaples For the graphcal exaples below, we wll assue we are axzng wth constrants. We have prevously consdered how we can tell atheatcally f soe arbtrary vector, s, ponts downhll. hat condton s, s f 0. We developed ths condton by notng that
2 Chapter 6: Constraned Optzaton any vector s could be resolved nto vector coponents whch le n the tangent plane and along the gradent (or negatve gradent) drecton. Now suppose we have the stuaton shown n Fg. 6. below. We are axzng. We have contours ncreasng n the drecton of the arrow. he gradent vector s shown. What s the set of drectons whch proves the objectve? It s the set for whch s f 0. We show that set as a se-crcle n Fg. 6. Fg. 6. Gradent and set of drectons whch proves objectve functon. Now suppose we add n a less-than nequalty constrant, g( x ) 0. Contours for ths constrant are gven n Fg. 6.. he trangular arkers ndcate the contour for the allowable value and pont towards the drecton of the feasble space. What s the set of drectons whch s feasble? It s the set for whch s f 0. hat set s shown as a se-crcle n the fgure. Fg. 6. Gradent and set of feasble drectons for a constrant. Now suppose we overlay these two sets of contours on top of each other, as n Fg Where does the optu le? By defnton, a constraned optu s a pont for whch there s no feasble drecton whch proves the objectve. We can see that that condton occurs when the gradent for the objectve and gradent for the constrant le on top of each other. When ths happens, the set of drectons whch proves the objectve (dashed se-crcle) does not overlap wth the set of feasble drectons (sold se-crcle.)
3 Chapter 6: Constraned Optzaton Fg. 6.3 An optu for one bndng constrant occurs when the gradent vectors overlap. When ths condton occurs, no feasble pont exsts whch proves the objectve. Matheatcally we can wrte the above condton as f x g x (6.) where s a postve constant. Now consder a case where there are two bndng constrants at the soluton, as shown n Fg. 6.4 Fg. 6.4 wo bndng constrants at an optu. As long as the objectve gradent s wthn the cone of the constrant gradents, no feasble pont exsts whch proves the objectve. We see that the objectve gradent vector s contaned nsde the constrant gradent vectors. If the objectve gradent vector s wthn the constrant gradent vectors, then no 3
4 Chapter 6: Constraned Optzaton drecton exsts whch sultaneously proves the objectve and ponts n the feasble regon. We can state ths condton atheatcally as: f x g x g x (6.) where, as wth the sngle constrant case, and are postve constants. Havng graphcally otvated the developent of the an atheatcal condtons for a constraned optu, we are now ready to state these condtons..3 he Kuhn-ucker Condtons he Kuhn-ucker condtons are the necessary condtons for a pont to be a constraned local optu, for ether of the general probles gven below. (he K- equatons also work for an unconstraned optu, as we wll explan later.) If x s a local ax for: Max s.t.: f x (6.3) 0,, 0,, g x b k (6.4) g x b k (6.5) Or f x s a local n for: Mn s.t.: f x (6.6) 0,, 0,, g x b k (6.7) g x b k (6.8) and f the constrant gradents at the optu, g x, are ndependent, then there exst λ, called Lagrange ultplers, such that x and λ satsfy the followng syste of equatons, g x b (6.9) s feasble,, g g b 0,, k f x x 0 (6.0) x (6.) 0,, k (6.) unrestrcted for k,, (6.3) or 4
5 Chapter 6: Constraned Optzaton Note n the above equatons,,, k ndcates nequalty constrants, k,, ndcates equalty constrants, and,, ndcates all constrants. Just as wth the necessary condtons for an unconstraned optu, the K- condtons are necessary but not suffcent condtons for a constraned optu. We wll now explan each of these condtons. Equaton (6.9) requres that a constraned optu be feasble wth respect to all constrants. Equaton (6.0) requres the objectve functon gradent to be a lnear cobnaton of the constrant gradents. hs nsures there s no drecton that wll sultaneously prove the objectve and satsfy the constrants. Equaton (6.) enforces a condton known as copleentary slackness. Notce that ths condton s for the nequalty constrants only. hs condton states that ether an nequalty constrant s bndng, or the assocated Lagrange ultpler s zero. Essentally ths eans that nonbndng nequalty constrants drop out of the proble. Equaton (6.) states that the Lagrange ultplers for the nequalty constrants ust be postve. Equaton (6.3) states that the Lagrange ultplers for the equalty constrants can be ether postve or negatve. Note that (6.0) above, whch s gven n vector for, represents a syste of n equatons. We can rewrte (6.0) as: f g g g 0 x x x x f g g g 0 x x x x n n n n (6.4) We note there s a Lagrange ultpler,, for every constrant. Recall, however, that f the constrant s not bndng then ts Lagrange ultpler s zero, fro (6.). aken together, the K- condtons represent +n equatons n +n unknowns. he equatons are the n equatons gven by (6.4) (or (6.0)) and the constrants ( (6.9)). he unknowns are the n eleents of the vector x and the eleents of the vector 5
6 Chapter 6: Constraned Optzaton.4 Exaples of the K- Condtons.4. Exaple : An Equalty Constraned Proble Usng the K- equatons, fnd the optu to the proble, Mn f x x 4x s.t. g: 3xx A pcture of ths proble s gven below: Fg. 6.5 Contours of functons for Exaple. Snce the constrant s an equalty constrant, we know t s bndng, so the Lagrange ultpler wll be non-zero. Wth two varables and one constrant, the K- equatons represent three equatons n three unknowns. he K- condtons can be wrtten: f g x x f g x x g x 0 b 0 evaluatng these expressons: 0 6
7 Chapter 6: Constraned Optzaton 4x 3 0 8x 0 3x x 0 whch we can wrte n atrx for as: 4 0 3x x he soluton to ths syste of equatons s x 3.77, x.0909, he value of the objectve at ths pont s f 6.8. hs optu occurs where the gradent vectors of the constrant and objectve overlap, just as ndcated by the graphcal dscusson. We should verfy to ake sure ths s a constraned n and not a constraned ax, snce the K- equatons apply at both. Because the objectve functon for ths proble s quadratc and the constrant s lnear, the K- equatons are lnear and thus easy to solve. We call a proble wth a quadratc objectve and lnear constrants a quadratc prograng proble for ths reason. Usually the K- equatons are not lnear. However, the SQP algorth attepts to solve these equatons by solvng a sequence of quadratc progra approxatons to the real progra thus the nae of Sequental Quadratc Prograng..4. Exaple : An Inequalty Constraned Proble In general t s ore dffcult to use the K- condtons to solve for the optu of an nequalty constraned proble (than for a proble wth equalty constrants only) because we don t know beforehand whch constrants are bndng at the optu. hus we often use the K- condtons to verfy that a pont we have reached s a canddate optal soluton. Gven a pont, t s easy to check whch constrants are bndng. Verfy that the pont x s an optu to the proble: Mn f x x x s.t. g: x 4x Step : Put proble n proper for: Mn f x x x s.t. g x4x 0 Step : See whch constrants are bndng: 7
8 Chapter 6: Constraned Optzaton Snce ths constrant s bndng, the assocated Lagrange ultpler s solved for. (If t were not bndng, the Lagrange ultpler would be zero, fro copleentary slackness.) Step 3: Wrte out the Lagrange ultpler equatons represented by (6.0): g f x x x f g x x x Step 4: Substtute n the gven pont: () 0 (4) (6.5) (6.6) Fro (6.5),.48 Fro (6.6),.48 Snce these 's are consstent and postve, the above pont satsfes the K- equatons and s a canddate optal soluton..4.3 Exaple 3: Another Inequalty Constraned Proble Gven the proble: See f x 0 3 Mn f x x x s.t. g x x x g x x x 90 0 satsfes the K condtons. Graphcally the proble looks lke, 8
9 Chapter 6: Constraned Optzaton Fg Contour plot and proposed pont for Exaple 3. At the proposed pont constrant g s bndng; g s not. Step : Change proble to be n the for of ( ): Max f x x x s.t. gx x x g x x x : 90 : 0 Step : See whch constrants are bndng: In ths case we can check constrants graphcally. Because fro (6.). However, s solved for snce g x s bndng. g x s not bndng, 0 Step 3: Wrte out the Lagrange ultpler equatons represented by (6.0): f g g x ( x) 0 x x x f g g x 0 x x x (6.7) (6.8) 9
10 Chapter 6: Constraned Optzaton Step 4: Substtute n the gven pont: At 0 3 x, (6.7) vanshes; fro (6.8): 3 6 so a vald set of s,.e., 0 6 has been found and the K condtons are satsfed. hs s therefore a canddate optal soluton..4.4 Exaple 4: Another Pont for the Sae Proble Check to see f 0 x satsfes the K condtons for the proble gven n Exaple 3 above. hs pont s shown n Fg. 6.7 Fg. 6.7 Contour plot and proposed pont for Exaple 4. Step : Change proble to be n the for of ( ): Max x f x x s.t. gx x x g x x x : 90 : 0 0
11 Chapter 6: Constraned Optzaton Step : See whch constrants are bndng: Fro Fg. 6.7 we can see that 0 g x s not bndng and therefore 0 ; g Step 3: Wrte out the Lagrange ultpler equatons represented by (6.0): x s bndng so f g g x () 0 x x x f g g 0 x x x (6.9) (6.0) Step 4: Substtute n the gven pont: Substtutng x 0 fro (6.9) fro (6.0) Snce we cannot fnd a consstent set of s, and the s are negatve as well (ether condton beng enough to dsqualfy the pont), ths pont does not satsfy the Kuhn-ucker condtons and cannot be a constraned optu. Queston: In Exaples 3 and 4 we have looked at two ponts a constraned n, and pont whch s not an optu. Are there any other ponts whch would satsfy the K- condtons for ths proble? Where would a constraned ax be found? Would the K- condtons apply there?.5 Unconstraned Probles We entoned the K- condtons also apply to unconstraned probles. hs s fortunate snce a constraned optzaton proble does not have to have a constraned soluton. he optu ght be an unconstraned optu n the nteror of the constrants. If we have an unconstraned optu to a constraned proble, what happens to the K- condtons? In ths case none of the constrants are bndng so all of the Lagrange ultplers are zero, fro (6.), so (6.0) becoes, = 0 f x gx f x 0 hus we see that the K- equatons splfy to the necessary condtons for an unconstraned optu when no constrants are bndng.
12 Chapter 6: Constraned Optzaton 3 he Lagrangan Functon 3. Defnton It wll be convenent for us to defne the Lagrangan Functon: L x, λ f x g x b (6.) Note that the Lagrangan functon s a functon of both x and. hus the gradent of the Lagrangan functon s ade up of partals wth respect to x and : L L L L L x, λ (6.) x xn We wll evaluate soe of these partals to becoe falar wth the, he partal L x s: L f g x x x L Slarly, s gven by, x L f g x x x he partal L L s: g b It s convenent, gven these results, to splt the gradent vector of the Lagrangan functon nto two parts: the vector contanng the partal dervatves wth respect to x, wrtten L, x and the vector contanng the partals wth respect to, wrtten L. he gradent of the Lagrangan functon wth respect to x can be wrtten n vector for as: x L f x g x (6.3) so that we could replace (6.0) by L x 0 f we wshed. he gradent of the Lagrangan functon wth respect to s:
13 Chapter 6: Constraned Optzaton L g g x x g b b b x (6.4) 3. he Lagrangan Functon and Optalty For a proble wth equalty constrants only, we can copactly state the K- condtons as, xl L L 0 (6.5) For a proble wth nequalty constrants as well, the an condton of the K- equatons, (6.0), can be stated as, L x 0 (6.6) hus we can consder that at an optu, there exst and x such that x s a statonary pont of the Lagrangan functon. he Lagrangan functon provdes nforaton about how the objectve and bndng constrants together affect an optu. Suppose we are at a constraned optu. If we were to change the objectve functon, ths would clearly have an effect on the soluton. Lkewse, f we were to change the bndng constrants (perhaps by changng the rght hand sdes), these changes would also affect the value of the soluton. he Lagrangan functon tells how these changes trade-off aganst each other, x λ x x L, f g b he Lagrange ultplers serve as weghtng factors between the ndvdual constrants and the objectve. Approprately, the ultplers have unts of (objectve functon/constrant functon). hus f our objectve had unts of pounds, and constrant had unts of nches, Lagrange ultpler would have unts of pounds per nch. 3.3 Interpretng Values of Lagrange Multplers hus far we have solved for Lagrange ultplers, but we have not attached any sgnfcance to ther values. We wll use the Lagrangan functon to help us nterpret what ther values ean. We wll start wth the Lagrangan functon at an optu: 3
14 Chapter 6: Constraned Optzaton L x, λ f x g b x (6.7) Suppose now we consder the rght-hand sde of constrant, b, to be a varable. How does df the optal soluton change as we change b? o answer ths queston, we need to fnd db. hat s, we need to fnd how the optal value of the objectve changes as we change the rght hand sde, b. he Lagrangan functon, whch relates how constrants and the objectve nteract at the optu, can gve us ths nforaton. We wll be concerned only wth sall perturbatons at the optu. hs allows us to gnore nonbndng nequalty constrants, whch wll be treated as f they were not there. hus nstead of constrants, we wll have constrants, whch s the set of equalty and bndng nequalty constrants. At an optu, the value of L becoes the sae as the value of f. hs s because all of the ters n braces go to zero, = 0 L x, λ f x g b x f x df dl snce all constrants n our set are bndng. At the optu therefore,. db db As we change b, we would expect x and to change, so we need to consder x and x b, λ b. hen by the chan rule: theselves to be functons of b,.e., df db db b b b dl x L xl λ L (6.8) At the optu xl 0 and L 0, leavng only Fro the Lagrangan functon, hus we have the result, L, b L. b df (6.9) db he Lagrange ultplers provde us wth senstvty nforaton about the optal soluton. hey tell use how the optal objectve value would change f we changed the rght-hand sde of the constrants. hs can be very useful nforaton, tellng us, for exaple, how we 4
15 Chapter 6: Constraned Optzaton could expect the optal weght of a truss to change f we relaxed the rght-hand sde of a bndng stress constrant (.e. ncreased the allowable value of the stress constrant). Cauton: he senstvty nforaton provded by the Lagrange ultplers s vald only for sall changes about the optu (snce for nonlnear functons, dervatves change as we ove fro pont to pont) and assues that the sae constrants are bndng at the perturbed optu Exaple 5: Interpretng the Value of the Lagrange Multplers In Exaple, Secton.4., we solved the followng proble Mn f x x 4x s.t. g: 3x x We found the optal soluton to be: x 3.77, x.0909, , at whch pont f 6.8. What would be the expected change n the objectve be f we ncreased the rght-hand sde of the constrant fro to 3? Fro (6.9), f b For b, the change n the objectve should be approxately If we change the rght hand sde and re-optze the proble, the new optu s, x , x.88, 4.77, at whch pont f he actual change n the objectve s (Indeed, t s the average of the two 's.) hus, wthout optzng, the Lagrange ultpler provdes an estate of how uch the objectve would change per unt change n the constrant rght hand sde. hs helps us evaluate how senstve the optu s to changes. Soetes the objectve represents proft, and the rght hand sde of a constrant s vewed as a resource whch can be purchased. he value of the Lagrange ultpler s a breakpont between realzng a net ncrease n proft or a net loss. If, for a bndng constrant, we can purchase ore rght hand sde for less than the Lagrange ultpler, net proft wll be postve. If not, the cost of the resource wll outwegh the ncrease n proft. 3.4 Necessary and Suffcent Condtons he K- Condtons we have presented n prevous sectons are necessary condtons for a constraned optu. hat s, for a pont to be a canddate optal soluton, t ust be possble to fnd values of that satsfy (6.9)-(6.3). If we cannot fnd such, then the canddate pont cannot be a constraned optal soluton. 5
16 Chapter 6: Constraned Optzaton Note, however, that as part of the K condtons we requre the constrant gradents, g x, to be ndependent at the optal soluton; otherwse t s possble, although unlkely, that we could have a pont be a constraned optu and not satsfy the K condtons. If a pont satsfes the K condtons, then t s a canddate optal soluton. As we have seen, the necessary condtons can hold at a constraned ax, constraned n, or a pont that s nether. An exaple showng how ths ght occur s gven below: A, C - Locally Constraned Opta B - Saddle Pont Decreasng Objectve X A B K Condtons can be satsfed at ponts A, B, and C. Infeasble Space C X Fg Ponts where the K- equatons would be satsfed. For an unconstraned optu we saw that suffcent condtons for a nu were that f 0 and, the Hessan, f ( x ), s postve defnte. Lkewse, for a constraned optu, suffcent condtons for a pont to be a constraned nu are the K- equatons are satsfed ( ) and the Hessan of the Lagrangan L, x x λ, s postve defnte, where, functon wth respect to x, f g L x, λ x x (6.30) x Soe further dscusson s needed, however. If we wrte the condton of postve defnteness as, x he vectors y ust satsfy, y L x, λ y 0 (6.3) Jx () y 0 (6.3) 6
17 Chapter 6: Constraned Optzaton Where Jx ( ) s the Jacoban atrx of the constrants (atrx whose rows are the gradents of the constrants) actve at x. hese vectors coprse a tangent plane and are orthogonal to the gradents of the actve constrants. For ore nforaton about the suffcent condtons, see Luenberger, (984), Fletcher (987) or Edgar et al. (00). 4 References Fletcher, R., Practcal Methods of Optzaton nd Ed., Wley, 987. Luenberger, D. G., Lnear and Nonlnear Prograng, nd Ed, Addson Wesley, 984 Edgar,. F., Helblau, D. M., Lasdon, L. S., Optzaton of Checal Processes, nd Ed., McGraw-Hll, 00. 7
CHAPTER 7 CONSTRAINED OPTIMIZATION 1: THE KARUSH-KUHN-TUCKER CONDITIONS
CHAPER 7 CONSRAINED OPIMIZAION : HE KARUSH-KUHN-UCKER CONDIIONS 7. Introducton We now begn our dscusson of gradent-based constraned optzaton. Recall that n Chapter 3 we looked at gradent-based unconstraned
More informationXiangwen Li. March 8th and March 13th, 2001
CS49I Approxaton Algorths The Vertex-Cover Proble Lecture Notes Xangwen L March 8th and March 3th, 00 Absolute Approxaton Gven an optzaton proble P, an algorth A s an approxaton algorth for P f, for an
More informationPreference and Demand Examples
Dvson of the Huantes and Socal Scences Preference and Deand Exaples KC Border October, 2002 Revsed Noveber 206 These notes show how to use the Lagrange Karush Kuhn Tucker ultpler theores to solve the proble
More informationWhat is LP? LP is an optimization technique that allocates limited resources among competing activities in the best possible manner.
(C) 998 Gerald B Sheblé, all rghts reserved Lnear Prograng Introducton Contents I. What s LP? II. LP Theor III. The Splex Method IV. Refneents to the Splex Method What s LP? LP s an optzaton technque that
More informationPHYS 705: Classical Mechanics. Calculus of Variations II
1 PHYS 705: Classcal Mechancs Calculus of Varatons II 2 Calculus of Varatons: Generalzaton (no constrant yet) Suppose now that F depends on several dependent varables : We need to fnd such that has a statonary
More informationCHAPTER 7 CONSTRAINED OPTIMIZATION 2: SQP AND GRG
Chapter 7: Constraned Optmzaton CHAPER 7 CONSRAINED OPIMIZAION : SQP AND GRG Introducton In the prevous chapter we eamned the necessary and suffcent condtons for a constraned optmum. We dd not, however,
More informationPhysics 5153 Classical Mechanics. Principle of Virtual Work-1
P. Guterrez 1 Introducton Physcs 5153 Classcal Mechancs Prncple of Vrtual Work The frst varatonal prncple we encounter n mechancs s the prncple of vrtual work. It establshes the equlbrum condton of a mechancal
More informationFermi-Dirac statistics
UCC/Physcs/MK/EM/October 8, 205 Fer-Drac statstcs Fer-Drac dstrbuton Matter partcles that are eleentary ostly have a type of angular oentu called spn. hese partcles are known to have a agnetc oent whch
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationExcess Error, Approximation Error, and Estimation Error
E0 370 Statstcal Learnng Theory Lecture 10 Sep 15, 011 Excess Error, Approxaton Error, and Estaton Error Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton So far, we have consdered the fnte saple
More informationSolutions to exam in SF1811 Optimization, Jan 14, 2015
Solutons to exam n SF8 Optmzaton, Jan 4, 25 3 3 O------O -4 \ / \ / The network: \/ where all lnks go from left to rght. /\ / \ / \ 6 O------O -5 2 4.(a) Let x = ( x 3, x 4, x 23, x 24 ) T, where the varable
More informationLeast Squares Fitting of Data
Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC http://www.geoetrctools.co/ Copyrght c 1998-2014. All Rghts Reserved. Created: July 15, 1999 Last Modfed: February 9, 2008 Contents 1 Lnear Fttng
More informationSystem in Weibull Distribution
Internatonal Matheatcal Foru 4 9 no. 9 94-95 Relablty Equvalence Factors of a Seres-Parallel Syste n Webull Dstrbuton M. A. El-Dacese Matheatcs Departent Faculty of Scence Tanta Unversty Tanta Egypt eldacese@yahoo.co
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationElastic Collisions. Definition: two point masses on which no external forces act collide without losing any energy.
Elastc Collsons Defnton: to pont asses on hch no external forces act collde thout losng any energy v Prerequstes: θ θ collsons n one denson conservaton of oentu and energy occurs frequently n everyday
More informationPhysics 5153 Classical Mechanics. D Alembert s Principle and The Lagrangian-1
P. Guterrez Physcs 5153 Classcal Mechancs D Alembert s Prncple and The Lagrangan 1 Introducton The prncple of vrtual work provdes a method of solvng problems of statc equlbrum wthout havng to consder the
More informationThe General Nonlinear Constrained Optimization Problem
St back, relax, and enjoy the rde of your lfe as we explore the condtons that enable us to clmb to the top of a concave functon or descend to the bottom of a convex functon whle constraned wthn a closed
More informationRecap: the SVM problem
Machne Learnng 0-70/5-78 78 Fall 0 Advanced topcs n Ma-Margn Margn Learnng Erc Xng Lecture 0 Noveber 0 Erc Xng @ CMU 006-00 Recap: the SVM proble We solve the follong constraned opt proble: a s.t. J 0
More information14 Lagrange Multipliers
Lagrange Multplers 14 Lagrange Multplers The Method of Lagrange Multplers s a powerful technque for constraned optmzaton. Whle t has applcatons far beyond machne learnng t was orgnally developed to solve
More informationLeast Squares Fitting of Data
Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC http://www.geoetrctools.co/ Copyrght c 1998-2015. All Rghts Reserved. Created: July 15, 1999 Last Modfed: January 5, 2015 Contents 1 Lnear Fttng
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationAssortment Optimization under MNL
Assortment Optmzaton under MNL Haotan Song Aprl 30, 2017 1 Introducton The assortment optmzaton problem ams to fnd the revenue-maxmzng assortment of products to offer when the prces of products are fxed.
More informationMore metrics on cartesian products
More metrcs on cartesan products If (X, d ) are metrc spaces for 1 n, then n Secton II4 of the lecture notes we defned three metrcs on X whose underlyng topologes are the product topology The purpose of
More informationSlobodan Lakić. Communicated by R. Van Keer
Serdca Math. J. 21 (1995), 335-344 AN ITERATIVE METHOD FOR THE MATRIX PRINCIPAL n-th ROOT Slobodan Lakć Councated by R. Van Keer In ths paper we gve an teratve ethod to copute the prncpal n-th root and
More informationOPTIMISATION. Introduction Single Variable Unconstrained Optimisation Multivariable Unconstrained Optimisation Linear Programming
OPTIMIATION Introducton ngle Varable Unconstraned Optmsaton Multvarable Unconstraned Optmsaton Lnear Programmng Chapter Optmsaton /. Introducton In an engneerng analss, sometmes etremtes, ether mnmum or
More informationApplied Mathematics Letters
Appled Matheatcs Letters 2 (2) 46 5 Contents lsts avalable at ScenceDrect Appled Matheatcs Letters journal hoepage: wwwelseverco/locate/al Calculaton of coeffcents of a cardnal B-splne Gradr V Mlovanovć
More informationCOS 511: Theoretical Machine Learning
COS 5: Theoretcal Machne Learnng Lecturer: Rob Schapre Lecture #0 Scrbe: José Sões Ferrera March 06, 203 In the last lecture the concept of Radeacher coplexty was ntroduced, wth the goal of showng that
More informationOn Pfaff s solution of the Pfaff problem
Zur Pfaff scen Lösung des Pfaff scen Probles Mat. Ann. 7 (880) 53-530. On Pfaff s soluton of te Pfaff proble By A. MAYER n Lepzg Translated by D. H. Delpenc Te way tat Pfaff adopted for te ntegraton of
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationAN ANALYSIS OF A FRACTAL KINETICS CURVE OF SAVAGEAU
AN ANALYI OF A FRACTAL KINETIC CURE OF AAGEAU by John Maloney and Jack Hedel Departent of Matheatcs Unversty of Nebraska at Oaha Oaha, Nebraska 688 Eal addresses: aloney@unoaha.edu, jhedel@unoaha.edu Runnng
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationMachine Learning. What is a good Decision Boundary? Support Vector Machines
Machne Learnng 0-70/5 70/5-78 78 Sprng 200 Support Vector Machnes Erc Xng Lecture 7 March 5 200 Readng: Chap. 6&7 C.B book and lsted papers Erc Xng @ CMU 2006-200 What s a good Decson Boundar? Consder
More informationRevision: December 13, E Main Suite D Pullman, WA (509) Voice and Fax
.9.1: AC power analyss Reson: Deceber 13, 010 15 E Man Sute D Pullan, WA 99163 (509 334 6306 Voce and Fax Oerew n chapter.9.0, we ntroduced soe basc quanttes relate to delery of power usng snusodal sgnals.
More informationACTM State Calculus Competition Saturday April 30, 2011
ACTM State Calculus Competton Saturday Aprl 30, 2011 ACTM State Calculus Competton Sprng 2011 Page 1 Instructons: For questons 1 through 25, mark the best answer choce on the answer sheet provde Afterward
More information( ) 2 ( ) ( ) Problem Set 4 Suggested Solutions. Problem 1
Problem Set 4 Suggested Solutons Problem (A) The market demand functon s the soluton to the followng utlty-maxmzaton roblem (UMP): The Lagrangean: ( x, x, x ) = + max U x, x, x x x x st.. x + x + x y x,
More informationComputational and Statistical Learning theory Assignment 4
Coputatonal and Statstcal Learnng theory Assgnent 4 Due: March 2nd Eal solutons to : karthk at ttc dot edu Notatons/Defntons Recall the defnton of saple based Radeacher coplexty : [ ] R S F) := E ɛ {±}
More informationLecture 10 Support Vector Machines. Oct
Lecture 10 Support Vector Machnes Oct - 20-2008 Lnear Separators Whch of the lnear separators s optmal? Concept of Margn Recall that n Perceptron, we learned that the convergence rate of the Perceptron
More informationThe Multiple Classical Linear Regression Model (CLRM): Specification and Assumptions. 1. Introduction
ECONOMICS 5* -- NOTE (Summary) ECON 5* -- NOTE The Multple Classcal Lnear Regresson Model (CLRM): Specfcaton and Assumptons. Introducton CLRM stands for the Classcal Lnear Regresson Model. The CLRM s also
More informationInner Product. Euclidean Space. Orthonormal Basis. Orthogonal
Inner Product Defnton 1 () A Eucldean space s a fnte-dmensonal vector space over the reals R, wth an nner product,. Defnton 2 (Inner Product) An nner product, on a real vector space X s a symmetrc, blnear,
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationStructure and Drive Paul A. Jensen Copyright July 20, 2003
Structure and Drve Paul A. Jensen Copyrght July 20, 2003 A system s made up of several operatons wth flow passng between them. The structure of the system descrbes the flow paths from nputs to outputs.
More informationOn the number of regions in an m-dimensional space cut by n hyperplanes
6 On the nuber of regons n an -densonal space cut by n hyperplanes Chungwu Ho and Seth Zeran Abstract In ths note we provde a unfor approach for the nuber of bounded regons cut by n hyperplanes n general
More informationFinite Vector Space Representations Ross Bannister Data Assimilation Research Centre, Reading, UK Last updated: 2nd August 2003
Fnte Vector Space epresentatons oss Bannster Data Asslaton esearch Centre, eadng, UK ast updated: 2nd August 2003 Contents What s a lnear vector space?......... 1 About ths docuent............ 2 1. Orthogonal
More informationAPPENDIX A Some Linear Algebra
APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly,
More informationn α j x j = 0 j=1 has a nontrivial solution. Here A is the n k matrix whose jth column is the vector for all t j=0
MODULE 2 Topcs: Lnear ndependence, bass and dmenson We have seen that f n a set of vectors one vector s a lnear combnaton of the remanng vectors n the set then the span of the set s unchanged f that vector
More informationCHAPTER 3 UNCONSTRAINED OPTIMIZATION
. Prelmnares CHAPER 3 UNCONSRAINED OPIMIZAION.. Introducton In ths chapter we wll examne some theory for the optmzaton of unconstraned functons. We wll assume all functons are contnuous and dfferentable.
More informationSolving Fuzzy Linear Programming Problem With Fuzzy Relational Equation Constraint
Intern. J. Fuzz Maeatcal Archve Vol., 0, -0 ISSN: 0 (P, 0 0 (onlne Publshed on 0 Septeber 0 www.researchasc.org Internatonal Journal of Solvng Fuzz Lnear Prograng Proble W Fuzz Relatonal Equaton Constrant
More informationDifference Equations
Dfference Equatons c Jan Vrbk 1 Bascs Suppose a sequence of numbers, say a 0,a 1,a,a 3,... s defned by a certan general relatonshp between, say, three consecutve values of the sequence, e.g. a + +3a +1
More information,..., k N. , k 2. ,..., k i. The derivative with respect to temperature T is calculated by using the chain rule: & ( (5) dj j dt = "J j. k i.
Suppleentary Materal Dervaton of Eq. 1a. Assue j s a functon of the rate constants for the N coponent reactons: j j (k 1,,..., k,..., k N ( The dervatve wth respect to teperature T s calculated by usng
More informationXII.3 The EM (Expectation-Maximization) Algorithm
XII.3 The EM (Expectaton-Maxzaton) Algorth Toshnor Munaata 3/7/06 The EM algorth s a technque to deal wth varous types of ncoplete data or hdden varables. It can be appled to a wde range of learnng probles
More information6.854J / J Advanced Algorithms Fall 2008
MIT OpenCourseWare http://ocw.mt.edu 6.854J / 18.415J Advanced Algorthms Fall 2008 For nformaton about ctng these materals or our Terms of Use, vst: http://ocw.mt.edu/terms. 18.415/6.854 Advanced Algorthms
More informationSection 8.3 Polar Form of Complex Numbers
80 Chapter 8 Secton 8 Polar Form of Complex Numbers From prevous classes, you may have encountered magnary numbers the square roots of negatve numbers and, more generally, complex numbers whch are the
More informationLECTURE :FACTOR ANALYSIS
LCUR :FACOR ANALYSIS Rta Osadchy Based on Lecture Notes by A. Ng Motvaton Dstrbuton coes fro MoG Have suffcent aount of data: >>n denson Use M to ft Mture of Gaussans nu. of tranng ponts If
More informationWeek3, Chapter 4. Position and Displacement. Motion in Two Dimensions. Instantaneous Velocity. Average Velocity
Week3, Chapter 4 Moton n Two Dmensons Lecture Quz A partcle confned to moton along the x axs moves wth constant acceleraton from x =.0 m to x = 8.0 m durng a 1-s tme nterval. The velocty of the partcle
More informationf(x,y) = (4(x 2 4)x,2y) = 0 H(x,y) =
Problem Set 3: Unconstraned mzaton n R N. () Fnd all crtcal ponts of f(x,y) (x 4) +y and show whch are ma and whch are mnma. () Fnd all crtcal ponts of f(x,y) (y x ) x and show whch are ma and whch are
More informationMMA and GCMMA two methods for nonlinear optimization
MMA and GCMMA two methods for nonlnear optmzaton Krster Svanberg Optmzaton and Systems Theory, KTH, Stockholm, Sweden. krlle@math.kth.se Ths note descrbes the algorthms used n the author s 2007 mplementatons
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationLecture 20: Lift and Project, SDP Duality. Today we will study the Lift and Project method. Then we will prove the SDP duality theorem.
prnceton u. sp 02 cos 598B: algorthms and complexty Lecture 20: Lft and Project, SDP Dualty Lecturer: Sanjeev Arora Scrbe:Yury Makarychev Today we wll study the Lft and Project method. Then we wll prove
More informationGradient Descent Learning and Backpropagation
Artfcal Neural Networks (art 2) Chrstan Jacob Gradent Descent Learnng and Backpropagaton CSC 533 Wnter 200 Learnng by Gradent Descent Defnton of the Learnng roble Let us start wth the sple case of lnear
More informationErrors for Linear Systems
Errors for Lnear Systems When we solve a lnear system Ax b we often do not know A and b exactly, but have only approxmatons  and ˆb avalable. Then the best thng we can do s to solve ˆx ˆb exactly whch
More informationThe Minimum Universal Cost Flow in an Infeasible Flow Network
Journal of Scences, Islamc Republc of Iran 17(2): 175-180 (2006) Unversty of Tehran, ISSN 1016-1104 http://jscencesutacr The Mnmum Unversal Cost Flow n an Infeasble Flow Network H Saleh Fathabad * M Bagheran
More informationDenote the function derivatives f(x) in given points. x a b. Using relationships (1.2), polynomials (1.1) are written in the form
SET OF METHODS FO SOUTION THE AUHY POBEM FO STIFF SYSTEMS OF ODINAY DIFFEENTIA EUATIONS AF atypov and YuV Nulchev Insttute of Theoretcal and Appled Mechancs SB AS 639 Novosbrs ussa Introducton A constructon
More informationELASTIC WAVE PROPAGATION IN A CONTINUOUS MEDIUM
ELASTIC WAVE PROPAGATION IN A CONTINUOUS MEDIUM An elastc wave s a deformaton of the body that travels throughout the body n all drectons. We can examne the deformaton over a perod of tme by fxng our look
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationThe equation of motion of a dynamical system is given by a set of differential equations. That is (1)
Dynamcal Systems Many engneerng and natural systems are dynamcal systems. For example a pendulum s a dynamcal system. State l The state of the dynamcal system specfes t condtons. For a pendulum n the absence
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationSingular Value Decomposition: Theory and Applications
Sngular Value Decomposton: Theory and Applcatons Danel Khashab Sprng 2015 Last Update: March 2, 2015 1 Introducton A = UDV where columns of U and V are orthonormal and matrx D s dagonal wth postve real
More informationScattering by a perfectly conducting infinite cylinder
Scatterng by a perfectly conductng nfnte cylnder Reeber that ths s the full soluton everywhere. We are actually nterested n the scatterng n the far feld lt. We agan use the asyptotc relatonshp exp exp
More informationTransfer Functions. Convenient representation of a linear, dynamic model. A transfer function (TF) relates one input and one output: ( ) system
Transfer Functons Convenent representaton of a lnear, dynamc model. A transfer functon (TF) relates one nput and one output: x t X s y t system Y s The followng termnology s used: x y nput output forcng
More informationCollege of Computer & Information Science Fall 2009 Northeastern University 20 October 2009
College of Computer & Informaton Scence Fall 2009 Northeastern Unversty 20 October 2009 CS7880: Algorthmc Power Tools Scrbe: Jan Wen and Laura Poplawsk Lecture Outlne: Prmal-dual schema Network Desgn:
More informationChapter 12 Lyes KADEM [Thermodynamics II] 2007
Chapter 2 Lyes KDEM [Therodynacs II] 2007 Gas Mxtures In ths chapter we wll develop ethods for deternng therodynac propertes of a xture n order to apply the frst law to systes nvolvng xtures. Ths wll be
More informationANSWERS. Problem 1. and the moment generating function (mgf) by. defined for any real t. Use this to show that E( U) var( U)
Econ 413 Exam 13 H ANSWERS Settet er nndelt 9 deloppgaver, A,B,C, som alle anbefales å telle lkt for å gøre det ltt lettere å stå. Svar er gtt . Unfortunately, there s a prntng error n the hnt of
More information1 Review From Last Time
COS 5: Foundatons of Machne Learnng Rob Schapre Lecture #8 Scrbe: Monrul I Sharf Aprl 0, 2003 Revew Fro Last Te Last te, we were talkng about how to odel dstrbutons, and we had ths setup: Gven - exaples
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationMathematical Economics MEMF e ME. Filomena Garcia. Topic 2 Calculus
Mathematcal Economcs MEMF e ME Flomena Garca Topc 2 Calculus Mathematcal Economcs - www.seg.utl.pt/~garca/economa_matematca . Unvarate Calculus Calculus Functons : X Y y ( gves or each element X one element
More informationModule 3: Element Properties Lecture 1: Natural Coordinates
Module 3: Element Propertes Lecture : Natural Coordnates Natural coordnate system s bascally a local coordnate system whch allows the specfcaton of a pont wthn the element by a set of dmensonless numbers
More informationMath1110 (Spring 2009) Prelim 3 - Solutions
Math 1110 (Sprng 2009) Solutons to Prelm 3 (04/21/2009) 1 Queston 1. (16 ponts) Short answer. Math1110 (Sprng 2009) Prelm 3 - Solutons x a 1 (a) (4 ponts) Please evaluate lm, where a and b are postve numbers.
More informationBezier curves. Michael S. Floater. August 25, These notes provide an introduction to Bezier curves. i=0
Bezer curves Mchael S. Floater August 25, 211 These notes provde an ntroducton to Bezer curves. 1 Bernsten polynomals Recall that a real polynomal of a real varable x R, wth degree n, s a functon of the
More informationPHYS 1443 Section 002 Lecture #20
PHYS 1443 Secton 002 Lecture #20 Dr. Jae Condtons for Equlbru & Mechancal Equlbru How to Solve Equlbru Probles? A ew Exaples of Mechancal Equlbru Elastc Propertes of Solds Densty and Specfc Gravty lud
More informationExercise Solutions to Real Analysis
xercse Solutons to Real Analyss Note: References refer to H. L. Royden, Real Analyss xersze 1. Gven any set A any ɛ > 0, there s an open set O such that A O m O m A + ɛ. Soluton 1. If m A =, then there
More information4 Column generation (CG) 4.1 Basics of column generation. 4.2 Applying CG to the Cutting-Stock Problem. Basic Idea of column generation
4 Colun generaton (CG) here are a lot of probles n nteger prograng where even the proble defnton cannot be effcently bounded Specfcally, the nuber of coluns becoes very large herefore, these probles are
More informationTHE CHINESE REMAINDER THEOREM. We should thank the Chinese for their wonderful remainder theorem. Glenn Stevens
THE CHINESE REMAINDER THEOREM KEITH CONRAD We should thank the Chnese for ther wonderful remander theorem. Glenn Stevens 1. Introducton The Chnese remander theorem says we can unquely solve any par of
More informationCHAPTER 10 ROTATIONAL MOTION
CHAPTER 0 ROTATONAL MOTON 0. ANGULAR VELOCTY Consder argd body rotates about a fxed axs through pont O n x-y plane as shown. Any partcle at pont P n ths rgd body rotates n a crcle of radus r about O. The
More informationIntegrals and Invariants of Euler-Lagrange Equations
Lecture 16 Integrals and Invarants of Euler-Lagrange Equatons ME 256 at the Indan Insttute of Scence, Bengaluru Varatonal Methods and Structural Optmzaton G. K. Ananthasuresh Professor, Mechancal Engneerng,
More informationy new = M x old Feature Selection: Linear Transformations Constraint Optimization (insertion)
Feature Selecton: Lnear ransforatons new = M x old Constrant Optzaton (nserton) 3 Proble: Gven an objectve functon f(x) to be optzed and let constrants be gven b h k (x)=c k, ovng constants to the left,
More informationProseminar Optimierung II. Victor A. Kovtunenko SS 2012/2013: LV
Prosemnar Optmerung II Vctor A. Kovtunenko Insttute for Mathematcs and Scentfc Computng, Karl-Franzens Unversty of Graz, Henrchstr. 36, 8010 Graz, Austra; Lavrent ev Insttute of Hydrodynamcs, Sberan Dvson
More informationNatural Language Processing and Information Retrieval
Natural Language Processng and Informaton Retreval Support Vector Machnes Alessandro Moschtt Department of nformaton and communcaton technology Unversty of Trento Emal: moschtt@ds.untn.t Summary Support
More informationCanonical transformations
Canoncal transformatons November 23, 2014 Recall that we have defned a symplectc transformaton to be any lnear transformaton M A B leavng the symplectc form nvarant, Ω AB M A CM B DΩ CD Coordnate transformatons,
More informationLecture 14: Bandits with Budget Constraints
IEOR 8100-001: Learnng and Optmzaton for Sequental Decson Makng 03/07/16 Lecture 14: andts wth udget Constrants Instructor: Shpra Agrawal Scrbed by: Zhpeng Lu 1 Problem defnton In the regular Mult-armed
More informationSolution Thermodynamics
Soluton hermodynamcs usng Wagner Notaton by Stanley. Howard Department of aterals and etallurgcal Engneerng South Dakota School of nes and echnology Rapd Cty, SD 57701 January 7, 001 Soluton hermodynamcs
More informationHow Strong Are Weak Patents? Joseph Farrell and Carl Shapiro. Supplementary Material Licensing Probabilistic Patents to Cournot Oligopolists *
How Strong Are Weak Patents? Joseph Farrell and Carl Shapro Supplementary Materal Lcensng Probablstc Patents to Cournot Olgopolsts * September 007 We study here the specal case n whch downstream competton
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More informationPROBLEM SET 7 GENERAL EQUILIBRIUM
PROBLEM SET 7 GENERAL EQUILIBRIUM Queston a Defnton: An Arrow-Debreu Compettve Equlbrum s a vector of prces {p t } and allocatons {c t, c 2 t } whch satsfes ( Gven {p t }, c t maxmzes βt ln c t subject
More information15 Lagrange Multipliers
15 The Method of s a powerful technque for constraned optmzaton. Whle t has applcatons far beyond machne learnng t was orgnally developed to solve physcs equatons), t s used for several ey dervatons n
More informationSolutions HW #2. minimize. Ax = b. Give the dual problem, and make the implicit equality constraints explicit. Solution.
Solutons HW #2 Dual of general LP. Fnd the dual functon of the LP mnmze subject to c T x Gx h Ax = b. Gve the dual problem, and make the mplct equalty constrants explct. Soluton. 1. The Lagrangan s L(x,
More informationLagrange Multipliers. A Somewhat Silly Example. Monday, 25 September 2013
Lagrange Multplers Monday, 5 September 013 Sometmes t s convenent to use redundant coordnates, and to effect the varaton of the acton consstent wth the constrants va the method of Lagrange undetermned
More informationMachine Learning. Support Vector Machines. Eric Xing , Fall Lecture 9, October 6, 2015
Machne Learnng 0-70 Fall 205 Support Vector Machnes Erc Xng Lecture 9 Octoer 6 205 Readng: Chap. 6&7 C.B ook and lsted papers Erc Xng @ CMU 2006-205 What s a good Decson Boundar? Consder a nar classfcaton
More informationTemperature. Chapter Heat Engine
Chapter 3 Temperature In prevous chapters of these notes we ntroduced the Prncple of Maxmum ntropy as a technque for estmatng probablty dstrbutons consstent wth constrants. In Chapter 9 we dscussed the
More informationThe Order Relation and Trace Inequalities for. Hermitian Operators
Internatonal Mathematcal Forum, Vol 3, 08, no, 507-57 HIKARI Ltd, wwwm-hkarcom https://doorg/0988/mf088055 The Order Relaton and Trace Inequaltes for Hermtan Operators Y Huang School of Informaton Scence
More information