Differentiable Convex Functions
|
|
- Frederick Dean
- 6 years ago
- Views:
Transcription
1 Differetiable Covex Fuctios The followig picture motivates Theorem 11. f ( x) f ( x) f '( x)( x x) ˆx x 1 Theorem 11 : Let f : R R be differetiable. The, f is covex o the covex set C R if, ad oly if for all x, xc. f( x) f( x) f( x) T ( xx) 1
2 Differetiable Covex Fuctios f C x Proof : Assume is covex o. The, for, x C we have f ( x(1 ) x) f( x) (1 ) f( x ) all [0, 1] or f( x( xx)) f( x) ( f( x) f( x)) all [0, 1] or f( x( xx)) f( x) f( x) f( x) all (0, 1] T f( x ( xx)) f( x) f( x) ( xx) lim f( x) f( x) 0 2
3 Differetiable Covex Fuctios Coversely, assume f( x) f( x) f( x) T ( xx) for all x, xc Let x, x C, ad let [0, 1]. Defie x x (1 ) x ; the f f f 1 1 ( x ) ( x) ( x) T ( x x) ad: f f f 2 2 ( x ) ( x) ( x) T ( x x) f f f f 1 2 T 1 2 ( x ) (1 ) ( x ) ( x) ( x) ( x (1 ) x x) f 1 2 ( x) f( x (1 ) x ) 3
4 Differetiable Covex Fuctios HW 31 : We already ow, for ay covex fuctio f : R * * ad ay covex set C, that if x is a local miimum the x is a global miimum. Without usig this owledge, show directly that the first order ecessary coditios of Theorem 4 are also sufficiet for * x to be a global miimum for mi f ( x) whe f ad C are covex. xc R 1 4
5 Differetiable Covex Fuctios We ow provide a ecessary ad sufficiet coditio for f : R to be covex whe, i additio, f has cotiuous secod partials. Before doig this, ote the followig rather obvious geometric fact. 1 1 If f : R R is covex ad f has a secod derivative f '', 1 the f ''( x) 0 all x R. We will ot prove this result. R 1 5
6 Differetiable Covex Fuctios 1 Theorem 12 : Let C R be covex ad ope ad let f : C R have cotiuous secod partials o. The f is covex o C if, ad oly if, the Hessia matrix of f, H, is PSD o all of C. C xc (i.e., Hx ( ) is PSD for all ) Proof : Assume H is PSD o C. By Taylor s Theorem we have, x xc for ay,, 1 f x f x f x xx xx H x x xx 2 T f( x) f( x) ( xx) T T ( ) ( ) ( ) ( ) ( ) ( (1 ) )( ) 6
7 Differetiable Covex Fuctios ad therefore, by Theorem 11, f is covex o C. Coversely, assume f is covex o C. Let xc ad let d R ad defie g by g( ) f( xd) Sice C is ope ad f is covex o C, it follows that g is covex i some eighborhood of 0. Therefore, it must be the case that g ''(0) 0 ad, i fact, g ''( ) 0 i this eighborhood of 0. Therefore dhxd Hx is PSD for all xc T 0 g ''(0) ( ) ( ) 7
8 Differetiable Covex Fuctios HW32 : I additio to the hypothesis of Theorem 12, assume H is PD o C. Does this imply f is strictly covex o C? If f is strictly covex o C is it true that H is PD o C? 8
9 Feasible Directio Philosophy The cocept of directioal derivative is very strogly used for may types of algorithms for solvig the problem mi f ( x) xfr 1 where f : R R is differetiable. The simple idea is merely: whe at a poit x F move away from x i the best directio ad as far as possible. 9
10 Feasible Directio Philosophy Step 0 : Set 0 ad select x F. Step 1 : Fid a d D( x, F) which provides a strict improvemet for f. If oe, go to Step 3. Step 2 : Let 0 be chose so that f( x d ) f( x ) ad 1 x d F. Set x x d, 1, ad retur to Step 1. Step 3 : Stop. 10
11 Remars o the Above Procedure Some remars are i order. R1 (Step 1) : This step is ofte carried out by solvig the optimizatio problem (which we ll call the directio fidig problem, deoted by D ) T mi f ( x ) d s. t. d D( x, F) (D ) d 1 11
12 Remars o the Above Procedure Note that if (P) is a ucostraied problem (i.e., F R ) ad if we tae d i1 d 2 i 1 2 the the optimal vector, or directio, for D is, of course, d f ( x ) f ( x ) 12
13 Remars o the Above Procedure HW33 : Assume, agai, F R ad we are usig d the so-called sup orm. Write a expressio for d, a optimal solutio for D. max { d } j1,, j [Note : D ca equivaletly be writte by replacig d 1 with d 1. Why?] 13
14 Remars o the Above Procedure HW34 : Repeat HW33 by usig d d j. If we let d D( x, F) deote a optimal vector for D, ad if it turs out that j1 f T ( x ) d 0 the the directioal derivative, from x, i every feasible directio d D( x, F) is positive ad, therefore, a move i ay such directio will actually icrease the objective from its curret value f ( x ). Therefore, by our above procedure, we go to Step 3, i.e., we stop at x F. 14
15 Remars o the Above Procedure T HW35 : Does f ( x ) d 0 imply there is o feasible directio of strict decrease for f? Note that D may ot be easily solved. However, if F is covex, D x F yx yf, y x the sice (, ) may be tae to be we see that D ca the be writte as T mi f ( x ) ( yx ) s. t. y F y x 1 (DC ) where (DC ) meas the directio fidig problem whe F is covex. 15
16 Remars o the Above Procedure Note also that if, where X xr x0 R, we have that (DC ) becomes mi f ( x ) T ( yx ) Ay b y 0 y x ad further ote that, if we use F x X Ax b s. t. 1 z max ( z ) i1,, i this directio fidig problem is a liear programmig problem. 16
17 Remars o the Above Procedure R2 (Step 2) : If a d D( x, F) is foud at Step 1 which does lead to a strict decrease for f, from x, the ofte is chose as a miimizer for the oe-dimesioal optimizatio problem, ofte also called the lie-search problem mi f ( x d ) 0 s.t. x d F (LS ) T Note : If d D( x, F ) is such that f ( x ) d 0 the (if it exists) is such that 0. 17
18 Remars o the Above Procedure R3 (Step 3) : Whe we eter Step 3 (if we ever do) there is certaily o guaratee that x is optimal. T HW36 : Provide a example where f ( x ) d 0 all dd( x, F) x is globally optimal for mi f ( x). What about locally optimal? xf However, because of HW31, if f is covex o F covex ad T if f ( x ) d 0 for all d D( x, F) the it does follow that x is globally optimal. 18
19 Remars o the Above Procedure R4 : Note also that we have ot discussed covergece i.e., if we ever eter Step 3, does the algorithm produce a sequece of poits which evetually comes close to some poit from which we caot escape (by usig the algorithm)? Such topics will be discussed later. 19
20 Cauchy s Steepest Descet Algorithm For the ucostraied differetiable problem mi ( x) xfr the above feasible directio procedure is called Cauchy s Steepest 2 Descet Algorithm whe (i) we use x x i (i.e., the directio vector at Step 1 is tae to be f ( x ) d f ( x ) f ad (ii) at Step 2 we solve mi f ( x d ) 0 20
21 Cauchy s Steepest Descet Algorithm HW37 : f ( x ) (a) Show that f ( x ) 0 implies d is a directio f ( x ) (feasible) of strict decrease for f at x (for the ucostraied problem). (b) Show that Cauchy s method for the ucostraied problem zig-zags 1 (i.e., dd 0 ). 21
22 Cauchy s Steepest Descet Algorithm HW38 : (a) Do 2 or 3 iteratios of Cauchy s method for solvig mi x 2x ( x, x ) R 1 2 startig with the poit. x 0 2 (1,1) (b) Now mae the substitutio y 2x ad use Cauchy s method for mi ( x, y) R 1 2 x y What happes? 2 22
23 Newto-Raphso Algorithm Agai, cosider the ucostraied differetiable optimizatio problem mi f ( x) xfr Let A be a symmetric positive defiite matrix. Cosider the fuctio defied by : R R A 1 x A ( x T Ax) 1/2 23
24 Newto-Raphso Algorithm HW39 : Show that is a orm. That is, show A (a) x 0 for all x R ad x 0 x0. A (b) x x for all 1 x R ad all R. A (c) xy x y. A A A A A 24
25 Newto-Raphso Algorithm HW40 : Let x R ad cosider the directio fidig problem mi f ( x ) T d d A 1 where A is symmetric ad PD. Show that d A 1 1 f A ( x ) A f( x ) f f f ( x ) T 1 1 ( x ) A ( x ) A 1/2 is a optimal vector for D K. 25
26 Newto-Raphso Algorithm If each time we eter Step 1, we solve the directio fidig problem D K usig as our orm o the fuctio (If the Hessia is PD R H( x ) at x ) we see that a optimal directio (because of HW 40) is d 1 H x f x f ( ) ( ) ( x ) 1 H ( x ) Whe this directio is used for the ucostraied differetiable optimizatio problem the feasible directio algorithm above is called the Newto-Raphso procedure. Also, whe Step 2 is etered, is chose accordig to mi f ( x d ). 0 26
27 Newto-Raphso Algorithm HW41 : 1 (a) Show that if A is PD the so is A. (b) Show that if f ( x ) 0 ad if Hx ( ) is PD the d is a directio of strict decrease for f at x ad, therefore, at Step 2 we ll have 0. What may happe if f ( x ) 0 ad if 1 H ( x ) is ot PSD; that is if we still use 1 d H ( x ) f ( x ). Is it ecessarily true that d will be a directio of strict decrease for f? 27
28 Newto-Raphso Algorithm 1 HW42 : Let f : R R be differetiable ad cosider the Newto-Raphso procedure for mi f ( x). Show that at each iteratio (assume Hx ( ) is xfr PD o R ) the directio 1 ( ) ( d H x f x ) poits, from x, to the vector which miimizes the quadratic approximatio for f at x. 1 T 1 T HW43 : Let f : R R be defied by f ( x) c x x Qx where Q is 2 a symmetric positive defiite matrix. Cosider mi f ( x) startig from xfr 0 ay poit x R. Show that the Newto-Raphso procedure termiates i 1 0 oe step. (Note: the problem of HW38 is of this form : c 0, Q )
29 A The Fra-Wolfe Algorithm Let be a m matrix ad let xr { xr x0}. 1 Let f : R R be differetiable ad cosider the problem mi f ( x) Axb x0 I this case, the directio fidig problem, at iteratio, is T mi f ( x ) ( y x ) s. t. Ay b y 0 y x 1 29
30 The Fra-Wolfe Algorithm Note that if F { x0 Axb} is bouded, the we ca also D tae to be T mi f ( x ) ( yx ) s. t. Ay b y b sice feasible y s ca ot get too large i magitude F (i.e., y is bouded o ). 30
31 The Fra-Wolfe Algorithm HW44 : Suppose F is ot bouded ad at some iteratio, mi f ( x ) T ( yx ). y0 Ayb Show that we ca still choose a directio d which yields a strict decrease for f at x. [Hit: Cosider the extreme-ray foud by the simplex algorithm whe solvig D.] 31
32 The Fra-Wolfe Algorithm If y F is optimal for D ad f ( x ) T ( yx ) 0, the Step 2 is etered with d y x ad we the solve for 0 by solvig [0,1] mi f x ( y x ) 32
33 The Fra-Wolfe Algorithm x y x F Note: By covexity of F we have ( ) for all [0,1]. If ( ) T ( f x yx ) 0, we eter Step 3 with x. Note: If f is covex, the x is optimal if T f ( x ) ( yx ) 0. 33
34 The Fra-Wolfe Algorithm Proof: By defiitio of y we have that, for all y F, T f ( x ) ( yx ) 0. By covexity of f we have, for all y f( y) f( x ) f( x ) T ( yx ) f( x ) To be specific we formally state the F.-W. algorithm for mi f ( x) Ax b x 0 F 34
35 The Fra-Wolfe Algorithm The emphasis is upo the otio of feasible directio we are ot ecessarily recommedig this particular method for solvig oliear problems with liear costraits. 35
36 The Fra-Wolfe Algorithm Step 0: Set 0 ad select x F{ x0 Axb }. Step 1: Let T y solve f x yx Ayb y0 mi ( ) ( ) (D ) ad set T d y x. If f ( x ) d 0, go to Step 4. T If f ( x ) d 0, go to Step 3. If, o the other had, D has o optimal solutio, go to Step 2. 36
37 The Fra-Wolfe Algorithm Step 2: Let y be the extreme ray located by the simplex D algorithm whe solvig the liear program. Set d y x ad go to Step 3. Step 3: Let 0 solve mi f ( x d ) ad set [0,1] 1 x x d. Set 1 Step 4: Stop at the vector x. ad retur to Step 1. 37
38 The Fra-Wolfe Algorithm Note that if we do eter Step 4 at some iteratio the x is globally optimal if f is covex. If f is ot covex, the it will geerally be the case that x is ot globally optimal. 1 Defiitio: A differetiable fuctio f : R R (or f : C where C is a ope covex set) is said to be pseudocovex if f ( x) ( xx ) 0 implies f ( x) f ( x) (or, T equivaletly, f ( x) f ( x ) implies f ( x) T ( xx) 0 ). R 1 38
39 The Fra-Wolfe Algorithm Pseudo-covex Not pseudo-covex It follows immediately from the defiitio that a ecessary ad sufficiet coditio for x * F covex to globally optimize mi f ( x) where f is pseudo-covex o R (or xf o some ope covex set cotaiig F ) is for all xf. f * T * ( x ) ( x x ) 0 39
40 The Fra-Wolfe Algorithm HW45: (a) Show this latter claim (b) Show Step 4 of the F.-W. algorithm implies global optimality of x if f is pseudo-covex (c) Show that a differetiable covex fuctio is pseudocovex. Note: f is pseudo-cocave if f is pseudo-covex T (or f ( x) ( xx ) 0 f( x) f( x) ). 40
41 The Fra-Wolfe Algorithm Note: We have ot yet discussed covergece of the F.-W. algorithm. That is, if we ever eter Step 4, does the algorithm evetually come close to some poit at which we may wish to stop? We ll discuss this later i the course. Note that i the F.-W. algorithm (as well as for Cauchy s Steepest Descet, ad the Newto-Raphso method (if Hx ( ) is PD) we have 1 f( x ) f( x ) for all. Therefore, a upper boud for mi f ( x) is f ( x ) at iteratio. For the F.-W. algorithm, whe f is covex, we ca also develop a lower boud. xf 41
42 The Fra-Wolfe Algorithm By covexity we have T f( x ) f( x ) ( yx ) f( y), for all yf T f( x ) f( x ) ( yx ) mi f( y), at each iteratio ad, therefore, at iteratio K we may tae 0,1,, K Therefore, at iteratio K of the F.-W. algorithm we have a lower K boud L ad a upper boud U f( x ). K yf L max f( x ) f( x )( yx ) K 42
43 The Fra-Wolfe Algorithm Defiitio: Cosider a arbitrary optimizatio problem mi f( x). The vector 1 xf x F is said to be a -optimal solutio ( 0) 1 * * if f( x ) f( x ) where x is optimal. Therefore, whe f is covex ad we are applyig the F.-W. algorithm if ever we have U L K K for some preassiged tolerace parameter 0, the we may stop at the vector x with the assurace that f ( x ) differs form the optimal value by o more tha. 43
Brief Review of Functions of Several Variables
Brief Review of Fuctios of Several Variables Differetiatio Differetiatio Recall, a fuctio f : R R is differetiable at x R if ( ) ( ) lim f x f x 0 exists df ( x) Whe this limit exists we call it or f(
More informationIntroduction to Optimization Techniques. How to Solve Equations
Itroductio to Optimizatio Techiques How to Solve Equatios Iterative Methods of Optimizatio Iterative methods of optimizatio Solutio of the oliear equatios resultig form a optimizatio problem is usually
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationOptimization Methods MIT 2.098/6.255/ Final exam
Optimizatio Methods MIT 2.098/6.255/15.093 Fial exam Date Give: December 19th, 2006 P1. [30 pts] Classify the followig statemets as true or false. All aswers must be well-justified, either through a short
More informationChapter 6 Infinite Series
Chapter 6 Ifiite Series I the previous chapter we cosidered itegrals which were improper i the sese that the iterval of itegratio was ubouded. I this chapter we are goig to discuss a topic which is somewhat
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationHomework 4. x n x X = f(x n x) +
Homework 4 1. Let X ad Y be ormed spaces, T B(X, Y ) ad {x } a sequece i X. If x x weakly, show that T x T x weakly. Solutio: We eed to show that g(t x) g(t x) g Y. It suffices to do this whe g Y = 1.
More informationMath 312 Lecture Notes One Dimensional Maps
Math 312 Lecture Notes Oe Dimesioal Maps Warre Weckesser Departmet of Mathematics Colgate Uiversity 21-23 February 25 A Example We begi with the simplest model of populatio growth. Suppose, for example,
More informationMa 530 Introduction to Power Series
Ma 530 Itroductio to Power Series Please ote that there is material o power series at Visual Calculus. Some of this material was used as part of the presetatio of the topics that follow. What is a Power
More informationMath 113, Calculus II Winter 2007 Final Exam Solutions
Math, Calculus II Witer 7 Fial Exam Solutios (5 poits) Use the limit defiitio of the defiite itegral ad the sum formulas to compute x x + dx The check your aswer usig the Evaluatio Theorem Solutio: I this
More informationWe are mainly going to be concerned with power series in x, such as. (x)} converges - that is, lims N n
Review of Power Series, Power Series Solutios A power series i x - a is a ifiite series of the form c (x a) =c +c (x a)+(x a) +... We also call this a power series cetered at a. Ex. (x+) is cetered at
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More information( ) (( ) ) ANSWERS TO EXERCISES IN APPENDIX B. Section B.1 VECTORS AND SETS. Exercise B.1-1: Convex sets. are convex, , hence. and. (a) Let.
Joh Riley 8 Jue 03 ANSWERS TO EXERCISES IN APPENDIX B Sectio B VECTORS AND SETS Exercise B-: Covex sets (a) Let 0 x, x X, X, hece 0 x, x X ad 0 x, x X Sice X ad X are covex, x X ad x X The x X X, which
More informationMATH 413 FINAL EXAM. f(x) f(y) M x y. x + 1 n
MATH 43 FINAL EXAM Math 43 fial exam, 3 May 28. The exam starts at 9: am ad you have 5 miutes. No textbooks or calculators may be used durig the exam. This exam is prited o both sides of the paper. Good
More informationMATH 10550, EXAM 3 SOLUTIONS
MATH 155, EXAM 3 SOLUTIONS 1. I fidig a approximate solutio to the equatio x 3 +x 4 = usig Newto s method with iitial approximatio x 1 = 1, what is x? Solutio. Recall that x +1 = x f(x ) f (x ). Hece,
More informationTHE SOLUTION OF NONLINEAR EQUATIONS f( x ) = 0.
THE SOLUTION OF NONLINEAR EQUATIONS f( ) = 0. Noliear Equatio Solvers Bracketig. Graphical. Aalytical Ope Methods Bisectio False Positio (Regula-Falsi) Fied poit iteratio Newto Raphso Secat The root of
More informationIntroduction to Optimization Techniques
Itroductio to Optimizatio Techiques Basic Cocepts of Aalysis - Real Aalysis, Fuctioal Aalysis 1 Basic Cocepts of Aalysis Liear Vector Spaces Defiitio: A vector space X is a set of elemets called vectors
More informationSolutions to Tutorial 5 (Week 6)
The Uiversity of Sydey School of Mathematics ad Statistics Solutios to Tutorial 5 (Wee 6 MATH2962: Real ad Complex Aalysis (Advaced Semester, 207 Web Page: http://www.maths.usyd.edu.au/u/ug/im/math2962/
More informationMIDTERM 3 CALCULUS 2. Monday, December 3, :15 PM to 6:45 PM. Name PRACTICE EXAM SOLUTIONS
MIDTERM 3 CALCULUS MATH 300 FALL 08 Moday, December 3, 08 5:5 PM to 6:45 PM Name PRACTICE EXAM S Please aswer all of the questios, ad show your work. You must explai your aswers to get credit. You will
More informationMa 530 Infinite Series I
Ma 50 Ifiite Series I Please ote that i additio to the material below this lecture icorporated material from the Visual Calculus web site. The material o sequeces is at Visual Sequeces. (To use this li
More informationCHAPTER 10 INFINITE SEQUENCES AND SERIES
CHAPTER 10 INFINITE SEQUENCES AND SERIES 10.1 Sequeces 10.2 Ifiite Series 10.3 The Itegral Tests 10.4 Compariso Tests 10.5 The Ratio ad Root Tests 10.6 Alteratig Series: Absolute ad Coditioal Covergece
More information62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
More informationn=1 a n is the sequence (s n ) n 1 n=1 a n converges to s. We write a n = s, n=1 n=1 a n
Series. Defiitios ad first properties A series is a ifiite sum a + a + a +..., deoted i short by a. The sequece of partial sums of the series a is the sequece s ) defied by s = a k = a +... + a,. k= Defiitio
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationMA131 - Analysis 1. Workbook 3 Sequences II
MA3 - Aalysis Workbook 3 Sequeces II Autum 2004 Cotets 2.8 Coverget Sequeces........................ 2.9 Algebra of Limits......................... 2 2.0 Further Useful Results........................
More informationLesson 10: Limits and Continuity
www.scimsacademy.com Lesso 10: Limits ad Cotiuity SCIMS Academy 1 Limit of a fuctio The cocept of limit of a fuctio is cetral to all other cocepts i calculus (like cotiuity, derivative, defiite itegrals
More informationRecurrence Relations
Recurrece Relatios Aalysis of recursive algorithms, such as: it factorial (it ) { if (==0) retur ; else retur ( * factorial(-)); } Let t be the umber of multiplicatios eeded to calculate factorial(). The
More informationMarkov Decision Processes
Markov Decisio Processes Defiitios; Statioary policies; Value improvemet algorithm, Policy improvemet algorithm, ad liear programmig for discouted cost ad average cost criteria. Markov Decisio Processes
More informationINEQUALITIES BJORN POONEN
INEQUALITIES BJORN POONEN 1 The AM-GM iequality The most basic arithmetic mea-geometric mea (AM-GM) iequality states simply that if x ad y are oegative real umbers, the (x + y)/2 xy, with equality if ad
More informationChapter 2 The Solution of Numerical Algebraic and Transcendental Equations
Chapter The Solutio of Numerical Algebraic ad Trascedetal Equatios Itroductio I this chapter we shall discuss some umerical methods for solvig algebraic ad trascedetal equatios. The equatio f( is said
More informationIt is often useful to approximate complicated functions using simpler ones. We consider the task of approximating a function by a polynomial.
Taylor Polyomials ad Taylor Series It is ofte useful to approximate complicated fuctios usig simpler oes We cosider the task of approximatig a fuctio by a polyomial If f is at least -times differetiable
More informationv = -!g(x 0 ) Ûg Ûx 1 Ûx 2 Ú If we work out the details in the partial derivatives, we get a pleasing result. n Ûx k, i x i - 2 b k
The Method of Steepest Descet This is the quadratic fuctio from to that is costructed to have a miimum at the x that solves the system A x = b: g(x) = - 2 I the method of steepest descet, we
More informationMAS111 Convergence and Continuity
MAS Covergece ad Cotiuity Key Objectives At the ed of the course, studets should kow the followig topics ad be able to apply the basic priciples ad theorems therei to solvig various problems cocerig covergece
More informationAssignment 5: Solutions
McGill Uiversity Departmet of Mathematics ad Statistics MATH 54 Aalysis, Fall 05 Assigmet 5: Solutios. Let y be a ubouded sequece of positive umbers satisfyig y + > y for all N. Let x be aother sequece
More informationLinear Support Vector Machines
Liear Support Vector Machies David S. Roseberg The Support Vector Machie For a liear support vector machie (SVM), we use the hypothesis space of affie fuctios F = { f(x) = w T x + b w R d, b R } ad evaluate
More informationDefinitions and Theorems. where x are the decision variables. c, b, and a are constant coefficients.
Defiitios ad Theorems Remember the scalar form of the liear programmig problem, Miimize, Subject to, f(x) = c i x i a 1i x i = b 1 a mi x i = b m x i 0 i = 1,2,, where x are the decisio variables. c, b,
More informationSupplemental Material: Proofs
Proof to Theorem Supplemetal Material: Proofs Proof. Let be the miimal umber of traiig items to esure a uique solutio θ. First cosider the case. It happes if ad oly if θ ad Rak(A) d, which is a special
More informationTopics. Homework Problems. MATH 301 Introduction to Analysis Chapter Four Sequences. 1. Definition of convergence of sequences.
MATH 301 Itroductio to Aalysis Chapter Four Sequeces Topics 1. Defiitio of covergece of sequeces. 2. Fidig ad provig the limit of sequeces. 3. Bouded covergece theorem: Theorem 4.1.8. 4. Theorems 4.1.13
More informationAP Calculus BC Review Applications of Derivatives (Chapter 4) and f,
AP alculus B Review Applicatios of Derivatives (hapter ) Thigs to Kow ad Be Able to Do Defiitios of the followig i terms of derivatives, ad how to fid them: critical poit, global miima/maima, local (relative)
More informationReview Problems 1. ICME and MS&E Refresher Course September 19, 2011 B = C = AB = A = A 2 = A 3... C 2 = C 3 = =
Review Problems ICME ad MS&E Refresher Course September 9, 0 Warm-up problems. For the followig matrices A = 0 B = C = AB = 0 fid all powers A,A 3,(which is A times A),... ad B,B 3,... ad C,C 3,... Solutio:
More informationPAPER : IIT-JAM 2010
MATHEMATICS-MA (CODE A) Q.-Q.5: Oly oe optio is correct for each questio. Each questio carries (+6) marks for correct aswer ad ( ) marks for icorrect aswer.. Which of the followig coditios does NOT esure
More informationMath 210A Homework 1
Math 0A Homework Edward Burkard Exercise. a) State the defiitio of a aalytic fuctio. b) What are the relatioships betwee aalytic fuctios ad the Cauchy-Riema equatios? Solutio. a) A fuctio f : G C is called
More informationNUMERICAL METHODS FOR SOLVING EQUATIONS
Mathematics Revisio Guides Numerical Methods for Solvig Equatios Page 1 of 11 M.K. HOME TUITION Mathematics Revisio Guides Level: GCSE Higher Tier NUMERICAL METHODS FOR SOLVING EQUATIONS Versio:. Date:
More information15.081J/6.251J Introduction to Mathematical Programming. Lecture 21: Primal Barrier Interior Point Algorithm
508J/65J Itroductio to Mathematical Programmig Lecture : Primal Barrier Iterior Poit Algorithm Outlie Barrier Methods Slide The Cetral Path 3 Approximatig the Cetral Path 4 The Primal Barrier Algorithm
More informationSequences A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece 1, 1, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet
More informationSequences and Limits
Chapter Sequeces ad Limits Let { a } be a sequece of real or complex umbers A ecessary ad sufficiet coditio for the sequece to coverge is that for ay ɛ > 0 there exists a iteger N > 0 such that a p a q
More informationM17 MAT25-21 HOMEWORK 5 SOLUTIONS
M17 MAT5-1 HOMEWORK 5 SOLUTIONS 1. To Had I Cauchy Codesatio Test. Exercise 1: Applicatio of the Cauchy Codesatio Test Use the Cauchy Codesatio Test to prove that 1 diverges. Solutio 1. Give the series
More informationPROBLEM SET I (Suggested Solutions)
Eco3-Fall3 PROBLE SET I (Suggested Solutios). a) Cosider the followig: x x = x The quadratic form = T x x is the required oe i matrix form. Similarly, for the followig parts: x 5 b) x = = x c) x x x x
More informationMath Solutions to homework 6
Math 175 - Solutios to homework 6 Cédric De Groote November 16, 2017 Problem 1 (8.11 i the book): Let K be a compact Hermitia operator o a Hilbert space H ad let the kerel of K be {0}. Show that there
More informationA sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece,, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet as
More informationlim za n n = z lim a n n.
Lecture 6 Sequeces ad Series Defiitio 1 By a sequece i a set A, we mea a mappig f : N A. It is customary to deote a sequece f by {s } where, s := f(). A sequece {z } of (complex) umbers is said to be coverget
More informationMath 451: Euclidean and Non-Euclidean Geometry MWF 3pm, Gasson 204 Homework 3 Solutions
Math 451: Euclidea ad No-Euclidea Geometry MWF 3pm, Gasso 204 Homework 3 Solutios Exercises from 1.4 ad 1.5 of the otes: 4.3, 4.10, 4.12, 4.14, 4.15, 5.3, 5.4, 5.5 Exercise 4.3. Explai why Hp, q) = {x
More informationHomework Set #3 - Solutions
EE 15 - Applicatios of Covex Optimizatio i Sigal Processig ad Commuicatios Dr. Adre Tkaceko JPL Third Term 11-1 Homework Set #3 - Solutios 1. a) Note that x is closer to x tha to x l i the Euclidea orm
More informationLinear Programming and the Simplex Method
Liear Programmig ad the Simplex ethod Abstract This article is a itroductio to Liear Programmig ad usig Simplex method for solvig LP problems i primal form. What is Liear Programmig? Liear Programmig is
More informationsin(n) + 2 cos(2n) n 3/2 3 sin(n) 2cos(2n) n 3/2 a n =
60. Ratio ad root tests 60.1. Absolutely coverget series. Defiitio 13. (Absolute covergece) A series a is called absolutely coverget if the series of absolute values a is coverget. The absolute covergece
More informationSequence A sequence is a function whose domain of definition is the set of natural numbers.
Chapter Sequeces Course Title: Real Aalysis Course Code: MTH3 Course istructor: Dr Atiq ur Rehma Class: MSc-I Course URL: wwwmathcityorg/atiq/fa8-mth3 Sequeces form a importat compoet of Mathematical Aalysis
More informationLecture 6: Integration and the Mean Value Theorem. slope =
Math 8 Istructor: Padraic Bartlett Lecture 6: Itegratio ad the Mea Value Theorem Week 6 Caltech 202 The Mea Value Theorem The Mea Value Theorem abbreviated MVT is the followig result: Theorem. Suppose
More informationACO Comprehensive Exam 9 October 2007 Student code A. 1. Graph Theory
1. Graph Theory Prove that there exist o simple plaar triagulatio T ad two distict adjacet vertices x, y V (T ) such that x ad y are the oly vertices of T of odd degree. Do ot use the Four-Color Theorem.
More information2.1. The Algebraic and Order Properties of R Definition. A binary operation on a set F is a function B : F F! F.
CHAPTER 2 The Real Numbers 2.. The Algebraic ad Order Properties of R Defiitio. A biary operatio o a set F is a fuctio B : F F! F. For the biary operatios of + ad, we replace B(a, b) by a + b ad a b, respectively.
More information1. By using truth tables prove that, for all statements P and Q, the statement
Author: Satiago Salazar Problems I: Mathematical Statemets ad Proofs. By usig truth tables prove that, for all statemets P ad Q, the statemet P Q ad its cotrapositive ot Q (ot P) are equivalet. I example.2.3
More informationCHAPTER 5. Theory and Solution Using Matrix Techniques
A SERIES OF CLASS NOTES FOR 2005-2006 TO INTRODUCE LINEAR AND NONLINEAR PROBLEMS TO ENGINEERS, SCIENTISTS, AND APPLIED MATHEMATICIANS DE CLASS NOTES 3 A COLLECTION OF HANDOUTS ON SYSTEMS OF ORDINARY DIFFERENTIAL
More informationCALCULUS AB SECTION I, Part A Time 60 minutes Number of questions 30 A CALCULATOR MAY NOT BE USED ON THIS PART OF THE EXAM.
AP Calculus AB Portfolio Project Multiple Choice Practice Name: CALCULUS AB SECTION I, Part A Time 60 miutes Number of questios 30 A CALCULATOR MAY NOT BE USED ON THIS PART OF THE EXAM. Directios: Solve
More information1 Duality revisited. AM 221: Advanced Optimization Spring 2016
AM 22: Advaced Optimizatio Sprig 206 Prof. Yaro Siger Sectio 7 Wedesday, Mar. 9th Duality revisited I this sectio, we will give a slightly differet perspective o duality. optimizatio program: f(x) x R
More informationAdvanced Analysis. Min Yan Department of Mathematics Hong Kong University of Science and Technology
Advaced Aalysis Mi Ya Departmet of Mathematics Hog Kog Uiversity of Sciece ad Techology September 3, 009 Cotets Limit ad Cotiuity 7 Limit of Sequece 8 Defiitio 8 Property 3 3 Ifiity ad Ifiitesimal 8 4
More informationThe Simplex algorithm: Introductory example. The Simplex algorithm: Introductory example (2)
Discrete Mathematics for Bioiformatics WS 07/08, G. W. Klau, 23. Oktober 2007, 12:21 1 The Simplex algorithm: Itroductory example The followig itroductio to the Simplex algorithm is from the book Liear
More informationChapter 10: Power Series
Chapter : Power Series 57 Chapter Overview: Power Series The reaso series are part of a Calculus course is that there are fuctios which caot be itegrated. All power series, though, ca be itegrated because
More informationMath 341 Lecture #31 6.5: Power Series
Math 341 Lecture #31 6.5: Power Series We ow tur our attetio to a particular kid of series of fuctios, amely, power series, f(x = a x = a 0 + a 1 x + a 2 x 2 + where a R for all N. I terms of a series
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More information} is said to be a Cauchy sequence provided the following condition is true.
Math 4200, Fial Exam Review I. Itroductio to Proofs 1. Prove the Pythagorea theorem. 2. Show that 43 is a irratioal umber. II. Itroductio to Logic 1. Costruct a truth table for the statemet ( p ad ~ r
More information7 Sequences of real numbers
40 7 Sequeces of real umbers 7. Defiitios ad examples Defiitio 7... A sequece of real umbers is a real fuctio whose domai is the set N of atural umbers. Let s : N R be a sequece. The the values of s are
More informationLecture Notes for Analysis Class
Lecture Notes for Aalysis Class Topological Spaces A topology for a set X is a collectio T of subsets of X such that: (a) X ad the empty set are i T (b) Uios of elemets of T are i T (c) Fiite itersectios
More information18.657: Mathematics of Machine Learning
8.657: Mathematics of Machie Learig Lecturer: Philippe Rigollet Lecture 0 Scribe: Ade Forrow Oct. 3, 05 Recall the followig defiitios from last time: Defiitio: A fuctio K : X X R is called a positive symmetric
More informationSOLUTIONS TO EXAM 3. Solution: Note that this defines two convergent geometric series with respective radii r 1 = 2/5 < 1 and r 2 = 1/5 < 1.
SOLUTIONS TO EXAM 3 Problem Fid the sum of the followig series 2 + ( ) 5 5 2 5 3 25 2 2 This series diverges Solutio: Note that this defies two coverget geometric series with respective radii r 2/5 < ad
More informationChapter 3 Inner Product Spaces. Hilbert Spaces
Chapter 3 Ier Product Spaces. Hilbert Spaces 3. Ier Product Spaces. Hilbert Spaces 3.- Defiitio. A ier product space is a vector space X with a ier product defied o X. A Hilbert space is a complete ier
More information2 Banach spaces and Hilbert spaces
2 Baach spaces ad Hilbert spaces Tryig to do aalysis i the ratioal umbers is difficult for example cosider the set {x Q : x 2 2}. This set is o-empty ad bouded above but does ot have a least upper boud
More information[ 11 ] z of degree 2 as both degree 2 each. The degree of a polynomial in n variables is the maximum of the degrees of its terms.
[ 11 ] 1 1.1 Polyomial Fuctios 1 Algebra Ay fuctio f ( x) ax a1x... a1x a0 is a polyomial fuctio if ai ( i 0,1,,,..., ) is a costat which belogs to the set of real umbers ad the idices,, 1,...,1 are atural
More informationFind quadratic function which pass through the following points (0,1),(1,1),(2, 3)... 11
Adrew Powuk - http://www.powuk.com- Math 49 (Numerical Aalysis) Iterpolatio... 4. Polyomial iterpolatio (system of equatio)... 4.. Lier iterpolatio... 5... Fid a lie which pass through (,) (,)... 8...
More informationSection A assesses the Units Numerical Analysis 1 and 2 Section B assesses the Unit Mathematics for Applied Mathematics
X0/70 NATIONAL QUALIFICATIONS 005 MONDAY, MAY.00 PM 4.00 PM APPLIED MATHEMATICS ADVANCED HIGHER Numerical Aalysis Read carefully. Calculators may be used i this paper.. Cadidates should aswer all questios.
More informationThe picture in figure 1.1 helps us to see that the area represents the distance traveled. Figure 1: Area represents distance travelled
1 Lecture : Area Area ad distace traveled Approximatig area by rectagles Summatio The area uder a parabola 1.1 Area ad distace Suppose we have the followig iformatio about the velocity of a particle, how
More informationON WELLPOSEDNESS QUADRATIC FUNCTION MINIMIZATION PROBLEM ON INTERSECTION OF TWO ELLIPSOIDS * M. JA]IMOVI], I. KRNI] 1.
Yugoslav Joural of Operatios Research 1 (00), Number 1, 49-60 ON WELLPOSEDNESS QUADRATIC FUNCTION MINIMIZATION PROBLEM ON INTERSECTION OF TWO ELLIPSOIDS M. JA]IMOVI], I. KRNI] Departmet of Mathematics
More informationIntroduction to Extreme Value Theory Laurens de Haan, ISM Japan, Erasmus University Rotterdam, NL University of Lisbon, PT
Itroductio to Extreme Value Theory Laures de Haa, ISM Japa, 202 Itroductio to Extreme Value Theory Laures de Haa Erasmus Uiversity Rotterdam, NL Uiversity of Lisbo, PT Itroductio to Extreme Value Theory
More informationStatistical Inference Based on Extremum Estimators
T. Rotheberg Fall, 2007 Statistical Iferece Based o Extremum Estimators Itroductio Suppose 0, the true value of a p-dimesioal parameter, is kow to lie i some subset S R p : Ofte we choose to estimate 0
More informationINFINITE SEQUENCES AND SERIES
INFINITE SEQUENCES AND SERIES INFINITE SEQUENCES AND SERIES I geeral, it is difficult to fid the exact sum of a series. We were able to accomplish this for geometric series ad the series /[(+)]. This is
More informationSolutions to Homework 7
Solutios to Homework 7 Due Wedesday, August 4, 004. Chapter 4.1) 3, 4, 9, 0, 7, 30. Chapter 4.) 4, 9, 10, 11, 1. Chapter 4.1. Solutio to problem 3. The sum has the form a 1 a + a 3 with a k = 1/k. Sice
More informationw (1) ˆx w (1) x (1) /ρ and w (2) ˆx w (2) x (2) /ρ.
2 5. Weighted umber of late jobs 5.1. Release dates ad due dates: maximimizig the weight of o-time jobs Oce we add release dates, miimizig the umber of late jobs becomes a sigificatly harder problem. For
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationLecture 8: Solving the Heat, Laplace and Wave equations using finite difference methods
Itroductory lecture otes o Partial Differetial Equatios - c Athoy Peirce. Not to be copied, used, or revised without explicit writte permissio from the copyright ower. 1 Lecture 8: Solvig the Heat, Laplace
More informationChapter 7: The z-transform. Chih-Wei Liu
Chapter 7: The -Trasform Chih-Wei Liu Outlie Itroductio The -Trasform Properties of the Regio of Covergece Properties of the -Trasform Iversio of the -Trasform The Trasfer Fuctio Causality ad Stability
More informationSingular Continuous Measures by Michael Pejic 5/14/10
Sigular Cotiuous Measures by Michael Peic 5/4/0 Prelimiaries Give a set X, a σ-algebra o X is a collectio of subsets of X that cotais X ad ad is closed uder complemetatio ad coutable uios hece, coutable
More information10.1 Sequences. n term. We will deal a. a n or a n n. ( 1) n ( 1) n 1 2 ( 1) a =, 0 0,,,,, ln n. n an 2. n term.
0. Sequeces A sequece is a list of umbers writte i a defiite order: a, a,, a, a is called the first term, a is the secod term, ad i geeral eclusively with ifiite sequeces ad so each term Notatio: the sequece
More informationThe Method of Least Squares. To understand least squares fitting of data.
The Method of Least Squares KEY WORDS Curve fittig, least square GOAL To uderstad least squares fittig of data To uderstad the least squares solutio of icosistet systems of liear equatios 1 Motivatio Curve
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationReal Variables II Homework Set #5
Real Variables II Homework Set #5 Name: Due Friday /0 by 4pm (at GOS-4) Istructios: () Attach this page to the frot of your homework assigmet you tur i (or write each problem before your solutio). () Please
More informationNotes on iteration and Newton s method. Iteration
Notes o iteratio ad Newto s method Iteratio Iteratio meas doig somethig over ad over. I our cotet, a iteratio is a sequece of umbers, vectors, fuctios, etc. geerated by a iteratio rule of the type 1 f
More informationChapter Vectors
Chapter 4. Vectors fter readig this chapter you should be able to:. defie a vector. add ad subtract vectors. fid liear combiatios of vectors ad their relatioship to a set of equatios 4. explai what it
More informationPUTNAM TRAINING INEQUALITIES
PUTNAM TRAINING INEQUALITIES (Last updated: December, 207) Remark This is a list of exercises o iequalities Miguel A Lerma Exercises If a, b, c > 0, prove that (a 2 b + b 2 c + c 2 a)(ab 2 + bc 2 + ca
More informationSolutions to Tutorial 3 (Week 4)
The Uiversity of Sydey School of Mathematics ad Statistics Solutios to Tutorial Week 4 MATH2962: Real ad Complex Aalysis Advaced Semester 1, 2017 Web Page: http://www.maths.usyd.edu.au/u/ug/im/math2962/
More informationLecture 4. We also define the set of possible values for the random walk as the set of all x R d such that P(S n = x) > 0 for some n.
Radom Walks ad Browia Motio Tel Aviv Uiversity Sprig 20 Lecture date: Mar 2, 20 Lecture 4 Istructor: Ro Peled Scribe: Lira Rotem This lecture deals primarily with recurrece for geeral radom walks. We preset
More informationChapter 7 Isoperimetric problem
Chapter 7 Isoperimetric problem Recall that the isoperimetric problem (see the itroductio its coectio with ido s proble) is oe of the most classical problem of a shape optimizatio. It ca be formulated
More information