Itroductio to Optimizatio Techiques Basic Cocepts of Aalysis - Real Aalysis, Fuctioal Aalysis 1
Basic Cocepts of Aalysis Liear Vector Spaces Defiitio: A vector space X is a set of elemets called vectors together with two operatios. The first operatio is additio which associates with ay two vectors x, y X a vector x y X, the sum of x ad y. The secod operatio is scalar multiplicatio which associates with ay vector x X ad ay scalar a vector x ; the scalar multiple of x by. The set X ad the operatios of additio ad scalar multiplicatio are assumed to satisfy the followig axioms: 1. x y yx. (commutative law) 2. ( x y) z x( yz). (associative law) 3. There is a ull vector i X such that x x for all x i X. 2
Basic Cocepts of Aalysis 4. ( x y) xy. (distributive laws) 5. ( ) xxx. 6. ( ) x ( x). (associative law) 7. 0 x, 1 x x. Propositio: I ay vector space, the followig properties hold: 1. x y xz implies y z. 2. xy ad 0 imply x y. (cacellatio laws) 3. xx ad x imply. 4. ( ) xxx. 5. ( x y) xy. 6. 0 0. (distributive laws) 3
Basic Cocepts of Aalysis Examples of liear vector spaces Let R ( C ): -dimesioal real (complex) coordiate space xv, x(,,,, ) { } 1 2 k k k 1 The collectio of all ifiite sequeces of real umbers forms a vector space. V Cab [, ]: Vector space of real-valued cotiuous fuctios o [ ab, ] with a orm x max x( t) a t b l p, L p spaces (see pages 13 & 14) 4
Basic Cocepts of Aalysis Norm, Ier product Defiitio: A ormed liear vector space is a vector space X o which there is defied a real-valued fuctio which maps each elemet x i X ito a real umber x called the orm of x. The orm satisfies the followig axioms: 1. x 0 for all xx, x 0 iff x (Null vector) 2. x y x + y for each x, yx 3. x x for all scalar ad each xx 5
Basic Cocepts of Aalysis Defiitio: A pre-hilbert space is a liear vector space X together with a ier product defied o X X. Correspodig to each pair of vectors x, y i X the ier product [ xy, ] of x ad y is a scalar. The ier product satisfies the followig axioms: 1. [ x, y] [ y, x]: cojugate of [ y, x] 2. [ x y, z] [ x, z] [ y, z] 3. [ xy, ] [ xy, ] 4. [ xx, ] 0 ad [ xx, ] 0 iff x 6
7 C s 7 C s 1. Covexity 2. Closedess 3. Covergece 4. Cotiuity 5. Cauchy sequece 6. Completeess 7. Compactess 7
7 C s 1. Covexity Defiitio : A set K i a liear vector space is said to be covex if, give x1, x2 K, all poit of the form x1 (1 ) x2k for all 0,1 2. Closedess x X P 0 p P x p P P P Defiitio : A poit is said to be a closure poit of a set if, give, there is a poit satisfyig. The collectio of all closure poits of is called the closure of ad is deoted. Defiitio : A set P is said to be closed if P P. 8
7 C s P X p P P 0 x x p P P P Defiitio : Let be a subset of a ormed space. The poit is said to be a iterior poit of if there is a such that all vectors satisfyig are also members of. The collectio of all iterior poits of is called the iterior of ad is deoted. P P P Defiitio : A set is said to be ope if. P S( x, ) : ope sphere cetered at x with radius Sx (, ) { y: x y } 9
A poit is a iterior poit S( x, ) P 3. Covergece 7 C s x P 0 if there is such that Defiitio : I a ormed liear space a ifiite sequece of vectors is said to coverge to a vector if the sequece { x } x x x x. write of real umbers coverges to zero. I this case, we x Defiitio : A trasformatio from a vector space ito the space of real (or complex) scalars is said to be a fuctioal o X X 10
7 C s 4. Cotiuity T 0 Y x0 X 0 xx0 T( x) T( x ). Defiitio : A trasformatio mappig a ormed space ito a ormed space is cotiuous at if for every there is a such that implies that If is cotiuous at each poit, we say is cotiuous. 0 T x0 X T X 11
7 C s 5. Cauchy sequece Defiitio: A sequece { x } i a ormed space is said to be a Cauchy sequece if x x 0 as m, ; i.e., give 0, there is a iteger such that x x for all m, N. Every coverget sequece m N m Cauchy sequece 12
7 C s 6. Completeess Defiitio: A ormed liear vector space X is complete if every Cauchy sequece from X has a limit i X. A complete ormed liear vector space is called a Baach space. Examples of Baach spaces: C[0,1], l, L p p cosists of all sequeces of scalars {,, } for which lp 1 2 p p, x i i p i1 i1 1/ p 13
7 C s L [ cosists of those real-valued measurable fuctios o p a, b ] x the iterval [ ab, ] for which xt () p is Lebesque itegrable. The orm o this space is defied as b p 1/ () a x x t dt p 7. Compactess Defiitio: A set K i a ormed space X is said to be compact if, give a arbitrary sequece { x i } i K, there is a subsequece { } covergig to a elemet x K. p x i I fiite dimesios, compactess closed ad boudedess 14
7 C s f x0 X 0, 0 f( x) f( x0) x x0 Defiitio : A real valued fuctioal defied o a ormed space is said to be upper semicotiuous at if, give there is a such that for. Theorem: (Weierstrass) A upper semicotiuous fuctioal o a compact subset K of a ormed liear space X achieves a maximum o K. Defiitio : A complete pre-hilbert space is called a Hilbert space Defiitio : I a pre-hilbert space two vectors x, y are said to be orthogoal if [ xy, ] 0. We symbolize this by x y. A vector x is said to be orthogoal to a set S (writte x S ) if x s for each s S. X 15
Projectio Theorem Theorem : (The classical Projectio Theorem) Let H be a Hilbert space ad M a closed subspace of H. Correspodig to ay vector x H, there is a uique vector m0 M such that xm0 xm for all m M. Furthermore, a ecessary ad sufficiet coditio that m0 M be the uique miimizig vector is that x m0 be orthogoal to M x M x m 0 m 0 16
Projectio Theorem Defiitio : Give a subset S of a pre-hilbert space, the set of the all vectors orthogoal to S is called the orthogoal complemet of S ad is deoted S. Propositio : Let S ad T be subsets of a Hilbert space, the 1. S is a closed subspace. 2. S S. 3. If S T, the T S. 4. S S 5. S S, i.e., S is the smallest closed subspace cotaiig S 17
Gram-Schmidt Procedure Defiitio : A set S of vectors i a pre-hilbert space is said to be a orthogoal set if x y for each x, y S, x y. The set is said to be orthoormal if, i additio, each vector i the set has orm equal to uity. Propositio : A orthogoal set of ozero vectors is a liearly idepedet set. GSOP... Arbitrary liearly idepedet set orthoormal set. Theorem : (Gram-Schmidt) Let { x i } be a coutable or fiite sequece of liearly idepedet vectors i a pre-hilbert space X. The, there is a orthoormal sequece { e i } such that for each the space geerated by the first ei s is the space geerated by the first x i s ; i.e., for each we have S e, e,, e Sx, x,, x 1 2 1 2 18
Gram-Schmidt Procedure Proof. For the first vector, take e 1 x x S{ e} S{ x} 1 1 1 1 e 2 Form i two steps. First, put z x x, e e z2 e2 z 2 2 2 1 1 2 z 2 e 1, e2 e, 1 z2 S e e = Sx, x 1, 2 1 2 19
Gram-Schmidt Procedure z 2 x 2 x x, e e 2 2 1 1 Gram-Schmidt procedure x, e e 2 1 1 e 1 The remaiig s are defied by iductio. The vector is formed accordig to the equatio ad z e i 1 z x x, e e i i i1 e z z 20
Gram-Schmidt Procedure It is easily verified by direct computatio that i for all, ad sice it is a liear combiatio of idepedet vectors. S z e = Sx x x e1, e2,,,,, 1 2 z e i If the origial collectio of s is fiite, the process termiates; otherwise the process produces a ifiite sequece of orthoormal sequeces. x i 21
Approximatios We ivestigate the followig approximatio problem. y1, y2,..., y H Suppose, M = Sy y y x H,,..., 1 2 Give a arbitrary vector, we seek the ˆx M vector i which is closest to. x Let ˆ 1 12 2... x y y y The problem is equivalet to that of fidig, mi x 1y1 2y2... y i 1,2,..., i 22
Normal Equatios ad Gram Matrices Accordig to the projectio theorem, there exists the uique miimizig vector ˆx : orthogoal projectio of x o M. such that x xˆ M x1y12y2... y, yi 0 for Or, Equivaletly i 1, 2,..., [ y1, y1] 1 [ y2, y1] 2 [ y, y1] [ x, y1] [ y1, y2] 1 [ y2, y2] 2 [ y, y2] [ x, y2] [ y, y ] [ y, y ] [ y, y ] [ x, y ] 1 1 2 2 23
Normal Equatios ad Gram Matrices These equatios i the coefficiets are kow as Normal equatios for the miimizatio problem. Gram matrix: traspose of the coefficiet matrix of the ormal equatios G G( y, y,..., y ) 1 2 i [ y1, y1] [ y1, y2] [ y1, y] [ y2, y1] [ y2, y2] [ y2, y] [ y, y1] [ y, y2] [ y, y] 24