Optimization Theory Linear Operators and Adjoints
A transformation T. : X Y y Linear Operators y T( x), x X, yy is the image of x under T The domain of T on which T can be defined : D X The range of T { y Y xd, y T( x)} 2
Linear Operators T( S): the image of S in Y for S X T ( P): the inverse image of P { xx T( x) P, for PY} Linear transformation (linear operator) A, B, A x for A( x) RA ( ): The range of A : X Y R( A) Y N( A) { xx Ax }: The null space of A N( A) X 3
Proposition Linear Operators A linear operator on a normed space X is continuous at every point in X if it is continuous at a single point. Definition A linear operator A from a normed space X to a normed space Y is said to be bounded if a constant M such that Ax M x, x X. The smallest M which satisfies the above condition is denoted A and called the norm of A. Equivalent definition of the norm A Ax sup x x Ax sup x x 4
Linear Operators Proposition A linear operator is bounded iff it is continuous. If addition and scalar multiplication are defined by ( A A ) x Ax A x 2 2 ( A) x ( Ax), the linear operators from X to Y form a linear vector space. Definition The normed space of all bounded linear operators from the normed space X into the normed space Y is denoted BX (, Y). 5
Theorem Linear Operators Let X and Y be normed spaces with Y complete. Then the space BX (, Y) is complete. Products of operators S: X Y, T : Y Z, Define TS: X Z by TS( x) T( Sx), x X Proposition Let X, Y, Z be normed spaces and suppose S B( X, Y), T B( Y, Z). Then TS T S 6
Linear Operators Ex X C[0, ] A: X X A x K(, s t)() x t dt where the 0 function K is continuous on the unit square 0 s, 0 t. A is linear. Compute A Actually Ax max K( s, t) x( t) dt 0s 0 max Kst (, ) dtmax xt ( ) 0s 0 0t max Kst (, ) dt x 0s 0 A max Kst (, ) dt 0s 0 A max Kst (, ) dt 0s 0 7
Linear Operators Ex X n E, A : X 2 Ax [ x, AAx] X A: n n matrix A : transpose of A Denote AA Q A max [ x, Qx] x Q : symmetric, 0 Q : has nonnegative eigenvalues A ( Q) max 8
Inverse Operators Inverse Operators A : X Y, Consider A x y. For a given yy This equation may :. have a unique solution 2. have no solution x X 3. have more than one solution Cases 2 & 3 Optimization problem Condition holds for every y Y, iff A: X Y is one-to-one and RA ( ) Y The operator A has an inverse if Ax y, then x A y A 9
Inverse Operators Proposition If a linear operator A : X Y has an inverse, the A inverse is linear. The Banach Inverse Theorem A supporting role in the optimization theory Lemma (Baire) A Banach space X is not the union of countably many nowhere dense sets in X. Theorem (Banach Inverse Theorem) Let A be a continuous linear operator from a Banach space X onto a Banach space Y and suppose A A that the inverse operator exists. Then is continuous. 0
Adjoint Operators Adjoint Operators X Definition A transformation from a vector space into the space of real (or complex) scalars is said to be a functional on X. Ex. E n, a linear functional can be expressed f( x) kk for k where the are fixed scalars. x (, 2,..., n ), k n Ex. Ex. C[0, ], f( x) x linear functional 2 [0, ], f( x) y() t x() t dt, for y L 2 [0, ] L 2 0
Linear Functionals Linear functional Definition A linear functional f on normed space is bounded if there is a constant M f( x) M x, x X. The smallest such constant M is called the norm of f and is denoted f. Thus, Alternative definitions f inf { M : f( x) M x, xx} f sup x sup f( x) x sup f( x) x f( x) x 2
Linear Functionals and Dual Space Definition Let X be a normed linear vector space. The space of all bounded linear functionals on X is called the normed dual of X and is denoted X. The norm of an element f X is x, x 2 f sup f( x) x bounded linear functionals on X elements of X The value of the linear functional at the point is denoted by x ( x) or x, x x X x X 3
Linear Functionals and Dual Spaces Dual of E n : E n Dual of l, p pq p Dual of Dual space of p Dual space of l L [0, ], p, p q L p p : : l q L q Theorem (Riesz-Frechet, Riesz representation theorem) If f is a bounded linear functional on a Hilbert space H, there exists a unique vector y H such that for all x H, f( x) [ x, y]. Furthermore, we have f y and every y determines a unique bounded linear functional in this way. 4
Adjoint Operators Definition Let X and Y be normed spaces and let A BX (, Y). The adjoint operator A : Y X is defined by the equation x, A y Ax, y Given a fixed y Y, the quantity Ax, y is a linear functional on X Ax, y y Ax y A x bounded an element x of Define A y x, unique, linear X. 5
Adjoint Operators A : Y X X A Y Dual space X A Y An operator and its adjoints y Ax A y x x X ( ) ( )( ). We may write yaay functional on X functional on Y 6
Adjoint Operators Theorem The adjoint operator A of linear operator A BX (, Y) is linear and bounded with A A. Proposition Adjoints satisfy the following properties. If I is the identity operator on a normed space X, then 2. If A, A B( X, Y), then 2 3. If A BX (, Y) and is a real scalar, then 4. If A BX (, Y), ABY (, Z), then 2 ( A A ) A A 2 2 ( A A) ( A) A A 2 2 5. If A BX (, Y) and A has a bounded inverse, then A I I ( A ) ( A ) 7
An important special case Adjoint Operators A: H G, H, G Hilbert spaces If H, G are real, [ Ax, y] [ x, A y] H H, G G Definition of the adjoint operator on Hilbert spaces In Hilbert space, we have Definition A A A bounded linear operator A mapping a real Hilbert space into itself is said to be self-adjoint if A A. 8
Definition Self-Adjoint Operators A self-adjoint linear operator A on a Hilbert space H is said to be positive semidefinite if [ x, Ax] 0 xh. Ex. Let X Y E n. Then A : X X is represented by an n n matrix. n Thus the i-th component of Ax is ( Ax) i aijxj j 9
Compute A. For A A Ex. Where Then, is the matrix with elements: is the transpose of Adjoint Operators n n i j ij aji yy,[ Ax, y] ya x A a i ij j j ij i j i [, ] Let X Y L 2 [0, ] and define Ax K(, t s) x( s) ds, t [0,] 0 0 Kt (, s) 2 0 0 dsdt [ Ax, y] y( t) K( t, s) x( s) dsdt xs () Kt (, sytdtds ) () 0 0 0 n n x a y x A y 20
Adjoint Operators Or, by interchanging the role of and s t [ A x, y] x( t) K( s, t) y( s) dsdt 0 0 [ x, A y] where A y K(, s t)() y s ds 0 2
Ex Let X Y L 2 [0, ] and define t 0 Adjoint Operators Ax K(, t s) x( s) ds, t [0,] with Then 2 Kt (, s) dtds. 0 0 [ Ax, y] y( t) K( t, s) x( s) dsdt 0 0 t 0 0 t ytkt () (, sxsdsdt ) () 22
Adjoint Operators s s s t s t t t Integration vertically Integration horizontally and then horizontally and then vertically 23
Adjoint Operators [ Ax, y] y( t) K( t, s) x( s) dtds 0 s xs ()( Ktsytdt (,) () ) ds 0 s Or, interchanging the role of and t s [ Ax, y] x( t) ( K( s, t) y( s) ds) dt 0 t [ x, A y] where A y K(, s t) y() s ds t 24
Range and Null spaces Relations between Range and Null space Theorem Let X and Y be normed spaces and let A BX (, Y). Then [ RA ( )] NA ( ) Proof Let y N( A ) and y R( A). Then y Ax for some x X. y, y Ax, y x, A y 0 N( A ) [ R( A)] Assume y [ R( A)]. Then for every x X, x, A y 0 RA NA [ ( )] ( ) Ax y, 0 25
Range and Null spaces Theorem Let A be a bounded linear operator acting between two real Hilbert spaces. Then. 2. 3. 4. [ RA ( )] NA ( ) RA ( ) [ NA ( )] [ RA ( )] NA ( ) RA ( ) [ ( )] NA 26
Solving Equation Optimization in Hilbert Space A: G H, G, H : Hilbert spaces Linear equation A x y, may for a given y H. possess a unique solution x G 2. possess no solution 3. possess more than one solution Cases 2 & 3 : Optimization problem Normal equation Case 2 an approximate solution 27
Theorem Let G and H be Hilbert spaces and let A BG (, H). Then for a fixed y H the vector xg minimizes y Ax if and only if A Ax A y Proof Problem min y yˆ where yˆ R( A). By P.T. ŷ is a minimizing vector iff y yˆ R( A) ˆ [ ( )] ( ). y y R A N A Solving Equation Or ( ˆ) A y y A ya Ax y y R( A) RA ( ) ŷ ŷ Ax 28
If A A is invertible x ( A A) A y Ex. Basic approximation problem in Hilbert space { x, x2,, x n }: independent set of vectors M f { x, x2,, x n }. Find the best approximation to y H onto M. n yˆ aixi i n Define the operator A : E H by the equation n Aa ax i i where a i a a n Solving Equation T By Theorem minimize y Aa. AAa=A y 29
Solving Equation Find the operator A. n [ x, Aa] [ x, a x ] a[ x, x ] [ ] E n n A : H i i i i i i xx xx n z,a z,, 2 T n n A x [ x, x ][ x, x ] [ x, x n ] AA: E E nn E n T matrix AAa=A y [ x, x] [ x2, x] [ xn, x] a [ y, x] [, 2] [ 2, 2] [, 2] 2 [, 2] x x x x xn x a y x [ x, x ] [ x, x ] [ x, x ] a [ y, x ] n 2 n n n n n Normal equations a=(a A) A - y 30
Solving Equation Dual Problem Case 3 : the equation Ax y has more than one solution minimum norm problem N( A) {0} Theorem Let G and H be Hilbert spaces and let AB( GH, ) with range closed in H. Then the vector x of minimum norm satisfying Ax y is given by x A z where z is any solution of AAzy. Proof If x is a solution of Ax y, the general solution is x x u where u N( A) a unique vector ˆx of minimum norm satisfying Axˆ y, xˆ N( A) 3
Solving Equation xx N( A) V : linear variety ˆx xˆ N( A) N( A) ˆ [ ( )] ( ): x N A R A closed Hence xˆ A z for some z H. Axˆ y AA z y. If ( ), AA ˆ ( ) x A AA y 32
Ex. A linear dynamic system governed by a set of differential eqs: x () t Fx() t bu() t x : n vector F : n n matrix b : n vector u : scalar control function Find the minimum energy solution s.t. x(0) θ, x( T ) x x ( ) ( ) T F T t b ( ) 0 T e u t dt Define A : L [0, T] E A 2 T F( Tt) u e bu() t dt 0 Solving Equation n T 2 min ( ) 0 u u t dt 33
The problem: Find the u of minimum norm satisfying Au x. R( A) is finite dimensional closed. By Theorem u Az where AA z x Calculation of the operators A and AA For any ul y E where AA 2, n is the n nmatrix AA T F( Tt) [ ya, u] n y e bu( t) dt E 0 T F( Tt) F ( T t) e Ayb y T T t T t e bbe dt F( ) F( ) 0 If the matrix AA is invertible, Solving Equation 0 ye bu() t dt [ A y, u] u A ( AA ) x L 2 34
Definition Among all vectors Pseudoinverse Operators More general and more complete approach to the problem of finding approximate or minimum norm solutions to Ax y. pseudoinverse operator Suppose A BGH (, ) with R( A) closed x 0 x G Ax y min Ax y, x satisfying Let be the unique vector of minimum norm. The pseudoinverse A of A is the operator mapping y into its corresponding x 0 as y vaires over H. Discussion. min Ax y approximation of y on R( A) yˆ Ax is x unique, although x may not be. Ax yˆ is a linear variety, a translation of N( A) closed a unique x0 of minimum norm. A is well defined. 35
Geometric interpretation of A G N( A) N( A) H R( A) R( A) Pseudoinverse Operators ( R( A) is closed) A : The operator A restricted to N( A) N ( A) A : N( A) R( A) N ( A) onto one-to-one : has a linear inverse, bounded ( By the Banach inverse theorem) Definition We say that a vector space X is the direct sum of two subspaces M and N if every vector x X has a unique representation of the form x m n where m M and n N. We describe this situation by notation X M N. 36
Pseudoinverse Operators Inverse operator defines A on R( A). Its domain is extended to all of by defining Ayθ for y R( A) A H N( A ) R( A) A N( A ) R( A) A A θ θ The pseudo inverse operator 37
Verification of This Definition For y H, y yˆ y uniquely where yˆ R( A), y ˆ is the best approximation to y in R( A). y R( A) Ay A( yˆ y ˆ ) Ay. Define x0 A yˆ, then by definition Ax0 yˆ. Furthermore x0 N( A) minimum norm solution of Ax ˆ y x x 0 θ Linear variety of x s with Ax yˆ translation of N( A) A minimum norm sol y ŷ θ R( A) yˆ R( A) 38
Verification of This Definition Inverse of A : Ax y y R( A) no sol approximate ŷ For Ax yˆ, N( A) more than one solution minimum norm solution x 0 x A y, A y A yˆ A y 0 0 R( A) 39
Algebraic Properties of Pseudoinverse Proposition Let A be a bounded linear operator with closed range and A denotes its pseudoinverse. Then. A is linear 2. A is bounded 3. ( A ) A 4. ( A ) ( A ) 5. AAA A 6. AA A A 7. ( AA) AA 8. A ( A A) A 9. A A ( AA ) If AA is invertible, then If AA is invertible, then A ( A A) A A A ( AA ) 40