Inner Product Defnton 1 () A Eucldean space s a fnte-dmensonal vector space over the reals R, wth an nner product,. Defnton 2 (Inner Product) An nner product, on a real vector space X s a symmetrc, blnear, postve-defnte functon, : X X R (x,x) x,x. (Postve-defnte means x, x > 0 unless x = 0.) 1 2 Orthogonal Defnton 3 (Orthogonal) Two vectors x and x are orthogonal f ther nner product s zero, x,x =0. Geometrcally, orthogonal means perpendcular. Orthonormal Bass Defnton 4 (Orthonormal Bass) In a Eucldean space, an orthonormal bass s a bass x such that x,x j = Any two bass vectors are orthogonal. 1 f = j 0 f j. A Eucldean space has more than one orthonormal bass. 3 4 If then x= x x x = x x, x,x = x x. For the real numbers R, the nner product s just ordnary multplcaton. R Defnton 5 The Eucldean space R of real numbers s defned by the nner product x,x := x x. 5 6
R n The Eucldean space R n := R R (n tmes), n whch the elements are vectors wth n real components. By assumpton, the n vectors 1 0 0 1,,. 0. 0. form an orthonormal bass. The nner product of two vectors s then the sum of the component by component products. 7 Isomorphc In abstract algebra, somorphc means the same. If two objects of a gven type (group, rng, vector space, Eucldean space, algebra, etc.) are somorphc, then they are the same, when consdered as objects of that type. An somorphsm s a one-to-one and onto mappng from one space to the other that preserves all propertes defnng the space. Any n-dmensonal Eucldean space s somorphc to R n. Although two spaces may be somorphc as Eucldean spaces, perhaps the same two spaces are not somorphc when vewed as another space. 8 Coordnate-Free Versus Bass It s useful to thnk of a vector n a Eucldean space as coordnate-free. Gven a bass, any vector can be expressed unquely as a lnear combnaton of the bass elements. For example, f x= x x for some bass x, one can refer to the x as the coordnates of x n terms of ths bass. Many lnear algebra textbooks develop all the results n terms of a bass. In economc theory and econometrcs, typcally vectors are not seen as coordnate-free. A partcular bass s sngled out, and one works wth coordnates. Commonly there s a natural bass, but unfortunately the natural bass s perhaps not orthonormal. Despte ths tradton, the coordnate-free pont-of-vew s superor. Not usng coordnates reduces the use of subscrpts and makes expressons smpler, and theorems are easer to state and to prove. 9 10 Lnear Transformaton Defnton 6 (Lnear Transformaton) A lnear transformaton from a Eucldean space X to a Eucldean space Y s a functon such that A : X Y x y=ax A(x 1 + x 2 )=Ax 1 + Ax 2. 11 Adjont The followng proposton s a standard theorem of lnear algebra. Proposton 7 (Adjont) Gven a lnear transformaton A : X Y, then there exsts a unque lnear transformaton (the adjont) A : Y X that preserves the nner product: y,ax = A y,x for all x and y. 12 (1)
The adjont s very mportant n applcatons and has not been apprecated by economsts. The adjont s ndependent of any choce of bases, and n many applcatons one can determne t drectly, expressed n a coordnate-free way. The adjont then becomes a powerful tool, and one can easly obtan valuable results va the adjont, almost as f by magc. Typcally one does not calculate the adjont drectly. Instead one conjectures an expresson for the adjont, and then verfes that the adjont condton (1) holds. Matrx Representaton A matrx representaton for a lnear transformaton A : X Y s a matrx A j that shows how bass elements x j X map to a lnear combnaton of bass elements y Y: x j Ax j = A j y. 13 14 Adjont as Transpose If the bases for X and Y are each orthonormal, then the matrx representaton of the adjont s the transpose of the matrx representaton: A y = A j x j. j To prove ths relatonshp, verfy the adjont condton (1), for arbtrary bass elements: A y,x j = A k x k,x j k = A j (snce the bass x j s orthonormal) = y, A k j y k (snce the bass y s orthonormal) k = y,ax j, as desred. 15 16 On the other hand, f the bases are not orthonormal, then the transpose of the matrx representaton s not the matrx representaton of the adjont. Snce we want to see vectors as coordnate-free, however, the matrx representaton s of secondary mportance. Apart from smple cases, t may be dffcult to wrte down the matrx representaton explctly. At the same tme, one can descrbe the adjont easly, wthout reference to any bass. 17 18
For some y X, the adjont of the lnear functon Resz Representaton A fundamental theorem states that any lnear functon X R can be expressed as x y,x for a unque y. s y : R X z x=zy y : X R x z= y,x. 19 20 Verfy that the adjont condton (1) holds: Thus x,yz = x,y z= y,x z= y,x,z = y x= y,x. y x,z. Ether notaton s equvalent, but normally we employ the nner product notaton on the rght-hand sde. 21 Matrx Representaton Suppose y= y x, for a bass x. Let us use the natural orthonormal bass 1 for R. The matrx representaton of the lnear transformaton y s 1 y x, so the vector wth components y defnes the matrx representaton. For the adjont y, however, the matrx representaton s not the transpose of ths vector, unless the bass x s orthonormal. 22 The matrx representaton of the adjont s x j y,x j 1= y x,x j 1= x,x j y 1. For a nonorthonormal bass, the matrx representaton of the adjont s not x j y j 1. Fundamental Theorem of Lnear Algebra The fundamental theorem of lnear algebra states that the null space N(A) and the range R ( A ) are orthogonal, and any x X can be wrtten unquely as an element of N(A) plus an element of R ( A ). The same relatonshp holds for the range R(A) and the null space N ( A ). 23 24
Moore-Penrose Generalzed Inverse Usng the fundamental theorem of lnear algebra, we defne the Moore-Penrose generalzed nverse. Consder a lnear transformaton A : X Y x y=ax. The generalzed nverse A + s a lnear transformaton mappng Y X. Usng the fundamental theorem of lnear algebra, one can prove the followng. Proposton 8 (Inverse) The restrcton of A to R ( A ) ( A : R A ) R(A) x y=ax. s one-to-one and onto, so t has an nverse. 25 26 Defne the generalzed nverse A + : Y X va ths nverse mappng. For y R(A), defne A + y as the nverse of A. For y N ( A ), defne A + y=0. Ths defnton rests on the coordnate-free approach to lnear algebra. The relatonshp between A and A + s symmetrc. A lnear transformaton s the generalzed nverse of ts generalzed nverse: (A + ) + = A. And AA + A=A. The sngular-value decomposton obtans further results. If A s onto, then AA s nvertble, and A + = A (AA ) 1. 27 For the lnear equaton Lnear Equaton Ax=y, there s a soluton x f and only f y R(A). If there s a soluton, then the unque soluton n R ( A ) s A + y. Ths vector plus any element of N(A) s also a soluton. Hence the complete soluton set s { A + y } + N(A). 28 Defnton 9 (Partal Orderng) Gven a bass representaton Order The concept of a Eucldean space does not nvolve any concept of order, of one vector beng greater than another. However commonly one defnes the addtonal structure of a partal orderng va the representaton of a vector n a natural bass. then x= x j x j, j x 0 f every x j 0 x 0 f every x j 0 and some x > 0 x 0 f every x j > 0. 29 30
Defnton of the Inner Product To embed model structure nto the nner product smplfes the analyss. In a Eucldean space of random varables, one mght defne the nner product of two random varables as the covarance. Orthogonalty then means no correlaton. A dfferent defnton of the nner product derves from a partal orderng: one defnes a trace nner product consstent wth the orderng. 31