Itroducto to Matrces ad Matrx Approach to Smple Lear Regresso
Matrces Defto: A matrx s a rectagular array of umbers or symbolc elemets I may applcatos, the rows of a matrx wll represet dvduals cases (people, tems, plats, amals,...) ad colums wll represet attrbutes or characterstcs The dmeso of a matrx s t umber of rows ad colums, ofte deoted as r x c (r rows by c colums) Ca be represeted full form or abbrevated form: a a a j a c a a a j a c A aj,..., r; j,..., c a a aj a c a r ar arj arc
Square Matrx: Number of rows = # of Colums Specal Types of Matrces 0 3 50 b b A 8 4 B b b 8 46 60 Vector: Matrx wth oe colum (colum vector) or oe row (row vector) d 57 d C 4 D E' 7 3 ' f f f3 d F 3 r c 8 d4 Traspose: Matrx formed by terchagg rows ad colums of a matrx (use "prme" to deote traspose) 6 8 6 5 G ' 5 3 3 8 3 5 G 3 5 h h c h hr H h j,..., r; j,..., c ' h,..., ;,..., rc H j j c r cr hr h rc h c h rc Matrx Equalty: Matrces of the same dmeso, ad correspodg elemets same cells are all equal: 4 6 b b A = b 4, b 6, b, b 0 0 B b b
Regresso Examples - Toluca Data Respose Vector: ' Desg Matrx: ' 80 399 30 50 90 376 70 36 60 4 0 546 80 35 00 353 50 57 40 60 70 5 90 389 0 3 0 435 00 40 30 50 68 90 377 0 4 30 73 90 468 40 44 80 34 70 33
Matrx Addto ad Subtracto Addto ad Subtracto of Matrces of Commo Dmeso: 4 7 0 4 7 0 6 7 4 7 0 7 C 0 D 4 6 C D 0 4 6 4 8 C D 0 4 6 4 6 a a c b b c A rc a j,..., r; j,..., c b B j,..., r; j,..., c rc ar a rc br b rc a b a c b c A B aj b j,..., r; j,..., c rc a rbr arc b rc a b a c b c AB aj b,..., ;,..., rc j r j c ar br arc b rc Regresso Example: E E E E E sce E ε E ε E E E E
Matrx Multplcato Multplcato of a Matrx by a Scalar (sgle umber): 3() 3() 6 3 k 3 A 7 ka 3( ) 3(7) 6 Multplcato of a Matrx by a Matrx (#cols( A) = #rows( B)): If c r : A B A B AB = ab,..., r ; j,..., c r c r c r c A A B B A ab c r th th j sum of the products of the A B elemets of row of A ad j colum of B: 5 3 A 3 B 4 0 7 (3) 5() ( ) 5(4) 6 8 A B AB 3(3) ( )() 3( ) ( )(4) 7 7 0(3) 7() 0( ) 7(4) 4 8 3 3 3 B j A B c If c r c : A B AB = ab = a b,..., r ; j,..., c A B j k kj A B ra ca rb cb ra cb k
Matrx Multplcato Examples - I Smultaeous Equatos: ( equatos: x, x ukow): a x a x y ax ax y a x a x y a x a x y a a x y a a x y A = Sum of Squares: 4 3 4 3 0 0 0 Regresso Equato (Expected Values): 0 4 3 9
Matrx Multplcato Examples - II Matrces used smple lear regresso (that geeralze to multple regresso): ' ' ' 0 0 0 β 0
Specal Matrx Types Symmetrc Matrx: Square matrx wth a traspose equal to tself: A = A': 6 9 8 6 9 8 A 9 4 3 9 4 3 A' A 8 3 8 3 Dagoal Matrx: Square matrx wth all off-dagoal elemets equal to 0: 4 0 0 b 0 0 A 0 0 0 b 0 B Note:Dagoal matrces are symmetrc (ot vce versa) 0 0 0 0 b 3 Idetty Matrx: Dagoal matrx wth all dagoal elemets equal to (acts lke multplyg a scalar by ): 0 0 a a a3 a a a3 I 0 0 A a a a IA AI A a a a 3 3 33 33 0 0 a3 a3 a 33 a3 a3 a 33 Scalar Matrx: Dagoal matrx wth all dagoal elemets equal to a sgle umber" k 0 0 0 0 0 0 0 k 0 0 0 0 0 k k I 0 0 k 0 0 0 0 0 0 0 k 0 0 0 -Vector ad matrx ad zero-vector: 44 0 0 0 J 0 Note: ' r ' r rr r r r J rr rr
Lear Depedece ad Rak of a Matrx Lear Depedece: Whe a lear fucto of the colums (rows) of a matrx produces a zero vector (oe or more colums (rows) ca be wrtte as lear fucto of the other colums (rows)) Rak of a matrx: Number of learly depedet colums (rows) of the matrx. Rak caot exceed the mmum of the umber of rows or colums of the matrx. rak(a) m(r A,c a ) A matrx f full rak f rak(a) = m(r A,c a ) 3 A 3 Colums of are learly depedet rak( ) = 4 A A A A 0 A A 4 3 B 0 0 Colums of are learly depedet rak( ) = 4 B B B B 0 B B
Matrx Iverse Note: For scalars (except 0), whe we multply a umber, by ts recprocal, we get : (/)= x(/x)=x(x - )= I matrx form f A s a square matrx ad full rak (all rows ad colums are learly depedet), the A has a verse: A - such that: A - A = A A - = I 8 8 4 3 6 6 8 36 36 36 36 8 36 36 36 36 - - 0 A 4 A A A 4 4 4 8 8 3 4 0 I 36 36 36 36 36 36 36 36 4 0 0 / 4 0 0 4 / 4 0 0 0 0 0 0 0 0 0 0 0 0-0 / 0 - B B BB 0 0 0 0 / 0 0 0 0 0 0 I 0 0 6 0 0 / 6 0 0 0 0 0 0 0 0 6/ 6 0 0
Computg a Iverse of x Matrx a a A full rak (colums/rows are learly depedet) a a Determat of A A a a a a Note: If A s ot full rak (for some value k): a ka a ka A aa aa kaa aka 0 a a - A Thus does ot exst f s ot full rak a a A A A Whle there are rules for geeral r r matrces, we wll use computers to solve them Regresso Example: ' ' ' Note: '
Use of Iverse Matrx Solvg Smultaeous Equatos A = C where A ad C are matrces of of costats, s matrx of ukows - - - A A A C = A C (assumg A s square ad full rak) Equato : y 6y 48 Equato : 0y y 6 y 48 - A 0 y C = A C - 6 6 A ( ) 6(0) 0 84 0 - = A C 6 48 96 7 68 84 0 84 480 44 84 336 4 Note the wsdom of watg to dvde by A at ed of calculato!
Useful Matrx Results All rules assume that the matrces are coformable to operatos: Addto Rules: A B B A ( A B) C A ( B C) Multplcato Rules: (AB)C A(BC) C( A B) CA + CB k( A B) ka kb k scalar Traspose Rules: ( A ')' A ( A B)' A ' B ' ( AB)' B'A' (ABC)' = C'B'A' Iverse Rules (Full Rak, Square Matrces): - - - - - - - - - - - (AB) = B A (ABC) = C B A (A ) = A (A') = (A )'
Radom Vectors ad Matrces Show for case of =3, geeralzes to ay : Radom varables:,, 3 3 E Expectato: E E I geeral: Ej E,..., ; j,..., p p E 3 Varace-Covarace Matrx for a Radom Vector: E E E E ' E E E E 3 E3 3 E 3 E E E E 3 E 3 E E E E 3 E3 3 E3 E 3 E3 E 3 E 3 3 E 3 3 3 3
Lear Regresso Example (=3) Error terms are assumed to be depedet, wth mea 0, costat varace : j E 0, 0 j 0 0 0 0 ε E ε σ ε 0 0 I 3 0 0 0 σ σ β + ε = β + ε E E β + ε β E ε β 0 0 0 0 σ ε 0 0 I
Mea ad Varace of Lear Fuctos of A matrx of fxed costats k radom vector a a W a... a W A radom vector: k W a a W a... a k k k k k E W E W EW k E a... a a E... a E Eak... ak ake... ake a a E AE a a E k k σ W = E A - AE A - AE ' = E A - E A - E ' = E A - E - E 'A' = AE - E - E ' A' = Aσ A'
Multvarate Normal Dstrbuto μ = E Σ = σ Multvarate Normal Desty fucto: f / / - Σ exp - μ 'Σ - μ ~ Nμ, Σ ~,,..., N, j Note, f A W = A ~ N s a (full rak) matrx of fxed costats: Aμ, AΣA' j j
Smple Lear Regresso Matrx Form Smple Lear Regresso Model:,..., 0 0 0 Defg: 0 0 0 0 = β ε 0 = β + ε sce: β E Assumg costat varace, ad depedece of error terms : 0 0 0 0 σ = σ ε I 0 0 Further, assumg ormal dstrbuto for error terms : ~ N β, I
Estmatg Parameters by Least Squares Q Q Normal equatos obtaed from:, ad settg each equal to 0: 0 0 0 ( ) ( ) Note: I matrx form: ' 'β = ' β = ' ' - 0 ' Defg β Based o matrx form: Q - β ' - β = ' - 'β - β'' + β''β ' - 'β + β''β ' 0 0 0 Q 0 set 0 - ( Q) ' 'β 0 ' β ' β = ' ' β Q 0 Geeral Result for fxed symmetrc matrx A ad varable vector w : w Aw A' w waw Aw
0 b e 0 Ftted Values ad Resduals 0 - - β = ' ' = P P = ' ' 0 I Matrx form: P s called the "projecto" or "hat" matrx, ote that P s dempotet ( PP = P) ad symmetrc( P = P' ): e - = - β = - P = (I - P) - - - - - - PP = ' ' ' ' 'I ' ' ' ' P P' = ' ' ' = ' ' = P Note: - E = E P = PE = Pβ = ' 'β = β σ = Pσ IP' P MSE MSE E e = E I P = I P E = I P β β - β = 0 σ e = I P σ I I P ' I P s P s e = I P
Total (Corrected) Sum of Squares: Aalyss of Varace SSTO Note: ' SSTO ' ' 'J J 'J I J Defg SSE as the Resdual (Error) Sum of Squares: sce SSE e'e = - β ' - β = ' - 'β- β ' + β''β = ' - β '' = ' I - P β'' = '' ' - ' = 'P Defg SSR as the Regresso Sum of Squares: SSR SSTO SSE β'' 'P 'J ' P J Note that SSTO, SSR, ad SSE are all QUADRATIC FORMS: 'A for symmetrc ad dempotet matrces A
Ifereces Lear Regresso MSE where - - - β = ' ' E β = ' 'E = ' 'β = β - - - - - - σ β = ' 'σ ' = σ ' 'I ' = σ ' s β ' SSE s MSE Recall: - ' s β MSE MSE MSE MSE MSE Estmated Mea Respose at : h s MSE - h 0 h 'β h h h h's β h h' ' h h Predcted New Respose at - h 0 h 'β h s pred MSE h' ' h h :