Introduction to Matrices and Matrix Approach to Simple Linear Regression

Size: px

Start display at page:

Download "Introduction to Matrices and Matrix Approach to Simple Linear Regression"

Sharlene Green
6 years ago
Views:

1 Itroducto to Matrces ad Matrx Approach to Smple Lear Regresso

2 Matrces Defto: A matrx s a rectagular array of umbers or symbolc elemets I may applcatos, the rows of a matrx wll represet dvduals cases (people, tems, plats, amals,...) ad colums wll represet attrbutes or characterstcs The dmeso of a matrx s t umber of rows ad colums, ofte deoted as r x c (r rows by c colums) Ca be represeted full form or abbrevated form: a a a j a c a a a j a c A aj,..., r; j,..., c a a aj a c a r ar arj arc

3 Square Matrx: Number of rows = # of Colums Specal Types of Matrces b b A 8 4 B b b Vector: Matrx wth oe colum (colum vector) or oe row (row vector) d 57 d C 4 D E' 7 3 ' f f f3 d F 3 r c 8 d4 Traspose: Matrx formed by terchagg rows ad colums of a matrx (use "prme" to deote traspose) G ' G 3 5 h h c h hr H h j,..., r; j,..., c ' h,..., ;,..., rc H j j c r cr hr h rc h c h rc Matrx Equalty: Matrces of the same dmeso, ad correspodg elemets same cells are all equal: 4 6 b b A = b 4, b 6, b, b 0 0 B b b

4 Regresso Examples - Toluca Data Respose Vector: ' Desg Matrx: '

5 Matrx Addto ad Subtracto Addto ad Subtracto of Matrces of Commo Dmeso: C 0 D 4 6 C D C D a a c b b c A rc a j,..., r; j,..., c b B j,..., r; j,..., c rc ar a rc br b rc a b a c b c A B aj b j,..., r; j,..., c rc a rbr arc b rc a b a c b c AB aj b,..., ;,..., rc j r j c ar br arc b rc Regresso Example: E E E E E sce E ε E ε E E E E

6 Matrx Multplcato Multplcato of a Matrx by a Scalar (sgle umber): 3() 3() 6 3 k 3 A 7 ka 3( ) 3(7) 6 Multplcato of a Matrx by a Matrx (#cols( A) = #rows( B)): If c r : A B A B AB = ab,..., r ; j,..., c r c r c r c A A B B A ab c r th th j sum of the products of the A B elemets of row of A ad j colum of B: 5 3 A 3 B (3) 5() ( ) 5(4) 6 8 A B AB 3(3) ( )() 3( ) ( )(4) 7 7 0(3) 7() 0( ) 7(4) B j A B c If c r c : A B AB = ab = a b,..., r ; j,..., c A B j k kj A B ra ca rb cb ra cb k

7 Matrx Multplcato Examples - I Smultaeous Equatos: ( equatos: x, x ukow): a x a x y ax ax y a x a x y a x a x y a a x y a a x y A = Sum of Squares: Regresso Equato (Expected Values):

8 Matrx Multplcato Examples - II Matrces used smple lear regresso (that geeralze to multple regresso): ' ' ' β 0

9 Specal Matrx Types Symmetrc Matrx: Square matrx wth a traspose equal to tself: A = A': A A' A Dagoal Matrx: Square matrx wth all off-dagoal elemets equal to 0: b 0 0 A b 0 B Note:Dagoal matrces are symmetrc (ot vce versa) b 3 Idetty Matrx: Dagoal matrx wth all dagoal elemets equal to (acts lke multplyg a scalar by ): 0 0 a a a3 a a a3 I 0 0 A a a a IA AI A a a a a3 a3 a 33 a3 a3 a 33 Scalar Matrx: Dagoal matrx wth all dagoal elemets equal to a sgle umber" k k k k I 0 0 k k Vector ad matrx ad zero-vector: J 0 Note: ' r ' r rr r r r J rr rr

10 Lear Depedece ad Rak of a Matrx Lear Depedece: Whe a lear fucto of the colums (rows) of a matrx produces a zero vector (oe or more colums (rows) ca be wrtte as lear fucto of the other colums (rows)) Rak of a matrx: Number of learly depedet colums (rows) of the matrx. Rak caot exceed the mmum of the umber of rows or colums of the matrx. rak(a) m(r A,c a ) A matrx f full rak f rak(a) = m(r A,c a ) 3 A 3 Colums of are learly depedet rak( ) = 4 A A A A 0 A A 4 3 B 0 0 Colums of are learly depedet rak( ) = 4 B B B B 0 B B

11 Matrx Iverse Note: For scalars (except 0), whe we multply a umber, by ts recprocal, we get : (/)= x(/x)=x(x - )= I matrx form f A s a square matrx ad full rak (all rows ad colums are learly depedet), the A has a verse: A - such that: A - A = A A - = I A 4 A A A I / / / 0 - B B BB / I / / 6 0 0

12 Computg a Iverse of x Matrx a a A full rak (colums/rows are learly depedet) a a Determat of A A a a a a Note: If A s ot full rak (for some value k): a ka a ka A aa aa kaa aka 0 a a - A Thus does ot exst f s ot full rak a a A A A Whle there are rules for geeral r r matrces, we wll use computers to solve them Regresso Example: ' ' ' Note: '

13 Use of Iverse Matrx Solvg Smultaeous Equatos A = C where A ad C are matrces of of costats, s matrx of ukows A A A C = A C (assumg A s square ad full rak) Equato : y 6y 48 Equato : 0y y 6 y 48 - A 0 y C = A C A ( ) 6(0) = A C Note the wsdom of watg to dvde by A at ed of calculato!

14 Useful Matrx Results All rules assume that the matrces are coformable to operatos: Addto Rules: A B B A ( A B) C A ( B C) Multplcato Rules: (AB)C A(BC) C( A B) CA + CB k( A B) ka kb k scalar Traspose Rules: ( A ')' A ( A B)' A ' B ' ( AB)' B'A' (ABC)' = C'B'A' Iverse Rules (Full Rak, Square Matrces): (AB) = B A (ABC) = C B A (A ) = A (A') = (A )'

15 Radom Vectors ad Matrces Show for case of =3, geeralzes to ay : Radom varables:,, 3 3 E Expectato: E E I geeral: Ej E,..., ; j,..., p p E 3 Varace-Covarace Matrx for a Radom Vector: E E E E ' E E E E 3 E3 3 E 3 E E E E 3 E 3 E E E E 3 E3 3 E3 E 3 E3 E 3 E 3 3 E

16 Lear Regresso Example (=3) Error terms are assumed to be depedet, wth mea 0, costat varace : j E 0, 0 j ε E ε σ ε 0 0 I σ σ β + ε = β + ε E E β + ε β E ε β σ ε 0 0 I

17 Mea ad Varace of Lear Fuctos of A matrx of fxed costats k radom vector a a W a... a W A radom vector: k W a a W a... a k k k k k E W E W EW k E a... a a E... a E Eak... ak ake... ake a a E AE a a E k k σ W = E A - AE A - AE ' = E A - E A - E ' = E A - E - E 'A' = AE - E - E ' A' = Aσ A'

18 Multvarate Normal Dstrbuto μ = E Σ = σ Multvarate Normal Desty fucto: f / / - Σ exp - μ 'Σ - μ ~ Nμ, Σ ~,,..., N, j Note, f A W = A ~ N s a (full rak) matrx of fxed costats: Aμ, AΣA' j j

19 Smple Lear Regresso Matrx Form Smple Lear Regresso Model:,..., Defg: = β ε 0 = β + ε sce: β E Assumg costat varace, ad depedece of error terms : σ = σ ε I 0 0 Further, assumg ormal dstrbuto for error terms : ~ N β, I

20 Estmatg Parameters by Least Squares Q Q Normal equatos obtaed from:, ad settg each equal to 0: ( ) ( ) Note: I matrx form: ' 'β = ' β = ' ' - 0 ' Defg β Based o matrx form: Q - β ' - β = ' - 'β - β'' + β''β ' - 'β + β''β ' Q 0 set 0 - ( Q) ' 'β 0 ' β ' β = ' ' β Q 0 Geeral Result for fxed symmetrc matrx A ad varable vector w : w Aw A' w waw Aw

21 0 b e 0 Ftted Values ad Resduals β = ' ' = P P = ' ' 0 I Matrx form: P s called the "projecto" or "hat" matrx, ote that P s dempotet ( PP = P) ad symmetrc( P = P' ): e - = - β = - P = (I - P) PP = ' ' ' ' 'I ' ' ' ' P P' = ' ' ' = ' ' = P Note: - E = E P = PE = Pβ = ' 'β = β σ = Pσ IP' P MSE MSE E e = E I P = I P E = I P β β - β = 0 σ e = I P σ I I P ' I P s P s e = I P

22 Total (Corrected) Sum of Squares: Aalyss of Varace SSTO Note: ' SSTO ' ' 'J J 'J I J Defg SSE as the Resdual (Error) Sum of Squares: sce SSE e'e = - β ' - β = ' - 'β- β ' + β''β = ' - β '' = ' I - P β'' = '' ' - ' = 'P Defg SSR as the Regresso Sum of Squares: SSR SSTO SSE β'' 'P 'J ' P J Note that SSTO, SSR, ad SSE are all QUADRATIC FORMS: 'A for symmetrc ad dempotet matrces A

23 Ifereces Lear Regresso MSE where β = ' ' E β = ' 'E = ' 'β = β σ β = ' 'σ ' = σ ' 'I ' = σ ' s β ' SSE s MSE Recall: - ' s β MSE MSE MSE MSE MSE Estmated Mea Respose at : h s MSE - h 0 h 'β h h h h's β h h' ' h h Predcted New Respose at - h 0 h 'β h s pred MSE h' ' h h :

Dr. Shalabh. Indian Institute of Technology Kanpur

Dr. Shalabh. Indian Institute of Technology Kanpur Aalyss of Varace ad Desg of Expermets-I MODULE -I LECTURE - SOME RESULTS ON LINEAR ALGEBRA, MATRIX THEORY AND DISTRIBUTIONS Dr. Shalabh Departmet t of Mathematcs t ad Statstcs t t Ida Isttute of Techology