PubH 7405: REGRESSION ANALYSIS REGRESSION IN MATRIX TERMS

Size: px

Start display at page:

Download "PubH 7405: REGRESSION ANALYSIS REGRESSION IN MATRIX TERMS"

Annabella Spencer
5 years ago
Views:

1 PubH 745: REGRESSION ANALSIS REGRESSION IN MATRIX TERMS

2 A mtr s dspl of umbers or umercl quttes ld out rectgulr rr of rows d colums. The rr, or two-w tble of umbers, could be rectgulr or squre could be just oe row row mtr or row vector or oe colum colum mtr or colum vector. Whe t s squre, t could be smmetrc or dgol mtr o-zero etres re o the m dgol. The umbers of rows d of colums form the dmeso of mtr; for emple, 3 mtr hs three rows d two colums.

3 A etr or elemet of mtr eed two subscrpts for detfcto; the frst for the row umber d the secod for the colum umber: A [ j ] For emple, the followg mtr we hve 6, d , 3 A 33, 47, 35

4 BASIC SL REGRESSION MATRICES M M X X s specl, clled Desg Mtr

5 X: the Desg Mtr for MLR:... k k k k M M M M X Frst subscrpt: Vrble, oe colum for ech predctor; Secod subscrpt: Subject, oe row for ech subject

6 The dmeso of Desg Mtr X s chged to hdle more predctors: oe colum for ech predctor the umber of rows s stll the smple sze. The frst colum flled wth s stll optol ; ot cluded whe dog Regresso through the org.e. o tercept.

7 X s clled the Desg Mtr. There re two resos for the me: B the model, the vlues of X s colums re uder the cotrolled of vestgtors: etres re fed/desged, The desg/choce s cosequetl: the lrger the vrto s ech colum the more precse the estmte of the slope.

8 TRANSPOSE The trspose of mtr A s other mtr, deoted b A or A T, tht s obted b terchgg the colums d the rows of the mtr A; tht s: If A A 3 [ ] the A [ ] j ; A 4 3 The roles of rows d colums re swpped; f A s smmetrc mtr, j j, we hve A A j

9 The trspose of colum vector s row vector C 3 ; C [ 3 ]

10 MULTIPLICATION b Sclr Ag, sclr s ordr umber I multplg mtr b sclr, ever elemet of the mtr s multpled b tht sclr The result s ew mtr wth the sme dmeso A ka [ j [ k ] j ]

11 MULTIPLICATION of Mtrces Multplcto of mtr b other mtr s much more complcted. Frst, the order s mportt; AB s sd to be A s post-multpled b B or B s pre-multpled b A ; I geerl: AB BA There s strog requremet o the dmesos: AB s defed ol f the umber of colums of A s equl to the umber of rows of B. The product AB hs, s ts dmeso, the umber of rows of A d the umber of colums of B.

12 MULTIPLICATION FORMULA A rc B cs AB c [ j] rc[ bj ] cs [ k rs k b kj ] rs For etr,j of AB, we multpl row of A b colum j of B; Tht s wh the umber of colums of A should be the sme s the umber of rows of B.

13 EXAMPLES ] 5 3 [ 5 3 5] 3 [

14 OPERATION ON SLR BASIC DATA MATRICES L L L L L ] [ ] [ X Order s mportt; cot form X

15 OPERATION ON MLR BASIC DATA MATRICES k k k k k M M M M M M L L ] [ ] [ X

16 MORE REGRESSION EXAMPLE M L L X X Ag, X s referred to s the Desg Mtr

17 X X M k M k M k L L L L M MM k k L L L L M M L L L L k k k k M k X X s squre mtr flled wth sums of squres d sums of products; we c form XX but t s dfferet -b- mtr whch we do ot eed.

18 SIMPLE LINEAR REGRESSION MODEL ε X ε X or ;,,..., ; X X X X ε ε ε ε M M M M

19 MULTIPLE LINEAR REGRESSION MODEL k k k k k k X X X ε ε ε ε M M M M M M M...,,..., ;...

20 I σ X E ε X σ b b b b k k b b MLR MODEL IN MATRIX TERMS

21 OBSERVATIONS & ERRORS M ε ε ε ε M

22 : Regresso Coeffcet colum vector of prmeters k M k

23 LINEAR DEPENDENCE Cosder set of c colum vectors C, C,, C c rc mtr. If we c fd c sclrs k, k,, k c ot ll zero so tht: k C k C k c C c the c colum vectors re sd to be lerl depedet. If the ol set of sclrs for whch the bove equto holds s ll zero k k c, the c colum vectors s lerl depedet.

24 EXAMPLE depedet re lerl tors four colum vec The : Sce A

25 RANK OF A MATRIX I the prevous emple, the rk of A<4 Sce the frst, secod d fourth colums re lerl depedet o sclrs c be foud so tht k C k C k 4 C 4 ; the rk of tht mtr A s 3. The rk of mtr s defed s the mmum umber of lerl depedet colums the mtr.

26 SINGULAR/NON-SINGULAR MATRICES If the rk of squre r r mtr A s r the mtr A s sd to be osgulr or of full rk A r r mtr wth rk less th r s sd to be sgulr or ot of full rk.

27 INVERSE OF A MATRIX The verse of mtr A s other mtr A - such tht: A - A AA - I where I s the dett or ut mtr. A verse of mtr s defed ol for squre mtrces. M mtrces do ot hve verse; sgulr mtr does ot hve verse. If squre mtr hs verse, the verse s uque; the verse of osgulr or full rk mtr s lso osgulr d hs sme rk.

28 INVERSE OF A MATRICE A A; or "of " the s bc d D determ t A A D D c D b D d d c b

29 A sgulr mtr does ot hve verse becuse ts determts s zero: A 7 We hve 6 : d : D - 67

30 A SSTEM OF EQUATIONS It s etremel smple to wrte sstem of equtos the mtr form especll wth m equtos z z z z

31 A SIMPLE APPLICATION : otto mtr wrtte whch s 3 4 : equtos two sstem of Cosder Soluto: & 4

32 ANOTHER EXAMPLE z z

33 REGRESSION EXAMPLE X X D

34 We c esl see tht D. Ths propert would ot be true f X hs more colums Multple Regresso d colums re ot lerl depedet. If colums.e. predctors/fctors re hghl relted, Desg Mtr pprochg sgulr : Regresso fled!

35 REGRESSION EXAMPLE D X X X X

36 FOUR IMPORTANT MATRICES IN REGRESSION ANALSIS ] [ X X X X X

37 LEAST SQUARE METHOD SLR { }, : Dt Q Q Q δ δ δ δ

38 LEAST SQUARE METHOD MLR { } : equtos We solve...,,...,, : Dt k k k k k k k k k k k Q Q Q Q δ δ δ δ δ δ

39 NORMAL EQUATIONS : ottos I mtr : Equtos Norml b b b b b b Q Q X b X X δ δ δ δ

40 NORMAL EQUATIONS IN MLR k k k k k k k k k e Q e Q e Q δ δ δ δ δ δ

41 I MLR, these orml equtos look more complcted but wll led to the sme result mtr terms: X Xb X

42 LEAST SQUARE ESTIMATES : ottos I mtr : Equtos Norml X X X b X Xb X b b b b b b

43 SUMMAR REGRESSION RESULTS Norml Equtos : X Xb X Lest Squre Estmtes : b X X X

44 WE PROVE THE SAME RESULTS [ ][ ] / b X X X B X X X

45 MORE DIRECT APPROACH Isted of orml equtos, we could strt erler wth the Sum of Squred Errors SSE Sum of squred I mtr errors Q otto Q : : X X X X X... k k

46 X X Xb X X X δ δ X X X X X : Equtos Norml k Q Q Q Q Q δ δ δ δ δ δ L

47 We c prove the orml equtos b tkg dervtve of SSE: SSE - X X X To do tht, we eed to ler: Dervtve of [X ] s X Dervtve of [X X] s [X X]

48 Puttg together δ δ Q : Q X Note : X - X X Q δ δ δq δ X X X X b X X X X X

49 LEAST SQUARE ESTIMATE X X b X b X X X Note the error equto 6.5 of the tet

50 THE HAT MATRIX ^ Xb X[X X X ] [XX X X] H H XX X X s clled the " Ht Mtr"

51 IDEMPOTENC the " Ht Mtr" : H XX X X s "dempotet ": HH H

52 RANDOM VECTORS & MATRICES A rdom vector or rdom mtr cots elemets whch re rdom vrbles.

53 RANDOM VECTORS IN SLR M d ε ε ε M ε

54 EXPECTED VALUES ; M M Eε E ε E E E E

55 VARIANCE-COVARIANCE MATRIX The vrces of elemets of rdom mtr d the covrce betwee two elemets of elemets of rdom mtr re ssembled the vrce-covrce mtr deoted b ether Vr or σ or

56 EXAMPLE: BIVARIATE VECTOR : Mtr Covrce Vrce Me : : Vrble σ σ σ σ µ µ Σ μ

57 We hve for rdom vrbles: E * * E Vr * *vr Wht bout rdom vectors? S: A

58 AE A A E E E E E E E

59 A Aσ A A,,,, σ σ σ σ σ σ σ σ σ L C verf bckwrd too, strtg wth Aσ A

60 REGRESSION EXAMPLES I Vr I Vr ε ε E,,,,,, σ σ σ σ σ σ σ σ σ σ σ L M M M M L L

61 FITTED VALUES Xb ^ b b b b b b b b M M M

62 RESIDUALS Model : Ftted Vlue : ^ Xb Resduls : e X ^ Xb H I H ε Lke the Ht Mtr H, I-H s smmetrc & dempotet

63 VARIANCE OF RESIDUALS H I H I H HI I H HI I H II H I H I Hσ I e σ H I e MSE ^ σ σ σ σ

64 REGRESSION COEFFICIENTS δ δ Q X X X X X Xb b X X X [ X X X ] A A X X X A X X X

65 VARIANCE OF REGRESSION COEFFICIENTS σ s b b b [ X X A Aσ A X X σ σ A IA X X X X MSE X X σ ] IX X X

66 ] [ b b X X X X X X b b eg. results, :sme C verf EXAMPLE: SL REGRESSION

67 MSE MSE b s X X X X b s

68 , MSE b b s MSE b s MSE b s MSE s b Sme results for Vrces; Covrce s ew - Ol eed Me & Vrce of X

69 THE MEAN RESPONSE Let X h deote the level of X for whch we wsh to estmte the me respose,.e. E X h. The ol thg ew s tht X d h re vector; h h, h,, kh. The pot estmte of the respose s: E X b h b h ^ h b h... b k kh

70 I mtr terms: h h h h h h h X X X X X X X X bx σ X b X MSE s ^ ^ ^ σ σ

71 PREDICTION OF NEW OBSERVATION Let X h deote the level of X uder vestgto, t whch the me respose s E X h. Let hew be the vlue of the ew dvdul respose of terest. The pot estmte s stll the sme E X h : ^ h ew b b h b h... bk kh X hb

72 ^ Vr h ew σ { X h X X X h } s ^ h ew MSE{ X h X X X h }

73 The mtr otto d dervtos m be decevg becuse the hde eormous computtol completes. To fd the verse of mtr, for emple X X wth k 9, requres tremedous mout of computto. However, the ctul computtos wll be doe b computer; hece t does ot mtter to us whether X X represets or 66 mtr.

74 Emple: SBP versus AGE Totls Age SBP -sq -sq

75 ,64 5 5, , ,64 67,954 5, ,954 67, X X X X D

76 X X X X X b ,93 46,6 X X , X.93, ,6

77 VARIANCE OF REGRESSION COEFFICIENTS b X X X s b MSE X X MSE

78 SUMMARIES All results c be put the mtr forms If we c verse mtr d c multpl two mtrces, we c get ll umercl results eve wthout pckged computer progrm. I mtr ther forms, results c be eser geerlzed; the ol chge eeded s the Desg Mtr & ts dmeso so s to hdle more th oe predctors.

79 Redgs & Eercses Redgs: A thorough redg of the tet s sectos pp.76-9 d sectos pp.-47 s hghl recommeded. Eercses: The followg eercses re good for prctce, ll from chpter 5 of tet: 5.,5., d ; plus these from chpter 6 of tet: 6.5b-d, 6.7, 6.-d, d 6.5-f.

80 Due As Homework X #. The followg dt were collected durg epermet whch lbortor mls were oculted wth pthoge. The vrbles re Tme fter oculto X, mutes d Temperture, Celsus degrees. For the regresso of s depedet vrble o X s sole predctor, form these mtrces:, X, d X X #. Solve the followg sstem of equtos:

Chapter Unary Matrix Operations

Chapter Unary Matrix Operations Chpter 04.04 Ury trx Opertos After redg ths chpter, you should be ble to:. kow wht ury opertos mes,. fd the trspose of squre mtrx d t s reltoshp to symmetrc mtrces,. fd the trce of mtrx, d 4. fd the ermt