CORRELATION AND REGRESSION

Size: px

Start display at page:

Download "CORRELATION AND REGRESSION"

Kory Caldwell
6 years ago
Views:

1 : Coelato ad Regesso CORRELATION AND REGRESSION N. Okedo Sgh Ida Agcultual Statstcs Reseach Isttute, New Delh - okedo@as.es.. Coelato Whe a bvaate dstbuto (volves two vaables) s ude cosdeato, thee s geeall a eed to stud the smultaeous vaato betwee two vaables sa, X ad Y. Fo eample, the vaato betwee legths ad weghts of a goup of fshes, heghts ad weghts of pesos, demad ad suppl of commodtes, etc. I pactce, ofte data o moe tha oe chaacte o the same ut ae obtaed, fo eample, data o fetlze appled ad eld dffeet felds, measuemet of legth ad ccumfeece of ea-head, weathe ad dsease cdece dffeet eas, etc. The teest ma be to stud the tpe ad etet of elatoshp such pas of vaables. The fst step s to plot the obsevatos o a gaph. If the chage oe vaable affects a chage the othe vaable, the vaables ae sad to be coelated. Thee tpes of coelato ae: ) Postve coelato: Smultaeous ceases ad deceases. ) Negatve coelato: Oe vaable ceases ad othe deceases. 3) Spuous coelato: Nethe ceases o deceases.. Scatte Dagam Suppose thee ae dvduals (uts) gvg se to pas of obsevatos (, ),...,(, ) o two vaables X ad Y. These pas of data ae plotted o a pece of gaph pape gvg se to dots, oe coespodg to each ut o the gaph pape. Such a vsual dspla of dots s kow as scatte dagam o dot dagam. A vsual specto of a dot dagam pemts us to judge f the two vaables ae elated postvel o egatvel ad also the stegth of elatoshp.e., week o stog. Howeve, f the umbe of data pots s lage, we ma ot get a fal good dea about the elatoshp usg ths method. Coelato coeffcet s a measue of test o degee of lea elatoshp betwee two vaables. I othe wods, fo a gve pa of elated measues (X ad Y) o each of a set of tems, the coelato coeffcet () povdes a de of the degee to whch the paed measues co-va a lea fasho. Coelato coeffcet betwee 8

2 : Coelato ad Regesso two adom vaables X ad Y, usuall deoted b elatoshp betwee them ad s defed b O, XY XY Cov Va ( X, Y) ( X ).Va( Y). XY s a umecal measue of lea It ma be oted that coelato coeffcet gves etet of lea elatoshp. A othe fom of elatoshp caot be judged though ths coelato coeffcet. Thus the zeo value of coelato coeffcet dcates absece of lea elato but ot absece of elatoshp. Coelato coeffcet eve eceeds ut umecall. It alwas les betwee - ad +. If +, the coelato s pefect ad postve ad f -, coelato s pefect ad egatve. If a costat quatt s added o deleted fom eve obsevato o X o/ad Y the coelato coeffcet wll ema same. Smlal, f eve obsevato o X ad / o Y s multpled o dvded b a costat quatt, coelato coeffcet emas 9

3 : Coelato ad Regesso uchaged. I statstcal laguage, we ca sa that s depedet of chage of og ad scale. The estmated stadad eo of s gve b S.Ê. () The coelato coeffcet defed above (also kow as Kal Peaso's coeffcet of coelato) s ot adequate to ude the followg codtos: a) Sample sze s fal lage. b) Whe we deal wth qualtatve chaactestcs (attbutes). c) Coelato s affected b the outle o eteme values..3 Test of Sgfcace of Coelato Coeffcet Oe-Sample Case: It s assumed that these obsevatos ae a sample fom a bvaate populato. The populato coelato coeffcet s geeall deoted b ρ. It ma be of teest to test whethe the populato coelato coeffcet ρ fom whch the peset sample s daw, s o dffeet fom,.e. H : ρ agast, H : ρ. The appopate test statstc s gve b t If the computed value of t eceeds, the Table value of t wth (-) degees of feedom (d.f.) at a gve level of sgfcace (sa 5%), the ull hpothess s ejected ad we coclude that vaables ma be egaded as coelated the populato. Alteatvel, computed value of ca be compaed wth table value of coelato coeffcet at desed level of sgfcace fo (-) degees of feedom. Fo eample, cosde the value.597 obtaed fom a sample of sze 9. Fo 7 d.f. value of at 5% level of sgfcace fom the Table s.666. The obseved s theefoe ot sgfcat ad ull hpothess s ot ejected. Sometmes the teest ma be to test the hpothess that the sample s daw fom a populato whch ρ has some specfed value ρ othe tha zeo.e. H : ρ ρ agast, H : ρ ρ. I ths case, ad ρ ae tasfomed as follows: + Z log e + ρ Z log e ρ Ad the U s computed as U Z Z ( ) 3 If the absolute value of U.e. U >.96, we eject the hpothess, 5% level of sgfcace. H : ρ ρ Two Sample Case: Let ad be the sample coelato coeffcets betwee two vaables obtaed fom two depedet samples of szes ad espectvel. The teest ma be to test f t s easoable to assume that the two samples ae daw fom the same populato o fom dffeet two populatos wth the same coelato coeffcet ρ (sa).e., H : ρ (sa), agast H :. 3 at

4 : Coelato ad Regesso To test ths hpothess, followg s computed Z Z Z S.E. Z Z ( ) + Whee Z log e ;,, ad S.E. ( Z Z ) Ude the ull hpothess Z s N(,), so we appl the stadad omal devate test ude whch the hpothess s ejected at 5% level f Z >.96 ad at % level of sgfcace f Z > Patal Coelato Coeffcets Whle dealg wth two vaables, smple coelato coeffcet povdes the complete pctue of etet of assocato betwee the two vaables but ofte pactce, we ma have to deal wth moe tha two vaables whch ma be te-elated. Fo eample, eld pe ut s flueced b seed ate, sol fetlt, fetlze used, gato avalablt, weathe codtos, etc. Smlal, the sol fetlt s flueced b weathe factos, legumous cops sow, fetlze used ad so o. The coelato betwee cop eld Y ad sol fetlt X ma be patl due to the fact that weathe facto (sa afall), flueces X, Y both. I such cases, coelato coeffcets fo each pa of chaactes ca be obtaed but measues of assocato, thus obtaed wll ot be depedet. Fo eample eld ma be coelated wth tempeatue whch tu s assocated wth afall, the secod facto fluecg eld. I such cases, coelato betwee a pa of chaactes s woked out, keepg emag factos costat (elmatg the effects of othe factos). Ths s kow as patal coelato coeffcet. The patal coelato coeffcet betwee X ad Y keepg Z costat (.z ) s gve b zz.z ( z )( z ) The patal coelato coeffcets obtaed afte emovg the effect of oe vaable as dscussed above ae called patal coelato coeffcets of ode oe. I some stuatos, howeve, we ma have to obta the patal coelato coeffcets afte elmatg the effects of two o moe vaables. The umbe of vaables that ae used fo elmatg the effects s kow as the ode of the patal coelato coeffcet. It s smple coelato coeffcet betwee X (adjusted fo effect of Z) ad Y (adjusted fo effect of Z). The dea s to measue that pat of coelato betwee X ad Y that s ot smpl a eflecto of the elato wth Z. The epessos fo z. ad z. ca be wtte smlal. Hghe ode patal coelato coeffcet sa.34 meas coelato betwee fst ad secod vaables afte elmatg the effects of 3 d ad 4 th vaables fom each of the fst ad secod vaables. It ca be computed as ( 3.4 )( 3.4 ) O,.34 ( )( ) Test of Sgfcace of Patal Coelato Coeffcet The test fo the hpothess about the populato patal coelato coeffcets ca be made o the smla les as that of smple coelato ecept that the umbe of 3

5 : Coelato ad Regesso vaables kept costat s subtacted fom the degees of feedom.e. (-) s eplaced b (-k-) whee k s umbe of vaables kept costat. To test : ρ ( + ) agast : ρ ( + ), the coespodg test statstc s gve b: t H j... k j... ( k+ ) j... ( k+ ) k H j... k Whee k s the ode of the coeffcet. Ths statstc follows t-dstbuto wth (-k-) degees of feedom. We eject the ull hpothess f t s geate tha the coespodg Table value of t fo (-k-) d.f. at 5% level of sgfcace..6 Rak Coelato Sometmes the vaables ude stud ae such that eact magtude fo these caot be ascetaed but the dvduals ca be aked accodg to some ctea, fo eample, beaut, tellgece, leadeshp qualtes, aggessveess, domace, etc. I such cases both the vaables ae epessed as aks ad ak coelato coeffcet, due to Speama, s oda coelato coeffcet betwee aked value of the two vaables ude stud. The coelato betwee the aks of two chaactes s called the ak coelato. Let ( ; be the aks of the th, ),,... dvdual two chaactestcs of A ad B espectvel. We assume that o two dvduals ae backeted equal ethe classfcato. Peasoa coeffcet of coelato betwee the aks s ad s s called the ak coelato coeffcet (deoted b ρ ) betwee A ad B fo that goup of dvduals. The Speama's fomula fo the ak of coelato coeffcet s d ( ) 6 ρ( X,Y) whee, d ;,,... deotes the dffeece betwee the aks of th dvdual. The lmts fo ak coelato coeffcet ae ρ. Ted Raks: If some of the dvduals eceve the same ak a akg of met, the ae sad to be ted. Let us suppose that m of the dvduals, sa, ( k + ) th,( k + ) th,...,( k + m) th ae ted. The each of these m dvduals s assged a commo ak, whch s the athmetc mea of the aks k+, k+,,k+m. Thus, case of ted aks the above fomula becomes, 6 d + TX + TY ( X, Y) ρ s whee, X m j( m j ) ( ) t T ;,,...,s & TY m m k k ; k,,..., t. j k Suppose that m ad m of the dvduals ae ted the X ad Y-sees ad also thee ae s ad t sets of ted aks espectvel. j j ( ) k 3

6 : Coelato ad Regesso Note: () Speama s fomula s eas to udestad ad appl as compaed wth Kal Peaso s fomula. The value obtaed b the two fomulae, vz., Peasoa s, ad Speama s ρ, ae geeall dffeet. The dffeece ases due to the fact that whe akg s used stead of full set of obsevatos, thee s alwas some loss of fomato. Uless ma tes est, the coeffcet of ak coelato should be ol slghtl lowe tha the Peasoa coeffcet. () d ( ) calculatos., whch povdes a umecal check fo Eecse : Te studets got the followg pecetage of maks Ecoomcs ad Statstcs: Maks Ecoomcs (X): Maks Statstcs (Y) : Calculate the coelato coeffcet betwee X ad Y. Soluto: Calculatos fo coelato coeffcet () X Y X Y XY Total ; ( ) ( 66) 33

7 : Coelato ad Regesso (appo.) Eecse : Calculate a) the coelato coeffcet () ad b) ak coelato coeffcet (ρ ) fo the followg suvvablt ate ( pecetage) of paets (X) ad the offspgs (Y) foud beedg pogam o dffeet fsh vaetes at a patcula fsh fam: X: Y: Soluto: a) Calculatos fo Kal Peaso's Coeffcet of Coelato () X Y X Y XY Total ; (appo.) ( ) ( ) 34

8 : Coelato ad Regesso b) Calculato fo Rak Coelato Coeffcet (ρ ) X Y Rak of X Rak of Y d- d () () Total d d 7 I the X-sees we see that the value 75 occus tmes. The commo ak gve to these values s.5, whch s the aveage of ad 3, the aks whch these values would have take f the wee dffeet. The et value 68, the gets et ak whch s 4. Aga we see that value 64 occus thce. The commo ak gve to t s 6, whch s the aveage of 5, 6 ad 7. Smlal Y-sees, the value 68 occus twce ad ts commo ak s 3.5, whch s the aveage of 3 ad 4. As a esult of these commo akgs, the fomula has to be coected. To d m( m ) we have to add fo each value epeated, whee m s the umbe of tmes a value occus. I the X-sees the coecto s to be appled twce, oce fo the value 75 whch occus twce (m) ad the fo the value 64, whch occus thce (m3). The total coecto fo the X-sees s ( 4 ) 3( 9 ) 5 + T X Smlal, ths coecto fo the Y-sees whch the value 68 occus twce, s ( 4 ) T Y Thus, the equed ak coelato coeffcet s ρ (, Y) 6 d X T X ( ) 5 ( ) + T + Y 35

9 : Coelato ad Regesso. Regesso I a sstem whch vaable quattes chage, t s of teest to eame the effects that some vaables eet (o appea to eet) o othes. Sometmes two vaables ae lked b a eact lea elatoshp (mathematcal elatoshp). Fo eample, f the esstace of a smple ccut s kept costat, the cuet vaes dectl wth voltage appled (Ohm s law). Sometmes the lea elatoshp s ot eact. Fo eample, suppose heghts ad weghts of adult males fo some gve populato ae cosdeed ad pas of (Y, X) (weght, heght) ae plotted fo a gve heght, thee s a age of obseved weghts ad vce vesa. Ths vaato wll be patall due to measuemet eos ad pmal due to vaato betwee dvduals. Thus, o uque elatoshp betwee heght ad weght ca be epected but t ca be oted that the aveage obseved weght fo a gve obseved heght ceases as heght ceases. Ths cuve of aveage obseved weghts fo gve obseved heghts s called egesso cuve of weght o heght. If the cuve s a staght le, t s called lea egesso. Smlal, egesso cuve of heght o weght ca be defed. I geeal, these two cuves ae ot the same. Regesso s the lea o o-lea elatoshp betwee two o moe adom vaables. Regesso Aalss s a mathematcal measue of the aveage elatoshp betwee two o moe vaables tems of the ogal uts of the data. I othe wods, t s a techque to deteme the aveage elatoshp betwee two o moe vaables. Aga, egesso aalss ma be boadl defed as the aalss of elatoshps amog vaables. It s oe of the most wdel used statstcal tools because t povdes a smple method fo establshg a fuctoal elatoshp amog vaables. The elatoshp s epessed the fom of a equato coectg the espose o depedet vaable, ad oe o moe depedet vaables,,..., P. The multple egesso equato takes the fom: β + β + β β + ε p p whee β, β,..., βp ae called the egesso coeffcets ad ae detemed fom the data. A egesso equato cotag ol oe depedet vaable s called a smple egesso equato. A equato cotag moe tha oe depedet vaable s efeed to as a multple egesso equato. Thus, the esdual at each data pot s gve b: ε β β β... β The sum of the squae of the esduals s gve b: S ε To fd the costats o paametes of the multple lea egesso model, we put the devatves wth espect to ; (j,,,...,p) to zeo, that s, S S S β j ( β ) ( β β β... β )( ) ( β ) ( β β β... β )( ) ( βp ) ( β β β... βp p )( p ) Settg those equatos mat fom gves: p... p p p p p

10 : Coelato ad Regesso β... p β... p βp p p... p p The above smultaeous lea equatos ae solved fo (p+) costats, β, β,..., βp. I mat otato, we ca wte the model wth addtve eo tem as Xβ + ε ( ) ε E ( ) ( ) Whee ε ε, ε,..., ε a d D() I, ˆ β X X X' ε. Le of Regesso: The le of egesso s the le whch gves the best estmate to the value of oe vaable fo a specfc value of the othe vaable. Thus, the le of egesso s the le of best ft ad s obtaed b the pcples of least squae. Let us suppose that the bvaate dstbuto (, ),,,..., ; Y s depedet vaable ad X s depedet vaable. Let the le of egesso of Y o X be β + β. The paametes β ad β ae estmated b the method of least squaes whch volves mmzg the sums of squaes of the esduals, whee ad S β β ) ( β, β ) ε ( The omal equatos fo estmatg β ad β ae: β β + β + β O solvg, we get β β ; whee, ad β ( )( ) ( ) Cov Va (, ) ( ) Thus, the le of egesso of Y o X s ˆ β ( ) ( ) Smlal, the le of egesso of X o Y s ˆ β ( ) ( ) Thee ae alwas two les of egesso, oe of Y o X ad the othe of X o Y. The le of egesso of Y o X s used to estmate o pedct the value of Y fo a gve value of X.e., whe Y s a depedet vaable ad X s a depedet vaable. The estmate so obtaed wll be best the sese that t wll have mmum possble eo 37

11 : Coelato ad Regesso as defed b the pcple of least squaes. Smlal, the le of egesso of X o Y s used to estmate the value of X fo a gve value of Y. The two egesso equatos ae ot evesble o techageable. Regesso Coeffcet s the chage the value of depedet vaable fo a ut chage the value of depedet vaable. Some of the mpotat popetes ae lsted below: ) Coelato coeffcet s the geometc mea betwee the egesso coeffcets, ˆβ ad ˆβ..e., ˆ ˆ β β ± βˆ ˆ β ) If oe of the egesso coeffcets s geate tha ut, the othe must be less tha ut. 3) Athmetc mea of the egesso coeffcets f geate tha the coelato coeffcet, povded >. 4) Regesso coeffcets ae depedet of the chage of og but ot of scale. 5) The age of egesso coeffcet s to +. 6) The sgs of egesso coeffcets ad coelato coeffcet ae alwas same. 7) If the vaables X ad Y ae depedet, the egesso coeffcets ae zeo. Wh Do We Need Regesso Aalss? I ma stuatos the depedet vaable Y s such that t ca ot be measued dectl. But, wth the help of some cocomtat o aula vaables as depedet vaables a egesso equato, Y ca be estmated.. Testg of Paametes If a egesso s ftted, the fst test would be to test the sgfcat of oveall egesso. If t s ot sgfcat, thee s o eed of futhe testg. Suppose the oveall egesso comes out to be sgfcat the oe must test the sgfcace of a o othe egesso. Whe the ull hpothess specfes egesso though og, the t s coect to test the sgfcace of ˆβ befoe testg othe coeffcets. Fo the pupose of aalss of vaace oe eeds the followg sum of squaes. Sum of squae due to egesso (SS R ) ( ŷ ) 38

12 : Coelato ad Regesso Sum of squaes due to eos (esduals) (SS E ) ( ) Total sums of squae (SS T ) ( ) ŷ It s usuall ecessa to estmate. A ubased estmate of s gve b: SS ˆ s E ( p ) Whee s the umbe of obsevatos ad (p+) s the umbe of paametes the model. To deteme f thee s lea elatoshp betwee espose vaable Y ad a subset of egesso X, we set up the hpothess: H : β β... βp H : β j fo at least oe j. Rejecto of ou ull hpothess mples that at least oe of (,,..., p ) cotbutes sgfcatl to the model. The test statstc ude ull hpothess s: SSR p MSR F SSE ( p ) MSE We eject ou ull hpothess f F eceeds coespodg table value F. ANOVA Table: α,p, p Souce of vaato d.f. SS MSS F Regesso p SS R SS R p MS MS R R /MS E Eo (-p-) SS E SS E ( p ) MSE Total (-) SS T Ide of Ft The coeffcet of detemato R s gve b: SSR SSE R SST SST The R measues the educto vaablt of Y obtaed usg the egessos ( ). Also,. The lage values of R,,..., p R do ot ecessal mpl that model s good as addto of vaable wll alwas cease R. Fo ths easo, ma a tmes R adjusted s used as statstc fo the pupose ad t s gve b: SS ( ) ( ) ( ) ( ) ( ) E p R adj R SST p Geeall, the adjusted R statstc wll ot alwas cease as vaables ae added to the model. I fact, f u-ecessa vaables ae added to the model R adj wll ofte decease. A cauto, howeve, s that whle the R-squae s a pecet, the Adjusted R- squae s NOT ad should be efeed to as a de value. 39

13 : Coelato ad Regesso.3 Sub-hpothess Testg The hpothess fo testg a dvdual egesso coeffcet, sa H : β j H : β j If H s ot ejected, the ths mples that j ca be deleted fom the model. The test statstc ths case s: βˆ j t ˆ C jj Whee C jj s the j th dagoal elemet of ( X X). The quatt ˆ C jj s ofte called stadad eo of ˆβ j. βˆ j Theefoe, t SE( β ˆ ad we eject the ull hpothess f t s geate tha the j ) coespodg Table value of t fo (-p-) d.f. at 5% level of sgfcace. β j ae: Eecse 3: Obta the egesso of Y o X, ad X o Y fom the followg table ad estmate the wate tempeatue ( C) whe the a tempeatue ( C) s 5. A Temp. (X): Wate Temp. (Y): Soluto: X Y X Y XY X 6.7 Y 94.7 X Y XY 59.7 X ; Y We have the omal equatos Y β + βx ad XY β X + β X Substtutg the above kow values ad o solvg, we get β β.854 Now the egesso le of Y o X s Y X Whe X5, 4

14 : Coelato ad Regesso Y ( C). Smlal, fo the egesso le of X o Y, the followg omal equatos wll be used X β + βy ad XY β Y + β Y O solvg we ca get the values of β ad β. Thus, the egesso le ca be used to estmate a tempeatue (X) fo a gve vale of wate tempeatue. Note: ) The assumpto of omalt s ot equed fo the estmato of the coeffcets b least squaes method. To make tests ad estmate cofdece tevals, howeve, these assumptos ae equed: (a) the eos ae omall dstbuted wth mea zeo, (b) the eos have costat vaace, (c) the eos ae depedet of each othe. ) Coelato ad egesso aalss should be appled caefull. Vaables should be caefull selected such as the ae epected to be eall leal elated. Sometmes data ma gve a fa degee of coelato b chace eve though thee ma be o elatoshp betwee the vaables cosdeed. Fo eample, a postve coelato ma be obtaed a sample of dvduals, betwee sze of the shoe ad amout of moe the pocket whch s meagless. Smlal, egesso equato, cluso of a eta vaable as egesso wll cease R, howsoeve eglgble the cease ma be. Icluso of supefluous vaables wll ot mpove the model eal sese, eve f R ceases. Dffeet combatos of pedctos ma be compaed o the bass of adjusted R, ad ot o the bass of R, as wth cluso of eve ew egesso, eo d.f. wll be educed b oe ad, thus, R based o dffeet umbe of egessos wll have dffeet eo degees of feedom. 3) I case, some of the eal pedcto vaables ae left out, the model fttg wll ot be adequate, eos wll cease whch wll ot be eal eos but wll clude lack of ft of the model. If epeated obsevatos ae avalable, the esdual SS ca be splt up to pue eo ad lack of ft SS whch ca be used to test lack of ft agast pue eo. The testg of lack of ft agast pue eo wll povde the fomato o ths aspect. 4) Thee should be eough vaablt the data. 5) The vaablt should be such whch ca be captued b pedcto vaables. 6) Pedctos should be made wth the obseved age of pedcto vaables.e. etapolato ca be dageous. It wll be clea fom the followg eample: It s clea fom the fgue that fo the values of X betwee to 5, the elatoshp s lea. But f ths lea elatoshp s utlzed to pedct value of Y fo X > 5, the pedctos wll be fa fom ealt. Selecto of vaables, adequac of models 4

15 : Coelato ad Regesso ad some pactcal poblems alog wth the emedes wll be dscussed detal subsequet lectues. Suggested Readgs: ) Saea, H.C. (988). Elemeta Statstcs. 7 th Edto, S. Chad ad Compa (Pvt) Ltd. New Delh. ) Chattejee, S. ad Pce, B. (977). Regesso Aalss b Eample. Joh Wle ad Sos. New Yok. 3) Dape, N.R. ad Smth, H. Appled Regesso Aalss, Joh Wle & Sos. 4) Sedeco, G.W. ad Cocha, W.G. Statstcal Methods, Ofod ad IBH Publshg Co. 4

Professor Wei Zhu. 1. Sampling from the Normal Population

Professor Wei Zhu. 1. Sampling from the Normal Population AMS570 Pofesso We Zhu. Samplg fom the Nomal Populato *Example: We wsh to estmate the dstbuto of heghts of adult US male. It s beleved that the heght of adult US male follows a omal dstbuto N(, ) Def. Smple