Estimation of bivariate linear regression data via Jackknife algorithm

Transcription

1 Itertiol Jourl of Sttistics d Mthemtics Vol. 1(1), pp , April, ISSN: x IJSM Reserch Article Estimtio of ivrite lier regressio dt vi Jckkife lgorithm Osuji George A 1, Ekezie D D 2, Opr Jude 3 d Ogo Chukwudi J 4 1 Deprtmet of Sttistics, Nmdi Azikiwe Uiversity, PMB 5025, Awk Amr Stte Nigeri 2,3 Deprtmet of Sttistics, Imo Stte Uiversity, PMB 2000, Owerri, Nigeri. 4 Deprtmet of Sttistics, Federl Uiversity of Techology, PMB 1526, Owerri Nigeri. Correspodig uthor: Prof. Ekezie D D, Deprtmet of Sttistics, Imo Stte Uiversity, PMB 2000, Owerri, Nigeri, Tel: , E-mil: dekezie@yhoo.com This pper is o the estimtio of ivrite lier regressio dt usig Jckkife lgorithm. Jckkife delete-oe lgorithm ws used to provide estimtes of ivrite lier regressio coefficiet. The dt used for this reserch were collected from Orji Tow Primry School, Owerri North Imo Stte Nigeri. The dt were o heights d weights of 20 rdomly selected pupils i primry five d six. Prtil estimtes, Pseudo-Vlues, Jckkife Estimtes, Jckkife Stdrd Devitios, d the Jckkife Stdrd Error were lso computed. The result of the lysis reveled tht the is result ws positive, which implies tht the coefficiet of correltio over-estimtes the mgitude of the popultio correltio. For the regressio, the jckkife prmeters re lier fuctios of the stdrd estimtes, which mes tht the vlues of c e perfectly predicted from the vlues of d the cofidece itervl were lso estimted.. The jckkife predicted vlues Key words: Jckkife lgorithm, Bivrite lier regressio, Prtil estimtes, Pseudo-Vlues INTRODUCTION The jckkife or leve oe out procedure is cross-vlidtio techique first developed y Queouille (1956) to estimte the is of estimtor. Tukey (1958) the expded the use of the jckkife to iclude vrice estimtio d tilored the me of jckkife ecuse like jckkife- pocket kife ki to Swiss rmy kife d typiclly used y oy scouts-this techique c e used s quick d dirty replcemet tool for lot of more sophisticted d specific tools. The jckkife estimtio of prmeter is itertive process. First the prmeter is estimted from the whole smple. The ech elemet is, i tur, dropped from the smple d the prmeter of iterest is estimted from this smller smple. This estimtio is clled prtil estimte (or lso jckkife replictio). A pseudo-vlue is the computed s the differece etwee the whole smple estimte d the prtil estimte. These pseudo-vlues reduce the (lier) is of the prtil estimte (ecuse the is is elimited y the sutrctio etwee the two estimtes). The pseudo-vlues re the used i lieu of the origil vlues to estimte the prmeter of iterest d their stdrd devitio is used to estimte the prmeter stdrd error which c the e used for ull hypothesis testig d for computig cofidece itervls. The jckkife is strogly relted to the ootstrp (i.e., the jckkife is ofte lier pproximtio of the ootstrp) which is curretly the mi techique for computtiol estimtio of popultio prmeters. Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

2 Ekezie et l 009 As potetil source of cofusio, somewht differet (ut relted) method, lso clled jckkife is used to evlute the qulity of the predictio of computtiol models uilt to predict the vlue of depedet vrile(s) from set of idepedet vrile(s). Such models c origite, for exmple, from eurl etworks, mchie lerig, geetic lgorithms, sttisticl lerig models, or y other multivrite lysis techique. These models typiclly use very lrge umer of prmeters (frequetly more prmeters th oservtios) d re therefore highly proe to over-fittig (i.e., to e le to perfectly predict the dt withi the smple ecuse of the lrge umer of prmeters. ut eig poorly le to predict ew oservtios). I geerl, these models re too complex to e lyzed y curret lyticl techiques d therefore the effect of over-fittig is difficult to evlute directly. The jckkife c e used to estimte the ctul predictive power of such models y predictig the depedet vrile vlue of ech oservtio s if this oservtio were ew oservtio. I order to do so, the predicted vlue(s) of ech oservtio is (re) otied from the model uilt o the smple of oservtios mius the oservtio to e predicted. The jckkife, i this cotext, is procedure which is used to oti uised predictio (i.e. rdom effect) d to miimize the risk of over-fittig. The im of this reserch is to estimte the ivrite lier regressio coefficiet usig the jckkife delete-oe lgorithm. Review of relted Litertures Sho d Ro (1993) crried out reserch pper o Jckkife iferece for heteroscedstic lier regressio models. Iferece o the regressio prmeters i heteroscedstic lier regressio model with replictio is cosidered, usig either the ordiry lest-squres (OLS) or the weighted lest-squres (WLS) estimtor. A deletegroup jckkife method is show to produce cosistet vrice estimtors irrespective of withi-group correltios, ulike the delete-oe jckkife vrice estimtors or those sed o the customry δ-method ssumig withigroup idepedece. Fiite-smple properties of the delete-group vrice estimtors d ssocited cofidece itervls re lso studied through simultio. Hogchg d Yuhe (2013) worked o Jckkifed Liu estimtor i lier regressio models. I their pper, they itroduced geerlized Liu estimtor d jckkifed Liu estimtor i lier regressio model with correlted or heteroscedstic errors. Therefore, they exteded the Liu estimtor. Uder the me squre error (MSE), the jckkifed estimtor ws superior to the Liu estimtor d the jckkifed ridge estimtor. They lso gve method to select the isig prmeter for d. Furthermore, umericl exmple ws give to illustrte these theoreticl results. Wu (1986) reserched o Jckkife ootstrp d other re-smplig methods i regressio lysis; motivted y represettio for the lest squres estimtor, they proposed clss of weighted jckkife vrice estimtors for the lest squres estimtor y deletig y fixed umer of oservtios t time. They re uised for homoscedstic errors d specil cse, the delete-oe jckkife, is lmost uised for heteroscedstic errors. The method ws exteded to cover olier prmeters, regressio M-estimtors, olier regressio d geerlized lier models. Three ootstrp methods were cosidered. Two were show to give ised vrice estimtors d oe does ot hve the is-roustess property ejoyed y the weighted delete-oe jckkife. A geerl method for re-smplig residuls ws proposed, d some simultio results were reported. METHODOLOGY The gol of the jckkife is to estimte prmeter of popultio of iterest from rdom smple of dt from this popultio. The prmeter is deoted y, its estimte from smple is deoted y T, d its jckkife estimte is deoted y T. The smple of N oservtios (which c e uivrite or multivrite) is set deoted (X 1,...,X,..., X N ). The smple estimte of the prmeter is fuctio of the oservtios i the smple. I forml wy, T= f(x 1... X... X N ) (1) A estimtio of the popultio prmeter otied without the th oservtio, is clled the -th prtil predictio, d is deoted T -. Formlly: T - = f(x 1,, X -1, X +1,, X N ) (2) A pseudo-vlue estimtio of the th oservtio is deoted y T, it is computed s the differece etwee the prmeter estimtio otied from the whole smple d the prmeter estimtio otied without the th oservtio. Formlly: T = NT (N l)t - (3) The jckkife estimte of, deoted T is otied s the me of the pseudo-vlues. Formlly: Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

3 It. J. Stt. Mth. 010 T 1 where N T T (4) N T is the me of the pseudo-vlues. The vrice of the pseudo-vlues is deoted ˆ 2 T, d is otied with the formul: 2 2 (T T ) ˆ T N 1 (5) Tukey cojectured tht the T s could e cosidered s idepedet rdom vriles. Therefore the stdrd error 2 of the prmeter estimtes T, deoted, could e otied from the vrice of the pseudo-vlues from the ˆT usul formul for the stdrd error of the m s: ˆ T 2 ˆ T ( T T ) N N( N 1) 2 (6) This stdrd error c the e used to compute cofidece itervls for the estimtio of the prmeter. Uder the idepedece ssumptio, this estimtio is distriuted s studet's t distriutio with (N -1) degrees of freedom. Specificlly (1 - ) cofidece itervl c e computed s T t (7) ˆ, v T with t, v eig the α-level criticl vlue of Studet's t distriutio with v = N 1 degrees of freedom. Jckkife without pseudo-vlues Pseudo-vlues re importt to uderstd the ier workig of the jckkife, ut they re ot computtiolly efficiet. Altertive formuls usig oly the prtil estimtes c e used i lieu of the pseudo-vlues. Specificlly, if T deotes the me of the prtil estimtes d ˆ T their stdrd devitio, the T (cf. Equtio 4) c e computed s T = NT (N l) T (8) d ˆT (cf. Equtio 6) c e computed s N 1 ˆ 2 T ˆ T (T T ) (N 1) N N (9) Assumptios of the Jckkife Although the jckkife mkes o ssumptios out the shpe of the uderlyig proility distriutio, it requires tht the oservtios re idepedet of ech other. Techiclly, the oservtios re ssumed to e idepedet d ideticlly distriuted (i.e., i sttisticl jrgo, i.i.d. ). This mes tht the jckkife is ot i geerl pproprite tool for time series dt. Whe the idepedece ssumptio is violted the jckkife uderestimtes the vrice i the dt-set which mkes the dt look more relile. Becuse the jckkife elimites the is y sutrctio (which is lier opertio), it works correctly oly for sttistics which re lier fuctios or the prmeters or the dt, d whose distriutio is cotiuous or t lest smooth eough to e cosidered s such. I some cses, lierity c e chieved y trsformig the sttistics (e.g. usig Fisher Z-trsform for correltios, or logrithm trsform for stdrd devitios), ut some olier or o-cotiuous sttistics, such s the medi, will give very poor results with the jckkife o mtter wht trsformtio is used. Fisher developed trsformtio ow clled Fisher s Z trsformtio tht coverts Perso s r s to the ormlly distriuted vrile Z. The formul for the trsformtio is: Z 0.5 l(1 r) l(1 r)] (10) Give fisher trsformtio vlue (z), the Fisher iverse (r) is give y; 2z e 1 r 2z e 1 (11) Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

4 Ekezie et l 011 Bis estimtio The jckkife ws origilly developed y Queouille (1956) s oprmetric wy to estimte d reduce the is of estimtor of popultio prmeter. The is or estimtor is defied s the differece etwee the expected vlue of this estimtor d the true vlue of the popultio prmeter. So formlly, the is, deoted, of estimtio T of the prmeter is defied s = E{T} - (12) with E{T} eig the expected vlue of T. The jckkife estimte of the is is computed y replcig the expected vlue of the estimtor (i.e., E{T}) y the ised estimtor (i.e., T) d y replcig the prmeter (i.e., ) y the uised jckkife estimtor (i.e., T). Specificlly, the jckkife estimtor of the is, deoted B jck is computed s: jck = T - T. (13) Estimtio of regressio prmeters d is I this cse, the jckkife is used to estimte the itercept, the slope d the vlue of the coefficiet of correltio for the regressio. The procedure is to drop ech oservtio i tur d compute, for the slope d the itercept, the prtil estimtes (deoted - d - ) d pseudo-vlues (deoted d ). To compute the regressio equtio with the prtil estimtes of the slope d itercept, the formuls re give s; 1= X (14) This jckkife pseudo vlues for the th oservtio gives the followig: N (N 1) (15) N (N 1) (16) The jckkife estimtes of the regressio will give the followig equtio for the predictio of the depedet vrile (the predictio usig the jckkife estimtes is deoted ): = + X (17) Geerlizig the performce of predictive models The first techique preseted ove, estimtes popultio prmeters d their stdrd error. The secod techique evlutes the geerliztio performce of predictive models. I these models, predictor vriles re used to predict the vlues of depedet vrile(s). I this cotext, the prolem is to estimte the qulity of the predictio for ew oservtios. Techiclly spekig the gol is to estimte the performce of the predictive model s rdom effect model. The prolem of estimtig the rdom effect performce for predictive models is ecomig crucil prolem i domis such s, for exmple, io-iformtics d euroimgig (Kriegeskorte et l., 2009; Vul et l., 2009) ecuse the dt sets used i these domis re typiclly comprised of very lrge umer of vriles (ofte much lrger umer of vriles th oservtios A cofigurtio clled the smll N, lrge P prolem). This lrge umer of vriles mkes sttisticl models otoriously proe to over-fittig. Estimte of the geerliztio performce of the regressio I order to estimte the geerliztio performce of the regressio, we eed to evlute the performce of the model o ew dt. These dt re supposed to e rdomly selected from the sme popultio s the dt used to uild the model. The jckkife strtegy here is to predict ech oservtio s ew oservtio; this implies tht ech oservtio is predicted from its prtil estimtes of the predictio prmeter. Specificlly, if we deote y the jckkife predicted vlue of the th oservtio, the jckkife regressio equtio ecomes: jck, jck, X (18) I this cotext, the gol of the jckkife is to estimte how model would perform whe pplied to ew oservtio. This is doe y droppig i tur ech oservtio d fittig the model for the remiig set of oservtios. The model is the used to predict the left-out oservtio. With this procedure, ech oservtio hs ee predicted s ew oservtio. Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

5 It. J. Stt. Mth. 012 I some cses, jckkife c perform oth fuctios, therey geerlizig the predictive model s well s fidig the uised estimte of the prmeters of the model. We shll use these dt to illustrte how the jckkife c e used to (1) estimte the regressio prmeters d their is d (2) evlute the geerliztio performce of the regressio model. Dt Collectio Dt used i this pper were collected from Orji Tow primry school, Owerri North Imo Stte Nigeri. It cosists of height d weight of twety (20) rdomly selected primry five d six pupils. The respose vrile (Y ) is the weight mesured i kg, while the predictor vrile (X ) is the height mesured i cm. Dt Alysis As prelimiry step, the dt re lyzed y stdrd regressio lysis d we foud tht the regressio equtio is equl to: = X The predicted vlues re give i Tle 1. This regressio model correspods to coefficiet of correltio of r =.8586 (i.e., the correltio etwee the Y -s d the s is equl to ). Let us ow estimte the regressio prmeters d is for the jckkife. So, for exmple, whe we drop the first oservtio, we use the oservtio 2 through 20 to compute the regressio equtio with the prtil estimtes of the slope d itercept s (Equtio 2). From Equtio (14); = X 1 Tle 1: Estimtio of Jckkife Prmeters Os X Y jck, Usig Equtios (15) d (16), we oti for the first oservtio s; 1 20( ) 19( ) d Tle 2 shows the prtil estimtes d pseudo vlues for the itercept d slope of the regressio. The jckkife estimtes of the regressio equtio for the predictio of the depedet vrile usig Equtio (17) is; = X. Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

6 Ekezie et l 013 The predicted vlues usig the jckkife estimtes re give i Tle 1. It is worth otig, tht for regressio, the jckkife prmeters re lier fuctios of the stdrd estimtes. This implies tht the vlues of c e perfectly predicted from the vlues of. Specificlly, Therefore the correltio etwee the (predictio usig the jckkife estimtes) d the (the regressio equtio)is equl to oe, this, i tur, implies tht the correltio etwee the origil dt d the predicted vlues is the sme for oth d. The jckkife estimte is computed o these Z-trsformed vlues, d the fil vlue of the estimte of r is otied y usig the iverse of the Fisher Z-trsform (usig r rther th the trsformed Z vlues would led to gross over-estimtio of the correltio). Tle 2 produces the prtil estimtes for the correltio, the Z- trsformed vlues, d the Z-trsformed pseudo-vlues. From Tle 2, we fid tht the jckkife estimte of the Z-trsformed coefficiet of correltio is equl to Z = which, whe trsformed ck to correltio gives vlue of the jckkife estimte for the correltio of r = However, this vlue is very close to the vlue otied with other clssic ltertive popultio uised estimte clled the shruke r, which is deoted r ~, d computed s r 2 (N 1) 1 (1 r ) (N 2) ~ ( ) Tle 2: Prtil estimtes d pseudo-vlues for the regressio exmple of Height & Weight dt Oss. Prtil estimtes Pseudo-Vlues - - r - Z - Z Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

7 It. J. Stt. Mth. 014 Tle 2. Cot Me Z Jckkife Estimtes Z SD ˆ ˆ ˆZ ˆ ˆ ˆ Z SE Jckkife Stdrd Devitios ˆ ˆ ˆ ˆ ˆ Z ˆ Z N N N Jckkife Stdrd Error Cofidece itervls re computed usig Equtio (7). For istce, tkig ito ccout tht the =.05 criticl vlue for Studet's t distriutio for v = 19 degree of freedom is equl to t,v = 2.093, the cofidece itervl for the itercept is equl to: t ˆ, v The is of the estimte is computed from Equtio (13), the is of the estimtio of the coefficiet of correltio is equl to: jck (r) = r r = = The is is positive d this shows (s required) tht the coefficiet of correltio over-estimtes the mgitude of the popultio correltio. To predict ech oservtio s ew oservtio usig the Jckkife strtegy, for istce, the first oservtio is predicted from the regressio model uilt with oservtios 2 to 20, this gives the followig predictig equtio for (see Tles 1 d 2). Usig Equtio (18), jck,1 jck,1 = = The jckkife predicted vlues re reveled i Tle 1. The qulity of the predictio of these jckkife vlues c e evluted, oce gi, y computig coefficiet of correltio etwee the predicted vlues (i.e., the the ctul vlues (i.e., the Y ). jck, ) d CONCLUSION I this pper, the Jckkife delete-oe lgorithm hs ee used to estimte the regressio prmeters, d the is of the estimte, d it ws reveled tht the is result ws positive. For the regressio, the jckkife prmeters re lier fuctios of the stdrd estimtes, which implies tht the vlues of c e perfectly predicted from the vlues of. The jckkife predicted vlues d the cofidece itervl were clculted. Estimtio of ivrite lier regressio dt vi Jckkife lgorithm

8 Ekezie et l 015 REFERENCES Bissel TA. (1975). The jckkife-toy, tool or two-edged wepo? The Sttistici, 24: Kriegeskorte K, Simmos WK, Bellgow PSF, Bker CI (2009). Circulr lysis i Systems eurosciece: the dgers of doule dippig. Nture Neurosciece, 12: Hogchg H, Yuhe X (2013). Jckkifed Liu estimtor i lier regressio models. Wuh Ui. J. Nturl Sci., 18(4): Miller RG (1974). The jckkife: review. Biometrik, 61: 117. Queouille MH (1956). Notes o is i estimtio. Biometrik, 43: Sho J, Ro JNK (1993). Jckkife iferece for heteroscedstic lier regressio models. C. J.Stt., 21(4): Tukey JW (1958). Bis d cofidece i ot quite lrge smples (strct) Als of Mthemticl Sttistic, 29, 614. Tukey JW (1986). The future of processes of dt lysis i the Collected Works of Joh W. Tukey (Volume IV). New York: Wdsworth. pp Vul E, Hrris C, Wikielm P, Pshler H (2009). Puzzligly high correltios i FMRI studies of emotio, persolity, d socil cogitio. Perspectives i Psychologicl Sciece, 4: Wu CFJ (1986). Jckkife ootstrp d other re-smplig methods i regressio lysis. The Als of Sttistics, 14(4): Accepted 23 My, Cittio: Osuji GA, Ekezie DD, Opr J, Ogo CJ (2014). Estimtio of ivrite lier regressio dt vi Jckkife lgorithm. Itertiol Jourl of Sttistics d Mthemtics 1(1): Copyright: 2014 Ekezie et l. This is ope-ccess rticle distriuted uder the terms of the Cretive Commos Attriutio Licese, which permits urestricted use, distriutio, d reproductio i y medium, provided the origil uthor d source re cited. Estimtio of ivrite lier regressio dt vi Jckkife lgorithm