Randomized Quantile Residuals

Transcription

1 Randmized Quantile Residuals Peter K. Dunn and Grdn K. Smyth Department f Mathematics, University f Queensland, Brisbane, Q 47, Australia. 4 April 996 Abstract In this paper we give a general definitin f residuals fr regressin mdels with independent respnses. Our definitin prduces residuals which are eactly nrmal, apart frm sampling variability in the estimated parameters, by inverting the fitted distributin functin fr each respnse value and finding the equivalent standard nrmal quantile. Our definitin includes sme randmizatin t achieve cntinuus residuals when the respnse variable is discrete. Quantile residuals are easily cmputed in cmputer packages such as SAS, S-Plus, GLIM r LispStat, and allw residual analyses t be carried ut in many cmmnly ccurring situatins in which the custmary definitins f residuals fail. Quantile residuals are applied in this paper t three eample data sets. Keywrds: deviance residual; epnential regressin; generalized linear mdel; lgistic regressin; nrmal prbability plt; Pearsn residual. Intrductin Residuals, and especially plts f residuals, play a central rle in the checking f statistical mdels. In nrmal linear regressin the residuals are nrmally distributed and can be standardized t have equal variances. In nn-nrmal regressin situatins, such as lgistic regressin r lg-linear analysis, the residuals, as usually defined, may be s far frm nrmality and frm having equal variances as t be f n practical use. A particular prblem ccurs when the respnse variable is discrete and takes n a small number f distinct values, as fr Pissn data with mean nt far frm zer r binmial data with mean clse t either zer r the number f trials. In such situatins the residuals lie n nearly parallel curves crrespnding t distinct respnse values, and these spurius This preprint is nw published as: Dunn, K. P., and Smyth, G. K. (996). Randmized quantile residuals. J. Cmput. Graph. Statist., 5,

2 curves distract the eye seriusly frm any meaningful message that might be cntained in a residual plt. In this paper we give a general definitin f residuals fr regressin mdels with independent respnses. Our definitin prduces residuals which are eactly nrmal, apart frm sampling variability in the estimated parameters, by inverting the fitted distributin functin at each respnse value and finding the equivalent standard nrmal quantile. This apprach is clsely related t that f C and Snell (968), but whereas C and Snell cncentrate n mean and variance crrectins we cncentrate n the transfrmatin t nrmality. Our definitin includes sme randmizatin t achieve cntinuus residuals when the respnse variable is discrete. Quantile residuals are easily cmputed in cmputer packages such as SAS, S-Plus, GLIM r LispStat, and allw residual analyses t be carried ut in many cmmnly ccurring situatins in which the custmary definitins f residuals fail. Special cases f quantile residuals have been used by Brillinger and Preisler (983) and Brillinger (996). Fr ther wrk n residuals fr nn-nrmal regressin mdels see Pierce and Schafer (986) r McCullagh and Nelder (989) and the references therein. In the discussin at the end f the paper we briefly indicate hw quantile residuals may be etended t mdels with dependent respnses. Pearsn and Deviance Residuals Let y,..., y n be respnses and fr each i let i be a vectr f cvariates. The y i are assumed t be independent and t fllw a distributin P(µ i, φ) where µ i = E(y i ) and φ is a parameter vectr cmmn t all the y i. The µ i are assumed t depend n the i and a vectr f regressin parameters β. We have particularly in mind generalized linear mdels (McCullagh and Nelder, 989) in which the prbability density r mass functin f y i has the frm f(y; θ i, φ) = a(y, φ) ep[{yθ i κ(θ i )}/φ] where a() and κ() are knwn functins and µ i = κ (θ i ). In this mdel we have var(y i ) = φv (µ i ) where V (µ i ) = κ (θ i ). It is custmary t assume that g(µ i ) = T β where g() is a knwn link functin. The parameter φ is the prprtinality cnstant in the meanvariance relatinship and is knwn as the dispersin parameter. In the cntet f generalized linear mdels, tw definitins f residuals have been cmmnly used in practice. The Pearsn residual is defined by r p,i = y i ˆµ i V (ˆµ i ) / where ˆµ i is the fitted value fr µ i. The Pearsn residual has the advantage that its mean and variance are eactly zer and φ respectively, if sampling variability in ˆµ i is small. The deviance residuals are defined in terms f the unit deviances. Fr the abve mdel, let t(y, µ) = yθ κ(θ). Assuming that y is in the dmain f µ, the unit deviance is d(y, µ) = {t(y, y) t(y, µ)}

3 The deviance residual is r d,i = d(y i, ˆµ i ) / sign(y i ˆµ i ) Pierce and Schafer (986) have argued n theretical grunds that the deviance residuals shuld be mre nearly nrmal than the Pearsn. Indeed bth cnverge t nrmality as φ relative t the µ i, the Pearsn residuals at rate O(φ / ) by the Central Limit Therem and the deviance residuals at O(φ) by the saddle-pint apprimatin t f(y; θ i, φ). The Pearsn and deviance residuals cincide and are eactly nrmal, ignring variability in ˆµ i, fr the nrmal linear mdel. The deviance residual is als eactly nrmal when the respnse is inverse-gaussian. In ther cases and fr φ/µ large hwever, neither type f residual can be guaranteed t be clsely nrmal, and the deviance residuals d nt generally have zer means r equal variances even at the true values µ i. 3 Randmized Quantile Residuals Let F (y; µ, φ) be the cumulative distributin functin f P(µ, φ). If F is cntinuus, then the F (y i ; µ i, φ) are unifrmly distributed n the unit interval. In this case, the quantile residuals are defined by r q,i = Φ {F (y i ; ˆµ i, ˆφ)} where Φ() is the cumulative distributin functin f the standard nrmal. Apart frm sampling variability in ˆµ i and ˆφ, the r q,i are eactly standard nrmal. This implies that the distributin f r q,i cnverges t standard nrmal if β and φ are cnsistently estimated. The abve definitin is a special case f C and Snell s (968) crude residuals. Eample: Leukemia data. Feigl and Zelen (965) discuss sme data relating the survival times y i f leukemia patients t their initial white bld cell cunts i and t eistence f AG-factr. Fllwing Feigl and Zelen, we treat the survival times as epnential, y i Ep(µ i ). We wrk with a lg-linear mdel fr the means, including separate intercepts fr the tw AG-factr grups, { α + β lg lg µ i = i AG psitive α + β lg i AG negative C and Snell (968) cnsidered a subset f this data, and defined apprimately epnential crude residuals R i = y i /ˆµ i, where the ˆµ i are the estimated means. In this case the quantile residuals r q,i = Φ { ep(y i /ˆµ i )} are a simple transfrmatin f the R i. A nrmal prbability plt f the quantile residuals cnfirms the assumptin f an epnential distributin. Figure plts the quantile residuals versus the cvariate. The three residuals (cases 7, 3 and 33) in the upper right-hand crner f the plt are relatively separate frm the bdy f the ther residuals, and withut them there appears t be a marked negative trend. While the pattern is nt sufficient t cntradict the mdel assumptins, it raises the pssibility that cases 7, 3 3

4 Figure : Plt f quantile residuals versus the cvariate fr the leukemia data. Circles represent patients which are AG-psitive, crsses AG-negative Quantile Residual Lg White Bld Cell Cunt and 33 may be utliers, r that the dispersin f the residuals increases at the largest white bld cell cunts. In any case, the three cases identified appear frm the residual plt t be jintly influential. Assigning the identified cases zer weight increases ˆβ nearly three-fld, frm -.3 t -.84 cmpared with a standard errr f.4. If F is nt cntinuus, a mre general definitin f quantile residuals is required. Let a i = lim y yi F (y; ˆµ i, ˆφ) and b i = F (y i ; ˆµ i, ˆφ). We define the randmized quantile residual fr y i by r q,i = Φ (u i ) where u i is a unifrm randm variable n the interval (a i, b i ]. Again, the r q,i are eactly standard nrmal, apart frm sampling variability in ˆµ i and ˆφ. The randmizatin strategy emplyed here is similar t the strategy f jittering (Chambers et al, 983) t prevent masses f verlapping pints in plts. Whereas jittering applies a unifrm randm cmpnent t the respnse, ur unifrm randm cmpnent is n the cumulative prbability scale and is tailred t the actual prbability mass at the pint in questin. Our randmizatin is the minimum necessary s that n granularity remains in the resulting residual distributin. Eample : Simulated binmial data. A lgistic linear regressin was used t mdel 6 binmial bservatins with binmial denminatr n = 3, i.e., the respnses were assumed t be independently distributed as y i bin(n, p i ), with n = 3 and lgit(p i ) = β + β i were i is a cvariate. The first plt f Figure displays the deviance residuals versus 4

5 Figure : Deviance and quantile residuals versus the cvariate frm a lgistic regressin. The respnse is simulated bin(3, p) with lgit p depending quadratically n the cvariate. 3 3 Deviance Residuals - - Quantile Residuals Cvariate Cvariate the cvariate. The pints in this plt lie n fur parallel curves crrespnding t the fur pssible values fr the respnse. The curves make it difficult t see any ther pattern in the data. The secnd plt displays the quantile residuals versus the cvariate. In this plt is clear that the residuals fllw a quadratic pattern. The data fr this eample was in fact cmputer generated with lgit(p i ) depending quadratically n the i. Figure 3 shws the residual plts nce the quadratic term has been included in the regressin. The deviance residuals lie n prminent curves while the quantile residuals nw shw randm scatter. Eample 3: Fathers and sns ccupatins. Brwn (974) and Ktze and Hawkins (984) analyze a sparse 4 4 cntingency table shwing the crss-classificatin f ccupatins f fathers (rws) by ccupatins f sns (clumns). The data was riginally published by Pearsn (94) and appears als in Hand et al (994). Brwn, Ktze and Hawkins were interested in identifying thse cells which are utliers relative t the independence mdel. We take a similar apprach, with the difference that the quantile residual apprach allws us t lk fr utliers relative t a mre realistic mdel. Observing that there is an apriri epectatin that sns will be influenced by their father s ccupatin, we fit a lg-linear Pissn regressin mdel t the cunts with rw and clumn effects and with an effect fr equality f father s and sn s ccupatin, i.e., y ij Pis(µ ij ), with lg µ ij = µ + α i + β j + δ ij () and ij = if i = j and therwise. Figure 4 is a nrmal prbability plt f quantile residuals frm this mdel. The largest psitive residual crrespnds t the (,) cell: sns almst always cntinue t wrk in the Arts if their father did. Figure 4 shws evidence f large negative residuals as well as large psitive residuals. Althugh nne f the negative residuals are individually significant, and the actual cntingency table cells represented in the left tail f the prbability plt varies with each realizatin f the quantile residuals, the 5

6 Figure 3: Deviance and quantile residuals versus the cvariate fr a well fitting lgistic regressin. Deviance Residuals Quantile Residuals Cvariate Cvariate Figure 4: Nrmal prbability plt with identity line f the quantile residuals frm the fathers and sns ccupatin data. Sample Deviates Nrmal Deviates 6

7 verall pattern is preserved acrss realizatins. The quantile residual plt shws in this way that there are t many small cunts in the cntingency table t be cmpatible with the abve mdel. N ther methd which has been applied t this data in the literature is able t shw this aspect f the data. Althugh Figure 4 shws clear evidence f lack f fit, the mdel () and the mdels which arise frm it by deleting selected cells des give an appreciably better fit t this data than the independence mdels cnsidered by earlier authrs. 4 Discussin and Etensins In this paper quantile residuals are cmputed by finding the equivalent standard nrmal deviate fr each respnse bservatin. In principle, any reference distributin culd have been chsen fr the residuals. C and Snell (968) fr eample cmputed epnential residuals fr data f Eample and Brillinger and Preisler (983) and Brillinger (996) use unifrm residuals. Hwever asymmetry seems an unnecessary cmplicatin, and bunded distributins intrduce a spurius pattern (the bundary itself) and make it difficult t distinguish between large residuals and utright utliers. The nrmal distributin is recmmended in this paper n the basis that nrmal variatin is that which mst peple have practice interpreting graphically. Randmizatin is used t prduce cntinuusly distributed residuals when the respnse is discrete r has a discrete cmpnent. This means that the quantile residuals will vary frm ne realizatin t anther fr a given data set and fitted mdel. Fr the sake f brevity, we have given nly ne realizatin f the quantile residuals fr each eample in this paper. In practice thugh we have fund it useful t rutinely plt fur realizatins f the quantile residuals. Any pattern in the residuals which is nt cnsistent acrss the realizatins is then ignred. The idea f applying a cntinuus randm cmpnent t discrete respnses s that methds fr cntinuus variables can be applied is in fact very ld. See Pearsn (95) fr a discussin. As used in this paper, randmizatin is a device thrugh which the aggregate pattern f the residuals becmes apparent. Since decisins d nt depend n individual realizatins, the bvius bjectins t randmizatin which arise in the cntet f tests and cnfidence intervals d nt seem t apply. Quantile residuals can generalize any f the usual diagnstic methds which use residuals. Fr eample, an added variable plt (Ck and Weisberg, 98) culd be cmputed fr a generalized linear mdel by pltting the quantile residuals, fr the mdel ecluding, versus a, where a is adjusted fr the ther cvariates in the mdel. The vectr a wuld be chsen t be rthgnal t the ther cvariates, relative t the cvariance matri f the y i. It might be cmputed as the residuals frm weighted least squares regressin f n the ther cvariates, using as weights the wrking weights frm the generalized linear mdel. Independence f the respnse bservatins was assumed in this paper. The methd f quantile residuals can be etended t dependent data situatins by epressing the multivariate likelihd as a sum f univariate cnditinal likelihds. Fr eample we might define the ith cnditinal quantile residual frm the cnditinal distributin f y i 7

8 given y,..., y i instead f frm the marginal distributin f y i as in the paper. This wuld prvide independent, standard nrmal residuals. Finally we cnsider the sampling variability f the ˆµ i, which has fr simplicity been ignred thrughut this paper. Treating the ˆµ i as fied is apprpriate when gd infrmatin is available n the mdel parameters, but may be unrealistic fr eample fr designed eperiments in which the number f parameters is nt small cmpared t the number f bservatins. In nrmal linear mdels, REML estimatin f the variance structure is btained frm the marginal distributin f any set f zer mean cntrasts, Z T y say. In a similar way, independent and identically distributed residuals culd be btained by transfrming frm the y i t any rthnrmal set f zer mean cnstrasts. Etending this idea t nn-nrmal regressin is mre difficult, but culd in principle be dne using the cnditinal apprach f Smyth and Verbyla (995). In that paper, Smyth and Verbyla argue that REML estimatin fr generalized linear mdels shuld prceed by cnsidering the cnditinal distributin f the y i given ˆβ. Independent quantile residuals culd therefre be defined by cnsidering the cnditinal distributin f each y i given y,..., y i and ˆβ. Fr certain values f i this distributin wuld be degenerate; these values culd be ignred withut lss f infrmatin. References Brillinger, D. R. and Preisler, H. K. (983). Maimum likelihd estimatin in a latent variable prblem. In S. Karlin, T. Amemiya and L. A. Gdman, eds., Studies in Ecnmetrics, Time Series and Multivariate Statistics, Academic, New Yrk, pp Brillinger, D. R. (996). An analysis f an rdinal-valued time series. In P. M. Rbinsn and M. Rsenblatt, eds., Papers in Time Series Analysis: A Memrial Vlume t Edward J Hannan. Athens Cnference Vlume. Lecture Ntes in Statistics, Vlume 5, Springer, New Yrk. T appear. Brwn, M. B. (974). Identificatin f the surces f significance in tw-way cntingency tables. Appl. Statist., 3, Chambers, J. M., Cleveland, W. S., Kleiner, B., and Tukey, P. A. (983). Methds fr Data Analysis, Wadswrth, Belmnt, Califrnia. Graphical Ck, R. D. and Weisberg, S. (98). Residuals and Influence in Regressin. Chapman and Hall, New Yrk. C, D. R. and Snell, E. J. (968). A general definitin f residuals (with discussin). J. R. Statist. Sc., B, 3, Feigl, P. and Zelen, M. (965). Estimatin f epnential survival prbabilities with cncmitant bservatin. Bimetrics,,

9 Hand, D. J., Daly, F., Lunn, A. D., McCnway, K. J. and Ostrwski, E. (994). Handbk f Small Data Sets. Chapman & Hall, Lndn. Ktze, T. J. v W. and Hawkins, D. M. (984). The identificatin f utliers in tw-way cntingency tables using subtables. Appl. Statist., 33, 5 3. McCullagh, P. and Nelder, J.A. (989). Generalized linear mdels, nd ed. Chapman and Hall: Lndn. Pearsn, E. S. (95). On questins raised by the cmbinatin f tests based n discntinuus distributins. Bimetrika, 37, Pearsn, K. (94). On the thery f cntingency and its relatin t assciatin and nrmal crrelatin. Reprinted in 948 in Karl Pearsn s Early Statistical Papers, Cambridge University Press, Cambridge, Pierce, D. A., and Schafer, D. W. (986). Residuals in generalized linear mdels. J. Amer. Statist. Ass., 8, Smyth, G. K. and Verbyla, A. P. (996). A cnditinal apprach t REML in generalized linear mdels. J. Ry. Statist. Sc. B, 58,