A Design Effect Measure for Calibration Weighting in Cluster Samples

Transcription

1 JSM 04 - Survey Research Methods Secton A Desgn Effect Measure for Calbraton Weghtng n Cluster Samples Kmberly Henry and Rchard Vallant Statstcs of Income, Internal Revenue Servce 77 K Street, E, Washngton, DC 000 Unverstes of Mchgan & Maryland, College Park, MD Abstract We propose a model-based extenson of weghtng desgn-effect measures for two-stage samplng when calbraton weghtng s used. Our proposed desgn effect measure captures the jont effects of a nonepsem samplng desgn, unequal weghts produced usng calbraton adjustments, and the strength of the assocaton between an analyss varable and the auxlares used n calbraton. he proposed measure s compared to exstng desgn effect measures n an example nvolvng household-type data. Key words: Auxlary data; Ksh weghtng desgn effect; Spencer desgn effect; generalzed regresson estmator. Introducton he most popular measure for gaugng the effect of dfferental weghtng on the precson of an estmator s Ksh s (965, 99) desgn-based desgn effect. Gabler et al. (999) showed that, for cluster samplng, ths estmator s a specal form of a desgn effect produced usng varances from random effects models, wth and wthout ntra-class correlatons. Spencer (000) proposed a smple model-based approach that depends on a sngle covarate to estmate the mpact on varance of usng varable weghts. However, these approaches do not provde a summary measure of the mpact of the gans n precson that may accrue from samplng wth varyng probabltes and usng a calbraton estmator lke the general regresson (GREG) estmator. Whle the Ksh desgn effects attempt to measure the mpact of varable weghts, as noted n Ksh (99), they are nformatve only under specal crcumstances, do not account for alternatve varables of nterest, and can ncorrectly measure the mpact of dfferental weghtng n some crcumstances. Spencer s approach holds for wth-replacement sngle-stage samplng for a very smple estmator of the total constructed wth nverse-probablty weghts wth no further adjustments. here are also few emprcal examples comparng these measures n the lterature. Henry and Vallant (03) extended ths to gauge the mpact of dfferental calbraton weghts n sngle-stage samples. In partcular, the exstng measures, revewed n Secton, may not accurately produce desgn effects for unequal weghtng nduced by calbraton adjustments. hese are often appled to reduce varances and correct for undercoverage and/or nonresponse n surveys (e.g., Särndal and Lundström 005; Kott 009). When the calbraton covarates are correlated wth the coverage/response mechansm, calbraton weghts can mprove the mean squared error (MSE) of an estmator. In many applcatons, snce calbraton nvolves element-level adjustments, calbraton weghts can vary more than the base weghts or categorybased nonresponse or poststratfcaton adjustments (Kalton and Cervantes-Flores 003; Brck and Montaqula 009). hus, an deal measure of the mpact of calbraton weghts ncorporates not only the correlaton between the survey varable of nterest y and the weghts, but also the correlaton between y and the calbraton covarates x to avod penalzng weghts for the mere sake that they vary. 564

2 JSM 04 - Survey Research Methods Secton We extend these exstng desgn effects to produce a new measure that summarzes the mpact of calbraton weght adjustments before and after they are appled to two-stage cluster-sample survey weghts. he proposed measure n Secton 3 accounts for the jont effect of a non-epsem sample desgn and unequal weght adjustments n the larger class of calbraton estmators. Our summary measure ncorporates the survey varable lke Spencer s model, but also uses a generalzed regresson varance to reflect multple calbraton covarates and the cluster sample desgn. In secton 4, we apply the estmators n a case study nvolvng household-type survey data and demonstrate emprcally how the proposed estmator outperforms the exstng methods n the presence of unequal calbraton weghts.. Exstng Methods In ths secton, we specfy notaton and summarze the exstng desgn effect measures for cluster samplng. he assumptons used to derve each of these are also presented... otaton We consder that a fnte populaton of M elements s parttoned nto clusters, wth szes M, and denoted by U, j :,,, j,, M. On element, j, an analyss varable y j s observed. he populaton total of the y and M j M y Y j, where y s s M y j j and the populaton varance s M M and Y M. We select a sample of n clusters m elements wthn cluster, usng two-stage samplng from U and obtan a set of s, j :,, n, j,, m respondents. he total sample sze of elements s m m. In the followng sectons, when a two-stage desgn s consdered, we assume that clusters are selected wth replacement. he selecton probablty of cluster on a sngle draw s p. Wthn cluster, the sample of m elements s selected va smple random samplng wthout replacement (srswor). Assumng that we have probablty-wth-replacement (pwr) samplng of clusters, the probablty of selecton for n clusters s approxmately p np (f p s not too large), where p s the one-draw selecton probablty. he second-stage selecton probablty s j m M for element j n cluster. hen the overall selecton probablty s approxmately j j npm M. We estmate the total from the sample usng wth the pwr-estmator ˆ n m M n m pwr, y y j j w j j y j np m, w M np m. where j n 565

3 JSM 04 - Survey Research Methods Secton.. GREG Weght Adjustments Case weghts resultng from calbraton on benchmark auxlary varables can be defned wth a global regresson model for the survey varables (see Kott 009 for a revew). Devlle and Särndal (99) proposed the calbraton approach that nvolves mnmzng a dstance functon between the base weghts and fnal weghts to obtan an optmal set of survey weghts. Specfyng alternatve calbraton dstance functons produces alternatve estmators. Suppose that a sngle-stage probablty sample of m elements s selected wth beng the selecton probablty of element and x a vector of p auxlares assocated wth element. A least squares dstance functon produces the general regresson estmator (GREG): ˆ ˆ ˆ B ˆ g y, () GREG Hy x Hx s where ˆHy s y s the Horvtz-hompson estmator of the populaton total of y, ˆHx s x s the vector of H estmated totals for the auxlary varables, x x s the correspondng vector of known totals, ˆ B A X V Π y s the regresson coeffcent, wth s s ss s s As XsVss Πs X s, X s s the matrx of x values n the sample, ss dag v varance matrx specfed under the workng model (defned below), and Π dag x Hx s expresson for the GREG estmator n (), g v case weghts are w g for each sample element. V s the dagonal of the s. In the second ˆ A x s the g-weght, such that the he GREG estmator for a total s model-unbased under the assocated workng model, y x β, ~ 0, v. he GREG s consstent and approxmately desgn-unbased when the sample sze s large (Devlle and Särndal 99). When the model s correct, the GREG estmator acheves effcency gans. If the model s ncorrect, then the effcency gans wll be dampened (or nonexstent) but the GREG estmator s stll approxmately desgn-unbased. Relevant to ths work, the varance of the GREG estmator can be used to approxmate the varance of any calbraton estmator (Devlle and Särndal 99) when the sample sze s large. hs result holds for multstage samplng and allows us to produce one desgn effect measure applcable to all estmators n the famly of calbraton estmators..3. Drect Desgn-Effect Measures For a gven non-epsem sample and estmator ˆ for the fnte populaton total, one defnton for the drect desgn effect (Ksh 965) s ˆ ˆ srswr ˆ srswr Deff Var Var. () where ˆsrswr ks k M y m s the expanson estmator under smple random samplng wth ˆsrswr replacement (srswr) and Var M y m. We refer to ths as a drect estmator because t uses theoretcal varances n the numerator and denomnator. he alternatves that are presented subsequently use varous approxmatons to the components n (). he desgn effect n () measures the sze of the 3 566

4 JSM 04 - Survey Research Methods Secton varance of the estmator ˆ under the desgn, relatve to the varance of the estmator of the same total f an srswr of the same sze had been used. We can approxmate the varance of any calbraton estmator ˆcal usng the approxmate varance of the GREG (Devlle and Särndal 99), n whch case Deff ˆ Var ˆ Var ˆ. (3) cal GREG cal srswr srswr As descrbed n Sec. 3, our proposed desgn effect s a model-based approxmaton to (3)..4. Ksh s Haphazard-Samplng Desgn-Effect for Unequal Sngle-stage Sample Weghts Ksh (965, 990) proposed the desgn effect due to weghtng as a measure to quantfy the loss of precson due to usng unequal and neffcent weghts. For,, w w w m denotng the weghts from a smple random sample wthout replacement (srswor) of m elements, ths measure s deff K w CV w, (4) m w s w s where s CV w m w w w s the coeffcent of varaton of the weghts, and w m w. Expresson (4) s derved from the rato of the varance of the weghted survey mean s under dsproportonate stratfed srswor to the varance under proportonate stratfed srswor when all stratum element varances are equal (Ksh 99). Wth equal stratum varances, samplng wth a proportonal allocaton to strata s optmal, whch leads to all elements havng the same weght..5. Ksh s Measure for Cluster Samplng Weghts Ksh (987) proposed a smlar measure for cluster samplng. Assume that there are G unque weghts n s such that the m g elements wthn each cluster have the same weght, denoted by w g w g for g,, G, m g s the number of elements wthn weghtng class g and m m g g s the total number of elements n the sample. We estmate the populaton mean Y M usng the weghted sample mean G G w g jg gj gj g jg gj y w y w. Ksh s (987) decomposton model for w assumes that the G weghtng classes are randomly ( haphazardly ) formed wth respect to y j, assumng that the y j have a common varance and that s s an equal-probablty sample n whch the varaton among the m g s wthn s s not sgnfcant. he resultng desgn effect s G y 4 567

5 JSM 04 - Survey Research Methods Secton G m w gm g g KC w c G w g gm g deff y m n, (5) where m n m s the average cluster sze and c the measure of ntra-cluster homogenety. he frst component n (5) s the cluster-sample equvalent of (4), and can be wrtten n a smlar form usng the squared CV of the weghts f mg. he second (5) component s the standard desgn effect due to cluster samplng (e.g., Ksh 965). Expresson (5) may not hold f there s varaton n the m g across clusters (Park 004) or moderate correlaton between the survey characterstc and weghts (Park and Lee 004)..6. Gabler et al. s Measure for Cluster Samplng Gabler et al. (999) used a model to justfy measure (5) that assumes y j s a realzaton from a one-way random effects model (.e., a one-way AOVA-type model wth only a random cluster-level ntercept term plus an error) that assumes the followng covarance structure:, j j Covyj, y j e, j j. (6) 0 Cov yj, y j for, j j and 0 otherwse. Gabler et al. (999) take the rato of the model-based varance of the weghted survey mean under a model wth covarance structure (6) to the varance under the uncorrelated errors verson and derve If the elements are uncorrelated, then (6) reduces to G m w gm g g G wm g g deff y m w g e g where where n G G g g g g g g g G m w gm g g UB deff yw mg e G wm g g g m w m w m, (7). hey also establshed an upper bound for (7): n G n G g g g g g g g m m w m w m, (8) s a weghted average of cluster szes. Park (004) further extends ths approach to three-stage samplng, assumng that a systematc samplng s used n the frst stage to select the clusters. Lynn and Gabler (005) provde examples of specal cases of (7), such as equal samplng/coverage/response rates across domans

6 JSM 04 - Survey Research Methods Secton Lynn and Gabler (005) rewrte 3 m g G G G g, g g g g g mvar g w g g mw g g g Cov m m w m m w n Gabler et al. (999) s desgn effect. Assumng certan restrctons on the weghts, they propose a survey plannng approxmaton, mg m CVm n CV w, where G G CV m G m m uses the squared CV of cluster sample szes across clusters, g g g G G and CV w m w m g g g g g g g w uses the squared CV of weghts across observatons..7. Park and Lee s Measures for Unequal Cluster Samplng Park and Lee (004) extend the Gabler et al. (999) desgn effect to account for unequal samplng weghts wthn a two-stage cluster sample that selects m elements from each PSU: where ˆ * Hy y deff m W SyB m S y SyB M S y S yb M Y Y * m0 Q Y CV y y, CV m y pm Y 0 varablty,, (9), S M M y Y y j j s the wthn-cluster s the across-cluster varablty among cluster means, M W CVy y Y, CVy Sy Y, and Y y j M. he term Q MM pm s equal to zero f p M M or the cluster probablty of selecton s proportonal to the cluster sze M. he equvalent of (9) for the weghted sample mean s * deffpl yw m0 Wd, (0) * Wd m0 CVy Q pm D Y CVd m0, CVd Sd D, where M j M D d M, and D d M for the transformed varable dj yj Y. j j.8. Spencer s Model-based Measure for Sngle-Stage pwr Samplng Spencer (000) derves a desgn-effect measure to more fully account for the effect on varances of weghts that are correlated wth the survey varable of nterest. hs measure was not developed for cluster samplng, but we used ths modelng approach n Sec. 3 for a proposed desgn effect. he sample s assumed to be selected wth varyng probabltes and wth replacement (pwr). Suppose that p s the one-draw probablty of selectng element, whch s correlated wth y and that a lnear model holds for 6 569

7 JSM 04 - Survey Research Methods Secton y : y p. A partcular case of ths would be p x, where x s a measure of sze assocated wth element. If the entre fnte populaton were avalable, then the ordnary least squares and, where, estmates of and are A Y BP B U y Y p P U p P Y P are the fnte populaton means for y and p. he fnte populaton varance of the resduals, e y A Bp e yp y Y yp y, where yp s the fnte, s U populaton correlaton between y and he estmated total studed by Spencer s, ˆpwr, y Var n p y p U p. he usual base weght under pwr-samplng s w np ˆpwr y s. w y, wth desgn-varance n sngle-stage samplng. Spencer substtuted the model-based values for y nto the pwr-estmator s varance and took ts rato to the varance of the estmated total usng srswr to produce the followng desgn effect for unequal weghtng (see Appendx n Spencer 000; modfed for our notaton): A mw mw m w ew e Amew Deff e w S yp. () M M y My My Assumng that the correlatons n the last two terms of () are neglgble, Spencer approxmates () W w n p wth the frst two terms n (), where U U s the average weght n the populaton. When yp s zero and y s large, measure () s approxmately equvalent to Ksh s measure (4). However, Spencer s method does ncorporate the survey varable y, unlke (4), and mplctly reflects the dependence of y on the selecton probabltes p. 3. Proposed Desgn Effect Measures Henry and Vallant s (03) approach n sngle-stage samplng can be extended to produce a new weghtng desgn effect measure for a calbraton estmator n cluster samplng. We produce the desgn effect n four steps: () constructng a lnear approxmaton to the GREG estmator; () obtanng the varance of ths lnear approxmaton; (3) substtutng our model-based components nto the GREG varance; and (4) takng the rato of the model-based varance to the varance of the pwr-estmator of the total under srswr. Frst, a lnearzaton approxmaton to the GREG estmator (Exp n Särndal et al. 99) assumng clusters are selected pwr s ˆ ˆ ˆ ˆ Bˆ GREG pwr, y x pwr, x ˆ pwr, y x pwr, x BU we s js j j x BU () 7 570

8 JSM 04 - Survey Research Methods Secton x s the known populaton total of x, ˆ pwr, x ej yj j u GREG x U w s j s je j where s the vector of pwr-estmators, B U s the populaton coeffcent, x B s the resdual for element, j, and x j s a row vector of calbraton covarates. From (), ˆ B. Under the two-stage samplng assumptons (Sec..), w M np m j and the approxmaton varance of the GREG n () s also the varance of ˆ GREG x BU M s j s ej npm, (3) wˆ where w s e np and ˆ M m e. he desgn-varance of (3) s where e js j ˆ eu M m M GREG U j U Var p E e e n p np m M M E U e j j e U M m E U S Ue n p np m M, eu e ju j, S M Ue M e j e U j, and M e e M U j j, (4). he true deff for ˆGREG s defned as deff ˆ Var ˆ Var ˆ. (5) rue GREG GREG srswr 3.. Vallant et al. s Relatve Varance Deff Vallant, Dever, and Kreuter (03, Sec. 9..3) gve a desgn effect for totals n cluster samples usng the rato of the relatve varance of the estmator of the total when clusters are selected pwr and m elements wthn clusters are selected va srswor over the equvalent expresson under srswr. her measure can be modfed by usng (4) whch uses the GREG-based resduals. Assumng that m elements are selected n each cluster, the relatve varance of the GREG (.e., the varance of the GREG over the squared total) can be rewrtten as RelVar ˆ ˆ GREG RelVar srswr ke e m, (6) RelVar ˆsrswr y nmy e Be Be We, Be p eu p EU s the where, We M SUe e e e e e y, where ej yj j U between-cluster relatve varance component, k B W RelVar B W s the wthn-cluster component, x B s the resdual for element 8 57

9 JSM 04 - Survey Research Methods Secton j, E U x j s a row vector of calbraton covarates (ncludng an ntercept f one s used), and e. Usng the form of the relatve varance n (6), the assocated desgn effect s U ju j ˆ deffvdk cal ke e m. (7) When calbraton s more effcent for a gven y -varable, the term k e wll be less than and acts as a dampenng factor, reducng the overall desgn effect. 3.. An Alternatve Deff We follow Spencer s approach and substtute model values n varance (4) to formulate a desgn-effect measure. However, here we substtute n the model-based equvalent to e, not y, as Spencer does. Suppose that the second-stage samplng fracton s neglgble,.e., m M 0. o match the theoretcal varance formulaton n (4), consder the model y A x B e, where A, B are the fnte populaton model parameters, and j j U j U j x from prevous sectons s equal to j j U x x. Smlar to Henry and Vallant (03), we smplfy thngs by reformulatng the model as uj yj x jbu, such that e j u j U and ncorporate UM not as a cluster-level (random) ntercept, smply an algebrac expresson. he model assumng a random ntercept and error term s equvalent to Gabler et al. s (999) random effects model. Substtutng the GREG resduals e j nto the varance (4) and takng ts rato to the varance of the pwr- estmator n srswr, Var ˆ M nm wll produce our approxmate desgn effect due to srs srswr y unequal calbraton weghtng. Assumng that the M M and the wthn-cluster samplng fractons are neglgble, we obtan the followng approxmate desgn effect: deff u nmw U U nm U G M y nm y M are large enough such that nmw u, w u u. (8) where u yj xjbu ju, M M uw u w nm M ws U M y M y m, U u, UM, M, S U M u u j j u, U u U 9 57

10 JSM 04 - Survey Research Methods Secton, u u U u u U between the terms wthn each subscrpt., w W W, and. he -components are the fnte populaton correlatons Some approxmatons to (8) exst. If the correlatons n (8) are neglgble, then we obtan mw u U U m U M ws m U deff u G0 M M, whch can m y nm y M y be estmated wth the correlaton terms removed from (8). Assumng that M are close enough such that M M, M M, and 0, then (8) becomes mw u U M U m U deffg M y y M y nm w M uw u Uuuw nm M ws Uu. (9) M m y M y he Ksh measure s also a specal case of (9), when there are no cluster-level effects. In partcular, suppose that for all elements x j 0, and there s no cluster samplng. hen, uj yj,, U M MY, u y, and 0, and (9) reduces to nmw M. he estmator of ths from deff w, defned n (4). a partcular sample s Ksh s measure, K When the relatonshp between the calbraton covarates x and y s strong, the varance u should be smaller than y. In ths case, measure (8) s smaller than the Ksh and Park and Lee estmates. Varable weghts produced from calbraton adjustments are thus not as penalzed (shown by overly hgh desgn effects) as they would be usng the Ksh. However, f we have neffectve calbraton, or a weak relatonshp between x and y, then u can be greater than y, producng a desgn effect greater than one. hs s llustrated n Sec. 4 wth a populaton that mmcs household-type data. We also examne the extent to whch the correlaton components n our proposed desgn effect (8) are sgnfcant, or large enough to nfluence the exact measure. u y 4. Emprcal Example Usng Household Data he MDarea.pop dataset n the R Pracools package (Vallant et al. 04) s used n ths secton as an example. hs dataset contans 403,997 persons generated from data from the 000 decennal U.S. Census for Anne Arundel County n the state of Maryland and s descrbed n detal n Vallant et al. (03). Indvdual values for each person were generated usng models. Groupngs to form the varables PSU and SSU were done after sortng the census fle by tract and block group wthn tract

11 JSM 04 - Survey Research Methods Secton We treated block groups wthn Census tracts as the cluster (PSU) and selected persons as the elements (SSU). PSUs wth a small number of persons (less than 500) were excluded, leavng a pseudo-populaton of 74 PSU s and 397,065 elements. A poststratfed estmator was used, wth poststrata defned by the cross-classfcaton of gender and 5 age groups (less than 5, 5-9, 0-4, 5-7, 8-, -4, 5-9, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59, 60-64, 65 and older). hus, the weghtng groups are post-strata, whch cut across the PSU s and SSU s. We examne sx varables of nterest, y, y, and y3, fcttous contnuous varables on the fle, created an addtonal varable y4 usng a lnear model wth the poststratfcaton varables as covarates, and two bnary varables ndcatng presence/absence of nsurance coverage and a hosptal stay. Fgure shows a parwse plot of the pseudo-populaton, ncludng plots of the varable values aganst each other n the lower left panels, hstograms on the dagonal panels, and the correlatons among the varables n the upper rght panels. hs plot mmcs household-type data patterns. Fgure. Pseudopopulaton Values and Loess Lnes for Desgn Effect Evaluaton Eght desgn effects are compared, wth results shown n able : he drect desgn effect measure deff rue computed from (5). hs reflects the combned effects of cluster samplng and poststratfcaton; he Ksh measure he Ksh measure Park and Lee s measure deff K (4) computed usng the GREG weghts deff KC (5) for cluster samplng; deff PL n (0); deff VDK n (7); 574

12 JSM 04 - Survey Research Methods Secton hree proposed measures: () deff G, the exact proposed desgn effect n (8), () deff G0, the zerocorrelaton approxmaton to deff G, and deff G, the equal-cluster szed approxmaton (9). All of these are meant to show the precson gans (f any) of the cluster samplng combned wth GREG estmaton. ote that we do not nclude the Gabler et al. (999) or Lynn and Gabler (005) measures, whch correspond to desgn effects of the weghted survey mean. Park and Lee (004) descrbe the crcumstances n whch desgn effects for means and totals are dfferent. Snce our proposed desgn effect s for estmaton of totals, we focus our emprcal comparson on comparng t to only the exstng measures for totals. he results are shown n able. able. Populaton Desgn Effects, by Varable of Interest Varable of Interest Desgn Effects y * y * y 3 * y 4 * Hosptal Stay** Insurance Coverage** deff rue Ksh Sngle-stage deff K Cluster deff Park and Lee Vallant et al. KC deff PL deff VDK Proposed Exact deff G Zero-corr. approx. deff G Equal cluster approx. deff G * contnuous varables; ** categorcal varables he true populaton desgn effects, denoted by deff rue, for the calbraton totals range from 0. for y 4 to 5. for y 3. hese are populaton values, not subject to samplng varablty. otably they are less than one for y and y 4, whch means that poststratfyng by age and gender wll mprove estmates of these varables totals. he Ksh sngle-stage K CV w, whch s constant regardless of the varable of nterest. he cluster sample extenson deff are all very large, 3.0 or hgher. he Park and Lee deff s smply KC 575

13 JSM 04 - Survey Research Methods Secton deff PL s account for the unequal cluster weghts but all exceed one. hs also occurred for the approxmatons, deff G0 and deff G. hs means for all of these alternatves, the deff s are too large. In partcular, one would ncorrectly beleve that calbraton would not be benefcal for y and y 4. he Vallant et al. deff VDK equals deff rue snce they are based on the same formula; the alternatve deff. estmator deff G s very close to rue Whle the Park and Lee deff PL was not desgned to capture the gans n calbraton, t could by ncorporatng the k e -factor used n deff VDK. he other components between these two deff s were nearly dentcal. hs s shown n able. able. Vallant et al. and Park and Lee Deff Components, by Varable of Interest Varable of Interest Desgn Effect Component y * y * y 3 * y 4 * Hosptal Insurance Stay** Coverage** Park and Lee deff PL * W y m Vallant et al. e deff VDK k e m e ke me * contnuous varables; ** categorcal varables he and e values are close, such that dfferences between these deff s are due to the k e -factor. Whle the Park and Lee deff PL was not desgned to account for calbraton effcency, able demonstrates how t can easly be adapted by ncorporatng k e. 5. Dscusson, Lmtatons, and Conclusons We propose new desgn effects that gauge the mpact of calbraton weghtng adjustments on an estmated total n cluster samplng. Exstng desgn effects nclude Ksh s (965) desgn effect due to weghtng and Park and Lee s (004) for unequal two-stage cluster weghts. ether of these reflect effcency gans due to calbraton. he Ksh deff s a reasonable measure f equal weghtng s optmal or nearly so, but does not reveal effcences that may accrue from samplng wth varyng probabltes. he Park and Lee deff does sgnal whether the H (or pwr) estmator n varyng probablty samplng s more effcent than srs, but does not reflect any gans from usng a calbraton estmator

14 JSM 04 - Survey Research Methods Secton he new desgn effects ntroduced n Sec. 3 measure the mpact of both samplng wth varyng probabltes and of usng a calbraton estmator, lke the GREG, that takes advantage of auxlary nformaton. As we demonstrate emprcally, the proposed desgn effects do not penalze unequal weghts when the relatonshp between the survey varable and calbraton covarate s strong. We also demonstrated emprcally that the correlaton components n the proposed measure deff G can be mportant n some stuatons. It s not overly dffcult to calculate these components, so we recommend ncorporatng them when possble to avod overly hgh estmates of the desgn effects. In cases where the auxlary nformaton s neffectve or s not used, the proposed measure approxmates Ksh s deff. One of our proposed measures, deff G, uses the model underlyng the general regresson estmator to extend the Spencer measure. he other alternatve, deff VDK, has a form smlar to the famlar m found n many texts. he survey varable, covarates, and weghts are requred to produce both new desgn effects. Snce the varance (4) s approxmately correct n large samples for all calbraton estmators, the new desgn effects should reflect the effects of many forms of commonly used weghtng adjustment methods, ncludng poststratfcaton, other forms of the GREG estmator, and rakng. Although desgn effects that do account for these adjustments can be computed drectly from estmated varances, t s mportant for practtoners to understand that the exstng deff s do not reflect any gans from those adjustments. he alternatve deff s ntroduced n ths paper, thus, serve as a correctve to that defcency. K References Brck, M.J., and Montaqula, J. M. (009). onresponse and weghtng n D. Pfeffermann and C. R. Rao (eds.), Handbook of Statstcs, Sample Surveys: Desgn, Methods and Applcatons, 9A. Amsterdam: Elsever BV. Devlle, J.-C. and Särndal, C. E. (99). Calbraton Estmators n Survey Samplng. Journal of the Amercan Statstcal Assocaton, 87, Gabler, S., Haeder, S., and Lahr, P. (999). A model-based justfcaton of Ksh s formula for desgn effects for weghtng and clusterng. Survey Methodology, 5, Henry, K., and Vallant, R. (03). A Desgn Effect Measure for Calbraton Weghts n Sngle-Stage Samples. Proceedngs of the Secton on Survey Research Methods, Amercan Statstcal Assocaton, to appear. Kalton, G., and Flores-Cervantes, I. (003). Weghtng Methods. Journal of Offcal Statstcs, 9: Ksh, L. (965). Survey Samplng, ew York: John Wley & Sons. Ksh, L. (987). Weghtng n Deft. he Survey Statstcan, June

15 JSM 04 - Survey Research Methods Secton Ksh, L. (990). Weghtng: Why, When, and How? Proceedngs of the Jont Statstcal Meetngs, Secton on Survey Research Methods, Amercan Statstcal Assocaton, -9. Ksh, L. (99). Weghtng for unequal P. Journal of Offcal Statstcs, 8, Kott, P. (009). Calbraton weghtng: combnng probablty samples and lnear predcton models, n D. Pfeffermann and C. R. Rao (eds.), Handbook of Statstcs, Sample Surveys: Inference and Analyss, 9B. Amsterdam: Elsever BV. Lynn, P. and Gabler, S. (005). Approxmatons to b* n the predcton of Desgn effects due to clusterng, Survey Methodology, 3, Park, I. (004). Assessng Complex Sample Desgns va Desgn Effect Decompostons. Proceedngs of the Secton on Survey Research Methods, Amercan Statstcal Assocaton, Park, I. and Lee, H. (004). Desgn effects for the weghted mean and total estmators under complex survey samplng. Survey Methodology, 30, Särndal, C. E., and Lundström, S. (005). Estmaton n Surveys wth onresponse, ew York: John Wley and Sons. Särndal, C.E., Swensson, B. and Wretman, J. (99). Model Asssted Survey Samplng, Berln: Sprnger. Spencer, B. D. (000). An Approxmate Desgn Effect for Unequal Weghtng When Measurements May Correlate Wth Selecton Probabltes. Survey Methodology, 6, Vallant, R., Dever, J. A., and Kreuter, F., (03). Practcal ools for Desgnng and Weghtng Survey Samples. ew York: Sprnger. Vallant, R., Dever, J.A., and Kreuter, F. (04). Pracools: ools for Desgnng and Weghtng Survey Samples. R package verson