Natalie Shlomo, Chris J. Skinner and Barry Schouten Estimation of an indicator of the representativeness of survey response

Transcription

1 Natale Shlomo, Chr J. Sknner and Barry Schouten Etmaton of an ndcator of the repreentatvene of urvey repone Artcle (Accepted veron) (Refereed) Orgnal ctaton: Shlomo, Natale and Sknner, Chr J. and Schouten, Barry (0) Etmaton of an ndcator of the repreentatvene of urvey repone. Journal of tattcal plannng and nference, 4 (). pp. 0-. ISSN DOI: 0.06/j.jp Elever Th veron avalable at: Avalable n LSE Reearch Onlne: November 0 LSE ha developed LSE Reearch Onlne o that uer may acce reearch output of the School. Copyrght and Moral Rght for the paper on th te are retaned by the ndvdual author and/or other copyrght owner. er may download and/or prnt one copy of any artcle() n LSE Reearch Onlne to facltate ther prvate tudy or for non-commercal reearch. You may not engage n further dtrbuton of the materal or ue t for any proft-makng actvte or any commercal gan. You may freely dtrbute the RL ( of the LSE Reearch Onlne webte. Th document the author fnal manucrpt accepted veron of the journal artcle, ncorporatng any revon agreed durng the peer revew proce. Some dfference between th veron and the publhed veron may reman. You are adved to conult the publher veron f you wh to cte from t.

2 Etmaton of an Indcator of the Repreentatvene of Survey Repone Natale Shlomo, Chr Sknner, Barry Schouten Southampton Stattcal Scence Reearch Inttute, nverty of Southampton, Southampton SO7 BJ, nted Kngdom Stattc Netherland, Henr Faadreef 3, 49 JP Den Haag, The Netherland Abtract Nonrepone a major ource of etmaton error n ample urvey. The repone rate wdely ued to meaure urvey qualty aocated wth nonrepone, but nadequate a an ndcator becaue of t lmted relaton wth nonrepone ba. Schouten, Cobben and Bethlehem (009) propoed an alternatve ndcator, whch they refer to a an ndcator of repreentatvene or R-ndcator. Th ndcator meaure the varablty of the probablte of repone for unt n the populaton. Th paper develop method for the etmaton of th R-ndcator aumng that value of a et of auxlary varable are oberved for both repondent and nonrepondent. We propoe ba adjutment to the pont etmator propoed by Schouten et al. (009) and demontrate the effectvene of th adjutment n a mulaton tudy where t hown that the method vald, epecally for maller ample ze. We alo propoe lnearzaton varance etmator whch avod the need for computer-ntenve replcaton method and how good coverage n the mulaton tudy even when model are not fully pecfed. The ue of the propoed procedure alo llutrated n an applcaton to two bune urvey at Stattc Netherland. Key word: nonrepone; qualty; repreentatve; repone propenty; ample urvey.

3 . Introducton One of the mot mportant ource of etmaton error n urvey nonrepone. Survey organaton need ndcator of uch error for a varety of purpoe, for example to compare dfferent urvey, to montor change n a repeated urvey over tme or to montor change durng the feldwork of a ngle urvey, perhap to nform decon uch a when to end feldwork. An ndcator whch wdely ued for uch purpoe the repone rate, where a hgher repone rate taken to ndcate hgher qualty. However, there ha been much recent emprcal reearch (ee e.g. Grove (006), Grove and Peytcheva (008), Heerwegh, et al. (007) and reference theren) whch conclude that the repone rate nuffcent a an ndcator to meaure the potental error arng from nonrepone. Snce ample ze are uually large n urvey, the quared ba component of mean quared error wll typcally domnate the varance component and hence t derable that the ndcator reflect nonrepone ba. However, the emprcal evdence ugget that the repone rate only a weak predctor of nonrepone ba. There therefore much nteret n urvey organaton n the development of alternatve ndcator (Grove et al., 008). In th paper, we conder an ndcator propoed by Schouten, Cobben and Bethlemem (009, referred to hereafter a SCB). The bac dea that nonrepone ba depend crtcally on the contrat between the charactertc of repondent and nonrepondent. Th contrat can be aeed n term of the probablty of a unt repondng to the urvey. If all unt n the populaton hare the ame probablty of repondng then no nonrepone ba wll reult and the repone mechanm may be vewed a repreentatve. The ndcator propoed by SCB, termed the R-ndcator ( R for repreentatvene), meaure the extent to whch the repone probablte vary. An advantage of th ndcator (hared by the repone rate) for varou practcal applcaton that t provde a ngle meaure for the whole urvey. It hould be recognzed that nonrepone ba defned n relaton to a pecfc populaton parameter (and hence one or more urvey varable). Thu, for any one (multpurpoe) urvey there may be a very large number of nonrepone bae. It would be

4 feable to contruct ndcator whch are parameter-pecfc (Grove et al., 008, Wagner, 008), but here we uppoe the requrement for a ngle ndcator for the whole urvey. Further dcuon of the ratonale and applcaton of the R-ndcator provded by Cobben and Schouten (007) and Schouten and Cobben (007) n addton to the paper by SCB. The purpoe of th paper to conder n more detal ome of the etmaton ue aocated wth the R-ndcator. The R-ndcator propoed by SCB ubject to ba arng from the etmaton of the repone propente. The ba partcularly problematc for mall ample ze, and a ba adjutment developed. In addton, we develop lnearzaton varance etmator a an alternatve to the method of boottrappng propoed n SCB. We evaluate thee procedure n a mulaton tudy and demontrate the applcaton of thee procedure to a real bune urvey. We ntroduce the theoretcal framework and defne repone propente n Secton. The R-ndcator defned at the populaton level n Secton 3. The relaton of the R-ndcator to non-repone ba dcued n Secton 4. Pont etmaton of the R-ndcator ung ample data condered n Secton 5. The ba of the pont etmator and ba adjutment, varance etmaton and confdence nterval are condered n Secton 6. A mulaton tudy and reult of that tudy are decrbed n Secton 7 and reult from a real dataet are demontrated n Secton 8. Fnally, we conclude and dcu future work n Secton 9.. Prelmnare and Repone Propente We uppoe that a ample urvey undertaken, where a ample elected from a fnte populaton. The unt n are labelled =,, K, N, wth the ze of and denoted n and N, repectvely. A probablty amplng degn employed, where elected wth probablty p( ). The frt order ncluon probablty of unt denoted π and = d π the degn weght. 3

5 The urvey ubject to unt nonrepone, wth the et of repondng unt denoted r, o r. We denote ummaton over the repondent, ample and populaton by Σ r, Σ and Σ, repectvely. Let R be the repone ndcator varable o that R = f unt repond and R = 0, otherwe. Hence, r = { ; R = }. Let X be a vector of auxlary varable whch may nfluence nonrepone, e.g. a vector contng of age, gender, degree of urbanzaton of redence area and the oberved tatu of the dwellng for a houehold urvey or the type of bune and the number of employee for a bune urvey. We defne the repone propenty a the condtonal expectaton (under a model) of R gven the value of pecfed varable, uch a the auxlary varable above, and urvey condton (Lttle, 986, 988). If t neceary to clarfy that the condtonng on a vector of varable X, for example, then we wrte: = ( x ) = Er ( R X X = x ) to denote the condtonal expectaton of R gven that the vector of varable X for unt take the value x. Here E r (.) denote expectaton wth repect to the model underlyng the repone mechanm. We aume that R defned for each populaton unt, o that nonrepone what Rubn (pp. 30-3, 987) refer to a table, and alo defned for all. We alo aume that the R for dfferent unt are ndependent, condtonal on the pecfed varable and urvey condton. We hall further aume that the amplng degn and the nonrepone proce are unconfounded (pp , 987) o that the probablty of electng reman p( ), whatever the value of the R,. Thu, t aumed that nonrepone doe not depend on the confguraton of the ample. 3. Repreentatvene Indcator 4

6 The varaton n the repone propente may be vewed a reflectng the repreentatvene of the nonrepone. In SCB repone defned to be (trongly) repreentatve f the repone propente are the ame for all unt n the populaton, correpondng to the noton of mng completely at random (MCAR) (Lttle and Rubn, p., 00) gven the varable whch are condtoned upon when defnng. They defne a repreentatvene ndcator, termed the R-ndcator and denoted R, n term of the populaton tandard devaton of the repone propente: S = ( N ) ( ), where = / N. In order to facltate the nterpretaton of the ndcator, they defne t n term of S a follow: R = S, (3.) where th tranformaton of S enure that 0 nce t may be hown that R S ( ) 0.5. The value R = ndcate the mot repreentatve repone, where the dplay no varaton, and the value 0 ndcate the leat repreentatve repone, where the dplay maxmum varaton. 4. Relaton of R-Indcator to Non-repone Ba The R-ndcator may alo be motvated n term of nonrepone ba. Suppoe that the target of nference a populaton mean θ = N y of a urvey varable, takng value y for unt and oberved only for r. A tandard degn-weghted etmator of θ ˆ d R y / d R θ =. The ba of ˆ θ a an etmator of θ may be evaluated by takng expectaton wth repect to both the random amplng mechanm, denoted E, and the repone mechanm, denoted E r a n Secton. 5

7 We aume, for now, that the pecfed varable nclude y o that t may be treated a fxed. We then have: ( ˆ Er E θ ) = Er E dr y / dr y /, (4.) where n th cae = y, x ) condtonal on both Y and X and the YX ( approxmaton for large ample o that we take the frt term only n the Taylor expanon. In addton, we have ued the aumpton that the amplng and repone mechanm are unconfounded. Hence the ba depend on nonrepone only va. It follow that Ba( ˆ θ ) ( y θ ) / = corr S S /, (4.) y y where corr N y S S and y = ( ) ( )( θ) / y S = ( N ) ( y θ ). y Expreon (4.) alo obtaned n Bethlehem (988) and Särndal and Lundtröm (p.9, 005). An upper bound for the abolute ba can thu be expreed n term of the R-ndcator by ˆ ( R ) S y Ba( θ ) SS y / =. (4.3) A tandardzed meaure, whch free of y gven by: ( R ) B =. (4.4) 5. Etmaton of R-ndcator We uppoe that the data avalable for etmaton purpoe cont frt of the value { y ; r} of the urvey varable (or, more generally, a vector of urvey 6

8 varable), oberved only for repondent. Secondly, we uppoe that nformaton avalable on the value x (,,,,,, ) T = x x K xk of a vector X of auxlary varable for all ample unt,.e. for both repondent and non-repondent. We refer to th a ample-baed auxlary nformaton. Th a key aumpton and natural f, for example, the varable makng up x are avalable on a regter. Other poble aumpton about the avalablty of auxlary nformaton are dcued n Secton 9. Snce y only oberved for repondent, the repone propenty condtonal on y generally netmable wthout further aumpton. Intead, we propoe to take n the defnton of R n (3.) a condtonal on x,.e. to et = ( x ) = E ( R X = x ). X r Nonrepone mng at random, denoted MAR (Lttle and Rubn, p., 00), f R condtonally ndependent of y gven x. In th cae, we have Er ( R y, x ) = Er ( R x ) and = X( x ) = Y X( y, x ) and o y may mplctly be ncluded n the condtonng et. Hence the argument ued to obtan the ba bound n (4.3) tll apple f MAR hold. The ba bound and the R-ndcator telf may, however, be too conervatve. If MAR hold then ( y ) = E [ ( x X ) y ] and: Y r var( ) = var[ ( x )] = var{ E[ ( x ) y ]} + E{var[ ( x ) y ]} X X X = var[ ( y )] + E{var[ ( x X ) y ]} (5.) Y The frt term on the rght hand de of (5.) repreent the varaton of the condtonal probablte y ), whch we hould deally lke to ue n the R-ndcator Y ( f we are only concerned about nonrepone ba for a target parameter defned n term of y alone. The econd term repreent addtonal varaton whch unrelated to nonrepone ba for uch target parameter and may be vewed a redundant 7

9 varablty,.e. noe, n the for that purpoe. Kreuter et al. (00) preent example of auxlary varable whch are predctve of nonrepone but only weakly predctve of relevant y varable. In uch cae, the econd term n (5.) may be relatvely large and the R-ndcator and t aocated ba bound may be vewed a too conervatve. It therefore derable to elect only thoe auxlary varable whch are reaonably predctve of at leat one of the y varable of key nteret n the urvey. One pecal cae occur when nonrepone mng completely at random (MCAR) o that t ndependent of both x and y. In th cae, both X ( x ) and ( y ) are contant o that both term on the rght hand de of (5.) are zero. Y Hence, there no varablty n the and th doe, albet n a degenerate way, capture the fact that there nothng n the nonrepone proce that wll lead to nonrepone ba for etmaton related to y. If nonrepone not MAR then (5.) no longer hold. Intead, = X ( x ) wll repreent a moothed veron of X ( y, x ) and t not necearly the cae that Y var( ) wll be at leat a large a var[ Y ( y )]. Thu, we may fal to capture relevant feature of the nonrepone proce n the. In partcular, f R condtonally ndependent of x gven y then var[ ( y )] wll necearly be at leat a large Y a var( ),.e. var[ X ( x )] (followng a parallel argument to the MAR cae). It may be argued therefore that t derable to elect the auxlary varable conttutng x n uch a way that the MAR aumpton hold a cloely a poble. In any cae, t mut be emphazed that our defnton of = X ( x ) relate to a pecfc choce of auxlary varable x. A dfferent choce would generally reult n a dfferent. 8

10 We noted n Secton that we defne the repone propenty condtonal on the urvey condton that apply when the data are collected. We do not make th condtonng explct n our notaton, but t crucal to recognze th condtonng nce, a we noted n Secton, one of the objectve of contructng R-ndcator to be able to compare the repreentatvene of dfferent urvey and uch comparon become challengng when the defnton of the repone propenty for any one urvey dependent on the condton wth whch that urvey ha been mplemented, for example upon the mode of data collecton, the choce of ntervewer, the way thee ntervewer were traned and work and the contact trategy. Even for a ngle urvey repeated at dfferent pont n tme, uch condton may well not reman contant. 5. Nonrepone model In order to etmate the R-ndcator, we frt etmate the repone propente, = E( R x ). To do th, we aume that depend on x n a parametrc way va: g( ) = x ' β, (5.) where g(.) a pecfed lnk functon, β a vector of unknown parameter and x may nvolve the tranformaton of the orgnal auxlary varable for the purpoe of model pecfcaton. In partcular, we hall conder the logt lnk functon g( ) = log[ /( )] leadng to the logtc regreon model. We propoe to etmate β by peudo maxmum lkelhood (Sknner, pp , 989).e. β etmated by ˆβ, whch olve: 9

11 [ g ( x ' β)] x = 0 (5.3) d R where g (.) the nvere of the lnk functon. One reaon for ung the degn weght here becaue the objectve to etmate an R-ndcator whch provde a decrptve meaure for the populaton. The repone propenty then etmated by: ˆ = g ( x ' β ). (5.4) ˆ 5. Etmaton of R-ndcator A n SCB, we propoe to etmate R by: Rˆ = Sˆ, (5.5) where Sˆ ( ) ( ˆ ˆ = N d ), ˆ defned n (5.4), ˆ = ( ˆ ) / N and d N may be replaced by the SCB etmator. d f t unknown. We refer to the etmator n (5.5) a 6. Ba and Confdence Interval 6. Ba and Ba Adjutment We now conder the ba properte of the SCB etmator ˆR defned n (5.5). We hall aume that the vector of auxlary varable x gven o that no ba can are from pecfyng the wrong et of auxlary varable. We note, neverthele, that the choce of auxlary varable a crtcal decon n practce and we hall llutrate emprcally n Secton 7 how the R-ndcator can depend on th choce. holdng the We frt conder defnng the ba wth repect to the amplng mechanm, R fxed. nder th ource of random varaton, the peudo maxmum 0

12 lkelhood etmate ˆβ approxmately unbaed for the cenu parameter β whch olve: [ ( x ' β)] x = 0 (6.) R g (Sknner, p. 8, 003). The approxmaton here wth repect to an aymptotc framework, wth a equence of ample and populaton wth n and N ncreang. Th cenu parameter mple a correpondng repone propenty = g ( x ' β ) and R-ndcator R, defned n term of thee propente. We then have E ( ˆ R ) R. The dfference R R may be vewed a the ba arng from model mpecfcaton. Intead of defnng the ba wth repect to jut amplng varaton we could alo conder the repone mechanm. In a parallel way, we may wrte E ( ˆ r E R ) R 0, where R 0 the R-ndcator defned n term of the repone propente = g ( x ' β ) and β 0 the oluton of: 0 0 x β x. (6.) [ 0 g ( ' )] = 0 where 0 = Er ( R x ) the true repone propenty gven x and we uppoe that g( 0) not necearly lnear n x, a n (5.),.e. the latter model may be mpecfed. Thu, R 0 R may be vewed a the ba (wth repect to both amplng varaton and the repone mechanm) arng from model mpecfcaton. We may expect that R R = O N o that there wll uually be neglgble p ( ) dfference n practce between the two meaure R R or R 0 R of ba. In prncple, one mght conder way of aeng ether of thee meaure of ba, perhap by comparng the reult of ung the parametrc model n (5.) wth

13 thoe for ome knd of non-parametrc regreon. We do not purue th approach further here, however. Intead we conder the fnte ample ba E( Rˆ ) R, treatng R a the parameter of nteret, whch equvalent to aumng that the nonrepone model n (5.) correctly pecfed. We mght antcpate that the fnte ample ba of ˆR wll be non-neglgble, nce ˆR defned va the varance of the ˆ and we mght expect amplng varaton n thee quantte to nflate th varance. We approxmate th fnte ample ba of ˆR by frt conderng the ba of Ŝ defned below (5.5). We ue the decompoton: ˆ ˆ = ( ˆ ) + ( ) + ( ) + ( ˆ ), where = N d and ue the approxmaton E ( ˆ ) to obtan E ( ˆ ) and: r E ˆ ˆ V ˆ V ˆ r[( ) ] r ( ) + ( ) + ( ) + r ( ) Cov ( ˆ, ˆ ) ( )( ) r r = + V ˆ ˆ + ( ) r ( ) ( ) ( )( ) It follow that E Sˆ N d d V ˆ ˆ + r ( ) ( ) { ( ) r ( ) + Nˆ N Nˆ ( ) ( )( )} where N ˆ = d. Takng expectaton alo wth repect to the amplng degn, we obtan: ˆ EEr ( S ) S A A + + (6.3) where A = E {( N ) d V ( ˆ ˆ )} r A = E N Nˆ N Nˆ {( ) [ ( ) ( )( )]}

14 Both A and A are term of O(/ n ) and, followng tandard lnearzaton argument, we mplfy thee expreon by removng term of lower order. Frt, A aymptotcally equvalent to: λ = { ( )}. ˆ E N d Vr Turnng to the term A, we may wrte ˆ ˆ ˆ ( ) ( )( ) { }( ) ( ˆ = + )( ) N N N N N N N and, gnorng term of lower order, A aymptotcally equvalent to λ E + E N Nˆ {( ) } {( )( )} ˆ N N = var ( ) + cov (, ). Replacng A and A n (6.3) by λ and λ repectvely, we obtan λ + λ a the approxmate ba of Ŝ. We thu propoe a a ba-corrected SCB etmator of R : R % = S %. (6.4) where % ˆ ˆ ˆ = S λ λ and S ˆλ and ˆλ are etmator of λ and λ repectvely. An etmator of λ ˆ λ = N d Vˆ ( ˆ ), where Vˆ ( ˆ r ) an etmator of r V ( ˆ r ) and N may be replaced by N ˆ f t unknown. We propoe to ue the etmator V ˆ ( ˆ ) gven n the Annex. In the cae of contant weght d = N / n th gve: r ˆ λ = n h( x ' βˆ ) x '[ h( x ' βˆ ) x x '] x, j j j j where h( ' ˆ) = exp( ' ˆ) /[ + exp( ' ˆ)] x β x β x β. The econd term λ may, n general, be etmated ung degn-baed varance etmaton method. In the cae of contant weght the term N ˆ contant o λ 3

15 reduce to λ = ( ). nder mple random amplng, we may wrte V λ = ( n N ) S. It follow that a ba corrected etmator of S n the cae of mple random amplng : S% = Sˆ ˆ λ ˆ λ = ( + n N ) Sˆ n h( x ' βˆ ) x '[ h( x ' βˆ ) x x '] x. (6.5) j j j j 6. Standard Error and Confdence Interval A lnearzaton varance etmator for the SCB etmator ˆR now derved n term of a varance etmator ˆ v( S ) of ftted and hold. From (5.5) and ung lnearzaton we have ˆ var[ R ] S var( S ) Ŝ, aumng that a logtc regreon model ˆ. (6.6) To approxmate ˆ var( S ) we hall decompoe the dtrbuton of Ŝ nto the part nduced by the amplng degn for a fxed value of ˆβ and the part nduced by the dtrbuton of ˆβ. We take the latter to be β ˆ N( β, Σ), where: ( ) var{ d [ R h Σ = Jβ ( x ' β)] x} Jβ ( ) (6.7) and J( β) = E{ Iβ ( )} the expected nformaton rather than the oberved nformaton n (6.7). Thee two choce of nformaton are aymptotcally equvalent (to frt order) but the expected nformaton ha the advantage that Σ doe not depend on. We wrte var( S ˆ ˆ ˆ ) = Eˆ[var ( S )] + var ˆ[ E ( S )], (6.8) β β where the ubcrpt ˆβ denote the dtrbuton nduced by β ˆ N( β, Σ), whch may be nterpreted a arng from the repone proce. Followng uual lnearzaton argument we obtan: 4

16 ˆ var ( S ) var [ N d ( ) ] β=βˆ and, gven the contency of ˆβ for β (and for tandard knd of amplng degn), we have approxmately: E Sˆ N d. (6.9) β ˆ[var ( )] var [ ( ) ] Turnng to the econd component n (6.8), we may wrte: E ( Sˆ ) N ( ) β=βˆ. A a lnear approxmaton we have ˆ + z ( βˆ -β) where z = h( x ' β) x. Hence ( ) ( ) + ( )( z z ) ( β -β) β=βˆ + ( z z ) ( βˆ β)( βˆ β) ( z z ) ˆ where z z. = N In large ample, we aume that ˆβ normally dtrbuted o that ( βˆ β ) uncorrelated wth ( βˆ β)( βˆ β ) '. Hence, we have βˆ ˆ + ˆ ˆ ˆ tr β var [ E ( S )] 4AΣA var { [ Bβ ( -β)( β -β) ]}, (6.0) where A = ( )( z z ), N B ( z - z )( z z ) ' and Σ defned = N n (6.7). The econd term nvolve the fourth moment of ˆβ whch may alo be expreed n term of Σ nce ˆβ aumed normally dtrbuted. The varance of Ŝ may be etmated by the um of the etmated component of (6.8). The frt of thee appear n (6.9) and may be etmated by a tandard degnbaed etmator of var [ d ( ) ], where th treated a the varance of a 5

17 lnear tattc var [ u ] and u replaced by d ˆ ˆ ( ) n the expreon for the varance etmator. The econd component of the varance appear n (6.0). To etmate th term requre etmatng A, B and Σ. Frt, z may be etmated by zˆ = h( x ' βˆ ) x. Then A may be etmated by ˆ = N d ˆ ˆ ˆ ˆ A ( )( z z ), B may be etmated by ˆ = N d ˆ ˆ ˆ ˆ B ( z z )( z z )', where zˆ = N d z ˆ, and Σ may be etmated by a tandard etmator of the covarance matrx of ˆβ. Fnally, the varance of the SCB etmator ˆR may be etmated by pluggng the etmated varance of Ŝ nto (6.6) and replacng S by Ŝ. A confdence nterval for R wth level α gven by S% ± z v( Sˆ ). 0.5 α / 7. Smulaton Study of the Properte of the etmated R-ndcator 7. Degn of Smulaton Study In th Secton, we carry out a mulaton tudy to ae the amplng properte of the etmaton procedure decrbed n Secton 6. The tudy baed on repeated ample drawn from a fle (repreentng telf a 0% ample) from the 995 Irael Cenu. The fle contan 753,7 ndvdual aged 5 and over n 3,4 houehold. The ample are drawn ung degn ntended to be mlar to ome tandard houehold and ndvdual urvey carred out at natonal tattc nttute. We ue the followng ample degn n the mulaton: Houehold Survey the ample unt are houehold and all peron over the age of 5 n the ampled houehold are ntervewed. Typcally a proxy quetonnare 6

18 ued and therefore there no ndvdual non-repone wthn the houehold. In addton, we aume that every houehold ha an equal probablty to be ncluded n the ample. Indvdual Survey - the ample unt are ndvdual over the age of 5. We aume equal ncluon probablte. For each type of urvey, we carred out a two-tep degn to defne repone probablte n the populaton (cenu) fle. In the frt tep, we determned probablte of repone baed on explanatory varable that typcally lead to dfferental non- repone baed on our experence of workng wth urvey data collecton. A repone ndcator wa then generated for each unt n the populaton fle. In the econd tep, we ft a logtc regreon model and generate a true repone propenty for each unt n the populaton a predcted by the model. The dependent varable for the logtc model the repone ndcator and the ndependent varable of the model the explanatory varable ued n the frt tep (decrbed below). Th two-tep degn enure that we have a known model generatng the repone propente n the populaton and therefore can ae model mpecfcaton bede the amplng properte of the ndcator. The explanatory varable ued to generate the repone probablte are the followng: Houehold Survey Type of localty (3 categore), number of peron n houehold (,,3,4,5,6+), chldren n the houehold ndcator (ye, no). Indvdual Survey Type of localty (3 categore), number of peron n houehold (,,3,4,5,6+), chldren n the houehold ndcator (ye, no), ncome group (5 group), ex (male, female) and age group (9 group). 7

19 500 ample of ze n were drawn from the Cenu populaton of ze N at dfferent amplng fracton :50, :00, and :00. For each ample drawn, a ample repone ndcator wa generated from the true populaton repone probablty. The overall repone rate wa 8% for the houehold urvey and 78% for the ndvdual urvey. Repone propente and the R-ndcator were then etmated from the ample. Two choce of auxlary varable were condered, frt the true varable employed to generate the repone propente and, econd, a mpler et of varable, ntended to repreent a poble mpecfed model. 7. Reult Smulaton mean of the SCB etmator ˆR, defned n (5.5), and t propoed ba corrected veron R%, defned n (6.4), obtaned from the 500 repeated ample drawn for the Houehold Survey at dfferent amplng rate and for two dfferent model are reported n Table. Correpondng reult for the Indvdual Survey are preented n Table. Alo ncluded n the table the percentage of Relatve Ba 500 calculated from the mulaton tudy a: 00{[ ˆ = ( R B R ) / R ]/ 500} where Rˆ B B the value of Rˆ computed for the Bth mulaton ample and mlarly for R%. [PLACE TABLE HERE] [PLACE TABLE HERE] The reult for the true model provde evdence of downward ba n the SCB etmator, wth the (abolute) ze of the ba ncreang a the ample ze decreae. Th a expected. Samplng error tend to lead to overetmaton of the varablty of the etmated repone propente and th lead to underetmaton of the R- 8

20 ndcator. We oberve that the ba correcton reduce the (abolute) ba of the SCB etmator when the true model hold (although there ome evdence of overcorrecton n Table whch doe not dappear a the ample ze ncreae). The ba correcton decreae (n abolute value) wth the ncreae n ample ze and tend to tablze the SCB etmator. ng a le complex logtc model to etmate repone probablte reult n a moothng of the probablte and hence an ncreae n the value of the R- ndcator. We nclude n Table and value of R 0, whch the R-ndcator for the logtc model for the reduced et of auxlary varable whch bet ft the repone propente generated by the true model (for the complete et of auxlary varable) n the populaton. Treatng R 0 a the parameter of nteret, we oberve that the ba adjutment doe reduce the (abolute) ba for the houehold urvey but not necearly for the ndvdual urvey, where the ba correcton can lead to overetmaton. The ntablty of the ba correcton for the le complex et of auxlary varable lkely caued by the mpecfcaton of the model. Snce the ba correcton depend on the correct pecfcaton of the logtc model, t may not perform qute o well n thee cae. The Relatve Root Mean Square Error (RRMSE) were alo calculated from B = B R 500 the mulaton tudy a: ˆ 00{ R [ ( R ) / 500]} but not preented n the table. There wa no ytematc evdence of the ba adjutment leadng to larger RRMSE. For the Houehold Survey, the RRMSE wa table acro the SCB and propoed etmator for both type of model. For the Indvdual Survey, there wa more varablty n the RRMSE due to the fluctuatng ba correcton acro the 9

21 type of model. Th reult reflected by the Relatve Ba that preented n the table. Smulaton mean of the lnearzaton varance etmator (ee Secton 6.) are compared n Table 3 and 4 wth the mulaton varance (calculated acro the replcated ample) of the SCB etmator for the houehold and ndvdual urvey, repectvely. The table alo nclude the Coverage Rate defned a the percentage of tme that the true R ncluded n the confdence nterval calculated by the 500 lnearzaton varance etmator: 00{[ ˆ var ( ˆ = I ( R ± ))]/ 500} B RB B R B where var ( R ˆ B B ) the etmated lnearzaton varance for the Bth mulaton ample and I the ndcator functon. [PLACE TABLE 3 HERE] [PLACE TABLE 4 HERE] The lnearzaton varance etmator een to be approxmately unbaed acro the range of condton repreented n thee table under the dfferent ample ze wth good coverage a een by the coverage rate n the table. Fgure and preent box plot comparng the SCB etmator and t propoed ba adjuted veron for the Houehold and Indvdual Survey mulaton repectvely when fttng the true logtc regreon model. The gan from the ba adjutment are evdent. [PLACE FIGRE HERE] [PLACE FIGRE HERE] 0

22 8. Applcaton to Real Survey We demontrate the ue of R-ndcator on bune urvey undertaken for the 007 Dutch Short Term Stattc (STS) for retal and ndutry. Table 5 provde a bref decrpton of the two urvey. [PLACE TABLE 5 HERE] In the table, the urvey repone rate are gven for 5, 30, 45 and 60 day of feldwork. After 30 day STS need to provde data for monthly tattc. We examne both a complete et of auxlary varable contng of () bune ze cla (baed on number of employee), () bune ub-type and () VAT 006 a collected by the Tax Board and a reduced et contng of jut () and (). Table 6 provde the reult of the unadjuted and ba adjuted R-ndcator, 95% confdence nterval and the tandardzed maxmal ba (obtaned by pluggng etmated repone propente nto (4.4)) after 5, 30, 45 and 60 day of feldwork for each of the bune urvey. Becaue of the large ample ze, the ba adjutment had a mall mpact. [PLACE TABLE 6 HERE] The ample for the bune urvey are large and hence the confdence nterval are reduced wth wdth between % and.5%. The R-ndcator for STS retal after 30 day feldwork drop almot 7% when VAT added to the auxlary nformaton. For STS ndutry the decreae much reduced. Apparently, the ze of VAT n the prevou year doe not relate to repone very trongly. Wthout the VAT nformaton the retal repondent have a hgher R-ndcator than the ndutry repondent. When VAT added th pcture change and the retal repondent core

23 wore. STS retal how a reducton n the R-ndcator a the repone rate ncreae for the reduced et of auxlary varable. The man urvey tem of the STS urvey monthly turnover (ubdvded over dfferent actvte). A VAT n a prevou year can be expected to correlate trongly to turnover n the runnng year, t mportant that repreentatvene good wth repect to VAT. The man concluon that for Indutry, the R-ndcator goe up after 30 day, uggetng repone repreentatvene tll mprovng and one would deally wat longer than 30 day before producng tattc. For Retal, the R-ndcator lower, uggetng that repone le repreentatve than for Indutry, but there very lttle change when data collecton prolonged. Hence, t doe not pay off to wat longer than 30 day conderng the compoton of the repone. The only reaon to do o would be that the rk of nonrepone ba a reflected by the maxmal ba tll decreang a repone are comng n. 9. Dcuon In th paper we have condered a new ndcator, called the R-ndcator, degned to reflect the potental etmaton error arng from nonrepone. The ndcator defned at the populaton level and we have developed method for t etmaton ung ample data, ncludng method of ba adjutment and varance etmaton. The approxmate valdty of thee method ha been demontrated va mulaton. We have alo demontrated how the ndcator may be ued n real bune urvey a well a ocal urvey. The ba adjutment partcularly effectve for mall ample ze. In addton, the varance etmaton provde good coverage and avod the need for computer-ntenve reamplng method.

24 The ndcator defned wth repect to a et of auxlary varable. An R- ndcator cannot be vewed eparately from the auxlary vector X that wa ued to defne t. A uch, ndcator value hould alway be reported wth reference to the auxlary varable. Conequently, when comparng multple urvey wthn one urvey nttute, over urvey nttute or even over countre, one need to fx the et of auxlary varable ued for each of the urvey. There are two conflctng am that need to be balanced when electng auxlary varable n the comparon of dfferent urvey. On the one hand, t derable to chooe auxlary varable that are maxmally correlated wth the varable of analytc nteret n each urvey. On the other hand, the choce contraned to the et of auxlary varable that avalable for each of the urvey. The wder the cope of the comparon, the more retrctve the avalablty of varable wll be. Wthn one urvey nttute one lkely to ue one amplng frame, have acce to the ame regter data and collect mlar paradata for urvey. Multple countre, however, may have completely dfferent tradton and leglaton, whch wll lmt the et of auxlary varable that hared. More dcuon on the electon of auxlary varable n Schouten, Shlomo and Sknner (0). A key aumpton ha been that thee varable are meaured on both repondent and nonrepondent. Th aumpton may be reaonable n ome urvey ettng. For example, rch auxlary nformaton avalable at Stattc Netherland from a populaton regter. However, n other urvey ettng, the avalablty of untlevel auxlary nformaton on nonrepondent may be very lmted. Intead, aggregate nformaton on the populaton total of auxlary varable may be avalable. We are addreng the etmaton of R-ndcator ung uch nformaton n ubequent work. 3

25 Acknowledgement Th reearch wa undertaken a part of the RISQ (Repreentatvty Indcator for Survey Qualty) project, funded by the European 7th Framework Programme (FP7), a a jont effort of the natonal tattcal nttute of Norway, the Netherland and Slovena and the nverte of Leuven and Southampton. We hould lke to thank L-Chun Zhang, Jelke Bethlehem, Mattjn Morren and Ana Marujo for ther contrbuton. 4

26 Annex. Varance of ˆ for logtc regreon model For the logtc regreon model, wrte h( η) = g ( η) = exp( η) /[ + exp( η)]. The etmatng equaton n (5.3) may then be expreed a: d [ R h ( x ' β)] x = 0. (A) Let ˆβ olve (A). Then n large ample we may approxmate the dtrbuton of ˆβ wth repect to the amplng degn (c.f. Sknner, pp , 989) by the dtrbuton of : ˆ + ( ) [ h ( ) ] β β Iβ d R xβ x, (A) where β defned n (6.), Iβ ( ) = h ( x ' β) xx ' the nformaton matrx and d h( η) = h( η) / η = h( η)[ h( η)]. In partcular, the varance of ˆβ wth repect to the amplng degn n large ample V ( βˆ ) Iβ ( ) V { d [ R h( xβ )] x } Iβ ( ) (A3) and, nce ˆ = h( x ' β ˆ) from (5.4), we have V ( ˆ ) h( xβ ) x V ( βˆ ) x = h( xβ ) x Iβ ( ) V { d [ R h( xβ )] x } Iβ ( ) x j j j j j (A4) Th expreon treat the repone ndcator R j a fxed. To account for the repone mechanm alo, we may wrte 0 = E ( R x ) and r var( ˆ ) = E [ V ( ˆ )] + V [ E ( ˆ )] (A5) r r In large ample, we may wrte E ( ˆ ) h( x ' β ). Aumng 0 = E ( R x ), we r may wrte β β and 0.5 = 0 + Op ( N ) V E ˆ r[ ( )] O( N ) =. The frt term n (A5) generally of O( N ) and o the econd term may be treated a neglgble f the 5

27 amplng fracton n / N may be treated a neglgble. In th cae an expreon for var( ˆ ) may be obtaned by replacng β n (A4) by β 0. Reference Bethlehem, J.G. (988). Reducton of nonrepone ba through regreon etmaton. Journal of Offcal Stattc, 4, Cobben, F. and Schouten, B. (007). An emprcal valdaton of R-ndcator. Dcuon paper, CBS, Voorburg. Grove, R.M. (006). Nonrepone rate and nonrepone ba n houehold urvey. Publc Opnon Quarterly, 70, Grove, R.M. and Peytcheva, E. (008). The mpact of nonrepone rate on nonrepone ba: A meta-analy. Publc Opnon Quarterly, 7, Grove, R.M., Brck, J.M., Couper, M., Kalbeek, W. Harr-Kojetn, F. Kreuter, Pennell, B., Raghunathan, T., Schouten, B., Smth, T., Tourangeau, R., Bower, A., Jan, M., Kennedy, C., Leventen, R., Olon, K., Peytcheva, E., Znel, S. and Wagner, J. (008). Iue facng the feld: alternatve practcal meaure of repreentatvene of urvey repondent pool. Survey Practce, October Heerwegh, D., Abt, K. and Looveldt, G. (007). Mnmzng urvey refual and noncontact rate: Do our effort pay off? Survey Reearch Method,, 3-0. Kreuter, F., Olen, K., Wagner, J., Yan, T., Ezzat-Rce, T.M., Caa-Cordero, C., Lemay, M., Peytchev, A., Grove, R.M. and Raghunathan, T.E. (00) ng Proxy Meaure and other Correlate of Survey Outcome to Adjut for Non- 6

28 repone: Example from Multple Survey. Journal of the Royal Stattcal Socety, Sere A, 73, Lttle, R.J.A. (986). Survey nonrepone adjutment for etmate of mean. Internatonal Stattcal Revew, 54, Lttle, R.J.A. (988). Mng-data adjutment n large urvey. Journal of Bune and Economc Stattc, 6, Lttle, R.J.A. and Rubn, D.B. (00). Stattcal Analy wth Mng Data. nd Ed. Hoboken, NJ.: Wley. Rubn, D.B. (987). Multple Imputaton for Nonrepone n Survey. New York: Wley. Särndal, C-E. and Lundtröm, S. (005). Etmaton n Survey wth Nonrepone. Chcheter: Wley. Schouten, B. and Cobben, F. (007). R-ndcator for the comparon of dfferent feldwork tratege and data collecton mode. Dcuon paper 0700, CBS Voorburg. Schouten, B., Cobben, F. and Bethlehem, J. (009). Indcator for the repreentatvene of urvey repone. Survey Methodology, 35, 0-3. Schouten, B., Shlomo, N. and Sknner, C.J. (0). Indcator for montorng and mprovng repreentatvene of repone. Journal of Offcal Stattc (to be publhed). Sknner, C.J. (989). Doman mean, regreon and multvarate analy. In Sknner, C.J., Holt, D. and Smth, T.M.F. ed. Analy of Complex Survey, Chcheter: Wley. 7

29 Sknner, C.J. (003). Introducton to Part B. In Chamber, R.L. and Sknner, C.J. Analy of Survey Data, Chcheter: Wley. Wagner, J.R. (008). Adaptve urvey degn to reduce nonrepone ba. PhD dertaton, nverty of Mchgan, SA. 8

30 Table : Houehold Survey - Smulaton Mean of ˆR and t ba-corrected veron, R% and ther percent relatve ba (acro 500 mulated ample) Samplng Fracton (Sample Sze) :00 (n=,6) :00 (n=3,4) :50 (n=6,448) True Logtc Model Le Complex Logtc Model (Number of Peron, Localty Type, (Number of Peron) R 0 = Chld Indcator) R = SCB ˆR Propoed R% SCB ˆR Propoed R% Mean Relatve Ba (%) Mean Relatve Ba (%) Mean Relatve Ba (%) Mean Relatve Ba (%) Table : Indvdual Survey - Smulaton Mean of ˆR and t ba-corrected veron, % and ther percent relatve ba (acro 500 mulated ample) R Samplng Fracton (Sample Sze) :00 (n=3,769) :00 (n=7,537) :50 (n=5,074) True Logtc Model (Number of Peron, Sex, Age Group, Income Group, Localty Type, Chld Indcator) R = Le Complex Logtc Model (Number of Peron, Sex and Age Group) R 0 = SCB ˆR Propoed R% SCB ˆR Propoed R% Mean Relatve Ba (%) Mean Relatve Ba (%) Mean Relatve Ba (%) Mean Relatve Ba (%)

31 Table 3: Houehold Survey - Smulaton mean of lnearzaton etmator of varance of ˆR wth coverage rate and, mulaton varance (acro 500 mulated ample) (0-3 ) Samplng Fracton (Sample Sze) :00 (n=,6) :00 (n=3,4) :50 (n=6,448) True Logtc Model (Number of Peron, Localty Type, Chld Indcator) Smulaton Mean of Lnearzaton Etmator Coverage Rate Smulaton Varance Le Complex Logtc Model (Number of Peron) Smulaton Mean of Lnearzaton Etmator Coverage Rate Smulaton Varance % % % % % % 0. Table 4: Indvdual Survey - Smulaton mean of lnearzaton etmator of varance of ˆR wth coverage rate and mulaton varance (acro 500 mulated ample) (0-3 ) Samplng Fracton (Sample Sze) :00 (n=3,769) :00 (n=7,537) :50 (n=5,074) True Logtc Model (Number of Peron, Sex, Age Group, Income Group, Localty Type, Chld Indcator) Smulaton Mean of Lnearzaton Etmator Coverage Rate Smulaton Varance Le Complex Logtc Model (Number of Peron, Sex and Age Group) Smulaton Mean of Lnearzaton Etmator Coverage Rate Smulaton Varance % % % % % %

32 Table 5: Decrpton of 007 Dutch Bune Survey STS retal 007 STS ndutry 007 n=93,799 n=64,43 Repone=49.5% (5day) Repone=78.0% (30day) Repone=85.8% (45day) Repone=88.% (60day) Repone=48.8% (5day) Repone=78.7% (30day) Repone=85.7% (45day) Repone=88.3% (60day) All bunee retal All bunee ndutry Stratfed degn on ze cla and bune type Stratfed degn on ze cla and bune type unequal degn weght unequal degn weght Feldwork 90 day Feldwork 90 day Paper + web Paper + web Table 6: nadjuted (top) and Ba-adjuted (bottom) R-ndcator, 95% Confdence Interval and Standardzed Maxmal Ba (formula 4.4) for Dutch Bune Survey ung Reduced and Complete Set of Auxlary Varable Reduced Set Complete Set Survey 5d 30d 45d 60d 5d 30d 45d 60d R 9.9% 93.% 93.9% 94.% 90.% 9.6% 9.9% 93.% 9.% 93.3% 94.0% 94.% 90.5% 9.8% 93.% 93.3% Indutry CI B 6.% 8.5% 7.0% 6.6% 9.5% 0.4% 8.% 7.6% R 95.9% 94.5% 93.9% 94.0% 87.9% 87.8% 88.% 88.9% 96.% 94.6% 94.0% 94.% 88.% 87.9% 88.3% 89.0% Retal CI B 7.9% 6.9% 7.0% 6.7% 4.0% 5.5% 3.6%.5% 3

33 Fgure : Houehold Survey Box plot for SCB ˆR and t Ba-Corrected Veron, % for 500 mulated ample wth :00, :00 and :50 amplng fracton - R True R-Indcator = Fgure : Indvdual Survey Box plot for SCB ˆR and t Ba-Corrected Veron, % for 500 mulated ample wth :00, :00 and :50 amplng fracton - R True R-Indcator =