Measuring Survey Quality Through Representativeness Indicators Using Sample and Population Based Information

Transcription

1 Meaung Suvey Qualty Though Repeentatvene Indcato ng Sample and Populaton Baed Infomaton Ch Sknne, Natale Shlomo, Bay Schouten, L-Chun Zhang 3, Jelke Bethlehem Southampton Stattcal Scence Reeach Inttute, nvety of Southampton, e-mal: Cental Bueau of Stattc, Netheland, e-mal: 3 Stattc Noway, e-mal: lchunzhang@bno Abtact The RISQ (Repeentatvty Indcato fo Suvey Qualty) poject, funded by the Euopean 7th Famewok Pogamme, a jont effot of the NSI of Noway, The Netheland and Slovena, and the nvete of Leuven and Southampton to develop qualty ndcato fo uvey epone The epone ate alone nuffcent to meaue the potental mpact of non-epone Thee Repeentatvty Indcato (R-ndcato) ae developed to be ued a tool at dffeent tage of the data collecton poce: montong feld tatege, tagetng feld eouce and compang tatege fo nceang epone ate The auxlay nfomaton fo thee ndcato depend on po nfomaton about epondent and non-epondent and paadata that become avalable dung feldwok In many counte, po auxlay nfomaton fo non-epondent may be lmted Howeve, magnal dtbuton at the populaton level ae often avalable though egte and populaton etmate In th talk, we peent and compae poble R-ndcato and evaluate the amplng popete We alo popoe R- ndcato that ae baed entely on populaton total and the epondent et, and compae the popete to ample-baed R-ndcato though a mulaton tudy Keywod: epone ate, non-epone ba, qualty ndcato Intoducton One of the laget ouce of non-andom eo n uvey due to non-epone Reeach ha hown, ee eg Bethlehem (988) that the non-epone ba of etmate detemned by two facto: The amount to whch epondent and non-epondent dffe, on aveage, wth epect to the taget vaable The moe they dffe, the lage the ba wll be Th ometme called the contat between epone and non-epone; The amount of epone n the uvey The epone ate et a bound to the maxmal mpact of non-epone; the lowe the epone ate of the uvey, the lage the potental mpact of ba To ae the effect of non-epone on the qualty of etmato, one need to meaue both the epone ate and the contat between epone and non-epone

2 The epone ate alone an nuffcent qualty ndcato to meaue the potental mpact of non-epone Stattcal ndcato of the contat hould meaue the degee to whch epondent and non-epondent dffe fom each othe The poject RISQ (Repeentatvty Indcato fo Suvey Qualty), funded by the Euopean 7th Famewok Pogamme, a jont effot of the NSI of Noway, the Netheland and Slovena, and the nvete of Leuven and Southampton to develop qualty ndcato fo uvey epone Thee ndcato meaue the degee to whch the goup of epondent of a uvey eemble the complete ample When th the cae, the epone called epeentatve In uvey pactce, epone ate ae almot alway computed Howeve, an ndcaton of the contat eldom gven explctly nce nfomaton needed on chaactetc of houehold o entepe that dd not epond to the uvey Nonethele, when nfomaton avalable that auxlay to the uvey one can ndectly meaue pat of the contat It the objectve of the RISQ poject to tanlate auxlay nfomaton to Repeentatvty Indcato, to develop thee qualty ndcato, to exploe the chaactetc and to how how to mplement and ue them n a pactcal uvey In th pape, we decbe befly two Repeentatvty Indcato (hencefoth called R-ndcato) Moe dcuon and theoy of thee R-ndcato povded n Shlomo, Sknne, Schouten, Bethlehem and Zhang (008) We focu on two R-ndcato: a meaue baed upon the vaance of etmated epone pobablte, a dcued n the pape by Cobben and Schouten (005, 007) and Schouten et al (008) a elated meaue popoed by Sändal and Lundtöm (008), n the context of electng auxlay vaable fo weghtng adjutment The etmaton of the R-ndcato vey much dependent on the natue of avalable auxlay nfomaton We ntally aume that auxlay nfomaton avalable at the ample level In many cae, howeve, auxlay nfomaton may only be avalable n aggegated fom at the populaton level We ntoduce theoy fo etmatng R-ndcato when aggegated auxlay nfomaton known and compae them to the ample level ndcato n a mulaton tudy Secton contan a fomulaton of the theoetcal famewok fo the R-ndcato The two R-ndcato ae defned fomally at the populaton level n Secton 3 and the etmaton dcued n Secton 4 Secton 5 contan a mulaton tudy on the popete of ample baed and populaton baed etmated R-ndcato and we conclude n Secton 6 wth futue wok Theoetcal Famewok fo R-Indcato Geneal notaton and natue of avalable nfomaton We uppoe that a ample uvey undetaken, whee a ample elected fom a fnte populaton The ze of and ae denoted n and N, epectvely The unt n ae labelled =,, K, N The ample aumed to be dawn by a

3 pobablty amplng degn p (), whee the ample elected wth pobablty p( ) The ft ode ncluon pobablty of unt denoted π and d = π the degn weght In ome cae, we hall aume mple andom amplng wthout eplacement We uppoe that the uvey ubject to unt nonepone The et of epondng unt denoted Thu, we have We denote ummaton ove the epondent, ample and populaton by Σ, Σ and Σ, epectvely We let R be the epone ndcato vaable o that R = f unt epond and R = 0, othewe Hence, = { ; R = } We hall uppoe that the typcal taget of nfeence a populaton mean Y = N y of a uvey vaable, takng value y fo unt We uppoe that the data avalable fo etmaton pupoe cont ft of the value { y ; } of the uvey vaable, obeved only fo epondent Secondly, we uppoe that nfomaton avalable on the value of T = ( x,, x,, K, xk,, a vecto x ) of auxlay vaable We hall uually uppoe each x k, a bnay ndcato vaable, whee x epeent one o moe categocal vaable, nce th wll be the cae n the applcaton we conde, but ou peentaton allow fo geneal x k, value We aume that value of x ae obeved fo all epondent Fo the majoty of th document we hall alo aume that x known fo all ample unt, e fo both epondent and non-epondent We efe to th a ample-baed auxlay nfomaton Th a natual aumpton f, fo example, the vaable makng up x ae avalable on a egte Howeve, n many counte and uvey ettng the avalablty of auxlay nfomaton on non-epondent may be vey lmted, eg becaue of the abence of a egte In uch ccumtance, aggegate populaton-baed auxlay nfomaton may be avalable Th mght take the fom of a (fnte) populaton total and/o mean and/o covaance matx of x Repone popente We defne the epone popenty a a condtonal expectaton of the epone ndcato vaable R gven the value of pecfed vaable and uvey condton (Lttle, 986, 988): ρ ( x ) = E( R x ), whee the vecto of auxlay vaable X defned a n ecton Fo mplcty, we hall uually wte ρ = ρ ( x ) and X hence denote the epone popenty jut by ρ A detaled dcuon of the noton of epone popente and the popete peented n Shlomo, et al (008) In th dcuon t wa agued that t deable to elect the auxlay vaable conttutng x n uch a way that the mng at andom, denoted MAR (Lttle and Rubn, 00) hold a cloely a poble and that ou defnton of ρ = ρ ( x ) elate to a pecfc choce of auxlay vaable X x A dffeent choce would geneally eult n a dffeent ρ We note alo that we defne the epone popenty condtonal on the uvey condton that apply when the data ae collected n ode to be able to compae the epeentatvene of dffeent uvey 3

4 3 Non-epone model In ode to etmate R-ndcato, we hall ft etmate the epone popente, whee thee ae defned a ρ = E( R x ) a dcued n the pevou ecton In th pape, we hall ue paametc modellng aumpton about how ρ depend on x In Shlomo, et al, 008, we alo dcu non-paametc model and ome mplcaton of the complexty of the model fo the vaablty of the ρ A geneal cla of model epeentng the dependence of ρ on x may be expeed n the fom: g( ρ ) = x ' β, () whee g() a pecfed lnk functon, β a vecto of unknown paamete to be etmated, and x may nvolve the tanfomaton of the ognal auxlay vaable (eg by ncludng nteacton tem) fo the pupoe of model pecfcaton A tandad choce of lnk functon the logt functon, leadng to the logtc egeon model: log[ ρ /( ρ )] = x ' β () Anothe lnk functon wth mla behavou to the logt the pobt functon We hall alo conde the ue of the dentty lnk functon, whch gve the lnea pobablty model : ρ = x ' β, (3) nce th wll offe patcula mplfcaton n the cae of populaton-baed auxlay nfomaton Sändal and Lundtöm (008) conde the ecpocal lnk functon, whch gve: ρ = x ' λ, (4) and they efe to ρ a the nfluence and denote t φ They aume that the vecto x c x = fo all defned n uch a way that thee ext a contant vecto c uch that ' Th etcton wll n mot pactcal tuaton be met and effectvely equvalent to aumng that a contant ntecept tem ncluded n the auxlay nfomaton Sändal and Lundtöm (008) vew (4) a a hypothetcal model whch wll not hold n pactce and they ntead focu on a fnte populaton appoxmaton to th model Th appoxmaton obtaned by ft defnng a value λ of λ whch acheve the bet ft of model (4) n the fnte populaton Fo mathematcal convenence, they defne the ft a the weghted um of quaed dffeence ρ ( ρ x ' λ) and th mnmed when: λ ( ρ x x ') x =, (5) 4

5 povded x defned o that the nveted matx n (5) non-ngula Th mple that a fnte populaton appoxmaton to φ gven by: φ x '( ρ x x ') x = (6) We efe to thee quantte a the appoxmate nfluence 3 Defnton of R-Indcato at Populaton Level Let ρ = ( ρ, ρ,, ρ N )' denote the vecto of epone popente n the populaton Followng Schouten et al (008), the epeentatvty of the epone mechanm may be meaued by the vaaton between the ρ and n patcula by the tandad devaton of the epone popente gven by: S( ρ ) = ( ρ ρ ) N, (7) whee ρ = ρ / N It may be hown that: S( ρ ) ρ ( ) ρ Hence, tanfomng S( ρ) to: R( ρ) = S( ρ ) (8) enue that 0 R( ρ ) and, a dcued by Schouten et al (008), Rρ ( ) defne an R-ndcato whch take value on the nteval [0,] wth the value ndcatng the mot epeentatve epone, whee the ρ dplay no vaaton, and the value 0 ndcatng the leat epeentatve epone, whee the ρ dplay maxmum vaaton Sändal and Lundtöm (008) defne the followng R-ndcato: Q ( ρ ) = [ ρ ] [ ρ ( φ φ ) ] (9) ρ whee φ ρ the ρ - weghted mean of the φ gven by ρ ( ) ( ) φ ρ ρ φ = (0) Th quantty a weghted vaance of the appoxmate nfluence We may expect t magntude to be nveely elated to the magntude of Rρ ( ) Thu, n vey ough tem, we expect Rρ ( ) to deceae and nceae 4 Etmaton Q ( ρ ) to nceae a the vaablty of the ρ 4 Etmaton of populaton total fom ample and epondent data In the followng ecton, we hall ue the fact that, fo a gven vaable z, the degn-weghted ample total d z a degn-unbaed etmato of the populaton 5

6 total z and the degn-weghted epondent total d z an unbaed etmato of the ρ - weghted populaton total ρz 4 Etmaton of non-epone model The etmaton of the model n ecton 3 depend on the natue of the auxlay nfomaton Fo ample-baed auxlay nfomaton, the model n () can be etmated fom the data on epondent and non-epondent by maxmum peudo lkelhood (Sknne, 989) e the paamete vecto β n th model may be etmated by the value ˆβ, whch olve: d [ R g ( x ' β )] x = 0 () whee g () the nvee of the lnk functon One eaon fo ung the degn weght hee becaue the objectve to etmate an R-ndcato whch povde a decptve meaue fo the populaton The lnea pobablty model n (3) can be etmated n cloed fom by odnay leat quae o by weghted leat quae, whee the weght ae the degn weght Fo the ecpocal lnk functon model n (4), Sändal and Lundtöm (008) appoxmate the model by ρ x λ, whee λ defned n (5) and etmate th appoxmate model by etmatng λ fom the ample data by: ˆ λ = ( d x x ') d x () Note that th etmaton follow the tategy n ecton 4 and that t alo aume ample-baed auxlay nfomaton 43 Etmaton of epone popente Fo the genealzed lnea model n (), the uual etmato of the epone popenty ρ : ˆ ρ = g ( ' β ), (3) x ˆ whee ˆβ the etmato of β obtaned a dcued n the pevou ecton In the cae of the lnea pobablty model n (3), f β etmated by (degn-) weghted leat quae, the mpled etmato of ρ gven by: ˆ ρ x '( d x x ') d x R =, (4) OLS whch may alo be expeed a: OLS ˆ ρ x '( d x x ') d x = (5) In the appoach of Sändal and Lundtöm (008) wth the ecpocal lnk functon, φ etmated by: 6

7 ˆ φ = x ' ˆ λ, (6) whee ˆ λ defned n (), o that: ˆ φ x '( d x x ') d x = (7) and the eultng etmato of ρ ˆ φ Fo the logt o pobt lnk functon, the etmato ˆ ρ obtaned fom (3) mut fall n the feable nteval [0,] Th not necealy the cae fo ethe the etmato baed on the lnea pobablty model n (4) o the etmato ˆ φ of ρ baed on (7) In the cae of populaton-baed auxlay nfomaton, we note that d x and d x x ' ae unbaed fo x and x x ', epectvely and that n lage ample we may expect that d x x and d x x x x It follow fom (5) that we may appoxmate ˆ ρ OLS by: OLS % ρ x '( x x ') d x, (8) = and fom (7) that we may appoxmate ˆ φ by: % φ x '( d x x ') x (9) = Expeon (8) and (9) povde etmato of the epone popenty fo epondent when x not avalable fo ndvdual non-epondent but aggegate populaton-level nfomaton avalable The etmato n (8) eque knowledge of the populaton um of quae and co-poduct x x ' of the element of x If th unknown, we can etmate d x x n (5) by ewtng: d x x = d ( x x )( x x ) + Nx x, (0) whee x = x / n x can be eplaced by X The covaance matx: S = ( )( ) xx N d x x x x may be eplaced by the obeved covaance matx: S = ( ) ( )( ) xx d R d R x x x x, () whee = R x ) /( x ( R ) The etmato n (9) only eque knowledge of the populaton total of each of the element of x 7

8 44 Etmaton of R-ndcato Let ρˆ be an etmato of the epone pobablty ρ, a dcued n the pevou ecton Aumng that ample-baed auxlay nfomaton avalable, ρˆ may be computed fo each An etmato of ρ then gven by ˆ ρ = ( ˆ ρ ) / N () d Altenatvely, we could eplace N n the denomnato by ndcato Rρ ( ) by: ˆ R( ) ( ˆ ρ ˆ ρ ) N d We etmate the R- ρ = d (3) Agan, we could eplace N n th expeon by d If ρˆ only avalable fo epondent ( ), a n the cae of aggegated populaton-level auxlay nfomaton decbed at the end of the pevou ecton, a poble etmato of Rρ ( ) : ˆ R ( ρ ) = d ˆ ρ ( ˆ ρ ˆ ρ ) N whee ˆ ρ = ( d ) / N Th coect fo non-epone ba ung valdty of th coecton depend on the valdty of the etmate ˆ ρ ρˆ - weghtng The We now tun to the etmaton of Q ( ρ ) n (9) Sändal and Lundtöm (008) popoe the followng etmato: q ˆ [ d = ] [ d ( φ φ ) ], (4) whee ˆ φ defned n (7) and φ ( ˆ dφ ) /( d ) be e-expeed a φ = ( d ) /( d ) = They note that n fact φ can The etmato n (4) baed only upon epondent data Howeve, ˆ φ n (7) doe depend on d x whch may not be avalable n the cae of aggegated populaton level nfomaton In uch cae, we may eplace ˆ φ n (4) by % φ fom (9) Shlomo, et al (008) decbe ba adjutment fo Rρ ˆ( ) and q a well a confdence nteval fo the R-ndcato 5 Smulaton Study of the Popete of the Etmated R-ndcato 5 Degn of Smulaton Study In th ecton, we cay out mulaton tude to ae the amplng popete of the two R-ndcato: Rρ ˆ( ) defned n (3) and q defned n (4) The mulaton tudy 8

9 baed on ample dawn fom Cenu data fom the 995 0% Iael Cenu Sample contanng 753,7 ndvdual aged 5 and ove n 3,4 houehold The ample degn mla to a tandad houehold uvey caed out at NSI The ample unt ae houehold and all peon ove the age of 5 n the ampled houehold ae ntevewed Typcally a poxy quetonnae ued and theefoe thee no ndvdual non-epone wthn the houehold In th tudy, we aume that evey houehold ha an equal pobablty to be ncluded n the ample We cay out a two-tep degn to defne epone pobablte n the Cenu data In the ft tep, we detemne pobablte of epone baed on explanatoy vaable that typcally lead to dffeental non- epone baed on ou expeence of wokng wth uvey data collecton A epone ndcato wa then geneated fo each unt n the Cenu fom thee pobablte In the econd tep, we ft a logtc egeon model, a n (), to thee Cenu data and thu detemned a tue epone popenty fo each unt a pedcted by th model ftted to the populaton The dependent vaable of the model the epone ndcato and the ndependent vaable of the model the explanatoy vaable ued n the ft tep Th two-tep degn enue that we have a known model geneatng the epone popente and theefoe we can ae model mpecfcaton bede the amplng popete of the ndcato The explanatoy vaable ued to geneate the epone pobablte ae Type of localty (3 categoe), numbe of peon n houehold (,,3,4,5,6+), chlden n the houehold ndcato (ye, no) Sample of ze n ae dawn fom the Cenu populaton of ze N at dffeent amplng facton :50, :00, and :00 Fo each ample dawn, a ample epone ndcato geneated fom the tue populaton epone pobablty The oveall epone ate 8% Repone popente and R- ndcato ae then etmated fom the ample 5 Reult Thoughout th mulaton we examne the amplng popete of the R-ndcato a well a the mpact of model mpecfcaton on the popete Becaue malle ample ze geneally lead to the electon of a le complex model, we hall conde that mpecfcaton epeented by a mple model In Table, we examne ample dawn at dffeent amplng ate, etmate epone popente fo each ample and calculate the meaue Rρ ˆ( ) (defned n (3)) fo both ample and populaton baed auxlay vaable We peent eult fo both the tue model and a le complex model Table how that the ample baed etmato ˆ( ) Rρ nceae a the ample ze nceae If the pecfed model coect, thee ome downwad ba and th tend to nceae a the ample ze nceae Th a expected Samplng eo tend to lead to oveetmaton of the vaablty of the etmated epone popente and th lead to undeetmaton of the R-ndcato The degee of undeetmaton, howeve, mall nde the le complex model, etmaton of epone popente eult n a moothng of the popente and hence an 9

10 Table : Smulaton Mean of Rρ ˆ( ) fo Sample and Populaton Baed Auxlay Vaable fo 500 Sample ( Tue R-Indcato = 08780) Samplng Facton :00 (n=,6) :00 (n=3,4) :50 (n=6,448) Tue Model (Numbe of Peon, Localty Type, Chld Indcato) Rρ ˆ( ) Populaton Baed Sample Baed Covaance Matx Known Covaance Matx nknown Sample Baed Le Complex Model (Numbe of Peon) Rρ ˆ( ) Populaton Baed Covaance Matx Known Covaance Matx nknown oveetmaton n Rρ ˆ( ) Reult of Rρ ˆ( ) wth the ba coecton (not hown hee) have poduced a moe tablzed ndcato aco dffeent ample ze unde the coect model, but fo the le complex model wth oveetmaton of Rρ ˆ( ), the ba coecton can exacebate the oveetmaton Compang Rρ ˆ( ) when ung ample baed auxlay vaable and populaton baed auxlay vaable, Table how that the R-ndcato undeetmated when ung populaton baed auxlay vaable The vaaton of epone popente lage than the vaaton unde ample baed auxlay vaable Snce we ued mple andom ample and aumpton of MAR, thee eem to be lttle dffeence between the two populaton level ndcato of Rρ ˆ( ) baed on a known populaton covaance matx and an etmated covaance matx fom the epone et In Table, we examne the popete of q baed on the vaance of the etmated epone nfluence Fo th ndcato, we expect low value to eflect good qualty and mall non-epone ba We compae the full et of explanatoy vaable n the model ued n th mulaton to a le complex model a befoe 0

11 Table : Smulaton Mean of Vaable fo 500 Sample q fo Sample and Populaton Baed Auxlay Samplng Facton :00 (n=,6) :00 (n=3,4) :50 (n=6,448) Tue Model (Numbe of Peon, Localty Type, Chld Indcato) Q ( ρ) = Sample Baed Populaton Baed Le Complex Model (Numbe of Peon) Q ( ρ) = Sample Baed Populaton Baed Reult fom Table how the deceae n q a the ample ze nceae Shlomo, et al (008) dcu a ba coecton fo th ndcato a well The le complex model poduce moothe etmated nfluence and hence le vaaton Th expected nce the ndcato wa developed to ae the effectvene of auxlay vaable on ba educton ng populaton baed auxlay vaable have nceaed the vaaton of the etmated nfluence 6 Futue Wok Fom thee ntal mulaton, we contnue to develop theoy fo ung populaton baed auxlay nfomaton In patcula, we wll nvetgate an teatve OLS algothm fo the etmaton of ρˆ and βˆ, and ue popenty weghted total n the etmaton poce The fnal ρˆ can alo be ued to calculate popenty weghted total fo etmatng q We wll alo examne ung a GLM model ntead of the OLS model, although we have found no depatue fom the expected nteval of [0,] ung the OLS model Othe aea of wok fo the R-ndcato ae to apply ba coecton and conde the vaance of the etmato, potental vaance etmato and confdence nteval In addton, we can calculate maxmum bound on the non-epone ba though the ue of thee R-ndcato We wll focu on moe pactcal apect of mplementaton, uch a the choce of auxlay vaable and the pecfcaton of the model and compae the qualty of uvey n vaou epect: ubequent uvey on the ame topc wthn a county, dffeent uvey wthn a county, and uvey on the ame topc n dffeent counte The eult of thee compaon wll help to buld a bette Euopean Stattcal Sytem Acknowledgement: We thank the membe of the RISQ poject: Katja Ruta fom Stattčn ad Republke Slovenje, Geet Looveldt and Koen Beullen fom Katholeke nvetet, Leuven, Oyvn Kleven fom Stattk Sentalbyå, Noway, Fanne Cobben fom Centaal Bueau voo de Stattek, Netheland, Ana Maujo and Gab Duant fom the nvety of Southampton, K fo the valuable nput

12 Refeence Bethlehem, JG (988) Reducton of nonepone ba though egeon etmaton, Jounal of Offcal Stattc, 4, 5 60 Cobben, F and Schouten, B (005) Ba meaue fo evaluatng and facltatng flexble feldwok tatege, Pape peented at 6th Intenatonal Wokhop on Houehold Suvey Nonepone, Augut 8-3, Tällbeg, Sweden Cobben, F and Schouten, B (007), An empcal valdaton of R-ndcato, Dcuon pape, CBS, Voobug Lttle, RJA (986) Suvey nonepone adjutment fo etmate of mean Intenatonal Stattcal Revew, 54, Lttle, RJA (988) Mng-data adjutment n lage uvey Jounal of Bune and Economc Stattc, 6, Lttle, RJA and Rubn,DB (00) Stattcal Analy wth Mng Data, Hoboken, New Jeey: Wley Sändal, C-E and Lundtöm, S (008) Aeng auxlay vecto fo contol of nonepone ba n the calbaton etmato, Jounal of Offcal Stattc, 4, 67-9 Schouten, B, Cobben, F and Bethlehem, J (008) Indcato fo the Repeentatvene of Suvey Repone Suvey Methodology (to appea) Shlomo, N, Sknne, CJ, Schouten, B, Bethlehem, J, Zhang, L (008) Stattcal Popete of R-ndcato, Wok Package 3, Delveable 3, RISQ Poject, 7 th Famewok Pogamme (FP7) of the Euopean non Sknne, CJ (989) Doman mean, egeon and multvaate analy In Sknne, CJ, Holt, D and Smth, TMF ed Analy of Complex Suvey, Chchete: Wley