Potential gains from using unit level cost information in a model-assisted framework

Transcription

1 Unversty of Wollongong Research Onlne Centre for Statstcal & Survey Methodology Workng Paper Seres Faculty of Engneerng and Informaton Scences 2013 Potental gans from usng unt level cost nformaton n a model-asssted framework Davd Steel Unversty of Wollongong, dsteel@uow.edu.au Robert Graham Clark Unversty of Wollongong, rclark@uow.edu.au Publcaton Detals Ths workng paper was subsequently revsed and publshed as: Steel, D. G. & Clark, R. Graham. (2014. Potental gans from usng unt level cost nformaton n a model-asssted framework. Survey Methodology, 40 (2, See: Research Onlne s the open access nsttutonal repostory for the Unversty of Wollongong. For further nformaton contact the UOW Lbrary: research-pubs@uow.edu.au

2 Natonal Insttute for Appled Statstcs Research Australa The Unversty of Wollongong Workng Paper POTENTIAL GAINS FROM USING UNIT LEVEL COST INFORMATION IN A MODEL-ASSISTED FRAMEWORK Davd G. Steel and Robert Graham Clark Copyrght 2013 by the Natonal Insttute for Appled Statstcs Research Australa, UOW. Work n progress, no part of ths paper may be reproduced wthout permsson from the Insttute. Natonal Insttute for Appled Statstcs Research Australa, Unversty of Wollongong, Wollongong NSW Phone , Fax Emal: anca@uow.edu.au

3 POTENTIAL GAINS FROM USING UNIT LEVEL COST INFORMATION IN A MODEL-ASSISTED FRAMEWORK Davd G. Steel and Robert Graham Clark 1 Key Words: optmal allocaton; optmal desgn; sample desgn; samplng varance; survey costs Word Count: 2949 ABSTRACT In developng the sample desgn for a survey we attempt to produce a good desgn for the funds avalable. Informaton on costs can be used to develop sample desgns that mnmse the samplng varance of an estmator of total for fxed costs. Improvements n survey management systems mean that t s now sometmes possble to estmate the cost of ncludng each unt n the sample. Ths paper develops relatvely smple approaches to determne whether the potental gans arsng from usng ths unt level cost nformaton are lkely to be of practcal use. It s shown that the key factor s the coeffcent of varaton of the costs relatve to the coeffcent of varaton of the relatve error on the estmated cost coeffcents. 1 Natonal Insttute for Appled Statstcs Research Australa, Unversty of Wollongong, NSW Australa

4 1. INTRODUCTION Unequal unt costs have been reflected n sample desgns by usng smple lnear cost models. In stratfed samplng, a per-unt cost coeffcent can sometmes be estmated for each stratum. The resultng allocaton of sample to strata s proportonal to the nverse of the square root of the stratum cost coeffcents (Cochran In a multstage desgn the costs of ncludng the unts at the dfferent stages of selecton can be used to decde the number of unts to select at each stage (Hansen et al Whle ths theory s well establshed, unequal costs have not been used extensvely n practce (Brewer and Gregore 2009, perhaps because of a lack of good nformaton on costs, and because of a focus on sample sze rather than cost of enumeraton. Groves (1989 argued that lnear cost models are unrealstc, and that mathematcal cost modellng can dstract from more mportant decsons such as the mode of collecton, the number of callbacks and how the survey nteracts wth other surveys conducted by the same organsaton. Nevertheless, gven the pressures on survey budgets, the fnal desgn should reflect costs and varance n a ratonal way, wthout beng fxated on formal optmalty. Increasng use of computers n data collecton s leadng to more extensve and useful cost-related nformaton on unts on survey frames. In a programme of busness surveys conducted by a natonal statstcs nsttute, most medum and large enterprses wll be selected n some surveys at least every year or two. Ths may provde nformaton on costs for those busnesses, for example some busnesses may have requred extensve follow-up or edtng n a prevous survey. Drect experence s less lkely to be avalable for any gven small busness, but 2

5 datasets of costs could be modelled to gve predctons of lkely costs. Smlarly, computer asssted personal ntervewng and computer-asssted telephone ntervewng are wdely used n surveys of people and households. Survey management software routnely collects detaled nformaton on tme spent by ntervewers on dfferent tasks, leadng to a wealth of data relatng to survey costs, whch could then be used to model how cost relates to geography, age, sex and other varables. One example of ths trend s the creaton of an Operatons Research Unt by the Australan Bureau of Statstcs (ABS 2007, p151. The focus of ths unt s on the analyss of paradata about survey processes and costs to mprove collecton methodology, but a sde effect may be more detaled cost data for use n sample desgn. There are therefore ncreasng possbltes for complng fnely graned or even unt-specfc cost estmates, whch can potentally be used for sample desgn. However, unt cost estmates are lkely to be far from perfect, and gnorng the uncertanty attached to them may lead to less effcent desgns. Also, the cost of mantanng and modellng cost databases needs to be balanced aganst realstc evaluatons of the mprovement n samplng effcency. Ths paper develops relatvely smple approxmatons to the gans arsng from usng unt level cost nformaton n a model-asssted framework. Secton 2 contans notaton and some key expressons. Secton 3 s concerned wth the optmal desgn when cost parameters are known. Secton 4 analyses the use of estmated unt costs, and Secton 5 s a summary. 3

6 2. NOTATION AND OBJECTIVE CRITERION Consder a fnte populaton, U contanng N unts, consstng of values Y for U. A sample s U s to be selected usng an unequal probablty samplng scheme wth postve probablty of selecton π = P [ s] for all unts U. A vector of auxlary varables x s assumed to be avalable ether for the whole populaton, or for all unts s wth the populaton total, t x = x, also known. The auxlary varables could consst of, for example, ndustry, regon and sze n a busness survey, or age, sex and regon n a household survey. In the model-asssted approach (see for example Särndal et al. 1992, the relatonshp between a varable of nterest and the auxlary varables s captured n a model, typcally of the followng form n sngle-stage surveys: E M [Y ] = β T x var M [Y ] = σ 2 z Y ndependent of Y j for all j (1 where E M and var M denote expectaton and varance under the model, β and σ 2 are unknown parameters and z are assumed to be known for all U. Let E p and var p denote expectaton and varance under repeated probablty samplng wth all populaton values held fxed. The generalzed regresson estmator s a wdely used model-asssted estmator of t y : ˆt y = s (y ˆβ T x + ˆβ T t x (2 where ˆβ may be a weghted or unweghted least squares regresson coeffcent of y on x usng sample data. ] The antcpated varance of ˆt y s defned by E M var p [ˆt y t y, and s 4

7 approxmated by E M var p [ˆt y ] σ 2 ( π 1 1 z (3 for large samples (Särndal et al. 1992, formula , p451 under model (1. Model-asssted desgns and estmators should mnmse E M var p [ˆt y ] subject to approxmate desgn unbasedness, E p [ˆt y ] = ty. The antcpated varance has been used to motvate model-asssted sample desgns n one stage (Särndal et al and two stage samplng (Clark and Steel 2007; Clark One advantage of usng the antcpated varance for ths purpose s that t depends only on the selecton probabltes and a small number of model parameters, whch can be roughly estmated when desgnng the sample. In contrast, var p [ˆt y ] typcally depends on the populaton values of y and on jont probabltes of selecton, both of whch are dffcult to quantfy n advance. The cost of enumeratng a sample s assumed to be C = s c where c s the cost of surveyng a partcular unt. The values of c are usually assumed to be known. Typcally c are also assumed to be constant for all unts n the populaton, or constant wthn strata. Wth the generalzaton that c may be dfferent for every unt, the cost C depends on the partcular sample s selected. The expected cost s E p [C] = π c. The am s to mnmse the antcpated varance (3 subject to a constrant on the expected enumeraton cost, π c = C f. (4 There wll also be fxed costs that are not affected by the sample desgn and so do not have to be ncluded here. Some notaton for populaton varances and covarances s needed. Consder the pars (u, v, and let S uv = N 1 (u ū (v v denote ther populaton 5

8 covarance, and Su 2 = N 1 (u ū 2 denote the populaton varance of (u. Let ū and v be the populaton means of u and v. The populaton coeffcent of varaton of (u s C u = S u /ū. The populaton relatve covarance of (u, v s C u,v = S uv /ū v. A useful result s u v = Nū v (1 + C u,v. (5 3. OPTIMAL DESIGN WITH KNOWN COST AND VARIANCE PARAMETERS 3.1 Optmal Model-Asssted Desgn The values of (π : U whch mnmse (3 subject to (4 are π = C f z 1/2 c 1/2 j U z1/2 j c 1/2 j and the resultng antcpated varance s AV opt = E M var p [ˆt y ] = σ 2 C 1 f z 1/2 c 1/2 (6 ( 2 c 1/2 z 1/2 σ 2 z. (7 Ths can be easly derved usng Lagrange multplers or the Cauchy-Schwarz Inequalty, and generalzes Särndal et al. (1992, Result , p452 to allow for unequal costs. Hgher probablty of selecton s gven to unts whch have hgher unt varance or lower cost. However the square roots of z and c n (6 means that probabltes of selecton do not vary dramatcally n many surveys. It s assumed that the last term of (7, whch represents the fnte populaton correcton, s neglgble. Applyng (5 gves: AV opt σ2 C 1 f N 2 c z ( 1 + C c, 2 z (8 (1 + C c (1 2 + C z 2 6

9 where C c and C z refer to the populaton coeffcents of varaton of c and z, respectvely. To make our results nterpretable, we wll assume that unt costs c and varances σz are unrelated, so that C c, z = 0. Ths assumpton may not always be satsfed n practce, but any relatonshp between c and z wll be specfc to the partcular example, and could be ether postve or negatve. To dentfy general prncples, t makes sense to gnore any such relatonshp. In practce, t s often reasonable to also assume that Cc 2 and Cz 2 are small. A Taylor Seres expanson then shows that Cc 2 4C 2 c and Cz 2 4C 2 z. Puttng these approxmatons together, (8 becomes AV opt = σ 2 C 1 N 2 c z f ( C2 c ( C2 z (9 See the appendx for detals of these dervatons. Ignorng Costs If the costs are gnored, then (6 suggests that π z 1/2. To make comparsons for the same expected cost, C f, wth resultng antcpated varance ( AV nocosts = σ 2 C 1 f z 1/2 π = C f. (10 j U z1/2 j c j z 1/2 ( c z 1/2 Applyng dervatons smlar to those used n Secton 3.1, σ 2 z. (11 AV nocosts σ2 C 1 f N 2 c z (. ( C2 z (See the appendx for detals. Comparng (12 and (9, we see that takng costs nto account n the desgn results n dvdng the antcpated varance by ( C2 c. 7

10 4. THE EFFECT OF USING ESTIMATED COST PARAMETERS In practce, c are not known precsely. Suppose that estmates ĉ = b c are used nstead. Usng the auxlary varable and the estmated costs n the optmal probabltes mples π z 1/2 cost, ĉ 1/2 The resultng antcpated varance s AV ests = σ 2 C 1 f. To make comparsons for the same expected ĉ 1/2 j U z1/2 j ĉ 1/2 π = C f z 1/2 ( ĉ 1/2 z 1/2 ( j U j c j. z 1/2 j ĉ 1/2 j c j σ 2 z. (13 If we assume that the values of b are unrelated to the values of c and z, then AV ests = σ 2 C 1 f σ 2 ( c 1/2 z 1/2 2 N 2 ( b 1/2 ( b 1/2 z (14 (see the appendx for detals. If the coeffcent of varaton of b s small, then a Taylor Seres approxmaton gves N 2 b 1/2 1/2 b C2 b. Applyng ths, and the same approxmatons as n Subsecton 3.1, (14 becomes AV ests = σ2 C 1 f N 2 c z ( C2 b ( ( ( C2 c C2 z Comparng (15 and (12, the effect of usng estmated cost parameters rather than no costs at all s to multply the antcpated varance by ( ( C2 b / C2 c. Therefore cost nformaton s worth usng provded C b < C c. The coeffcent of varaton of the error factors has to be less than that of the true unt costs over the populaton. 8

11 5. EXAMPLES OF COST MODELS The key quanttes determnng the usefulness of the unt cost data are C b and C c. Optmal desgns usng unequal cost nformaton are not very common, so there s relatvely lttle lterature on the typcal values of these measures. Unequal costs may be drven by a varety of factors, ncludng mode effects, geography and wllngness to respond, and lterature on these ssues s helpful to gve a rough dea of cost models that may apply n practce. Groves (1989, p.538 compares per-respondent costs of telephone ntervewng ($38.00 and personal ntervewng ($84.90 of the general populaton. If the preference of all unts on a frame were known, and half preferred each mode, ths would mply C c = Greenlaw and Brown-Welty (2009 compared paper and web surveys, and found per-respondent costs of $4.78 and $0.64, respectvely, n a survey of members of a professonal assocaton. In a mxed mode opton, two thrds of respondents opted for the web opton. If preferences are known n advance, then C c = Another reason for varyng costs s that some respondents are more dffcult to recrut than others, requrng more vsts or remnders. Groves and Heernga (2006, Secton 2.2 tralled a survey where ntervewers classfed non-respondents from the frst approach as ether lkely or unlkely to respond. In subsequent follow-up, the frst group had a response rate of 73.7% compared to 38.5% for the second group. Ths suggests that the per-respondent cost for the second group would be at least 1.9 tmes hgher than the frst group. (In fact, the rato would be hgher, because more follow-up attempts would be made for the dffcult group. If 50% of respondents are n both groups, then C c =

12 Geography s another source of dfferental costs n ntervewer surveys. In the Australan Labour Force Survey, costs have been modelled as havng a per-block component and a per-dwellng component (Hcks 2001, Table n Secton 4.2 dependng on the type of area (15 types were defned. Assumng a constant 10 dwellngs sampled per block, the net per-dwellng costs range from $4.98 n Inner Cty Sydney and Melbourne to $6.71 n Sparse and Indgenous areas. Whle ths s a sgnfcant dfference n costs across area types, the great majorty of the populaton are n three area types (settled area, outer growth and large town where per-dwellng costs vary only between $5.71 and $6.07. As a result, C c s estmated at a very small Table 1 shows the approxmate percentage mprovement n the antcpated varance from usng estmated cost nformaton for dfferent values of C c and C b, some suggested by these examples. Negatve values ndcate that the desgn s less effcent than gnorng costs altogether. The table suggests that cost nformaton s only worthwhle provded there s a far varaton n the unt costs, otherwse the beneft s very small, and can be erased when there s even small mprecson n the estmated costs. Mxed mode surveys are probably the best canddate for explotng varyng unt costs n sample desgn. 10

13 Table 1: Percentage mprovement n antcpated varance from usng estmated cost nformaton compared to no cost nformaton Coeffcent of Possble Scenaro Coeffcent of Varaton Varaton of Unt of Error Factor (C b (% Costs (C c (% ntervewer travel due to remoteness response propensty mxed mode (phone/personal nt mxed mode (paper/web self-complete DISCUSSION Incorporatng unequal unt costs can mprove the effcency of sample desgns. For the gans to be apprecable, the unt costs need to vary consderably. Even wth no estmaton error, a coeffcent of varaton of 50% may lead to a gan of only 6% n the antcpated varance. When ths coeffcent of varaton s 75%, as can happen n a mxed mode survey, the reducton n the antcpated varance (or n the sample sze for fxed precson can be over 12%. Costs wll be estmated wth some error and ths reduces the gan by a factor determned by the relatve varaton of the relatve errors n estmatng the costs at the ndvdual level. 11

14 REFERENCES ABS (2007, Australan Bureau of Statstcs Annual Report , Brewer, K. and Gregore, T. G. (2009, Introducton to Survey Samplng, n Handbook of Statstcs 29A: Sample Surveys: Desgn, Methods and Applcatons, eds. Pfeffermann, D. and Rao, C. R., Amsterdam: Elsever/North-Holland, pp Clark, R. G. (2009, Samplng of subpopulatons n two-stage surveys, Statstcs n Medcne, 28, Clark, R. G. and Steel, D. G. (2007, Samplng wthn households n household surveys, Journal of the Royal Statstcal Socety: Seres A (Statstcs n Socety, 170, Cochran, W. (1977, Samplng Technques, New York: Wley, 3rd ed. Greenlaw, C. and Brown-Welty, S. (2009, A comparson of web-based and paper-based survey methods testng assumptons of survey mode and response cost, Evaluaton Revew, 33, Groves, R. M. (1989, Survey Errors and Survey Costs, New York: Wley. Groves, R. M. and Heernga, S. G. (2006, Responsve desgn for household surveys: tools for actvely controllng survey errors and costs, Journal of the Royal Statstcal Socety: Seres A (Statstcs n Socety, 169, Hansen, M., Hurwtz, W., and Madow, W. (1953, Sample Survey Methods and Theory Volume 1: Methods and Applcatons, New York: John Wley and Sons. 12

15 Hcks, K. (2001, Cost and Varance Modellng for the 2001 Redesgn of the Monthly Populaton Survey, Australan Bureau of Statstcs Methodology Advsory Commttee Paper. Särndal, C., Swensson, B., and Wretman, J. (1992, Model Asssted Survey Samplng, New York: Sprnger-Verlag. 13

16 APPENDIX: DETAILED DERIVATIONS Lemma 1: Let u be defned for U. Let u = ū + θe, where e = 0 and θ s small. Then: a. u = ū 1 8 θ2 ū 3/2 S 2 e + o (θ 2. b. S 2 u = 1 4 θ2 ū 1 S 2 e + o (θ 2 = 1 4ū 1 S 2 u + o (θ 2. c. N 2 ( u1/2 ( u 1/2 = θ2 ū 2 Se 2 + o (θ 2 = C2 u + o (θ 2. d. C 2 u = 1 4 θ2 ū 2 S 2 e + o (θ 2 = 1 4 C2 u + o (θ 2. The notaton o (C 2 u can be used n place of o (θ 2, snce C 2 u = θ 2 C 2 e. Ths wll be done n the remander of the Appendx. Proof: We start by wrtng u as a functon of θ: ū + θe u = N 1 u = N 1 Call ths g (θ, then dfferentatng about θ = 0 gves g(0 = ū, g (0 = 0 and g (0 = 1 4 N 1 ū 3/2 e 2 = 1 4ū 3/2 S 2 e. Hence u = g (θ = g(0 + g (0θ g (0θ 2 + o ( θ 2 = ū 1 8 θ2 ū 3/2 S 2 e + o ( θ 2 whch s result a. 14

17 Result b s proven usng result a: S 2 u = N ( 1 ( u 2 N 2 1 u ( u 2 = ū ( ū 1 = ū 8 θ2 ū 3/2 Se 2 + o ( θ 2 2 = ū (ū θ4 ū 3 S 4e 14 θ2 ū 1 S 2e + o ( θ 2 = 1 4 θ2 ū 1 Se 2 + o ( θ 2 = 4ū 1 1 Su 2 + o ( θ 2 To derve c, we frstly wrte N 1 u 1/2 as a functon g( of θ and take a Taylor Seres expanson: N 1 u 1/2 = N 1 (ū + θe 1/2 = g (θ = g(0 + g (0θ g (0θ 2 + o ( θ 2 = ū 1/2 + 0θ ū 5/2 N 1 e 2 θ 2 + o ( θ 2 2 = ū 1/ ū 5/2 S 2 e θ 2 + o ( θ 2 (16 Note thatn 1 u1/2 (16 gves N 2 ( = u 1/2 = u. Multplyng the expresson for u n result a and ( u 1/2 { ū 1 8 θ2 ū 3/2 Se 2 + o ( θ 2 } {ū 1/2 + 38ū 5/2 S 2e θ 2 + o ( θ 2} = ū 2 S 2 e θ 2 + o ( θ 2 = C2 u + o ( θ 2 whch s result c. For result d, frstly note that u = ū + o (θ from result a, and so, from a frst 15

18 order Taylor Seres, ( u 2 = ( ū 2 + o (θ = ū 1 + o (θ. Combnng ths wth result b, we obtan C 2 u = S 2 u ( u 2 = { 1 4 θ2 ū 1 Se 2 + o ( θ 2 } {ū 1 + o (θ } = 1 4 θ2 ū 2 S 2 e + o ( θ 2 = 1 4 C2 u + o ( θ 2 gvng result d. Dervaton of (8 For the specal case where u = v, (5 becomes u 2 = Nū ( Cu 2. (17 Applyng (5, c 1/2 z 1/2 = N c z ( 1 + C c, z (18 where c = N 1 c and z = N 1 z. Usng (17, we can express c n terms of c: c = N 1 c = N 1 ( c 2 = ( c 2 (1 + C 2 c. (19 Smlarly, z = ( z 2 (1 + C 2 z. (20 Assumng the last term of (7 s neglgble, applyng (18, (19 and (20 gves (8. 16

19 Dervaton of (9 Lemma 1d mples that C 2 c = 1 4 C2 c + o (Cc C2 c and C 2 z = 1 4 C2 z + o (Cz C2 z. Result (9 follows from (8 by usng these approxmatons, as well as assumng that C c, z = 0. Dervaton of (12 Frstly, c z 1/2 = N c z ( 1 + C c, z, from (5, where Cc, z s the populaton relatve covarance between the values of z 1/2 and c. It s assumed that the values of c and z are unrelated, so that C c, z = 0. It s also assumed that the second term of (11 s neglgble, correspondng to small samplng fracton. Hence (11 becomes: AV nocosts = σ 2 N 2 C 1 f c ( z 2. (21 From (20, and Lemma 1d, we have Substtutng nto (21 gves (12. ( z 2 = z 1 + C 2 z z C2 z Dervaton of (14 Two terms n (13 wll be smplfed usng (5. Frstly, ĉ 1/2 z 1/2 = b 1/2 c 1/2 z 1/2 ( = N N 1 b 1/2 ( N 1 c 1/2 z 1/2 + C b, cz (22 where C b, cz s the covarance between the populaton values of b1/2 and c 1/2 z 1/2. 17

20 Secondly, z 1/2 ĉ 1/2 c = b 1/2 c 1/2 z 1/2 ( = N N 1 b 1/2 ( N 1 c 1/2 z 1/2 + C 1/ b, cz (23 where C 1/ b, cz s the covarance between the populaton values of b 1/2 and c 1/2 z 1/2. If we assume that the populaton values of b are unrelated to the values of c and z, so that C b, cz = C 1/ b, cz = 0, and substute (22 and (23 nto (13, then we obtan (14. Dervaton of (15 We can express (14 n terms of AV opt whch s defned n (7, assumng the last term of (7 s neglgble, correspondng to small samplng fracton: Lemma 1c mples that AV ests AV opt N 2 b 1/2 b 1/2 (24 N 2 b 1/2 1/2 b = C2 b + o ( Cb C2 b. Substtutng ths, and (8, nto (24 gves (15. 18