Statistical registers by restricted neighbor imputation

Transcription

1 Statstcal regsters by restrcted neghbor mputaton an applcaton to the Norwegan Agrculture Survey Abstract Nna Hagesæther 1 and L-Chun Zhang Statstcs Norway In ths paper we mplement the method of Zhang and Nordbotten (28) for constructng a statstcal regster by the use of neghbor-mputaton wth restrcton (RENI) Emprcal results from the Norwegan Agrculture Survey are shown, and further research on the topc s dscussed 1 Introducton Based on a survey, one may want to estmate the populaton total or mean for several varables To use avalable data more effcently, we can combne data from admnstratve sources and statstcal surveys A way to do ths s to use a sample survey wth target varables and an admnstratve regster wth auxlary varables and construct survey weghts for the sample unts Ths method s common n statstcal producton at the Natonal Statstcal Insttutes An alternatve approach s to predct the values of the target varables for every unt outsde the sample and construct a statstcal regster Statstcal tables wll then be constructed based on all the populaton unts, each of them havng the weght equal to one If one assumes that the probablty of non-response s condtonally ndependent of the varable of nterests gven the auxlary varables, the non-respondent unts n the sample can be predcted n the same way as unts outsde the sample Zhang and Nordbotten (28) proposed three goals they would lke to acheve n constructng a statstcal regster: 1 It should yeld effcent estmates of populaton totals of nterest 2 It should contan correct co-varances among the survey varables, as well as between the survey and auxlary varables 3 It should be non-stochastc, such that the statstcs can be reproduced on repetton The frst goal s tradtonally assocated wth the producton of statstcal tables The second one s a natural requrement now that a statstcal regster s a complete mcro-data fle The last condton s mportant for the acceptablty and face-value n offcal statstcs: e f a procedure s repeated, the results should be exactly the same In the paper they present a table where common mputaton methods are classfed wth respect to the trple-goal crteron: 1 e-mal: nh@ssbno 1

2 Table 1: Trple-goal classfcaton of common mputaton methods, from Zhang and Nordbotten (28) Method (1) (2) (3) Regresson Predcton Not always No Yes Random regresson mputaton No No, f multvarate No Multple Imputaton Not always No, f multvarate No Predctve mean matchng Not always No, f multvarate Yes, n theory Artfcal Neural Network Usually not No, f categorcal Yes Nearest neghbor mputaton Usually not Yes, non-parametrc Yes, n theory Nearest neghbor mputaton (NNI) emerges as the only practcal approach n terms of preservng the co-varances among all the varables It s therefore a prncpal method for the constructon of a statstcal regster Consder the smple stuaton wth a sample ( x 1, y1),, ( x r, y r) where y s a vector of all target varables known for the respondents r, 1 r, and all x -values are observed The NNI method mputes y by y, where r+ 1 N and N s the number of unts n the populaton, and satsfes x x = mn x x l 1 l r and denotes the chosen dstance metrc In the case of multple nearest neghbors, the donor s randomly selected as one of them Chen and Shao (2) showed that NNI yelds asymptotcally consstent estmators of the populaton totals as well as the fnte populaton dstrbutons of nterest However, ther emprcal results also ndcate that the method s often not effcent Zhang and Nordbotten suggested a new approach called RENI (restrcted neghbor mputaton), by mposng restrctons n the mputed totals, whch may be obtaned separately from the NNI such as through a regresson predcton A smple modfcaton s ntroduced n order to make the NNI nonstochastc, and addtonal neghbors are ncluded as potental donors, not only the nearest In secton 2, the algorthm for the RENI approach s descrbed In secton 3, we show emprcal results where RENI s appled to real data from the Norwegan Agrculture Survey In secton 4 we dscuss some further work that we have planned 2 Algorthm for RENI The algorthm to produce a RENI based statstcal regster n Zhang and Nordbotten (28) s descrbed as follows The ump-start (JS) phase Denote by R the set of recevers Denote by D the set of donors Let x be the varable (or varables) based on whch the dstance metrc (and the NNI) s defned 1 Set the counter d = for all D 2 For each R : a) Let m be the number of nearest neghbor (NN) donors, where m 1 b) For each NN-donor, ncrease ts counter d by 1/m 2

3 3 Let Y R denote the column vector of margnal restrctons for the recevers Let y be the correspondng vector of varables for D Put d = d g and g = 1+ (Y Y % ) A% y ' T 1 R R where Y % R = dy and D T A % = dyy It s easly verfed that D dy ' D = Y R ' 4 Let d = a + u, where a s the largest nteger that satsfes a d Sort the recevers n the ncreasng order of m For = 1,, R : a) Fnd the frst NN donor wth a 1 Impute y = y, and decrease a by 1 b) Do nothng f there s no NN-donor wth postve a ' The fne-tune (FT) phase Denote by R the remanng set of recevers that have not yet been ' mputed Extend x to x so that for each R' there s now a unque orderng among the potental donors by x and, some addtonal nformaton z For nstance, z can be the post zp code, the dentfcaton number of unt, and so on Notce that z s not consdered nformatve The unque orderng ensures that the procedure s non-stochastc 1 Set k =1 For each R', fnd the NN-donor and set y y = Let 1 Δ be the dstance between YR' = YR y and YR';k 1 = y accordng to some chosen metrc R\R' = R' 2 Set k =2 For R', let D ;k = 2 contan ts two closest NN-donors a) For each R' b) For = 1,, R', set, fnd the NN-donor and set I I = = for D ;k = 2 that yelds a closer mputed total to c) Repeat step 2b untl no changes can be made Calculate Δ 2 between Y R' Y R' and R';k 2 Y = 3 Stop f Δ 2 =, and use the mputatons from Step 2 Otherwse, stop f Δ2 Δ 1, and use the mputaton from Step 1 Otherwse, set k = 3 and let D ;k = 3 contan the three closest NNdonors for R' a) For each R' b) For = 1,, R', set, fnd the NN-donor and set I I = = for D ;k = 3 that yelds a closer mputed total to c) Repeat step 3b untl no changes can be made Calculate Δ 3 between Y R' Y R' and R';k 3 Y = 4 Stop f Δ 3 =, and use the mputatons from Step 3 Otherwse, stop f Δ3 Δ 2, and use the mputaton from Step 2 Otherwse, set k = 4 and let D ;k = 4 contan the four closest NNdonors for R' 3

4 The followng observatons are worth notng: The JS s desgned to speed up the process Y R may consst of optonal target varables At each teraton of the FT phase the donor s the one among the k nearest neghbors that best satsfes the restrctons The consstency of the NNI remans, as long as the dfference between the condtonal expectatons of a unt and ts k -th nearest neghbor s bounded by the dstance between them through a fnte constant Devaton from the margnal restrctons of the mputed totals can be reduced n two ways Frstly, one may ncrease k to allow for greater combnatoral flexblty Secondly, one may reduce the amount of mputatons acheved by the JS, or even skppng completely over t In constructng Δ, one can put a larger weght on the target varables that are consdered more mportant In return these wll be closer to the restrcton, on the expense of the other varables consdered less mportant The weght s called varable-weght and denoted by W If one wants to ntroduce restrctons on a more detaled level than the mputaton class, ths can be ncorporated n the algorthm Y R can be extended to nclude sub-populatons of the mputaton class, and the Δ can be calculated also wth respect to the sub-populatons A weght between and 1, called margnal weght and denoted by W, may be ncluded, to balance between the mputaton class total and the sub-populaton total If W =, no subpopulaton totals are ncluded n Δ, and f W = 1, only sub-populaton totals are ncluded 3 Emprcal results We have tested RENI on the Norwegan Agrculture Survey 26 Agrculture Survey 26 There are 126 respondng unts whch are the donors There are 4993 non-respondng and out-of-sample unts whch are the recevers The Agrculture Survey 26 has 84 varables of nterest: 42 of them are sze varables and the other 42 are ndcator varables dependng on whether the sze s postve Leasng, nvestment and mantenance were the most mportant topcs ncludng the followng varables, all gven n Norwegan kroner (VAT excluded): Leasng: leasng (pad leasng rent for machnes), leasng1 (value of new leasng contract, contract 1) and leasng2 (value of new leasng contract, contract 2) Investment: _fxed (sum nvested n fxed techncal equpment n outbuldngs), _buld (sum nvested n outbuldngs), mach_new (purchase of other new machnes and tools) and mach_used (purchase of other used machnes and tools) We have also ncluded sale_mach (sales of other machnes and tools) Mantenance: m_buld (mantenance of buldngs and fxed equpment), m_tractor (mantenance of tractors and combne harvesters), m_car (mantenance of cars and trucks) and m_mach (mantenance of machnes and tools) 4

5 Estmated numbers are publshed at the followng classfcaton levels: Farmng system/actvty (FA): 1=gran and olseed crops, 2=other agrcultural crops, 3=garden crops, 4=cattle, mlk producton, 5=cattle, meat producton, 6=cattle, mlk and meat producton, 7=sheep, 8=other roughage anmals, 9=swne and poultry, 1=mxed crop producton, 11=mxed farm anmal producton, 12=crop and farm anmal producton Class of farmland n decares: 1=-4, 2=5-49, 3=5-99, 4=1-199, 5=2-499 and 6=5+ County The mputaton class s FA Ths means that recevers can only get a donor wth the same farmng actvty Due to lmted space we show only results four dfferent mputaton classes: FA-2: 266 recevers and 727 donors The sample fracton s about the same as for the whole populaton, and the proporton of recevers s about average FA-4: 9984 recevers and 3296 donors Ths s the largest mputaton class n ths study FA-1: 384 recevers and 243 donors Ths s the smallest mputaton class n ths study, wth a hgh samplng fracton above 6 percent FA-11: 586 recevers and 34 donors Ths s another one of the smallest mputaton classes The varable used to fnd the nearest neghbor wthn each mputaton class s farmland n decares and an dentty number of the farms, where the latter of whch s non-nformatve The restrctons are the publshed totals of each mputaton class, obtaned by stratfed rato estmaton 31 Performance n satsfyng the restrctons Frst we examne how well the algorthm s able to satsfy the mposed restrctons Ths s mportant n order to realze the effcency goal Fgure 1 below shows the effect of k on Δ, where Δ s calculated as the un-weghted sum of relatve dfferences between the restrcton and the mputed class total for all varables Recall that k =1 means that only the nearest neghbor may be the donor, k =2 means that the donor may be selected among the two nearest neghbors, and so on It s seen that Δ decreases rapdly as k ncreases from 1 to 3, and there s bascally no change n Δ beyond k =6 Fgure 1: How vares wth k, for farmng actvtes (FA) 2, 4 1 and FA-2 FA FA-1 FA-11 k 5

6 We examne next how the JS affects the computaton tme Fve dfferent scenaros of the JS phase where used, varyng from no JS at all to JS wth respect to all varables of nterest The results are summarzed n Table 2-5, where s the same as above We notce that the number of unts that s left for the FT phase depends mostly on whether the JS s employed or not, but less on the exact restrctons used n the JS phase The computaton tme depends on number of teratons and number of recevers at the FT phase Twce as many recevers, or twce as many teratons, takes about twce as long tme --- the relatonshp s lnear as shown n Fgure 2 Snce the number of teratons does not seem to depend on the JS at all, the JS can sgnfcantly reduce the computaton tme for large mputaton classes such as FA-4 Meanwhle, the restrctons n the JS do affect the fnal Generally, t seems that the sze varables more readly put the JS n the rght drecton, and usng all the restrctons at the JS acheves the best fttng of the fnal mputed totals In concluson, the JS performed as ntended, and usng all the restrctons at the JS yelded fnal mputaton totals that were bascally as good as wthout the JS at all The latter opton requres of course much more computaton tme n large mputaton classes Table 2: Shows how and computaton tme depend on restrctons and number of unts n the fne-tune phase JS = ump-start, FT = fne-tune Farmng actvty 2 Seconds Iteratons, k=6, k=1 Restrctons # n JS # n FT No JS ,16 22, Wth JS ,57 15,48 5 sze varables Wth JS ,48 12,7 All sze varables Wth JS 9 4 5,23 12,84 All ndcator varables Wth JS ,9 11,95 All varables Table 3: Shows how and computaton tme depend on restrctons and number of unts n the fne-tune phase JS = ump-start, FT = fne-tune Farmng actvty 4 Seconds Iteratons, k=6, k=1 Restrctons # n JS # n FT No JS ,17 33, Wth JS ,2 11,34 5 sze varables Wth JS 3 4 2,29 8,8 All sze varables Wth JS ,93 1,17 All ndcator varables Wth JS ,97 8,43 All varables Table 4: Shows how and computaton tme depend on restrctons and number of unts n the fne-tune phase JS = ump-start, FT = fne-tune Farmng actvty 1 Seconds Iteratons, k=6, k=1 Restrctons # n JS # n FT No JS 3,5 4 6,11 37, Wth JS 1,5 5 7,45 32,18 5 sze varables Wth JS 1,5 4 6,13 33,6 All sze varables Wth JS 1 4 8,95 3,8 All ndcator varables Wth JS 1,5 4 7,29 34,87 All varables

7 Table 5: Shows how and computaton tme depend on restrctons and number of unts n the fne-tune phase JS = ump-start, FT = fne-tune Farmng actvty 11 Seconds Iteratons, k=6, k=1 Restrctons # n JS # n FT No JS 9,5 5 4,1 29, Wth JS 1,7 4 7,49 3,31 5 sze varables Wth JS 2,5 5 4,74 27,78 All sze varables Wth JS 2, ,82 All ndcator varables Wth JS 2,4 4 2,8 28,8 All varables Fgure 2: Shows how computaton tme (measured n seconds) dvded on number of teratons vares wth number of unts n FT, for farmng actvty (FA) 2, 4, 1 and 11 Number of unts n FT Seconds used dvded on number of teratons, FA = 2 Number of unts n FT Seconds used dvded on number of teratons, FA = Number of unts n FT ,25,5,75 1 Seconds used dvded on number of teratons, FA = 1 Number of unts n FT ,5 1 1,5 2 Seconds used dvded on number of teratons, FA = 11 Table 6 and 7 show how the mputed totals n FA-4 and FA-1, respectvely, satsfy the restrctons on the 12 most mportant varables lsted above All the mputatons were carred out usng full JS and wth k =6 Fve dfferent ways of calculatng have been used For the frst scenaro, all varables are consdered equally mportant (that s, W=1), so that s calculated as the un-weghted sum of the relatve dfferences, and only totals at the mputaton class level are consdered when calculatng (that s, W =) For all other scenaros the weghts for the 12 mportant varables are ncreased to W=1, e these relatve dfferences weght ten tmes as much as the rest n the calculaton of For the thrd scenaro, W =1, whch means that the restrctons for sub-populatons (n ths case, margnals of class and regon) n calculaton s ncluded to a small extent For the fourth and ffth, the W s ncreased to 5 and 9 Notce that n scenaro three fve, the number of mputaton restrctons s ncreased to 84 from 84 n the frst two scenaros The results ndcate that there s a lmt on the number of restrctons that can be satsfed n any gven mputaton class Moreover, the varables wth skewed dstrbutons are more lkely to have problems Here a skewed dstrbuton s ndcated by a small number (or proporton) of donors wth postve 7

8 values It follows that n practce one may need to gve prorty to certan mputaton restrctons, for nstance the varables of the most nterest or the restrctons that have the lowest varances Table 6: Table of results from fve dfferent scenaros of RENI compared to restrcton, for the mportant varables The second column shows number of donors wthn the mputaton class that has a value larger than W = varable weght, W = margnal weght Farmng actvty 4 Varable Donors > Restrcton W=1, W'= W=1, W'= W=1, W'=1 W=1, W'=5 W=1, W'=9 leasng ,,,,, leasng ,,,, -,1 leasng ,1,,,1,11 m_buld ,,,,,2 _fxed ,,,, -,2 _buld ,,,,, m_tractor ,,,,, m_car ,,,,,4 m_mach ,,,,, mach_new ,,,,,1 mach_used ,,,, -,2 sale_mach ,,,,1,6 Table 7: Table of results from fve dfferent scenaros of RENI compared to restrcton, for the mportant varables The second column shows number of donors wthn the mputaton class that has a value larger than W = varable weght, W = margnal weght Farmng actvty 1 Varable Donors > Restrcton W=1, W'= W=1, W'= W=1, W'=1 W=1, W'=5 W=1, W'=9 leasng ,2,,,,23 leasng ,5,6,5,12,27 leasng ,4 -,4 -,4 -,4 -,4 m_buld ,,,, -,4 _fxed ,4 -,5,9,11,14 _buld ,9,,3,7,19 m_tractor ,,,,2,2 m_car ,11,,,, m_mach ,,1,,7,15 mach_new , -,1,,14,17 mach_used ,5,1 -,1,,1 sale_mach ,26,,5,21,25 32 Co-varances of the mputed values It s essental that the mposed restrctons do not damage the consstency property of the NNI for the estmaton of the fnte populaton dstrbutons In partcular, wth regard to the second goal, the mputed data should contan correct co-varances among the survey varables We now compare the varances and co-varances n the statstcal regster obtaned by the RENI and the correspondng estmates by the weghtng method Table 8 shows that the correlatons by and large are very close under the two approaches n FA-1, and the stuaton s very smlar n the other mputaton classes for whch the detals are omtted due to the lmt of space The same concluson holds n terms of the coeffcent of varaton under the two approaches (Table 9) For the co-varaton among ndcator varables, we look at ther cross-tables Table 1 show these are very close under the two approaches n FA-1 Agan, the stuaton s very smlar n the other mputaton classes such that the detals are omtted 8

9 Table 8: Co-varances for RENI (above dagonal) and weghtng (under dagonal) Farmng actvty 1 leasng leasng1 leasng2 m_buld _fxed _buld m_tractor m_car m_mach mach_new mach_used sale_mach leasng,59,24,25,44,37,3,25,37 -,3,12,7 leasng1,59,43,13,47,44,14,1,25 -,1,17,1 leasng2,25,44,12,5,16,11,9,21 -,2 -,2 -,2 m_buld,25,14,14,7,8,35,59,4,13,23,7 _fxed,45,47,5,6,8,5 -,2,14 -,3 -,2 -,2 _buld,37,42,15,8,79,7 -,2,17 -,3, -,2 m_tractor,28,13,12,35,5,8,42,53,17,13,11 m_car,21,9,1,61 -,1 -,1,47,49,11,15,11 m_mach,35,25,24,41,16,18,57,52,28,18,3 mach_new -,1 -,2 -,2,15 -,3 -,3,2,15,32,11,6 mach_used,11,18 -,1,27 -,2,,13,16,17,15,6 sale_mach,9,11 -,2,7 -,1 -,2,12,13,32,56,6 Table 9: Coeffcent of varaton for RENI and weghtng, for all farmng actvtes FA = 2 FA = 4 FA = 1 FA = 11 R-NNI Weghtng R-NNI Weghtng R-NNI Weghtng R-NNI Weghtng leasng,2,19,3,3,97,97,47,47 leasng1,29,29,6,6,191,28,98,95 leasng2,153,149,18,18,836,823,229,226 m_buld,1,1,2,1,84,82,25,25 _fxed,46,46,9,8,173,164,9,94 _buld,3,3,9,9,164,166,79,73 m_tractor,7,7,1,1,43,43,23,23 m_car,12,12,4,4,91,91,48,45 m_mach,1,1,1,1,53,53,28,31 mach_new,25,26,4,3,132,117,88,81 mach_used,36,34,5,5,237,25,66,67 sale_mach,31,31,6,6,225,227,126,134 Table 1: Cross-table for the ndcator varable of the mportant varables for the RENI (above the dagonal) and weghtng method (under the dagonal) Farmng actvty 1 I_leasng I_leasng1 I_leasng2 I_m_bul I fxed I buld I_m_tractor I_m_car I_m_mach I_mach_new I_mach_used I_sale_mach I_leasng I_leasng I_leasng I_m_buld I fxed I buld I_m_tractor I_m_car I_m_mach I_mach_new I_mach_used I_sale_mach

10 4 Summary and dscusson of future works Applcaton of the RENI to the Norwegan Agrculture Survey 26 suggests that the method s capable of fulfllng the trple-goal crteron for the constructon of statstcal regsters The most problematc ssue that we have encountered concerns how well the mputed totals satsfy the mposed restrctons A large number of restrctons can easly be obtaned by method of weghtng However, not all the weghted totals are equally relable One may choose those wth small coeffcents of varaton and/or those that are consdered mportant It may also be helpful to examne systematcally the effectve choce of mputaton restrctons For nstance, gven the number of restrctons to be mposed, whch ones should be chosen n order to obtan n addton the closest agreement of the rest totals? The defnton of the dstance metrc s another ssue one mght look nto For nstance, what s a sutable balance between the absolute and relatve dfferences? Partal mssng/non-response s as common as unt mssng/non-response In general, a donor may not have exactly the same values as the observed ones for the recevers, when observatons are mssng partally It seems natural n such a stuaton that the observed values of the donors should be adusted before they are mputed for the recever Otherwse, the covarance between the varables may be dstorted Ths requres extendng the NNI to partally mssng data More generally, the donor and recever may not exactly match on the x-values, ether Thus, the extenson may have a more general bearng on the methodology 5 References Zhang, L C and Nordbotten, S (28) Predcton and mputaton n ISEE: Tools for more effcent use of combned data sources Proceedngs from UN/ECE Workshop Sesson on Statstcal Edtng n Venna Geneva 28 Chen, J and Shao, J (2) Nearest neghbor mputaton for survey data Journal of Offcal Statstcs, vol 16,