Optimal Coordination of Samples in Business Surveys

Transcription

1 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Optimal Coordination of Sample in Buine Survey enka Mach, Ioana Şchiopu-Kratina, Philip T Rei, Jean-Marc Fillion Statitic Canada New York Univerity Child Study Center Abtract In thi paper we preent a new method for coordination of ample elected according to tratified imple random ampling without replacement (SRSWOR deign et ( denote a ample elected for the firt (econd urvey Sample are aid to be optimally coordinated if their joint probabilitie p(, maximize or minimize the expected overlap The overlap of the two ample, o(,, i the number of unit that and have in common We formulate our problem a a linear programming (P problem whoe objective function i the expected overlap under a joint probability ditribution of the integrated urvey The joint probabilitie are the unknown and the marginal probabilitie of and are the contraint We ue the exchangeability of the SRSWOR deign and reduce the P problem by grouping ample We decribe implementation of the propoed method and compare it with the popular Sequential SRSWOR method Keyword: Sample overlap, Integration of urvey, Survey population update, Tranportation problem Introduction When multiple ample urvey are conducted on overlapping population, it i often required to control the number of unit that the ample have in common, ie the overlap of the ample Coordinated ampling encompae a variety of technique and procedure ued by urvey-taker for controlling the overlap of ample A number of key overlap procedure are dicued and compared by Ernt (999 Sample coordination i needed for different reaon, for example to facilitate data collection, reduce repone burden, or improve the preciion of etimate of change between two occaion Many of the reaon, iue and procedure of ample coordination were dicued at the ICES-II invited eion entitled Coordinating Sampling Between and Within Survey ; ee McKenzie and Gro (000, Ohlon (000, Royce (000 and Ernt (000 The objective of coordinated ampling i to achieve either a higher or a lower ample overlap than if ample were elected independently The term poitive (negative coordination i ued when the goal i to increae (decreae the overlap We propoe to ue linear programming to integrate two tratified SRSWOR deign o that their expected overlap i either maximized or minimized over all poible integrating joint ditribution Thi lead to an P problem known a a Tranportation Problem (TP In Section we mathematically formulate ample coordination of two urvey In Section we et up the optimal coordination of two urvey a a TP, dicu how to reduce it ize, and olve the problem in two tage for tratified SRSWOR deign Section preent an example demontrating the two-tage procedure and it implementation for a population updated for death (deleted unit and birth (added unit A econd example, in Section, illutrate optimal poitive coordination after re-tratification For both example we compare the optimal olution with the one obtained via the Sequential SRSWOR method Coordination of Two Sample We aume two urvey are conducted on two overlapping finite population Thi cover the ituation of two ditinct urvey a well a two different election for the ame repeated urvey when the population ha been updated between the election et denote by S ( S the et of all poible ample in the firt (econd urvey The overlap of the ample S and S, denoted by o (,, i the number of unit that and have in common et { p( and { ( } S } S p define the probability ditribution on the et of ample for the two urvey We eek a joint (, uch that p, ditribution { } S S p(, = p( for all in S, and p(, = p( for all in S Two urvey are aid to be integrated if their joint probability ditribution preerve their repective marginal ditribution The expected overlap E ( o i given by E [ o( ] = o(, p(,, (

2 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Table : Tabular Preentation of Optimal Sample Coordination a a Tranportation Problem o i x = p, for each pair of ample Overlap (, and joint probability ( j t nd urvey ample urvey ample (, (, ( o o o, x (, x (, x (, o o o ij x (, x (, x (, o o o i j x o (, x o (, x o (, p ( i x p ( x p ( x p ( o( K, K K p ( j ( x o( K, x o( K, K x o ( K K, K p p ( p ( ( x p ( K p Two urvey are poitively (negatively coordinated if o p, o, p p (, ( > (< ( ( ( Optimal Sample Coordination Tranportation Problem Our objective i to integrate two urvey o that their expected overlap E ( o i maximized (minimized Thi can be viewed a a TP: Find maximum (minimum of E o, = o, p, [ ( ] ( ( = x ij = p(, S, over all { } { } S X ( ubject to p(, = p(, S, p(, = p(, S and p (, = A tabular repreentation of thi TP i in Table Thu, to achieve optimal coordination of two urvey one need to olve the TP given in ( However, for many practical ituation uch a TP ha too many variable and i too large even for today computer For example, if an SRSWOR ample of n=0 i elected from a population of N=0 unit for the firt urvey, then there would be N n =7,86,8,80 different ample in S Conequently, uing P technique to olve the optimal ample coordination problem could be a viable approach if the ize of the TP in ( could be reduced Reduced Tranportation Problem For tratified SRSWOR deign, the TP can be radically reduced by grouping ample and then uing a two-tage procedure to obtain an optimal olution The group of pair of ample ( o (, mut be uch that the matrix of, within each group i ymmetric a illutrated in Table et P (P denote the lit of unit in the firt (econd urvey population and let C = P P be the et of unit common to both urvey population Denote by c = c the number of unit in C, and imilarly by ( ( c = the number of unit in C Procedure - Stage : Form uper-row c by grouping ample with the ame number of unit from the common part C, ie the ame value of c For each uper-row, p( c = p( { : c( =c} Form uper-column c by grouping ample with the ame value of c For each uper-column, p( = p( { : ( = } c, and define block Create a matrix of block ( optimum o ( c o (, within each block, to be the maximum (minimum Table : Matrix of o (, within a block ( c =, = P ={ u, u, u, d d }, P = { u, u, u, b b } ud ud ud u d ud ud, n=, n =,, u u bb u ubb u ubb u u u u u u Notation: u k = unit in C, d k =unit in P P, b k = unit in P P

3 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Solve the reduced TP: Find joint probabilitie p ( c, for all block that will maximize (minimize E o c, = o c, p c, c, ( [ ( ] ( ( c ubject to p( c, = p( c for all uper-row and p( c, = p( for all uper-column c Procedure - Stage : Within each block, ditribute p ( c pair of ample (, with o (, = o ( c optimum, evenly among the,, the block To illutrate Stage, let u ue the block given in Table and aume that poitive coordination i required Then 6 o o c, = would be pair of ample with (, = ( aigned probability p (, = p( c, 6 remaining pair, p (, = 0 ymmetry ( (, = For each of the Becaue of the block o exactly once on each row and exactly twice on each column, every ample get the ame probability p ( c, 6, every ample get the ame probability p ( c, and hence the two SRSWOR deign are preerved To demontrate the magnitude of the TP reduction, let u conider the following example Example : Survey elect a ample of n=0 from a population of N=0 unit, and Survey elect n =0 from N = Both urvey ue the SRSWOR deign The two population are overlapping with C=7 common unit There are D= unit preent only in P and B= unit preent only in P The 0 0 ample will form uper-row a the poible value of c are 7, 8, 9, or 0, the 0 ample will form uper-column a c could equal 6, 7, 8, 9, or 0, and hence the reduced TP ha only 0 unknown Note that the marginal probabilitie are eaily obtained uing D= C=7 B= P P Figure the hypergeometric : The two overlapping ditribution: population of Example p ( c = C N c n D c n and ( p ( = C N c n n ( The two-tage procedure decribed here give a olution to (, X { } TP = p TP (, S, S, yielding the optimal expected overlap ETP [ o(, ] = Eopt ( o The implementation of thi olution i imple and i illutrated by example in Section and Variability of the Overlap So far, we have dicued optimization of the expected overlap In practice only one pair (, i elected and o, of the elected pair hence it i very deirable that ( i cloe to E ( o The likelihood of thi depend on the variance of the overlap V ( o given by: V [ o(, ] = { o(, E( o } p(, (6 It can be proved that, for maximization, all optimal olution have the ame variance V ( o and, for minimization, our two-tage procedure yield the minimum variance within the et of joint probability ditribution that are olution to ( To compare a uboptimal olution p * to the optimal olution, one could conider the mean-quare-error type meaure MSE *[ o(, ] = { o(, E ( o } p *(, (7 p opt Example : Negative Coordination We will ue Example introduced in Section to illutrate optimal negative coordination of two tratified SRSWOR deign when both urvey population have the ame tratum definition and the econd population ha been updated for death (unit in P but not in P and birth (unit in P but not in P In thi cae, we obtain an optimal olution eparately for each tratum The two overlapping population of Example, illutrated in Figure, repreent one tratum Two-Stage Solution for Example Procedure - Stage : Form four uper-row by grouping ample with the ame value of c, c =7, 8, 9, 0 Ue ( to calculate the marginal probability p(c for each uper-row

4 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Form five uper-column c by grouping ample with the ame value of c, c = 6, 7, 8, 9, 0 Ue ( to obtain p ( for each uper-column Order uper-row by acending c and upercolumn by decending c to create a matrix of block The block optimum i the minimum o (, within each block: o( c, = max{ 0, c + C} Solve the reduced TP: Find joint probabilitie p ( c, for all block that minimize E o c, = o c, p c, c, [ ( ] ( ( c ubject to p( c, = p( c, c =7, 8, 9, 0, and p( c, = p(, c = 6, 7, 8, 9, 0 c In thi cae, becaue the matrix of o ( c, ha very pecific propertie, an optimal olution can be obtained uing the Northwet Corner Rule (NWCR, a imple algorithm well known in optimization (Hoffman 98 The NWCR olution for Example i given in Table The reader can find more detail about optimal ample coordination via NWCR in Mach, Rei, and Şchiopu- Kratina (006 Procedure - Stage : Within each block, ditribute p ( c, evenly among the pair (, with o (, = o c, ( For example, conider block (7, 0 with o ( c, = 0 In thi block, there are: =,90,68,70 different ample (row =,90,68,70 different ample (column For each ample, there i exactly one ample uch that o (, =0, and vice vera, for each, there i exactly one uch that o (, =0 Hence the block ha the deired ymmetry and, by aigning probability of 009 to each pair with o (, =0, each,90,68,70 p c, and will get the ame hare of ( Table : Reduced TP for Example p c, aigned by NWCR Block optima o ( c, for minimization and joint probabilitie ( Super-row Super-column labeled by c labeled by c p ( p (c Table : Empirical block probabilitie for equential SRSWOR (baed on imulation with 0,000 repetition Super-row Super-column labeled by c labeled by c p ( p (c

5 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Implementation Once the olution of the reduced TP i obtained, the ample can be elected, either imultaneouly (both and at the ame time or equentially ( elected firt, elected econd, conditionally on Simultaneou Selection: Select one block uing the joint probabilitie given in Table (Thi can be done uing any oftware that ha method for unequal or proportional-to-ize ampling Given the block elected in, randomly elect the required number of unit from each et: C (common unit, D (death and B (birth Suppoe block (9, 8 with p ( c, =008 i elected at the firt tage To elect, randomly elect 9 of the 7 unit in C and of the unit in D To elect, take the remaining 8 unit from C and randomly elect of the unit in B Sequential Selection: Given ample drawn for the firt urvey, elect one block from the uper-row c ( uing the conditional probabilitie p {( c, c( } = p( c, p{ c( } calculated from Table Given the block elected in, randomly elect the required number of unit C and B to form Comparion with Sequential SRSWOR Method In thi ection we compare our method with the equential SRSWOR method that i often ued by urvey taker to coordinate tratified SRSWOR deign Thi method wa developed by Fan, Muller, and Rezucha (96 a a fat computer technique for electing an SRSWOR ample Note that here the adjective equential refer to electing a ample unitby-unit, which i different from the term equential election ued in Ohlon (99 decribed the ue of equential SRSWOR for ample coordination Table : Ditribution of o ( (, o NWCR p ( o ( o, for Example Sequential SRSWOR p o p( o o p( o The method belong to the cla of permanent random number (PRN method Briefly, all unit on the frame are independently and permanently aigned random number from the uniform ditribution on [0, ] Unit correponding to the firt n ordered PRN encountered when one tart from a point a [0, ] and move in a pecified direction (right or left are included in the ample Ohlon (99 proved that thi technique yield the SRSWOR deign Some propertie of the PRN method, uch a the expected overlap, are difficult to derive theoretically We undertook a imulation tudy to compare the empirical expectation and variance of o (, for equential SRSWOR with the value given by the NWCR method To negatively coordinate the two ample via the PRN method, we elected unit from P with the firt 0 ordered PRN, tarting at 0 and going right, and unit from P correponding to the firt 0 ordered PRN, tarting at and going left The aignment of PRN and the election of and wa repeated 0,000 time Within each block, the equential SRSWOR method,, with like our method, elected only the pair ( o (, = o ( c, Thu inight into how the method differ can bet be gleaned by comparing the block probabilitie p ( c, preented in Table and Table For Example, the PRN method aigned non-zero probabilitie to all 0 block while our optimal olution aigned non-zero probabilitie to 8 block with o ( c different ditribution of o (, =0 Thu the two method yield very,, a hown in Table The expected overlap and variance are zero for the o =076 and optimal olution, compared with E PRN ( V PRN ( o =0 Example : Poitive Coordination after Retratification Now we look at the ituation in which the tratification of the two population i not the ame and poitive coordination i required A typical example would be electing a ample for a repeated urvey, after it population ha been re-tratified, for intance due to change in indutrial claification or ize tratum boundarie It i important to minimize the impact of uch change on the urvey proce and etimate The following example how how an optimal olution i obtained and implemented 7

6 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Example : Conider one tratum of Survey, which we will refer to a the new tratum and denote by P Suppoe that it contain N = unit that come from different old trata P h, h =,,, each contributing N h unit to P et h denote an SRSWOR of P h, elected in Survey, h =,, n h out of, and N h unit in = Suppoe of n = unit i to be elected from the new tratum via SRSWOR Figure diplay the overlap of the old trata with the new tratum a well a the ize of the different ubet and ample et C h = P h P, denote by h ch ( h unit in C h h and, imilarly, by ch = ch ( number of unit in C h, for h =,, Two-Stage Solution for Example c = the number of the Procedure - Stage : Independently within each old tratum P h, group ample h with the ame value of c h The poible value are: c =0,,, c =0,,,, and c = Each different combination of thee three et of group form a uper-row and hence there are uper-row, each labeled c = ( c, c, c Ue a product of hypergeometric probabilitie to calculate p (c : C N C C N C c n c c n p( c = (8 N N n n c c, c, Form a uper-column labeled = ( c for each poible combination of h value uch that Old tratum P : N =0, n =0 Old tratum P : N = 6, n = Old tratum P : N =0, n = New tratum P : N =, n = C : N = C : N = C : N =0 Figure : Old trata overlapping new tratum in Example ch = n =; there are uper-column Ue h= multi-hypergeometric probabilitie to obtain p ( : C C C p( c = N n (9 Create the x matrix of block The block o, within each optimum i the maximum ( block: o ( c, = min( c, h= Solve the reduced TP: Find joint probabilitie p ( c, for all block that maximize E o c, = o c, p c, c, [ ( ] ( ( c ubject to p ( c, = p( c h h for all uper-row, and p ( c, = p( for all uper-column c The SAS/OR oftware olve a variety of optimization problem Two of the SAS/OR procedure, PROC P and PROC INTPOINT, can be ued to olve a TP Uing PROC P, we obtained different optimal olution for different ordering of the uper-row and uper-column In each PROC P olution, about out of block were aigned a poitive probability and the remaining block zero probability On the other hand, PROC INTPOINT yielded an optimal olution with 60 poitive block probabilitie All optimal E o = E o and have the ame olution reult in ( opt ( ( o V but yield variou conditional propertie on uperrow We plan to invetigate thee difference between the optimal olution in the future Procedure - Stage : Within each block, ditribute p ( c, evenly among the pair (, with o (, = o ( c, c =,,, =,, with Conider block { ( ( } o ( c, = In thi block, there are: =,78 x x different ample (row, - 0 = x x different ample (column In each row, there are exactly three ample uch o, =, and in each column, there are that ( exactly,78 ample with o (, = Hence the block ha the required ymmetry a hown in Table 6 8

7 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Table 6: Matrix of o (, within the block { c = (,,, = (,, } ,78 row,78 row Implementation A in example of Section, given a olution to the reduced TP, the ample for the two urvey in Example can be elected either imultaneouly or equentially We will illutrate the latter ince thi would occur more commonly in practice Suppoe the Survey ample belong to the uperrow labelled c = (,, Table 7 lit the block and their joint probabilitie p ( c, on thi uper-row aigned by SAS PROC P Sequential Selection: Ue the conditional probabilitie in the lat row of Table 7 to elect a block Suppoe uper-column c = (,, wa elected in Stage To form, keep the two unit from C and the two unit from C and randomly elect one unit from the three in C Comparion with Sequential SRSWOR Method Now we compare the propertie of the two-tage optimal olution with the empirical propertie of the equential SRSWOR method, baed on 00,000 repetition In our imulation, we elected unit in both the old trata and the new tratum from a lit ordered by PRN, tarting at 0 and going right Again, we oberved that within each block, equential SRSWOR elect only the pair (, with o (, = o ( c,, exactly like our method However, the block probability ditribution are quite different, with the PRN method electing pair (, from mot of the block Table 7: Reduced TP for Example Super-row c = (,, Block optima o( c, for maximization and joint probabilitie p( c, aigned by SAS PROC P,,,, 0,,,,,,,,0,, 0,,,0, 0,,,0, 0,0, ( c, ( c, {( c, c} o p p Table 8: Empirical block probabilitie for equential SRSWOR Super-row c = (,, (baed on imulation with 00,000 repetition,,,, 0,,,,,,,,0,, 0,,,0, 0,,,0, 0,0, ( c, ( c, {( c, c} o p p

8 Paper preented at the ICES-III, June 8-, 007, Montreal, Quebec, Canada Thi i demontrated in Table 7 and Table 8, which give the block probabilitie on the uper-row c = (,, for our optimal olution and equential SRSWOR repectively The two ditribution of o (, and their expectation are hown in Table 9 The optimal olution yield a higher expected overlap, but the difference may eem too mall to be practically ignificant However, mot buine urvey have hundred of trata, and over all trata, the expected overlap could be coniderably larger for the propoed method Alo conditionally, given uper-row c = (,,, the optimal olution ha much better propertie that the equential SRSWOR method 6 Concluion Optimal coordination of two urvey can be viewed a a TP and olved uing an P algorithm Unfortunately, in many practical ituation uch a TP would be too large to handle In thi article, we how that, for tratified SRSWOR deign, we can reduce the ize of the TP problem by grouping ample; the optimal olution i thu obtained in two tage We alo illutrate that thi two-tage optimal olution i eay to implement The tatitical propertie of our olution compare favourably with thoe of the popular equential SRSWOR method Acknowledgement The author would like to thank Pierre avallée and Dave MacNeil for reviewing the manucript and providing valuable comment and advice Reference Ernt, R (999, The Maximization and Minimization of Sample Overlap Problem: A Half Century of Reult, Bulletin of the International Statitical Intitute, Proceeding, Tome VIII, Book, pp 9-96 Ernt, R (000, Dicuion of Seion : Coordinating Sampling Between and Within Survey, ICES II, Invited Paper, American Statitical Aociation, pp 6-67 Fan, C T, Muller, M E, and Rezucha, I (96, Development of Sampling Plan by Uing Sequential (Item-by-Item Technique and Digital Computer, Journal of the American Statitical Aociation, 7, 87 0 Hoffman, A J (98, On Greedy Algorithm That Succeed, in Survey in Combinatoric, ed I Anderon, Cambridge, UK: Cambridge Univerity Pre, pp 97 Sample via the Northwet Corner Rule, Journal Table 9: Ditribution of o ( Optimal Solution by PROC P, for Example Sequential SRSWOR (, p ( o p ( o o Expectation E ( o V ( o ( o c =,, E 8 ( o c =,, V Mach,, Rei, PT, and Şchiopu-Kratina, I (006, Optimizing the Expected Overlap of Survey of the American Statitical Aociation, Vol 0, pp McKenzie, R and Gro, B (000, Synchronized Sampling, ICES II, Invited Paper, American Statitical Aociation, pp 7- Ohlon, E (99, SAMU, a Sytem for coordination of Sample From the Buine Regiter at Statitic Sweden: A Methodological Decription, Reearch and Development Report 99:8, Statitic Sweden Ohlon, E (99, Coordination of Sample Uing Permanent Random Number, in Buine Survey Method, ed B G Cox, D A Binder, D N Chinnappa, A Chritianon, M J Colledge, and P S Kott, New York: Wiley, pp 69 Ohlon, E (000, Coordination of PPS Sample Over Time, ICES II, Invited paper, American Statitical Aociation, pp -6 Royce, D (000, Iue in Coordinated Sampling at Statitic Canada, ICES II, Invited Paper, American Statitical Aociation, pp - 0