MODIFICATION OF FRIEDMAN-RUBIN'S CLUSTERING ALGORITHM FOR USE IN STRATIFIED PPS SAMPLING

Transcription

1 MODFCATON OF FREDMAN-RUBN'S CLUSTERNG ALGORTHM FOR USE N STRATFED PPS SAMPLNG Donna Kostanch, Davd udkns, Rajendra Sngh, and Mnd Schautz U.S. Bureau of e Census. NTRODUCTON Major research actvtes are currently underway at e Census Bureau to mprove e qualty of e Current Populaton Survey (CPS). Obtanng a meod whch reduces e frst-stage component of varance s a hgh prorty project at has been and s stll beng nvestgated. Ths paper deals w some of e research conducted us far on e use of a partcular clusterng algorm to acheve s varance reducton. The followng secton gves a bref dscusson of e survey's objectves and e reason for e concern n reducng s component of varance. Secton of s paper dscusses e general form of s varance component as t relates to bo e desgn of e CPS and a general clusterng stuaton. Also presented n s secton are e modfcatons made to Fredman and Rubn's clusterng algorm (see {2}) for e above stated purpose. A descrpton of e computer program at has been created to satsfy our modfcatons along w oer related optons s gven n Secton V. Emprcal results on e effect of ese modfcatons under varous realstc samplng stuatons are shown n Secton V.. BACKGROUND One of e major purposes of e CPS s to produce labor force statstcs at bo e natonal and state levels w e prmary objectve of obtanng relable estmates on e number of unemployed. The emphass of e CPS had recently, durng e latter part of 1978, shfted towards producng more relable unemployment estmates at a substate level. For a more n-dep dscusson on e desgn and objectves of e CPS durng e last decade see {4}. More recently (.e. early 1981), e am of e CPS has shfted back to obtanng relable state estmates. Ths shft was caused by budgetary problems. Addtonally, ere has been consderable nterest n mnorty estmates. The above mentoned changes n objectves, especally e antcpated shft towards substate estmates, had created concern over e frst-stage contrbuton of varance. For natonal estmates, s component of varance due to e selecton of prmary samplng unts (PSUs) s relatvely small compared to e total varance. Ths proporton ncreases for mnorty estmates at e natonal level. For some states, however, and even more so for some substate areas, s component of varance can be qute large. Therefore, major research s underway n order to reduce e between-psu varance for several varables at e state level.. GENERAL The between-psu varance n e CPS results from e selecton of one sample PSU per stratum w probablty proportonate to sze (PPS). n e current CPS desgn, PSUs are generally defned to be countes or a contguous group of countes w e excepton at mnor cvl dvsons form PSUs n some noreastern states. The results presented n s paper always consder sngle countes as PSUs. Ths component of varance for estmated levels w a gven characterstc s gven by e followng formula: g nh Ph...Ph \\ 2 (~" Var B = h=z = ---- Ph ---- Ph U h - U h ) x / where g = e number of strata n~ = e number of PSUs n e h L stratum Ph ~ = t~ populaton of e PSU n e h" stratum Ph = e populaton of e h stratum / \ /oe. Ph = zh Ph ),, =l Uh = e actual number of persons w att~ertan charac~t~rstcs n e PSU of e h "" stratum and U h = e number of persons w~h characterstc n e h / rl h \ [,.e. Uh = = Uh ) a certan stratum The above formula () can be rewrtten n terms of proportons by lettng Xh Uh / Uh = / Ph and X h = <. Ths gves: g nh (2) Var B = E Z Ph Ph \ fxh - Xh \2 h=l =l \ / Consequently, e purpose of s study s to produce a stratfcaton procedure at smultaneously reduces (2) for several key labor force statstcs for a gven number of strata. Alough g s not actually known, t can be assumed to be fxed snce ts value depends heavly on e desred stratum workload whch s fxed and dependent on e sze of e stratum. Oer factors ncludng relablty requrements and e value of Var B acheved for unemployed also affect e value of g. n attempts to accomplsh e above, a clusterng algorm suggested by Fredman and Rubn n {2} was used as a bass for reducng s frst stage varance component for several varables. n general, er clusterng algorm attempts to optmze a crteron functon for a fxed number of clusters. One of e crteron functons suggested n {2} to be mnmzed s e wn cluster sums of squared errors for several varables, whch s referred to as e trace W crteron. Note at W s defned to be e wn cluster dsperson matrx. Usng notaton defned above, an element of W, wj,j, s defned as: g n h ~, _ w.j,j., = h=z E l \Xh j - ~Xh j, - Xhj,, 285

2 for j, j' =, 2,... p where e addtonal subscrpt j s for desgnaton of e P varables and _ / n X Xhj = Z h Xhj~ /n h ', =l Thu s, P g n / _ \2 (3) Trace W = Z Z Z h ~\Xh j - Xhj ) j=l h=l =l j Note at f e between PSU varance gven n (2) was summed over e key labor force statstcs, s functon would be very smlar to at gven n (3) above. The only dfferences are e Ph and Ph terms and manner n whch e cluster means are computed. Thus, t seems approprate to modfy (3) to obtan a sum of varances n (2) to use as e crteron functon to mnmze n order to reduce e between-psu varance. Ths new crteron functon wll be defned as: P (4) Betvar = j_z_ VarB, where VarB,; s e quantty stated n formula () above for e j varable and p s e number of varables to be used n e clusterng algorm. Ths appears to be a worwhle addton because of e effect of dfferng PSU szes (Ph s) on bo e squared devatons and e clusters means. Ths modfcaton of trace W was en used as e crteron functon along w parts of Fredman and Rubn's clusterng algorm to form a stratfcaton algorm. t was later dscovered at an dentcal crteron functon had been used w a very smlar clusterng algorm for samplng purposes (see {}). The algorm specfed n {2} actually conssts of ree separate passes; a hll clmbng pass, a forcng pass and a reassgnment pass. The hll clmbng pass examnes each object (PSU) one at a tme and moves s object to e group at produces e most mprovement n e crteron functon. f no mprovement occurs e object remans n ts current cluster. A one move local mnmum occurs when an entre pass of e objects produces no moves. The oer two passes are heurstc procedures whch attempt to obtan a near global mnmum. The forcng pass attempts to optmze e crteron by placng groups of objects from one cluster nto e oer clusters. For e reassgnment pass e algorm consders movng all objects nto dfferent clusters at one tme. Due to e lmted avalablty of programmng resources, e effectveness of ese two passes was evaluated usng e wn cluster sums of squared errors gven n formula (3) above. These results, presented n {3}, show at a number of ntal starts wll produce results smlar to ose obtaned usng e forcng and reassgnment passes. Therefore, t was decded at programmng resources could be more effcently utlzed by ncorporatng random ntal starts and oer modfcatons nto e program raer an usng ese two passes. The next secton of s paper descrbes more oroughly e program created for e stratf- caton algorm ncludng oer addtons at have been made to assst n s research as t apples to samplng, V. STRATFCATON PROGRAM Ths stratfcaton program, developed at e Census Bureau, bascally conssts of e hllclmbng pass n {2} and a crteron functon (4) for between-psu varances. Oer optons have also been ncorporated nto s program for e prmary purpose of formng strata for PPS samplng, However, s program can be used for e more conventonal clusterng problem due to a number of optons avalable to e user. The program s run on a Unvac 1100/44 machne. The program sze s 39-43k, w 'k' beng defned as bt words. The sze of e program depends on e optons ncorporated n a partcular run. The program language s ALGOL. The clusterng program can be altered rough e selecton of avalable optons, whch nvolve usng alternate peces of code n e program. The followng are e optons at alter e clusterng program tself: ) OPTON FOR MAXMUM/MNMUM CLUSTER SZE CON- STRANTS. Any reasonable maxmum and mnmum cluster szes can be assgned, whch are referred to as sze constrants. Sze constrants have been ncorporated nto e algorm n order to balance ntervewer workloads. Only one mnmum and two maxmum sze constrantsmay be specfed. One maxmum sze constrant can be used to exclude unusually large elements or PSUs, whch wll en be sampled as self-representng PSUs, and e oer can be used as e maxmum sze constrant n e clusterng algorm. 2) OPTON FOR NTAL ALLOCATON OF PSUs TO CLUSTERS. The followng are e ree avalable procedures for assgnng e ntal allocaton of e elements or PSUs to e clusters: a) Ths allocaton s defned by an external procedure (such as ntuton, prncpal components, graphng, etc.). The number of elements to be n each cluster, nh, s also defned by e external procedure. A cluster code s placed on each record and e fle s en sorted usng s feld, The frst n elements are put nto e frst cluster and e next n 2 elements are placed nto e second cluster, etc. b) Anoer meod s to frst sort e fle by e predomnant varable(s), such as unemployment. The ntal allocaton s determned by formng approxmately equal szed clusters from s sorted fle. c) A random clusterng of e elements may also be acheved by usng a procedure at ncorporates a random number generator. The elements are randomly placed nto clusters so at e clustems meet any reasonable mnmum or maxmum sze constrants. Frst, PSUs or elements are assgned to meet e mnmum sze constrant. f, after e mnmum sze constrant s met, some elements are not yet assgned to a cluster, en e procedure attempts to randomly assgn all remanng elements to a cluster. f at any tme a clusterng scheme cannot be devsed such at all elements are ncluded n one of e groups and all clusters satsfy e sze constrants, e process s stopped and started agan by randomly 286

3 assgnng all elements. The number of random starts to use s left up to e user. 3) OPTON FOR CRTERON FUNCTONS. The avalable functons to be mnmzed are: a) Trace W; suggested by Fredman and Rubn and also gven n (3) b) Betvar; functon of varances gven n (4) c) Determnant W'; W' s specfed n Secton V. The Determnant W' opton s smlar to e Determnant W crtera gven n {2} but ncorporates e PSU szes nto e wn cluster dsperson matrx W. Some results showng e applcablty of s functon as a crteron to mnmze are ncluded n e next secton of s paper 4) OPTON TO STANDARDZE VARABLES. The varables beng used to determne e clusters may be standardzed by one of e followng meods (no standardzaton s also an opton): a) probablty proportonate to sze scalng. For s opton e scalng factor, sf;, appled to all observatons on e j varable, s determned by],. max P1 P (Xl _ Xl )2 sf. = k V =l l,k,k / N \ -E- P Pl (Xl,j -Xl,j)e for j and k =, 2,... p where p s e number of varables, N s e total number of PSUs or elements and all PSUs are assumed to be n one cluster (.e. h = mples at Y n h = N and E Ph = P )" Note at P~ s e populaton of e entre NSR area. b) equal sze scalng. For s opton all observatons or elements are consdered equally mportant n determnng e scalng factors sf ' appled to all observatons on e j These factors are obtaned by: max ~ ~ (- _ ~, )2 k =l Xl,k k sf. = N ~ =~(Xl,j - X,j)2 varable.. where j, k, p and N are defned above n a). Scalng meods a.) and b.) above, respectvely, equalze e between-psu varance or e sums of squares for all varables gven at all elements are n one cluster. 5) OPTON TO CHOOSE NUMBER OF CLUSTERS. The number of clusters to be formed s determned by e user; however, e actual number of clusters formed may be nternally reduced by: a) e removal of any ndvdual element whose sze (Ph) s greater an e maxmum allowable cluster sze; b) e creaton of an ntal empty cluster rough e random starts procedure. Ths was allowed n order to avod complextes, snce an empty cluster would not meet e mnmum sze constrant. The possblty of s occurrng s very slght f one chooses a reasonable number of clusters and reasonable sze constrants. 6) OPTON TO CHOOSE PREFERENCE FACTORS. Any preference factors or varable weghts may be assgned to each varable. f factors are n- dcated here, ey are ncorporated nto e Betvar crteron functon n order to weght e contrbuton from some varables more an oers. Consequently, s forces a greater reducton n between-psu varance for e more heavly weghted varables as determned by e user. f e varables were not standardzed ntally, e contrbuton of each varable to e Betvar crteron would dffer. These contrbutons would be dependent on e general magntude of e varances raer an e mportance of e actual varable. f s opton s not specfed, all preference factors are assumed equal to one. V. EMPRCAL RESULTS Some prelmnary results showng e effectveness at e above mentoned modfcatons have on reducng e between-psu varance are presented n e attached tables. Strata were formed wn ree test states w PSUs beng defned as countes. Not all countes n a state were ncluded n s testng; only ose consdered to be nonself-representng (NSR) were used to form strata. Countes excluded were assumed to be self-representng and were desgnated as such snce er populaton would yeld a sample large enough to support an ntervewer. For example, countes ncluded n e Pttsburgh SMSA were not stratfed and would be n sample w certanty. These ree states were arbtrarly selected for testng as ey seemed to be somewhat representatve of e country. For each state e algorm was tested usng four key labor force varables, whch are mportant n e CPS. Each of ese states ncludes e number of unemployed and e number n e cvlan labor force (CLF) as varables and 2, respectvely, and e resultng between-psu varances are desgnated by VarB,l and Var B,2. Varables 3 and 4 represent smlar mnorty characterstcs of unemployed and CLF respectvely, where Spansh was used n and blacks n and. The number of strata formed n each state s an approxmaton of e number at would be needed to satsfy e antcpated relablty requrements on e state estmate of e number of unemployed for e redesgned CPS. The NSR PSUs n each of ese states were clustered nto strata usng e followng crtera: 1.) Betvar j=l VarB, 4 g n 2.) Trace W = j=l r.. h=l r. =l r'h (Xhj - Xhj ) 2 3.) Determnant W'; where W' can be wrtten as: ~ -VarB, Cov B,1,2 Cov B,1,3 C VB, 1 4-] " C VB,1, 2 VarB, 2 C VB,2, 3 C VB' 214 C VB, 1,3 C VB,2,3 VarB,3 C VB, 3, ~ 1 C VB, 1, 4 C VB,2,4 C VB,3, 4 VarB, 4 by defnng e covarance terms as: C VB,j,j ' = h~l Ph Ph (Xhj-Xhj )(Xh j '-~j,) 287

4 4 4.) Betvar = j_e_ Pfj VarB, j where e Pf s represent preference factors ds- cussed n secton V of s paper, for whch e followng values were assgned: Pfl = 2 and Pf2 = Pf3 = Pf4 = Ths was tested n order to examne e effect of weghtng e most mportant labor force varable, unemployed, more heavly. 5.) Betvar = Var B, 6.) Betvar = Vat B,2 7.) Betvar = Vat B,3 8.) Betvar = Var B,4 Note at ese last four crtera produce a stratfcaton based on only one of e four key labor force varables and e results gve some ndcaton of e amount of reducton n between- PSU varance at can be acheved for each varable. Each of e above eght crtera were used to cluster NSR PSUs nto strata n each of e ree test states. Also for each crteron and state a stratfcaton was produced by mposng ree types of sze constrants on e strata, none, loose and tght. Three random starts were mplemented for each type of sze constrant and e resultng average between-psu varances, Va--~ B,j' were obtaned and are presented n e attached tables. Note also at for a gven sze constrant, dentcal ntal starts were used n clusterng e PSUs w each of e eght crtera. For all stratfcatons e actual varables were frst standardzed w scalng factors determned by e probablty proportonate to sze meod dscussed n secton V of s paper. Tables, 2, 3 and 4 present e average between-psu varances at were obtaned from e frst four crtera gven above, respectvely. The average between-psu varances resultng from stratfyng on each varable separately (.e. crtera 5.)-8.) above) are shown n Table 5. For purposes of comparson, e percent reductons n between-psu varance for each varable are also gven n ese tables. These percent reductons are based on what e between-psu varance would be f e NSR PSUs were not stratfed and g PSUs were selected w probablty proportonate to sze w replacement. These nonstratfed between-psu varances, Var~ were calculated for each test state by: '' N _-E.l Pl P1 (Xlj-Xj)2 VarB,j = g for j = 1,2, 3 and4 where N represents e number of NSR PSUs n e state. Chart A summarzes e nonstratled between-psu varances for each state and varable and also gves oer stratfcaton parameters used n mplementng e algorm. The effect of ncorporatng probabltes of selecton n e clusterng crteron can be seen by comparng e results n Table w ose n Table 2. Greater reductons n between PSU varances occurred w e Betvar crteron for all states and all varables when no sze constrants were mposed on e strata. n fact, n one nstance e Trace W crteron produced an ncrease n e between-psu varance for a partcular varable. Ths happened n to varable 2, CLF, when no sze constrants were mposed; as can be seen from e negatve percent reducton n Table 2. When sze constrants are mposed on e strata, generally e Betvar crteron produces gans over e Trace W crteron alough ese gans are smaller. n and, w tght sze constrants, one of e varables, varable 3 - Spansh unemployed n and varable - unemployed n, acheved greater varance reducton w e Trace W crteron; however, for all varables concerned e Betvar crteron produced a better stratfcaton. For, when loose sze constrants were mposed e Trace W crteron produced an overall better stratfcaton (.e. ree varables acheved a greater varance reducton). However w tght sze constrants half e varances were smaller usng e Trace W crteron e oers were reduced more w e Betvar crteron. Ths prelmnary comparson shows qute strongly at e modfcatons made to e Trace W crteron wll produce sgnfcant reductons n e frst stage varance component when no sze constrants are mposed. t also appears at n general and under certan stuatons smaller gans n varance reducton can be acheved even w sze constrants on strata. The varances for stratfcatons produced by mnmzng e Determnant W' are shown n Table 3. The applcablty of s crteron to stratfcaton s of nterest snce t s a functon of bo between-psu varances and wn cluster covarances. By comparng ese results w ose obtaned by mnmzng only e between-psu varances, (see Table ), t appears at e wn cluster covarances do not n general mprove e stratfcaton. Most cases where greater varance reductons are acheved w e Determnant, e Betvar crteron does not produce substantally larger varances. However, an unusual stuaton dd occur for unemployed n, smaller varances were consstently obtaned from e Determnant for all ree types of sze constrants. Also ncluded n s study s an example of e effect of preference factors, pf~ w s, on e stratfcatons. As prevously dscussed e pf.'s are ncorporated n e Betvar crteron 3 n order to weght partcular varables more heavly an oers. Ths example smply weghts e VarB, (varance for unemployed) twce at of e oers. As antcpated, greater reductons n e between-psu varance for unemployed (.e. see Tables and 4) dd occur, but by dfferng amounts. The decreases were generally substantal, partcularly when no sze constrants were mposed. t appears at e between-psu varances for e oer ree varables can also ncrease consderably. The results presented n Table 5 somewhat ndcate e amount of varance reducton at s possble for a varable under varous condtons, snce ese stratfcatons were based on only one varable. t should be noted at s procedure wll not produce e mnmal 288

5 varance because results are somewhat dependent on e ntal clusterng due to e fact at e algorm only produces a local mnmum. However, ese varances can serve as a bass for determnng e effectveness of a multvarate stratfcaton. Also wory of menton s at greater reductons n between-psu varances can be acheved f strata szes are allowed to vary. Ths can be seen n all e stratfcatons produced, but s especally emphaszed when only one stratfcaton varable s used (.e. Table 5). Alough, e results presented n s paper are not conclusve, research s currently focused on ways to acheve greater varance reducton when sze constrants are mposed on strata. Ths s beng nvestgated under e assumpton at e Betvar crteron along w preference factors wll be e most practcal way to form strata for e CPS. FOOTNOTE Some of e average between-psu varances for were obtaned from only two random starts. BBLOGRAPHY {} DahmstrSm, P. and Hagnell, M. "Multvarate Stratfcaton of Prmary Samplng Unts n a General Purpose Survey." Scand.. Statst. 1978: 5, {2} Fredman, H. P. and Rubn,. (1967). "On Some nvarant Crtera for Groupng Data." ournal of e Amercan Statstcal Assocaton {3} u--~ns, D. R. and Sngh, R. P. "Usng Clusterng Algorms to Stratfy Prm~y Samplng Unts." Presented at e 141-'" Annual Meetng of e Amercan Statstcal Assocaton, August {4} Moore, T.; Bettn, P.; Kostanch, D.; and Shapro, G. "Overvew of Current Populaton Survey Sample Desgn." Presented at 139 Annual Meetng of e Amercan Statstcal Assocaton, August Chart A Summary of Stratfcaton Parameters and Nonstratfed Varances --~-~- ~ State Parameter... ~--_~.._._..._=~ Number of PSUs(N) Number of Strata(g) Vary, (x 105 ) Vary, 2 (x l0 S ) Vary, 3 (x l0 s ) Vary, 4 (x l0 s ) Scalng factors: sf sf 2 sf 3 sf , , t , , , , State and Sze Constrants Table. Reductons n Varance Due to Stratfcaton (Based on Betvar / Crteron) ~'~B, ~/ (% Reducton) -~B,2 ~/ (% Reducton) V-~rB,3~/ (% Reducton) V'-~B,4 ~/ (% Reducton) (62%) (65%) (41%) (32%) (36%) (34%) 3.62 (76%) 6.69 (55%) 6.35 (58%) (78%) (64%) (52%) (83%) (84%) (56%) (81%) 9.32 (93%) (76%) 9.55 (92%) (61%) (75%) (88%) (87%) 1, (72%) (85%) (61%) (53%) 8, (81%) 13, (67%) 16, (61%).71 (92%) 4.35 (52%) 4.35 (52%) (96%) 1, (52%) 1, (52%) 4 Ths crteron s defned as: Betvar = Y~ Var B j=l ' ~/ The average between-psu varances are x

6 18.91 (82%) (75%) (78%) (68%) (59%) (50%) Table 2. Reductons n Varance Due to Stratfcaton Based on Trace W Crteron) "State and Sze Constrants ~'~B,-- /, Reducton 1~'~-arB, 2-/ (% Reducton) ~T~B,3~/ (% Reducton)=~ B 4~/(% - Reduct~: (% t (45%) 1, (-0%~ 4.98 (67%) } (71%) (35%) (28%) 8.58 (43%) (42%) (31%) (33%) 6.02 (60%) (48%) MsssspP (80%) 8, (80%) (75%) 9, (78%) [ (48%) 12, (71%) _/ The average between-psu (90%) (84%) (90%) (85%) (75%) 1, (68%) 1.75 (81%) (89%) 2.25 (75%) (64%) 3.94 (57%) 1, (50%) l State and Sze ~ / (% Reducton) '' VarB,2-- / (% Reducton)... ~ VarB, 31/ -- (% Reducton) _ / (% Reducton Constrants arb,1-- VarB, (56%) (30%) (41%) r80%) %~ (58%~ (87%~ (67%) (64%)... / The average between-psu varances are x 105. Reductons n Varance Due to Stratfcaton (Based on Determnant W" Crteron) ,62 (49%) 7.30 (51%) (37%) 6.29 (58%) (28%) 7.68 (49%) (66%) (86%) (58%) (80%) (47%) (68) 10, (76%~ 1.01 (89%) 16, (60%) 4.39 (52%) 15, (63%) 4.41 (51%) (60%) (65%) (55%) (84%) 1, (78%) 2, (55%) (95%) 1, (50%) 1, (50%) Table 4. Reductons n Varance Due to Stratfcaton (Based on B e t vat-- / Crteron w Preference Factors) sta... d Sze W- 2/ (% Reducton~ ' ~~ 2/ C% Reducton) VarB,3 ~/ (% Reducton) -V~B,4~/ (% Reducton) Constrants arb, - arb, (78%) (50%) (44%) 8.74 (92%) (85%) (75%) 7O7.98 (39%) (23%) (25%) L (70%) (60%) 1, (33%) (91%) 111, (73%) (72%) ~ 17, (59%) (61%) 24, (42%) ~ (75%) 7.19 (52%) 7.37 (51%) (91%) (82%) (70%) (88%) 5.35 (41%) 4.52 (50%) (74%) (58%) (48%) (85%) 1, (69%) 2, (45%) (95%) 1, (48%) 1, (50%) 4 / Ths crteron s defned as: Betvar = ~ pf~ VarB, j w Pfl = 2 and Pf2 = Pf3 = Pf4 =. jl The average between-psu varances are x 105. ~Crteron State and Sze Constrants M s s.ss pp.. VarB.l "-~ / (% Reducton) VarB, (94%) (56%) (51%) 1.20 (99%) (79%) (82%) Reductons n Varance Due to a One-Varable Stratfcaton (Based on a dfferent Betvar Crteron for Each Varable) r....varb, 2 VarB,3 L... VarB~ V / (% Reducton... / ~VarB (% Reducton) arb, 2- VarB,3 - (% Reducton),4 / r (95%) (51%) (39%) (99%) (82%) (81%) ' %) (98%) (75%) 10, %) (51%) 14, (66%) (96%) 6.08 (59%) 6.01 (60%) (99%) (84%) (81%),..07 (99%) 4.55 (50%) 4.05 (55%) %) %) (58%) (99%) (94%) (83%) 9.03 (99.7%) 1, (54%) 1, (54%) / The average between-psu 105. Note at e average between-psu varances for e four varables represent dfferent stratfcatons. 2.q0