ANALYSIS OF RECORD LINKAGE DECISION RULES. William E. Winkler, Rm , Bureau of the Census, Washington, DC 20223

Size: px

Start display at page:

Download "ANALYSIS OF RECORD LINKAGE DECISION RULES. William E. Winkler, Rm , Bureau of the Census, Washington, DC 20223"

Verity Lucas
5 years ago
Views:

1 COMPARATIV ANALYSIS OF RCORD LINKAG DCISION RULS Willia. Winkle, R , Bueau of he Census, Washingon, DC Keywods: M Algoih, Log, linea, Dependence, Sing Copaao Meic This pape povides an epiical copaison of decision ules in he Fellegi-Sune odel of ecod linkage. Using files fo which ue linkage saus is known, he esuls of applying vaious paaee-esiaion/decision-ule saegies fo designaing links and nonlinks ae copaed. The xpecaion-maxiizaion Algoih povides esiaes of paaees fo loglinea odels of laen classes in siuaions whee he undelying pobabiliy disibuions of ageeens on idenifies such has sunae, house nube and age saisfy a condiional independence assupion and in siuaions whee oe geneal ineacions ae allowed. 1. INTRODUCTION This pape descibes ehods of esiaing pobabiliy disibuions fo laen class odels and applicaions of associaed ecod linkage decision ules. The bes pevious decision ules wee geneally based on exensive odelling using aining ses fo which ue aching saus was known. The bes ehod of his pape does no equie aining ses and ay allow auoaic ceaion of opial decision ules fo elaively exensive ses of files. Sih and Newcobe (1975) fis obseved ha, when linking files of individuals, ageeen on faily (o household) idenifies such as las nae, house nube and see nae is no necessaily independen of ageeen on individual idenifies such as fis nae, age and sex. When all idenifies wee used in he basic decision ule (e.g., Newcobe e al. 1959, Newcobe 1988), hen diffeen pesons (fo a pai of ecods fo he wo files) in he sae household could ge weighs (o scoes) ha ae highe han weighs of a single peson (again fo a pai of ecods fo wo files) ha is lised as esiding in wo diffeen locaions. Thei soluion (which ipoved on he basic decision ule) was o develop a scoing and decision ule echanis ha ceaed scoes based on faily and individual idenifies independenly and hen o cobine he wo scoes in he decision ule ha deeined which pais wee links. An alenaive o he Sih-Newcobe pocedue is o have a geneal fiing echanis ha esiaes undelying pobabiliy disibuions when ageeens on idenifies ae dependen. In such siuaions, one weigh would esul and he oiginal decision ule of Newcobe (which had been shown o be opial by Fellegi and Sune 1969) could be applied. The advanages of he geneal ehods ae ha hey would wok wih abiay ses of idenifies and would auoaically deeine dependencies whee none wee suspeced o exis. The ouline of he pape is as follows. The fis secion gives backgound on ecod linkage and xpecaion-maxiizaion (M, e.g., Depse, Laid, and Rubin 1977, Habean 1975, 1979) echniques fo esiaing paaees. Fo laen class loglinea odels, appopiae M efeences ae Winlde (1989) and Meng and Rubin (1992). The second secion descibes he vaious odels and decision ules fo which epiical esuls ae povided. While he M fiing pocedues ae applied o all pais, he decision ules foce 1-1 aching (i.e., each ecod can be ached wih a os one ohe) using a linea su assignen pocedue (Jao 1989). In he hid secion esuls abou he esiaed pobabiliy disibuions and associaed decision ules ae pesened. The fouh secion discusses why he fiing pocedues ha yield esiaed pobabiliy disibuions ha ae quie close o he ue disibuions do no necessaily yield he bes decision ules. The las secion is a suay. 2. BACKGROUND 2.1. Fellegi-Sune Model of Recod Linkage The ecod linkage pocess aeps o classify pais in a poduc space A B fo wo files A and B ino M, he se of ue links, and U, he se of ue nonlinks. Making igoous conceps inoduced by Newcobe (e.g., Newcobe e al. 1959), Fellegi and Sune (1969) consideed aios of pobabiliies of he fo: R = P( y im) / P( y FlU) (2.1) whee ~, is an abiay ageeen paen in a copaison space F. Fo insance, F igh consis of eigh paens epesening siple ageeen o no on sunae, fis nae, and age. Alenaively, each ~, ~ F igh addiionally accoun fo he elaive fequency wih which specific sunaes, such as Sih o Zabinsky, occu. The decision ule is given by: If R > UPPR, hen designae pai as a link. If LOWR R UPPR, hen designae pai as a possible link and hold fo cleical eview. (2.2) If R < LOWR, hen designae pai as a nonlink. Fellegi and Sune (1969, Theoe) showed ha he decision ule is opial in he sense ha fo any pai of fixed uppe bounds on he aes of false links and false nonlinks, he cleical eview egion is iniized ove all decision ules on he sae copaison space F. The cuoff hesholds UPPR and LOWR ae deeined by he eo 829

bounds. We call he aio R o any onoonely inceasing ansfoaion of i (such as given by a logaih) a aching weigh o oal ageeen weizh. In acual applicaions, he opialiy of he decision ule (2.

2 bounds. We call he aio R o any onoonely inceasing ansfoaion of i (such as given by a logaih) a aching weigh o oal ageeen weizh. In acual applicaions, he opialiy of he decision ule (2.2) is heavily dependen on he accuacy of he esiaes of he pobabiliies given in (2.1). The pobabiliies in (2.1) ae called aching paaees. One way o evaluae how accuaely esiaes of pobabiliies in (2.1) agee wih he uh is o plo he cuulaive esiaed condiional pobabiliies given links and nonlinks agains he coesponding cuulaive condiional pobabiliies based on he uh. The cuulaion is by he odeing of he aio R in (2.2) induced by he esiaed pobabiliies xpecaion-maxiizaion Algoih Fo each ~, F and each pai of subses C~ and C2 ha paiion A B conside P(~') = P(~' I Cx) P(Cx) + P(~' I C2) P(C2), (2.3) whee C1 and C2 igh epesen M and U, especively. We can obseve he popoion of pais having epesenaion ~, F. If,/ epesens a siple agee/disagee paen, we can esiae he pobabiliies on he igh hand side via he xpecaion-maxiizaion (M) Algoih (see e.g., Depse, Laid, and Rubin 1977). If we assue ha ageeens on diffeen chaaceisics ae condiionally independen wihin C1 and C2, hen he axiizaion sep is in closed-fo and he M Algoih is quie saighfowad o apply (Jao 1989). Moe geneally, o accoun fo dependencies of he ageeens of diffeen aching fields (e.g., Thibaudeau 1989), we apply a vaian of an algoih of Habean (1975, see also Winkle 1989). As hee ae en aching vaiables, we only have sufficien degees of feedo o fi all 3-way ineacions (see e.g., Bishop, Fienbeg, and Holland 1975, Habean 1979). We also paiion A x B ino hee ses of pais C~, C2, and C3 using an equaion analogous o (2.3). The M pocedues ae hen divided ino 3-class o 2-class pocedues. When appopiae, wo of he hee classes ae cobined ino eihe a se which epesens M o U wih he eaining class epesening is copleen. Fo pobabiliies copued unde he independence assupion and wih daa fo ecod linkage seings, he 2-class M Algoih ypically conveges o a unique liiing soluion ove a wide ange of plausible saing poins (Thibaudeau 1989, Winkle 1989). Ou expeience is ha he independen, 3-class M also conveges o a unique liiing soluion which we, in un, use as he saing poin fo he 3-class, 3-way ineacion M. The disadvanage of any of he M pocedues is ha hey ay divide A x B ino wo ses ha diffe significanly fo he desied ses of links M and nonlinks U. An enhanceen o he basic M pocedues is o pu addiional convex (affine) consains on soe of he condiional pobabiliies and popoions o assue ha he soluions ae close o he known ue values. Fo insance he popoion in he fis class of he 3-class odel igh be bounded above by 0.1 o he pobabiliy of disageeing on fis nae condiional on being in he fis class igh be bounded above by The inuiive idea of applying convex consains is ha he M pocedues igh be given a pedisposiion owad placing ceain pais ino diffeen classes based on pio knowledge of he chaaceisics of links and nonlinks Decision Rules Fo copaaive puposes, we conside seveal eleenay decision ules and a vaiey of inceasingly oe coplicaed ules fo designaing links and nonlinks. Fo evey decision ule we foce 1-1 aching via a linea su assignen pocedue inoduced by Jao (i.e., each ecod fo one file can be linked wih a os one ecod fo anohe) and we use sing copaao eics (see e.g., Jao 1989, Winkle 1990). The eason we conside 1-1 aching ehods is ha hey can daaically lowe he size of he cleical eview egion. Fo insance, wihin a household say, he fahe-fahe, ohe-ohe, son-son, and daughedaughe pais igh be kep. The eaining welve pais (which igh be cleically eviewed if 1-1 aching wee no foced) igh be designaed as nonlinks. As he ae of ypogaphical eo fo idenifies (e.g., Sih vesus Soh) aong links is quie high fo he epiical files of his pape (25% of fis naes and 15% of las naes, Winkle 1990), we use he sing copaaos o ge bee decision ules in hose cases whee idenifes agee alos exacly bu no exacly. Fo he bes decision ules we also incopoae fequency-based (value-specific) enhanceens ha accoun fo he elaive fequency of occuence of specific fields such as las nae o fis nae (Fellegi and Sune 1969). In all cases he elaive fequency weighs ae pecisely scaled o he basic yes/no ageeen weighs obained fo he fiing sofwae o by guesses. The basic ules ae lised in Table 1. Fo he independence odel, he ules essenially diffe in how he aginal pobabiliies ae obained. The ules lised unde e-ehods ge pobabiliies via he e algoih. The ule lised unde he sc-ehod uses he aginal pobabiliies based on he known uh. In he ideal siuaion such ieaive fiing ehods ha equie exensive huan inevenion a ineediae seps as used in Saisics Canada's aching syse GRLS-V2 (Hill 1992) could yield he ue aginal pobabiliies. 830

3 Table 1. Decision Rules Independen (I) 3-class, e, feq eplace yes/no pobabliies in class 1 fo fis and las nae wih elaive fequency ones. (2) sc, feq use - and u-pobs based on uh, elaive fequencies fo fis and las nae. 3-Way Ineacion (3) 3-class, e, double double coun inceenal disinguishing powe of fis and las nae. (4) 3-class, convex sae as (4) bu apply addiional convex consains. Fo 3-way ineacion odels, we apply ules ha use basic pobabiliies fo M pocedues, ha use an enhanceen ha double-couns he inceenal disinguishing powe of fis and las nae, and ha fi wih addiional convex consains. By inceenal disinguishing powe, we ean he condiional pobabiliy of ageeen on fis o las nae in class one given he pobabiliies associaed wih he eaining vaiables. The addiional convex consains esic he popoion of pais in he fis of he hee classes o be less han and esic he pobabiliy of disageeen on fis nae given he pai is in class one o be less han Maching Fields and Daa Files The en fields available fo aching ae he six individual idenifies: fis nae, age, sex, aial saus, elaionship o head of household, and ace, and he fou faily (o household) idenifies: las nae, house nube, see nae, and elephone nube. The file sizes ae appoxiaely 12,000 and 15,000. Slighy less han 9,900 pais of ecods ae ue links and ae idenified in he files. The idenificaion was based on exensive anual eview and field followup fo a se of blocks in S Louis, Missoui. 3. RSULTS 3.1. Disibuions Resuls of fiing wih eihe wo o hee classes unde vaious independence and ineacion assupions show ha he basic 3-class, 3-way ineacion odel gives by fa he bes fi (Table 2). Appoxiae chi-squaes values ae copued accoding o Habean (1979, p. 562) and Z- values via noal appoxiaion. The esiaed cuulaive disibuion condiional on links (Figues 1, 2, and 3) is uch close o he uh when he 3- class, 3-way ineacion odel is used han when vaious independence odels ae used. The coesponding cuves fo nonlinks (no shown) also yield ha he 3-class, 3-way ineacions odel gives he bes fi. Table 2. Chi-Squae Fis, Degees of Feedo, and Z-Values unde Vaious Models Chi DOF Z 2-class Independen class Independen class 3-way class 3-way, convex If we wee o conside all pais (ahe han he subse obained unde he 1-1 aching esain), hen he 3-class, 3-way ineacion pobabiliies would yield he bes decision ules and easonably accuae esiaes of eo aes. The las fi illusaes how fis degade when convex consains ae iposed ha cause only ild deviaions fo he fi unde no consains. While convex fis can yield fis in diffeen classes ha ae close o he ue popoions and os of he ue pobabiliies, hey have no ye ipoved he decision ules and ae no consideed fuhe Decision Rules unde 1-1 Maching The decision ules ((1)-3-class independen M, (2)- independen wih agins based on uh, & (3)-3-class, 3- way M) ae oughly equivalen a eo levels of and above (Table 3). A he eo level of 20 false links (0.2 pecen false link ae), ules (1), (2), and (3) designae 9808, 9813, and 9601 pais as links, especively and, a eo level 50 false links (0.5 pecen false link ae), he ules designae 9875, 9881, and 9802 pais as links, especively. If we adjus he pobabiliies of secion 3.1 o he subses of pais consideed by he 1-1 aching ules (he subses ae dependen on he weigh esiaes), hen he 3-class, independen e-pobabiliies sill deviae quie subsanially fo he uh (Figues 4 and 6). While he 3-class, 3-way ineacion pobabiliies geneally deviae fo he uh fo links (Figues 5), hey eain vey close o he uh a eo levels of less han 10%. They also ae easonably close o he uh fo nonlinks (Figue 7). 831

4 Table 3. # False Nube of Pais by Maching a Diffeen Levels of False Links 1-1 Maching, Fequency-Based Nube of Pais e sc e indep indep 3-way (1) (2) (3) / i/ Highes nube of ue links achievable wih he 1-1 aching esicion. 4. DISCUSSION 4.1. Pais Via 3-class and 2-class M Pocedues When 2-class M pocedues ae applied o pais of files having boh individual idenifies such as fis nae, age, and sex and faily (household) idenifies such as las nae, house nube, and see nae, he se of pais naually divides ino hose ageeing on household idenifies and hose ha do no. The 3-class M ceaes a oe naual paiioning because i basically divides he se of pais ino (1) links wihin he sae household, (2) nonlinks wihin he sae household, and (3) nonlinks ouside he sae household. Fo he decision ules, he pobabiliies associaed wih classes (2) and (3) ae cobined o yield he pobabiliies fo he se of nonlinks U Decision Rules On an absolue basis, he bes wo decision ules ha equie neihe knowledge of he uh no aining ses ((1)- 3-class independen M and (3)-3-class, 3-way ineacion M) boh wok vey well. A he 0.5% false link ae ule (1) yields 99.7% of he ue links (( )/9859) while ule (3) yields 98.9% of he ue links (( )/9861)). The bes 3-way ineacion odel woks slighly wose han he bes of he independence odels because he 3-way ineacion odel places wo addiional ypes of pais in Class C ha he independence odel does no. The fis ype basically consiss of husband-wife pais which agee on age and which agee on a iskeyed sex code. The second ype includes fahe-son pais ha agee on nae. The fields ha bes allow us o disinguish individuals wihin a household ae fis nae, age, and sex. ach of hese wo ypes of pais agee on wo of he hee fields. If 30 pais of hese wo ypes (epesening appoxiaely 0.03% of he 116,305 pais used in he ea-esiaion pocedues) wee shifed fo Class C1 o Class C2, hen ule (3) would wok as well as ule (1). A pesen, we suspec oe caeful odelling using seleced subses of ineacion es geae han he hid ode o caeful applicaion of cobinaions of convex consains will yield an ipoveen o ule (3). The disadvanage of using seleced subses of he highe ode ineacion es is ha such odelling is quie difficul in he bes of cicusances (Bishop, Fienbeg, and Holland 1975) and ay be soewha specific o he pais of files being ached. The advanage of he convex consains is ha hey can easily be applied based on pio aching siuaions. Fo insance, we igh esic he pobabiliy of siulaneous disageeen on fis nae and sex in he fis class o be less han Pobabiliy Disibuions When decision ules ha do no foce 1-1 aching ae applied, pobabiliy disibuions obained unde he 3-class, 3-way ineacion odels ae sufficienly accuae ha hey can be used o esiae ue eo aes. They have he addiional inuiive feaue ha if a pai agees on house nube and see nae, hen he inceenal weigh associaed wih las nae is vey sall. If he pai disagees on household coponens such as house nube and see nae, hen he inceenal disinguishing powe of las nae is quie lage. o aes fo decision ules based on pobabiliy disibuions esiaed unde he independence assupion (whehe o no 1-1 aching is foced) ae sufficienly inaccuae ha hey ae unusable (e.g., Figues 1, 2, 4, and 6). Fo esiaing eo aes in he independen case when an 1-1 decision ule wih good disinguishing powe is used, we would use a ehod of Belin and Rubin (1991). Unlike he M odelling ehod of his pape, he Belin-Rubin ehod geneally equies ha a epesenaive aining se be available fo odelling ceain paaees in hei odel. Fo he weighs aising fo he 3-class, 3-way ineacion odel (whehe o no 1-1 aching is foced), Belin-Rubin fiing sofwae will no always convege due o he fac ha he cuve of naual logaih vesus weigh is no clealy biodal as i was in ohe applicaions fo which he sofwae had been developed. We noe ha he Belin-Rubin 832

pocedues use only he suaizing infoaion conained in he aching weigh while he 3-class, 3-way ineacion odels use all he infoaion fo he vaious ageeen paens. 4.

5 pocedues use only he suaizing infoaion conained in he aching weigh while he 3-class, 3-way ineacion odels use all he infoaion fo he vaious ageeen paens Diffeen Saing poins fo 3-Way Ineacion Model Using a vaiey of diffeen saing poins, hee appea o be a leas hee local axia, all of which give he sae value of he likelihood funcion o five significan digis. Aong he liiing soluions, he popoions assigned o diffeen classes vay subsanially and he condiional pobabiliies geneally vay. If saing poins wee chosen faily close o he soluion of he 3-class, independen odel (which appeas o yield unique soluions), hen convegence was always o he fis local axia. We noe ha soluions can, a bes, be unique up o peuaion (Habean 1979, Chape 10) Geneal Applicabiliy of Mehods The use of he 3-class, independen M pocedue should geneally yield good esuls in he ype of 1-1 aching decision ules eployed in his pape. In fou ohe pais of files of individuals in which household idenifies wee pesen and fo which ue aching saus known, he 3- class, independen M pefoed a leas as well i did fo he pai of files in his pape. 5. SUMMARY This pape consides ehods fo aching individuals using a cobinaion of individual idenifies such as fis nae, age, and sex and faily idenifies such as house nube and see nae. A pesen, he bes decision oles (based on eihe false link ae o size of cleical eview egion) allow a os one individual fo one file o be ached agains one fo anohe (i.e., foce 1-1 aching) and use pobabiliy disibuions ha saisfy an independence assupion. If all pais ae consideed (i.e., 1-1 aching is no foced), hen he bes decision ules use pobabiliy disibuions fi unde 3-way ineacion odels and allow esiaion of eo aes. When 1-1 aching is foced, howeve, he 3-way ineacion decision ules pefo soewha wose han he bes of he independen ules. Fo 1-1 aching 3-way ineacion pobabiliies fo all pais can be adjused o yield easonably accuae esiaes of eo aes bu independen pobabiliies can no. *This pape epos views of he auho ha do no necessaily epesen hose of he Bueau of he Census. The auho hanks Yves Thibaudeau, Cal Konshnik, and Phillip Seel of he Bueau of he Census and Michael Lasen of Havad Univesiy fo coens. RFRNCS Belin, T. R. and Rubin, D. B. (1991) "Recen Developens in Calibaing o Raes fo Copue Maching," P0c. of he 1991 Census Annual Reseach Conf., Bishop, Y. M. M., Fienbeg, S.., and Holland, P. W. (1975), Discee Mulivaiae Analysis, MIT Pess, Cabidge, MA. Depse, A. P., Laid, N. M., and Rubin, D. B. (1977) "Maxiu Likelihood fo Incoplee Daa via he M Algoih," J. Royal Sa. Soc. B, Fellegi, I. P., and Sune, A. B. (1969), "A Theoy fo Recod Linkage," Jounal of he Aeican Saisical Associaion, Habean, S. J. (1975), "Ieaive Scaling fo Log-Linea Model fo Fequency Tables Deived by Indiec Obsevaion," Poceedings of he Secion on Saisical Copuing, Aeican Saisical Associaion, pp Habean, S. (1979), Analysis of Qualiaive Daa, Acadeic Pess. New Yok. Hill, T. (1991), "GRLS-V2, Release of 22 May 1991," (Available fo Geneal Syses, R.H. Coas Bldg, 14-O, Saisics Canada, Oawa, Onaio KIA 0T6.) Jao, M. A. (1989), "Advances in Recod-Linkage Mehodology as Applied o Maching he 1985 Census of Tapa, Floida," Jounal of he Aneican Saisical Associaion, 8.._99, Meng, X. and Rubin, D. B., (1992) "Maxiu Likelihood Via he CM Algoih: A Geneal Faewok," Bioeika, o appea. Newcobe, H. B. (1988) Handbook of Recod Linkage: Mehods fo Healh and Saisical Sudies, Adinisaion, and Business. Oxfod: Oxfod Univ. Pess. Newcobe, H. B., Kennedy, J. M., Axfod, S. J., and Jaes, A. P. (1959), "Auoaic Linkage of Vial Recods," Science Sih, M.., and Newcobe, H. B. (1975), "Mehods of Copue Linkage of Hospial Adission-Sepaaion Recods ino Cuulaive Healh Hisoies," Meh. Info. Med ,=.._ Thibaudeau, Y. (1989), "Fiing Log-Linea Models When Soe Dichooous Vaiables ae Unobsevable," in.p. oceedings of he Secion on Saisical Copuing, Aeican Saisical Associaion, pp Winkle, W.. (1989), "Nea Auoaic Weigh Copuaion in he Fellegi-Sune Model of Recod Linkage," Poceedings of he Fifh Census Bueau Annual Reseach Confeence, Winkle, W.. (1990), "Sing Copaao Meics and nhanced Decision Rules in he Fellegi-Sune Model of Recod Linkage," Poc. of he Secion on Suvey Reseach Mehods, Ae. Saisical Assoc., pp

6 Figue 1. sgae8 v8 Tuh Cuulaive Disibuion of Maches 1- Independen M Rgu 2. siaes v8 Tulh Cuulaive Dlsdbullon of Maches 2- Independen, Salsldcs Canada Rgu 3. siaes s Tuh Cuulaive Disibuion of Maches 3-3-way Ineacion M ~.o 11 I O.l il o.-, e do...= =- ++" z.o I '. 1 j o.1, II o.,. 1 " do.0, z.o~ e! ~ io,. " B do-*' p o ~ o.i B ID o-~ I I I Y o.~ o.o o.z o.1 o.s o.+ o.= o.. +.o TnJe ~ po.8. Go,. b I... i ~" y " o.o..... o.-, o.o o.a o.= o.~ o.= o.. a.o po.a. oo,. b " B ~ o.~, Ij o.=. o~. s I, pc', S' o.o o.3. o.= o.s o.i o.. o.- o.o ~..o Figue 4. siss v8 "lh Cuulaive Dislibuion of Maches 1- Independen M, 1-1 Maching Figue 5. siaes vs Tulh CuulaUve Disdbulion of Maches 3-3-way Ineacion M, 1-1 Maching Rgue 8. siaes s Tuh Cuulaive Disibuion of Nonaches 1- Independen M, 1-1 Maching U o.i 0 d P.1, o.= o d p o.~. II o.~ IS" & Y s~ 1 Y Y o.o o.x Tue ~ l y o.o o.o o.z T~ I:~Xu~lly o.o o.o o.a o.- o.: Figue 7. siaes vs Tuh Cuulaive Disibuion of Nonaches 3-3-way Ineacion M, 1-1 Maching Cuulaion Is by deoeaslng eslaed weigh Sall p o b g b l l l l e ool-el ~ond o lall eo Qe 45 dog0"oo Il~o ao 'ofe'e~go... o.~. ' a 0 ~. 0 sl d / P f 0 b a o.~. b I i,,/f f-.. Y! o.o. o.o o.~ o. o.: 834

International Journal of Pure and Applied Sciences and Technology

International Journal of Pure and Applied Sciences and Technology In. J. Pue Appl. Sci. Technol., 4 (211, pp. 23-29 Inenaional Jounal of Pue and Applied Sciences and Technology ISS 2229-617 Available online a www.ijopaasa.in eseach Pape Opizaion of he Uiliy of a Sucual