Design-based analysis of embedded experiments with applications in the Dutch Labour Force Survey

Transcription

1 Design-based analysis of ebedded experiens wih applicaions in he Duch Labour Force Survey Discussion paper Jan van den Brael The views expressed in his paper are hose of he auhor(s) and do no necessarily reflec he policies of Saisics Neherlands Saisics Neherlands Voorburg/Heerlen, 007

2 Explanaion of sybols. = daa no available * = provisional figure x = publicaion prohibied (confidenial figure) = nil or less han half of uni concerned 0 (0,0) = less han half of uni concerned = (beween wo figures) inclusive blan = no applicable = 005 o 006 inclusive 005/006 = average of 005 up o and including / 06 = crop year, financial year, school year ec. beginning in 005 and ending in / / 06 = crop year, financial year, ec. 003/ 04 o 005/ 06 inclusive Due o rounding, soe oals ay no correspond wih he su of he separae figures. Publisher Saisics Neherlands Prinses Bearixlaan XZ Voorburg Cover design WAT onwerpers, Urech Prepress Saisics Neherlands - Faciliy Services Inforaion E-ail: infoservice@cbs.nl Via conac for: Where o order E-ail: veroop@cbs.nl Inerne hp:// Saisics Neherlands, Voorburg/Heerlen, 007. Reproducion is peried. Saisics Neherlands us be quoed as source. ISSN: X-0

3 SUMMARY: In a series of papers a design-based analysis procedure is proposed for experiens ebedded in coplex sapling designs in which he uliae sapling unis of an ongoing saple survey are randoized over differen reaens according o copleely randoized designs or randoized bloc designs. Design-based Wald and -saisics are applied o es hypoheses abou differences beween saple eans observed under differen survey ipleenaions, Van den Brael and Renssen (998, 005) and Van den Brael and Van Berel (00). In his paper, his approach is generalized o experienal designs in which clusers of sapling unis are randoized over he differen reaens. Furherore, es saisics are derived o es hypoheses abou raios of wo saple esiaes. The ehods are illusraed wih a siulaion sudy and real life applicaions of experiens ebedded in he Duch Labour Force Survey. The funcionaliy of a sofware pacage developed o conduc hese analyses is described. Keywords: probabiliy sapling, randoized experiens, easureen error odels, X-ool.. Inroducion Randoized experiens ebedded in saple surveys are frequenly used o es he effecs of one or ore adusens in a survey process on response raes or paraeer esiaes of an ongoing survey. In survey ehodology lieraure one finds any references o experienal sudies on iproving he qualiy or efficiency of survey processes. For exaple sudies o copare he effec of differen quesionnaire designs, daa collecion odes or approach sraegies on he ain oucoes of a saple survey, wih he purpose of reducing response bias or iproving response raes. A naional saisical offices such experiens are paricularly useful o quanify disconinuiies in he series of repeaed surveys due o adusens in he survey process. The Duch Labour Force Survey (LFS), for exaple, is coninuous and aes up a series ha describes he developen of indicaors abou he siuaion on he labour are. Coparabiliy over ie is a ey aspec of he relevance of hese figures. Modificaions in he survey process should no resul in unexplained differences in he series of he eployed and uneployed labour force. This paper deals wih a series of experiens ebedded in he LFS aied o quanify he effec of alernaive quesionnaires, daa collecion odes and approach sraegies. These applicaions are used o illusrae he conceps for ebedding experiens in ongoing saple surveys and he need for a design-based heory for he analysis of such experiens. The idea of ebedding experiens in ongoing saple surveys was probably firs inroduced by Mahalanobis (946) o es inerviewer variance in survey sapling. Fienberg and Tanur (987, 988, 989, 998) discussed he fundaenals, he 3

4 parallels and he differences beween randoized sapling and randoized experiens and deailed he sraegies for design and analysis of ebedded experiens. Two oher ey references are Fellegi (964) and Harley and Rao (978). Ebedding experiens in saple surveys iplies ha firs a saple is drawn fro a finie arge populaion by eans of he probabiliy saple of he saple survey. Nex, he saple is randoly divided ino K subsaples according o an experienal design. Each subsaple is assigned o one of he K alernaive survey procedures or reaens ha are copared in he experien. The obecive in hese applicaions is o esiae finie populaion paraeers under he differen survey ipleenaions or reaens, and o es hypoheses abou he differences beween hese paraeer esiaes. Van den Brael (00), Van den Brael and Van Berel (00), and Van den Brael and Renssen (998, 005) developed a design based procedure for he analysis of copleely randoized designs (CRD s) and randoized bloc designs (RBD s). In his approach, a design based esiaor for he populaion paraeer observed under each reaen, as well as he covariance arix of he differences beween hese esiaes, is derived using he Horviz- Thopson esiaor or he generalized regression esiaor. These esiaors accoun for he probabiliy srucure iposed by he sapling design, he randoizaion echanis of he experienal design, and he weighing procedure applied in he ongoing survey for he esiaion of arge paraeers. This gives rise o a design-based Wald or -saisic, o es hypoheses abou differences beween saple survey esiaes for populaion eans or oals. In his approach uliae sapling unis are randoized over he reaens. Due o fieldwor resricions, clusers of sapling unis insead of separae sapling unis are randoized over he reaens in any pracical applicaions. As a resul, he level of randoizaion in he experienal design does no always coincide wih he uliae sapling unis in he sapling design. For exaple, i ay be necessary fro a pracical poin of view o randoize priary sapling unis (PSU s), or clusers of sapling unis ha belong o he sae household, or are assigned o he sae inerviewer over he differen reaens in he experienal design. In his paper he design-based approach of Van den Brael e al. (00, 005) is exended o experiens where clusers of sapling unis are randoized over he differen reaens. Furherore, he ehods are exended o es hypoheses abou populaion paraeers ha are defined as he raio of wo populaion oals. Secion deals wih a series of experiens ebedded in he Duch LFS. Soe aspecs of designing ebedded experiens are discussed in secion 3. The echnical deails of a design-based analysis where clusers of sapling unis of he sapling design are he experienal unis in he experien and hypoheses abou raios are esed, are deailed in secion 4. In secion 5 he properies of he proposed variance esiaor and Wald-saisic are furher invesigaed wih a siulaion sudy. A Saisics Neherlands, a sofware pacage was developed o suppor he proposed analysis procedures. The funcionaliy of his pacage is described in secion 6. In secion 7 he ehodology is illusraed wih a real life experien ebedded in he Duch Labour Force survey o es he effec of differen incenives on response raes and response bias. Soe general rears are ade in secion 8. 4

5 . Exaples of experiens ebedded in he Duch Labour Force Survey The survey design of he LFS is suarised in secion.. Five illusraive experiens are described in he oher secions.. Survey design The LFS is based on a roaing panel survey. Each onh a sraified wo-sage cluser saple of abou addresses is drawn fro a regiser of all nown addresses in he Neherlands. Sraa are fored by geographical regions, unicipaliies are considered as PSU s, and addresses as SSU s. Addresses of people aged 65 and over are undersapled, since he arge paraeers of he LFS concern people aged 5 hrough 64. All households, wih a axiu of hree, residing on an address, are included in he saple. In he firs wave, daa are colleced by eans of copuer assised personal inerviewing (CAPI) using lapops. Inerviewers collec daa for he LFS in areas close o where hey live. Deographic variables are observed for all ebers of he seleced households. Only persons aged 5 years and over are inerviewed for he arge variables. When a household eber canno be conaced, proxy-inerviewing is allowed wih ebers of he sae household. Households in which one or ore seleced persons do no respond heselves or in a proxy-inerview, are reaed as non-responding households. The respondens are re-inerviewed four ies a quarerly inervals. In hese four subsequen waves, daa are colleced by eans of copuer assised elephone inerviewing (CATI). During hese re-inerviews a condensed quesionnaire is applied o esablish changes in he labour are posiion of he household ebers aged 5 years and over. Proxy-inerviewing is also allowed during hese re-inerviews. The weighing procedure of he LFS is based on he generalized regression esiaor. The inclusion probabiliies reflec he under-sapling of addresses described above as well as he differen response raes beween geographical regions. The weighing schee is based on a cobinaion of differen sociodeographic caegorical variables. The inegraed ehod for weighing persons and failies of Leaîre and Dufour (987) is applied o obain equal weighs for persons belonging o he sae household. The os iporan paraeers of he LFS are oal uneployen and he uneployed labour force. The uneployed labour force is defined as he raio of esiaed oal uneployen and he esiaed oal labour force.. Experiens wih new quesionnaire designs The LFS quesionnaire was revised in 999. The quesions were grouped in a differen order, he wording of quesions changed, and a bloc of quesions abou receiving social benefis was deleed since his inforaion is also available fro regisraions. I was anicipaed ha he inroducion of he new quesionnaire would change he easureen errors ha are induced by he design of he quesionnaire, and have syseaic effecs on he oucoes of he LFS. Therefore a large scale field 5

6 experien was conduced o quanify he effecs of he new quesionnaire on he ain paraeer esiaes of he LFS. This enabled us o separae he real developen of he eployed and uneployed labour force fro he syseaic effec of he new quesionnaire on hese paraeer esiaes. Fro April 999 hrough Sepeber 999 he onhly saple was divided in wo subsaples in a randoized experien. Abou 80% of he onhly saples were assigned o he regular quesionnaire. The daa obained in his subsaple are used for regular publicaion purposes of he LFS bu also served as he conrol group of he experien. The reaining 0% of he sapling unis were assigned o he new quesionnaire. I was decided o assign inerviewers o one of he wo quesionnaires only o avoid confusion of he quesionnaires by he inerviewers during he daa collecion. Furherore, i was ipossible o run boh quesionnaires on he sae hand-held copuer, since he new quesionnaire was suppored by a Windows version of Blaise while he regular quesionnaire was suppored by a DOS version of Blaise. I is no feasible for inerviewers o visi households wih wo hand-held copuers. Based on hese consideraions, he experien was designed as a wo-reaen RBD. Each bloc consised of wo neighbouring inerview areas of wo inerviewers. The wo inerviewers as well as he addresses in he onhly saple of he LFS in each bloc were randoly assigned o he experienal and he conrol group. Wihin each bloc he inerviewer visis he addresses ha are assigned o his or her reaen. The purpose of his ype of experien is o esiae he eployed and uneployed labour force using he daa obained wih he regular and he new quesionnaire, and es hypoheses abou differences beween hese esiaes. Such an analysis should ypically accoun for he saple design and esiaion procedure of he LFS as described in secion.. In Van den Brael and Van Berel (00) such a designbased approach is proposed for wo-reaen experiens and used o analyse his experien. The revisions of he quesionnaire resuled in an increase of he uneployed labour force and he regisered a he eployen exchange. See Van den Brael and Van Berel (00) for a deailed discussion. A he reques of he Minisry of Social Services and Eployen, he LFS quesionnaire was exended in 005 wih a odule conaining quesions abou cobining paid eployen and care aciviies for ill parners or oher relaives. The exension of he LFS quesionnaire wih his odule us no resul in inexplicable disconinuiies in he series abou he eployed and uneployed labour force. Therefore he effec of adding his odule on he ain paraeers of he LFS was esed in a large-scale field experien. This experien was conduced fro January hrough June 005. The households included in he onhly saple were assigned o inerviewers. Subsequenly hree quarers of he households of each inerviewer were randoly assigned o he regular quesionnaire and a quarer o he quesionnaire wih he addiional odule. This iplies ha he experien is designed as an RBD were inerviewers are he bloc variable. In his case, he inerviewers collec daa wih boh quesionnaires siulaneously. I was expeced ha he inerviewers would no confuse boh 6

7 quesionnaires, since he only difference beween he regular and he new quesionnaire was an addiional, clearly separaed bloc of quesions. The daa obained in boh subsaples are used o esiae he eployed and uneployed labour force according o he esiaion procedure described in secion.. Subsequenly hypoheses abou differences beween hese esiaes are esed using he design-based approach of Van den Brael and Van Berel (00) wih he sofware pacage X-ool, described in secion 6. In his experien a oal of 5750 and 7,500 households copleed a new and a regular quesionnaire. Wih his saple size, a difference of abou 55,000 in he esiaed oal uneployed labour force could be observed a a significance level of 95%. The oal uneployen in 005 aouned abou 500,000 which iplies ha syseaic differences saller han % are no observed wih his saple size. Under he new quesionnaire he oal uneployen dropped wih 5,000 people (p-value 0.60) and uneployed labour force wih 0.4 percen poins (p-value 0.5), so he null hypoheses ha he paraeer esiaes under boh quesionnaires are equal, could no be reeced a a significance level of 95%..3 Experien wih differen daa collecion odes In 00 an experien was conduced o es he effec on he esiaes of he eployed and uneployed labour force if he daa in he firs wave is colleced by eans of CATI insead of CAPI. In his period here was a srucural lac of capaciy for he CAPI field wor organizaion, paricularly in he ore urban regions of he Neherlands. One soluion o his capaciy proble is o conduc he daa collecion in he firs wave parially by eans of CATI. Before he daa collecion in he firs wave swiched fro a uni ode design by eans of CAPI o a ixed ode design hrough CAPI and CATI, i should be esablished ha his produces no large ode effecs on he esiaes of he eployed and uneployed labour force. Fro Augus hrough Deceber 00 an experien was conduced o quanify ode effecs. Households in he onhly saples wih a lised peranen elephone connecion were randoized over he CAPI and CATI daa collecion ode by eans of an RBD, using geographical regions as he bloc variable. Abou 90% of hese households were assigned o he regular daa collecion ode, i.e. CAPI. The reaining 0% were assigned o he CATI ode. In his experien 9900 households responded under he CAPI and 00 under he CATI ode. Based on he daa observed in boh subsaples, hypoheses were esed abou ode effecs in he paraeer esiaes of he eployed and uneployed labour force, again using he design-based approach of Van den Brael and Van Berel (00) wih X-ool. The hypoheses ha here are no ode effecs was reeced, since he uneployed labour force dropped by abou. percen poins (p-value 0.07) under he CATI ode and he eployed labour force increased by.5 percen poins (p-value 0.008). Possible explanaions are he increased fracion of proxi inerviews under he CATI ode, differences in he privacy percepion of he responden, and differences in inerview speed beween hese odes. To avoid disconinuiies in he series of he eployed and uneployed labour force, i was decided no o change he daa collecion ode in he firs wave. 7

8 .4 Experien wih a new advance leer In an aep o iprove he LFS response raes, a new ore inforal advance leer for he LFS was developed. The effec on he response raes of his new advance leer was esed by eans of an experien fro January hrough March 004. The purpose of his experien was o deec sall differences in response and refusal raes. This could be achieved wihou aing subsanial addiional coss, since in his applicaion he households assigned o he sandard as well as he experienal advance leer are boh used for he regular publicaion purposes of he LFS. During hree onhs one hird of he saple addresses of each inerviewer are randoly assigned o he new leer and wo hirds o he regular leer. This resuled in an RBD where inerviewers are bloc variables wih a gross subsaple size of 8073 households for he new leer and 655 households for he regular leer. Wih an average response rae of 55%, his saple size gives rise o an experien where differences of.3% could be deeced a a significance level of 95%. Finally, he response rae obained wih he new leer was.4% saller and he refusal rae was.8% higher han wih he regular leer. A logisic regression analysis where inerviewers (blocs) and leer (reaen) were used as he explanaory variables deonsraed ha he new leer had a significan negaive effec on response behaviour. So he regular advance leer was no replaced..5 Experien wih incenives Saisics Neherlands does no pay respondens for heir paricipaion in a survey. Therefore, here are generally no incenives in he approach sraegies for surveys conduced by Saisics Neherlands. There are, however, any references o experiens deonsraing he posiive effec of incenives on response raes (Groves and Couper, 998). To explore he possibiliies o iprove response raes and reduce non-response bias in he Duch LFS, an ebedded experien wih sall prepaid incenives was conduced in 005. In his experien he effec of hree differenly valued incenives were copared wih a conrol group, where no incenive was applied. Sap booles of differen values are used as a prepaid incenive, see Table.. The sap booles are included wih he advance leer sen o he sapled households prior o he inerview for he firs wave. This experien was conduced in Noveber and Deceber 005. The gross saple was randoly divided ino four subsaples according o an RBD where inerviewers are he bloc variable. The fracions used o spli he saple ino four subsaples are specified in able.. Tabel.: Overview reaens Treaen Subsaple Nuber Descripion Value fracion no incenive 48% sap boole conaining five saps.95 4% 3 sap booles conaining en saps in oal 3.9 4% 4 4 sap booles conaining weny saps in oal 7.8 4% 8

9 The purpose of his experien was o es hypoheses abou incenives on response raes and response bias. A logisic regression analysis was applied o es hypoheses abou effecs on response and refusal raes. To invesigae he effec on response bias, hypoheses are esed abou differences beween he paraeer esiaes of he eployed and uneployed labour force obained under he subsaples assigned o he four differen incenives. This experien is analysed in secion Design of ebedded experiens 3. Ebedding experiens in saple surveys A aor advanage of ebedded experiens is he rando selecion of he sapling unis fro a finie arge populaion. This aes he appropriae o es wheher a odificaion in he survey process or a coplee survey redesign yields a higher response rae or lower response bias, or wheher cheaper ehods do no yield a lower response or less daa qualiy. The exaples in he previous secion illusrae how experiens are used in he LFS o avoid ha inended odificaions or redesigns of he survey process will resul in unexplained disconinuiies in he series of he eployed and uneployed labour force. Running he regular and he new approach in parallel by eans of an ebedded experien provides a safe survey ransiion process, since he new approach is conduced in a full scale saple before is foral ipleenaion. This reduces several coninuiy riss. Quanifying and explaining he effec of a redesign avoids he confounding of real developens described by he series wih he effec of he adusens in he underlying survey process. This reduces he negaive effec of he redesign of a repeaedly conduced survey on he coninuiy of he series. Finally, if he new approach urns ou o be a failure, his sill leaves he possibiliy of eeping he old approach for regular publicaion purposes wihou having a period for which no reliable figures are available. The experien wih he advance leer e.g. showed ha he new leer resuled in a reduced response rae while he opposie effec was expeced. The experien wih he alernaive daa collecion ode showed ha he inroducion of a ixed daa collecion ode would give probles wih he inegriy of he daa colleced in he firs wave. Based on he resuls of hese experiens, he inended odificaions in he survey process were no inroduced. Ebedding experiens in saple surveys is efficien fro a financial poin of view. In he applicaions of secion here is one relaive large subsaple ha is assigned o he regular survey, which serves no only as he official publicaion purposes for he LFS bu also as he conrol group in he experien. In soe siuaions even he daa obained in he subsaples assigned o he alernaive reaens can be used for he regular publicaion purposes of he ongoing survey, e.g. he experien wih he advance leers in secion.4. Neverheless i should be realized ha wo ore or less copeing obecives are cobined in an ebedded experien. The purpose of he regular survey is o esiae populaion paraeers as precisely as possible, so he subsaple assigned o regular survey should be axiized. The purpose of he experien, on he oher hand, is o esiae 9

10 conrass beween he populaion paraeers observed under differen survey ipleenaions as precisely as possible. This iplies ha he subsaple sizes should preferably be equal, since balanced designs axiize he power of he ess abou reaen effecs (see e.g. Mongoery, 00). The fracions used o spli he regular survey ino subsaples in he differen exaples of secion are always a rade-off beween an accepable loss of precision for he ongoing survey, he required saple size o deec pre-specified differences a a cerain power and significance level, and he ie available o conduc he experien. 3. Design consideraions A CRD is he os sraighforward approach o randoly divide a saple in K subsaples. However, he applicaion of unresriced randoizaion is generally no he os efficien design available. Fienberg and Tanur (987, 988, 989) argued ha he applicaion of an RBD wih sapling srucures lie sraa, PSU s, clusers, inerviewers and he lie as bloc variables, igh iprove he precision of an experien considerably. The response obained fro sapling unis ha are drawn fro he sae sraa, cluser or PSU, or are assigned o he sae inerviewer are generally ore hoogenous han sapling unis fro oher sraa, clusers or inerviewers. Using hese sapling srucures as a bloc variable in an experien increases he power of he experien and also guaranees ha each srau or PSU is sufficienly represened in each subsaple. This las propery is paricularly iporan if he subsaples assigned o he alernaive reaens are sall copared o he subsaple assigned o he regular survey or conrol group. All exaples discussed in secion are indeed designed as RBD s. Inerviewers require special aenion in he planning and designing sage of an experien. I should be considered carefully wheher inerviewers are assigned o all or only one of he reaens in he experien. There is a rade-off beween he increased power of he experien if inerviewers are used as a bloc variable, and he sipliciy of he fieldwor organizaion if inerviewers are assigned o only one of he reaens. Using inerviewers as bloc variables iplies ha each inerviewer us conduc each reaen. This igh resul in pracical probles wih conducing he fieldwor. Fro a saisical poin of view, i is worhwhile o ae an all-ou effor o use inerviewers as a bloc variable in an RBD. A par of he variaion in response raes is deerined by he inerviewers personal capabiliies o persuade respondens o paricipae in he survey. I is also nown ha inerviewers induce addiional variance, since hey ay affec he responses given by respondens in personal inerviews. This iplies ha he power of an experien can be iproved if inerviewers are used as he bloc variable in an RBD. Wheher i is achievable o use inerviewers as blocs depends on he nuber and ype of reaens and he field saff s experience wih collecing daa under differen reaens siulaneously. For exaple, differen wordings in differen versions of a quesionnaire igh be ixed up easily by inexperienced inerviewers and haper he applicaion of an RBD wih inerviewers as he bloc variable. In he firs experien wih a new LFS quesionnaire, discussed in secion., i was decided o assign inerviewers o one reaen only for differen reasons. The quesion blocs were organized in a differen order, he rouing changed, quesions 0

11 were dropped, and he wording changed. I was anicipaed ha he inerviewers easily ix up he differen reaens, since here was hardly experience wih daa collecion under differen reaen seings wihin he sae survey a he ie ha his experien was conduced. Since hen inerviewers a Saisics Neherlands have frequenly been used as he bloc variable in an RBD, as he experience of he field saff wih ebedded experiens increased, as in he second experien wih an alernaive quesionnaire discussed in secion. or he experien wih incenives in secion.5. I was no feasible o use inerviewers as blocs in he experien wih differen daa collecion odes since he daa collecion by eans of CAPI and CATI is organised in differen deparens wih heir own inerviewers. In he experien wih a new advance leer, discussed in secion.4, he ineracion beween inerviewers and respondens is hardly affeced by he differen reaens, so he inerviewers could be used as blocs wihou ris. Assigning inerviewers o one reaen only can be accoplished as follows in a CATI survey. Sapling unis and inerviewers are randoly assigned o he differen reaens, independen of each oher. Subsequen sapling unis are assigned o inerviewers wihin each subsaple or reaen. In a CAPI survey, where inerviewers are woring on he daa collecion in relaively sall areas around heir own place of residence, unresriced randoizaion of sapling unis and inerviewers over he reaens is ofen no applicable. This randoizaion echanis igh resul in an unaccepable increase of he ravel disance for inerviewers, paricularly if he saple sizes of he subsaples assigned o he alernaive reaens are sall. An alernaive is o assign sapling unis o inerviewers. Subsequenly he inerviewers wih heir cluser of sapling unis are randoized over he reaens of he experien. Here however, he inerviewers are he experienal unis insead of he sapling unis, which decreases he effecive saple size for variance esiaion and power for esing hypoheses. One coproise is o use geographical regions which are lined adacen inerviewer regions as a bloc variable. The sapling unis wihin each bloc are randoized over he reaen cobinaions, and each inerviewer wihin each bloc is randoly assigned o one of he reaen cobinaions. This iplies ha he nuber of inerviewers in each bloc us be equal o he nuber of reaens. Subsequenly each inerviewer visis he sapling unis assigned o his or her reaen cobinaion. This resuls in a relaive sall increase in he ravel disance for he inerviewers and no increase of variance, since he sapling unis are he experienal unis in his design. The firs experien discussed in secion. wih a new quesionnaire is an exaple of his design. Anoher consideraion is wheher he inerviewers should be infored ha hey are paricipaing in an experien or no. The advanage of eeping inerviewers uninfored is ha hey do no adus heir behaviour because hey are aware ha heir perforance is supervised in an experien. I depends on he reaens wheher i is possible o eep inerviewers uninfored and o apply an RBD where inerviewers are he bloc variable. In he LFS exaples he inerviewers were infored ha hey paricipae in an experien. No only because hey did have o collec daa under differen reaens, hey also had o share heir pracical nowledge in he design and analysis hrough debriefings. Finally here are ehical

12 reasons. A ris of eeping inerviewers uninfored is ha heir loyaly in fuure proecs ay suffer if hey accidenally find ou ha hey were involved in an experien. An experien ebedded in a wo-sage sapling design can be designed as an RBD where PSU s are he bloc variable and secondary sapling unis (SSU s) he experienal unis, or as an experien where PSU s are he experienal unis. Ofen, he variaion beween SSU s wihin PSU s is sall copared o he variaion beween he PSU s. This iplies ha he power of he experien is increased if PSU s are he bloc variables and SSU s he experienal unis, since ) he variaion beween PSU s is eliinaed fro variance of he reaen effecs and ) he effecive saple size for variance esiaion is increased because SSU s are he experienal unis insead of PSU s. I is ofen no feasible o apply differen reaens wihin he sae PSU fro a pracical poin of view, e.g. if PSU s are households and he reaens concern differen approach sraegies. A he cos of reduced power, PSU s are randoized over he reaens and consequenly he experienal unis do no coincide wih he uliae sapling unis of he sapling design. 4. Analysis of experiens wih differen randoizaion levels The purpose of he experiens discussed in secions.,.3 and.5 is o esiae he arge paraeers of an ongoing survey under he differen reaens and o es hypoheses abou he observed differences. This iplies ha for each subsaple paraeer and variance esiaors are required ha accoun for ) he sapling design of he ongoing survey used o draw a probabiliy saple fro he finie arge populaion, ) he experienal design used o divide his saple in subsaples and 3) he esiaion procedure of he ongoing survey o esiae arge paraeers. This gives rise o a design-based Wald-saisic o es hypoheses abou subsaple esiaes ha are defined as eans, oals and raios. In his secion, such a designbased analysis procedure is derived for an RBD ebedded in a wo-sage sapling design where he PSU s are he experienal unis. Subsequenly i is indicaed in secion 4.6 how resuls for oher designs enioned above are obained as a special case, e.g. if clusers of sapling unis assigned o he sae inerviewer are randoized over he reaens. Van den Brael and Renssen (005) discuss in deail why a design-based linear regression analysis is less appropriae in hese applicaions. 4. Hypohesis esing Tesing hypoheses abou response bias in finie populaion paraeer esiaes due o differen survey ipleenaions iplies he exisence of easureen errors. Therefore he radiional noion ha observaions obained fro sapling unis are rue fixed values observed wihou error (e.g. Cochran, 977) is unenable, and a easureen error odel is assued o lin syseaic differences beween a finie populaion paraeer observed under differen survey ipleenaions or reaens. Consider a finie populaion ha consiss of M PSU s. The -h PSU

13 consiss of N SSU s. The populaion size is given by N = N =. Le y il denoe he observaions for he arge paraeer obained fro sapling uni i belonging o PSU ha is assigned o inerviewer l and reaen. I is assued ha he observaions for his paraeer are a realizaion of he easureen error odel yil = u i + β + γ l + εi. (4.) Here u i is he rue inrinsic value of sapling uni (i,), β an addiive fixed effec of reaen, γ l an effec of inerviewer l and ε i a easureen error of sapling uni (i,) observed under reaen. The reaen effecs β can be inerpreed as he bias induced by -h reaen or survey ipleenaion used o easure he populaion paraeer. The odel allows for inerviewer effecs, i.e. γ l = ψ + ξl, where ψ denoes a syseaic inerviewer bias and ξ l he rando effec of he l-h inerviewer. Le E and Cov denoe he expecaion and he covariance wih respec o he easureen error odel. I is assued ha E ( ξ l ) = 0, Var ( ξ l ) = σ l and ha rando inerviewer effecs beween inerviewers are independen. Furherore, i is assued ha E ( ε i ) = 0, Var ( ε i ) = σ i + σ, he covariance beween easureen errors fro he sae PSU equals Cov( ε i, εi' ) = σ and ha easureen errors beween differen PSU s are independen. Hence and E ( y ) u + β +ψ, il = i σ i + σ + σ l σ + σ l Cov( yil, yi' ' ' l' ) = σ l 0 : : : : i = i', i i', i i', i i', M = ', l = l' = ', l = l'. ', l = l' ', l l' Noe ha he easureen errors of each inerviewer igh have a separae variance. Separae variances are also allowed for he easureen errors of he differen PSU s and SSU s under he differen reaens. The easureen error odel allows for correlaed responses beween differen sapling unis assigned o he sae inerviewer. The easureen error odel also allows for correlaed responses beween sapling unis ha belong o he sae PSU. Such correlaion arises for exaple if PSU s correspond o households and proxy-inerviewing is allowed by oher ebers of he sae household for seleced persons which canno be conaced, which is for exaple he case in he LFS exaples, see secion.. Le Y denoe he populaion ean of a arge paraeer observed under reaen =,..., K. Under a coplee enueraion of he populaion under reaen, he populaion ean is given by Y = u + β + γ + ε = u + β + ψ, where u,γ and ε are he populaion eans of he inrinsic values, inerviewer effecs and easureen errors in he finie populaion. Then Y = ( Y,..., Y K ) denoes he K-diensional vecor wih populaion eans observed under he differen reaens of he experien. A linear easureen error odel, lie (4.) is appropriae for quaniaive variables. In he LFS exaples, however, he arge variables are binary. In hese applicaions, 3

14 he observaions y il are indicaors aing values one if he sapling uni (i,) under he -h reaen repors o be uneployed and zero oherwise. The inrinsic variables u i igh also be considered as binary variables aing values one if he sapling uni is uneployed and zero oherwise. In his case, he populaion ean u denoes he real fracion of eployed or uneployed persons in he finie populaion. The reaen effecs β can be inerpreed as he average effec a his fracion if his finie populaion paraeer is easured under he -h reaen. I igh be ore appealing o inerpre he inrinsic variables u i as he probabiliy ha he response of he sapling uni equals one. The real populaion paraeer u sill denoes he real fracion of eployed or uneployed persons in he finie populaion and he reaen effecs β can be inerpreed as he average effec a he probabiliy ha he sapling unis response under he -h reaen equals one. The inerviewer effecs can be inerpreed in an analogous way. This approach appears o be rigid, since logisic odels are ore naural in he case of binary response variables. The linear odel, however, is required o develop a design-based analysis ha accouns for he generalised regression esiaion ha is used in he LFS o esiae figures abou he labour are (see secion.). Furherore, he linear easureen error odel (4.) is very appropriae o lin syseaic differences beween a finie populaion paraeer ha is observed under differen survey ipleenaions, i.e. he K differen values for Y, and he real populaion value u. The purpose of he experien is o es he hypohesis ha he populaion eans observed under he differen reaens are equal agains he alernaive ha a leas one pair is significanly differen. Only syseaic differences beween he reaens, refleced by β, should lead o a reecion of he null hypohesis. Rando deviaions due o easureen errors and inerviewer effecs should no lead o significan differences in he analysis. This is accoplished by forulaing hypoheses abou Y in expecaion over he easureen error odel, ha is H 0 : CE ( Y) = 0. (4.) H : CE ( Y) 0 Here C = ( I) denoes a ( K ) K conras arix, where denoes a ( K ) vecor wih each eleen equal o one and I a ( K ) ( K ) ideniy arix. The conrass beween he populaion paraeers in (4.) exacly correspond o he conrass beween he reaen effecs β represened by easureen error odel (4.). Hypohesis (4.) can be esed by esiaing Y, where we accoun for he sapling design, he experienal design and he weighing procedure of he regular saple survey. If Ŷ denoes such a design-unbiased esiaor and V ( CY ) he covariance arix of he conrass beween Ŷ, hen (4.) can be esed wih he Wald-saisic Y C V CY W = [ ( )] CY. Paraeer and variance esiaors for his Wald-es are wored ou in he nex secions. For noaional convenience he subscrip l will be oied in y il if possible. 4. Paraeer esiaion To es hypohesis (4.), a wo-sage saple s, drawn fro he finie arge populaion I is available. Le denoe he firs order inclusion probabiliy of he -h PSU in he II firs sage of he sapling design and he firs order inclusion probabiliy of he i 4

15 i-h SSU in he second sage given he realizaion of he firs sage saple. In he case of a CRD, he saple of PSU s is randoized over he K reaens. Le denoe K he nuber of PSU s assigned o subsaple s. Then + = = denoes he oal nuber of PSU s in s. The condiional probabiliy ha PSU is assigned o reaen, given he realizaion of he firs sage equals / +. In he case of an RBD, he PSU s are deerinisically divided in B blocs s b, b =,..., B. The PSU s wihin each bloc are randoized over he K reaens. Inerviewers or sraa of he firs sage design are poenial bloc variables in his siuaion. Le b denoe he K nuber of PSU s assigned o reaen in bloc b. Then b + = = b denoes B he nuber of PSU s in bloc b, + = b = b he nuber of PSU s in subsaple B K s and + + = b = = b he oal nuber of PSU s in s. The condiional probabiliy ha PSU is assigned o reaen, given he realizaion of he firs sage and ha PSU sb, equals b / b +. Each subsaple s can be considered as a wo-phase saple, where he firs phase corresponds o he sapling design used o draw saple s and he second phase corresponds o he experienal design used o divide s ino K subsaples s. Consequenly i follows ha he firs order inclusion * I I probabiliy of he -h PSU in he firs sage of s equals = ( / + ) in he * I I case of a CRD or = ( b / b + ) in he case of an RBD. The firs order I II inclusion probabiliy of he i-h SSU in subsaple s is given by * i. The Horviz-Thopson esiaor for Y is given by y y n i Y = * I II * I N s i= i N s, (4.3) wih n he nuber of SSU s drawn fro PSU in he second sage and ŷ he Horviz-Thopson esiaor for he populaion oal of he -h PSU assigned o he -h reaen. In any saple surveys, including he Duch LFS, he generalized regression esiaor is used o calibrae he saple weighs o a se of auxiliary variables for which he populaion oals are nown. To es hypoheses abou subsaple esiaes, he analysis procedure for ebedded experiens should be based on he generalized regression esiaor. This has he addiional advanage ha i aes he analysis ore accurae since he generalized regression esiaor generally reduces he design variance of he Horviz-Thopson esiaor and correcs, a leas parially, for selecive non-response. Le x i = ( x i,..., xih ) denoe a vecor conaining H auxiliary variables x ih of sapling uni (i,). I is assued ha hese auxiliary variables are inrinsic variables ha are observed wihou easureen errors and are no affeced by he reaens. According o he odel assised approach of Särndal e al. (99), he inrinsic values u i in he easureen error odel for each uni in he populaion are assued o be an independen realizaion of he linear regression odel: = b x e, (4.4) u i i + i where b denoes an H-vecor wih regression coefficiens and e i he residuals of he regression odel. Le ω i denoe he variance of e i. I is assued ha all ω i are nown up o a coon scale facor; ha is ω i = viω, wih v i nown. The 5

16 generalized regression esiaor for Y based on he observaions in s is given by (Särndal e al., 99) Y = Y + b ( X X ). (4.5) ; greg Here X denoes an H-diensional vecor, conaining he nown populaion eans of he auxiliary variables, X = N x n i * I II s i= i N s x * I he Horviz-Thopson esiaor for X and x he Horviz-Thopson esiaor for he populaion oal of he -h PSU. Finally, an esiaor for he regression coefficiens b, based on he observaions in s is given by n n xixi xi yi b =. * * s i= ωi i s i= ωi i The regression coefficiens b canno be copued, even in he case of a coplee enueraion of he populaion, since (4.4) is he regression of he inrinsic values u i on x i and he inrinsic values are syseaically biased by he reaen effecs β. This iplies ha b is an approxiaely design unbiased esiaor for b, i.e. he finie populaion regression coefficiens obained under a coplee enueraion under he -h reaen wih he expecaion over he easureen error odel (see forula (A.) in he appendix for an expression). Now he generalized regression esiaor Y Y greg,..., Y GREG = ( ; K ; greg ) is an approxiaely design-unbiased esiaor for Y and E. Y 4.3 Variance esiaion The nex sep is he derivaion of a design-based esiaor for he covariance arix of he conrass beween Ŷ GREG. The subsaple esiaes are correlaed since he subsaples are drawn wihou replaceen fro a finie populaion. A design-based esiaor for his covariance arix requires ha for each sapling uni an observaion under each of he K reaens is obained. These paired observaions are, however, no available since he sapling unis are assigned o one of he K reaens only. This proble can also be saed in ore echnical ers by noing ha an esiaor for he design-covariances requires oin inclusion probabiliies for he sapling unis (i,) and (i, ) ha are assigned o reaens and. The oin inclusion probabiliy ha a sapling uni is assigned o wo differen reaens, i.e. i = i', = ' and ', equals zero. This hapers a direc esiaion of he design-covariance arix of Ŷ. GREG To es hypohesis (4.) i is however sufficien o have a design-based esiaor for he covariance arix of he K- conrass beween Ŷ GREG. Under easureen error odel (4.) and a weighing odel for he generalized regression esiaor ha a leas uses he size of he finie populaion as auxiliary inforaion, i follows ha an approxiaely design-unbiased esiaor for he covariance arix of he conrass 6

17 of Ŷ GREG is given by CDC where D is a diagonal arix. In he case of an RBD, he diagonal eleen are given by d = N B ( ) e b+ I b= b b sb b ' sb e b+ ' I ', (4.6) wih e = y b x n i II i= i i. An ouline of he proof of his resul is given in he appendix. An expression for he diagonal eleens d and d under a CRD follows as a special case fro (4.6) by aing B =, b =, and b + = +. If (4.6) is copared wih forula (4.5.3) of Särndal e al., (99), hen i can be recognised ha CDC has he srucure as if he K subsaples were drawn independenly fro each oher, where he PSU s are seleced wih unequal I I probabiliies / + in he case of a CRD and / b+ in he case of an RBD. In survey sapling his variance esiaor is used o approxiae he variance under coplex ulisage sapling designs (Särndal e al., 99, secion 4.6). For he ebedded RBD s and CRD s, his esiaor is design unbiased for he variance of he conrass beween wo subsaple esiaes. No oin inclusion probabiliies and no covariances are required in his variance esiaor. Besides soe echnical deails his is he resul of he superiposiion of he experienal design on he sapling design in cobinaion wih he fac ha we focus on he variances abou he conrass beween subsaple esiaes and he assupion ha easureen errors beween PSU s are independen. Due o he superiposiion of he experienal design on he sapling design, he randoizaion echanis of he experienal design doinaes he variance srucure of he K- conrass beween Ŷ GREG. Noe ha he randoizaion echanis of an RBD can be considered as he selecion of K subsaples by eans of sraified siple rando sapling wihou replaceen were he PSU s are he sapling unis and he blocs of he experienal design are he sraa. In a siilar way, a CRD can be considered as selecing K subsaples fro he iniial saple by eans of siple rando sapling. In he variance of he conrass under (sraified) siple rando sapling, he finie populaion correcion of he subsaple eans cancels ou agains he covariance beween hese subsaple eans. See Van den Brael and Renssen (005) for a ore a echnical and deailed discussion why oin inclusion probabiliies vanish in CDC. Resuls for he Horviz-Thopson esiaor follow as a special case fro he resuls obained for he generalized regression esiaor wih he coon ean odel as weighing schee (Särndal e al., 99, secion 7.4), i.e. ( x i ) = and ω i = ω. Under his weighing schee i follows ha Y n n y i ; greg = * I II * I II s i= i s i= i ~ Y, (4.7) 7

18 and b = Y ~. An approxiaely design-unbiased esiaor for he covariance arix of he conrass beween he subsaple esiaes is given by (4.6), where ~ n e = y Y N, wih = II N i = / i. The saed condiion ha a leas he size of he finie populaion is used as auxiliary inforaion in he generalized regression esiaor holds for weighing odels ha conain an inercep or one or ore caegorical variables ha pos-sraify he populaion in subpopulaions. This condiion does no hold for he raio esiaor, since he raio odel only conains a single real valued auxiliary variable, see Särndal e al. (99), secion 7.3. Under his weighing odel he proposed variance esiaor is design-unbiased under he null hypohesis of no reaen effecs bu no under he alernaive hypohesis. Paricularly if he nuber of experienal unis wihin each bloc is sall, he variance esiaion procedure igh be iproved by pooling he variance esiaors for he separae subsaples, d, p = N B K) K e b+ ' I ( b= b b+ ' = sb ' b ' ' sb ' e b+ ' ' I '. (4.8) Wih his pooled variance esiaor i is assued ha he easureen errors of he PSU s and SSU s under he differen reaen have equal variances, i.e. σ = σ = σ and σ = σ = σ. ' ' 4.4 Wald-es I i i' ' ' II To es hypohesis (4.), he subsaple esiaes and he covariance arix of he conrass beween he subsaple esiaes give rise o he following design-based Wald-saisic: GREGC CDC ( ) CY GREG W = Y. Due o he diagonal srucure of D his Wald-saisic can be siplified o (Van den Brael and Renssen 005) K Y ; K K greg Y ; greg W. (4.9) = = d = d = d To calculae p-values or criical regions for W, i is usually conecured for generally coplex sapling schees ha under he null hypohesis W is asypoically chisquared disribued wih K- degrees of freedo. See Van den Brael and Renssen (005) for a ore deailed discussion abou he lii disribuion of (4.9). The siulaion resuls discussed in secion 5 also confir his conecure. 4.5 Analysis of raios In he LFS exaples he ain arge paraeers are defined as he raio of wo populaion oals. The uneployed labour force, e.g. is defined as he oal uneployen divided by he oal labour force. Therefore he design-based analysis 8

19 procedure developed in he preceding secions for populaion eans and oals, is now exended o raios. Le R = Y / Z denoe he raio of wo populaion eans observed under reaen =,..., K. Then R = ( R,..., R K ) denoes he K-diensional vecor wih raios observed under he differen reaens of he experien. The hypohesis of no reaen effecs for raios can be esed wih he Wald-saisic R C [ V( CR W = )] CR, where R denoes a design-based esiaor for R. Analogous o (4.) hypoheses are forulaed abou he raios were he nueraor and he denoinaor boh denoe he populaion oal in expecaion over he easureen error odel. Le y il denoe he observaions for he paraeer in he nueraor and z il he observaions for he paraeer in he denoinaor for sapling unis (i,) assigned o he -h reaen and he l-h inerviewer. I is assued ha he observaions z il are a realizaion of he sae ype of easureen error odel as defined for y il in (4.). The generalized regression esiaor for Z based on he observaions obained in subsaple s, is defined in a siilar way as Y ; greg in (4.5). The generalized regression esiaor for R is given by = /. (4.0) R ; greg Y ; greg Z; greg Finally R R greg,..., R GREG = ( ; K ; greg ) denoes he generalized regression esiaor for R. In he appendix, i is derived ha under he null hypohesis an approxiaely unbiased esiaor for he covariance arix of he conrass of R GREG is given by C D ( R) C ( ) where D R is a diagonal arix. For an RBD he diagonal eleens are defined as: d B ( R) b+ e + = b e ' I I N Z b= b ( b ) sb b ' s ; greg b ' (4.) and e = b x f x n yi i R ; greg ( zi i ) II i= i. (4.) Here f denoes he H-diensional vecor wih he Horviz-Thopson ype esiaor for he regression coefficiens of he regression funcion of z i on x i, which is (R) ( ) defined in a siilar way as b in secion 4.. Expressions for d and d R under a CRD follows as a special case fro (4.) and (4.) wih B =, b =, and b + = +. In secion 4.4 i was ephasized ha he variance esiaion procedure under he generalized regression esiaor wih a raio odel is only unbiased under he null hypohesis. Analogous o his propery, he esiaor for he covariance arix of he conrass beween raios of wo populaion oals is unbiased under he null hypohesis bu no under he alernaive hypohesis. In secion 5 a siulaion is described ha is aied o invesigae he perforance of (4.) as an esiaor for he real covariance arix of he conrass beween raios. These siulaions do no indicae ha esiaor (4.) is biased under he alernaive hypohesis. 9

20 Expressions for he Horviz-Thopson esiaor are obained in a sraighforward anner, i.e. R ~ ~ ~ = Y / Z, where Y ~ is defined in (4.7) and Z ~ is defined in a siilar way. An approxiaion for he covariance arix of he conrass beween he ~ ~ ~ subsaple esiaes is given by (4.), where e [ ( = y Y N R z Z N )], n wih = II N i = / i. The pooled variance esiaor (4.8) can be used as an alernaive o obain ore sable variance esiaes if he nubers of sapling unis wihin he blocs are sall. The hypohesis of no reaen effecs is esed wih Wald-saisic (4.9), where and d are replaced by and ( R ) d. 4.6 Special cases Y ; greg R ; greg 4.6. Randoizing inerviewers wih heir cluser of sapling unis over he reaens Now consider an experien where clusers of sapling unis ha are assigned o he sae inerviewer are randoized over he reaens. The analysis of his ype of experiens can be conduced wih he procedure proposed in his secion by I II aing = for all and considering i = i as he firs order inclusion probabiliies of he sapling design. Furherore, b denoes he nuber of inerviewers in bloc b which are assigned o reaen, b+ he nuber of inerviewers in bloc b, + he nuber of inerviewers assigned o reaen and ++ he oal nuber of inerviewers in he experien. This resul is obained by concepually dividing he arge populaion in M subpopulaions, wih M he nuber of inerviewers available for he daa collecion. Each subpopulaion consiss of he sapling unis ha are inerviewed by he sae inerviewer if hey are included in he saple. These M subpopulaions are included in he firs sage of he saple and randoized over he reaens Randoizing he uliae sapling unis over he reaens Expressions for he paraeer and variance esiaes for experiens where he I sapling unis are randoized over he reaens are obained by aing = II for all and considering i = i as he firs order inclusion probabiliies of he sapling design. This resul can be derived analogous o he ouline of he proof given in he appendix bu requires a easureen error odel were he easureen errors beween he uliae sapling unis are independen, i.e. σ = 0 for all and Two reaen experiens A special case of he experiens discussed in his secion are he wo-reaen experiens. These experiens are analyzed wih a design-based version of he - es, see Van den Brael and Van Berel (00). The paraeer and variance esiaes obained in his secion can be insered ino his design based -es, for he analysis of experiens where clusers of uliae sapling unis are randoized over he reaens and o es hypoheses abou raios. 0