The British Crime Survey Sample: A response to Elliott and Ellingworth

Transcription

1 The British Crime Survey Sample: A response to Elliott and Ellingworth Peter Lynn This is a pre-copyedited, author-produced PDF of an article accepted for publication in Sociological Research Online following peer review. The version of record (published version) [The British Crime Survey Sample: A response to Elliott and Ellingworth; Peter Lynn; Sociological Research Online 1998] is available online at: and should be cited as follows: Lynn, Peter (1998) The British Crime Survey Sample: A response to Elliott and Ellingworth. Sociological Research Online, 3(1). 1. Introduction Elliott and Ellingworth draw our attention to an important issue, namely the need to assess the quality of survey samples. In particular, they stress the effect that survey response rate can have on sample representativeness. They rightly implore analysts of survey data to pay more attention to this issue, to consider the possible impacts on their analyses, and to contemplate the development of weighting for non-response if appropriate. They use the British Crime Survey (BCS) as their example. However, Elliott and Ellingworth s article contains a number of important misunderstandings. These are addressed in sections 3, 4 and 5 below. Also, the methods used by Elliott and Ellingworth have certain limitations which have implications for their conclusions. Those limitations are described in section 5 below. Section 6 then presents some alternative assessments of non-response bias and attempts to draw some revised conclusions about the impacts of survey non-response on survey estimates. 2. Survey error Before any systematic assessment of components of survey error can proceed, it is important to define the components of interest. Elliott and Ellingworth make no reference to the vast literature on survey error sources (Andersen et al, 1979; Dippo, 1997; Groves, 1989; Hansen et al, 1961; Kish, 1965 chapter 13; O Muircheartaigh, 1997). Instead, they present their own rather unhelpful trichotomous classification of sources of survey error, using rather unconventional terminology. They label the

2 sources, misoperationalisation of the population, sampling error and responding error. This makes it difficult for the reader to be clear about exactly which error sources are being addressed in the paper. Relating Elliott and Ellingworth s error sources to the classification conventionally used in the survey error literature (see, for example, Groves, 1989, p.17), it seems to me that by misoperationalisation of the population they essentially mean coverage errors, by sampling error they mean the net effect of sampling variance and sampling bias and by responding error they mean specifically unit non-response bias. It is important to understand that Elliott and Ellingworth s responding error does not include response error, or item non-response error, or the variance component of unit non-response error (their paragraph 2.8 makes it clear that it is bias with which they are concerned). In fact, Elliott and Ellingworth seem unclear about the distinction between the bias and variance components of error. For example, in paragraph 2.6 they describe the weighting which is made available with the public access version of the BCS data, without explaining that the point of this weighting is to remove sampling bias, i.e. to produce designunbiased estimates. They also seem a little confused about the meaning of unit nonresponse, as they suggest (section 5.13) that deadwood is a contributory factor. 3. Coverage error on the BCS As an example of coverage error, Elliott and Ellingworth refer to the decision to change the sampling frame for the BCS from the electoral registers to the PAF. However, they seem to have misunderstood the way in which the electoral registers were used as a sampling frame, implying that only people whose name appeared on the registers had a chance of inclusion in the survey. In fact, all residents at addresses which appeared on the registers (i.e. at which at least one person was registered) were given a chance of inclusion, so the coverage bias involved was perhaps rather less than Elliott and Ellingworth might suppose. Earlier studies have investigated this coverage error in some detail, both with respect to the BCS specifically (Lynn, 1997a) and with respect to social surveys more generally (Lynn and Taylor, 1995). 4. Sampling error on the BCS In their discussion of sampling error, Elliott and Ellingworth state that sampling theory identifies a simple random sample as the method that will [minimise sampling error] most effectively. They go on to claim that the BCS sample design is not truly random as it is a stratified, multi-stage design. They seem to have rather misunderstood sampling theory. One of the main purposes of stratification (though not the only use to which it is put) is to reduce sampling error relative to a simple random sample (Kish, 1965 chapter 3; Moser and Kalton, 1971 section 5.3; Stuart, 1984 section 21). This is precisely the reason it is employed on the BCS. (For an example of the beneficial effects of stratification on a UK social survey, see Barton, 1996.) The motivation for multi-stage sampling is the same - to reduce sampling error for a given level of expenditure. Although clustering per se tends to increase sampling

3 variance, the reduction in field work costs permits a larger sample size, thus reducing sampling variance. If the latter effect outweighs the former, which it certainly does on the BCS, then the overall effect is a reduction in sampling error for a fixed survey budget. Thus, in summary, techniques like stratification and multi-stage selection are not used on the BCS and other social surveys as a second best because researchers cannot afford simple random samples. Rather, the designers of survey samples utilise the full range of sampling techniques at their disposal to produce the design that is estimated to deliver maximum accuracy (minimum total survey error) for a fixed cost (or alternatively, to deliver a prescribed accuracy for minimum cost) (Lynn and Lievesley, 1991). Elliott and Ellingworth also discuss the ethnic minority booster sample component of the BCS. Here they do the survey a great disservice in claiming that the aim of randomness was abandoned. It is not made clear why they think this. In fact, the ethnic minority sample too is an unbiased random probability sample. The selection procedures are documented in Hales (1992). 5. Non-response bias on the BCS The main section of Elliott and Ellingworth s paper attempts to estimate the extent of unit non-response bias on the 1992 BCS. However, the methods that they use have two severe limitations. In addition, they appear to misinterpret some of their results. The first limitation relates to the way response rates are calculated. They are calculated for each primary sampling unit (postcode sector) by dividing the number of achieved interviews by 17.5, where 17.5 represents the total target achieved sample size divided by the total number of primary sampling units. The problem with this approach is that it fails to take into account the distribution of ineligible addresses across primary sampling units. Consequently, response rate is completely confounded with eligibility rate, and the rates used in the analysis are in fact the product of response rate and eligibility rate. Thus, when the authors claim to have found an association between these rates and area crime rates (paragraphs 4.6 and 4.7), there is in fact no way of knowing whether the association is with response or with eligibility. As eligibility is essentially an estimate of the proportion of addresses in the area that are residential and occupied, it is plausible that this could be correlated with response rate. The second limitation is that the assessments of non-response made by Elliott and Ellingworth all rely on external comparisons, in other words they compare respondents with population rather than respondents with non-respondents (as they acknowledge in passing in paragraph 2.8). This has two drawbacks. First, there is a risk of the comparisons themselves being biased if there are systematic differences between the two data sources in terms of, for example, definitions of categories, reference time point, reference population, data collection method, etc. Second, even in the absence of any such bias, random sampling variance will be present and will reduce the precision of comparisons. The comparisons made with 1991 Census data (paragraphs 3.9 and 3.10) could well be flawed by systematic differences between the two data sources, as the authors have compared a number of variables that can be rather sensitive to differences in reference

4 time point and/or questioning method. For example, the male unemployment rate is found to be significantly different. But the BCS data was collected around a year after the Census, during which time the male unemployment rate moved by around two percentage points (Employment Department, 1992). Another variable compared was ethnic group. Reported ethnicity has been found to be sensitive to data collection mode (Johnson, 1974; OPCS, 1987 page 14; Sillitoe, 1977; Smith, 1997) - the Census involved self-completion and, often, proxy reporting, whereas the BCS involved interviewer-administered self-reporting. Unfortunately, Elliott and Ellingworth do not sufficiently define the variables they have used to permit comment on possible differences of definition between the two sources. Neither is it made clear whether their Census estimates are based on the same target population as the BCS sample (residents of England and Wales only, aged 16 or over, in private households). In paragraphs 3.13 and 3.14, having calculated the sums of estimated squared errors across postcode sectors, Elliott and Ellingworth attempt to search for patterns to see whether, for example, relatively deprived groups are the least accurately sampled. They see no pattern amongst the variables they have chosen, and conclude that sample accuracy cannot be seen to be related to deprivation. What they apparently fail to realise is that whether or not a household is deprived are opposite sides of the same coin. Consequently, for example, the total squared errors associated with owner occupiers and renters must by definition be the same. If one group is over-represented, then the other must be under-represented by the same amount. A meaningful comparison should take into account the direction of the error, for example to see whether the deprived tend always to be under-represented (Lynn, 1996b). In paragraph 3.16, commenting on a table showing ten estimates of squared bias, each of which takes one of only two values (0.01 or 0.02), the authors claim to see a pattern! Moreover, they claim that one observation is an exception to the pattern. Such statistical fallacy does not encourage the reader to take their conclusions seriously. The reader should perhaps also be wary of the conclusions drawn in paragraph The authors first argue that if the representation of demographic variables shows no pattern over crime deciles then the variation in response rate over the deciles must be due to demographic differences rather than crime rate per se (paragraph 4.9). They then find just that - no pattern - and proceed to conclude the opposite, namely that differences in response rates result from differences in crime rate. In any case, this whole approach ignores the possibility of other unmeasured variables accounting for the relationship. In paragraph 5.4 the authors speculate about the possible effects of non-response bias on estimated crime rates. They suggest that there are two likely scenarios - either nonrespondents experience more crime than respondents or there is no significant difference. They point out that the crime experiences of non-respondents on the 1992 BCS are unknown. However, on the 1996 BCS special attempts were made to administer a very short questionnaire (taking about two minutes) to each non-responding household, designed to permit estimation of six key crime victimisation rates. There was some evidence of non-response bias for two of the six rates (Lynn, 1997b). In both cases, the non-respondents appeared to have experienced less crime than the respondents.

5 6. Alternative assessments of non-response bias There are of course many ways of assessing non-response bias (Armstrong and Overton, 1977; Barnes, 1992; Groves, 1989 chapter 4; Hansen and Hurwitz, 1946; Kalton, 1983; Lynn, 1996a; Moser and Kalton, 1971 section 7.4). A key distinction is between internal and external methods. The methods used by Elliott and Ellingworth fall in to the latter category, relying as they do on comparisons with data sources external to the sample. Internal methods have the strong advantage of not suffering from sampling error or from potential bias due to differences in data collection mode, definitions or reference time point (Lynn, 1996a). Their disadvantage, however, is that they can only be applied with respect to data items that can be collected for both respondents and nonrespondents. On the BCS, various sources of such data have been explored. These include data that can be linked to the sampling frame (Census small area data, population densities, geographical areas), data that can be collected by interviewer observation (characteristics of the area, type of dwelling, characteristics of the dwelling, etc) and data that can be collected through limited questioning of nonrespondents (the six quick questions approach). Non-response bias has been estimated by SCPR with respect to variables collected from all of these sources (Lynn 1996b, 1997b). Those studies revealed a number of systematic differences between responding and non-responding households. For example, table 1 shows the relationship between response rate and certain attributes of the dwelling, while table 2 shows the relationship of response rate with population density. Table 1: Response Rates, by Housing Type (BCS 1996) Housing type Response rate Base (eligible addresses) House: Detached House: Semi-detached House: End-terrace House: Mid-terrace Maisonette Flat: Converted Flat: Purpose-built Rooms/ bedsit At least one visible security ,408 device at address No visible security devices ,304 Entryphone at address ,729 No entryphone ,999 Source: Lynn, 1997b Note: Security devices include burglar alarms, security gates, window bars or grilles, security patrols and other devices thought by the interviewer to be for security purposes.

6 Table 2: Response Rates, by Population Density (BCS 1996) Population density (persons per hectare) of postcode sector Response Rate Base (eligible addresses) Less than ,121 5 to , to , to , or more ,701 Source: Lynn, 1997b Furthermore, there are differences between different categories of non-response. In summary, BCS addresses where a refusal is received tend to be disproportionately purpose-built flats, flats on 4th floor or above, dwellings in buildings with a common entrance, dwellings with no visible security devices and addresses where there are no registered electors. There is also a suggestion that blacks may be less likely to respond than other ethnic groups. On the other hand, non-contacts occur disproportionately at flats or maisonettes, flats on the tenth floor or above, dwellings in buildings with a lockable common entrance, dwellings with an entryphone system, addresses in areas of poor condition housing, addresses in areas of high population density and in mainly nonresidential areas (see Lynn 1996b, 1997b for more details). Attempts directly to estimate non-response bias in estimated crime victimisation rates (Lynn, 1997b) has suggested that, if anything, non-respondents may experience slightly less crime than respondents. These internal comparisons of respondents and non-respondents provide robust evidence of the nature of non-response bias on the BCS. The comparisons that have been made directly in terms of crime victimisation measures (Lynn 1997b) are as yet preliminary. The methodology has yet to be proven. But the comparisons in terms of characteristics of the dwelling and the immediate local area (Lynn 1996b) are reliable and informative and could be used to develop a weighting strategy that would remove that component of non-response bias that is explained by these variables. 7. Conclusions In summary, quite a lot is known about non-response bias on the BCS. The Home Office, as funders and main users of the BCS data, have chosen not to incorporate this information into the sample weighting made available on the archived data set, but this does not mean that they are blind to the issues. Ultimately, I support the suggestion that The Home Office might be well advised to consider incorporating a system of weighting for non-response on the BCS (Lynn, 1996a). However, I suspect that the simplistic model proposed by Elliott and Ellingworth, where the primary sampling units form the weighting classes, is likely to be sub-optimal and possibly even counter-productive. With such small weighting classes, there will be a lot of random variation in response rates and therefore in the resulting weights. The increase in variance could well outweigh any reduction in bias, thereby increasing the total survey error. Rather, I would suggest that a number of

7 substantively more relevant variables should be used to define the classes and that the classes should be developed using an empirical multivariate model (Lynn, 1996b). References ANDERSEN, R., KASPER, J., FRANKEL, M.R. and Associates (1979) Total Survey Error. San Francisco: Jossey-Bass. ARMSTRONG, J. and OVERTON, T. (1977) Estimating nonresponse bias in mail surveys, Journal of Marketing Research, vol. 14, pp BARNES, B. (1995) Studies of non-respondents in OPCS household surveys using Census data, Survey Methodology Bulletin, no. 30 pp BARTON, J. (1996) Selecting stratifiers for the Family Expenditure Survey (FES), Survey Methodology Bulletin, no. 39 pp DIPPO, C. (1997) Survey measurement and process improvement: concepts and integration, in L. Lyberg, P. Biemer, M. Collins, E. deleeuw, C. Dippo, N. Schwarz and D. Trewin (editors) Survey Measurement and Process Quality. New York: Wiley. ELLIOTT, C. and ELLINGWORTH, D. (1997) Assessing the Representativeness of the 1992 British Crime Survey: The Impact of Sampling Error and Response Biases, Sociological Research Online, vol. 2, no. 4, < EMPLOYMENT DEPARTMENT (1992) Labour Market Data: Unemployment GB Summary, Employment Gazette vol. 100 no. 6, pp. S16-S17. GROVES, R.M. (1989) Survey Errors and Survey Costs. New York: Wiley. HALES, J. (1992) British Crime Survey (England and Wales) Technical Report. London: SCPR. HANSEN, M. and HURWITZ, W. (1946) The problem of nonresponse in sample surveys, Journal of the American Statistical Association, vol. 41, pp HANSEN, M.H., HURWITZ, W.N. and BERSHAD, M.A. (1961) Measurement errors in censuses and surveys, Bulletin of the International Statistical Institute, vol. 38 no. 2, pp JOHNSON, C. E. (1974) Consistency of reporting of ethnic origin in the Current Population Survey. Technical Paper no. 31. Washington DC: Bureau of the Census. KALTON, G. (1983) Compensating for Missing Survey Data. ISR Research Report. Michigan: Survey Research Center, University of Michigan. KISH, L. (1965) Survey Sampling. New York: Wiley.

8 LYNN, P. (1996a) Weighting for non-response, in R. Banks, J.Fairgrieve, L. Gerrard, T. Orchard, C. Payne and A. Westlake (editors) Survey and Statistical Computing 1996: Proceedings of the Second ASC International Conference. Chesham: Association for Survey Computing. LYNN, P. (1996b) Who responds to the British Crime Survey?, Paper presented to Essex 96, the Fourth International ISA Conference on Social Science Methodology, July , University of Essex, Colchester UK, July 1-4. LYNN, P. (1997a) Sampling frame effects on the British Crime Survey, Journal of the Royal Statistical Society Series A, vol. 160 no. 2, pp LYNN, P. (1997b) Collecting Data About Non-respondents to the British Crime Survey, Unpublished report to The Home Office. LYNN, P. and LIEVESLEY, D. (1991) Drawing General Population Samples in Great Britain. London: SCPR. LYNN, P. and TAYLOR, B. (1995) On the bias and variance of samples of individuals: a comparison of the electoral registers and postcode address file as sampling frames, The Statistician, vol. 44 no. 2, pp MOSER, C.A. and KALTON, G. (1971) Survey Methods in Social Investigation (second edition). Aldershot: Gower. O MUIRCHEARTAIGH, C. (1997) Measurement error in surveys, in L. Lyberg, P. Biemer, M. Collins, E. deleeuw, C. Dippo, N. Schwarz and D. Trewin (editors) Survey Measurement and Process Quality. New York: Wiley. OPCS (1987) General Household Survey London: Her Majesty s Stationery Office. SMITH, T. (1997) Measuring race by observation and self-identification. GSS Methodological Report no. 89. Chicago: National Opinion Research Center. STUART, A. (1984) The Ideas of Sampling. London: Griffin.