Measuring Accuracy II

Transcription

1 SESSION 38 Measuring Accuracy II chair Ugo Trivellato - University of Padova, Italy Total Quality Management in Statistics Finland s Interview Data Collection Process Matti Simpanen, Eero Tanskanen, Kai Vikki - Statistics Finland The quality reports of surveys mostly focus on the quality of sampling or the quality of analysis. The quality of one of the most important phases, namely data collection process, is quite often missing. We argue that several factors have impact on the quality of interview data collection. In all the Social Survey Unit of Statistics Finland has approximately 35 monthly paid permanent employees and more than 200 hourly paid interviewers. The interviewers have permanent, part-time employment contracts with Statistics Finland. The other quality factor is the standardised interviewing method. At Statistics Finland the standardised interviewing method means systematic interviewing techniques and guidelines on the interaction between interviewer and interviewees and the style of interviewing. The measuring of the interviewers professional skills starts already when new interviewers are recruited. New interviewers start their work with a one week basic training period covering the items that form the basis of the standardised interviewing method. Later on there is further training covering more specific items to maintain the interviewers professional skills and to deepen their understanding of the survey process and data collection. We set different kinds of goals both for the projects and for the interviewers (e.g. timetables and quality issues, etc.). The achievement of goals is monitored (by project and by interviewer). The quality of data collection is monitored using e.g. recordings of the interviews and checking the numbers of interviews and other parameters. The salary system is based on a fixed part and a part determined by the employee's personal performance. Depending on the personal results and the quality of the work, it is possible to raise the part determined by personal performance. The collection of feedback is the way to maintain the high quality of the survey processes. At Statistics Finland we collect systematic feedback both from the interviewers and the interviewees. This validation is part of our survey processes. To maintain and improve quality and processes, the Social Survey Unit collects feedback for and from its customers (customers in- and outside of Statistics Finland). Statistics Finland sells data collection and survey services also to customers outside Statistics Finland. Our main goal is to describe the quality of data collection and issues affecting that quality. In the paper we emphasise that the interview work has to be seen as an integrated part of the whole survey process. Also see the abstract: Eero Tanskanen, Matti Simpanen, Kai Vikki: "A Day in the Life of an Interviewer". matti.simpanen@stat.fi PERMANENT PROFESSIONAL INTERVIEW STAFF STANDARDISED INTERVIEWING METHOD QUALITY CONTROL AND FEEDBACK SYSTEM Q2008 European Conference on Quality in Official Statistics 177

2 AREA FRAME SAMPLE SURVEY VALIDATION MEASUREMENT ERRORS Comparison of Validation Procedures to Detect Measurement Errors in an Area Frame Sample Survey Laura Martino, Marco Fritz, Marjo Kasanko - Eurostat (European Commission), Luxembourg This paper compares various validation procedures for improving the quality of statistical results coming out from an area frame sample survey. Those methods have been evaluated according to their ability to detect and reduce measurement errors. The EUROSTAT LUCAS survey has been taken as a case-study. The Land Use/Cover Area frame statistical Survey (LUCAS) is a project launched in 2001 by EUROSTAT in close cooperation with DG AGRI (Martino et al. 2008). Data collection is conducted directly in the field at geo-referenced points, which belong to a double-phase stratified sample. Stratification of the first phase sample was carried out in At that time, points were photo-interpreted to be classified into 7 main land cover classes. In 2006 and 2007, a second phase sample was selected, mainly targeted to observe agro-environmental land cover and land use (two previous campaigns were also carried out in 2001 and 2003 but using a different sampling scheme). During the ground visit, the surveyors were also requested to take standardized pictures of the landscape into the four cardinal directions, of the situation of the point and a close-up of the crop. Based on the ground visit outcomes, estimates of the extent of the main land cover/use classes are computed applying, sampling procedures, data collection methods, nomenclature and statistical estimators, which are all harmonized at European level. LUCAS was a pilot project until Thus a special attention was devoted to the assessment and documentation of the quality of both, the data collected and the procedures adopted during all steps of the survey. To the scope of this paper, four main validation procedures, applied during the two campaigns of the survey, have been selected and compared: cross-check of orto-photo interpretation and ground-survey outcomes; double-blind survey on a subsample of points surveyed in 2006; check of ground data through visual comparison with pictures taken on the points; cross-check of 2006 LUCAS sample points against CORINE Land Cover classification (EEA 2007). Some indicators have been computed. They provide an evaluation of the potential reduction of the measurement error that could be achieved applying the validation methods previously listed. Finally the efficacy of the different methods is commented on. References European Environmental Agency (2007): CLC technical guidelines. EEA technical report n. 17/2007. Copenhagen. Martino L. & Fritz M. (2008): LUCAS Land Use/Cover Area frame sample Survey: methodology and tools. Eurostat: Statistics in Focus. Agriculture and fisheries. To be published laura.martino@ec.europa.eu Reliability of responses at the 14 th Italian Population and Housing Census in 2001 Marcello D Orazio - Istat, Italy The 14 th Population and Housing Census, held in Italy in 2001, was followed by a quality control survey (the Post Enumeration Survey) aimed at evaluating the impact on the final census estimates of both coverage and response errors. In practice, a random sample of Census Enumeration Areas (EA) was selected and a 178 Q2008 European Conference on Quality in Official Statistics

3 questionnaire (a subset of about 15 questions of the Census form) was submitted to Households found in the sampled EAs. The reinterview study was designed as a perfect replication the Census data collection phase in order to obtain a independent perfect replication of Census response process. The comparison of reinterview responses with original census responses was allowed by a complex record linkage procedure aimed at finding persons that were covered by both the Census and the reinterview survey. The reliability for the responses for subset of the main Census questions was evaluated by computing the gross difference rate (GDR), i.e. the rate of responses that do not agree, and the associated index of inconsistency. The half of the GDR provides an estimate of the simple response variance (SRV), under the assumption that reinterview responses are i.i.d. replications of the original responses (cf. Biemer and Forsman, 1992). The index of inconsistency I compares SRV to its maximum value (for details see Biemer, 2004); hence, it takes values between 0 and 1 and, according to commonly used empirical rule (US Census Bureau, 1985), the SRV is considered high (low reliability) when I 0.5, moderate when 0.2 < I < 0.5; low (high reliability) if I 0.2. In 2001 Population and Housing Census, the estimated I resulted smaller than 0.2 in most of the cases with a couple of exceptions in which values among 0.20 and 0.30 were found. In this work, the evaluation of reliability is extended to case when two variables of the original survey are crossed to build a two way contingency table. In fact, in most of the cases, the final Census reports include many two way contingency tables. Two separate cases are considered according to the fact that the two survey questions are involved or not in a questionnaire skip (e.g. type of Occupation, where the latter has to be answered only by occupied people). In these cases, obviously, the level of reliability depends on the level of reliability of each single variable involved in table and on their association. In the case of variables involved in a questionnaire skip the relationship among the variables is a crucial aspect giving that responding to a variable depends on the responses to the other one. In this case, the usage of Latent Class Models (LCM) may represent a useful tool in order to assess reliability. References Biemer, P.P., and Forsman, G. (1992) On the quality of reinterview data with application to the current population survey. Journal of the American Statistical Association, 87, Biemer, P.P. (2004) The Twelfth Morris Hansen Lecture. Simple response Variance: Then and Now. Journal of Official Statistics, 3, US Census Bureau (1985) Evaluating Censuses of population and Housing. Statistical Training Document ISP-TR-5. madorazi@istat.it RESPONSE ERRORS TWO WAY CONTINGENCY TABLES LATENT CLASS MODELS Associated papers The Quality of Farm Structure Survey: the Measurement Error and the Interviewer Characteristics Massimo Greco, Matteo Mazziotta - Istat, Italy The Farm Structure Survey (FSS) is the most important agricultural sample survey carried out by Italian National Statistical Institute (Istat). The Target population of the survey includes all farms with one hectare or more of UAA (Utilised Agricultural Area) or, if they are less than one hectare, producing a certain pro- Q2008 European Conference on Quality in Official Statistics 179

4 MEASUREMENT ERROR RE-INTERVIEW INTERVIEWER CHARACTERISTICS portion for sale (2.066 ) or exceeding certain physical thresholds. Data are collected with a random sample selected according to a stratified sample design with a take all stratum containing the biggest farms. The sample size in 2005 survey is about 55,000 selected from the target population. Italian national statistical Institute (Istat) is in charge of the survey and avails oneself of Regional Bodies (Statistical offices or Technical Agricultural Offices) to carry out the data collection. The data collection is performed by interviewers recruited by Regions (an average of 36 units for each interviewer). The activities of interviewers are monitored by the regional offices in charge of the survey. Data are collected by "face-to-face" interviews using personalised paper questionnaires supplied by Istat. The filled questionnaires are manual checked and recorded (data entry) at Regional level by the interviewers or their supervisors. Most of the data entry has been performed by the interviewers or by staff close to interviewers at regional level. It is clear that the role of interviewer is very important and above all the monitoring system of his work. For this reason, Istat collects information on the characteristics of the interviewers utilised by the Regions (code, name, gender, age, educational qualification, labour contract, survey s experiences) and produces reports on the errors by interviewers. In order to evaluate the measurement error of FSS, Istat carries out a reinterview survey with Cati techniques based on replicated measurements of the same units and it is finalize to supply sufficiently reliable estimates of the measurement error and their main components (response variance and bias). That is, for a sample of units a set of questions from the original interview is asked once again (the reinterview) and the two answers given by the same units to the same question are then matched. When the responses obtained during the reinterview differ from those obtained in the original interview, the difference can be evaluated through the so-called reconciliation. In this work it is proposed a complex analysis where the authors have crossed some quality indicators of FSS with regard to the interviewers characteristics and the results of the reinterview survey. The main aim is to find, when it exists, a relation about some interviewer characteristics (age, educational qualification, labour contract, survey s experiences, etc.) and the quality of the survey. This goal is obtained by an accurate territorial analysis of the results using the interviewers code. msmagrec@istat.it, mazziott@istat.it Interviewer Falsification in the Italian Labour Force Survey: Prevention and Detection in Fieldwork Monitoring Nunzia Balì, Nicoletta Ferrante, Gianluca Giuliani, Rita Ranaldi - Istat, Italy In 2002, Italian National Institute of Statistics (Istat) undertook a whole project aimed at redesigning the Labour Force Survey (LFS) in order to fulfil the European Union (EU) Regulation. The main project objectives have been not only achieving a complete fulfilment the EU requirements but also deploying a survey system based on high quality levels in every task composing the survey process. The new LFS is characterized for a big investment on quality that involves contents, organization, methodology, information systems and, in particular, quality monitoring. Every task composing the survey process is monitored, passing from a quality control only oriented to product to a quality control of the process. In this context, Istat has given great attention, also in term of resources (time, staff, costs), to fieldwork monitoring in order to control the correct fulfilment of all activities, in particular in the prevention and detection of anomalous situations as interviewer falsification. By interviewer falsification we mean the intentional departure from the 180 Q2008 European Conference on Quality in Official Statistics

5 designed interviewer guidelines or instructions, unreported by the interviewer, which could result in the contamination of data. It includes fabricated interviews (full or partial), that is the deliberate creation of survey responses by the interviewer without input from sampled respondent. If interview data do not reflect the answers or characteristics of the respondent but rather are the invention of the interviewer, accuracy of survey results is seriously affected and the population estimates may be biased. So, survey researchers have to work in order to identify methods and implement procedures that discourage and detect interviewer cheating. Falsification in surveys, as known in literature, is a rare event especially when an effective strategy of deterrence and detection is used. This paper describes the current methods and tools used in the Italian LFS to prevent and control data falsification. The effective control of interviewer cheating is not the result of a single method, but the combined result of many actions in different phases of the survey process, in the selection and training of interviewers, in the fieldwork, in the monitoring system. Also two significant tools for the households sensitization, as the informative letter signed by Istat President and the free toll number at households disposal, become an instrument for preventing and discouraging falsification. But prevention methods are not sufficient without a suitable interviewer quality control. Procedures for detecting interviewer falsification include recontact methods with a questionnaire containing a small set of factual questions. Daily monitoring of the interviewers performance and, at the same time, a periodical use of multidimensional analysis techniques permit the research staff to target interviewers who appear more likely to have falsified data. rita.ranaldi@istat.it INTERVIEWER FALSIFICATION MONITORING PROCESS LABOUR FORCE SURVEY An Exploration of the Role of Interviewers on Survey Nonresponse in UK Government Surveys Gabriele B. Durrant - University of Southampton, UK Despite new modes of data collection, face-to-face interviewing is still one of the most important ways of collecting survey data. Interviewers have a vital role in contacting sample members and achieving their cooperation, and can be seen as a hierarchical component in the response process. A better understanding of interviewer effects is important for the improvement of data quality, survey design, interviewer training and for informing strategies to maximise response. This paper investigates the influence of interviewers on unit-nonresponse in six UK government surveys. The data available have a hierarchical structure with households nested within interviewers and interviewers being cross-classified with geographical areas. The paper explores the use of multilevel cross-classified models to analyse the effects of interviewers taking account of household level and area level characteristics. The influence of socio-demographic characteristics of the interviewer, interviewer experience, the interaction process between household and interviewer, interviewing strategies and interviewer behaviours will be analysed. Some survey specific and survey independent effects will be discussed. The study makes use of the 2001 UK Census Link Study, a unique dataset, which links the survey outcome of six major UK government surveys to a rich set of auxiliary variables for both responding and nonresponding households, including socio-demographic information from the 2001 census, detailed information about the interviewer and some information about the geographical area. An advantage of this study is that information on households is linked to rich interviewer characteristics, which also allows the investigation of possible interaction effects SURVEY UNIT NONRESPONSE INTERVIEWER EFFECTS MULTILEVEL MODELLING Q2008 European Conference on Quality in Official Statistics 181

6 between households and interviewers. It should be noted that, due to cost constraints, as in many other surveys, none of the six surveys employed an interpenetrated sampling scheme where interviewers are allocated at random to households. The paper outlines possibilities for analysis of interviewer effects in such a setting. However, some confounding of area and interviewer effects will remain. g.durrant@soton.ac.uk 182 Q2008 European Conference on Quality in Official Statistics