Response Errors and Interviewer Characteristics: A Multidimensional Analysis

Transcription

1 Proceedings of Q2008 European Conference on Quality in Survey Statistics Response Errors and Interviewer Characteristics: A Multidimensional Analysis Massimo Greco, Matteo Mazziotta, Adriano Pareto 1 Keywords: Response error, reinterview, multidimensional analysis 1. Introduction The aim of this work is to analyse the relation between the interviewer profile and the quality of the 2005 Farm Structure Survey (FSS). This goal is obtained matching some quality indicators of 2005 FSS with the interviewers characteristics (age, educational level, labour contract, survey experiences, etc.). FSS is the most important agricultural sample survey carried out by Italian National Statistical Institute (Istat). Data collection is performed face-to-face by interviewers recruited by Regions (an average of 36 units for each interviewer). Because of the crucial role of the interviewers in the survey, Istat monitors and collects information on their main characteristics. In order to evaluate the measurement error of 2005 FSS, Istat has carried out a reinterview survey with CATI technique based on replicated measurements of the same units. It is finalized to supply sufficiently reliable estimates of the measurement error and its main components. The error is calculated by the difference (in modulus) between the first value collected during the FSS and the second one collected in the reinterview (without reconciliation). The errors have been linked to the characteristics of the interviewers in charge of that wrong data. Then, a study of the relationships between the error rates and the variables describing the interviewer characteristics was conducted using Multiple Correspondence Analysis (MCA). In the sections 2 and 3, FSS and reinterview survey are described respectively; in the section 4 it is shown the data analysis, in particular a descriptive study in the sub-section 4.1 and a multivariate analysis in the sub-section 4.2; the conclusions are presented in the section Farm Structure Survey FSS has the scope to collect information on agricultural holdings in the Member States at different geographic levels (Member States, regions, districts) and over periods (follow 1 Massimo Greco, Italian National Statistical Institute, via Ravà 150, Roma, Italy (msmagrec@istat.it); Matteo Mazziotta, Italian National Statistical Institute, via Ravà 150, Roma, Italy (mazziott@istat.it); Adriano Pareto, Italian National Statistical Institute, via Ravà 150, Roma, Italy (pareto@istat.it).

2 up the changes in agricultural sector), thus provide a base for decision making in the Common Agricultural Policy. Two kinds of FSSs are carried out by Member States: a basic survey (full scope Agricultural Census - AC) every 10 years; several sample based intermediate surveys between them. The calendar for the surveys to be held in all Member States is agreed by the Agricultural Statistics Committee of the European Commission. For a given survey year, Member States have to conduct their surveys within the agreed time-frame, thus all the data are as comparable as possible. The FSSs are organised in all Member States on a harmonised base. Whereas the characteristics are based on community legislation, the same data are available for all countries in case of each survey. The variables are arranged into four groups: one general overview with the key variables, and three specialized ones containing detailed data on: land use; livestock; special interest topics: farm labour force, rural development issues as well as management and practices. Data for basic surveys are available in a three-level geographical breakdown of the whole country, the regions and the district; while data for intermediate surveys are only available upon the two-levels of country and regions. Istat is in charge of the survey in Italy and avails itself of Regions statistical offices to carry out data collection. FSS 2005 has been carried out in Italy at the end of the agricultural year 2005 (1st November st October 2005). The Target population of the survey is defined as the set of agricultural holdings with the following characteristics in the 2005 agricultural year: the agricultural area utilised for farming is one hectare or more, or; the agricultural area utilised for farming is less than one hectare if they produce a certain proportion for sale ( 2,500) or if their production unit has exceeded certain physical threshold. Data have been collected with a random sample selected according to a stratified sample design with a take all stratum containing the biggest farms. The sample size is 56,540 selected from the target population. Furthermore all farms resulting from a splitting or a merging of a sampling unit have been added to the sample by the interviewers. Data collection was performed face-to-face by 1,590 interviewers recruited by Regions and using personalised paper questionnaires supplied by Istat. Interviewers training has been carried out by meetings at regional or provincial levels. regional coordinators and supervisors have been assisted by Istat regional staff. The activities of interviewers in the field were monitored by regional offices in charge of the survey. Furthermore, an help desk for interviewers was available to solve any problem met during the data collection or data entry. Each submitted question has been classified by topic and the answer has been sent to every person of our mailing list. A regional or provincial interviewers supervisor has collected the filled in questionnaires in order to check the quality of the data and the work carried out by each interviewer. Data entry has been performed by the interviewers or by staff close to interviewers.

3 3. The reinterview Istat carried out in outsourcing a sample survey using a CATI technique aimed at evaluating the quality of collected data. The sample survey is based on replicated measurement on the same units interviewed by 2007 FSS (McClung, Tolomeo and Pafford, 1990). A set of questions from the original interview is asked once again to a sample of units (reinterview) and the two answers given by the same units to the same question are then matched. When the responses obtained during the reinterview differ from those obtained in the original interview, the difference can be evaluated through the so-called reconciliation (Forsman and Schreiner, 1991). Through this survey methodology, it is possible to estimate the bias, the total response variance and the simple response variance (Lessler and Kalsbeek, 1992): the bias is the difference between the FSS value and the true value (obtained through the reinterview); the total response variance is the sum of the three components of the statistic error: the sampling variance and the correlated and uncorrelated response variance (due to interviewer effect); the simple (or uncorrelated) response variance is the average variance of responses to an item over repeated applications of the measurement process. The goal to estimate the parameters is both to understand the source of the statistic error and to improve the survey quality, operating on the different aspects of the survey phases (survey technique, questionnaire, interviewer training, etc). The reinterview, carried out from May to August 2006, has been based on 2,304 farms interviewed by 8 telephone interviewers divided in early shift and afternoon shift. During the phone interview a comparison with data collected by FSS is performed on the main items of the following sections of the questionnaire: - Agricultural land arable land; permanent crops; fruit trees; permanent grass land and meadow; - Livestock bovines; buffaloes; sheep; goats; pigs; poultries; - Labour force non-family workers regularly employed; non-family workers not regularly employed.

4 4. Data Analysis 4.1 Descriptive Analysis In descriptive analysis, the goal is to quantify the observations by statistic tools in order to investigate and to highlight phenomena. This analysis is based on the percentage errors of the variables of interest; where error is the difference (in modulus) between the first value collected during the FSS and the second one collected in the reinterview (without reconciliation). Two composite measures of response error were generated: the mean absolute percentage error (MAPE) and the proportion of biased responses (PBR). The first is a measure of intensity and the second is a measure of frequency. The mean absolute percentage error measures the average bias in the original FSS responses and it was calculated using the differences between the FSS responses and their corresponding CATI responses for each item of interest. The mean absolute percentage error for the i-th observation is given by: MAPE i 100 = m i m i y ij1 y y j = 1 ij1 ij 2 where y ij1 and y ij2 are, respectively, the original response and the final one in item j for the i-th observation and m i is the number of items of interest. The proportion of biased responses measures the frequency of the errors and is defined as follows: PBR i 100 = m i m i j = 1 x ij where 1 if y ij 1 y ij 2 x ij =. 0 otherwise Table 1 shows, with regard to absolute percentage errors (APE), the minimum, the maximum, the mean, the standard deviation and the median. In the maximum column, all variables (excepting buffaloes) present very high values (in the order of thousands), demonstrating the presence of outliers; it is also confirmed both by the difference between the mean and the median and by the high variability of the APE distributions. The MAPE, calculated on the 2,304 farms, presents a maximum value equal to 6,021.4, the measures of central tendency are not similar and the standard deviation is rather high: these are possible signs about the presence of outliers.

5 Table 1 Descriptive statistics of the APE of the variables of interest Variable N. of Farms Minimum Maximum Mean Std. Dev. Median Arable land 2, , Permanent crops 1, , Fruit trees 1, , Permanent grass land and meadow , Bovines , Buffaloes Sheep Goats Pigs Poultries Non-family workers regularly employed , Non-family workers not regularly employed MAPE 2, , Table 2 presents, concerning the MAPE, the descriptive statistics by geographical area. The results of North West demonstrate that, in this area, the influence of outliers is very strong: the mean and the median values are constantly over the Italy values. The results of Centre and South appear similar to Italy; the North East presents lower values for the mean, the median and the standard deviation. The geographical area distributions of PBR present similar values, excepting for North West that has the median equal to 100 and the mean value is higher than the others. The survey monitoring system forecasts to collect a set of information about the interviewer characteristics (age, sex, degree, work experiences, etc). Not all Regions provide this information to Istat. In the figure 1 it is possible to see the percentage distributions, by Regions, of the presence of interviewer data. From the figure, it seems that in the North and in the Centre of Italy the interviewers data are more complete than the South. In Val d Aosta it doesn t exist monitoring and in Molise data are all incomplete. Data of Trentino Alto Adige, Liguria, Marche, Campania and Basilicata are fully complete. In the Islands (Sicilia and Sardegna) the presence of interviewer data is decidedly low and this is a problem since they are very important regions by agricultural point of view. The red zone size of Piemonte, Lombardia, Veneto, Friuli, Umbria and Abruzzo is very small.

6 Table 2 Descriptive statistics of the MAPE and PBR by geographical area Geographical Area N. of Farms Minimum Maximum Mean Std. Dev. Median MEAN ABSOLUTE PERCENTAGE ERROR (%) North West , North East , Centre , South , ITALY 2, , PROPORTION OF BIASED RESPONSES (%) North West North East Centre South ITALY 2, Figure 1 Bar diagram of the presence of interviewer data by region Italian Regions Piemonte Valle d'aosta Lombardia Trentino-Alto Adige Veneto Friuli-Venezia Giulia Liguria Emilia-Romagna Toscana Umbria Marche Lazio Abruzzo Molise Campania Puglia Basilicata Calabria Sicilia Sardegna Presence of Interviewer Data (%) Complete Data Incomplete Data

7 4.2 The study of the relationships between response error and interviewer characteristics Table 3 shows the main descriptive statistics of the two measures of response bias for each level of the interviewer characteristics (Norris and Hatcher, 1994). The differences across the various interviewer characteristics are not particularly large. The strongest relationship is for age of interviewer. Older (61 and over) interviewers have a median MAPE of 31,5% compared to 21,7% for younger (up to 30) interviewers. Males and interviewers who work in the public sector have slightly higher error rates. It is noteworthy that interviewing experience in other surveys has not a positive effect on the MAPE and PBR and, in fact, interviewers with no previous experience have lower error rates compared to those with previous experience. Finally, as regards educational level, interviewers who are graduate in agrarian sciences have the lowest median MAPE (24,3%). Table 3 Descriptive statistics of the MAPE and PBR by interviewer characteristics Variable N. of Farms MAPE (%) PBR (%) Mean Std. Dev. Median Mean Std. Dev. Median SEX Male 1, Female AGE Up to and over EDUCATION Agrarian Diploma Other Diploma Agrarian Degree Other Degree EXPERIENCE No surveys or more surveys WORK SECTOR Public Private

8 The analysis of the relationships between the error rates and the variables describing the interviewer characteristics was conducted using MCA (Bouroche and Saporta, 1980). The analysis was restricted to 1,411 farms for whom the interviewer characteristics data were completely available. This subset accounted for 61.2% of the sample size. The Kolmogorov-Smirnov two-sample test and Mann-Whitney test were used to determine if complete and incomplete datasets differ significantly. The first test is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples, while the second compares the medians. Figure 2 displays the distribution of the MAPE for the two datasets. Curves are clearly skewed to the right (positive skew): the right tails are indeed longer and the mass of the distributions is concentrated on the left of the plots. Moreover, there are some particularly high values (outliers). Figure 2 Distribution of MAPE scores by presence of interviewer data 500 Kolmogorov-Smirnov Test: Z=1.110 (p=0.17) Mann-Whitney Test: Z adj = (p=0.44) 400 Number of Farms Interviewer Data: Yes MAPE (%) Interviewer Data: No The Kolmogorov-Smirnov test did not indicate any significant difference between the two datasets (Z=1.110, p=0.17) and also the Mann-Whitney test resulted not significative (Z adj =-0.766, p=0.44). The same results were obtained with the PBR.

9 In order to perform the MCA, MAPE and PBR were recoded as categorical variables. The levels were defined as follows: MAPE - Negligible ( 5%), Low (>5% and 20%), Medium (>20% and 50%), High (>50% and 200%), Outlier (>200%); PBR - 0 (0%), 0-50 (>0% and 50%), (>50% and <100%), 100 (100%). Figure 3 shows the projections on the first two dimensions of the different levels of the variables of interest. Figure 3 Multiple Correspondence Analysis biplot 1,2 Other degree 0 1,0 0,8 Up to 30 0 Negligible Other diploma 61 and over Dimension 2 0,6 0,4 0,2 0,0-0,2-0,4-0, North-East Female Low Private Agr. degree 3 or more Centre Public Male Medium North-West Agr. diploma 100 Outlier High South -0,8-1,6-1,4-1,2-1,0-0,8-0,6-0,4-0,2 0,0 0,2 0,4 0,6 0,8 1,0 1,2 Dimension 1 MAPE (%) PBR (%) Geogr. Area Sex Age Education Experience Work Sector The results shows that age and experience are positively correlated with the first factor, which represents also sex and work sector. The second factor may instead be interpreted as a global measure of response error (MAPE and PBR). Besides, each quadrant represents a different level of education: the bottom quadrants depict agrarian studies, while the upper quadrants depict other type of studies. Overall, there seems to be a slight degree of association between error rates and interviewer characteristics. In particular, the proximity between and outliers means that particularly high MAPE values tend to appear more frequently for middleaged interviewers. On the contrary, negligible errors are closely related to the presence of non-agrarian studies.

10 Furthermore, lower error rates occur much more in females and private sector interviewers, whereas higher error rates occur much more in males and public sector interviewers. In short, the results of MCA confirm what is reported in table Conclusions This paper has the aim to study the relationships between response error and interviewer characteristics of the 2005 FSS and to find a specific interviewer profile particularly absent-minded. In order to achieve this goal, it is fundamental to use the reinterview, based on a subset data from the FSS, carried out with CATI technique. In fact, the second survey is used to individuate the errors of the first one and then it is possible to link these errors with the characteristics of their authors. The results, obtained from descriptive and multivariate analysis, firstly indicate that the interviewer young, female, employed in the private sector tends to have lower error rates. However, it is found that high educational level and interviewing experience in other surveys have not a positive effect on the measures of response bias. As regards the geographical divisions, the North-East is the area where the error frequency is lower and, in general, in the North and in the Centre, the monitoring system seems to be better than others areas. Overall, there not seems to be a clean and evident association between error rates and variables describing the interviewer characteristics. These results say that the most important interviewer characteristics are the motivation and the desire to show his skill in the job. In this context, it seems to increase the importance of the interviewers training because it is the best way to motivate and improve the sense of responsibility. Besides, it is necessary to have an efficiency and detailed monitoring system in order to check particular situations and to correct errors. References Bouroche, J.M., and Saporta, G. (1980), L analyse des données, Paris: Presses Universitaires de France. Forsman, G., and Schreiner, I. (1991), The Design and Analysis of Reinterview: an Overview, in Biemer, P., Groves, R.M., Lyberg, L., Mathiowetz, N., Sudman S. (eds.), Measurement Errors in Surveys, New York: Wiley, pp Lessler, J.T., and Kalsbeek, W.T. (1992), Nonsampling errors in surveys, New York: Wiley. McClung, G., Tolomeo, V., Pafford, B. (1990), The Measurement of Response Bias in March 1988 Quarterly On-farm Grain Stocks Data, United States Department of Agriculture, Research and Applications Division, SRB Norris, D.A., and Hatcher, J. (1994), The Impact of Interviewer Characteristics on Response in a National Survey of Violence Against Women, Proceedings of the Survey Research Methods Section, American Statistical Association, pp