Deriving indicators from representative samples for the ESF

Transcription

1 Deriving indicators from representative samples for the ESF Brussels, June 17, 2014 Ralf Münnich and Stefan Zins Lisa Borsi and Jan-Philipp Kolb GESIS Mannheim and University of Trier

2 Outline 1 Choosing a Sample Size 2 Estimation methods for Indicators 3 Weighting 4 Non-response 5 ESF/YEI Sampling and Estimation Recommendations

3 Choosing a Sample Size

4 Variance and Sample Size Estimates can be expected to be more accurate with a higher sample size, in the sense of being closer to the expected value of the estimator. What we would like to do, is to determine a sample size that will assure that our estimate will not exceed a certain error e with probability 1 α. That is, ( ) Prob. ˆθ(s) θ < e 1 α If we can assume that the estimator ˆθ is approximately normal distributed, then the above condition is fulfilled if e ) V (ˆθ z 1 α. 2

5 Necessary Sample Size y Error e is the half CI length for symmetric CIs: e = z 1 α σy 2 n ( e = z 1 α Sy 1 n ) 2 n N (SRSWR) (SRSWoR) With given prior information for σ or S we get n z 2 1 α 2 n e 2 z 2 1 α 2 σ 2 y σ2 y e 2 + σ2 y N (SRSWR) (SRSWoR)

6 Necessary Sample Size p y I For binary response, i.e. µ = P y, we can write P y (1 P y ) n e 2 ( 1 n 1 ) z1 2 α 2 N 1 (SRSWR) assuming n 1 N 1 n N we get n P y(1 P y ) e 2 + z 2 1 α 2 n P y(1 P y ) e 2 z 2 1 α 2 Py (1 Py ) N (SRSWoR) (SRSWR)

7 Necessary Sample Size p y II Necessary sample size for different maximal errors to estimate P y = 0.5, and P y = 0.2 or 0.8, given a confidence level of 95%. P y = 0.5 P y = 0.2 N e = 0.03 e = 0.05 e=0.03 e=

8 Necessary Sample Size p y III The maximum of P y (1 P y ) is 0.25, therefore if p y is used as an estimator for P y a sample size of n z 2 1 α 2 4e 2 should always ensure the necessary precision on a 95% confidence level.

9 Stratified Designs

10 Necessary sample size StrSRS I For y StrSRS we have under WoR the problem is to minimize the sum n = H h=1 n h under the constrains 0 < n h N h, h = 1,..., H and side condition, e z1 2 H α 2 h=1 γ 2 h S 2 y h n h ( 1 n ) h. N h In praxis there are often minimal value m h and maximal value M h for all n h, (0 m h M h N h ). Usually m h 2 if N h 2. Thus we have the solve the above problem under constrains m h n h M h, h = 1,..., H.

11 Necessary Sample Size StrSRS III Necessary sample sizes for two-way stratification: Education Employment high middle low Total Employed N 11 N 12 N 13 N 1. Unemployed N 21 N 22 N 23 N 2. Total N.1 N.2 N.3 N Minimize H h Gg n gh, under m hg n hg M hg, h = 1,..., H, g = 1,..., G, and side conditions for the necessary sample size for each cell in the above table.

12 Necessary Sample Size StrSRS IV Solution via specialised box-constraints optimization algorithm Exact: Gabler, Ganninger and Münnich (2012), Metrika, and Numerical: Münnich, Sachs and Wagner (2012), Journal of Global Optimization, and For integer solution: Friedrich, Münnich, de Vries, Wagner, in preparation. Remark: Overallocation of optimal allocation may be handled easily!

13 Complex Designs

14 Design Effect Working with a more complex sampling design, e.g. one with cluster sampling, will usually have an effect on the variance of the estimators used. Design effect describes this difference: ) V (ˆθp p deff (θ) p = ) V (ˆθSRS SRS where ˆθ p is the used estimator under the complex design p( ) and ˆθ SRS the estimator under SRS, usually WoR. To have a fair comparison, the same (expected) sample size n is assumed under design p( ) and the SRS design.

15 Effective Sample Size The ratio n n eff = deff (θ) p is referred to as the effective sample size and it is the sample size required in a SRS which yields the same precision on a certain estimator as under a given complex sample design. If we select n eff as the sample size under SRS to achieve a certain level of precision, multiplying it by the design effect of the complex sample we get the sample size n p, which should assure the same under the complex design. Hence, n p = n eff deff (θ) p

16 deff Behaviour of the Design Effect Cluster size (b) Homogeneity (ρ)

17 Estimation methods for Indicators

18 The main types of statistics that are of interest are totals and means of the indicator variables. In the following we show how τ y and P y can be estimated for any sampling design introducing the unifying concept of inclusion probabilities.

19 First order inclusion probabilities The selection probability of element k in draw l is of special interest and denoted by ψ k,l. These probabilities vary from draw to draw when sampling without replacement is chosen. We now define: First order inclusion probabilities The first order inclusion probability π k is the probability, that element k is selected in the sample (independently from the draw!). Then: π k = s k p(s) In general, we use the inverse inclusion probability d k = 1/π k (IIP) and refer the quantity d k to as design weight.

20 Properties of first order inclusion probabilities The total probability over all samples: p(s) = 1 s S For fixed sample size n designs, we get π k = n k U and k s 1 π k = N

21 Second order inclusion probabilities In order to gain variances or variance estimates, we need also the second order inclusion probabilities: Second order inclusion probabilities The probability that both elements k and l are drawn in the sample is denoted by π kl = s {k,l} p(s), and is called second order inclusion probability. From this definition, we can conclude that π kk = π k holds.

22 SRS (revisited) As inclusion probabilities for SRSWoR we get π k = n N n(n 1) π kl = N (N 1) for all k U for all k l U

23 StrSRS (revisited) As inclusion probabilities for StrSRSWoR we get π k = n h N h π kl = n h(n h 1) N h (N h 1) π kl = n hn g N h N g for all k U h for all k l U h for all k U h, l U g, and h g

24 The Horvitz-Thompson estimator (HT) I The HT estimator for the total τ y of a variable y is Ŷ HT = i s y i = d i y i. π i i s The HT estimator is unbiased for any design! The variance of Ŷ HT is V (ŶHT ) = k s l s (π kl π k π l ) y k π k y l π l.

25 Horvitz-Thompson estimator II An unbiased estimate of the variance of the HT estimator is ) ˆV (ŶHT = i s j s (π ij π i π j ) π ij y i y j. π i π j The above estimator is only unbiased for designs with strictly positive second order inclusion probabilities.

26 Horvitz-Thompson estimator III As HT-type of estimator for a mean is given by ˆP yht = l s d iy i l s d i = ŶHT ˆN Variance estimation is more complex, as the estimator in not linear, but there exist approximate solutions.

27 Properties of the HT estimator HT estimators: a class linear unbiased estimators. Negative variance estimates may result in some designs! The design effect should be considered. Cassel, C.; Särndal, C.-E.; Wretman, J.H. (1977): Foundations of inference in survey sampling. Wiley. Gabler, S. (1990): Minimax Solutions in Sampling from Finite Populations. Lecture Notes in Statistics, 64. Springer. Hedayat, A.S.; Sinha, B.K. (1991); Design and Inference in Finite Population Sampling, Wiley.

28 General Regression Estimator (GREG) Ŷ GREG = Ŷ HT + (τ x X HT ) ˆB (cf. Särndal et al. 1992, pp. 225) with ˆB as estimate of the coefficients of a linear regression model to explaining the variable of interest y with the auxiliary variables x. Or: Ŷ GREG = i S g i d i y i where g i is the adjustment to the design weight by the linear model.

29 The Idea of Regression Estimators Sample data for two varaibles, y and x: Y µ x x µ^y y

30 Weighting

31 The Hurvitz-Thompson Estimator Revisited The Hurvitz-Thompson estimator Ŷ HT = i S d i y i The HT has very good properties. But, it does not gain in efficiency by any available auxiliary information. Cannot directly consider non-response or other disproportions Additional weights may help!

32 Poststratification The stratification is used after drawing a simple random sample (Lohr, 1999) SRS yields approximately equal proportions with respect to a stratification Correction of over- and under-representation of the categories (strata) Variance formulae of the proportional allocation are applied Poststratification is applied once the necessary information for stratification is not available in the sampling frame but from other sources, e.g. register. Example: nationality (German, non-german) and gender Note: stressing poststratification with many variables may lead to erroneous results

33 Example A simple random sample yields: n y sy 2 W M The proportion of women in the universe is 60%. Hence: SRS PostStr The variability of the sample size in the denominator of ˆV (y PS ) SRSWR was not considered (may be ignored in practice if not too low)!

34 Example A simple random sample yields: n y sy 2 W M The proportion of women in the universe is 60%. Hence: SRS y = ( ) = ˆV (y) SRSWR = ( ) = PostStr The variability of the sample size in the denominator of ˆV (y PS ) SRSWR was not considered (may be ignored in practice if not too low)!

35 Example A simple random sample yields: n y sy 2 W M The proportion of women in the universe is 60%. Hence: SRS y = ( ) = ˆV (y) SRSWR = ( ) = PostStr y PS = = ˆV (y PS ) SRSWR = L γq 2 s2 q = q=1 n q The variability of the sample size in the denominator of ˆV (y PS ) SRSWR was not considered (may be ignored in practice if not too low)!

36 The Calibration and GREG Estimator We learned that the GREG can be written as a HT with specialised weights: Ŷ GREG = i S g i d i y i. The weights come from a regression model which assists the estimation in order to improve the estimates. The better the model, the better the estimates. At least aggregate information is needed for covariates.

37 Weights and Estimation Why are weights so essentially important? Münnich and Burgard (2012) Weights can be used for Correction of the disproportions in the sample E.g. poststratifiction for gender Controlling for response propensities Correction for non-response Control for the variation of design weights Cf. Gelman (2007) discussion Known estimators using weights: GREG family Calibration estimators Expected solution: does there exist a weighting vector such that all (most / small area) estimates of the main variables are considered as calibration constraints? Extension: Benchmarking by penalized calibration

38 Our goal is Get weights w k for estimating ˆτ y = w k y k = d k g k y k, k s k s τ x = k s w k x k whereas these weights satisfy benchmarks by certain areas and domains, certain auxiliary variables. d k = πk 1 k : desgin weights g k = the ajustement weight w k = d k g k k : calibrated weights

39 Non-response

40 Missing Data Everybody has them, nobody wants them! Missing Completely at Random (MCAR) Complete random pattern. The pattern is neutral, i.e. not systematic. Missing at Random (MAR) The pattern is random but conditional on other data. The pattern is systematic, but can be explained by the observed auxiliary data. Missing not at random (MNAR) The pattern cannot be modelled as a random process. The pattern is systematic but cannot be explained.

41 Non-response Bias Source: Schnell, Rainer (2002): Antworten auf non-response.

42 Methods to Handle Missing data Procedures based on the available cases only, i.e., only those cases that are completely recorded for the variables of interest Weighting procedures such as Horvitz-Thompson type estimators or raking estimators that adjust for non-response Single imputation and correction of the variance estimates to account for imputation uncertainty Multiple imputation (MI) according to Rubin (1978, 1987) and standard complete-case analysis We regard multiple imputation as most flexible for multipurpose complex surveys

43 Complete Case Analysis Delete the observation with missing values from the analysis/estimation. Is an elimination procedure. Results in a reduction of observations. All incomplete observed rows of a data matrix are deleted. Large loss of information in case of item non-response. Remaining sample can be falsified, if missing data pattern is systematic (example income) Estimates will be biased if the data was not MCAR!

44 Single Imputation Methods Imputations are means or draws from a predictive distribution of the missing values, and require a method of creating a predictive distribution for the imputation based on the observed data (Little and Rubin, 2002, p. 59). With respect to Little and Rubin(2002) the single imputation methods can be divided into explizit and implizit modeling. When the modeling is explicit a formal statistical model (e.g. multivariate normal) is used for the predictive distribution. When the modeling is implicit the emphasis is on an algorithm, which implies an underlying model.

45 Explicit Modeling Mean Imputation: The mean of the responding units is used as imputed value. Regression Imputation: We have auxiliary variables x i = (x ig,..., x ig ) that are fully observed for all elements in the sample. Then we have the varaible of interest y i that has missing values for some elements in the sample. The imputed values ŷ i are obtained by using the auxiliary data of the nonrespondents in the regression equation: G ŷ i = ˆα + ˆβ g x ig, for all missing y i s. g=1 Ratio Imputation: Similar to regression imputation. For the respondent units the ratio of the sum of their values for the variable of interest Y and a auxiliary variable X is computed. ( The ) imputed value yobs for an certain nonrespondent is receivey by: cf. Little and Rubin(2002), Lee et al. (1994) ( xobs ) x mis

46 Implicit Modeling The Hot Deck Imputation: The imputed value is obtained from the observed data set. A commonly used procedure is to defined a pool of so called donors which are quite similar to the nonrespondent (may be with recourse to auxiliary data). The imputed value is received by drawing recorded values with SRS with replacement from the pool of donors. Nearest Neighbor Imputation: A special case of the hot deck imputation. A distance measure (e.g. absolute deviation) is used to define the distance based on auxiliary variables between a nonrespondent and the respondent units. As imputed values the value of the respondent is chosen which is the closest to the nonrespondent. Cold Deck Imputation: The imputed value comes from an external sources, e.g. a previous realization of the same survey. cf. Little and Rubin(2002), Longford (2005)

47 Response Homogeneity Classes In practice often the different imputation methods are performed within different groups or classes. The groups consist of similar elements To identify such groups or classes auxiliary variables may be used This procedure ensures that imputed values are obtained which are close to similar respondents Cf. Särndal and Lundstrøm (2005)

48 Weighting Methods

49 Calibration for Non-response Using the calibration approach directly yields biased estimates Update of calibration weights Correction of response probability Response homogeneity classes before calibrating the first order inclusion probabilities π k π k;nr We need auxiliary information with which we can explaine the response behavior sufficiently good. For example, if we know that employed and unemployed persons have different response propensities than we should calibrate on the variable employment.

50 Example: Unit Non-response in volunteer panels sample Population Microcensus Access Panel D-SILC 1 % Microcensus (MC) unit non-response 1% sample of German population 4 rotational quarters Access Panel (DSP) D-SILC strat. sample n=14,100 Recruitment out of the latest MC quarter Unit non-response due to self-selection (recruitment rates vary ernomously) Stratified sample from access panel Population estimates using weights

51 Multiple Imputation

52 The Multiple Imputation Principle (1) Y 1 Y 2 Y Estimate 1 Y 1 Y 2 Y 3 NA NA NA NA 1 Y 1 Y 2 Y Y 1 Y 2 Y 3 3 Estimate 2 MI estimate MI inference Estimate 3

53 The Multiple Imputation Principle (2) ˆθ, ˆV (ˆθ) Imputed data set 1 ˆθ (1), ˆV (ˆθ(1) ) Imputed data set 2 ˆθ (2), ˆV (ˆθ(2) ) Complete data missing values Incomplete data ˆθ MI ˆV MI (ˆθ) Imputed data set m ˆθ (m), ˆV (ˆθ(m) )

54 Variance Estimation under Multiple Imputation Multiple imputation (Rubin, 1987): ˆθ (j ) and ˆV (ˆθ(j ) ) Multiple imputation point estimate ˆθ MI = 1 m Multiple imputation variance Estimate T = W + (1 + 1 m )B with within imputation variance W = 1 m between imputation variance B = 1 m 1 m j =1 m m ˆθ (j ) j =1 ˆV (ˆθ (j ) ) und j =1 (ˆθ (j ) ˆθ MI ) 2 Problem: the imputation has to be proper in Rubin s sense.

55 Miscellaneous for Imputation Development for MI is on-going Hiding values to achieve anonymity is unlikely to be good! Regional patterns and other peculiarities have to be considered Imputer and analyst should not differ(future task) Software available Mice Baboon (R - language and environment for statistical computing and graphics)

56 Final methodological remarks

57 Recent developments in survey statistics Nowadays, survey optimization is under reconstruction Model-based estimation becomes more and more important Researchers want to use data (later) Sophisticated design weights yield problems Many sources of information are to be considered Administrative data Register data Big data Data are used for deriving regional policies Small area and domain estimation Unit- versus area-level models Designs weights have to be considered properly Münnich and Burgard (2012) Burgard, Münnich, and Zimmermann (2014, 2015)

58 Idea of small area estimation Parallel estimation of many areas Borrow strength from info from other areas (Rao, 2003) Small area model (Battese, Harter, and Fuller, 1988): The model has two components, a design based one similar to a GREG, and model based one. The two components are weighted according to amount of variation that can be explained by the membership of an area/domain. If much of the variation can be explained by area/domain membership more weight is given to the design part and vice versa. Fay and Herriot (1979) area-level model similar. Planned and unplanned areas/domains.

59 Small area estimation in practice 15 Variable of interest Auxiliary information

60 ESF/YEI Sampling and Estimation Recommendations

61 Representative Requirements Representative samples are randomly selected at the level of the socio-economic characteristics (variables) of the participants as captured by the output indicators covering personal data (gender, employment status, age, educational attainment and household situation). Representativeness shall also relate to the regional dimension of the output indicators. Regional representativeness shall be ensured to one NUTS level lower than the level of the programme.

62 StrSRSWoR and Allocation How could this requirement be addressed? An obvious choice would be to use a stratified random sample, with a simple random sample without replacement in each stratum and a proportional allocation of the sample size. To solve the rounding problem a Cox algorithm could be used. And a two-way stratification table, with region being on dimension and the cross-classification of all output indicators the other. This approach would answer to the naïve understanding of representativeness.

63 StrSRSWoR and Allocation Alternatively, selecting necessary or optimal stratum specific samples size to obtain a certain degree of accuracy or error tolerance. Using box-constraints optimization, as we have seen in the section on selecting sample sizes.

64 Problems with High Stratification There could be strata that are very sparsely populated. For proportional allocation this could result into the selection of only one or zero elements in a stratum. Exact variance estimation becomes then impossible. However, approximations or conservative variance estimates (SRS) can be used instead. If stratum specific minimal sample size are set the required total sample size might be to high.

65 Domain Estimation Stratification can be tool to select at least a minimum number of elements for each domain of interest (e.g. region X all output indicators). This would be necessary to use the types of (design-based) estimators we saw to estimate the indicators for each stratum/domain of interest. If extensive stratification is not feasible some of the strata resulting from crossing region and output indicators have to be collapsed. Estimates for each domain of interest can still be obtain by the using so called model-based estimators. This would however require having auxiliary information on the elements in the non-sampled domains. Results will be more prone to bias.

66 Sampling Frame Is it possible to build sampling frames that cover the populations of interest and segregate them from each other? Are the frames enriched with the stratification variables? Are the investment priorities identifiable? Non-overlapping Samples Selection non-overlapping samples from the same frame. Selection non-overlapping samples from different overlapping frames. If multiple samples have to be selected that covering the same, or parts of the same, population, then this can sprout highly complex sampling designs. Inclusion probabilities can become very difficult to determine and to calculate. Select as few samples as possible to cover the same population!

67 Non-response It is difficult to give specific recommendations in advance without having any knowledge of the response behavior. In general: If a survey has single variable that is of major interest (e.g. a particular indicator), single imputation or calibration might by appropriate. If the survey has multiple equally important variables all affect by nonresponse then it might be worth to take multiple imputation into consideration.

68 Cross National Comparability For the comparability of results across countries two important things: Sample sizes should be planned with the same error in all countries. Translation of a common questionnaire (e.g. in English) into national languages. For further good practice on cross naional comparability see ESS Methodoloy: