EC969: Introduction to Survey Methodology Peter Lynn Tues 1 st : Sample Design Wed nd : Non-response & attrition Tues 8 th : Weighting Focus on implications for analysis
What is Sampling? Identify the population of interest Select (sample) some members of the population (units) Study the sample Draw inferences about the population Examples: Sampling pasta from a pan Sampling apples from a market stall
Some Terminology Sample statistics (e.g. mean income in sample, x ) can be used as estimates of population parameters (e.g. mean income in population, X ) i.e. Xˆ = x
Sampling Distribution I The set of estimates that could be obtained from all the samples that could be selected under the chosen sample design; SD is simply a statement of what values the estimate could take, and the frequency with which it will take them; For a single survey, frequency is equivalent to probability.
Sampling Distribution II Can be drawn as histogram or graph, or presented as a table For most sample designs and estimates, it will approximate to a Normal distribution Consequently, it has known statistical properties (e.g. symmetrical) A (Normal) Sampling Distribution: 5000 4000 Frequency 3000 000 1000 0 1 9 17 5 33 41 49 57 65 73 Estimate
Sampling Variance Sampling variance is a measure of the spread of the sampling distribution Small sampling variance is desirable General form of variance: ( i S = X -X) (N -1) where the sum is made over the N values in the distribution; X i is the i th value (i = 1,,... N); and X = X N i S is standard deviation If the distribution is a sampling distribution, then S is called the sampling variance (SE ) and S is called the standard error (SE)
Sampling Variance - Example From Stuart (1984), population of 6 individuals with associated measures: A B C D E F 6 8 10 10 1 Want to sample and estimate mean. There are 15 possible samples of from 6 Sample 1 3 4 5 6 7 8 9 10 11 1 13 14 15 Members of sample A B A C A D A E A F B C B D B E B F C D C E C F D E D F E F Values in sample 6 8 10 10 1 6 8 6 10 6 10 6 1 8 10 8 10 8 1 10 10 10 1 10 1 Sample mean 4 5 6 6 7 7 8 8 9 9 9 10 10 11 11 The bottom row of the table above is the sampling distribution - a list of all the possible estimates that could arise from this design of sampling n= from N=6.
Example continued We can draw this sampling distribution as a histogram or graph: 3.5 3.5 Frequency 1.5 1 0.5 0 4 5 6 7 8 9 10 11 Estimate 3.5 Frequency 1.5 1 0.5 0 3 4 5 6 7 8 9 10 11 1 Estimate
Example continued We can calculate the variance of this sampling distribution: Sample Mean Frequency Mean x freq Deviation from mean (8) Squared deviation from mean Freq. x Sq.dev. X i X i -X ( X i -X) ( X i -X) 4 1 4-4 16 16 5 1 5-3 9 9 6 1-4 8 7 14-1 1 8 16 0 0 0 9 3 7 1 1 3 10 0 4 8 11 3 9 18 N = 15 10 64 X = 10/ 15 = 8 S = 64/14 Note that just one of the 15 samples contributes a quarter of the total variance.
Sampling Bias Sampling bias is a measure of the location of the sampling distribution, relative to the true population value A sample design is unbiased if the mean of all possible estimates equals the true population value i.e. an unbiased design will have a sampling distribution that is centred around the true population value (Technically, bias is a property of the estimator used, but for brevity we shall refer to sample designs that permit the use of design-unbiased estimators as unbiased sample designs)
Sampling Error Sampling error is difference (due to sampling) between sample estimate and true value It is therefore the product of the effects of variance and bias In sample design, we wish to minimise expected sampling error Terminology: Small (expected) sampling error = High accuracy Small sampling variance = High precision So high accuracy requires high precision and absence of (or minimal) bias
Key Aim of Sample Design To minimise the cost required to deliver a given level of accuracy or To maximise the accuracy delivered by a given budget
Assessing a Sample Design Knowledge that a survey estimate is in error does not tell us whether or not the sample design was biased Conversely, failure to detect any bias does not mean none is present The only way to know if a sample is biased is to evaluate the design, not the sample We should strive to design samples in a way that ensures bias cannot arise
Other Sources of Survey Error Survey error is difference between survey estimate and true value Sampling error is not only component of survey error. Survey error includes both sampling error and non-sampling error. Sources of nonsampling error include: Non-response bias (if non-respondents differ systematically from respondents) Response bias (e.g. due to leading questions, poor interviewer training,... ) Coder effects Interviewer effects etc... See Groves (1989)
Mean Square Error Variance Bias Errors of Non-observation Observational Errors Errors of Non-observation Observational Errors Coverage Sampling Non-response Interviewer Respondent Instrument Coder Coverage Sampling Non-response Interviewer Respondent Instrument Coder
Random (Probability) Sampling Selections are made from a specified and defined population Each unit is selected with a known and nonzero probability, so that every unit in the population has a known chance of selection The method of selection is specified, objective and replicable Simple Random Sampling (SRS) is a special case of random sampling, where every unit in the population has an equal and independent probability of selection.
Other Sampling Approaches Convenience Sampling: choose the most readily available units to study Purposive Sampling: use (expert) judgement to choose a representative sample Quota Sampling: combination of convenience and purposive. Quota Sampling is quicker and cheaper than probability sampling but it does not ensure absence of bias. Neither does it permit any assessment of bias or variance. Bias will arise if survey measures are correlated with accessibility and/ or willingness of respondents to take part.
Advantages of Random Sampling Protects against selection bias Enables precision of estimates to be estimated, and hence confidence intervals Likely to suffer less non-response bias (for interview surveys) Enables non-response bias to be assessed
Trade Offs In practice, sampling usually involves trade offs, e.g.: Sampling error vs. non-sampling error Sampling variance vs. sampling bias One source of sampling variance vs. another All sources of error vs. cost
Var y Mean Squared Error ( ) = Ey [ Ey] MSE( y) = E y Y ( ) ; ( ) = ( ) [ ] Ey Ey + ( Ey ( ) Y) = Var( y) + Bias ; Similarly, Root Mean Square Error (RMSE): RMSE( y) = MSE( y). The RMSE is the expected value of the sampling error - taking into account both bias and variance. For unbiased designs, RMSE = SE.
Confidence Intervals I CIs follow from sampling distribution e.g. in the sampling distribution discussed on 1.10-1.1, eleven out of the fifteen samples gave an estimate within plus or minus of the true population mean of 8 So, we would be 73% confident (11/15) that our sample estimate was within of true value In other words, the estimate plus or minus would constitute a 73% confidence interval for the population value BUT, we can only say this because in this case we know all the values in the sampling distribution. This would not normally be the case.
Confidence Intervals II Pr [Est-C(SE) < True value < Est+C(SE)] = 100(1-a). e.g.: Pr [Est-SE < True value < Est+SE] = 0.95 We need to estimate SE This can be done using only the survey data (an amazing result!)
Estimating SE For simplicity, consider SRS. S is the population variance - for example the variance of household income across all households in GB s is the sample variance - the variance of household income across all households in our SRS SE is the sampling variance - the variance of mean household income across all the samples that we could happen to select using SRS. For SRS, SE S = 1 n n N n 1 is known as the finite population N correction factor, and can usually be ignored
Estimating SE continued Thus, sampling variance (SE ) depends only on the population variance (S ) and the sample size (n), in the case of SRS. (We will see later in the course that in the case of complex sample designs, sampling variance also depends on other aspects of the design) Now, s is an unbiased estimator of S (intuitively obvious), so we can estimate SE as follows: SE s = n $ So, using just the information collected from the sample, we can assess the precision of estimates.
Design Effects We have seen that sampling variance depends upon population variance and sample size Sampling variance is also affected by aspects of sample design: clustering, selection probabilities, stratification etc. The effect of sample design on sampling variance is called the Design Effect (DEFF) Note that the effect can (and will) be different for different estimates from the same survey We should not refer to the design effect for a particular sample design, though we can usefully refer to predicted average design effects under certain simplifying assumptions.
Definition of Design Effect DEFF is the ratio of the actual sampling variance to SRS sampling variance for sample of same size: DEFF = S.E. S.E. c SRS where S.E. C is the sampling variance of the complex design under consideration, and.e. SRS S is the sampling variance of a simple random sample of the same size.
Design Factor; Effective Sample Size The equivalent ratio of standard errors is known as the design factor, DEFT: DEFT = S.E. S.E. C SRS = DEFF The effective sample size, neff, is the size of a simple random sample that would have produced the same precision as the actual (complex) sample design under consideration: neff = n DEFF ; DEFF = n neff. e.g. if DEFF= and n = 1000, then neff = 500
Components of DEFF The overall design effect can be split in components due to different aspects of sample design. For example, DEFF Strat is the design effect due to stratification; DEFF Clus is the design effect due to sample clustering, etc. If these were the only two aspects of sample design that affect sampling variance (relative to SRS), then: DEFF = DEFF DEFF Tot Strat Clus Later in the course, we will learn how to control and how to estimate the main components of DEFF.
Uses of DEFF/ neff DEFF is a useful concept - we can use it to compare the precision of alternative designs. This is important at the sample design stage. Prediction of DEFFs for different designs allows us to make informed design choices. Neff can be useful for explanation of precision to clients, data users etc. both at design stage and post data collection. At reporting stage, it is important to correctly estimate and report standard errors. This requires correct estimation of design effects. Often, design effects are published only for a range of key estimates for a small selection of key subgroups.
Variable Sampling Fractions I Random sampling does not require that selection probabilities are equal, only that they are known and non-zero. We sometimes sample with unequal probabilities. Think of the population as being divided into I subsets (i = 1,... I), with N i units in the i th subset. If we sample separately from each subset, then we call the subsets sampling strata. If we sample n i units from stratum i, then the sampling fraction (selection probability) in that stratum is n i /N i. f i = n N i i
VSFs II For unbiased estimation, each sampled unit must be assigned a weight in inverse proportion to its selection probability. This is usually referred to as the sampling weight or design weight, w i:
Standard Errors for Stratified Sampling I Var ( x) Ni = S i 1 N n where S i = ( x) stratum i i ni N i - (4-1) Var i is the variance within ( Remember,.E.( x) Var( x) S = ) As with SRS, we estimate standard errors by simply substituting s i for S i : Vˆar ( x) Ni si = 1 N n i ni N i - (4-)
Design Effect due to VSFs If we can assume the stratum variances to be equal, the expression for the effective sample size becomes: DEFF VSF = i n n i N N i where n i and N i are respectively the sample size and population size in stratum i and n and N are the total sample and population size.
Examples Design 1: Stratum N i % n i % England 38,600,000 84.5 600 60.0 Scotland 4,800,000 10.5 00 60.0 Wales,300,000 5.0 00 0.0 45,700,000 1,000 DEFF = 1.6 Design : Stratum N i % n i % England 38,600,000 84.5 400 40.0 Scotland 4,800,000 10.5 300 30.0 Wales,300,000 5.0 300 30.0 45,700,000 1,000 DEFF = 1.83
Graphical illustration of n êff The graph on the next page shows the relationship between the proportion of the sample taken from a stratum with a relatively high sampling fraction (x-axis) and the consequent loss of precision, as measured by the design effect (y-axis). The three lines relate to three different possible relative weights: :1, 4:1 and 10:1 (i.e. w1=1 in all cases). Obviously, these examples are all for the simple case of i=. It demonstrates the first two bullet points on the previous page.
Graphical illustration of continued n êff 3.4 3 DEFF VSF.6. 1.8 1.4 1 0 0. 0.4 0.6 0.8 1 n1/n w= w=4 w=10
Multi-Stage Sampling The units in the population are arranged hierarchically A 3-stage design would entail: Primary sampling units (PSUs) Secondary sampling units (SSUs) Sample elements It would be necessary to assign every element uniquely to one SSU and every SSU uniquely to one PSU Stage 1: select sample of PSUs Stage : select sample of SSUs within each selected PSU Stage 3: select sample of elements within each selected SSU
PSUs, SSUs, Elements Example: general population survey PSUs might be postcode sectors SSUs might be households Elements might be persons Example: business survey PSUs might be companies SSUs might be workplaces Elements might be employees Note that there could be any number of stages:, 3 or 4 are common
Why Multi-Stage Sampling? No frame of elements available, but frame of PSUs available (example: national sample of school pupils, where schools could be PSUs) Cost of data collection (example: general population sample involving face-to-face interviewing) Access to elements may only be via gatekeepers (examples: students, employees, trainees) Data quality (example: in the case of face-toface interviewing, field work can be better supervised if in clusters)
Design Choices (clustering): Some General Points Larger clusters will generally result in larger design effects due to clustering (see later) But larger clusters will also generally result in larger cost savings (e.g. field interviewers, gatekeepers) Necessary to make an appropriate compromise: i.e. where cost saving outweighs loss in precision, to produce higher overall accuracy per unit cost
Selection Probabilities With multi-stage sampling, the selection probability of each element is the product of the (conditional) selection probabilities at each stage e.g. probability of sampling unit i in SSU j in PSU k is Pr ijk = Pr (k) x Pr (j k) x Pr (i j,k) For unbiased estimation, we need to weight each sampled element ijk by w ijk = 1/Pr ijk So, it is important to control and record the selection probabilities at each stage.
Design Effects due to Clustering Clustering tends to increase sampling variance (but this is partly offset by the fact that a larger sample size can be obtained for any given cost). This is because units within a cluster tend to be more homogeneous than units as a whole. Clustering is therefore tending to have the opposite effect to stratification.
Example of Homogeneity of Clusters Population of 6 people, with values of 1, 1,,, 3, and 3. The population variance is: Var (X) = (4 x 1 + x 0 )/6 = /3. a) divide population into 3 clusters: (1,1) (,) (3,3). No variance within clusters. But variance between the cluster means is: Var (X B ) = ( x 1 + 1 x 0 )/3 = /3. b) divide the population into clusters: (1,,3) (1,,3). No variance between cluster means. But variance within each cluster is: Var (X W ) = ( x 1 + 1 x 0 )/3 = /3. So, with design a) all the variance is between clusters - clusters are perfectly homogeneous. With design b), clusters are as heterogeneous as the population as a whole, so cluster sampling would not cause a loss in precision.
Example continued If we sample one cluster (and then include all elements), design a) has a sampling variance of /3; design b) has a sampling variance of 0. This illustrates the general point that sampling variance will be greater if clusters are relatively homogeneous - i.e. not like b). Typically, the sorts of units that we tend to use as PSUs are relatively homogeneous, so in practice clustering nearly always results in a design effect due to clustering which is greater than one. Examples: people within postcode sectors, pupils within schools, students within classes employees within firms.
Intra-Cluster Correlation The design effect due to clustering is: ( b ) DEFFCL = 1+ 1 ρ where b is sample size per cluster (in practice b may vary slightly, in which case mean cluster size provides an adequate approximation), andρ (roh) is the intra-cluster correlation. ρ = 0: randomly sorted clusters ρ = 1: perfectly homogeneous clusters Note that ρ is a population characteristic relating to the chosen definition of PSU, but sample design should involve a careful choice of b. e.g. b=10: if ρ = 0 then DEFF CL = 1; if ρ = 1 then DEFF CL = 10; more realistically, if ρ = 005. then DEFF CL = 145..
Example of Intra-Cluster Correlations From the British Social Attitudes Survey: Variable $ρ b DEFT $ DEFT $ if b=10 Household size 0.070 16.6 1.45 1.8 Owner-occupier 0.31 16.5.14 1.75 Has telephone 0.10 16.5 1.61 1.38 Asian 0.334 8.3 1.86 1.53 Roman Catholic 0.037 16.4 1.5 1.15 Not racially prejudiced 0.01 8.4 1.08 1.03 Extra-marital sex wrong 0.044 8.3 1.15 1.08 Dodging VAT is OK 0.01 8. 1.07 1.04 Note $ρ is low for attitudinal variables, so design effects small. But $ρ large for variables related to ethnicity and housing type. Thus, the most effective degree of clustering might be greater for an attitude survey (fewer, larger clusters) than for a housing survey.
Example: Estimating Design Effects in Stata set mem 00m (04800k) use "M:\Peter\BritCrime\Sep0 on\bcs01_pt1.dta" svyset psu area svyset strata pfa svyset pweight hhdwgt svymean vandalis, subpop(typea) Survey mean estimation pweight: hhdwgt Number of obs = 18199 Strata: pfa Number of strata = 4 PSU: area Number of PSUs = 98 Subpop.: typea==1 Population size = 433336 Mean Estimate Std. Err. [95% Conf. Interval] Deff Vandalis.16683.0070467.1088381.1364985 1.5799
Sampling: Summary I Survey error: bias and variance components Sample design: can contribute to both Bias: Design bias: Is removed by design weighting Coverage bias: can be reduced by weighting (see later) Variance, depends on: Sample size Population variance (Proportionate) Stratification Variable selection probabilities (disproportionate stratification) Clustering
Sampling: Summary II Important to estimate standard errors appropriately (design effects) Use software that allows you to do this easily (STATA, SUDAAN, SAS?) Make sure you understand the design, so that you can specify it correctly (PSUs, strata, selection probabilities at each stage) Make sure relevant variables are available on the data set (PSU, strata, design weights) In general, s.e.s that assume SRS (e.g. SPSS base) will be underestimates if sample is clustered and/or weighted. Models will be over-fitted (true alpha larger than notional alpha).