Cross-Sectional Vs. Time Series Benchmarking in Small Area Estimation; Which Approach Should We Use? Danny Pfeffermann

Transcription

1 Cross-Sectional Vs. Time Series Benchmarking in Small Area Estimation; Which Approach Should We Use? Danny Pfeffermann Joint work with Anna Sikov and Richard Tiller Graybill Conference on Modern Survey Statistics, Colorado State University, Fort Collins, 2013

2 What is benchmarking? Y - target characteristic in area d at time t, y- direct survey estimate, Y ˆ model - estimate obtained under a model. d 1,2,..., D Areas, t 1,2,... Time Benchmarking: modify model based estimates to satisfy: ˆ D model by d 1 B t ; 12 t =,,... ( B t known, e.g., B D b y ). t d 1 b fixed coefficients (relative size, scale factors, ). Requirement: B t sufficiently close to true value D by d 1. When B t function of { y} internal benchmarking. 2

3 Why benchmark? 1. The area model-dependent estimators may not agree with the direct survey estimator for an aggregate of the small areas, for which the direct estimator is deemed accurate. Guarantees consistency of publication between modeldependent small area estimators and design-based estimator in a larger area, which is required by statistical bureaus 2. Corrects for the bias resulting from model misspecification? 3

4 3. Problems considered in present presentation Consider e.g., a repeated monthly survey with sample estimates y in area d at month t. Denote by Yˆ time-series ( Yˆ cross-section ) the optimal unbenchmarked predictor under an appropriate times series (cross-sectional) model. 1- Should we benchmark, Yˆ time-series or Yˆ cross-section? 2- Does it matter which cross-sectional benchmarking method we use? Distinguish between correct model specification and when the model is misspecified. 4

5 Problems considered in present presentation (cont.) 3- Develop a time series two-stage benchmarking procedure for small area hierarchical estimates. First stage: benchmark concurrent model-based estimators at higher level of hierarchy to reliable aggregate of corresponding survey estimates. Second stage: benchmark concurrent model-based estimates at lower level of hierarchy to first stage benchmarked estimate of higher level to which they belong. 5

6 Example: Labour Force estimates in the U.S.A 6

7 Content of presentation 1. Review cross-sectional and times series benchmarking procedures for small area estimation; 2. Compare cross-sectional and times series benchmarking: 2.a Under correct model specification 2.b When models are misspecified 3. Review two-stage cross-sectional and times series benchmarking procedures; (if time allows!!) 4. Apply time series two-stage benchmarking to monthly unemployment series in the U.S.A. 7

8 Single- stage cross-sectional benchmarking procedures (Pfeffermann & Barnard 1991): No BLUP benchmarked predictor exists!! Pro-rata (ratio) benchmarking (in common use), Limitations: Y ˆ Y ˆ b y b Y ˆ ) ; bmk model D D model d, R d / d1 d d k1 k k 1- Adjusts all the small area model-based estimates the same way, irrespective of their precision, 2- Benchmarked estimates not consistent: if sample size in area d increases but sample sizes in other areas unchanged, Yˆ bmk d, R does not converge to true population value Y d. 8

9 Additive single-stage cross-sectional benchmarking Y ˆ Y ˆ ( b y by ˆ ); bmk model D D model d, d 1 k k A d k k1 k k D b d 1 d d 1. Coefficients { d } measure precision (next slide); distribute difference between benchmark and aggregate of model-based estimates between the areas. Unbiased under correct model. If d 0 nd model ˆbmk Y Yˆ A Y consistent. d, d d If ˆ model k Plim( Y y ) 0 Area k with accurate estimate n k k does not contribute to benchmarking in other areas. ˆ bmk Easy to estimate variance of Yd, A. 9

10 Examples of additive cross-sectional benchmarking Wang et al. (2008) minimize model s.t. by ˆ d d by d d, D D bmk d1 d1 ( ˆ ) D bmk φ EY d 1 d Y d d, A A. Sol.: d 2 under F-H 1 D 1 2 d bd / k 1 k bk λ. { φ d } represent precision of direct or model-based estimators. ˆ model 1 φ d [ Var( Y d )] Battese et al model D ˆ ˆmodel 1 φ bd[ Cov( Yd, d Y )] 1 k Pfeffermann & Barnard φd [ Var( y d )] Isaki et al k In practice, model parameters replaced by estimates. Bell et al. (2012) extend results to multiple constraints. 10

11 Self benchmarking and other benchmarking procedures You and Rao (2002) propose self-benchmarked estimators for unit-level model by modifying the estimator of β. Approach extended by Wang et al. (2008) to F-H model. Wang et al. (2008) consider alternative self-benchmarking, 2 x b σ. obtained by extending the covariates x i to (x, ) 11 i i i Di Ugarte et al. (2009) benchmark the BLUP under unit-level model to synthetic estimator for all areas under regression model with heterogeneous variances. Lahiri (1990), You et al. (2004), Datta et al. (2011), Nandram and Sayit (2011) and Steorts and Ghosh (2013) consider Bayesian & empirical Bayesian benchmarking.

12 Small area single-stage time series benchmarking Basic idea: Fit a time series model to direct estimators in all the areas, incorporate benchmark constraints into model equations. Variances obtained as part of model fitting. Pfeffermann & Tiller (2006) consider the following model for total unemployment in census divisions. yt ( y t,, ydt) = = sampling errors. Yt ( Y1 t,..., YDt) = true division totals, 1 direct estimates, et ( e1 t,, edt) 2 2 y Y e; E( e ) 0, E e e Σ Diag[,, ] t t t t t τt 1,, t D,, t. Division sampling errors independent between divisions but highly auto-correlated within a division, and heteroscedastic. Sampling rotation: (4 in, 8 out, 4 in). 12

13 Time series model for division d Totals Y assumed to evolve independently between divisions according to basic structural model (BSM, Harvey 1989). Model accounts for covariates, stochastic trend, stochastically varying seasonal effects and random irregular terms. Model written as: Y z; Tdd, t1 (state-space) Errors mutually independent white noise, E( ) Q. d ARIMA, regression with random coefficients and unit & area level models can all be expressed in state-space form. 13

14 Combining the separate division models y Y e Z e (measurement equation) ; t t t t t t (,..., ) 1, t t D t t t1 t t D D d T (state equation) ; Z z, T T ; - block diagonal E 0, E Q Q, E 0, t. t t t D d t Benchmark constraints D D MODEL D b y b z b Y d1 d1 d1 be =0, t = 1,2,... D d=1 (imposed in model equations). 14

15 Time series benchmarked predictor Denote by α ˆt,u the state predictor at time t without imposing D the constraint be = 0 at time t, but imposing the d=1 constraints in previous time points. Define, [ ( ˆ ft Var bz, u )] ; D d 1 Cov[( ˆ ), b z ( ˆ )]. dft, u d 1, u D ˆ bmk z 1 ˆ z D D Y ˆ, u dft b y b z ft d 1 d 1, u D = 1 b d 1 z dft ft 1. ( ). 15

16 Properties of time series benchmarked predictor Y ˆ z ˆ z ( b y b z ˆ ) bmk 1 D D, u dftft d 1 d 1, u. Y ˆ bmk member of cross-sectional benchmarked predictors, Y ˆ Y ˆ λ ( b y by ˆ ) (Wang et al. 2008). bmk model D D model d, d k 1 k k A d k1 k k ˆ model Y = z αˆ un-benchmarked predictor at time t;,u b ; λ b 1 D ˆmodel ˆmodel φ 1 2 = =b Cov Y D -Y, b k=1 kt Ykt Y kt zδdft k 1 kt b kt Pfeffermann & Barnard (1991). { [( ) ( )]} -1 16

17 Properties of time series benchmarked predictor (cont.) (a)- unbiasedness: if ˆ bmk EY ( d,t-1 Yd, t 1) 0 ( ˆ bmk EY Y ) 0. To warrant unbiasedness under the model, it suffices to initialize at time t= 1 with unbiased predictor. (b)- Consistency: Plim( y Y ) 0 & Plim( ˆ bmk Y y ) 0 n d Plim( ˆ bmk Y Y ) 0 (even if model misspecified). n d (c) ( ˆ bmk Var Y Y ) accounts for variances and covariances of sampling errors, variances and covariances of benchmark errors D d1 d1 D b y b z and their covariances with division sampling errors, and variances of model components. n d 17

18 Comparison: cross-sectional & time series benchmarking Generate series { y, t 1,2,...,75} for areas i 1,...,25 independently between the areas from model: ti y ti Y e ; e N(0, ) ti Y x L S ; ti ti ti ti 1 ti ti, ee tti E ti i t i, t, 1,2,...,75. L L R ; R R, ti t1, i t1, i Lti ti t1, i Rti N N 2 2 Lti (0, L), Rti (0, R) ; S cos S sin S ; ~ N(0, ) * 2 jti j jt, 1, i j jt, 1, i jti jti S S sin S cos S ; ~ N(0, ) * * * * 2 jti j j, t1, i j j, t1, i jti jti S 6 S S ; 2 j/12, j ti jti j j1 Used by BLS for production of Labor Force estimates. 18

19 Cross-sectional model operating at time t By repeated substitutions, for fixed starting values { L, R, S, S, j 1,...,6} (same in all areas), the cross-sectional * 0 0 j0 j0 model operating in area i at time t is the F-H model y x u e, ti 0 1 ij ti ti j 6 5 * 0 L0tR0( Sj0cos tsj0sin t), j1 6 j1 6 t t 6 t 5 t j * j ti Lli ( ) Rli jli cos ( ) jli sin ( ) l1 l1 j1 l1 6 j1 l1 6 u t l t l t l Eu ( ti ) 0, β 0 and 19 j t ( ti) ut L R ( ) S6 l1 Var u t t l t. σ constant across the areas for given t. 2 ut.

20 Simulation set up Divided the 25 areas into 5 equal groups. 2 Var( ) 0, Var( ) , 2 4 R Rti Var( ) S Sti Variances of similar magnitude to variances estimated by BLS for division total unemployment series (after scaling). L Lti Sampling errors e ti in first 4 groups generated from AR(1) models eti et 1, i ti, with different s and 2 e Var( eti) in 2 2 different groups: ( ρ 1,σ 1e ) = (0.15,3), ( ρ 2,σ 2e ) = (0.75,3), ( ) (0.15,5.5) 2 2 ρ 3,σ 3e =, 4 4e ( ρ,σ ) = (0.75,5.5). 20

21 Simulation set up (cont.) For group 5 used variances and autocorrelations estimated for scaled total unemployment series in division of West North Central. 4.4 tti First three autocorrelations are 0.38, 0.21, Autocorrelations at lags around β 1 = 1.5 and started with R 0, L 1, S S 0, j 1,...,6, * j 0 j such that E( Y ) = (1+1.5 x ) for all ( t,i ). ti i We compare the bias and RPMSE at t=25,50,75. 21

22 Simulation set up (cont.) For each method re-estimated model parameters at each of three time points with 2 u estimated by F-H method and BSM variances estimated by mle. True σ at three time points are σ , σ , σ u Variances & autocorrelations of sampling errors known. Same model parameters for five areas in each group. Used true U.S.A State x-values. Results in figures below are group averages of empirical bias and RPMSE over 400 simulated series. 22

23 Root Prediction MSE under correct model specification 23

24 Root Prediction MSE under correct model specification 24

25 Root Prediction MSE under correct model specification 25

26 Prediction bias when excluding covariate from the model 26

27 Prediction bias when excluding covariate from the model 27

28 Prediction bias when excluding covariate from the model 28

29 Mean R- Square when regressing y on x Group Time σ ut t= t= t=

30 Root PMSE when excluding covariate from the model 30

31 Root PMSE when excluding covariate from the model 31

32 Root PMSE when excluding covariate from the model 32

33 Conclusions 1- The cross-sectional procedures perform similarly and with 25 areas, there is no loss in efficiency from benchmarking. 2- Cross-sectional benchmarking does not correct for bias. 3- For sufficiently long time series, the extra efforts of fitting time series models and benchmarking the predictors are well rewarded in terms of the RPMSE. 4- The length of time series required for dominating the crosssectional benchmarked methods depends on variances of sampling errors and correlations between them. Time series predictors can be enhanced by smoothing. 33

34 Two-stage time series benchmarking First stage: benchmark concurrent model-based estimates at higher level of hierarchy to reliable aggregate of corresponding survey estimates. Second stage: benchmark concurrent model-based estimates at lower level of hierarchy to first stage benchmarked estimate of higher level to which they belong. Ghosh and Steorts (2013) consider Bayesian two-stage cross-sectional benchmarking, applied in a single run. 34

35 Why not benchmark second level areas in one step under time series benchmarking? 1- May not be feasible in real time production: For USA-CPS series, proposed procedure requires joint modeling of all the areas state-space model of order Delay in processing data for one second level area could hold up all the area estimates. 3- When 1 st level hierarchy composed of homogeneous 2 nd level areas, benchmarking more effectively tailored to 1 st level characteristics. 35

36 Second-stage time series benchmarking Suppose S states in Division d; Direct Target y Y e ; E( e ) 0, Cov( e, e ) 2 ds, t ds, t ds, t ds, t ds, ds*, t s, s* ds, t Y z ds, t ds, t ds, t ; T, E( ) 0, E( ) Q ds, t ds ds, t1 ds, t ds, t ds, t ds,* t t,* t ds, t S Benchmark: b y,, b z ˆ,,, s1 ds t ds t s1 ds t ds t ds t S Y ˆ bmk Guarantees that model-based estimates at lower level add up to published estimate at higher level to which they belong. bmk Benchmark error: ˆ bmk bmk S r ( Y Y ) ( z ˆ b z ds, t ds, tds, t ). s1 36

37 Benchmarking of second-level estimates (cont.) (,...,, ˆ bmk y y y Y ) d t d1, t ds, t d (, ˆ bmk y Y ) t bmk e ( e,..., e, r ) ( e, r ). d bmk d t d1, t ds, t t (,..., ), d t d1, t ds, t d S T, d ds Zt S z ds, t T, d d Z t Z t. bd1, tz d1, t,..., bds, tz ds, t Combined model: y Z e ; T d d d d d d d d t t t t t t1 t E( ) Q Q ; E( e e ) d d d d d t t S ds, t d t t. 37

38 Empirical results Total unemployment, CPS-USA, Jan Dec2009. First level- Census divisions, Second level- States. Illustrate: 1- Consistency of benchmarked predictors 2- Robustness to model failure 38

39 Consistency of benchmarked predictors S.E.( y ) /y (left) Y ˆ bmk / y (right) ds,t ds,t Massachusetts ds,t ds,t Jan-90 Jan-94 Jan-98 Jan-02 Jan Jan-90 Jan-94 Jan-98 Jan-02 Jan-06 39

40 Consistency of benchmarked predictors [S.E.( y ds,t ) /y ds,t] (left) [ Y ˆ bmk ds,t / y ds,t ] (right) New Hampshire Jan-90 Jan-94 Jan-98 Jan-02 Jan Jan-90 Jan-94 Jan-98 Jan-02 Jan-06 40

41 Direct, Benchmarked and Unbenchmarked estimates of Total Unemployment, Georgia (numbers in 000 s) CPS BMK UNBMK Jan-00 Jan-02 Jan-04 Jan-06 Jan-08 41

42 Direct, Benchmarked and Unbenchmarked estimates of Total Unemployment, Alabama (numbers in 000 s). 170 CPS BMK UNBMK Jan-00 Jan-02 Jan-04 Jan-06 Jan-08 42

43 Conclusions 1. Two-stage time series benchmarking consistent and protects against abrupt changes in time series model. 2. Due to bias correction, two-stage time series benchmarking occasionally produces predictors with lower RPMSE than unbenchmarked predictors (not illustrated). 3. Methodology can be extended to other time series models or to three- or higher-stage benchmarking. 4. Proposed method produces variances of benchmarked predictors as part of the model fitting. 43