The Econometrics of Unobservables: Identification, Estimation, and Empirical Applications

Transcription

1 The Econometrics of Unobservables: Identification, Estimation, and Empirical Applications Yingyao Hu Department of Economics Johns Hopkins University March 25, 2018 Yingyao Hu (JHU) Econometrics of Unobservables / 80

2 Economic theory vs. econometric model: an example economic theory: Permanent income hypothesis econometric model: Measurement error model y = βx + e x = x + v y : observed consumption x : observed income x : latent permanent income v : latent transitory income β : marginal propensity to consume maybe the most famous application of measurement error models Yingyao Hu (JHU) Econometrics of Unobservables / 80

3 A canonical model of income dynamics: an example permanent income: a random walk process transitory income: an ARMA process η t : ɛ t : xt : v t : x t = x t + v t x t = x t 1 + η t v t = ρ t v t 1 + λ t ɛ t 1 + ɛ t permanent income shock in period t transitory income shock latent permanent income latent transitory income Can a sample of {x t } t=1,...,t uniquely determine distributions of latent variables η t, ɛ t, x t, and v t? Yingyao Hu (JHU) Econometrics of Unobservables / 80

4 Road map 1 empirical evidences on measurement error 2 measurement models: observables vs unobservables definition of measurement and general framework 2-measurement model 2.1-measurement model 3-measurement model dynamic measurement model estimation (closed-form, extremum, semiparametric) 3 empirical applications with latent variables auctions with unobserved heterogeneity multiple equilibria in incomplete information games dynamic learning models effort and type in contract models unemployment and labor market participation cognitive and noncognitive skill formation two-sided matching income dynamics 4 conclusion Yingyao Hu (JHU) Econometrics of Unobservables / 80

5 Measurement error: empirical evidences and assumptions Kane, Rouse, and Staiger (1999): Self-reported education x conditional on true education x. (Data source: National Longitudinal Class of 1972 and Transcript data) f x x (x i x j ) x true education level x self-reported education x 1 no college x 2 some college x 3 BA + x 1 no college x 2 some college x 3 BA Finding I: more likely to tell the truth than any other possible values f x x (x x ) > f x x (x i x ) for x i = x. = error equals zero at the mode of f x x ( x ). Finding II: more likely to tell the truth than to lie. f x x (x x ) > 0.5. = invertibility of the matrix [ f x x (x i x j ) ] in the table above. i,j Yingyao Hu (JHU) Econometrics of Unobservables / 80

6 Measurement error: empirical evidences and assumptions Chen, Hong & Tarozzi (2005): ratio of self-reported earnings x vs. true earnings x by quartiles of true earnings. (Data source: 1978 CPS/SS Exact Match File) Finding I: distribution of measurement error depends on x. Finding II: distribution of measurement error has a zero mode. Yingyao Hu (JHU) Econometrics of Unobservables / 80

7 Measurement error: empirical evidences and assumptions Bollinger (1998, page 591): percentiles of self-reported earnings x given true earnings x for males. (Data source: 1978 CPS/SS Exact Match File) Finding I: distribution of measurement error depends on x. Finding II: distribution of measurement error has a zero median. Yingyao Hu (JHU) Econometrics of Unobservables / 80

8 Measurement error: empirical evidences and assumptions Self-reporting errors by gender Yingyao Hu (JHU) Econometrics of Unobservables / 80

9 Graphical illustration of zero-mode measurement error Yingyao Hu (JHU) Econometrics of Unobservables / 80

10 Latent variables in microeconomic models empirical models unobservables observables measurement error true earnings self-reported earnings consumption function permanent income observed income production function productivity output, input wage function ability test scores learning model belief choices, proxy auction model unobserved heterogeneity bids contract model effort, type outcome, state var Yingyao Hu (JHU) Econometrics of Unobservables / 80

11 Our definition of measurement X is defined as a measurement of X if cardinality of support(x ) cardinality of support(x ). there exists an injective function from support(x ) into support(x ). equality holds if there exists a bijective function between two supports. number of possible values of X is not smaller than that of X X discrete {x 1, x 2,..., x L } discrete {x1, x 2,..., x K } L K continuous discrete {x1, x 2,..., x K } continuous continuous X X : measurement error (classical if independent of X ) X Yingyao Hu (JHU) Econometrics of Unobservables / 80

12 A general framework observed & unobserved variables X measurement observables X latent true variable unobservables economic models described by distribution function f X f X f X f X X f X (x) = f X X (x x )f X (x )dx X : latent distribution : observed distribution : relationship between observables & unobservables identification: Does observed distribution f X uniquely determine model of interest f X? Yingyao Hu (JHU) Econometrics of Unobservables / 80

13 Relationship between observables and unobservables discrete X {x 1, x 2,..., x L } and X X = {x 1, x 2,..., x K } matrix expression f X (x) = x X f X X (x x )f X (x ), p X = [f X (x 1 ), f X (x 2 ),..., f X (x L )] T p X = [f X (x 1 ), f X (x 2 ),..., f X (x K )] T M X X = [ f X X (x l x k ) ] l=1,2,...,l;k=1,2,...,k. p X = M X X p X. given M X X, observed distribution f X uniquely determine f X if Rank ( M X X ) = Cardinality (X ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

14 Identification and observational equivalence two possible marginal distributions p a X and p b X are observationally equivalent, i.e., p X = M X X p a X = M X X p b X that is, different unobserved distributions lead to the same observed distribution M X X h = 0 with h := p a X p b X identification of f X requires M X X h = 0 implies h = 0 that is, two observationally equivalent distributions are the same. This condition can be generalized to the continuous case. Yingyao Hu (JHU) Econometrics of Unobservables / 80

15 Identification in the continuous case define a set of bounded and integrable functions containing f X { } L 1 bnd (X ) = h : h(x ) dx < and sup x X h(x ) < X define a linear operator operator equation L X X : L 1 bnd (X ) L 1 bnd (X ) ( LX X h ) (x) = f X X (x x )h(x )dx X f X = L X X f X identification requires injectivity of L X X, i.e., L X X h = 0 implies h = 0 for any h L 1 bnd (X ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

16 A 2-measurement model definition: two measurements X and Z satisfy X Z X two measurements are independent conditional on the latent variable f X,Z (x, z) = matrix expression x X f X X (x x )f Z X (z x )f X (x ) M X,Z = [f X,Z (x l, z j )] l=1,2,...,l;j=1,2,...,j M Z X = [ f Z X (z j xk ) ] j=1,2,...,j;k=1,2,...,k D X = diag {f X (x1 ), f X (x2 ),..., f X (xk )} M X,Z = M X X D X M T Z X suppose that matrices M X X and M Z X have a full rank, then Rank (M X,Z ) = Cardinality (X ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

17 2-measurement model: binary case a binary latent regressor Y = βx + η (X, X ) η X, X {0, 1} measurement error X X is correlated with X in general f (y x) is a mixture of f η (y) and f η (y β) f (y x) = 1 f (y x )f X X (x x) x =0 = f η (y)f X X (0 x) + f η (y β)f X X (1 x) f η (y)p x + f η (y β)(1 P x ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

18 2-measurement model: binary case observed distributions f (y x = 1) and f (y x = 0) are mixtures of f (y x = 1) and f (y x = 0) with different weights P 1 and P 2 f (y x = 1) f (y x = 0) = [f η (y β) f η (y)](p 0 P 1 ) if P 0 P 1 1, then f (y x = 1) f (y x = 0) f (y x = 1) f (y x = 0) leads to partial identification Yingyao Hu (JHU) Econometrics of Unobservables / 80

19 2-measurement model: binary case parameter of interest β = E (y x = 1) E (y x = 0) bounds β E (y x = 1) E (y x = 0) If Pr(x = 0 x = 0) > Pr(x = 0 x = 1), i.e., P 0 P 1 > 0, then sign {β} = sign {E (y x = 1) E (y x = 0)} Yingyao Hu (JHU) Econometrics of Unobservables / 80

20 2-measurement model: binary case measurement error causes attenuation f(y x=0) f(y x=1) f(y x * =0) observed true f(y x * =1) Yingyao Hu (JHU) Econometrics of Unobservables / 80

21 2-measurement model: discrete case a discrete latent regressor y = βx + η (X, X ) η X, X {x 1, x 2,..., x K } Chen Hu & Lewbel (2009): point identification generally holds general models without (X, X ) η : partial identification see Bollinger (1996) and Molinari (2008) Yingyao Hu (JHU) Econometrics of Unobservables / 80

22 2-measurement model: linear model with classical error a simple linear regression model with zero means Y = βx + η X = X + ε X ε η β is generally identified (from observed f Y,X ) except when X is normal (Reiersol 1950) Yingyao Hu (JHU) Econometrics of Unobservables / 80

23 2-measurement model: Kotlarski s identity a useful special case: β = 1 Y = X + η X = X + ε Yingyao Hu (JHU) Econometrics of Unobservables / 80

24 2-measurement model: Kotlarski s identity a useful special case: β = 1 Y = X + η X = X + ε distribution function & characteristic function of X (i = 1) f X (x ) = 1 2π e ix t Φ X (t)dt Φ X = E itx ] [e Yingyao Hu (JHU) Econometrics of Unobservables / 80

25 2-measurement model: Kotlarski s identity a useful special case: β = 1 Y = X + η X = X + ε distribution function & characteristic function of X (i = 1) f X (x ) = 1 2π e ix t Φ X (t)dt Φ X = E itx ] [e Kotlarski s identity (1966) Φ X (t) = exp [ t 0 ie [ Ye isx ] ] ds Ee isx Yingyao Hu (JHU) Econometrics of Unobservables / 80

26 2-measurement model: Kotlarski s identity a useful special case: β = 1 Y = X + η X = X + ε distribution function & characteristic function of X (i = 1) f X (x ) = 1 2π e ix t Φ X (t)dt Φ X = E Kotlarski s identity (1966) [ t ie [ Ye isx ] ] Φ X (t) = exp 0 Ee isx ds itx ] [e latent distribution f X is uniquely determined by observed distribution f Y,X with a closed form Yingyao Hu (JHU) Econometrics of Unobservables / 80

27 2-measurement model: Kotlarski s identity Kotlarski s identity (1966) Φ X (t) = exp [ t 0 ie [ Ye isx ] ] ds Ee isx intuition: Var(X ) = Cov(Y, X ) All the moments of X may be written as a function of joint moments of Y and X with a closed form first introduced to econometrics by Li and Vuong (1998). Li (2002, JoE) first used the result to consistently estimate regression models with classical measurement errors. Yingyao Hu (JHU) Econometrics of Unobservables / 80

28 2-measurement model: nonlinear model with classical error a nonparametric regression model Y = g(x ) + η X = X + ε X ε η Schennach & Hu (2013 JASA): g( ) is generally identified except some parametric cases of g or f X a generalization of Reiersol (1950, ECMA) 2-measurement model needs strong specification assumptions for nonparametric identification: additivity, independence Yingyao Hu (JHU) Econometrics of Unobservables / 80

29 2.1-measurement model 0.1 measurement refers to a 0-1 dochotomous indicator Y of X definition of 2.1-measurement model: two measurements X and Z and a 0-1 indicator Y satisfy for y {0, 1} f X,Y,Z (x, y, z) = X Y Z X x X f X X (x x )f Y X (y x )f Z X (z x )f X (x ) an important message: adding 0.1 measurement in a 2-measurement model is enough for nonparametric identification, i.e., under mild conditions, f X,Y,Z uniquely determines f X,Y,Z,X f X,Y,Z,X = f X X f Y X f Z X f X a global nonparametric point identification (exact identification if J = K = L) Yingyao Hu (JHU) Econometrics of Unobservables / 80

30 Identification: discrete case (Hu, 2008, JE) Let x, x {x 1, x 2, x 3 } and z {z 1, z 2, z 3 }, e.g., education levels. M x x = M x z = D y x = M y;x z = f x x (x 1 x 1 ) f x x (x 1 x 2 ) f x x (x 1 x 3 ) f x x (x 2 x 1 ) f x x (x 2 x 2 ) f x x (x 2 x 3 ) f x x (x 3 x 1 ) f x x (x 3 x 2 ) f x x (x 3 x 3 ) f x z (x 1 z 1 ) f x z (x 1 z 2 ) f x z (x 1 z 3 ) f x z (x 2 z 1 ) f x z (x 2 z 2 ) f x z (x 2 z 3 ) f x z (x 3 z 1 ) f x z (x 3 z 2 ) f x z (x 3 z 3 ) f y x (y x 1 ) f y x (y x 2 ) f y x (y x 3 ) = error structure = IV structure = latent model f y;x z (y, x 1 z 1 ) f y;x z (y, x 1 z 2 ) f y;x z (y, x 1 z 3 ) f y;x z (y, x 2 z 1 ) f y;x z (y, x 2 z 2 ) f y;x z (y, x 2 z 3 ) f y;x z (y, x 3 z 1 ) f y;x z (y, x 3 z 2 ) f y;x z (y, x 3 z 3 ) = observed info. M y;x z contains the same information as f y,x z. Yingyao Hu (JHU) Econometrics of Unobservables / 80

31 Matrix equivalence The main equation for a given y f y,x z (y, x z) = x f x x (x x )f y x (y x )f x z(x z) M y;x z = M x x D y x M x z Yingyao Hu (JHU) Econometrics of Unobservables / 80

32 Matrix equivalence The main equation for a given y f y,x z (y, x z) = x f x x (x x )f y x (y x )f x z(x z) Similarly, M y;x z = M x x D y x M x z f x z (x z) = x f x x (x x )f x z(x z) M x z = M x x M x z Yingyao Hu (JHU) Econometrics of Unobservables / 80

33 Matrix equivalence The main equation for a given y f y,x z (y, x z) = x f x x (x x )f y x (y x )f x z(x z) M y;x z = M x x D y x M x z Similarly, f x z (x z) = x f x x (x x )f x z(x z) M x z = M x x M x z Eliminate M x z M y;x z M 1 = ( ) ) M x z x x D y x M x z (M 1 x z M 1 x x = M x x D y x M 1 x x. Yingyao Hu (JHU) Econometrics of Unobservables / 80

34 An inherent matrix diagonalization An eigenvalue-eigenvector decomposition: M y;x z M 1 = M x z x x D y x M 1 x x f x x (x 1 x 1 ) f x x (x 1 x 2 ) f x x (x 1 x 3 ) = f x x (x 2 x 1 ) f x x (x 2 x 2 ) f x x (x 2 x 3 ) f x x (x 3 x 1 ) f x x (x 3 x 2 ) f x x (x 3 x 3 ) f y x (y x 1 ) f y x (y x 2 ) f y x (y x 3 ) f x x (x 1 x 1 ) f x x (x 1 x 2 ) f x x (x 1 x 3 ) 1 f x x (x 2 x 1 ) f x x (x 2 x 2 ) f x x (x 2 x 3 ) f x x (x 3 x 1 ) f x x (x 3 x 2 ) f x x (x 3 x 3 ) For {x 1, x 2, x 3 }, i.e., an index of eigenvalues and eigenvectors: eigenvalues: f y x (y ) eigenvectors: [ f x x (x 1 ), f x x (x 2 ), f x x (x 3 ) ] T Yingyao Hu (JHU) Econometrics of Unobservables / 80

35 Ambiguity Inside the decomposition Ambiguity in indexing eigenvalues and eigenvectors, i.e., {,, } 1-to-1 {x 1, x 2, x 3 } Decompositions with different indexing are observationally equivalent, M y;x z M 1 = M x z x x D y x M 1 x x f x x (x 1 ) f x x (x 1 ) f x x (x 1 ) = f x x (x 2 ) f x x (x 2 ) f x x (x 2 ) f x x (x 3 ) f x x (x 3 ) f x x (x 3 ) f y x (y ) f y x (y ) f y x (y ) f x x (x 1 ) f x x (x 1 ) f x x (x 1 ) 1 f x x (x 2 ) f x x (x 2 ) f x x (x 2 ) f x x (x 3 ) f x x (x 3 ) f x x (x 3 ) Identification of f x x boils down to identification of symbols,,. Yingyao Hu (JHU) Econometrics of Unobservables / 80

36 Restrictions on eigenvalues and eigenvectors Eigenvalues are distinctive if x is relevant, i.e., f y x (y x i ) = f y x (y x j ) with x i = x j for some y. Symbols,, are identified under zero-mode assumption. For example, error distribution f x x is the same as in Kane et al (1999). no clg. x 1 : some clg. x 2 : BA + x 3 : f x x (x 1 ) f x x (x 2 ) f x x (x 3 ) = zero-mode assumption x 2 = arg max xi f x x (x i ) arg max xi f x x (x i ) = x 2 is the mode truth at the mode }{{} = x 2 (some college) Similarly, we can identify and. = The model f y x and the error structure f x x are identified. Yingyao Hu (JHU) Econometrics of Unobservables / 80

37 Uniqueness of the eigen decomposition uniqueness of the eigenvalue-eigenvector decomposition (Hu 2008 JE) 1. distinctive eigenvalues: a nontrivial set of y, s.t., f (y x 1 ) = f (y x 2 ) for any x 1 = x 2 2. eigenvectors are colums in M X X, i.e., f X X ( x ). A natural normalization is x f X X (x x ) = 1 for all x 3. ordering of the eigenvalues or eigenvectors That is to reveal the value of x for either f X X ( x ) or f (y x ) from one of below a. x is the mode of f X X ( x ): very intuitive, people are more likely to tell the truth; consistent with validation study b. x is a quantile of f X X ( x ): useful in some applications c. x is the mean of f X X ( x ): useful when x is continuous d. E (g(y) x ) is increasing in x for a known g, say Pr(y > 0 x ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

38 2.1-measurement model: geometric illustration Eigen-decomposition in the 2.1-measurement model Eigenvalue: λ i = f Y X (1 xi ) Eigenvector: p i = [ ] p X x = f X X (x 1 x i i ), f X X (x 2 xi ), f X X (x 3 xi T ) Observed distribution in the whole sample: q 1 = [ ] T p X z1 = f X Z (x 1 z 1 ), f X Z (x 2 z 1 ), f X Z (x 3 z 1 ) Observed distribution in the subsample with Y = 1 : y q 1 = [ ] T p y1,x z = f 1 Y,X Z (1, x 1 z 1 ), f Y,X Z (1, x 2 z 1 ), f Y,X Z (1, x 3 z 1 ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

39 Discrete case without ordering conditions: finite mixture conditional independence with general discrete X, Y, Z, and X (Allman, Matias and Rhodes, 2009, Ann Stat) advantages: 1 cardinality of X can be larger than that of X or Z or both 2 a lower bound on the so-called Kruskal rank is sufficient for identification up to permutation. (but ordering is innocuous) disadvantages: 1 Kruskal rank is hard to interpret in economic models, not testable as regular rank 2 not clear how to extend to the continuous case cf. classic local parametric identification condition: Number of restrictions Number of unknowns cf. 2.1 measurement model: 1 reach the lower bound on the Kruskal rank: 2Cardinality (X ) directly extend to the continuous case 3 values of X may have economic meaning Yingyao Hu (JHU) Econometrics of Unobservables / 80

40 2.1-measurement model: continuous case X, Z, and X are continuous f (y, x, z) = f (y x )f (x x )f (x, z)dx share the same idea as the discrete case in Hu (2008) from matrix to integral operator diagonal matrix diagonal operator (multiplication) matrix diagonalization spectral decomposition eigenvector eigenfunction nontrivial extension, highly technical Hu & Schennach (2008, ECMA) Yingyao Hu (JHU) Econometrics of Unobservables / 80

41 From conditional density to integral operator From 2-variable function to an integral operator ( Lx x g ) (x) = f x x ( ) f x x (x x ) g (x ) dx for any g. Operator L x x transforms unobserved f x to observed f x, i.e., f x = L x x f x. ( f x (x ) distribution of x ) ( Lx x = f x x ( ) is called the kernel function of L x x. f x (x) distribution of x ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

42 Identification: from matrix to integral operator From matrix to integral operator L y;x z g = f y,x z (y, z) g (z) dz L x z g = f x z ( z) g (z) dz L x x g = f x x ( x ) g (x ) dx L x zg = f x z ( z) g (z) dz D y;x x g = f y x (y ) g ( ). L y;x z : y viewed as a fixed parameter. D y;x x : diagonal operator (multiplication by a function). Yingyao Hu (JHU) Econometrics of Unobservables / 80

43 Identification: operator equivalence The main equation for a function g, [ ] L y;x z g (x) = = = = = = L y;x z = L x x D y;x x L x z. f y,x z (y, x z) g (z) dz f x x (x x ) f y x (y x ) f x z (x z) dx g (z) dz f x x (x x ) f y x (y x ) f x z (x z) g (z) dzdx [ ] f x x (x x ) f y x (y x ) L x z g (x ) dx [ ] f x x (x x ) D y;x x L x z g (x ) dx [ ] L x x D y;x x L x z g (x). Similarly, L x z = L x x L x z. Yingyao Hu (JHU) Econometrics of Unobservables / 80

44 Identification: a necessary condition on error distribution Intuition: if f x x is known, we want f x to be identifiable from f x. That is, if f x and f x are observationally equivalent as follows: f x (x) = f x x (x x )f x (x ) dx = f x x (x x ) f x (x ) dx, then f x = f x. In other words, let h = f x f x, we want f x x (x x )h (x ) dx = 0 for all x = h = 0. An equivalent condition: Assumption 2(i): L x x is injective. Implications: Inverse L 1 exists on its domain. x x Assumption 2(i) is implied by bounded completeness of f x x, e.g., exponential family. Yingyao Hu (JHU) Econometrics of Unobservables / 80

45 A necessary condition on instrumental variable Intuition: same as before f x z(x z)h (x ) dx = 0 for all z = h = 0 Implications: It is equivalent to the injectivity of L x z. Inverse L 1 exists on its domain. x z Used in Newey & Powell (2003) and Darolles, Florens & Renault (2005). It is a necessary condition to achieve point identification using IV. Implied by the bounded completeness of f x z, e.g., exponential family. Since L x z = L x x L x z and L x x is injective, the injectivity of L x z is implied by: Assumption 2(ii): L x z is injective. Yingyao Hu (JHU) Econometrics of Unobservables / 80

46 An inherent spectral decomposition L 1 x x and L 1 x z exist = an inherent spectral decomposition L y;x z L 1 = ( L x z x x D y;x x L ( ) 1 x z) Lx x L x z = L x x D y;x x L 1 x x. An eigenvalue-eigenfunction decomposition of an observed operator on LHS Eigenvalues: f y x (y x ), kernel of D y;x x. Eigenfunctions: f x x ( x ), kernel of L x x. Yingyao Hu (JHU) Econometrics of Unobservables / 80

47 Identification: uniqueness of the decomposition Assumption 3: sup y Y sup x X f y x (y x ) <. = boundedness of L y;x z L 1, the observed operator on the LHS. x z Theorem XV.4.5 in Dunford & Schwartz (1971): The representation of a bounded linear operator as a weighted sum of projections is unique. Each eigenvalue λ = f y x (y x ) is the weight assigned to the projection onto a linear subspace S (λ) spanned by the corresponding eigenfunction(s) f x x ( x ). However, there are ambiguities inside weighted sum of projections. = We need to freeze these degrees of freedom to show that L x x and D y;x x are uniquely determined by L y;x zl 1 x z. Yingyao Hu (JHU) Econometrics of Unobservables / 80

48 A close look at weighted sum of projections Discrete case: L y;x z L 1 x z = L x x D y;x x L 1 x x = f y x (y x 1 ) L x x + f y x (y x 2 ) L x x + f y x (y x 3 ) L x x L 1 x x L 1 x x L 1 x x Continuous case: L y;x z L 1 x z = σ λp (dλ) Yingyao Hu (JHU) Econometrics of Unobservables / 80

49 Identification: uniqueness of the decomposition Ambiguity I: Eigenfunctions f x x ( x ) are defined only up to a constant: Solution: Constant determined by f x x (x x ) dx = 1. Intuition: Eigenfunctions are conditional densities, therefore, are automatically normalized. Ambiguity II: If λ is a degenerate eigenvalue, more than one possible eigenfunctions. Solution: Assumption 4: for all x 1, x 2 X, the set { y : fy x (y x 1 ) = f y x (y x 2 ) } has positive probability whenever x 1 = x 2. Intuition: eigenvalues f y x (y 1 x ) and f y x (y 2 x ) share the same eigenfunction f x x ( x ). Therefore, y is helpful to distinguish eigenfunctions. Note: this assumption is weaker than (or implied by) the monotonicity assumptions typically made in the nonseparable error literature Yingyao Hu (JHU) Econometrics of Unobservables / 80

50 Identification: uniqueness of the decomposition Ambiguity III: Freedom in indexing eigenvalues: e.g., use x or (x ) 3? Solution: the zero location assumption, i.e., Assumption 5: there exists a known functional M such that x = M [ f x x ( x ) ] for all x. Intuition: Consider another variable x related to x by x = R (x ). = M [ f x x ( x ) ] = M [ f x x ( R ( x )) ] = R ( x ) = x. = Only one possible R: the identity function. Examples of M error has a zero mean: error has a zero mode: M [f ] = xf (x)dx (thus, allow classical error) M [f ] = arg max x f (x) error has a zero τ-th quantile: M [f ] = inf { x : 1 (x x ) f (x)dx τ } Importance: this assumption is based on the findings from validation studies. Yingyao Hu (JHU) Econometrics of Unobservables / 80

51 2.1-measurement model: continuous case key identification conditions: 1) all densities are bounded 2) the operators L X X and L Z X are injective. 3) for all x = x in X, the set { y : f Y X (y x ) = f Y X (y x ) } has positive probability. 4) there exists a known functional M such that M [ f X X ( x ) ] = x for all x X. then f X,Y,Z uniquely determines f X,Y,Z,X with f X,Y,Z,X = f X X f Y X f Z X f X a global nonparametric point identification Yingyao Hu (JHU) Econometrics of Unobservables / 80

52 3-measurement model definition: three measurements X, Y, and Z satisfy X Y Z X can always be reduced to a 2.1-measurement model. all the identification conditions remain with a general Y. doesn t matter which is called dependent variable, measurement, or instrument. examples: Hausman Newey & Ichimura (1991) add x = γz + u, z instrument, g( ) is a polynomial Schennach (2004): use a repeated measurement x 2 = x + ε 2 general g( ), use ch.f. Kotlarski s identity Schennach (2007): use IV: x = γz + u u z general g( ), use ch.f. similar to Kotlarski s identity Yingyao Hu (JHU) Econometrics of Unobservables / 80

53 Hidden Markov model: a 3-measurement model an unobserved Markov process X t+1 {X s } s t 1 X t. a measurement X t of the latent X t satisfying a hidden Markov model X t {X s, X s } s =t X t. X t 1 X t X t+1 Xt 1 Xt X t+1 a 3-measurement model X t 1 X t X t+1 X t, Yingyao Hu (JHU) Econometrics of Unobservables / 80

54 dynamic measurement model {X t, X t } is a first-order Markov process satisfying Flow of chart f Xt,X t X t 1,X t 1 = f X t X t,x t 1 f X t X t 1,X t 1. X t 2 X t 1 X t X t+1 Xt 2 Xt 1 Xt X t+1 Hu & Shum (2012, JE): nonparametric identification of the joint process Special case with Xt = Xt 1 needs 4 periods of data. cf. 6 periods in Kasahara and Shimotsu (2009) Yingyao Hu (JHU) Econometrics of Unobservables / 80

55 dynamic measurement model Hu & Shum (2012): nonparametric identification of the joint process. (use Carroll Chen & Hu (2010, JNPS)) key identification assumptions: 1) for any x t 1 X, M Xt x t 1,X t 2 is invertible. 2) for any x t X, there exists a (x t 1, x t 1, x t ) such that M Xt+1,x t x t 1,X t 2, M Xt+1,x t x t 1,X t 2, M Xt+1,x t x t 1,X t 2, and M Xt+1,x t x t 1,X t 2 are invertible and that for all x t = x t in X xt xt 1 ln f Xt X t,x t 1 (x t ) = xt xt 1 ln f Xt X t,x t 1 ( x t ) 3) for any x t X, E [X t+1 X t = x t, Xt = xt ] is increasing in xt. joint distribution of five periods of data f Xt+1,X t,x t 1,X t 2,X t 3 uniquely determines Markov transition kernel f Xt,Xt X t 1,Xt 1 Yingyao Hu (JHU) Econometrics of Unobservables / 80

56 Other approaches: use a secondary sample {Y, X }, {X } (administrative sample) Hu & Ridder (2012) {Y, X }, {X, X } (validation sample) Chen, Hong & Tamer (2005) among many other papers in econometrics & statistics {Y, X, W }, {Y a, X a, W a } (auxiliary survey sample) Carroll, Chen & Hu (2010) with model of interest f (Y X, W ) = f (Y a X a, W a ) also related to literature on missing data, where X can be considered as missing Yingyao Hu (JHU) Econometrics of Unobservables / 80

57 Estimation: discrete case Estimate the matrices directly L y;x,z = Use sample proportion f y;x z (y, x 1, z 1 ) f y;x z (y, x 1, z 2 ) f y;x z (y, x 1, z 3 ) f y;x z (y, x 2, z 1 ) f y;x z (y, x 2, z 2 ) f y;x z (y, x 2, z 3 ) f y;x z (y, x 3, z 1 ) f y;x z (y, x 3, z 2 ) f y;x z (y, x 3, z 3 ) Use kernel density estimator with continuous covariates Identification is globe, nonparametric, and constructive Mimic identification procedure: a unique mapping from f y,x,z to f y x, f x x, and f x,z Easy to compute without optimization or iteration May have problems with a small sample: estimated prob outside [0,1] Yingyao Hu (JHU) Econometrics of Unobservables / 80

58 Estimation: discrete case Eigen decomposition holds after averaging over Y with a known ω (.) E [ω (Y ) X = x, Z = z] f X,Z (x, z) = f X X (x x )E [ω (Y ) x ] f Z X (z x )f X (x ) x X Define M X,ω,Z = [E [ω (Y ) X = x k, Z = z l ] f X,Z (x k, z l )] k=1,2,...,k;l=1,2,...,k D ω X = diag {E [ω (Y ) x1 ], E [ω (Y ) x2 ],..., E [ω (Y ) xk ]} M X,ω,Z M 1 X,Z = M X X D ω X M 1 X X The matrix M X,ω,Z can be directly estimated as [ ] N 1 M X,ω,Z = N ω (Y i ) 1 (X i = x k, Z i = z l ) i=1 Estimation mimics identification procedure k=1,2,...,k;l=1,2,...,k Yingyao Hu (JHU) Econometrics of Unobservables / 80

59 Estimation: discrete case May also use extremum estimator with restrictions ( MX X, Dω X ) = arg min M,D M ( ) 1 X,ω,Z MX,Z M M D such that 1) each entry in M is in [0, 1] 2) each column sum of M equals 1 3) D is diagonal 4) entries in M satisfies the ordering Assumption See Bonhomme et al. (2015, 2016) for more extremum estimators Yingyao Hu (JHU) Econometrics of Unobservables / 80

60 Closed-form estimators Global nonparametric identification elements of interest can be written as a function of observed distributions continuous case: Kotlarski s identity nonparametric regression with measurement error: Schennach (2004b, 2007), Hu and Sasaki (2015) discrete case: eigen-decomposition in Hu (2008) Closed-form estimator mimic identification procedure don t need optimization or iteration less nuisance parameters than semiparametric estimators but may not be efficient Yingyao Hu (JHU) Econometrics of Unobservables / 80

61 Closed-form estimators a 3-measurement model x 1 = g 1 (x ) + ɛ 1 x 2 = g 2 (x ) + ɛ 2 x 3 = g 3 (x ) + ɛ 3 normalization: g 3 (x ) = x Schennach (2004b): g 2 (x ) = x Hu and Sasaki (2015): g 2 is a polynomial Hu and Schennach (2008): g 1 and g 2 are nonparametrically identified Open question: Do closed-form estimators for g 1 and g 2 exist? Yingyao Hu (JHU) Econometrics of Unobservables / 80

62 Estimation: a sieve semiparametric MLE Based on : f y,x z (y, x z) = f y x (y x )f x x (x x )f x z(x z)dx Approximate -dimensional parameters, e.g., f x x, by truncated series f 1 (x x ) = i n j n i=0 j=0 γ ij p i (x)p j (x ), where p k ( ) are a sequence of known univariate basis functions. Sieve Semiparametric MLE α = ( β, η, f 1, f 2 ) 1 = arg max (β,η,f 1,f 2 ) A n n n i=1 ln f y x (y i x ; β, η)f 1 (x i x )f 2 (x z i )dx β : parameter vector of interest η, f 1, f 2 : -dimensional nuisance parameters A n : space of series approximations Yingyao Hu (JHU) Econometrics of Unobservables / 80

63 Estimation: handling moment conditions Use η to handle moment conditions: For parametric likelihoods: omit η. For moment condition models: need η. Model defined by: E [m (y, x, β) x ] = 0. Method: Define a family of densities f y x (y x, β, η) such that m (y, x, β) f y x (y x, β, η) dx = 0, x, β, η. Use sieve MLE α = ) ( β, η, f 1, f 2 = n 1 arg max (β,η,f 1,f 2 ) A n n ln i=1 f y x (y i x ; β, η)f 1 (x i x )f 2 (x z i )dx. Yingyao Hu (JHU) Econometrics of Unobservables / 80

64 Estimation: consistency and normality Consistency of α Conditions: too technical to show here. Theorem (consistency): Under sufficient conditions, we have α α 0 s = o p (1). Proof: use Theorem 4.1 in Newey and Powell (2003). Asymptotic normality of parameters of interest β. Conditions: even more technical. Theorem (normality): Under sufficient conditions, we have n ( β β 0 ) d N ( 0, J 1 ). Proof: use Theorem 1 in Shen (1997) and Chen and Shen (1998). Yingyao Hu (JHU) Econometrics of Unobservables / 80

65 Empirical applications with latent variables auctions with unknown number of bidders auctions with unobserved heterogeneity auctions with heterogeneous beliefs multiple equilibria in incomplete information games dynamic learning models effort and type in contract models unemployment and labor market participation cognitive and noncognitive skill formation dynamic discrete choice with unobserved state variables two-sided matching income dynamics Yingyao Hu (JHU) Econometrics of Unobservables / 80

66 First-price sealed-bid auctions Bidder i forms her own valuation of the object: x i Bidders values are private and independent Common knowledge: value distribution F, number of bidders N Bidder i chooses bid b i to maximize her expected utility function U i = (x i b i ) Pr(max b j < b i ) j =i Winning probability Pr(max b j < b i ) depends on bidder i s belief about j =i her opponents bidding behavior Perfectly correct beliefs about opponents bidding behavior Nash equilibrium Yingyao Hu (JHU) Econometrics of Unobservables / 80

67 Auctions with unknown number of bidders An Hu & Shum (2010, JE): IPV auction model: N : # of potential bidders A: # of actual bidders b: observed bids bid function b(x i ; N ) = { x i xi r F N (s) N 1 ds F N (x i ) N 1 for x i r 0 for x i < r. conditional independence f (A t, b 1t, b 2t b 1t > r, b 2t > r) = N f (A t A t 2, N ) f (b 1t b 1t > r, N ) f (b 2t b 2t > r, N ) f (N b 1t > r, b 2t > r) Yingyao Hu (JHU) Econometrics of Unobservables / 80

68 Auctions with unobserved heterogeneity s t is an auction-specific state or unobserved heterogeneity b it = s t a i (x i ) 2-measurement model b 1t b 2t s t and ln b 1t = ln s t + ln a 1 ln b 2t = ln s t + ln a 2 in general b 1t b 2t b 3t s t Li Perrigne & Vuong (2000), Krasnokutskaya (2011), Hu McAdams & Shum (2013 JE) Yingyao Hu (JHU) Econometrics of Unobservables / 80

69 Auctions with heterogeneous beliefs An (2016): empirical analysis on Level-k belief in auctions Bidders have different levels of sophistication Heterogenous (possibly incorrect) beliefs about others behavior Beliefs (types) have a hierarchical structure Type Belief about other bidders behavior 1 all other bidders are type-l0 (bid naïvely) 2 all other bidders are type-1.. k all other bidders are type-(k 1) Specification of type-l0 is crucial, assumed by the researchers Help explain overbidding and non-equilibrium behavior Observe joint distribution of a bidder s bids in three auctions, assuming bidder s belief level doesn t change across auctions three bids are independent conditional on belief level Yingyao Hu (JHU) Econometrics of Unobservables / 80

70 Multiple equilibria in incomplete information games Xiao (2014): a static simultaneous move game utility function u i (a i, a i, ɛ i ) = π i (a i, a i ) + ɛ i (a i ) expected payoff of player i from choosing action a i a i π i (a i, a i ) Pr (a i ) + ɛ i (a i ) Π i (a i ) + ɛ i (a i ) Bayesian Nash Equilibrium is defined as a set of choice probabilities Pr (a i ) s.t. ({ }) Pr (a i = k) = Pr Π i (k) + ɛ i (k) > max Π i (j) + ɛ i (j) j =k let e denote the index of equilibria a 1 a 2... a N e Yingyao Hu (JHU) Econometrics of Unobservables / 80

71 Dynamic learning models Hu Kayaba & Shum (2013 GEB): observe choices Y t, rewards R t, proxy Z t for the agent s belief Xt Z t : eye movement a 3-measurement model Y t 1 Y t Y t+1 Xt 1 Xt X Z t 1 Z t Z t+1 Z t Y t Z t 1 X t t+1 learning rule Pr ( X t+1 X t, Y t, R t ) can be identified from Pr (Z t+1, Y t, R t, Z t ) = Pr (Z t+1 Xt+1) Pr (Z t Xt ) Pr (Xt+1, Xt, Y t, R t ). Xt+1 Xt Yingyao Hu (JHU) Econometrics of Unobservables / 80

72 Effort and type in contract models: Xin (2018) Online credit markets for peer-to-peer lending attract dispersed and anonymous borrowers and lenders, and often require no collateral. The problems of asymmetric information are two-fold: (1) Borrowers differ in their inherent risks = Adverse Selection; (2) Additional incentives are necessary to motivate borrowers to exert effort = Moral Hazard. Xin (2018, Job market paper) sets up a dynamic structural model to formalize (1) borrowers repayment decisions, (2) lenders investment strategies, (3) websites pricing schemes, when both hidden information (adverse selection) and hidden actions (moral hazard) are present. identification strategies to recover the dist. of borrowers private types and costs of effort, and utility primitives, and estimate the model using a large dataset from Prosper.com. Yingyao Hu (JHU) Econometrics of Unobservables / 80

73 Effort and type in contract models: Xin (2018) Let the index for two loans be t 1 and t. Key elements in the model: (1) Outcomes of the loan (default, late payment): O t, O t 1 ; (2) Observed characteristics (debt-to-income ratio, credit grade): X t, X t 1 ; (3) Effort choices: e t, e t 1 ; (4) Borrower s type: c. Dynamic structure motivated by the model: Yingyao Hu (JHU) Econometrics of Unobservables / 80

74 Effort and type in contract models: Xin (2018) Step 1: Identify Type Distribution Observables, X t = {Financial Status(Z t ), Credit Grade(K t )}. Three pieces of information, independent conditional on type. f (O t, X t, O t 1, X t 1 ) = c f (c, X t 1, O t 1 ) f (X t X t 1, O t 1, c) }{{}}{{} Init. Char. Transition of States f (O t c, X t ) }{{} Outcome Realized Type distribution f (c X t 1, O t 1 ) is identified for borrowers with multiple loans. (Hu and Shum, 2012) Yingyao Hu (JHU) Econometrics of Unobservables / 80

75 Effort and type in contract models: Xin (2018) Step 2: Identify Effort Choice Probabilities Loan outcomes include borrowers default and late payment performances, O t = {D t, L t }. f (O t c, X t ) = }{{} f (D t e t )f (L t e t )f (e t c, X t ) e t identified (1) Conditional on effort, default and late payment are independent. (2) Effort choice is related to borrower s type. Following Hu (2008), effort choice probabilities and outcome realization process are identified. Yingyao Hu (JHU) Econometrics of Unobservables / 80

76 Unemployment and labor market participation Feng & Hu (2013 AER): Let Xt self-reported labor force status. monthly CPS {X t+1, X t, X t 9 } i local independence assume Pr (X t+1, X t, X t 9 ) = X t+1 and X t denote the true and X t X t 9 Pr (X t+1 X t+1) Pr (X t X t ) Pr (X t 9 X t 9) Pr (X t+1, X t, X t 9). a 3-measurement model Pr (X t+1 X t, X t 9) = Pr (X t+1 X t ) Pr (X t+1, X t, X t 9 ) = Pr (X t+1 Xt ) Pr (X t Xt ) Pr (Xt, X t 9 ), Xt Yingyao Hu (JHU) Econometrics of Unobservables / 80

77 Cognitive and noncognitive skill formation Cunha Heckman & Schennach (2010 ECMA) Xt = ( XC,t, X N,t) cognitive and noncognitive skill I t = (I C,t, I N,t ) parental investments for k {C, N}, skills evolve as X k,t+1 = f k,s (X t, I t, X P, η k,t), where XP = ( XC,P, X N,P) are parental skills latent factors ( {X X } T = C,t t=1, { ) XN,t} T t=1, {I C,t} T t=1, {I N,t} T t=1, X C,P, X N,P measurements of these factors key identification assumption a 3-measurement model X j = g j (X, ε j ) X 1 X 2 X 3 X Yingyao Hu (JHU) Econometrics of Unobservables / 80

78 Dynamic discrete choice with unobserved state variables Hu & Shum (2012 JE) W t = (Y t, M t ) Y t agent s choice in period t M t observed state variable X t unobserved state variable for Markovian dynamic optimization models f Yt M t,xt f Mt,Xt Y t 1,M t 1,Xt 1 f Wt,X t W t 1,X t 1 = f Y t M t,x t f M t,x t Y t 1,M t 1,X t 1 conditional choice probability for the agent s optimal joint law of motion of state variables f Wt+1,W t,w t 1,W t 2 uniquly determines f Wt,X t W t 1,X t 1 Yingyao Hu (JHU) Econometrics of Unobservables / 80

79 Two-sided matching model Agarwal & Diamond (2013): an economy containing n workers with characteristics (X i, ε i ) and n firms described by (Z j, η j ) researchers observe X i and Z j a firm ranks workers by a human capital index as v (X i, ε i ) = h (X i ) + ε i. (1) the workers preference for firm j is described by u (Z j, η j ) = g (Z j ) + η j. (2) the preferences on both sides are public information in the market. Researchers are interested in the preferences, including functions h, g, and distributions of ε i and η j. a pairwise stable equilibrium, where no two agents on opposite sides of the market prefer each other over their matched partners. Yingyao Hu (JHU) Econometrics of Unobservables / 80

80 Two-sided matching model when the numbers of firms and workers are both large, The joint distribution of (X, Z ) from observed pairs then satisfies f (X, Z ) = 1 0 f (X q) f (Z q) dq ( f (X q) = f ε F 1 V (q) h(x )) ( f (Z q) = f η F 1 U (q) g(z )) a 2-measurement model h and g may be identified up to a monotone transformation. intuition: f Z X (z x 1 ) = f Z X (z x 2 ) for all z implies h (x 1 ) = h (x 2 ) in many-to-one matching f (X 1, X 2, Z ) = a 3-measurement model 1 0 f (X 1 q) f (X 2 q) f (Z q) dq Yingyao Hu (JHU) Econometrics of Unobservables / 80

81 Income dynamics Arellano Blundell & Bonhomme (2014): nonlinear aspect of income dynamics pre-tax labor income y it of household i at age t y it = η it + ε it persistent component η it follows a first-order Markov process η it = Q t (η i,t 1, u it ) transitory component ε it is independent over time {y it, η it } is a hidden Markov process with y i,t 1 y it y i,t+1 η it a 3-measurement model Yingyao Hu (JHU) Econometrics of Unobservables / 80

82 A canonical model of income dynamics: a revisit Permanent income: a random walk process Transitory income: an ARMA process η t : ɛ t : xt : v t : x t = x t + v t x t = x t 1 + η t v t = ρ t v t 1 + λ t ɛ t 1 + ɛ t permanent income shock in period t transitory income shock latent permanent income latent transitory income Can a sample of {x t } t=1,...,t uniquely determine distributions of latent variables η t, ɛ t, x t, and v t? Yingyao Hu (JHU) Econometrics of Unobservables / 80

83 A canonical model of income dynamics: a revisit Define Estimate AR coefficient Use Kotlarski s identity x t+1 = x t+1 x t ρ t+1 1 ρ t+2 1 ρ t+1 = cov ( x t+2, x t 1 ) cov ( x t+1, x t 1 ) x t = v t + xt x t+2 ρ t+2 1 x t+1 = v t + λ t+2ɛ t+1 + ɛ t+2 + η t+2 η t+1 ρ t+2 1 Joint distribution of {x t } t=1,...,t 3 uniquely determines distributions of latent variables η t, ɛ t, x t, and v t. (Hu, Moffitt, and Sasaki, 2016) Yingyao Hu (JHU) Econometrics of Unobservables / 80

84 Conclusion The Econometrics of Unobservables a solution to the endogeneity problem integration of microeconomic theory and econometric methodology economic theory motivates our intuitive assumptions global nonparametric point identification and estimation flexible nonparametrics applies to large range of economic models latent variable approach allows researchers to go beyond observables Yingyao Hu (JHU) Econometrics of Unobservables / 80

85 Conclusion See updated lecture notes for details The Econometrics of Unobservables Latent Variable and Measurement Error Models and Their Applications in Empirical Industrial Organization and Labor Economics at Yingyao Hu s webpage Comments are welcome. Thank you for your interest. Yingyao Hu (JHU) Econometrics of Unobservables / 80