NUCLEAR NORM PENALIZED ESTIMATION OF INTERACTIVE FIXED EFFECT MODELS. Incomplete and Work in Progress. 1. Introduction

Transcription

1 NUCLEAR NORM PENALIZED ESTIMATION OF IERACTIVE FIXED EFFECT MODELS HYUNGSIK ROGER MOON AND MARTIN WEIDNER Incomplete and Work in Progress. Introduction Interactive fixed effects panel regression models have been widely studied in recent panel literature. The interactive fixed effect model parsimoniously represents heterogeneity in both dimensions of the panel and includes the conventional additive error component model as a special case. One of widely used estimation methods of the interactive fixed effects panel regression is the fixed effect approach (or principal component approach in a linear model) that treats the interactive fixed effects in a factor form as parameters to estimate. For example, Bai (2009), Moon and Weidner (205a), and Moon and Weidner (205b)) investigated asymptotic properties of the fixed effect least squares estimator for linear panel regression models with interactive fixed effects, and Fernández-Val and Weidner (206) studied the fixed effect maximum likelihood estimator for nonlinear panel regressions with interactive fixed effects. The main advantage of the fixed effect approach is that it does not restrict the relationship between the unobserved heterogeneity and the observed explanatory variables. On the other hand, the computation of the fixed effects estimator requires solving a non-convex optimization problem with respect to high dimensional fixed effects parameters. Also, establishing consistency of the fixed effect estimator as an M-estimator is quite challenging due to the presence of the incidental parameters in both directions of panel. In this paper, we investigate a nuclear (trace) norm penalized estimator of interactive fixed effects models that is motivated by the two difficulties of the fixed effects estimator of interactive fixed effects panel regressions. This paper is incomplete and very preliminary. We provide asymptotic analysis of the nuclear norm penalized estimator with a baseline model where the relationship between the dependent variable and the regressors are linear, the panel is balanced, and the rank of the interactive fixed effects is finite and known. Date: January 28, 206. Other estimation methods in the interactive fixed effects literature include the quasi-difference approach in Holtz-Eakin, Newey, and Rosen (988) and the common correlated random effect method (e.g., Pesaran (2006)).

2 2 HYUNGSIK ROGER MOON AND MARTIN WEIDNER Extensions are in progress now. In the extensions, we consider more general models that allow heterogeneous regression coefficients, nonlinearity, and unbalanced panel. We also work on the estimators whose penalty threshold is small, which is closely related with unknown dimension of the interactive fixed effects. Potential empirical applications we consider include estimation of the students education performance model with teacher and student specific effects (generalized value added models), or modeling college admission decision as binary choice (with college and student specific effect). 2. General Model Let m(w it, z it ) be the objective function for i =,..., N and t =,..., T, where w it is observed, and z it = g it (x it ) + λ if t is a scalar single index that depends on the observed covariates x it and the unknown f it (.) and λ R N R and f R T R. The function m(.,.) is known. We define m it (z) = m(w it, z), which is a random function of the single index. We require that m it (z) is convex in z, and we assume that the true parameters solve the following population FOC for all i, t E [ z m it (zit) ] 0 zit 0 = 0, z 0 it = git(x 0 it ) + λ 0 i ft 0. The set of functions g it (x it ) can either be linearly parameterized in terms of parameters β such as a linear single index. Examples for m it (z) include (i) (Weighted) Least Squares Estimation with Interactive Effects: m it (z) = s it (y it z) 2, w it = (y it, s it ). The weights s it could be population weights, or could be binary variables that indicate missing data. (ii) MLE: such that log p(y it z) is concave in z. m it (z) = log p(y it z), (iii) (Smoothed) Quantile Regression. Without smoothing: where ρ τ (u) = u (τ (u < 0)). Examples for g it (x it ) are m it (z) = ρ τ (y it z), (i) Homogeneous Coefficients: g it (x it ) = g(x it ) = β x it. (ii) Heterogeneous Coefficient: g it (x it ) = (α i + γ t ) x it, where β it = α i + γ t and β = [β it ] it.

3 NUCLEAR NORM PENALIZED ESTIMATION OF IERACTIVE FIXED EFFECT MODELS 3 The fixed effects estimator studied in the existing literature is an M-estimator that solves the following minimize problem: ( β, λ, f ) Q (β, λf ) = argmin Q (β, λf ), β B,λ R N R,f R T R N i= T m it ((β X ) it + λ if t ). t= There are two difficulties with the M-estimator ( β, λ, f ). One is a computational issue and the other one is a theoretical issue. (i) Computational Issue: Calculating β, λ, f is complicated by the fact that the objective function Q (β, λ f ) is non-convex in the parameters and can have multiple local minima. For fixed β to optimization over λ and f only becomes easy for non-weighted balanced least-squares (principal components), but for weighted least squares (including missing data), MLE and quantile regression becomes challenging. In any case, after profiling out λ and f the resulting profile objective is in general still non-convex in β, so that high-dimensionality of β also causes potentially serious computational problems. (ii) Theoretical Issue: Consistency of β, λ, f is only shown in some cases, but for many of the above examples the consistency problem is unresolved. For the penalized estimator that we consider, we can show consistency for a much larger class of models. The consistency then carries over to the improved estimator that we consider in the second step that improved estimator may often be (asymptotically) equivalent to β, λ, f, but we do not show this, and it may not always be the case. 3. Penalization Estimation In the general model in Section 2, notice that only the N T matrix Γ = λf enters into the objective function Q (β, Γ). When the function m it (z) is convex in z and z is linear in β and Γ, the objective function Q (β, Γ) is now a convex function of the parameters (β, Γ). Assumption (Convexity of Q (β, Γ)). We assume that the objective function Q (β, Γ) is convex in β and Γ. Even thought the objective function Q (β, Γ) is convex in all parameters, the optimization problem with the rank restriction, min min Q (β, Γ) s.t. rank(γ) = (or ) R (3.) β Γ is non-convex because the constraint rank(γ) = (or ) R is non-convex. This implies that the objective function Q (β, λf ) is non-convex (in (β, λ, f).

4 4 HYUNGSIK ROGER MOON AND MARTIN WEIDNER Notice that the rank-constraint in (3.) can be considered as an L 0 penalty on the singular values of Γ. When replacing the L 0 penalty with the convex L penalty, then we obtain the convex objective function Q ψ (β, Γ) = Q (β, Γ) + 2 ψ s r (Γ), where ψ = ψ > 0 is a penalty parameter that needs to be chosen and s r (A) denotes the r th largest singular value of Γ. The penalty function Γ := s r (Γ) is a matrix norm called called trace norm, nuclear norm, Schatten -norm, or Ky Fan n- norm in literature. Notice that the nuclear norm can equivalently be defined as Γ = Tr [ (ΓΓ ) /2], or Γ = sup B Tr(BΓ). It satisfies A + B A + B (norm) and AB A B (submultiplicative). 2 By definition, we have c Γ + ( c )Γ 2 c Γ + ( c )Γ 2 = c Γ + ( c ) Γ 2. This implies that the penalty function Γ := s r (Γ) is convex in Γ. The estimator we consider in this paper is the following nuclear norm penalized estimator: ( β ψ, Γ ψ ) = argmin β B, Γ R N T Q ψ (β, Γ). 4. Asymptotics of the Nuclear Norm Panelized Estimator in a Baseline Model 4.. Baseline Model. Before we discuss the general model, we consider the following baseline model: Y it = β X it + λ if t + e it, which we write in N T matrix notation as Y = β X +λf +e. The baseline model assumes the following: (i) The panel is balanced, that is, Y it and X it are observed for all pairs i, t. (ii) The model is linear, given by (4.) above. (iii) The penalization parameter ] ψ = ψ is chosen such that as N, T, the probability of rank [ Γ(b, ψ ) = R 0 goes to one, for all b in an appropriate shrinking neighborhood of β. This means that asymptotically we estimate the correct number of factors. In practice, this essentially means that R 0 should be known. 2 The last inequality can be strengthened to AB A B.

5 NUCLEAR NORM PENALIZED ESTIMATION OF IERACTIVE FIXED EFFECT MODELS 5 Defining Γ := λf the model can equivalently be written as Y = β X + Γ + e, rank(γ) R 0. (4.) Here, β and Γ are the true unknown parameters, and we usually use b and G for generic parameter values. R 0 denotes the true number of factors. Notation: Matrix norms:. 2 spectral norm,. F Frobenius norm. s r (.) for r th largest singular value of a matrix. A for Moore-Penrose pseudo-inverse of A. Least Squares (or Fixed Effects) Estimator. For given R {0,, 2,...} the least squares estimator is given by [ β LS (R), Γ LS (R)] := argmin {b R K, G R N T : rank(g) R} Alternatively, we can obtain β LS (R) as β LS (R) = argmin b R K Q (b, R) := Q (b, R), min {G R N T : rank(g) R} Y b X G 2 F. Y b X G 2 F. Here, Q (b, R) is the profile least squares objective function. We also define Γ LS (b, R) := argmin {G R N T : rank(g) R} Y b X G 2 F. Nuclear Norm Penalized Estimator. For given ψ > 0 we define [ β(ψ), Γ(ψ) ] := argmin b R K, G R N T Y b X G 2 F + 2 ψ G. The additional factor 2 in the penalty term turns out to be convenient below. Alternatively, we can obtain β(ψ) as β(ψ) = argmin b R K S (b, ψ) := min G R N T S (b, ψ), ( Y b X G 2 F + 2 ψ G ). Here, S (b, ψ) is the profile penalized objective function. We also define. Γ(b, ψ) := argmin G R N T Y b X G 2 F + 2 ψ G 4.2. Asymptotic Results for the Baseline Case Least Squares Estimator. Here, we briefly summarize some known results on the least squares profile objective Q (b, R), and on the corresponding estimators β LS (R) and Γ LS (R).

6 6 HYUNGSIK ROGER MOON AND MARTIN WEIDNER Those results will be used afterwards to derive the asymptotic properties of the penalized estimator. For the profile least squares objective it is well-known that Q (b, R) := r=r+ [s r (Y β X)] 2. This representation of the profile objective in terms of singular values, or eigenvalues, 3 is useful both for the numerical evaluation of Q (b, R), and for the derivation of the asymptotic properties of the least squares estimator. Assumption 2. As N, T we have (i) s R0 (Γ/ ) P c > 0. (ii) e 2 = O P ( max(n, T ) ). (iii) X k 2 = O P ( ), for all k =,..., K. For k, l {,..., K} we define W,kl := Tr(M λx k M f X l), C (),k := Tr(M λ X k M f e ), C (2),k := [ Tr ( em f e M λ X k Γ ) + Tr ( e M λ e M f X k (Γ ) ) Let W be the K K matrix with elements W,kl, and let C () + Tr ( e M λ X k M f e (Γ ) ) ]. and C(2) be the K-vectors with elements C (),k M λ = M Γ and M f = M Γ. We prefer to write M λ and M f, because we find it slightly more and C(2),k, respectively. Remember that Γ = λf. We therefore have transparent. Notice also that Γ = f(f f) (λ λ) λ, and (Γ ) = (Γ ). The results in the following are known from (Moon and Weidner 205b). 4 Lemma 3. Let Assumption 2 hold. Consider N, T with N/T a > 0. Then, ( ) Q (b, R 0 ) = Q (β, R 0 ) 2 (b β) C () + C(2) + (b β) W (b β) + Q (rem) (b), 3 s r (Y β X) 2 is the r th largest eigenvalue of (Y β X)(Y β X). 4 The result for Γ LS (b, R 0 ) is not explicit in (Moon and Weidner 205b), but can easily derived, because Γ LS (b, R 0 ) = Γ (b β) X + e ê R0 (b), and results for ê R0 (b) are given.

7 and NUCLEAR NORM PENALIZED ESTIMATION OF IERACTIVE FIXED EFFECT MODELS 7 Γ LS (b, R 0 ) = Γ K (b k β k ) (X k M λ X k M f ) + e M λ e M f + Γ (rem) (b), k= where the remainder terms Q (rem) (b) and Γ (rem) (b) satisfy for any sequence c 0, Q (rem) (b) ( ) sup ( {b: b β c } + ) 2 = o P, b β Γ (rem) (b) sup ( 2 {b: b β c } + ) 2 = O P (). N b β Notice that Assumption 2(i) is slightly weaker than the strong factor assumption in (Moon and Weidner 205b), but sufficient to get all the results. We can give a more precise expansion of Γ LS (b, R 0 ), but it is not required for the following. If β LS (R 0 ) P β and W P W > 0 and C () = O P (), then using the expansion of Q (b, R 0 ) in Lemma 3 one can show that ( ) [ β LS (R 0 ) β] P C () + C(2). Using this one can derive the asymptotic distribution of β LS (R 0 ). W Nuclear Norm Penalized Estimator. For ψ > 0 we define g ψ : R + 0 R + 0 : s min [ψ 2, s 2 ] + 2ψ max [0, s ψ]. Thus, for s [0, ψ] we have g ψ (s) = s 2, and for s > ψ we have g ψ (s) = 2ψs ψ 2. Notice that g ψ (s) is continuous and convex. We then have S (b, ψ) := g ψ [s r (Y β X)]. (4.2) The derivation of equation (4.2) is given in the appendix. Define R (b, ψ) = I{s r (Y b X) > ψ}, the number of singular values of Y β X that is larger than ψ. Notice that R (b, ψ) = rank[ Γ(b, ψ)]. Comparing those formulas for S (b, ψ) and Q (b, R), we find that for small singular values, namely for s r (Y β X) ψ and r > R, respectively, the contribution of the singular value s r (Y β X) to both profile objective functions is given by the square of the singular value.

8 8 HYUNGSIK ROGER MOON AND MARTIN WEIDNER We therefore have 5 S (b, ψ) = Q (b, R (b, ψ)) + 2 ψ = Q (b, R (b, ψ)) + 2 ψ R (b,ψ) s r (Y β X) R (b, ψ) ψ 2 R (b, ψ) ψ 2. tr Γ LS (b, R (b, ψ)) In the last step we used that the non-zero singular values of Γ LS (b, R) are equal to s r (Y β X), r =,..., R, which is also well-known from principal components analysis. Asymptotic expansions of Q (b, R 0 ) and Γ LS (b, R 0 ) are given in Lemma 3. Using the expansion of Γ LS (b, R 0 ) one can show that under the assumptions of Lemma 3, and also assuming that P λ ep f 2 = O P (), we have, uniformly in any N-shrinking neighborhood of β, that Γ LS (b, R 0 ) = Γ (b β) D + O P (), where D is the K-vector with components D,k := Tr [ (λ λ) /2 λ X k f(f f) /2]. Notice that Tr [ (λ λ) /2 λ X k f(f f) /2] can also be written as Tr [X k B], where B := f(f f) /2 (λ λ) /2 λ satisfies B 2 and Γ = Tr(ΓB). This explains why Γ + = Γ + Tr( B) + O( 2 2). See also (Watson 992). From these, it is possible to deduce the following theorem: Theorem 4. Let Assumption 2 hold. Furthermore, assume that P λ ep f 2 = O P () and Tr(eX ) = O P (N /2 ). Consider N, T with N/T a > 0. Consider a sequence ψ > 0 such that R (β, ψ ) = R 0, wpa. Then, S (b, ψ ) = S (β, ψ ) 2 ψ (b β) D + (b β) W (b β) + S (rem) (b), where the remainder term S (rem) (b) satisfies for any sequence c 0, S (rem) (b) sup +ψ {b: b β c and R (b,ψ )=R 0 } log N + N+ψ b β + b β = o 2 P (). From the expansion in Theorem 4, we want to conclude that β(ψ ) β = ψ ( ) W D ψ + o P. (4.3) 5 One can also show that the nonzero singular values of Γ(b, ψ) are equal to s r (Y β X) ψ, for r =,..., R (b, ψ). We therefore have S (b, ψ) = Q (b, R (b, ψ)) + 2 ψ Γ(b, ψ) / + R (b, ψ) ψ 2 /.

9 NUCLEAR NORM PENALIZED ESTIMATION OF IERACTIVE FIXED EFFECT MODELS 9 However, the bound on the remainder term of the expansion is only applicable if R (b, ψ ) = R 0. Therefore, we can only conclude (4.3) if R (b, ψ ) = R 0 holds within a sufficiently large neighborhood of b = β + ψ W D. This can be achieved by choosing ψ appropriately, which will now be discussed. The strong factor Assumption 2(i) guarantees that for ψ = o P ( ) we have, in any shrinking neighborhood of β, that s R0 (Y b X) > ψ, implying that R (b, ψ ) R 0. We furthermore have s R0 + (Y b X) = s [ê LS (b, R 0 ) ] = s [M λ (Y b X)M f ] + lower order terms e + M λ [(b β) X]M f + lower order terms. The last bound is crude and needs to be improved if we want to consider ψ = O P ( N). However, if ψ / N, then e in the last expression can be neglected for the question whether s R0 + (Y b X) < ψ. Thus, if ψ = o P ( ) and ψ / N and M λ [(b β) X]M f /ψ < ɛ, wpa, for some ɛ > 0, then R (b, ψ ) = R 0, wpa. We thus obtain the following. Theorem 5. Let Assumption 2 hold. Furthermore, assume that P λ ep f 2 = O P () and Tr(eX ) = O P (N /2 ). Consider N, T with N/T a > 0. Consider a sequence ψ > 0 such that ψ = o P ( ) and ψ / N. Assume furthermore that W P W > 0 and that there exists ɛ > 0 such that K ( ) W M λ X k M f D < ɛ, wpa. (4.4) k Then we have k= ψ ] [ β(ψ ) β P W D Bias Corrected Estimattor. For the baseline case (linear model, balanced panel, known R 0 ) a good way to calculate an improved estimator is to start from β (0) = β(ψ ) and then iterate the following: () Given β (j), calculate λ (j) and f (j) as the R 0 principal components of Y β (j) X. (2) Given λ (j) and f (j) calculate the K K matrix Ŵ (j) kl = Tr(M λ(j)x k M f (j)x l ) and the K-vector Â(j) k = Tr(M λ(j)x k M f (j)y ), and update the estimator for β as β (j+) = (Ŵ (j)) Â(j).

10 0 HYUNGSIK ROGER MOON AND MARTIN WEIDNER Starting with ψ = a N log N, for some constant a, ander the above assumptions, as N, T grow at the same rate, this will give an estimator that is asymptotically equivalent to after very few iterations. Alternatively, we could bias correct β(ψ ) directly, e.g. function S BC (b, ψ ) = S (b, ψ ) + 2 ψ (b β) D, β LS by modifying the objective where the estimator D uses the first stage estimator for λ and f. While this method seems overly complicated in the baseline case, it might be a very convenient method for non-linear models or unbalanced panels. 5. Extension to General Model Extensions under progress include models that allow heterogeneous regression coefficients, some nonlinear regression model, unbalanced panel. We also work on the estimators whose penalty threshold is small, which is closely related with unknown dimension of the interactive fixed effects.

11 NUCLEAR NORM PENALIZED ESTIMATION OF IERACTIVE FIXED EFFECT MODELS Appendix A. Derivation of Equation (4.2) S (b, ψ) = min Q (β, Γ) Γ R N T = min Γ R N T = min { [sr (Y β X Γ)] 2 + 2ψs r (Γ) } min min { γ R γ 0} { U R N U U=I} { V R T V V =I} = min { γ R γ 0} = min { γ R γ 0} = = min { α R α 0} {[s r (Y β X Udiag(γ)V )] 2 + 2ψγ r } { [sr ( )] } Y β X UY β X diag(γ)v Y 2 β X + 2ψγr { [sr (Y β X) γ r ] 2 + 2ψγ r } { [sr (Y β X) α] 2 + 2ψα } { min [ ψ 2, s r (Y β X) 2] + 2ψ max [0, s r (Y β X) ψ] }. Here, in the first step we rewrote the sum of squared residuals as the sum of squared singular values. In the second step we introduced the singular value decomposition Γ = Udiag(γ)V, thus replacing the minimization over Γ by a minimization over the singular vector matrices U and V and the singular value vector γ. In the third step we use that for any given γ the minimizing U and V are equal to the corresponding singular vector matrices of Y β X, denoted by U Y β X and V Y β X (This is the only step in the derivation that is not straightforward, we might want to provide a lemma in the appendix for this. Notice also that the minimizing singular vectors might not be unique, in particular they are not unique for γ r = 0.). In the fourth step we use that the singular values of Y β X U Y β X diag(γ)v Y β X are equal to s r (Y β X) γ r (This assumes s r (Y β X) γ r 0, which is always satisfied for the optimal γ.). The fifth step interchanges the sum and the minimization. The final step solves the minimization problem over α (the optimal α is α = max[0, s r (Y β X) ψ]). References Bai, J. (2009): Panel data models with interactive fixed effects, Econometrica, 77(4),

12 2 HYUNGSIK ROGER MOON AND MARTIN WEIDNER Fernández-Val, I., and M. Weidner (206): Individual and time effects in nonlinear panel data models with large N, T, forthcoming in Journal of Econometrics. Holtz-Eakin, D., W. Newey, and H. S. Rosen (988): Estimating Vector Autoregressions with Panel Data, Econometrica, 56(6), Moon, H., and M. Weidner (205a): Dynamic Linear Panel Regression Models with Interactive Fixed Effects, Forthcoming in Econometric Theory. Moon, H. R., and M. Weidner (205b): Linear Regression for Panel With Unknown Number of Factors as Interactive Fixed Effects, Econometrica, 83(4), Pesaran, M. H. (2006): Estimation and Inference in Large Heterogeneous Panels with a Multifactor Error Structure, Econometrica, 74(4), Watson, G. A. (992): Characterization of the subdifferential of some matrix norms, Linear Algebra and its Applications, 70, Department of Economics and USC Dornsife INET, University of Southern California, Los Angeles, CA 90089, U.S.A. Department of Economics, University College London, Gower Street, London WCE 6BT, U.K., and CeMMaP.