Semiparametric Estimation of Markov Decision Processes with Continuous State Space

Transcription

1 Semiarametric Estimation of Markov Decision Processes with Continuous State Sace Sorawoot Srisuma and Oliver Linton London School of Economics and Political Science he Suntory Centre Suntory and oyota International Centres for Economics and Related Discilines London School of Economics and Political Science Discussion aer no.: Houghton Street EM/200/550 London WC2A 2AE August 200 el: We thank Xiaohong Chen, Phili Schmidt-Dengler, and seminar articiants at the 9th EC- Squared Conference on.recent Advances in Structural Microeconometrics. in Rome, and Worksho in Banff Semiarametric and Nonarmetric Methods in Econometrics.in Banff, for helful comments. his research is orted by the ESR, United Kingdom. Deartment of Economics, London School of Economics, Houghton Street, London, WC2A 2AE, United Kingdom address: s.t.srisumalse.ac.uk Deartment of Economics, London School of Economics, Houghton Street, London, WC2A 2AE, United Kingdom. address: o.lintonlse.ac.uk

2 Abstract We roose a general two-ste estimation method for the structural arameters of oular semiarametric Markovian discrete choice models that include a class of Markovian Games and allow for continuous observable state sace. he estimation rocedure is simle as it directly generalizes the comutationally attractive methodology of Pesendorfer and Schmidt-Dengler (2008) that assumed finite observable states. his extension is non-trivial as the value functions, to be estimated nonarametrically in the first stage, are defined recursively in a non-linear functional equation. Utilizing structural assumtions, we show how to consistently estimate the infinite dimensional arameters as the solution to some tye II integral equations, the solving of which is a well-osed roblem. We rovide sufficient set of rimitives to obtain root- consistent estimators for the finite dimensional structural arameters and the distribution theory for the value functions in a time series framework. Keywords: Discrete Markov Decision Models, Kernel Smoothing, Markovian Games, Semi-arametric Estimation, Well-Posed Inverse Problem. he authors. All rights reserved. Short sections of text, not to exceed two aragrahs, may be quoted without exlicit ermission rovided that full credit, including notice, is given to the source.

3 Introduction he inadequacy of static frameworks to model economic henomena led to the develoment of recursive methods in economics. he mathematical theory underlying discrete time modelling is dynamic rogramming develoed by Bellman (957); for a review of its revalence in modern economic theory, see Stokey and Lucas (989). In this aer we study the estimation of structural arameters and their functionals that underlie a class of Markov decision rocesses (MDP) with discrete controls and time in the in nite horizon setting. Such models are oular in alied work, in articular in labor and industrial organization. he econometrics involved can be seen as an extension of the classical discrete choice analysis to a dynamic framework. Discrete choice modelling has a long established history in the structural analysis of behavioral economics. McFadden (974) ioneered the theory and methods of analyzing discrete choice in a static framework. Rust (987), using additive searability and conditional indeendence assumtions, show that a class of dynamic discrete choice models can naturally reserve the familiar structure of discrete choice roblems of the static framework. In articular, Rust roosed the Nested Fixed Point (NFP) algorithm to estimate his arametric model by the maximum likelihood method. However, in ractice, this method can ost a considerable obstacle due to its requirement to reeatedly solve for the xed oint of some nonlinear ma to obtain the value functions. he two-ste aroach of Hotz and Miller (993) avoided the full solution method by relying on the existence of an inversion ma between the normalized value functions and the (conditional) choice robabilities, which signi cantly reduces the comutational burden relative to the NFP algorithm. he two-ste estimator of Hotz and Miller is central to several methodologies that followed, esecially in the recent develoment of the estimation of dynamic games. A class of stationary in nite horizon Markovian games can be de ned to include the MDP of interest as a secial case. Various estimation rocedures have been roosed to estimate the structural arameters of dynamic discrete action games; Pakes, Ostrovsky and Berry (2004), and Aguirregabiria and Mira (2007), considered two-ste method of moments and seudo maximum likelihood estimators resectively, which are included in the general class of asymtotic least square estimators de ned by Pesendorfer and Schmidt-Dengler (2008); Bajari, Benkard and Levin (2007) generalizes the simulation-based estimators of Hotz et al. (994) to the multile agent setting. However, in both single and multile agent settings, the aforementioned work assumed the observed state sace is nite whenever the transition distribution of the observed state variables is not seci ed arametrically. As noted by Aguirregabiria and Mira (2002,2007), we should be able to relax this requirement and allow for uncountable observable state sace. In this aer we roose a simle two-ste semiarametric aroach that falls in the general

4 class of semiarametric estimation discussed in Pakes and Olley (995), and Chen, Linton and van Keilegom (2003). he criterion function will be based on some conditional moment restrictions that requires consistent estimators of the value functions. he additional di culty here is due to the fact that the in nite dimensional arameter is de ned through a linear integral equation of tye II. he study of the statistical roerties of solutions to integral equations falls under the growing research area on inverse roblem in econometrics, see Carrasco, Florens and Renault (2007) for a survey. ye II integral equations are found, amongst others, in the study of additive models, see Mammen, Linton and Nielson (995). We show that our roblem is generally well-osed and utilize the aroach similar to Linton and Mammen (2005) to estimate and rovide the distribution theory for the in nite dimensional arameters of interest. Our estimation strategy can be seen as a generalization of the unifying method of Pesendorfer and Schmidt-Dengler (2008) that allows for continuous comonents in the observable state sace. he novel aroach of Pesendorfer and Schmidt-Dengler relies on the attractive feature of the in nite time stationary model, where they write their ex-ante value function as the solution to a matrix equation. We show that the solving of an analogous linear equation, in an in nite dimensional sace, is also a well-osed roblem for both oulation and emirical versions (at least for large samle size). 2 We note that an indeendent working aer of Bajari, Chernozhukov, Hong and Nekielov (2008) also rooses a sieve estimator for a closely related Markovian games, which allows for continuous observable state sace. herefore our methods are comlementary in lling this ga in the literature. However, our estimation strategy, which is simle and intuitive like its redecessor. We use the local aroach of kernel smoothing, under some easily interretable rimitive conditions, to rovide exlicit ointwise distribution theory of the in nite dimensional arameters that would otherwise be elusive with the series or slines exansion. Since the in nite dimensional arameters in MDP are the value functions, they may be of considerable interest themselves. Another advantage for the local estimator includes the otimality in the minimax sense for local linear estimators, see Fan (993). In addition, we exlicitly work under time series framework and rovide the tye of rimitive conditions required for the validity of the methodology. Since the main idea can be fully illustrated in the single agent setu, for most arts of the aer we consider the single agent setu and leave the discussion of the Markovian game estimation to the latter section. he aer is organized as follows. Section 2 de nes the MDP of interest, A closely related technique is also used in the estimating a dynamic auction game of Jofre-Bonet and Pesendorfer (2003). 2 We only focus on the estimation asect as, taking the aroach of Magnac and hesmar (2002), one can simly write down extensions of nonarametric identi cation results on the er eriod ayo functions of Pesendorfer and Schmidt-Dengler (2003,2008). 2

5 motivates and discusses the estimation strategy and the related linear inverse roblem. Section 3 describes in detail the ractical imlementation of the rocedure to obtain the feasible conditional choice robabilities. In Section 4, rimitive conditions and the consequent asymtotic distribution are rovided, the semiarametric ro led likelihood estimator is illustrated as a secial case. Section 5 discusses the extension to dynamic game setting. Section 6 resents a small scale Monte Carlo exeriment to study the nite samle erformance of our estimator. Section 7 concludes. 2 Markov Decision Processes We de ne our time homogeneous MDP and introduce the main model assumtions and notation used throughout the aer. he sources of the comutational comlexity for estimating MDP are brie y reviewed, there we focus on the reresentation of the value function as a solution to the olicy value equation that can generally be written as an integral equation, in 2.2. We discuss the inverse roblem associated with solving such integral equations in De nitions and Assumtions We consider a decision rocess of a forward looking agent who solves the following in nite horizon intertemoral roblem. he random variables in the model are the control and state variables, denoted by a t and s t resectively. he control variable, a t, belongs to a nite set of alternatives A = f; : : : ; Kg. he state variables, s t, has ort S R L+K. At each eriod t, the agent observes s t and chooses an action a t in order to maximize her discounted exected utility. he resent eriod utility is time searable and is reresented by u (a t ; s t ). he agent s action in eriod t a ects the uncertain future states according to the ( rst order) Markovian transition density (ds t+ js t ; a t ). he next eriod utility is subjected to discounting at the rate 2 (0; ). Formally, for any time t, the agent is reresented by a trile of rimitives (u; ; F ), who is assumed to behave according to an otimal decision rule, f (s )g =t, in solving the following sequential roblem " # X V (s t ) = max E u (a (s ) ; s ) fa (s )g =t s t s.t. a (s ) 2 A for all t: () =t Under some regularity conditions, see Bertsekas and Shreve (978) and Rust (994), Blackwell s heorem and its generalization ensure the following imortant roerties. First, there exists a stationary (time invariant) Markovian otimal olicy function : S! A so that (s t ) = (s t+ ) for any s t = s t+ and any t;, where (s t ) = arg max a2a fu (a; s t) + E [V (s t+ ) js t ; a t = a]g : 3

6 Secondly, the value function, de ned in (), is the unique solution to the Bellman s equation V (s t ) = max a2a fu (a; s t) + E [V (s t+ ) js t ; a t = a]g : (2) We now introduce the following set of modelling assumtions. Assumtion M: (Conditional Indeendence) he transitional density has the following factorization: (dx t+ ; d" t+ jx t ; " t ; a t ) = q (d" t+ jx t+ ) f X 0 jx;a (dx t+ jx t ; a t ), where the rst moment of " t exists and its conditional distribution is absolutely continuous with resect to the Lebesgue measure in R K, we denote its density by q. he conditional indeendence assumtion of Rust (987) is fundamental in the current literature. It is a subject of current research on how to nd a ractical methodology that can relax this assumtion, for examle Arcidiacono and Miller (2008). he continuity assumtion on the distribution of " t ensures we can aly Hotz and Miller s inversion theorem. Assumtion M2: he ort of s t = (x t ; " t ) is X E, where X is a comact subset of R L, in articular, x t = x c t; x d t 2 X C X D, and E = R K. In order to avoid a degenerate model, we assume that the state variables s t = (x t ; " t ) 2 X R K can be searated into two arts, which are observable and unobservable resectively to the econometrician; see Rust (994a) for various interretations of the unobserved heterogeneity. Comactness of X is assumed for simlicity, in articular to X C can be unbounded. Assumtion M3: (Additive Searability) he er eriod ayo function u : A X E! R is additive searable w.r.t. unobservable state variables, u (a t ; x t ; " t ) = (a t ; x t ) + P K k= " a k;t [a t = k]. he combination of M and M3 allows us to set our model in the familiar framework of static discrete choice modelling. We shall introduce the structural arameters 2 R M that arameterize later in Section 3 to kee the notation of the general discussion simle. It is indeed our goal to estimate as well as some functionals deending on them. Conditions M - M3 are crucial to the estimation methodology we roose. hese conditions are standard in the literature. In articular, M2 is weaker than the usual nite X assumtion when no arametric assumtion is assumed on f X 0 jx;a (dx t+ jx t ; a t ) in the in nite horizon framework. For deartures of this framework see the discussion in the survey of Aguirregabiria and Mira (2008) and the references therein. Henceforth Conditions M - M3 will be assumed and later strengthened as aroriate. 4

7 2.2 Value Functions Similarly to the static discrete choice models, the choice robabilities lay a central role in the analysis of the controlled rocess. here are two numerical asects that we need to consider in the evaluation of the choice robabilities. he rst are the multile integrals, that also arise in the static framework, where in ractice many researchers avoid this issue via the use of conditional logit assumtion of McFadden (974). 3 he second is regarding the value function - this is unique to the dynamic setu. o see recisely the roblem we face, we rst udate the Bellman s equation (2) under the assumtions M - M3, V (s t ) = max a2a f (a; x t) + " a;t + E [V (s t+ ) jx t ; a t = a]g : Denoting the future exected ayo E [V (s t+ ) jx t ; a t ] by g (a t ; x t ), and the choice seci c value, net of " a;t, (a t ; x t ) + g (a t ; x t ) by v (a t ; x t ), the otimal olicy function must satisfy (x t ; " t ) = a, v (a; x t ) + " a;t v (a 0 ; x t ) + " a 0 ;t for a 0 6= a: (3) he conditional choice robabilities, fp (ajx)g, are then de ned by P (ajx) = Pr [v (a; x t ) + " a;t v (a 0 ; x t ) + " a 0 ;t Z for a 0 6= ajx t = x] (4) = [ (x; " t ) = a] q (d" t jx) : Even if we knew v, (4) will generally not have a closed form and the task of erforming multile integrals numerically can be non-trivial, see Hajivassiliou and Ruud (994) for an extensive discussion on an alternative aroach to aroximating integrals. For some seci c distributional assumtions on " t, for examle using the oular i.i.d. extreme value of tye I - we can avoid the multile integrals as (4) has the well known multinomial logit form P (ajx) = ex (v (a; x)) X ex (v (a 0 ; x)) : a 0 2A Our estimation strategy accommodates for general form of distribution. However, the roblem we want to focus on is the fact that we generally do not know v, as it deends on g that is de ned through some nonlinear functional equation that we need to solve for. Next, we outline a characterization of the value function that motivates our aroach to estimate g (and v). he main insight to the simlicity of our methodology is motivated from the geometric series reresentation for the value function that is commonly used in dynamic rogramming theory, for 3 Unlike in static models, we do not su er from the undesirable I.I.A. when use i.i.d. extreme values errors of tye I in the dynamic framework. 5

8 an examle see Bertsekas and Shreve (978, Chater 9). More seci cally, one can de ne the value function corresonding to a articular stationary Markovian olicy by " # X V (s t ; ) = E u ( (s ) ; s ) s = s t ; =t which is the solution to the following olicy value equation V (s t ; ) = u ( (s t ) ; s t ) + E [V (s t+ ; ) js t ] : In this aer we only consider values corresonding to the the otimal olicy, to reduce the notation, so we ress the exlicit deendence on the olicy. herefore, by de nition of the otimal olicy, the solution to (2) is also the solution to the following olicy value equation V (s t ) = u ( (s t ) ; s t ) + E [V (s t+ ) js t ] : (5) If the state sace S is nite, then V is a solution of a matrix equation above since the conditional exectation oerator here can be reresented by a stochastic transitional matrix. By the dominant diagonal theorem, the matrix reresenting (I E [js]) s2s is invertible and (5) has a unique solution, solvable by direct matrix inversion or aroximated by a geometric series (see the Neumann series below). he notion of simly inverting a matrix has an obvious aeal over Rust s xed oint iterations. In the in nite dimensional case, the matrix equation generalizes to an integral equation. In the resence of some unobserved state variables, we can also de ne the conditional value function as a solution to the following conditional olicy value equation, taking conditional exectation on (5) w.r.t. x t yields E [V (s t ) jx t ] = E [u ( (s t ) ; s t ) jx t ] + E [E [V (s t+ ) js t ] jx t ] = E [u ( (s t ) ; s t ) jx t ] + E [E [V (s t+ ) jx t+ ] jx t ] ; where the last equality follows from the law of iterated exectations and M. Noting that, again by M, g (a t ; x t ) can be written as E [m (x t+ ) jx t ; a t ], where m (x t ) = E [V (s t ) jx t ], then we have m as a solution to some articular integral equation of tye II; more succinctly, m satis es m = r + Lm; (6) where r is the ex-ante exected immediate ayo given state x t, namely E [u ( (s t ) ; s t ) jx t = ]; and the integral oerator L generates discounted exected next eriod values of its oerands, e.g. Lm (x) = E [m (x t+ ) jx t = x] for any x 2 X. If we could solve (6) then we need another level of smoothing on m to obtain the choice seci c value v. In articular, we can de ne g through the following linear transform g = Hm; (7) 6

9 where H is an integral oerator that generates the choice seci c exected next eriod values of its oerands oerator, e.g. Hm (x; a) = E [m (x t+ ) jx t = x; a t = a] for any (x; a) 2 X A. herefore we can write the choice seci c value net of unobserved states in a linear functional notation as v = + Hm: (8) In Section 3 we discuss in details on how to use the olicy value aroach to estimate the model imlied transformed of the value functions and choice robabilities. 2.3 Linear Inverse Problems Before we consider the estimation of v, we need to address some issues regarding the solution of integral equations (6). It is natural to ask the fundamental question whether our roblem is wellosed, more seci cally, whether the solution of such equation exist and if so, whether it is unique and stable. he study of the solution to such integral equations falls in the general framework of linear inverse roblems. he study of inverse roblems is an old roblem in alied mathematics. he tye of inverse roblems one commonly encounters in econometrics are integral equations. Carrasco et al. (2007) focused their discussion on ill-osed roblems of integral equations of tye I where recent works often needed regularizations in Hilbert Saces to stabilize their solutions. Here we face an integral equation of tye II, which is easier to handle, and in addition, the convenient structure of the olicy value equations allows us to easily show that the roblem is well-osed in a familiar Banach Sace. We now de ne the normed linear sace and the oerator of interest, and roof this claim. We shall simly state relevant results from the theory of integral equations. For de nitions, roofs and further details on integral equations, readers are referred to Kress (999) and the references therein. From the Riesz heory of oerator equations of the second kind with comact oerators on a normed sace, say A : X! X, we know that I A is injective if and only if it is surjective, and if it is bijective, then the inverse oerator (I A) : X! X is bounded. For the moment, ose that X D is emty, we will be working on the Banach sace (B; kk), where B = C (X) is a sace of continuous functions de ned on the comact subset of R L, equied with the -norm, i.e. kk = x2x j (x)j. L is a linear ma, L : C (X)! C (X), such that, for any 2 C (X) and x 2 X, Z L (x) = (x 0 ) f X 0 jx (dx 0 jx) ; X where f XjX (dx t+ jx t ) denotes the conditional density of x t+ given x t. In this case since we know the existence, uniqueness and stability of the solution to (6) are assured for any r = 2 C (X) as we can show L is a contraction. o see this, take any 2 C (X) and 7

10 x 2 X, Z jl (x)j since the discounting factor 2 (0; ), X j (x 0 )j f X 0 jx (dx 0 jx) j (x)j ; x2x klk kk ) klk < : his imlies that our inverse is well-osed. Further, the contraction roerty means we can reresent the solution to (6) using the Neumann series: m = (I L) r (9) X = lim! L r: herefore the in nite series reresentation of the inverse suggests one obvious way of aroximating the solution to the integral equation which will converge geometrically fast to the true function. If X is countable, then L would be reresented by a -ste ahead transition matrix (scaled by ). Note that the oerator for the (uncountable) in nite dimensional case share the analogous interretation of -ste ahead transition oerator with discounting. Since our roblem is well-osed, then it is reasonable to exect that with su ciently good estimates of (r; L; H), our estimated integral equation is also well-osed and will lead to (uniform) consistent estimators for (m; g; v). Our strategy is to use nonarametric methods to generate the emirical versions of (6) and (7), then use them to rovide an aroximate for v necessary for comuting the choice robabilities. = 3 Estimation Given a time series fa t ; x t g t= generated from the controlled rocess of an economic agent reresented by (u 0 ; ; ), for some 0 2, where u re ects the arameterization of by. In this section we rovide in details the rocedure to estimate 0 as well as their corresonding conditional value functions. We based our estimation on the conditional choice robabilities. We de ne the model imlied choice robabilities from a family of value functions, fv g 2, induced by underlying otimal olicy that generates the data. In articular, for each, V satis es (cf. equation (5)) V (s t ) = u ( (s t ) ; s t ) + E [V (s t+ ) js t ] : he olicy value V has the interretation of a discounted exected value for an economic agent whose ayo function is indexed by but behaves otimally as if her structural arameter is 0. By 8

11 de nition of the otimal olicy, V coincides with the solution of a Bellman s equation in (2) when = 0. We then de ne the following (otimal) olicy-induced equations to analogous to (6), (7) and (8), resectively for each : m = r + Lm ; (0) g = Hm ; () v = + Hm ; (2) where r is the ex-ante exected ayo given state x t, namely E [u ( (s t ) ; s t ) jx t = ]; and the integral oerators L and H are the same as in Section 2.2. he functions m ; g and v are de ned to satisfy the linear equation and transforms resectively. Naturally, for each (a; x) 2 A X, P (ajx) is then de ned to satisfy P (ajx) = Pr [v (a; x t ) + " a;t v (a 0 ; x t ) + " a 0 ;t for a 0 6= ajx t = x] ; which is analogous to (4). Our methodology roceeds in two stes. In the rst ste, we nonarametrically comute estimates of the kernels of L; H and for each, estimate r, which are then used to estimate m by solving the emirical version of the integral equation (0) and estimate g analogously from an emirical version of (). he second ste is the otimization stage, the model imlied choice seci c value functions are used to comute the choice robabilities that can be used to construct various objective functions to estimate the structural arameter Estimation of r ; L and H here are several decisions to be made to solve the emirical integral equation in (0). We need to rst decide on the nonarametric method. We will focus on the method of kernel smoothing due to its simlicity of use as well as its well established theoretical grounding. Our nonarametric estimation of the conditional exectations will be based on the Nadaraya-Watson estimator. However, since we will be working on bounded sets, it is necessary to address the boundary e ects. he treatment of the boundary issues is straightforward, the recise trimming condition is described in Section 4. So we will assume to work on a smaller sace X X where X = X C; XD denotes a set where the ort of the uncountable comonent is some strict comact subset of X C but increases to X C in. When allowing for discrete comonents we simly use the frequency aroach, smoothing over the discrete comonents is also ossible, see the monograh by Li and Racine (2006) for a recent udate on this literature. We will also need to make a decision on how to de ne and interolate the solution to the emirical version of (0) in ractice. We discuss two asymtotically equivalent 9

12 otions for this latter choice, whether the size of the emirical integral equation does or does not deend on the samle size, as one may have a reference given the relative size of the number of observations. We now de ne the nonarametric estimators, (br ; L; b H), b of (r ; L; H). Any generic density of a mixed continuous-discrete random vector w t = wt; c wt d, fw : R lc R ld! R + for some ositive integers l C and l D, is estimated as follows, bf w w c ; w d = X K h (wt c w c ) wt d = w d ; t= where K is some user chosen symmetric robability density function, h is a ositive bandwidth and for simlicity indeendent of w c. K h () = K (=h) =h and if l C > then K h (w c t w c ) = l C Q l= K hl w c t;l w c l, [] denotes the indicator function, namely [A] = if event A occurs and takes value zero otherwise. Similar to the roduct kernel, the contribution from a multivariate discrete variable is reresented by roducts of indicator functions. he conditional densities/robabilities are estimated using the ratio of the joint and marginal densities. he local constant estimator of any generic regression function, E [z t jw t = w] is de ned by, Estimation of r For any x 2 X ; he rst term can be estimated by P t= be [z t jw t = w] = z tk h (wt c w c ) wt d = w d : (3) bf w (w) r (x) = E [u (a t ; x t ; " t ) jx t = x] = E [ (a t ; x t ) jx t = x] + E [" at jx t = x] = ; (x) + 2 (x) : b ; (x) = X a2a bp (ajx) (a; x) ; (4) or, alternatively, the Nadaraya-Watson estimator, e ; (x) = E b [ (a t ; x t ) jx t = x] : In (4), fp b (ajx)g a2a is a kernel estimator of the choice robabilities. We also comment that it might be more convenient to use b ; over e ;, as we shall see, since the nonarametric estimates for the choice robabilities are required to estimate 2. 0

13 he conditional mean of the unobserved states, 2, is generally non-zero due to selectivity. By Hotz and Miller s inversion theorem, we know 2 can be exressed as a known smooth function of the choice robabilities. An estimator of 2 can therefore be obtained by lugging in the local constant (linear) estimator of the choice robabilities. For examle, the i.i.d. tye I extreme value errors assumtion will imly that 2 (x) = + X P (ajx) log (P (ajx)) ; (5) a2a where is the Euler s constant. Our rocedure is not restricted to the conditional logit assumtion. Although other distributional assumtion will generally not rovide a closed form exression for 2 in fp (ajx)g, it can be comuted for any (a; x) 2 A X, for examle see Pesendorfer and Schmidt- Dengler (2003) who assume the unobserved states are i.i.d. standard normals. Note also that 2 is indeendent of as the distribution of " t is assumed to be known; in rinciles; our rocedure can be written to easily accommodate the case when the conditional distribution of " t is known u some nite dimensional arameters. Estimation of L and H For the ease of notation let s ose X D is emty. For the integral oerators L and H, if we would like to use the numerical integration to aroximate the integral, we only need to rovide the nonarametric estimators of their kernels, resectively, f b X 0 jx (dx t+ jx t ) and f b X 0 jx;a (dx t+ jx t ; a t ). For any 2 C (X ), the emirical oerators are de ned as, Z bl (x) = (x 0 ) f b X 0 jx (dx 0 jx) ; (6) X Z bh (x; a) = (x 0 ) f b X 0 jx;a (dx 0 jx; a) : (7) X So L b and H b are linear oerators on the Banach sace of continuous functions on X with range C (X ) and C (X A) resectively under -norm. Alternatively, we could use the Nadaraya- Watson estimator, de ned in (3), to estimate the oerators, el (x) = E b [ (x t+ ) jx t = x] ; eh (x; a) = E b [ (x t+ ) jx t = x; a t = a] : his aroach may be more convenient when samle size is relatively small, and we want to solve the emirical version of (0) by using urely nonarametric methods for interolation, where we could use the local linear estimator to address the boundary e ects. Note that, if X is nite then the integrals in (6) and (7) will be de ned with resect to discrete measures, then ( L; b H) b and ( L; e H) e can be equivalently reresented by the same stochastic matrices.

14 3.2 Estimation of m ; g and v We rst describe the rocedure used in Linton and Mammen (2005), by using ( b L; b H), to solve the emirical integral equation. We de ne bm as any sequence of random functions de ned on X that aroximately solves bm = br + b L bm. Formally, we shall assume that bm is any random sequence of functions that satisfy I 2;x2X L b bm (x) br (x) = o =2 ; (8) i.e., the right hand side of (8) is aroximately zero. We allow this extra generality like Pakes and Pollard (989) and Linton and Mammen (2005). In ractice, we solve the integral equation on a nite grid of oints, which reduces it to a large linear system. Next we use bm to de ne bg, seci cally we de ne bg as any random sequence of functions that satisfy bg (a; x) b H bm (a; x) = o =2 : (9) 2;a2A;x2X Once we obtain bg, the estimator of v is de ned by 2;a2A;x2X jbv (a; x) (a; x) bg (a; x)j = o =2 : (20) For illustrational uroses, ignoring the trimming factors, we will assume that X = [x; x] R. For any integrable function on X, de ne J () = R (t) dt. Given an ordered sequence of n nodes ft j;n g [a; b], and a corresonding sequence of weights f! j;n g such that P n j=! j;n = b a, a valid integration rule would satisfy lim n () n! = J () nx J n () =! j;n (t j;n ) ; for examle Simson s rule and Gaussian quadrature both satisfy this roerty for smooth. herefore the emirical version of (0) can be aroximated for any x 2 [a; b] by bm (x) = br (x) + j= nx! j;nfx b 0 jx (t j;n jx) bm (t j;n ) : (2) j= So the desired solution that aroximately solves the emirical integral equation will satisfy the following equation at each node ft j;n g, bm (t i;n ) = br (t i;n ) + nx! j;nfx b 0 jx (t j;n jt i;n ) bm (t j;n ) : j= 2

15 his is equivalent to solving a system of n equations with n variables, the linear system above can be written in a matrix notation as bm = br + L b bm ; (22) where bm = ( bm (t ;n ) ; : : : ; bm (t n;n )) > ;br = (br (t ;n ) ; : : : ; br (t n;n )) > ; I n is an identity matrix of order n and L b is a square n matrix such that ( L) b ij =! j;nfx b 0 jx (t j;n jt i;n ). Since f b X 0 jx (jx) is a roer density for any x, with a su ciently large n, (I n L) b is invertible by the dominant diagonal theorem. So there is a unique solution to the system (22) for a given br. In ractice we have a variety of ways to solve for bm with one obvious candidate being the successive aroximation as mentioned in (??). Once we obtain bm, we can aroximate bm (x) for any x 2 X by substituting bm into the RHS of (2). his is known as the Nyström interolation. We need to aroximate another integral to estimate g. his could be done using the conventional method of kernel regression as discussed in Section 3., or by aroriately selecting sequences of r nodes fq j;r g and weights j;n so that bg (j; x) = rx b j;nfx 0 jx;a (q j;r jx; j) bm (q j;r ) ; j= where the comutation for this last linear transform is trivial. See Judd (998) for a more extensive review of the methods and issues of aroximating integrals and also the discussion of iterative aroaches in Linton and Mammen (2003) for large grid sizes. Alternatively, we can form a matrix equation of size, em = er + e L em ; to estimate equation (0) at the observed oints with the t-th element. For each t, let em (x t ) = er (x t ) + P t= em (x t+ ) K h (x t x) P = K : h (x x t ) By the dominant diagonal theorem, the matrix equation above always has a unique solution for any 2. Once solved, the estimators of em can be interolated by em (x) = er (x) + b E [m (x t+ ) jx t = x] ; for any x 2 X. Similarly, eg and ev can be estimated nonarametrically without introducing any additional numerical error. Clearly, the more observation we have, the latter method will be more di cult as dimension of the matrix reresenting e L is large whilst the grid oints for the former emirical equation is user-chosen. 3

16 3.3 Estimation of By construction, when = 0, the model imlied conditional choice robability P coincides with the underlying choice robabilities de ned in (4). herefore one natural estimator for the nite dimensional structural arameters can be obtained by maximizing a likelihood criterion. De ne Q () = X log P (a t jx t ) ; t= b Q () = X c t; log P b (a t jx t ) : (23) t= Here fc t; g is a triangular array of trimming factors, more discussion on this can be found in Section 4. In ractice, we relace P (ajx) by bp (ajx) = Pr [bv (a; x t ) + " a;t bv (a 0 ; x t ) + " a 0 ;t for a 0 6= ajx t = x] ; where bv satis es condition (20). Of articular interest is the secial case of the conditional logit framework, as discussed in Section 2, where we have bp (ajx) = ex (bv (a; x)) X ex (bv (a 0 ; x)) : a 0 2A herefore b Q denotes the feasible objective function, which is identical to Q when the in nite dimensional comonent bv is relaced by v. We de ne our maximum likelihood estimator, b, to be any sequence that satisfy the following in equality bq ( b ) 2 bq () o =2 : (24) Alternatively, a class of criterion functions can be generated from the following conditional moment restrictions E [ [a t = a] P (ajx t ) jx t ] = 0 for all a 2 A when = 0 : Note that these moment conditions are the in nite dimensional counterarts (with resect to the observable states) of equation (8) in Pesendorfer and Schmidt-Dengler (2008) for a single agent roblem. here are general large samle theory of ro led semiarametric estimators available that treat the estimators de ned in our models. 4 In articular, the work of Pakes and Olley (995) and Chen, Linton and van Keilegom (2003) rovide high level conditions for obtaining root consistent estimators are directly alicable. he latter is a generalization of the work by Pakes and Pollard (989), who rovided the asymtotic theory when the criterion function is allowed to be non-smooth, which 4 We can generally write the objective functions from the likelihood criterion (via rst order conditions) and the 4

17 may arise if we use simulation methods to comute the multile integral of (4), to the semiarametric framework. In Section 4, as an illustration, we derive the asymtotic distribution of the semiarametric likelihood estimator under a set of weak conditions in the conditional logit framework. 3.4 Practical Discussion We re ect on the comutational e ort required of the roosed method. We only discuss the estimation of the conditional value functions. It will be helful to have in mind the methodology of Pesendorfer and Schmidt-Dengler (2008) as our methods coincide when the X is nite and there is only layer in the game (vice versa, extending from a single agent decision rocess to a dynamic game). For each, the nonarametric estimates of (r ; L; H) have closed form and are very easy to comute even with large dimensions, further, the emirical integral oerators (or their aroximations) only need to be comuted once at the required nodes since they do not deend on. Solving the emirical integral equation to obtain bm, in (22), is the only otential comlication that does not exist in a static roblem. However, in this setu, this reduces to the need to invert a large matrix that aroximates (I L) that only need to be done once at the beginning and stored for future comutation with any other. Estimators of (m ; g ; v ) are obtained trivially for any, by simle matrix multilication, once the emirical oerator of (I L) is obtained. We note that further comutational gain is ossible if is linear in. More seci cally, if = > 0 for some known functions 0 then r = > r 0 + 2, where r 0 () = P a2a P (aj) 0 (a; ). Utilizing the fact that the inverse of (I L) is a linear oerator, we have m = > (I L) r 0 + (I L) 2, where the estimates of (I L) r 0 and (I L) 2 only need to be comuted once. See Hotz, Miller, Sanders and Smith (994) and Bajari, Benkard and Levin (2007) for related utilization of the reeated substitution concet. However, it is imortant to note that, as we have decided on the kernel smoothing aroach there is an issue of bandwidth selection which is imortant for small samle roerties. Further, it is easy moment restrictions in the following way: M () = X q (a t ; x t ; ; v ) ; t= c M () = X c t; q (a t ; x t ; ; bv ) ; t= where c M is the feasible counterart of M. De ne the limiting objective function M () = lim! EM () ; which is assumed to exist and is uniquely minimized at = 0. We then de ne our estimator to be any sequence that satisfy the following inequality, M c ( b ) inf M c () + o =2 : 2 5

18 to see that the invertibility of the matrix (I L) b and (I L) e are not deendent on the number of continuous and/or discrete comonents. Clearly, there are a lot of choices available regarding integral aroximation and matrix inversion methods. It is beyond the scoe of this aer to analyze the nite samle erformance of these various methodologies. 4 Distribution heory In this section we rovide a set of rimitive conditions and derive the distribution theory for the estimators b, as de ned in (24), and ( bm ; bg ) as de ned in (8) and (9) resectively when the unobserved state variables is distributed as i.i.d. extreme value of tye I. his distributional assumtion is the most commonly used in ractice as it yields closed-form exressions for the choice robabilities. We also restrict the dimensionality of X C to be a subset of R, the reason being this is the scenario that alied researchers may refer to work with. hese seci cs do not limit the usefulness of the rimitives rovided. For other estimation criteria, since two-ste estimation roblems of this tye can be comartmentalized into nonarametric rst stage and otimization in the second stage, the rimitives below will be directly alicable. In articular, the discussions and results in 4. are indeendent of the choice of the objective functions chosen in the second stage. here might be other intrinsically continuous observable state variables that require discretizing but with increasing dimension in X C, the ractitioners will need to emloy higher order kernels and/or undersmooth in order to obtain the arametric rate of convergence for the nite structural arameters, adatation of the rimitives are straightforward and will be discussed accordingly. 4. In nite Dimensional Parameters he relevant large samle roerties for the nonarametric rst stage, under the time series framework, for the ointwise results see the results of Roussas (967,969), Rosenblatt (970,97) and Robinson (983). Roussas rst rovided central limit results for kernel estimates of Markov sequences, Rosenblatt established the asymtotic indeendence and Robinson generalized such results to the -mixing case. he uniform rates have been obtained for the class of olynomial estimators by Masry (996), in articular, our method is closely related to the recent framework of Linton and Mammen (2005) who obtained the uniform rates and ointwise distribution theory for the solution of a linear integral equation of tye II. We begin with some rimitives. In addition to M - M3, they are not necessary and only su cient but they are weak enough to accommodate most of the existing emirical works in alied labor and industrial organization involving estimation of MDP. 6

19 We denote the strong mixing coe cient as (k) = t2n A2F t+k ;F t jpr (A \ B) Pr (A) Pr (B)j for k 2 Z; where Fa b denotes the sigma-algebra generated by fa t ; x t g b t=a. Our regularity conditions are listed below: B X is a comact subset of R J R L with X C = [x; x]. B2 he rocess fa t ; x t g t= is strictly stationary and strongly mixing, with a mixing coe cient (k),such that for some C 0 and some, ossibly large > 0; (k) Ck. B3 he density of x t is absolutely continuous f X C ;X D dx t; x d t for each x d t 2 X D. he joint density of (a t ; x t ) is bounded away from zero on X C and is twice continuously di erentiable over X C for each x d t ; a t 2 X D A. he joint density of (x t+ ; x t ; a t ) is twice continuously di erentiable over X C X C for each x d t+; x d t ; a t 2 X D X D A. B4 he mean of the er eriod ayo function u (a t ; x t ) is twice continuously di erentiable on X C for each x d t ; a t 2 X D A. B5 he kernel function is a symmetric robability density function with bounded ort such that for some constant C; jk (u) K (v)j C ju vj. De ne j (K) = R u j K (u) du and j (K) = R K j (u) du. B6 he bandwidth sequence h satis es h = 0 ( ) =5 and 0 ( ) bounded away from zero and in nity. B7 he triangular array of trimming factors fc t; g is de ned such that c t; = x c t 2 X C where X = [x + c ; x c ] and fc g is any ositive sequence converging monotonically to zero such that h < c. B8 he distribution of " t is known to be distributed as i.i.d. extreme value of tye I across K alternatives, and is mean indeendent of x t and is i.i.d. across t. he comactness of the arameter sace in B is standard. Comactness of the continuous comonent of the observable state sace can be relaxed by using an increasing sequence of comact sets that cover the whole real line, see Linton and Mammen (2005) for the modelling in the tails of the distribution. he dimension of X C is assumed to be for exositional simlicity, discussion on this is follows the theorems below. On the other hand, it is a trivial matter to add arbitrary ( nite) number of discrete comonents to X D. 7

20 Condition B2 is quite weak desite the value of can be large. he assumtions of B3, B4 and B5 are standard in the kernel smoothing literature using second order kernel. Here in B6 we use the bandwidth with the otimal MSE rate for a regular -dimensional nonarametric estimates. he trimming factor in B7 rovides the necessary treatment of the boundary e ects. his would ensure all the uniform convergence results on the exanding comact subset fx g whose limit is X. In ractice we will want to minimize the trimming out of the data, we can choose c to h to do this. close enough Condition B8 is not necessary for consistency and asymtotic normality for any of the arameters below. he only requirement on the distribution of " t, for our methodology to work, is that it allows us to emloy Hotz and Miller s inversion theorem. A su cient condition for that is the distribution of " t is known and satisfy M. In articular, B8 yields us the simle multinomial logit form that is often used in ractice. For other distribution will result in the use of a more comlicated inversion ma, for examle see Pesendorfer and Schmidt-Dengler (2003) for the Gaussian case. Next we rovide ointwise distribution theory for the nonarametric estimators obtained from the rst stage, as described in Section 3, for any given set of values of the structural arameters. he bias and the variance terms are comlicated, the exlicit formulae can be found along with all roofs in the Aendix. heorem. Suose B m; and! m; such that for each x 2 int (X) ; h bm (x) m (x) 2 2h 2 m; (x) where bm (x) is de ned as in (8) and: B8 hold. hen for each 2, there exists deterministic functions =) N (0;! m; (x)) ; m; (x) = (I L) r; + L; (x) ; 2! m; (x) = f X (x) 2 var (m (x t+ ) jx t = x) +! r; (x) : Some comonents of the bias and variance are comlicated, in articular the exlicit form of r;, L; and! r; can be found below in (39),(48) and (40) resectively. he estimators bm (x) and bm (x 0 ) are also asymtotically indeendent for any x 6= x 0. Furthermore, j bm (x) m (x)j = o =4 : (x;)2x he ointwise rate of convergence, 2=5, is the usual otimal rate (in the MSE sense) of a dimensional nonarametric function. he above is obtained by using analogous arguments of 8

21 Linton and Mammen (2005) after showing that the conditional density estimator that de ne the emirical integral oerator converges uniformly (see Masry (996)) over its domain. heorem, we also obtain the following results for the estimator of g. heorem 2. Suose B B8 hold. hen for each 2 ; x 2 int (X) and a 2 A; h bg (a; x) g (a; x) 2 2h 2 g; (a; x) =) N (0;! g; (a; x)) ; where bg (a; x) is de ned as in (9) and: g; (a; x) = H (I L) r; + L; (a; x) + H; (a; x) ; 2! g; (a; x) = f X;A (x; a) var (m (x t+ ) jx t = x; a t = a) : Similar to he exlicit form of r;, L; and H; can be found in (39),(48) and (49) resectively. bg (a; x) and bg (a 0 ; x 0 ) are also asymtotically indeendent for any x 6= x 0 and any a. Furthermore, jbg (a; x) g (a; x)j = o =4 : (x;a;)2x A We end with a brief discussion of the change in rimitives required to accommodate the case when the dimension of X C is higher than. Clearly, using the otimal (MSE) rates for h, dim X C cannot exceed 3 with second order kernel if we were to have the uniform rate of convergence for our nonarametric estimates to be faster than =4 that is necessary for consistency of the nite dimensional arameters. It is ossible to overcome this by exloiting additional smoothness (if available) of our densities. his can be done by using higher order kernels to control the order of the bias, for details of their constructions and usages see Robinson (988) and also Powell, Stock and Stoker (989). 4.2 Finite Dimensional Parameters In order to obtain consistency result and the arametric rate of convergence for b, we need to adjust some assumtions described in the revious subsection and add an identi cation assumtion. Consider: B6 0 he bandwidth sequence h satis es h 4! 0 and h2! B9 he value 0 2 int () is de ned by, for any " > 0 Q ( 0 ) Q () > 0; k 0 k" where Q () denotes the limiting objective function of Q (de ned in (23)), namely Q () = lim! EQ (). 9

22 he rate of undersmoothing (relative to B6) in Condition B6 0 ensures that the bias from the nonarametric estimation disaears su ciently quickly to obtain arametric rate of convergence for b. o accommodate for higher dimension of X C, we generally cannot just roceed by undersmoothing but combining this with the use higher order kernels, again, see Robinson (988) and also Powell, Stock and Stoker (989). Condition B9 assumes the identi cation of the arametric art. his is a high level assumtion that might not be easy to verify due to the comlication with the value function. In ractice we will have to check for local maxima for robustness. We note that this is the only assumtion concerning the criterion function, for other tye of objective functions, obvious analogous identi cation conditions will be required. he roerties of b can be obtained by alication of the asymtotic theory for semiarametric ro le estimators. his requires uniform exansion bg (and hence bm ) and their derivatives with resect to. heorem 3. Suose B B5; B6 0 and B7 B9 hold. hen t= b 0 =) N 0; J IJ ; where I is a comlicated term reresenting the asymtotic variance of the leading terms in X q(a t;x t; 0 ;bg 0 ) (see Aendix A) and 2 q (a t ; x t ; 0 ; g 0 ) J = E : > he root- rate of convergence is common for such semiarametric estimators when the dimension of the continuous comonent of X is not too large under some smoothness assumtions. We next resent the results for the feasible estimators of m and g: heorem 4. Suose B B5; B6 0 and B7 B9 hold. hen for any arbitrary estimator e such that jj e 0 jj = O =2 and x 2 int (X), h bm e (x) m 0 (x) =) N (0;! m;0 (x)) ; where bm ; m; and! m; are de ned as those in heorem and, bm e (x) and bm e (x 0 ) are asymtotically indeendent for any x 6= x 0. Similarly, for g we have the following result. heorem 5. Suose B B5; B6 0 and B7 B9 hold. hen for any arbitrary estimator e such that jj e 0 jj = O =2 ; x 2 int (X) and a 2 A; h bg e (a; x) g 0 (a; x) =) N (0;! g;0 (a; x)) ; 20

23 where bg ; g; and! g; are de ned as those in heorem 2 and, bg e (a; x) and bg e (a 0 ; x 0 ) are asymtotically indeendent for any x 6= x 0 and any a. Given the exlicit forms of the bias and variance terms rovided in the above theorems, inference can be conducted using large samle aroximation based on obvious lug-in estimators. However, due to their comlicated form, bootstra rocedures would most likely be referred in ractice. Nevertheless, the exlicit exressions we derive in the Aendix are still useful as they rovide insights into to the variation in the bias and variance of our estimators. 5 Markovian Games he develoment of emirical dynamic games is of recent interest esecially in the industrial organization literature. See Ackerberg et al. (2005) for an excellent survey. In this section we brie y summarize how we can use the methodology discussed in revious sections to estimate a class of Markovian games. Similar to Bajari et al. (2008), we consider the same class of dynamic games described in Aguirregabiria and Mira (2007), Bajari et al. (2007), Pakes et al. (2004) and Pesendorfer and Schmidt-Dengler (2008), and allowing the observable state variable to have continuous comonent. We refer the reader to our working aer version, Srisuma and Linton (2009), for a detailed discussion. First, note that Pesendorfer and Schmidt-Dengler s (2008) results on the characterization (Proosition ) and the existence (heorem ) of the Markov erfect equilibrium can be readily extended to this more general framework. 5 o avoid reetition, we roceed directly to the olicy value equation for each layer i, induced by the equilibrium best resonses, f i g N i=, which generate the observed data. For any, we have V i; (s it ) = E [u i; ( i (s it ) ; a it ; s it ) js it ] + E [V i; (s it+ ) js it ] ; where a it denotes the usual notation of the actions of all other layers excet layer i. Following the arguments in Section 3, where x t now reresents observed ublic information (to all the layers and econometricians), the olicy value equation can be used to obtain its conditional counterart E [V i; (s it ) jx t ] = E [u i; ( i (s it ) ; a it ; s it ) jx t ] + E [E [V i; (s it+ ) jx t+ ] jx t ] ; 5 his follows since: (Proosition ) the arguments in the roof of their Proosition is done ointwise on the ort of the observable state sace; (heorem ) heir equation () (on age 909) becomes a continuous functional equation in an in nite dimensional sace. We can aeal to xed oint theorems in in nite dimensional saces under some weak smoothness conditions on the rimitve functions (such as Schauder or ikhonov xed oint theorems, see Granas and Dugundji (2003)). 2