Evaluating enterprise supprt: state f the art and future challenges Dirk Czarnitzki KU Leuven, Belgium, and ZEW Mannheim, Germany
Intrductin During the last decade, mircecnmetric ecnmetric cunterfactual impact evaluatins have becme an imprtant tl in the area f enterprise supprt plicies. It became standard t use ecnmetric methds, such as Matching estimatrs (Cnditinal) Difference-in-Difference regressins Instrumental variable regressins Mre recently: Regressin Discntinuity Designs Nt standard (yet): randmized cntrl trials natural experiments
Intrductin Why are these methds imprtant? Firms select themselves int the prgrams Gverments pick winners Result: treated firms cannt be cmpared with nn-treated firms withut further adjustment fr deriving effects f plicies Treatment is an endgenus variable in (OLS) regressin mdels, and results wuld be biased if endgeneity is nt addressed prperly
Intrductin Therefre all ecnmetric evaluatin methds seek t establish a crrect cntrl grup apprach t derive e.g. the treatment effect n the treated, i.e. Hw many jbs wuld a treated firm have created if it had nt been treated? Hw much wuld have a firm invested in innvatin activities if it had nt been subsidized? Which sales with new prducts wuld a firm have achieved if it had nt gtten a start-up grant?
What is the current evidence? Recent general review* f the available empirical evidence by the What Wrks Centre fr Lcal Ecnmic Grwth Led by the Lndn Schl f Ecnmics and Plitical Science www.whatwrksgrwth.rg 1,700 wrks (academic jurnals and plicy reprts) reviewed Classied accrding t the strength f the empirical evidence Using a variant f the Scientic Maryland Scale Results: nly view achieve highest methdlgical standards Hwever, limited data availability critically determines the researchers ptins f applicatins. * nt limited t enterprise supprt within Chesin Plicy; review cvers all fields f ecnmic evaluatins.
Mdified Scientific Maryland Scale 5 4 3 2 1 Randmized cntrl trials, natural experiments, n selective sample attritin Instrumental variable techniques r RDD, prper balancing (OLS, matching), attritin discussed but nt addressed Difference-in-Differences, balancing (OLS, matching), but uncntrlled differences likely remain Befre and after cmparisns, r a cmparisn grup but withut balancing f cvariates Crrelatin analysis, n cntrl grup, n attempt at establishing a cunterfactual
Methds are imprtant, but it is equally imprtant HOW we use the methds! What des this mean? Example: Evaluatin f Eureka s Eurstars prgramme* (ttal budget 500 millin EUR - c-financed by EC = 100 millin EUR) Dirk estimated: treatment effect f Eurstars with respect t jb creatin amunts t a 3.1% higher average annual emplyment grwth-rate when cmpared t the cunterfactual situatin f n Eurstars grant Extraplatin frm regressin sample t ttal prgramme impact yields abut 7,800 jbs created * http://ec.eurpa.eu/research/sme-techweb/pdf/ejp_final_reprt_2014.pdf
What is useful infrmatin? Is this infrmatin useful fr the plicy maker? Partially yes prgramme has an estimated psitive impact HOWEVER, prgramme might cntinue t exist anyways. Mre useful infrmatin wuld be: Hw can we make the prgramme better, i.e. mre effective and/r mre efficient? Search fr hetergeneus treatment effects See e.g. Czarnitzki/Lpes-Bent (2013)
Hetergeneus treatment effects within a plicy scheme
Hetergeneus treatment effects Mnetary value f grant Subsidy rate Firm size Hetergeneity f cnsrtia start-ups, large firms, universities Prpsal quality (Peer-review scre) Multiple grants Hünermund/Czarnitzki (2016) find that treatment effect varies with the peer-review scre. Better prpsals als yield higher treatment effects (but effect is nt linear) Nte: LATE in RDD vs. ATT btained with ther estimatrs.
-2 0 2 4 6 8 Hetergeneus treatment effects in Eurstars accrding t peer-review scre (prpsal quality) 400 450 500 Scre
Hetergeneus treatments and their effects acrss plicy instruments
Hetergeneus treatments Instead f explring hetergeneus treatment effects within a plicy scheme, it is als pssible t search fr hetergenus effects acrss schemes Prblem: very data hungry E.g. Czarnitzki et al. (2016): enterprise supprt in German Chesin Plicy schemes Als: dynamic treatment effects Treatment effect culd evlve ver time rather than ccuring in a single perid
-.2 -.2 -.2 -.2 0 0 0 0.2.4.6.2.4.6.2.4.6.2.4.6 -.2 -.2 -.2 -.2 0 0 0 0.2.4.6.2.4.6.2.4.6.2.4.6 Treatment effect ver time fr ln(emplyment) by grant type General Cnsulting Innvatin/R&D Training -3-2 -1 0 1 2 3 4 5-3 -2-1 0 1 2 3 4 5-3 -2-1 0 1 2 3 4 5-3 -2-1 0 1 2 3 4 5 Netwrking Marketing Investment Labr supprt -3-2 -1 0 1 2 3 4 5-3 -2-1 0 1 2 3 4 5-3 -2-1 0 1 2 3 4 5-3 -2-1 0 1 2 3 4 5 95% cnfidence interval
Indirect effects Plicy scheme may have indirect effects Example Eurstars: even rejected applicatins may have effects Beware: cntaminated cntrl grup
Cnclusins The use f apprpriate ecnmetrics methds increased significantly in the last decades. Next steps: There is still rm fr imprvement with respect t identificatin Explit discntinuities, instruments, experiments apply methds in a mre useful way fr plicy making (i.e. beynd hmgenus treatment effects n the treated f a single prgramme) Design f plicy schemes Selectin f plicy schemes