The evaluation of policy framework reforms in network industries empirical and methodological issues

The evaluation of policy framework reforms in network industries empirical and methodological issues Andrea Bastianin (joint with P. Castelnovo and M. Florio)

Introduction

Introduction Regulatory policy Regulatory policy: along with fiscal and monetary policies, is a tool that policy makers can use to influence the welfare of a country. When Reforms might be useful? market failures: When people do not pay for the consequences of their actions (...) This is the greatest market failure the world has seen. It is an externality that goes beyond those of ordinary congestion or pollution... (Sir Nicholas Stern). guess what is the quote about? socially unacceptable income and wealth distributions; provide merit goods that would otherwise under-consumed and/or under-produced (e.g. vaccination).

Introduction Regulatory policy Aim: to enhance efficiency and ultimately to boost economic growth. Empirical evidence highlights that changes in policy can account for some variability of economic growth.

Introduction Reforms and growth: empirics Source: Prati, A., Onorato, M. G., & Papageorgiou, C. (2013). Which reforms work and under what institutional environment? Evidence from a new data set on structural reforms. Review of Economics and Statistics, 95(3), 946-968.

Introduction Regulatory policy Aim: to enhance efficiency and ultimately to boost economic. Empirical evidence highlights that changes in policy can account for some variability of economic growth. Reforms are often invoked by policy makers. Structural reforms increase both potential output and the resilience of the economy to shocks. This makes structural reforms relevant for any central bank, but especially in a monetary union (Mario Draghi, 2015) [Source: Introductory speech by Mario Draghi, President of the ECB, ECB Forum on Central Banking, Sintra, 22 May 2015.]

Introduction Regulatory policy Aim: to enhance efficiency and ultimately to boost economic. Empirical evidence highlights that changes in policy can account for some variability of economic growth. Reforms are often invoked by policy makers. desirable reforms must yield economic and social benefits that outweigh their costs: social welfare function (SWF) must increase. Implement reform if DSFW > 0 A seemingly easy task: empirical and methodological issues

Introduction Regulatory policy Aim: to enhance efficiency and ultimately to boost economic. Empirical evidence highlights that changes in policy can account for some variability of economic growth. Reforms are often invoked by policy makers. desirable reforms must yield economic and social benefits that outweigh their costs: social welfare function (SWF) must increase. Implement reform if DSFW > 0 A seemingly easy task: empirical and methodological issues

Theory

Theory The RSG approach: a primer The Ramsey-Samuelson-Guesnerie (RSG) approach: Reform D vector of signals Signals = variables affecting the behaviour and welfare of individuals and firms (e.g. prices, rations, taxes, interest rates) Aim: to approximate how the social welfare function changes in response to a marginal change in the vector of signals controlled by the government. Theoretical models: mathematical description of functional relations linking government objectives, signals, constraints and private agents reactions and welfare.

Theory The RSG approach: shortcomings Real world reforms D vector of signals. They do not necessarily involve an immediate change of signals, nor an instantaneous variation in the SWF. Implemented with complex legislative packages: Regulation of public utility services in the EU involve the adoption of both legislative and non-legislative acts: Legislative acts (secondary law): directives, regulations and decisions. Non-legislative acts/soft-laws: communications, green and white papers (i.e. provide a correct interpretation of the primary and secondary laws). Aim: to provide mechanisms that, under certain circumstances, should pave the way for a change of signals.

Theory Why the RSG fail? An example Local loop unbundling in European member states Aim is to give the right to entrants to use the incumbent telecom operator s local broadband network to offer their services directly to customers. What are relevant and measurable signals? A better description: a modification of the regulatory policy framework that is expected to create the conditions for a change in signals

Theory Policy Framework Reforms LLU is an example of a Policy Framework Reforms (PFR) The RSG approach is neither applicable, nor appropriate. Empirical measurement of PFR is complex Empirical analyses are difficult, but feasible and necessary Care needed to formulate policy recommendations.

What are PFR in practice?

What are PFR in practice? Exclusion criteria: regulatory packages not immediately translated into a change in the vector of signals That do not automatically and directly affect private agents utility function that cannot be analyzed within the RSG framework.

What are PFR in practice? Some examples Washington Consensus 1 Other reforms: opening to trade, financial liberalization, judicial reform, privatization of state enterprises, reduction of entry barriers, tax reform, removal of targeted industrial subsidies, and central bank independence. 1 See: https://en.wikipedia.org/wiki/washington_consensus

What are PFR in practice? Some examples Other examples: opening to trade financial liberalization judicial reform privatization of state enterprises reduction of entry barriers tax reform removal of targeted industrial subsidies central bank independence

What are PFR in practice? Definitional attempts A list is never exhaustive Rely on a broader definition: regulatory policy is about achieving government's objectives through the use of regulations, laws, and other instruments to deliver better economic and social outcomes and thus enhance the life of citizens and business. OECD, 2012. Recommendation of the Council on regulatory policy and governance. OECD Publishing, Paris.

How are policy framework reforms measured?

Measuring PFR Given that real world reforms D vector of signals No natural unit of measure (i.e. %D interest rate or $D gasoline price) Artificial indices of reforms: No natural unit of measure Summarize different data sources (e.g. institutional and survey data) Scoring system Multi-dimensional Rely on arbitrary weighting scheme for aggregation (typically equal weights)

Measuring PFR: examples The OECD indicators of Energy, Transport and Communications Regulation (ETCR). The World Economic Forum s Global Competitiveness Index and its sub-indices. Indices central bank independence developed by various authors. Indices of the quality of institutions (see: http://qog.pol.gu.se/).

Measuring PFR: the ETCR database It provides information about regulatory structures and policies for OECD and some non-oecd countries. Annual time series starting in the mid-1970s. Information is collected with a questionnaire and complemented with publicly available data.

Measuring PFR: the ETCR database Questionnaires: Addressed to a country s government closed questions answered with numerical values (e.g. what is the market share of the largest company in the sector?) or by selecting from a set of pre-specified answers (e.g. a question that can be answered with yes or no, such as: is unbundling of the local loop required?). Qualitative information is coded into quantitative measures and then all answers are normalized in a range from zero to six, where values near zero indicate fewer restrictions to competition.

Measuring PFR: the ETCR database ETCR - Electricity

Measuring PFR: the ETCR database Overall ETCR index = equally weighted index of comprising sub-indices for seven network sectors 1. Telecom 2. Electricity 3. Gas 4. Post 5. air transport 6. rail transport 7. road transport. For each sector, there are up to four sub-indices that cover different dimensions of the reforms: entry regulation, public ownership, vertical integration and market regulation.

What are PFR in practice?

Measuring PFR: the ETCR database Trends of ECTR reform indicators in EU-15 countries for the natural gas industry, 1975 2013.

A prototypical econometric model for the analysis of PFR

Econometrics A survival guide Econometric method to use is jointly determined by: Aim of analysis Data availability Nature of dependent and independent variables PFR analyses typically rely on panel data

Econometrics A survival guide i t price Regulation Reg01 Italy 2012 12 3 1 France 2012 13 2 0 Germany 2012 15 4 1 Nature of dataset? Sample size? Nature of variables?

Econometrics A survival guide i t price Regulation Reg01 Italy 2010 12 6 1 Italy 2011 11 5 1 Italy 2012 12 3 1 Nature of dataset? Sample size? (time periods & sample frequency)

Econometrics A survival guide i t price Regulation Reg01 Italy 2010 12 6 1 Italy 2011 11 5 1 Italy 2012 12 3 1 France 2010 14 2 0 France 2011 12 2 0 France 2012 13 2 0 Germany 2010 13 6 1 Germany 2011 15 5 1 Germany 2012 15 4 1 Nature of dataset? Sample size? (time periods & sample frequency)

Econometrics A survival guide Typically the aim is to measure the impact of PFR on some outcome variable, y, possibly linked with social welfare (e.g. consumers prices) A linear model: y it = a + b PFR it + c x it + u it for i = 1,,n and t = 1,,T Define / identify the following: Dependent variable Explanatory/Independent variables or regressors Control variables (what can they include) Sample size Parameter VS estimates of parameters OLS Regression Fitted VS Actual; residuals VS errors

Econometrics A survival guide Typically the aim is to measure the impact of PFR on some outcome variable, y, possibly linked with social welfare (e.g. consumers prices) A linear model: y i = a + c x i + u i for i = 1,,n Interpret estimates of : a (geometrically and economically, if y continous in R) c with x continuos (geometrically and economically) What happens to y if x increases by 1-unit? Dy = y -y = (a + c x ) (a + c x) = c Dx Dx = x -x = (x+1)-x = 1 If Dx = 1 then Dy = c What happens to y if x doubles?

Econometrics A survival guide A linear model with a dummy: y t = a + c PFR t + b x t + u t for t = 1990,,2016 where PFR t = 1 if t>2000 and 0 otherwise. If t<2000, PFR t = 0 : y t = a (for x t = 0) If t>2000, PFR t = 1 : y t = a+c (for x t = 0)

Econometrics A survival guide A linear model with a dummy: y t = a + d (PFR t * x t )+ b x t + u t for t = 1990,,2016 where PFR t = 1 if t>2000 and 0 otherwise. y t = a (for x t = 0 and any t) If t<2000, PFR t = 0 : Dy t = b (for Dx t = 1) If t>2000, PFR t = 1 : Dy t = b+d (for Dx t = 1)

Econometrics A survival guide A linear model with a dummy: y t = a + c PFR t + d(pfr t *x t )+ b x t + u t for t = 1990,,2016 where PFR t = 1 if t>2000 and 0 otherwise. If t<2000, PFR t = 0 : y t = a (for x t = 0) If t>2000, PFR t = 1 : y t = a+c (for x t = 0) If t<2000, PFR t = 0 : Dy t = b (for Dx t = 1) If t>2000, PFR t = 1 : Dy t = b+d (for Dx t = 1)

Econometrics A survival guide Something useful that often students do not like: logs Variables in econometric models are often logtransformed: why?

Econometrics A survival guide 1. Ensure positivity: p = a + b*pfr with b<0 can lead to p<0 nonsensical! Then log(p) = a + b*pfr, so that p>0 always!! i.e. p = exp(a+b*pfr)

Econometrics A survival guide 1. Ensure positivity 2. Linearize otherwise nonlinear relations: Y = al b K (1-b) log(y) = log(a) + b*log(l) + (1-b) log(k)

Econometrics A survival guide 1. Ensure positivity 2. Linearize otherwise nonlinear relations: % Urban Pop VS per-capita GDP % Urban Pop VS log(p.c. GDP)

Econometrics A survival guide 1. Ensure positivity 2. Linearize otherwise nonlinear relations 3. «Makes things look normal» (consistently with OLS assumptions)

Econometrics A survival guide 1. Ensure positivity 2. Linearize otherwise nonlinear relations 3. «Makes things look normal» 4. In log-log models estimates = elasticities (unitless and dear to economists)

Econometrics A survival guide 1. Ensure positivity 2. Linearize otherwise nonlinear relations 3. «Makes things look normal» 4. In log-log models estimates = elasticities (unitless and dear to economists) Rewrite log y = a + b log x as: y = exp(a)x b and recall that y x x y elasticity = y x y x = exp(a)bxb-1 x = exp a bxb 1 = exp a x b bxb 1 x 1 b = b x y

Econometrics A survival guide 1. Ensure positivity 2. Linearize otherwise nonlinear relations 3. «Makes things look normal» 4. In log-log models estimates = elasticities (unitless and dear to economists) So in log y = a + b log x if b = -0.8, it means that: if Dx = 1% Dy = -0.8% Note: b 100*(y -y)/y where y is y evaluated for x = 1.01x

Econometrics A survival guide Note: b 100*(y -y)/y where y is y evaluated for x = 1.01x y = exp(a)(x^b) 2.718 y' = exp(a)(x'^b) 2.740 (y' - y) 0.022 (y' - y) / y 0.008 100*(y' - y) / y 0.799 a 1 b 0.8 x 1 x' = 1.01*x 1.01

Econometrics A survival guide Note: b 100*(y -y)/y where y is y evaluated for x = 1.01x This interpretation does not apply when x is a dummy When x is binary y does not make sense neither from x mathematical, nor from an economic perspective In this case need to compute 100*(y 1 -y 0 )/y 0 where y 1 is y evaluated for x = 1 and y 1 is y evaluated for x = 0

Potential pitfalls of policy framework reforms data

Potential pitfalls of PFR data PFR proxies is prone to several types of errors: 1. Conceptual errors; 2. Discretization, definition of orderings and metrics. 3. Aggregation errors. Notice that issues 1-3 all cause of a broader class of empirical issues that goes under the header of measurement error.

Conceptual errors PFR indicator might also be the outcome of some other macroeconomic shock Example, unbundling: The regulator forces the incumbent to divest generation capacity PFR proxy = Market Share (MS) of the largest electricity producer in a country MS can change because (1) the policy (2) an exogenous technological shock has changed the optimal production plan not only for the incumbent, but for any firm in a sector In this case, the technological shock is a confounding factor, that causes an omitted variable bias and hence ultimately affects the evidence in favor or against the reform.

Conceptual errors In this case, the technological shock is a confounding factor, that causes an omitted variable bias and hence ultimately affects the evidence in favor or against the reform. a policy-unrelated change might be attributed to a PFR. Control variables or use alternative proxies can mitigate the omitted variable problem

Discretization Discretization = conversion of a continuous variable into a discrete one. The reason for relying on discretized variables is the need to find a unit of measure for PFR proxies. Drawbacks: useful variability is discarded loss of variability reduces the precision of in-sample and out-of-sample predictions. interpretational issues: using a dummy variable within a regression to measure, say an anti-monopoly policy, implies that competition increases by the same magnitude in all countries (statistical units). it is often completely arbitrary

Discretization Arbitrary classification Loss of variability

Aggregation Equal weights are often assigned to different aspects of PFR Upper-level indices are simple averages or equally weighted sums of the lower-level variables. Equal weights are neither theoretically-driven nor the result of estimation: how to weight entry and unbundling scores? Aggregation might bias estimates The problem can be either attenuated by relying on data-reduction techniques, or completely avoided by using lower-level indicators. The use of aggregate PFR measures in place of sub-indices lead to a loss of information in that it does not allow to identify the effects resulting from distinct aspects of the regulation. A further alternative to mitigate the aggregation bias is to check the robustness of results to different weighting schemes with a random weights analysis.

Aggregation Equal weights are often assigned to different aspects of PFR Upper-level indices are simple averages or equally weighted sums of the lower-level variables.

Measurement error Measurement error = any deviation from the true value of variable, dependent or independent, that arises in the definitional or measurement stage It subsumes all the issues discussed so far. PFR proxies rely on several non-standard data sources such as media coverage, legal and non-legal acts, expert surveys. Misreporting by subjects, coding and other errors, such as those discussed above, are likely to inflate the measurement error. Also missing data are a limiting special case of mismeasurement that arises when there is no prior information about the true unobserved values of the variable of interest. Conceptual errors are another source of mismeasurement: observed data often do not correspond to the exact concept the analyst is interested in (e.g. the use of years of schooling as a proxy for human capital).

Measurement error Measurement errors can be purely random or have both a casual and a nonrandom component. Nonrandom measurement errors arise when the deviation from the true value of variable is systematically influenced by some factors that introduce an upward or downward bias in the observations. data are often more accurate for some countries than for others long time series are more precisely measured in the present than in the past.

Measurement error Random or classical measurement error: True, but latent (unobserved) relation: y i = b 0 x * i + u i for i = 1,,n Observed value (proxy) of latent x * i : x i = x * i + e i e i = random Estimate: y i = b x i + u i Can show that OLS estimate of b, denoted as b, is downard biased (or attenuated) also asymptotically: bias = (b - b 0 ) < 0 even if n In other settings (multiple x, nonlinear models, nonlinear transformatios ) cannot determine a-priori the sign of the bias.

Measurement error Monte Carlo: Suppose we estimate y i = b x i + u i many times (M) At each run we draw a new error term u, compute y, estimate b and save the OLS estimate b Repeat this M = 1000 times to obtain the distribution of b

Measurement error An unbiased and consistent estimator

Measurement error A biased and inconsistent estimator

What should I do??? Econometric analysis is useful, but complex. Some rules: 1. Understand your problem 2. Undestand your dataset and variables 3. Make sensible things and comments. 4. Be courious and dig deeper: are results robust?? What if I add more variables, change a variable.

An example

An example log