Lecture 12 Robust Estimation

Transcription

1 Lecture 12 Robust Estimation Prof. Dr. Svetlozar Rachev Institute for Statistics and Mathematical Economics University of Karlsruhe Financial Econometrics, Summer Semester 2007

2 Copyright These lecture-notes cannot be copied and/or distributed without permission. The material is based on the text-book: Financial Econometrics: From Basics to Advanced Modeling Techniques (Wiley-Finance, Frank J. Fabozzi Series) by Svetlozar T. Rachev, Stefan Mittnik, Frank Fabozzi, Sergio M. Focardi,Teo Jaši `c.

3 Outline Robust statistics. Robust estimators of regressions. Illustration: robustness of the corporate bond yield spread model.

4 Robust Statistics Robust statistics addresses the problem of making estimates that are insensitive to small changes in the basic assumptions of the statistical models employed. The concepts and methods of robust statistics originated in the 1950s. However, the concepts of robust statistics had been used much earlier. Robust statistics: 1. assesses the changes in estimates due to small changes in the basic assumptions; 2. creates new estimates that are insensitive to small changes in some of the assumptions. Robust statistics is also useful to separate the contribution of the tails from the contribution of the body of the data.

5 Robust Statistics Peter Huber observed, that robust, distribution-free, and nonparametrical actually are not closely related properties. Example: The sample mean and the sample median are nonparametric estimates of the mean and the median but the mean is not robust to outliers. In fact, changes of one single observation might have unbounded effects on the mean while the median is insensitive to changes of up to half the sample. Robust methods assume that there are indeed parameters in the distributions under study and attempt to minimize the effects of outliers as well as erroneous assumptions on the shape of the distribution.

6 Robust Statistics: Qualitative and Quantitative Robustness Estimators are functions of the sample data. Given an N-sample of data X = (x 1,..., x N ) from a population with a cdf F (x), depending on parameter Θ, an estimator for Θ is a function ˆϑ = ϑ N (x 1,..., x N ).. Consider those estimators that can be written as functions of the cumulative empirical distribution function: F N (x) = N 1 N i=1 I (x i x) where I is the indicator function. For these estimators we can write ˆϑ = ϑ N (F N )

7 Robust Statistics: Qualitative and Quantitative Robustness Most estimators, in particular the ML estimators, can be written in this way with probability 1. In general, when N then F N (x) F (x) and ˆϑ N ϑ in probability. The estimator ˆϑ N is a random variable that depends on the sample. Under the distribution F, it will have a probability distribution L F (ϑ N ). Statistics defined as functionals of a distribution are robust if they are continuous with respect to the distribution.

8 Robust Statistics: Qualitative and Quantitative Robustness In 1968, Hampel introduced a technical definition of qualitative robustness based on metrics of the functional space of distributions. It states that an estimator is robust for a given distribution F if small deviations from F in the given metric result in small deviations from L F (ϑ N ) in the same metric or eventually in some other metric for any sequence of samples of increasing size. The definition of robustness can be made quantitative by assessing quantitatively how changes in the distribution F affect the distribution L F (ϑ N ).

9 Robust Statistics: Resistant Estimators An estimator is called resistant if it is insensitive to changes in one single observation. Given an estimator ˆϑ = ϑ N (F N ),we want to understand what happens if we add a new observation of value x to a large sample. To this end we define the influence curve (IC), also called influence function. The IC is a function of x given ϑ, and F is defined as follows: IC ϑ,f (x) = lim s 0 ϑ((1 s)f + sδ x ) ϑ(f ) s where δ x denotes a point mass 1 at x.

10 Robust Statistics: Resistant Estimators As we can see from its previous definition, the IC is a function of the size of the single observation that is added. In other words, the IC measures the influence of a single observation x on a statistics ϑ for a given distribution F. In practice, the influence curve is generated by plotting the value of the computed statistic with a single point of X added to Y against that X value. Example: The IC of the mean is a straight line.

11 Robust Statistics: Resistant Estimators Several aspects of the influence curve are of particular interest: Is the curve bounded as the X values become extreme? Robust statistics should be bounded. That is, a robust statistic should not be unduly influenced by a single extreme point. What is the general behavior as the X observation becomes extreme? For example, does it becomes smoothly down-weighted as the values become extreme? What is the influence if the X point is in the center of the Y points?.

12 Robust Statistics: Breakdown Bound The breakdown (BD) bound or point is the largest possible fraction of observations for which there is a bound on the change of the estimate when that fraction of the sample is altered without restrictions. Example: We can change up to 50% of the sample points without provoking unbounded changes of the median. On the contrary, changes of one single observation might have unbounded effects on the mean.

13 Robust Statistics: Rejection Point The rejection point is defined as the point beyond which the IC becomes zero. Note: The observations beyond the rejection point make no contribution to the final estimate except, possibly, through the auxiliary scale estimate. Estimators that have a finite rejection point are said to be redescending and are well protected against very large outliers. However, a finite rejection point usually results in the underestimation of scale.

14 Robust Statistics: Main concepts The gross error sensitivity expresses asymptotically the maximum effect that a contaminated observation can have on the estimator. It is the maximum absolute value of the IC. The local shift sensitivity measures the effect of the removal of a mass at y and its reintroduction at x. For continuous and differentiable IC, the local shift sensitivity is given by the maximum absolute value of the slope of IC at any point. Winsor s principle states that all distributions are normal in the middle.

15 Robust Statistics: M-Estimators M-estimators are those estimators that are obtained by minimizing a function of the sample data. Suppose that we are given an N-sample of data X= (x 1,..., x N ). The estimator T (x 1,..., x N ) is called an M-estimator if it is obtained by solving the following minimum problem: } N T = arg min t {J = ρ(x i, t) i=1 where ρ(x i, t) is an arbitrary function.

16 Robust Statistics: M-Estimators Alternatively, if ρ(x i, t) is a smooth function, we can say that T is an M-estimator if it is determined by solving the equations: N ψ(x i, t) = 0 i=1 where ψ(x i, t) = ρ(x i, t) t

17 Robust Statistics: M-Estimators When the M-estimator is equivariant, that is T (x 1 + a,..., x N + a) = T (x 1,..., x N ) + a, a R, we can write ψ and ρ in terms of the residuals x t. Also, in general, an auxiliary scale estimate, S, is used to obtain the scaled residuals r = (x t)/s. If the estimator is also equivariant to changes of scale, we can write ( x t ) ψ(x, t) = ψ = ψ(r) S ( x t ) ρ(x, t) = ρ = ρ(r) S

18 Robust Statistics: M-Estimators ML estimators are M-estimators with ρ = log f, where f is the probability density. The name M-estimators means maximum likelihood-type estimators. LS estimators are also M-estimators. The IC of M-estimators has a particularly simple form. In fact, it can be demonstrated that the IC is proportional to the function ψ: IC = Constant ψ

19 Robust Statistics: L-Estimators Consider an N-sample (x 1,..., x N ). Order the samples so that x (1) x (2) x (N). The i-th element X = x (i) of the ordered sample is called the i-th order statistic. L-estimators are estimators obtained as a linear combination of order statistics: N L = a i x (i) i=1 where the a i are fixed constants. Constants are typically normalized so that N a i = 1 i=1 An important example of an L-estimator is the trimmed mean. It is a mean formed excluding a fraction of the highest and/or lowest samples.

20 Robust Statistics: R-Estimators R-estimators are obtained by minimizing the sum of residuals weighted by functions of the rank of each residual. The functional to be minimized is the following: { } N arg min J = a(r i )r i where R i is the rank of the i-th residual r i and a is a nondecreasing score function that satisfies the condition i=1 N a(r i ) = 0 i=1

21 Robust Statistics: The Least Median of Squares Estimator The least median of squares (LMedS) estimator referred to minimizing the median of squared residuals, proposed by Rousseuw. This estimator effectively trims the N/2 observations having the largest residuals, and uses the maximal residual value in the remaining set as the criterion to be minimized. It is hence equivalent to assuming that the noise proportion is 50%. LMedS is unwieldy from a computational point of view because of its nondifferentiable form. This means that a quasi-exhaustive search on all possible parameter values needs to be done to find the global minimum.

22 Robust Statistics: The Least Trimmed of Squares Estimator The least trimmed of squares (LTS) estimator offers an efficient way to find robust estimates by minimizing the objective function given by { } h J = where r(i) 2 is the i-th smallest residual or distance when the residuals are ordered in ascending order, that is: r(1) 2 r (2) 2 r (N) 2 and h is the number of data points whose residuals we want to include in the sum. This estimator basically finds a robust estimate by identifying the N h points having the largest residuals as outliers, and discarding (trimming) them from the dataset. i=1 r 2 (i)

23 Robust Statistics: Reweighted Least Squares Estimator Some algorithms explicitly cast their objective functions in terms of a set of weights that distinguish between inliers and outliers. These weights usually depend on a scale measure that is also difficult to estimate. Example: The reweighted least squares (RLS) estimator uses the following objective function: { } arg min J = N i=1 ω i r 2 i where r i are robust residuals resulting from an approximate LMedS or LTS procedure. The weights ω i trim outliers from the data used in LS minimization, and can be computed after a preliminary approximate step of LMedS or LTS.

24 Robust Statistics: Robust Estimators of the Center The mean estimates the center of a distribution but it is not resistant. Resistant estimators of the center are the following: Trimmed mean. Suppose x (1) x (2) x (N) are the sample order statistics (that is, the sample sorted). The trimmed mean T N (δ, 1 γ) is defined as follows: T N (δ, 1 γ) = 1 U N L N U N j=l N +1 x j δ, γ (0, 0.5), L N = floor[nδ], U N = floor[nγ]

25 Robust Statistics: Robust Estimators of the Center Winsorized mean. The Winsorized mean is the mean X W of Winsorized data: x IN +1 j L N y j = x j L N + 1 j U N x j = x UN +1 j U N + 1 X W = Ȳ Median. The median Med(X) is defined as that value that occupies a central position in a sample order statistics: { x((n+1)/2) if N is odd Med(X ) = ((x (N/2) + x (N/2+1) )/2) if N is even

26 Robust Statistics: Robust Estimators of the Spread The variance is a classical estimator of the spread but it is not robust. Robust estimators of the spread are the following: Median absolute deviation. The median absolute deviation (MAD) is defined as the median of the absolute value of the difference between a variable and its median, that is, MAD = MED X MED(X ) Interquartile range. The interquartile range (IQR) is defined as the difference between the highest and lowest quartile: IQR = Q(0.75) Q(0.25) where Q(0.75) and Q(0.25) are the 75th and 25th percentiles of the data.

27 Robust Statistics: Robust Estimators of the Spread Mean absolute deviation. The mean absolute deviation (MeanAD) is defined as follows: 1 N N x j MED(X ) j=1 Winsorized standard deviation. The Winsorized standard deviation is the standard deviation of Winsorized data, that is, σ W = σ N (U N L N )/N

28 Robust Statistics: Illustration of Robust Statistics To illustrate the effect of robust statistics, consider the series of daily returns of Nippon Oil in the period 1986 through The following was computed: Mean = e-005 Trimmed mean (20%) = e-004 Median = 0 In order to show the robustness properties of these estimators, lets multiply the 10% highest/lowest returns by 2. Then Mean = e-004 Trimmed mean (20%) = e-004 Median = 0

29 Robust Statistics: Illustration of Robust Statistics

30 Robust Statistics: Illustration of Robust Statistics We can perform the same exercise for measures of the spread. We obtain the following results: Standard deviation = IQR = MAD = Lets multiply the 10% highest/lowest returns by 2. The new values are: Standard deviation = IQR = MAD = If we multiply the 25% highest/lowest returns by 2, then Standard deviation = IQR = MAD =

31 Robust Estimators of Regressions Identifying robust estimators of regressions is a rather difficult problem. In fact, different choices of estimators, robust or not, might lead to radically different estimates of slopes and intercepts. Consider the following linear regression model: Y = β 0 + N β i X i + ε i=1

32 Robust Estimators of Regressions If data are organized in matrix form as usual, Y 1 1 X 11 X N1 Y =., X = Y T 1 X 1T X NT β = β 1. β N, ε = then the regression equation takes the form, ε 1. ε T, Y = X β + ε

33 Robust Statistics: Illustration of Robust Statistics The standard nonrobust LS estimation of regression parameters minimizes the sum of squared residuals, T ε 2 t = i=1 T Y i i=1 N β ij X ij j=0 or, equivalently solves the system of N + 1 equations, T N Y i β ij X ij X ij = 0 i=1 j=0 or, in matrix notation, X X β = X Y. The solution of this system is ˆβ = (X X ) 1 X Y 2

34 Robust Estimators of Regressions The fitted values (i.e, the LS estimates of the expectations) of the Y are Ŷ = X (X X ) 1 X Y = HY The H matrix is called the hat matrix because it puts a hat on, that is, it computes the expectation Ŷ of the Y. The hat matrix H is a symmetric T T projection matrix; that is, the following relationship holds: HH = H. The matrix H has N eigenvalues equal to 1 and T N eigenvalues equal to 0. Its diagonal elements, h i h ii satisfy: 0 h i 1 and its trace (i.e., the sum of its diagonal elements) is equal to N: tr(h) = N

35 Robust Estimators of Regressions Under the assumption that the errors are independent and identically distributed with mean zero and variance σ 2, it can be demonstrated that the Ŷ are consistent, that is, Ŷ E(Y ) in probability when the sample becomes infinite if and only if h = max(h i ) 0. Points where the h i have large values are called leverage points. It can be demonstrated that the presence of leverage points signals that there are observations that might have a decisive influence on the estimation of the regression parameters. A rule of thumb suggests that values h i 0.2 are safe, values 0.2 h i 0.5 require careful attention, and higher values are to be avoided.

36 Robust Estimators of Regressions: Robust Regressions Based on M-Estimators The LS estimators ˆβ = (X X ) 1 X Y are M-estimators but are not robust. We can generalize LS seeking to minimize T N J = ρ Y i β ij X ij i=1 j=0 by solving the set of N + 1 simultaneous equations T N ψ Y i β ij X ij X ij = 0 i=1 j=0 where ψ = ρ β

37 Robust Estimators of Regressions: Robust Regressions Based on W-Estimators W-estimators offer an alternative form of M-estimators. They are obtained by rewriting M-estimators as follows: N N N ψ Y i β ij X ij = w Y i β ij X ij Y i β ij X ij j=0 j=0 Hence the N + 1 simultaneous equations become N N w Y i β ij X ij Y i β ij X ij = 0 or, in matrix form j=0 where W is a diagonal matrix. j=0 X WX β = X WY j=0

38 Robust Estimators of Regressions: Robust Regressions Based on W-Estimators The above is not a linear system because the weighting function is in general a nonlinear function of the data. A typical approach is to determine iteratively the weights through an iterative reweighted least squares (RLS) procedure. Clearly the iterative procedure depends numerically on the choice of the weighting functions. Two commonly used choices are the Huber weighting function w H (e) and the Tukey bisquare weighting function w T (e).

39 Robust Estimators of Regressions: Robust Regressions Based on W-Estimators The Huber weighting function defined as { 1 for e k w H (e) = k/ e for e > k The Tukey bisquare weighting function defined as w T (e) = { (1 (e/k) 2 ) 2 for e k 0 for e > k where k is a tuning constant often set at (standard deviation of errors) for the Huber function and k = (standard deviation of errors) for the Tukey function.

40 Illustration: Robustness of the Corporate Bond Yield Spread Model To illustrate robust regressions, we use the illustration of the spread regression (Chapter 4) to show how to incorporate dummy variables into a regression model. The leverage points are all very small. We therefore expect that the robust regression does not differ much from the standard regression. We ran two robust regressions with the Huber and Tukey weighting functions. The estimated coefficients of both robust regressions were identical to the coefficients of the standard regression.

41 Illustration: Robustness of the Corporate Bond Yield Spread Model For the Huber weighting function the tuning parameter (k) was set at 160, that is, the standard deviation of errors. The algorithm converged at the first iteration. For the Tukey weighting function the tuning parameter (k) set at 550, that is, the standard deviation of errors. The algorithm converged at the second iteration.

42 Illustration: Robustness of the Corporate Bond Yield Spread Model Another example:

43 Illustration: Robustness of the Corporate Bond Yield Spread Model Now suppose that we want to estimate the regression of Nippon Oil on this index; that is, we want to estimate the following regression: R NO = β 0 + β 1 R Index + Errors Estimation with the standard least squares method yields the following regression parameters: R 2 : Adjusted R 2 : Standard deviation of errors:

44 Illustration: Robustness of the Corporate Bond Yield Spread Model When we examined the diagonal of the hat matrix, we found the following results Maximum leverage = Mean leverage = e-004 suggesting that there is no dangerous point. Robust regression can be applied; that is, there is no need to change the regression design. We applied robust regression using the Huber and Tukey weighting functions with the following parameters: Huber (k = standard deviation) and Tukey (k = standard deviation)

45 Illustration: Robustness of the Corporate Bond Yield Spread Model The robust regression estimate with Huber weighting functions yields the following results: R 2 = Adjusted R 2 = Weight parameter = Number of iterations = 39

46 Illustration: Robustness of the Corporate Bond Yield Spread Model With the Tukey weighting functions: R 2 = Adjusted R 2 = Weight parameter = Number of iterations = 88 Conclusion: all regression slope estimates are highly significant; the intercept estimates are insignificant in all cases. There is a considerable difference between the robust (0.40) and the nonrobust (0.45) regression coefficient.

47 Illustration: Robust Estimation of Covariance and Correlation Matrices Variance-covariance matrices are central to modern portfolio theory. Suppose returns are a multivariate random vector written as r t = µ + ε t The random disturbances ε t is characterized by a covariance matrix Ω. ρ X,Y = Corr(X, Y ) = Cov(X, Y ) Var(X )Var(Y ) = σ X,Y σ X σ Y The correlation coefficient fully represents the dependence structure of multivariate normal distribution.

48 Illustration: Robust Estimation of Covariance and Correlation Matrices The empirical covariance between two variables is defined as where ˆσ X,Y = 1 N 1 X = 1 N N (X i X )(Y i Ȳ ) i=1 N X i, i=1 Ȳ = 1 N are the empirical means of the variables. N i=1 Y i

49 Illustration: Robust Estimation of Covariance and Correlation Matrices The empirical correlation coefficient is the empirical covariance normalized with the product of the respective empirical standard deviations: ˆρ X,Y = ˆσ X,Y ˆσ X ˆσ Y The empirical standard deviations are defined as ˆσ X = 1 N (X i X ) N 2, ˆσ Y = 1 N (Y i Ȳ ) N 2 i=1 Empirical covariances and correlations are not robust as they are highly sensitive to tails or outliers. Robust estimators of covariances and/or correlations are insensitive to the tails. i=1

50 Illustration: Robust Estimation of Covariance and Correlation Matrices Different strategies for robust estimation of covariances exist; among them are: Robust estimation of pairwise covariances Robust estimation of elliptic distributions We discuss only the robust estimation of pairwise covariances.

51 Illustration: Robust Estimation of Covariance and Correlation Matrices The following identity holds: cov(x, Y ) = 1 [var(ax + by ) var(ax by )] 4ab Assume S is a robust scale functional: S(aX + b) = a S(X ) A robust covariance is defined as C(X, Y ) = 1 4ab [S(aX + by )2 S(aX by ) 2 ]

52 Illustration: Robust Estimation of Covariance and Correlation Matrices Choose a = 1 S(X ), b = 1 S(Y ) A robust correlation coefficient is defined as c = 1 4 [S(aX + by )2 S(aX by ) 2 ] The robust correlation coefficient thus defined is not confined to stay in the interval [-1,+1]. For this reason the following alternative definition is often used: r = S(aX + by )2 S(aX by ) 2 S(aX + by ) 2 + S(aX by ) 2

53 Illustration: Applications Robust regressions have been used to improve estimates in the area of the market risk of a stock (beta) and of the factor loadings in a factor model. Martin and Simin propose a weighted least-squares estimator with data-dependent weights for estimating beta, referring to this estimate as resistant beta, and report that this beta is a superior predictor of future risk and return characteristics than the beta calculated using LS. The potential dramatic difference between the LS beta and the resistant beta:

54 Illustration: Applications Fama and French studies find that market capitalization (size) and book-to-market are important factors in explaining cross-sectional returns. These results are purely empirically based. The empirical evidence that size may be a factor that earns a risk premia was first reported by Banz. Knez and Ready reexamined the empirical evidence using robust regressions (the least-trimmed squares regression). First, they find that when 1% of the most extreme observations are trimmed each month, the risk premia for the size factor disappears. Second, the inverse relation between size and the risk premia no longer holds when the sample is trimmed.

55 Final Remarks The required textbook is Financial Econometrics: From Basics to Advanced Modeling Techniques. Please read Chapter 12 for today s lecture. All concepts explained are listed in page 428 of the textbook.