Data-based Automatic Discretization of Nonparametric Distributions

Transcription

1 Data-based Automatic Discretization of Nonparametric Distributions Alexis Akira Toda arxiv: v1 [q-fin.ec] 2 May 2018 May 3, 2018 Abstract Although using non-gaussian distributions in economic models has become increasingly popular, currently there is no systematic way for calibrating a discrete distribution from the data without imposing parametric assumptions. This paper proposes a simple nonparametric calibration method that combines the kernel density estimator and Gaussian quadrature. Applications to an asset pricing model and an optimal portfolio problem suggest that assuming Gaussian instead of nonparametric shocks leads to a 15 25% reduction in the equity premium and overweighting in the stock portfolio because the investor underestimates the probability of crashes. Keywords: calibration, discrete approximation, Gaussian mixture, Gaussian quadrature, kernel density estimator. JEL codes: C63, C65, G11. 1 Introduction This paper studies the following problem, which applied theorists often encounter. A researcher would like to calibrate the parameters of a stochastic model. One of the model inputs is a probability distribution of shocks, which is to be approximated by a discrete distribution. Due to computational considerations, the researcher would like this distribution to have as few support points (nodes) as possible, say five. Given the data of shocks, how should the researcher calibrate the nodes and probabilities of this five-point distribution? While there are many established methods for discretizing processes with Gaussian shocks such as Tauchen(1986), Tauchen and Hussey(1991), and Rouwenhorst (1995), 1 discretizing non-gaussian distributions remains relatively unexplored. However, it has become increasingly common in economics to study models with non-gaussian shocks. For example, the rare disasters model(rietz, 1988; Barro, 2006; Gabaix, 2012) uses rare but large downward jumps to explain asset pricing puzzles. One issue with discretizing non-gaussian distributions is how to calibrate them. If we have a parametric density, it is possible to discretize it using the Gaussian quadrature as in Miller and Rice (1983) or the maximum entropy Department of Economics, University of California San Diego. atoda@ucsd.edu. 1 See Farmer and Toda (2017) and the references therein for a detailed literature review. 1

2 methodasintanaka and Toda(2013,2015)providedthatwecancomputesome moments. However, it is not obvious how to obtain an N-point distribution that approximates the data well without imposing parametric assumptions. Given the data, this paper proposes a new and simple method for automatically calibrating a discrete distribution with a specified number of grid points. The method combines two well-known techniques, the (Gaussian) kernel density estimation and Gaussian quadrature. 2 In the first step, we nonparametrically estimate the probability distribution using the kernel density estimator with the Gaussian (normal) kernel. Because the kernel density estimator can be viewed as a Gaussian mixture, we can compute the moments of order up to 2N efficiently. Using these moments, in the second step we compute the nodes and weights of the N-point Gaussian quadrature. These nodes and weights give us the N-point discretization of the nonparametric distribution. Since this method does not involve optimization (it is a matter of recursively computing moments and solving for the eigenvalues/vectors of a sparse matrix), the implementation is easy and fast. As applications, I discretize the U.S. historical data on aggregate consumption growth and stock returns and solve an asset pricing model and an optimal portfolio problem with constant relative risk aversion utility. I consider two cases in which the investor uses the nonparametric and Gaussian densities. I show that when the investor incorrectly believes that the consumption growth/stock returns distribution is lognormal, the equity premium shrinks by 16 24% and the stock portfolio is overweighted by 16 26% because he underestimates the probability of recessions/crashes. These examples show that the choice of the calibration method may matter quantitatively. 1.1 Related literature The closest paper to mine is Miller and Rice (1983), who use the Gaussian quadrature to discretize distributions. While they consider only the discretization of parametric distributions, my focus is on the discretization of nonparametric distributions estimated from data. Tanaka and Toda (2013) consider the discretization of distributions on preassigned nodes by matching the moments using the maximum entropy principle, and Tanaka and Toda(2015) prove convergence and obtain an error estimate. Farmer and Toda (2017) consider the discretization of general non-gaussian Markov processes by applying the Tanaka-Toda method to conditional distributions. In one of the applications, they discretize a nonparametric density on a preassigned grid by approximating it with a Gaussian mixture. Since computing the kernel density estimator does not require optimizing over parameters (unlike the maximum likelihood estimation of Gaussian mixture parameters or solving the maximum entropy problem), my method is easier and faster to implement, and the grid is chosen endogenously. 2 In this paper the term Gaussian has two different meanings. Gaussian quadrature refers to the quadrature (numerical integration) formula that integrates the maximum number of monomials. Gaussian mixture/distribution refers to the normal mixture/distribution in statistics. The widely used Gaussian quadrature for the Gaussian (normal) distribution is called Gauss-Hermite quadrature. 2

3 2 Method 2.1 Discretizing a density using Gaussian quadrature Suppose for the moment that the nonparametric density f(x) is known. Since stochasticmodelsofteninvolveexpectations,wewouldliketofindnodes{x n } N n=1 and weights {w n } N n=1 such that E[g(X)] = g(x)f(x)dx w n g(x n ), (2.1) where g is a general integrand and X is a random variable with density f(x). The right-hand side of (2.1) defines an N-point quadrature formula. When (2.1) is exact (i.e., becomes =) for all polynomials of degree up to D, we say that the quadrature formula has degree of exactness D. Since the degree of freedom in an N-point quadrature formula is 2N (because there are N nodes and N weights), we cannot expect to integrate more than 2N monomials f(x) = 1,x,...,x 2N 1 exactly. When the quadrature formula (2.1) is exact for these monomials, or equivalently when it has degree of exactness 2N 1, we call the formula Gaussian. The following Golub-Welsch algorithm provides an efficient way to compute the nodes and weights of the Gaussian quadrature. (Appendix B provides more theoretical background.) n=1 Algorithm 1 (Golub and Welsch, 1969). 1. Select a number of quadrature nodes N N. 2. For k = 0,1,...,2N, compute the k-th moment of the density m k = x k f(x)dx. 3. DefinethematrixofmomentsM = (M ij ) 1 i,j N+1 bym ij = m i+j Compute the Cholesky factorization M = R R. Let R = (r ij ) 1 i,j N Define α 1 = r 12 /r 11, α n = rn,n+1 r nn rn 1,n r n 1,n 1 (n = 2,...,N), and β n = rn+1,n+1 r nn (n = 1,...,N 1). Define the N N symmetric tridiagonal matrix α 1 β β 1 α 2 β.. 2. T N =. 0 β 2 α (2.2) βn β N 1 α N 6. Compute the eigenvalues {x n } N n=1 of T N and the correspondingeigenvectors {v n } N n=1. {x n} N n=1 are the nodes of the Gaussian quadrature and the weights {w n } N n=1 are given by w n = m 0 vn1 2 / v n 2 > 0, where v n = (v n1,...,v nn ). 3

4 Once we compute the nodes {x n } N n=1 and weights {w n} N n=1, we can use them as the discrete approximation of the density f. 2.2 Computing moments using kernel density estimator The only inputs to the Golub-Welsch algorithm are the number of nodes N and the moments m k = x k f(x)dx of the density f, where k = 0,...,2N. We can compute these quantities using the kernel density estimator, as follows. Given the data {x i } I i=1, the kernel density estimator (with Gaussian kernel) is defined by f h (x) = 1 Ih I ( ) x xi φ, (2.3) h i=1 where h > 0 is the bandwidth and φ(x) = 1 2π e x2 /2 is the standard normal density. I do not take a stance on the choice of the bandwidth h. When f is itself Gaussian, it has been shown that ( ) 1/5 4 h = σ (2.4) 3I is optimal in the sense of the mean integrated squared error, where σ is the sample standard deviation (Silverman, 1986). Note that the kernel density estimator(2.3) can be viewed as a Gaussian mixture density with I components, where the proportion, mean, and standard deviation of the i-th component are 1/I, x i, and h, respectively. As such, it is straightforward to compute its moments recursively. First, assume that the number of mixtures is 1, so the Gaussian mixture is just N(µ,σ 2 ). Let m k = E[X k ] be the k-th noncentral moment. Clearly m 0 = 1 and m 1 = µ. For k 2, using integration by parts we can compute m k = = =µ x k 1 2πσ 2 e (x µ)2 /2σ 2 dx x k 1 1 [ σ 2( e (x µ)2 /2σ 2) ] +µe (x µ) 2 /2σ 2 dx 2πσ 2 x k 1 1 2πσ 2 e (x µ)2 /2σ 2 dx σ 2 [ [ x k 1 2πσ 2 e (x µ)2 /2σ 2 ] ] (x k ) 1 /2σ 2 dx 2πσ 2 e (x µ)2 =µm k 1 +σ 2 (k 1)m k 2. (2.5) Therefore we can iterate this equation to compute m 2,m 3,...,m 2N. The k-th moment of the kernel density estimator (2.3) is m k = x k fh (x)dx = 1 I I m k (x i,h), (2.6) where m k (µ,σ) is the k-th moment of N(µ,σ 2 ) computed as in (2.5). Since the moments of the Gaussian mixture can be explicitly calculated this way, it is straightforward to compute the nodes and weights of the corresponding Gaussian quadrature using the Golub-Welsch algorithm 1. i=1 4

5 2.3 Summary of algorithm Summarizing the above observations, we obtain the following algorithm for the data-based automatic discretization of nonparametric distributions. Algorithm 2 (Automatic discretization of nonparametric distributions). 1. Given the data {x i } I i=1, choose a bandwidth h (for example, Silverman s rule of thumb (2.4)). 2. Given the number of discrete points N, for k = 0,...,2N compute the k-th moment of the kernel density estimator recursively as in (2.6) and (2.5). 3. Usethese moments {m k } 2N k=0 in the Golub-Welschalgorithm1to compute the nodes { x n } N n=1 and weights{w n} N n=1. The desired discretization assigns probability w n on the point x n. Appendix A shows that the accuracy of the proposed method exceeds that of using a parametric distribution when the latter is misspecified. 3 Applications In this section I illustrate the usefulness of the proposed method using minimal economic examples. In the first application, I solve a consumption-based asset pricing model by calibrating the consumption growth distribution from the data. In the second application, I solve a static optimal portfolio problem by calibrating the stock returns distribution from the data. In each case I compare the nonparametric solution to the case when the distribution is assumed to be lognormal and show that the choice of the discretization method matters quantitatively. 3.1 Asset pricing model Consider a plain-vanilla Lucas (1978) consumption-based asset pricing model, where a representative agent with constant relative risk aversion (CRRA) utility consumes the aggregate endowment. Endowment growth is assumed to be independent and identically distributed (i.i.d.) over time with a general distribution. A stock is modeled as a claim to the aggregate endowment, and the supply of the risk-free asset is zero. Let R > 0 be the gross stock return (which is an i.i.d. random variable) and R f > 0 be the gross risk-free rate. Letting γ > 0 be the relative risk aversion coefficient of the representative agent, it is well known that the log equity premium is given by E[logR] logr f = logm(1)+logm( γ) logm(1 γ) > 0, (3.1) 5

6 where M(s) = E[e sxt ] is the moment generating function of the log endowment growth x t = log(e t /e t 1 ). 3 I obtain the annual data on U.S. real per capita consumption for the period from the spreadsheet of Robert Shiller 4 and apply Algorithm 2 to the log consumption growth logc t /C t 1 to obtain a discrete distribution with nodes { x n } N n=1 and weights {w n} N n=1, where I choose the number of points N = 5 (increasing the number of points further does not change the results). I then compute the moment generating function as M(s) = N n=1 w ne s xn and substitute into (3.1) for s = 1, γ,1 γ to compute the log equity premium for the range γ (0,10]. Probability density Histogram Nonparametric Gaussian Log equity premium (%) Nonparametric Gaussian Log consumption growth (a) Histogram and densities Relative risk aversion (b) Equity premium. 25 Equity premium error (%) Relative risk aversion (c) Relative error. Figure 1: Consumption growth distribution and equity premium. Figure 1a shows the histogram of the log consumption growth distribution as well as the nonparametric kernel density estimator and the Gaussian distribution fitted by maximum likelihood. We can see that the histogram and the nonparametric density have slightly longer tail (expansions and recessions) than the Gaussian distribution. Figure 1b shows the log equity premium (3.1) for the two models. 5 With nonparametric shocks, the equity premium increases by 3 Mylecturenote athttps://sites.google.com/site/aatoda111/file-cabinet/272_l04.pdf contains a proof of (3.1). With i.i.d. endowment growth, it is easy to show that the log equity premium depends only on the risk aversion and it does not matter whether the utility is additive of recursive (Epstein-Zin) To make the results comparable, I discretize the Gaussian density using the Golub-Welsch algorithm with the corresponding Gaussian density, which is equivalent to the well-known 6

7 0.3 percentage points when γ = 10. Figure 1c shows the relative increase in the equity premium when using the nonparametric distribution, which ranges 16 24%. Thus using nonparametric distributions help to explain asset pricing puzzles. 3.2 Optimal portfolio problem Consider a CRRA investor with relative risk aversion γ > 0. Letting R > 0 be the gross stock return, R f > 0 be the gross risk-free rate, and θ be the fraction of wealth (portfolio share) invested in the stock, the investor s optimal portfolio problem is max θ 1 1 γ E[(Rθ +R f(1 θ)) 1 γ ]. (3.2) I obtain the annual data on U.S. nominal stock returns, risk-free rate, and inflation for the period from the spreadsheet of Amit Goyal. 6 For the stock returns I use the CRSP volume-weighted index including dividends. I convert these returns into real log returns and calibrate the log risk free rate as the sample average. The result is R f = I then apply Algorithm 2 to the log excess returns logr logr f to obtain a discrete distribution with nodes { x n } N n=1 and weights {w n} N n=1, where I choose the number of points N = 5 (increasing the number of points further does not change the results). The gross stock return in state n is defined by R n = R f e xn, which occurs with probability w n. Finally, I numerically solve the optimal portfolio problem (3.2). Figure 2 shows the results when we changethe relative risk aversionin the range γ [1,7]. Figure 2a shows the histogram of the log excess returns distribution as well as the nonparametric kernel density estimator and the Gaussian distribution fitted by maximum likelihood. We can see that the histogram and the nonparametric density have a long left tail corresponding to stock market crashes, which the Gaussian distribution misses. Figure 2b shows the optimal portfolio θ for the two models. We can see that when the investor incorrectly believes that the stock returns distribution is lognormal, he overweights the stock portfolio because he underestimates the probability of crashes. Figure 2c shows the percentage of this overweight(portfolioerror)θ G /θ NP 1, wheregand NPstandforGaussian and nonparametric densities. The portfolio error is substantial, in the range of 16 26%. 4 Concluding remarks This paper has proposed a simple, automatic method for discretizing a nonparametric distribution, given the data. Using an asset pricing model and an optimal portfolio problem as a laboratory, I showed that the error from using a parametric distribution (such as the Gaussian distribution) can be substantial. A natural extension is to consider the discretization of Markov processes with nonparametric shocks. For example, one may be tempted to apply the Gauss-Hermite quadrature. 6 The spreadsheet is at Using monthly or quarterly data give qualitatively similar results, though slightly less extreme quantitatively. 7

8 Probability density Histogram Nonparametric Gaussian Optimal portfolio Nonparametric Gaussian Log excess returns (a) Histogram and densities Relative risk aversion (b) Optimal portfolio. 25 Portfolio error (%) Relative risk aversion (c) Portfolio error. Figure 2: Excess returns distribution and numerical solution. kernel density estimation and Gaussian quadrature in the Tauchen and Hussey (1991) method to discretize the AR(1) process x t = ρx t 1 +ε t, where ρ < 1 and the innovations {ε t } t=0 are independent and identically distributed according to some probability density function f. However, it is well known that the Tauchen-Hussey method is not accurate when the persistence ρ is moderately high (Flodén, 2008). Using the AR(1) asset pricing model in the Online Appendix of Farmer and Toda (2017) to evaluate the solution accuracy, I found that the Gaussian quadrature-based methods for discretizing Markov processes is even less accurate when the shock distribution is nonparametric. Therefore for such processes, it is preferable to use the Farmer and Toda (2017) maximum entropy method with an even-spaced grid (see their Section for an example). Fortunately, since the inputs to the Farmer-Toda method are the probability density function and some low order moments (which we can compute exactly as in Section 2.2), the proposed method based on the kernel density estimator is still applicable. Finally, although I proposed my method as a tool for discretization, it can also be used as a quadrature method. For example, Pohl et al. (2018) solve the Bansal and Yaron (2004) long run risks model using the projection method and Gauss-Hermite quadrature, but that is because the model is assumed to have Gaussian shocks. If instead a researcher wishes to use nonparametric shocks, my method can be directly used to construct a quadrature rule from data. 8

9 References Ravi Bansal and Amir Yaron. Risks for the long run: A potential resolution of asset pricing puzzles. Journal of Finance, 59(4): , August doi: /j x. Robert J. Barro. Rare disasters and asset markets in the twentieth century. Quarterly Journal of Economics, 121(3): , doi: /qjec Leland E. Farmer and Alexis Akira Toda. Discretizing nonlinear, non-gaussian Markov processes with exact conditional moments. Quantitative Economics, 8(2): , July doi: /qe737. Martin Flodén. A note on the accuracy of Markov-chain approximations to highly persistent AR(1) processes. Economics Letters, 99(3): , June doi: /j.econlet Xavier Gabaix. Variable rare disasters: An exactly solved framework for ten puzzles in macro-finance. Quarterly Journal of Economics, 127(2): , doi: /qje/qjs001. Gene H. Golub and John H. Welsch. Calculation of Gauss quadrature rules. Mathematics of Computation, 23(106): , may doi: /s Robert E. Lucas, Jr. Asset prices in an exchange economy. Econometrica, 46 (6): , November doi: / Allen C. Miller, III and Thomas R. Rice. Discrete approximations of probability distributions. Management Science, 29(3): , March doi: /mnsc Walter Pohl, Karl Schmedders, and Ole Wilms. Higher-order effects in asset pricing models with long-run risks. Journal of Finance, doi: /jofi Thomas A. Rietz. The equity risk premium: A solution. Journal of Monetary Economics, 22(1): , July doi: / (88) K. Geert Rouwenhorst. Asset pricing implications of equilibrium business cycle models. In Thomas F. Cooley, editor, Frontiers of Business Cycle Research, chapter 10, pages Princeton University Press, Bernard W. Silverman. Density Estimation for Statistics and Data Analysis, volume 26 of Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, New York, Ken ichiro Tanaka and Alexis Akira Toda. Discrete approximations of continuous distributions by maximum entropy. Economics Letters, 118(3): , March doi: /j.econlet Ken ichiro Tanaka and Alexis Akira Toda. Discretizing distributions with exact moments: Error estimate and convergence analysis. SIAM Journal on Numerical Analysis, 53(5): , doi: /

10 George Tauchen. Finite state Markov-chain approximations to univariate and vector autoregressions. Economics Letters, 20(2): , doi: / (86) George Tauchen and Robert Hussey. Quadrature-based methods for obtaining approximate solutions to nonlinear asset pricing models. Econometrica, 59 (2): , March doi: / A Accuracy As in any numerical method, evaluating the accuracy is very important. In this section I evaluate the accuracy of the proposed method using the optimal portfolio problem in Section 3.2 as a laboratory. I design the numerical experiment as follows. First I fit a Gaussian mixture distribution with two components to the annual log excess returns data. The proportion, mean, and standard deviation of each mixture components are p = (p j ) = (0.1392,0.8608), µ = (µ j ) = ( ,0.1064), and σ = (σ j ) = (0.2164, ), respectively. Next, I assume that the true excess returns distribution is this Gaussian mixture and solve the optimal portfolio problem for relative risk aversion γ {2, 4, 6} using the Gaussian quadrature (Golub-Welsch algorithm 1) for Gaussian mixtures with 11 points. Finally, I generate random numbers from this Gaussian mixture with various sample sizes, discretize these distributions with various methods, and compute the optimal portfolio. I repeat this procedure with M = 1,000 Monte Carlo replications and compute the relative bias and mean absolute error (MAE) Bias = 1 M MAE = 1 M M m=1 ( θm /θ 1), (A.1a) M θ m /θ 1, (A.1b) m=1 where θ m is the optimal portfolio from simulation m and θ is the theoretical optimal portfolio. For the sample size I consider T = 100,1,000,10,000, and for the number of quadrature nodes I consider N = 3,5,7,9. For the discretization method I consider three cases. The first is the kernel density estimator-gaussian quadrature method (Algorithm 2), which I refer to as KDE-GQ. The second is the Gauss-Hermite quadrature, where the mean and standard deviation are estimated by maximum likelihood. This is the most natural method if the returns distribution is lognormal. The third is the maximum entropy method proposed by Tanaka and Toda (2013, 2015) and Farmer and Toda (2017) where the kernel density estimatoris fed into, which I referto as KDE-ME.For this method one needs to assign the grid and the number of moments to match. Following Corollary 3.5 of Farmer and Toda (2017), I use an even-spaced grid centered at the sample mean that spans 2(N 1) times the sample standard deviation at both sides, where N is the number of grid points. I match 4 moments whenever possible for N 5, and otherwise I match 2 moments (mean and variance). For more details on the exact algorithm, please refer to Tanaka and Toda (2013, 2015) and Sections 2 and 3.2 of Farmer and Toda (2017). 10

11 Tables 1 and 2 show the relative bias and mean absolute error of the optimal portfolio, respectively. As expected, the optimal portfolio computed using Gauss-Hermite is biased upwards because it uses the Gaussian distribution, which underestimates the probability of crashes. KDE-GQ and KDE-ME perform similarly for N 5 grid points, although KDE-GQ has slightly better small sample properties (T = 100). For N = 3 grid points, in which case it is impossible to match 4 moments with KDE-ME, the proposed KDE-GQ method performs better. Finally, increasing N beyond 5 does not improve the bias or the mean absolute error, which suggests that using a five-point distribution is enough (at least for solving this portfolio problem). Table 1: Relative bias of the optimal portfolio. Method KDE-GQ Gauss-Hermite KDE-ME T N γ = γ = γ = ,000 10, Note: the table reports the relative bias of the optimal portfolio defined by (A.1a). For discretization methods, KDE-GQ uses Algorithm 2, Gauss-Hermite uses the Gauss- Hermite quadrature (with mean and standard deviation estimated by maximum likelihood), and KDE-ME uses the maximum entropy method with the kernel density estimator. T is the sample size in each simulation. N is the number of nodes in the quadrature formula. γ is the relative risk aversion. All results are based on 1,000 Monte Carlo replications. B Gaussian quadrature (not for publication) In this appendix we prove some properties of the Gaussian quadrature. For notational simplicity let us omit a,b (so means b a ) and assume that w(x)x n dx exists for all n 0. For functions f,g, define the inner product (f,g) by (f,g) = b a w(x)f(x)g(x) dx. (B.1) As usual, define the norm of f by f = (f,f). The first step is to construct orthogonal polynomials {p n (x)} N n=0 corresponding to the inner product (B.1). Definition 1 (Orthogonalpolynomial). The polynomials {p n (x)} N n=0 are called orthogonal if (i) degp n = n and the leading coefficient of p n is 1, and (ii) for all m n, we have (p m,p n ) = 0. 11

12 Table 2: Relative mean absolute error of the optimal portfolio. Method KDE-GQ Gauss-Hermite KDE-ME T N γ = γ = γ = ,000 10, Note: the table reports the relative mean absolute error of the optimal portfolio defined by (A.1b). See Table 1 for the definition of variables. Some authors require that the polynomials are orthonormal, so (p n,p n ) = 1. In this paper we normalize the polynomials by requiring that the leading coefficient is 1, which is useful for computation. The following three-term recurrence relation (TTRR) shows the existence of orthogonal polynomials and provides an explicit algorithm for computing them. Proposition2(Three-termrecurrencerelation,TTRR). Let p 0 (x) = 1, p 1 (x) = x (xp0,p0), and for n 1 define p 0 2 p n+1 (x) = ( x (xp n,p n ) p n 2 ) p n (x) p n 2 Then p n (x) is the degree n orthogonal polynomial. p n 1 2p n 1(x). (B.2) Proof. Let us show by induction on n that (i) p n is an degree n polynomial with leading coefficient 1, and (ii) (p n,p m ) = 0 for all m < n. The claim is trivial for n = 0. For n = 1, by construction p 1 is a degree 1 polynomial with leading coefficient 1, and since p 0 (x) = 1, we obtain (p 1,p 0 ) = (( x (xp 0,p 0 ) p 0 2 )p 0,p 0 ) = (xp 0,p 0 ) (xp 0,p 0 ) = 0. Suppose the claim holds up to n. Then for n + 1, by (B.2) the leading coefficient of p n+1 is the same as that of xp n, which is 1. If m = n, then (( ) ) (p n+1,p n ) = x (xp n,p n ) p n 2 p n p n 2 p n 1 2p n 1,p n = (xp n,p n ) (xp n,p n ) p n 2 p n 1 2(p n 1,p n ) = 0. 12

13 If m = n 1, then (p n+1,p n 1 ) = (( x (xp n,p n ) p n 2 ) ) p n p n 2 p n 1 2p n 1,p n 1 = (xp n,p n 1 ) (xp n,p n ) p n 2 (p n,p n 1 ) p n 2 = (p n,xp n 1 ) p n 2. Since the leading coefficients of p n,p n 1 are 1, we can write xp n 1 (x) = p n (x)+ q(x), where q(x) is a polynomial of degree at most n 1. Clearly q can be expressed as a linear combination of p 0,p 1,...,p n 1, so (p n,q) = 0. Therefore (p n+1,p n 1 ) = (p n,p n +q) p n 2 = p n 2 +(p n,q) p n 2 = 0. Finally, if m < n 1, then (( (p n+1,p m ) = x (xp n,p n ) p n 2 ) ) p n p n 2 p n 1 2p n 1,p m = (xp n,p m ) (xp n,p n ) p n 2 (p n,p m ) p 2 n p n 1 2(p n 1,p m ) = (p n,xp m ) = 0 because xp m is a polynomial of degree 1+m < n. The following lemma shows that an degree n orthogonal polynomial has exactly n real roots (so they are all simple). Lemma 3. p n (x) has exactly n real roots on (a,b). Proof. By the fundamental theorem of algebra, p n (x) has exactly n roots in C. Suppose on the contrary that p n (x) has less than n real roots on (a,b). Let x 1,...,x k (k < n) those roots at which p n (x) changes its sign. Let q(x) = (x x 1 ) (x x k ). Since p n (x)q(x) > 0 (or < 0) almost everywhere on (a,b), we have (p n,q) = w(x)p n (x)q(x)dx 0. On the other hand, since degq = k < n, we have (p n,q) = 0, which is a contradiction. The following theorem shows that using the N roots of the degree N orthogonal polynomial p N (x) as quadrature nodes and choosing specific weights, we can integrate all polynomials of degree up to 2N 1 exactly. Thus Gaussian quadrature always exists. Theorem 4 (Gaussian quadrature). Let a < x 1 < < x N < b be the N roots of the degree N orthogonal polynomial p N and define w n = w(x)l n (x)dx 13

14 for n = 1,...,N, where L n (x) = m n x x m x n x m is the degree N 1 polynomial that takes value 1 at x n and 0 at x m (m {1,...,N}\n). Then w(x)p(x) dx = w n p(x n ) n=1 for all polynomials p(x) of degree up to 2N 1. Proof. Since degp 2N 1 and degp N = N, we can write p(x) = p N (x)q(x)+r(x), (B.3) where degq,degr N 1. Since q can be expressed as a linear combination of orthogonal polynomials of degree up to N 1, we have (p N,q) = 0. Hence w(x)p(x)dx = (p N,q)+ w(x)r(x) dx = w(x)r(x) dx. On the other hand, since {x n } N n=1 are roots of p N, we have for all n, so in particular p(x n ) = p N (x n )q(x n )+r(x n ) = r(x n ) w n p(x n ) = n=1 w n r(x n ). Therefore it suffices to show (B.3) for polynomials r of degree up to N 1. Let us show that r(x) = r(x n )L n (x) n=1 identically. To see this, let r be the right-hand side. Since L n (x m ) = δ mn (Kronecker s delta), we have r(x m ) = r(x n )L n (x m ) = n=1 n=1 δ mn r(x n ) = r(x m ), so r and r agree on N distinct points {x n } N n=1. Since each L n(x) is a degree N 1 polynomial, we have deg r N 1. Therefore it must be r = r. Since r can be represented as a linear combination of L n s, it suffices to show (B.3) for all L n s. But since by definition the claim is true. w(x)l n (x)dx = w n = n=1 w m δ mn = m=1 w m L n (x m ), m=1 14

15 In practice, how can we compute the nodes {x n } N n=1 and weights {w n} N n=1 of the N-point Gaussian quadrature? The solution is given by the following Golub-Welsch algorithm. Theorem 5 (Golub and Welsch, 1969). For each n 1, define α n,β n by α n = (xp n 1,p n 1 ) p n 1 2, β n = p n p n 1 > 0. Define the N N symmetric tridiagonal matrix T N as in (2.2). Then the Gaussian quadrature nodes {x n } N n=1 are eigenvalues of T N. Letting v n = (v n1,...,v nn ) be an eigenvector of T N corresponding to eigenvalue x n, then the weights {w n } N n=1 are given by w n = v2 n1 v n 2 w(x)dx > 0. (B.4) Proof. By (B.2) and the definition of α n,β n, for all n 0 we have p n+1 (x) = (x α n+1 )p n (x) β 2 np n 1 (x). Note that this is true for n = 0 by defining p 1 (x) = 0 and β 0 = 0. For each n, let p n(x) = p n (x)/ p n be the normalized orthogonal polynomial. Then the above equation becomes p n+1 p n+1 (x) = p n (x α n+1 )p n (x) p n 1 β 2 n p n 1 (x). Dividing both sides by p n > 0, using the definition of β n,β n+1, and rearranging terms, we obtain β n p n 1 (x)+α n+1p n (x)+β n+1p n+1 (x) = xp n (x). In particular, setting x = x k (where x k is a root of p N ), we obtain β n p n 1(x k )+α n+1 p n(x k )+β n+1 p n+1(x k ) = x k p n(x k ). for all n and k = 1,...,N. Since β 0 = 0 by definition and p N (x k) = 0 (since x k is a root of p N and hence p N = p N/ p N ), letting P(x) = (p 0(x),...,p N 1 (x)) and collecting the above equation into a vector, we obtain T N P(x k ) = x k P(x k ) for k = 1,...,N. Define the N N matrix P by P = (P(x 1 ),...,P(x N )). Then T N P = diag(x 1,...,x N )P, so x 1,...,x N are eigenvalues of T N provided that P is invertible. Now since {p n} N 1 n=0 are normalized and Gaussian quadrature integrates all polynomials of degree up to 2N 1 exactly, we have δ mn = (p m,p n ) = N w(x)p m (x)p n (x)dx = w k p m (x k)p n (x k) for m,n N 1. Letting W = diag(w 1,...,w N ), this equation becomes PWP = I. Therefore P,W are invertible and x 1,...,x N are eigenvalues of T N. Solving for W and taking the inverse, we obtain k=1 W 1 = P P 1 N 1 = p k w (x n) 2 > 0 n 15 k=0

16 for all n. To show (B.4), let v n be an eigenvector of T N corresponding to eigenvalue x n. Then v n = cp(x n ) for some constant c 0. Taking the norm, we obtain N 1 v n 2 = c 2 P(x n ) 2 = c 2 k=0 p k(x n ) 2 = c2 w n w n = c2 v n 2. Comparing the first element of v n = cp(x n ), noting that p 0 (x) = 1 and hence p 0 = p 0/ p 0 = 1/ p 0, we obtain c 2 = vn1 2 p 0 2 = vn1 2 w(x)p 0 (x) 2 dx = vn1 2 w(x) dx, which implies (B.4). 16