American Society for Quality

Transcription

1 American Society for Quality Ridge Regression: Biased Estimation for Nonorthogonal Problems Author(s): Arthur E. Hoerl and Robert W. Kennard Source: Technometrics, Vol. 42, No. 1, Secial 40th Anniversary Issue (Feb., 2000), Published by: American Statistical Association and American Society for Quality Stable URL: htt:// Accessed: 24/10/ :25 Your use of the JSTOR archive indicates your accetance of JSTOR's Terms and Conditions of Use, available at htt:// JSTOR's Terms and Conditions of Use rovides, in art, that unless you have obtained rior ermission, you may not download an entire issue of a journal or multile coies of articles, and you may use content in the JSTOR archive only for your ersonal, non-commercial use. Please contact the ublisher regarding any further use of this work. Publisher contact information may be obtained at htt:// Each coy of any art of a JSTOR transmission must contain the same coyright notice that aears on the screen or rinted age of such transmission. JSTOR is a not-for-rofit organization founded in 1995 to build trusted digital archives for scholarshi. We work with the scholarly community to reserve their work and the materials they rely uon, and to build a common research latform that romotes the discovery and use of these resources. For more information about JSTOR, lease contact suort@jstor.org. American Statistical Association and American Society for Quality are collaborating with JSTOR to digitize, reserve and extend access to Technometrics. htt://

2 Ridge Regression: Biased Estimation for Nonorthogonal Problems Arthur E. HOERL* and Robert W. KENNARD* University of Delaware and E. I. du Pont de Nemours & Co. (Deceased 1994) (2632 Horseshoe Court, Cocoa, FL 32926) In multile regression it is shown that arameter estimates based on minimum residual sum of squares have a high robability of being unsatisfactory, if not incorrect, if the rediction vectors are not orthogonal. Proosed is an estimation rocedure based on adding small ositive quantities to the diagonal of X'X. Introduced is the ridge trace, a method for showing in two dimensions the effects of nonorthogonality. It is then shown how to augment X'X to obtain biased estimates with smaller mean square error. 0. INTRODUCTION Consider the standard model for multile linear regression, Y = X/3 + E, where it is assumed that X is (n x ) and of rank,/3 is ( x 1) and unknown, E[e] = 0, and E[se'] = u2in. If an observation on the factors is denoted by x, = {xi,,x22,...,zv}, the general form X/3 is { ip_1 3iOi(x^)} where the Oi are functions free of unknown arameters. The usual estimation rocedure for the unknown 3 is Gauss-Markov-linear functions of Y = {y,} that are unbiased and have minimum variance. This estimation rocedure is a good one if X'X, when in the form of a correlation matrix, is nearly a unit matrix. However, if X'X is not nearly a unit matrix, the least squares estimates are sensitive to a number of "errors." The results of these errors are critical when the secification is that X/3 is a true model. Then the least squares estimates often do not make sense when ut into the context of the hysics, chemistry, and engineering of the rocess which is generating the data. In such cases, one is forced to treat the estimated redicting function as a black box or to dro factors to destroy the correlation bonds among the Xi used to form X'X. Both these alternatives are unsatisfactory if the original intent was to use the estimated redictor for control and otimization. If one treats the result as a black box, he must caution the user of the model not to take artial derivatives (a useless caution in ractice), and in the other case, he is left with a set of dangling controllables or observables. Estimation based on the matrix [X'X + ki], k > 0 rather than on X'X has been found to be a rocedure that can be used to hel circumvent many of the difficulties associated with the usual least squares estimates. In articular, the rocedure can be used to ortray the sensitivity of the estimates to the articular set of data being used, and it can be used to obtain a oint estimate with a smaller mean square error. 1. PROPERTIES OF BEST LINEAR UNBIASED ESTIMATION Using unbiased linear estimation with minimum variance or maximum likelihood estimation when the random vector, e, is normal gives 3 -(X'X) 1X'Y (1.1) as an estimate of /3 and this gives the minimum sum of squares of the residuals: 03) =- (Y- X/)'(Y - X/). (1.2) The roerties of P are well known (Scott 1966). Here the concern is rimarily with cases for which X'X is not nearly a unit matrix (unless secified otherwise, the model is formulated to give an X'X in correlation form). To demonstrate the effects of this condition on the estimation of,3, consider two roerties of 13-its variance-covariance matrix and its distance from its exected value. (i) VAR(3) = c2(x'x)-1 (1.3) (ii) L1 = Distance from 3 to 3. or equivalently L =- (O - )'(P - 3) (1.4) E[L2] = r2trace(x'x)-1 E[/'P] = /3' + a2trace(x'x)-l When the error e is normally distributed, then VAR[L2] = 2cr4Trace(X'X)-2. (1.5) (1.5a) (1.6) These related roerties show the uncertainty in / when X'X moves from a unit matrix to an ill-conditioned one. If the eigenvalues of X'X are denoted by Amax = 1 > 2 > *' > \ = Amin > 0, (1.7) then the average value of the squared distance from / to /3 is given by E[LI2] = a2 (1/Ai) i=l (1.8)? 1970 American Statistical Association and the American Society for Quality 80

3 and the variance when the error is normal is given by VAR[L2] = 2-4 (1/Ai)2. Lower bounds for the average and variance are a2/amin and 2o4/A i,, resectively. Hence, if the shae of the factor sace is such that reasonable data collection results in an X'X with one or more small eigenvalues, the distance from /3 to,3 will tend to be large. Estimated coefficients, f3i, that are large in absolute value have been observed by all who have tackled live nonorthogonal data roblems. The least squares estimate (1.1) suffers from the deficiency of mathematical otimization techniques that give oint estimates; the estimation rocedure does not have built into it a method for ortraying the sensitivity of the solution (1.1) to the otimization criterion (1.2). The rocedures to be discussed in the sections to follow ortray the sensitivity of the solutions and utilize nonsensitivity as an aid to analysis. 2. RIDGE REGRESSION A. E. Hoerl first suggested in 1962 (Hoerl 1962; Hoerl and Kennard 1968) that to control the inflation and general instability associated with the least squares estimates, one can use 3* = [X'X + kj]-x'y; k > (2.1) = WX'Y. (2.2) The family of estimates given by k > 0 has many mathematical similarities with the ortrayal of quadratic resonse functions (Hoerl 1964). For this reason, estimation and analysis built around (2.1) has been labeled "ridge regression." The relationshi of a ridge estimate to an ordinary estimate is given by the alternative form 3* [I + k(x'x)-1]-1 (2.3) Z3. (2.4) This relationshi will be exlored further in subsequent sections. Some roerties of 3*, W, and Z that will be used are: (i) Let (i(w) and (i(z) be the eigenvalues of W and Z, resectively. Then (i(w) = 1/(A, + k) (2.5) i(z) = Ai/(A, + k) (2.6) where Ai are the eigenvalues of X'X. These results follow directly from the definitions of W and Z in (2.2) and (2.4) and the solution of the characteristic equations IW--I[ = 0 and IZ - II = 0. (ii) Z = I- k(x'x + ki)-1 = I- kw (2.7) The relationshi is readily verified by writing Z in the alternative form Z = (X'X + ki)- X'X = WX'X and multilying both sides of (2.7) on the left by W-1. (iii) /* for k y 0 is shorter than /3, i.e., RIDGE REGRESSION 81 By definition /* = Z/. From its definition and the assumtions on X'X, Z is clearly symmetric ositive definite. Then (1.9) the following relation holds (Sheff6 1960): (d*)'(3*) _< ax(z)i'2. (2.9) But ~max(z) = A1/(A1 + k) where A1 is the largest eigenvalue of X'X and (2.8) is established. From (2.6) and (2.7) it is seen that Z(0) = I and that Z aroaches 0 as k -+ oo. For an estimate /* the residual sum of squares is b*(k) = (Y - X3*)'(Y - X/*) (2.10) which can be written in the form 0*(k) = Y'Y - (d*)'x'y - k(3*)'(3*). (2.11) The exression shows that * (k) is the total sum of squares less the "regression" sum of squares for /3* with a modification deending uon the squared length of 3*. 3. THE RIDGE TRACE a. Definition of the Ridge Trace When X'X deviates considerably from a unit matrix, that is, when it has small eigenvalues, (1.5) and (1.6) show that the robability can be small that 3 will be close to /3. In any excet the smallest roblems, it is difficult to untangle the relationshis among the factors if one is confined to an insection of the simle correlations that are the elements of X'X. That such untangling is a roblem is reflected in the "automatic" rocedures that have been ut forward to reduce the dimensionality of the factor sace or to select some "best" subset of the redictors. These automatic rocedures include regression using the factors obtained from a coordinate transformation using the rincial comonents of X'X, stewise regression, comutation of all 2P regressions, and some subset of all regressions using fractional factorials or a branch and bound technique (Beale, Kendall, and Mann 1967; Efroyson 1960; Garside 1965; Gorman and Toman 1966; Hocking and Leslie 1967; Jeffers 1967; Scott 1966). However, with the occasional excetion of rincial comonents, these methods don't really give an insight into the structure of the factor sace and the sensitivity of the results to the articular set of data at hand. But by comut- ing /*(k) and O*(k) for a set of values of k, such insight can be obtained. A detailed study of two nonorthogonal roblems and the conclusions that can be drawn from their ridge traces is given in James and Stein (1961). b. Characterization of the Ridge Trace Let B be any estimate of the vector /. Then the residual sums of squares can be written as =(Y - XB)'(Y- XB) =(Y - X3)'(Y - X3) + (B -/)'X'X(B - 3) = >min +?(B) (3.1).on-to o.a I a Contours oft constant 0 are the surfaces of hyerellisoids (2.8) centered at /, the ordinary least squares estimate of /3. The

4 82 ARTHUR E. HOERL AND ROBERT W. KENNARD value of 0 is the minimum value,?mi, lus the value of the quadratic form in (B - /). There is a continuum of values of Bo that will satisfy the relationshi q = Omin + -o where 00 > 0 is a fixed increment. However, the relationshis in Section 2 show that on the average the distance from to 3 will tend to be large if there is a small eigenvalue of X'X. In articular, the worse the conditioning of X'X, the more 3 can be exected to be too long. On the other hand, the worse the conditioning, the further one can move from,3 without an areciable increase in the residual sums of squares. In view of (1.5a) it seems reasonable that if one moves away from the minimum sum of squares oint, the movement should be in a direction which will shorten the length of the regression vector. The ridge trace can be shown to be following a ath through the sums of squares surface so that for a fixed X a single value of B is chosen and that is the one with minimum length. This can be stated recisely as follows: Minimize B'B subject to (B - /)'X'X(B - 0) = 0. (3.2) As a Lagrangian roblem this is Minimize F = B'B + (1/k)[(B - 3)'X'X(B - )' - 0o] where (1/k) is the multilier. Then OF This reduces to (3.3) = 2B + (1/k)[2(X'X)B - 2(X'X)3] = 0 (3.4) B = /* = [X'X + ki]-x'y (3.5) where k is chosen to satisfy the restraint (3.2). This is the ridge estimator. Of course, in ractice it is easier to choose a k > 0 and then comute o0. In terms of 3* the residual sum of squares becomes 0*(k) = (Y- X/*)'(Y - XP*) =- min + k2 3*/(XIX)-1l*. (3.6) A comletely equivalent statement of the ath is this: If the squared length of the regression vector B is fixed at R2, then /3* is the value of B that gives a minimum sum of squares. That is, /3* is the value of B that minimizes the function F1 = (Y - XB)'(Y - XB) + (1/k)(B'B - R2). (3.7) With (3.1) in 3b, this shows that an increase in the residual sum of squares is equivalent to a decrease in the value of the likelihood function. So the contours of equal likelihood also lie on the surface of hyerellisoids centered at 3. The ridge trace can thereby be interreted as a ath through the likelihood sace, and the question arises as why this articular ath can be of secial interest. The reasoning is the same as for the sum of squares. Although long vectors give the same likelihood values as shorter vectors, they will not always have equal hysical meaning. Imlied is a restraint on the ossible values of /3 that is not made exlicit in the formulation of the general linear model given in the Introduction. This imlication is discussed further in the sections that follow. 4. MEAN SQUARE ERROR PROPERTIES OF RIDGE REGRESSION a. Variance and Bias of a Ridge Estimator To look at 3* from the oint of view of mean square error it is necessary to obtain an exression for E[L2(k)]. Straightforward alication of the exectation oerator and (2.3) gives the following: E[L 2(k)] =E[(* - -)'(* - )] = E[(/ - 3)'Z'Z(/ -/3)] + (Z/3-3)'(Z/3 -/3) (4.2) = -2 Trace(X'X)-lZ'Z + /(Z - I)'(Z - I)P/ (4.3) = 02[Trace(X'X + ki)-1 - k Trace(X'X + ki)-2] = + k2/3'(x'x + ki)-23 f2 Z Ai/(Ai + k)2 + k2f3(x'x + ki)-2/ 1 = 1 (k) + 2(k) (4.4) (4.5) (4.6) The meanings of the two elements of the decomosition, 7yi(k) and 72(k), are readily established. The second element, 72(k), is the squared distance from Z/( to 3. It will be zero when k = 0, since Z is then equal to I. Thus, 72 (k) can be considered the square of a bias introduced when /* is used rather than /. The first term, 71 (k), can be shown to be the sum of the variances (total variance) of the arameter estimates. In terms of the random variable Y, c. Likelihood Characterization of the Ridge Trace Using the assumtion that the error vector is Normal (0, a2in) the likelihood function is (27ru2)-n/2 ex{-(1/2cr2)(y - X/)'(Y - X/3)}. (3.8) The kernel of this function is the quadratic form in the exonential which can be written in the form (Y - X3)'(Y - X3) = (Y - X/)'(Y - X3) + (/ - f/)'x'x(3 - /3). (3.9) Then /* = Z/3 = Z(X'X) 1X'Y. (4.7) VAR[3*] = Z(X'X)-1X'VAR[Y]X(X'X)-1Z' = r2z(x'x)-lz. (4.8) The sum of the variances of all the /* is the sum of the diagonal elements of (4.8).

5 RIDGE REGRESSION 83 Figure 1 shows in qualitative form the relationshi between the variances, the squared bias, and the arameter k. The total variance decreases as k increases, while the squared bias increases with k. As is indicated by the dot- ted line, which is the sum of 71 (k) and Y2 (k) and thus is E[L2(k)], the ossibility exists that there are values of k (admissible values) for which the mean square error is less for 3* than it is for the usual solution 3. This ossibility is suorted by the mathematical roerties of 71(k) and 72(k). [See Section 4b.] The function 71 (k) is a monotonic decreasing function of k, while 7Y2(k) is monotonic increasing. However, the most significant feature is the value of the derivative of each function in the neighborhood of the origin. These derivatives are: Lim(dTy/dk) - -2a2E(1/A2) (4.9) k-o+ b. Theorems on the Mean Square Function Theorem 4.1. The total variance y71(k) is a continuous, monotonically decreasing function of k. Corollary The first derivative with resect to k of the total variance 7 (k), aroaches -oc as k and A -X0. Both the theorem and the corollary are readily roved by use of lyi(k) and its derivative exressed in terms of Ai. Theorem 4.2. The squared bias 72(k) is a continuous, monotonically increasing function of k. Proof From (4.5) Y2(k) = k2/'(x'x + ki)-2/3. Corollary The first derivative of the total variance, y1 (k), aroaches -oo as k and the matrix X'X becomes singular. Both the theorem and the corollary are readily roved by use of 7 (k) and its derivative exressed in terms of Ai. Theorem 4.2. The squared bias 72(k) monotonically increasing function of k. is a continuous, Lim (d72/dk) - 0. (4.10) k-o+ Proof: From (4.5) Y22(k) = k2,'(x'x + ki)-2/3. If A is the matrix of eigenvalues of X'X and P the orthogonal transformation such that X'X = P'AP, then Thus, -yi(k) has a negative derivative which aroaches -2r2 as k for an orthogonal X'X and aroaches -oo as X'X becomes ill-conditioned and A -- 2 (k) = k2 Ca2/(Xi + k)2 (4.11) 0. On the 1 other hand, as k -X 0+, (4.10) shows that 72(k) is flat and zero at the origin. These roerties lead to the conclusion that it is ossible to move to k where a > 0, take a little P/3. (4.12) bias, and substantially reduce the variance, thereby imroving Since Ai > 0 for all i and k > 0, each element (Ai + k) is the mean square error of estimation and rediction. An ex- ositive and there are no singularities in the sum. Clearly, istence theorem to validate this conclusion is given in Sec- 72(0) = 0. Then 72(k) is a continuous function for k > 0. tion 4b. For k > 0 (4.11) can be written as "Y2(k)- E oz2/[1 + (Ai/k) ]2. (4.13) Since Ai > 0 for all i, the functions Ai/k are clearly monotone decreasing for increasing k and each term of 'y2(k) is monotone increasing. So 72(k) is monotone increasing. q.e.d. f1 Corollary The squared bias 72(k) aroaches /3't as an uer limit. Proof: From (4.13) limk-ooy72(k) /3'P'P/3 = '/3 q.e.d. 1Zai - = a-a I- i t n Corollary The derivative 72 (k) aroaches zero as k -X 0+. Proof From (4.11) it is readily established that d'y2(k)/dk = 2k Aia2/(Ai + k)3. (4.14) k Figure 1. Each term in the sum 2kAia2/(Ai + k)3 is a continuous function. And the limit of each term as k -* 0+ is zero. q.e.d. Theorem 4.3. (Existence Theorem) There always exists a k > 0 such that E[L2(k)] < E[L2(0)] = C2 P (1/Ai).

6 84 ARTHUR E. HOERL AND ROBERT W. KENNARD Proof. From (4.5), (4.11), and (4.14) de[li(k)]/dk - dy(k)/dk + d2(k)/dk -2a2 Ai/(A, + k)3 + 2k A Ai?a2/(Ai + k)3. (4.15) First note that - 3y1(0) u2e(1/ai) and 72(0) = 0. In Theorems 4.1 and 4.2 it was established that -y1(k) and Y2 (k) are monotonically decreasing and increasing, resectively. Their first derivatives are always non-ositive and non-negative, resectively. Thus, to rove the theorem, it is only necessary to show that there always exists a k > 0 such that de[l2(k)]/dk < 0. The condition for this is shown by (4.15) to be: k < O /amax q.e.d. (4.16) c. Some Comments On The Mean Square Error Function The roerties of E[L2(k)] -=' (k) + Ty2(k) show that it will go through a minimum. And since 72(k) aroaches 3'/3 as a limit as k -+ oo, this minimum will move toward k = 0 as the magnitude of /3' increases. Since 3'/3 is the squared length of the unknown regression vector, it would aear to be imossible to choose a value of k 0 and thus achieve a smaller mean square error without being able to assign an uer bound to /3'3. On the other hand, it is clear that /3' does not become infinite in ractice, and one should be able to find a value or values for k that will ut /3* closer to /3 than is /. In other words, unboundedness, in the strict mathematical sense, and ractical unboundedness are two different things. In Section 7 some recommendations for choosing a k > 0 are given, and the imlicit assumtions of boundedness are exlored further. 5. A GENERAL FORM OF RIDGE REGRESSION It is always ossible to reduce the general linear regression roblem as defined in the Introduction to a canonical form in which the X'X matrix is diagonal. In articular there exists an orthogonal transformation P such that X'X = P'AP where A = (6ijAi) is the matrix of eigenvalues of X'X. Let and where X= X*P (5.1) Y = X*t + e (5.2) ac = PO, (X*)/(X*) = A, and a'a = '3. (5.3) Then the general ridge estimation rocedure is defined from where c* = [(X*)'(X*) + K] -(X*)'Y (5.4) K = (ijki), ki > 0. All the basic results given in Section 4 can be shown to hold for this more general formulation. Most imortant is that there is an equivalent to the existence theorem, Theorem 4.3. In the general form; one seeks a ki for each canonical variate defined by X*. By defining (L)2 = (&* - )'(* - a) it can be shown that the otimal values for the ki will be ki = a2/a2. There is no grahical equivalent to the RIDGE TRACE but an iterative rocedure initiated at ki = a 2/a2 can be used. (See Section 7) 6. RELATION TO OTHER WORK IN REGRESSION Ridge regression has oints of contact with other aroaches to regression analysis and to work with the same objective. Three should be mentioned. * In a series of aers, Stein (1960, 1962) and James and Stein (1961) investigated the imrovement in mean square error by a transformation on /3 of the form C3, 0 < C < 1, which is a shortening of the vector 3. They show that such a C > 0 can always be found and indicate how it might be comuted. * A Bayesian aroach to regression can be found in Jeffreys (1961) and Raiffa and Schlaifer (1961). Viewed in this context, each ridge estimate can be considered as the osterior mean based on giving the regression coefficients, /, a rior normal distribution with mean zero and variance-covariance matrix E = (6ij62/k). For those that do not like the hilosohical imlications of assuming P to be a random variable, all this is equivalent to constrained estimation by a nonuniform weighting on the values of 3. * Constrained estimation in a context related to regression can be found in Balakrishnan (1963). For the model in the resent aer, let /3 be constrained to be in a closed, bounded convex set C, and, in articular, let C be a hyershere of radius R. Let the estimation criterion be minimum residual sum of squares? = (Y-XB)'(Y-XB) where B is the value giving the minimum. Under the constraint, if /3'3 < R2, than B is chosen to be /3; otherwise B is chosen to be 3* where k is chosen so that (/3*)'(*) R2. 7. SELECTING A BETTER ESTIMATE OF / In Section 2 and in the examle of Section 3, it has been demonstrated that the ordinary least squares estimate of the regression vector 3 suffers from a number of deficiencies when X'X does not have a uniform eigenvalue sectrum. A class of biased estimators /*, obtained by augmenting the diagonal of X'X with small ositive quantities, has been introduced both to ortray the sensitivity of the solution to X'X and to form the basis for obtaining an estimate of /3 with a smaller mean square error. In examining the roerties of /3*, it can be shown that its use is equivalent to making certain boundedness assumtions regarding either the individual coordinates of /3 or its squared length, /3'3. As Barnard (1963) has recently ointed out, an alternative to unbiasedness in the logic of the least squares estimator /3 is the rior assurance of bounded mean square error with no boundedness assumtion on /3. If it is ossible to make secific mathematical assumtions about /3, then it is ossible to constrain the estimation rocedure to reflect these assumtions.

7 RIDGE REGRESSION 85 The inherent boundedness assumtions in using 3* make estimation can be continued until there is a stability it clear that it will not be ossible to construct a clear-cut, achieved in (a*)'(a*) and kio -= 2/(&o)2. automatic estimation rocedure to roduce a oint estimate (a single value of k or a secific value for each ki) as can be 8. CONCLUSIONS constructed to roduce 3. However, this is no drawback to It has been shown that when X'X is such that it its use has because with any given set of data it is not difficult a nonuniform to select a 3* that is better than /. In fact, ut in eigenvalue sectrum, the estimates of 3 in context, Y = X3 + e, based on the criterion of any set of data which minimum is a candidate residfor analysis using lin- ual sum of ear regression has imlicit in it restrictions on the squares, can have a high robability of being ossible far removed from 3. values This of the estimates that can be consistent with known unsatisfactory condition manifests itself in estimates that roerties of the data are too generator. Yet it is difficult to be exlarge in absolute value and some licit about these restrictions; it is esecially difficult to be may even have the wrong sign. By adding a small mathematically exlicit. In a recent ositive quantity to each aer (Clutton-Brock diagonal element the system 1965) it has been shown that for the roblem of [X'X + K]/* = X'Y acts more like an orthogonal system. estimating When K = the ki and mean /u of all a distribution, a solutions set of in data the has in it interval 0 < k < 1 imlicit are restrictions on the values of a that can be obtained, it is logical contenders ossible to obtain a two-dimensional charas acterization of the generators. Of course, in linear regression the roblem is system and a ortrayal of the kinds of much more difficulties caused difficult; the number of ossibilities is so by the intercorrelations among the relarge. dictors. First, there A is the number of arameters involved. To have study of the roerties of the estimator /3* shows ten to that it can be used to twenty regression coefficients is not uncommon. And imrove the mean square error of estheir signs have to be considered. Then there is X'X timation, and the and the magnitude of this imrovement increases with (2) different an factor increase correlations in and the ways in which sread of the they eigenvalue sectrum. An can be related. estimate Yet in based the on final analysis these many differ- /3* is biased and the use of a biased esent influences can timator be integrated to make an assessment as imlies some rior bound on the regression vector to whether the estimated values are consistent with the data /3. However, the data in any articular roblem has inforand the roerties of mation in the it data that can show the class generator. Guiding one of generators 3 that are along the way, of reasonable. course, is The the objective of the study. In Hoerl urose of the ridge trace is to ortray this and Kennard (1970) it is information shown for two roblems how such exlicitly and, hence, guide the user to a better estimate /*. an assessment can be made. Based on exerience, the best method for achieving a better estimate /3* is to use ki = k for all i and use the Ridge Trace to select a single value of k and a unique 3*. These kinds of things can be used to guide one to a choice. * At a certain value of k the system will stabilize and have the general characteristics of an orthogonal system. * Coefficients will not have unreasonable absolute values with resect to the factors for which they reresent rates of change. * Coefficients with aarently incorrect signs at k = 0 will have changed to have the roer sign. * The residual sum of squares will not have been inflated to an unreasonable value. It will not be large relative to the minimum residual sum of squares or large relative to what would be a reasonable variance for the rocess generating the data. Another aroach is to use estimates of the otimum values of ki develoed in Section 5. A tyical aroach here would be as follows: * Reduce the system to canonical by the transformations X = X*P and ca = P/. * Determine estimates of the otimum ki's using kio = &2/&2. Use the kio to obtain /*. * The ki0 will tend to be too small because of the tendency to overestimate a'a. Since use of the kio will shorten the length of the estimated regression vector, kio can be re-estimated using the 6&. This re- NOMENCLATURE 3= (X'X)- X'Y 3* = f*(k) = [X'X + ki]-1x'y; k > 0 W = W(k) = [X'X + ki]-1 = I - kw Z = Z(k)= [I + k(x'x)-1]- Ai = Eigenvalue of X'X; A1 > A2 >. > A > 0 A = (SijAi) = the matrix of eigenvalues P = An orthogonal matrix such that P'AP = X'X L (k) = E[(3)* - )'( *-)] = (k) + 72(k) 71 (k) = Variance of the estimate * 72(k) = Squared bias of the estimate 3* K = (&iki); ki > \- t - xj / I - negative constants. = PP X* = XP' A* = [(X*)'(X*) + K] -(X*)'Y W* = [(X*)'(x*) + K]-1 Z* = {I + [(X*)'(X*)]-1K]} = I- KW 0 A diagonal matrix of nnn- vt _-...L...1,.t1 t./1 111~./11 [Received August Revised June 1969.] REFERENCES Balakrishnan, A. V. (1963), "An Oerator Theoretic Formulation of a Class of Control Problems and a Steeest Descent Method of Solution," Journal on Control, 1, Barnard, G. A. (1963), "The Logic of Least Squares," Journal of the Royal Statistical Society, Ser. B, 65, Beale, E. M. L., Kendall, M. G., and Mann, D. W. (1967), "The Discarding of Variables in Multivariate Analysis," Biometrika, 54,

8 86 ARTHUR E. HOERL AND ROBERT W. KENNARD Clutton-Brock, M. (1965), "Using the Observations to Estimate Prior Distribution," Journal of the Royal Statistical Society, Ser. B, 27, Efroymson, M. A. (1960), "Multile Regression Analysis," in Mathematical Methodsfor Digital Comuters, eds. A. Ralston and H. S. Wilf, New York: Wiley, cha. 17. Garside, M. J. (1965), "The Best Subset in Multile Regression Analysis," Alied Statistics, 14. Gorman, J. W., and Toman, R. J. (1966), "Selection of Variables for Fitting Equations to Data," Technometrics, 8, Hocking, R. R., and Leslie, R. N. (1967), "Selection of the Best Subset in Regression Analysis," Technometrics, 9, Hoerl, A. E. (1962), "Alication of Ridge Analysis to Regression Problems," Chemical Engineering Progress, 58, (1965), "Ridge Analysis," Chemical Engineering Progress Symosium Series, 60, Hoerl, A. E., and Kennard, R. W. (1968), "On Regression Analysis and Biased Estimation," Technometrics, 10, (1970), "Ridge Regression. Alications to Non-orthogonal Problems," Technometrics, 12. James, W., and Stein, C. M. (1961), "Estimation With Quadratic Loss," Proceedings of the 4th Berkeley Symosium, 1, Jeffers, J. N. R. (1967), "Two Case Studies in the Alication of Princial Comonent Analysis," Alied Statistics, 16, Jeffreys, H. (1961), Theory of Probability (3rd ed.), London: Oxford University Press. Raiffa, H., and Schlaifer, R. (1961), Alied Statistical Decision Theory, Boston: Harvard University Press. Riley, J. D. (1955), "Solving Systems of Linear Equations With a Positive Definite, Symmetric, but Possibly Ill-conditioned Matrix," Mathematics of Comutation, 9, Scheffe, H. (1960), The Analysis of Variance, New York: Wiley. Scott, J. T., Jr. (1966), "Factor Analysis and Regression," Econometrica, 34, Stein, C. M. (1960), "Multile Regression," in Essays in Honor of Harold Hotelling, Stanford, CA: Stanford University Press, cha. 37. (1962), "Confidence Sets for the Mean of a Multivariate Normal Distribution," Journal of the Royal Statistical Society, Ser. B, 24,