WEIGHTED LEAST SQUARES - used to give more emphasis to selected poits i the aalysis What are eighted least squares?! " i=1 i=1 Recall, i OLS e miimize Q =! % =!(Y - " - " X ) or Q = (Y_ - X "_) (Y_ - X "_) I eighted least squares, e miimize Q = DDe œ DDcY Y b" X ) d The ormal equatios become b! D + b" DX = DY! " i bdx + bdx = DXY For the itermediate calculatios e get (here is the eight) t D D X, DD Y, DD X Y, DD X, DD Y, DD i j Calculatio of the corrected sum of squares is DD y = D D (Y Y..) = DDY i j the slope is " " " = b = DD xy DDYX DD x œ DDX ( DDY X ) i j D ( DDX) i j DD i j ( DDY ) DD the itercept is calculated ith X ad Y as usual, but these are calculated as DD Y Y.. = ad X.. = DD DDX DD It the eights are 1, the all results are the same as OLS The variace is 5 = 5 = 5 = 5 % Y
WEIGHTED LEAST SQUARES hadout! " i=1 i=1 I OLS e miimize Q =! =! % (Y - " - " X ) or Q = ( Y - X ") ( Y - X ") I eighted least squares, e miimize Q = De i œ DcY b! - b" Xd i If the eights are 1, the all results are the same as OLS The ormal equatios become b! D + b" DX = DY b! DX + b" DX i = DXY For the itermediate calculatios e get (here is the eight) t D D X, DD Y, DD X Y, DD X, DD Y, DD i j Calculatio of the corrected sum of squares is _ DD y = D D (Y Y..) = DDY i j the slope is DDYX DDX ( DDY X ) i j D ( DDX) i j DD i j ( DDY ) DD " " = b " = the itercept is calculated ith X ad Y as usual, but the meas are calculated as _ DDY _ Y.. = ad X.. = DD 5 % Y DDX DD The variace is 5 = 5 = 5 = For multiple regressio Q = DDe = DDcY b! b" X " i b# X # i... b: -" X :-",i) d the ormal equatios (X WX)B = (X WY) the regressio coefficiets -" B = (X WX) (X WY) the variace-covariace matrix; 5, -" = 5 (X WX) is estimated by MSE, based o eighted deviatios; MSE = 5 (Y Y) A A p D
I a multiple regressio, e miimize Q = DDe œ DDcY Y b" X " i b# X # i... b: -" X :-",i) d The matrix cotaiig eights is W = 88 x Ô " 0 0 0 0 # 0 0 Ö Ù 0 0... 0 Õ 0 0 0 Ø 8 The matrix equatios for the regressio solutios are, the ormal equatios the regressio coefficiets (X WX)B = (X WY) -" B = (X WX) (X WY) -", the variace-covariace matrix; 5 = 5 (X WX) here 5 is estimated by MSE A, based o eighted deviatios MSE = = A D(Y Y) D / p p
Estimatig eights For ANOVA this may be relatively easy if eough observatios are available i each group. Variaces ca be estimated directly. Hoever, i regressio e usually have a "smooth" fuctio of chagig residuals ith chagig X values Þ To estimate these e ote that 5 ca be estimated by the residuals e. We ca the estimate the residuals ith OLS, ad regress the residuals (squared or usquared) o a suitable variable (usually oe of the X variables or Y ). The procedure ca be iterated. That is, if the e regressios coefficiets differ substacially from the old, the residuals ca be estimated agai from the eighted regressio ad the eights ca be calcalated agai ad the regressio fit agai ith the e eights.
Weightig may be used for various cases 1) Oe commo use is to adjust for o-homogeeous variace. Oe commo approach is assume that the variace is o-homogeeous because it si a fuctio of X. The e must determie hat that fuctio is, for simple liear regressio, the fuctio is commoly 5 = 5 X, 5 = 5 X, 5 = 5 ÈX to adjust for this, e ould eight by the iverse of the fuctio ) Aother commo case is here the fuctio is ot ko, but the data ca be subset ito smaller groups. These may be separate samples, or they may be cells from a aalysis of variace (ot regressio, but eightig also orks for (ANOVA). The e determie a value of 5 for each cell i, ad the eightig fuctio 1 is the reciprocal (iverse) of the variace for each subgroup 5 ) The values aalyzed are meas, the mea ill differ betee observatios" if the sample size are ot equal. Sice e may still assume that the variace of the origial observatios is homogeeous, The the variace of each poit is a simple fuctio of (here the variace of a mea is 5 1 ) e ould eight by the iverse of the coefficiet, of simply If e could ot assume that the variaces ere equal the for each mea e ould have the variace of 5, ad the eight ould be 5 NOTE: the actual value of a eight is ot importat, oly the proportioal value betee eights, so all eights could be multiplied or divided by some costat value. Some people recommed eightig by ithout 5 cocer for the homogeeity of variace. Sice all 5 are equal to 5 he the variace is homogeeous, this is the same as takig all eights, ad multiplyig by a costat 1 5 5, or
8. ROBUST REGRESSION There are may regressios developed that call themselves "robust". Some are based o the media deviatio, others o trimmed aalyses. The oe e ill discuss uses eighted least squares, ad is called ITERATIVELY REWEIGHTED LEAST SQUARES. a) ordiary least squares is cosidered "robust", but is sesitive to poits hich deviate greatly from the lie this occurs (1) ith data hich is ot ormal or symmetrical data ith a large tail () he there are outliers (cotamiatio) b) ROBUST REGRESSION is basically eighted least squares here the here the eights are a iverse fuctio of the residuals (1) poits ithi a rage "close" to the regressio lie get a eight of 1 if all eights are 1 the the result is ordiary least squares regressio () poits outside the rage get a eight less tha 1 () ordiary least squares miimize D (Y X ") = D(Y Y..) = (residual) robust regressio miimizes D Ò(Y X " ) c5 here is some fuctio of the stadardized residual ad should be chose to miimize the ifluece of large residuals. The chose is a eight, ad if the proper fuctio is chose, the calculatios may be doe as a eighted least squares, miimizig D (Y X ")
(4) the fuctio chose by Huber as chose such that if the residual is small ( Ÿ 1.45 5 ) there is o eight First e defie a robust estimate of variace as called the media absolute 7/.+8Öl/ 7/.+8Ð/ Ñl deviatio (MAD) such that MAD =!Þ'(%&, here the costat 0.6745 ill give a ubiased estimate of 5 for idepedet observatios from a ormal distributio. MAD is a alterative estimate of The e defie a scaled residual as the eight = 1 if le l Ÿ 1. $%&5 "Þ$%& = l. l for ke k > 1.45 5 ÈMSE.ß here. = / QEH here 1.45 is called the tuig costat, ad is chose to make the techique robust ad 95% efficiet. Efficiecy refers to the ratio of variaces from this model relative to ormal regressio models. (5) The solutio is iterative (a) do regressio, obtai e values (b) compute the eights (c) ru the eighted least squares aalysis usig the eights (d) iteratively recalculate the eights ad reru the eighted least squares aalysis UNTIL (1) there is o chage i the regressio coefficiets to some predetermied level of precisio () ote that the solutio may ot be the oe ith the miimum sum of squares deviatios
(6) PROBLEMS (a) this is ot a ordiary least squares, but the hypotheses are likely to be tested as if they ere the distributioal properties of the estimator are ot ell documeted (b) Schreuder, Boos ad Hafley (Forest Resources Ivetories Workshop Proc., Colorado State U., 1979) suggest ruig both regressios if "similar", use ordiary least squares if "differet" fid out hy; if because of a outlier the robust regressio is "probably" better they further suggest usig robust regressio as "a tool for aalysis, ot a cure all for bad data" The techique is useful for detectig outliers.