Linear Regression Demystified
|
|
- Angelina Powell
- 5 years ago
- Views:
Transcription
1 Liear Regressio Demystified Liear regressio is a importat subject i statistics. I elemetary statistics courses, formulae related to liear regressio are ofte stated without derivatio. This ote iteds to derive these formulae for studets with more advaced math backgroud. Simple Regressio: Oe Variable. Least Square Prescriptio Suppose we have a set of data poits (x i, y i ) (i =,,, ). The goal of liear regressio is to fid a straight lie that best fit the data. I other words, we wat to build a model ŷ i = β 0 + β x i () ad tue the parameters β 0 ad β so that ŷ i is as close to y i as possible for all i =,,,. Before we do the math, we eed to clarify the problem. How do we judge the closeess of ŷ i ad y i for all i? If the data poits (x i, y i ) do ot fall exactly o a straight lie, y i ad ŷ i is ot goig to be the same for all i. The deviatio of y i from ŷ i is called the residual ad is deoted by ɛ i here. I other words, y i = ŷ i + ɛ i = β 0 + β x i + ɛ i. () The least square prescriptio is to fid β 0 ad β that miimize the sum of the square of the residuals SSE: SSE = ɛ i = (y i ŷ i ) = (y i β 0 β x i ). (3) If you are familiar with calculus, you will kow that the miimizatio ca be doe by settig the derivatives of SSE with respect to β 0 ad β to 0, i.e. SSE/ β 0 = 0 ad SSE/ β = 0. The resultig equatios are (y i ŷ i ) = ɛ i = 0 ad x i (y i ŷ i ) = x i ɛ i = 0. (4) We ca combie the first ad secod equatio to derive a alterative equatio for the secod equatio: ɛ i = 0 xɛ i = 0 Subtractig this equatio from the secod equatio of (4) yields Equatio (4) ca be expressed as (y i ŷ i ) = ɛ i = 0 (x i x)ɛ i = 0. ad (x i x)(y i ŷ i ) = (x i x)ɛ i = 0. (5) If you are ot familiar with calculus, we ca show you a proof usig algebra. I actually prefer his proof sice it is more rigorous. Before presetig the proof, let s remid you some cocepts i statistics. Careful studets may realize that this oly gauratees that SSE is statioary but ot ecessarily miimum. There is also a cocer whether the resultig solutio is a global miimum or a local miimum. Detail aalysis (see Sectio.3) reveals that the solutio i our case is a global miimum.
2 . Mea, Stadard Deviatio ad Correlatio The mea ad stadard deviatio of a set of poits {u i } are defied as ū = u i, SD u = (u i ū). (6) The Z score of u i is defied as Z ui = u i ū SD u. (7) The correlatio of two set of poits (with equal umber of elemets) {x i } ad {y i } is defied as r xy = r yx = Z xi Z yi = The Cauchy-Schwarz iequality states that for two set of real umbers {u i } ad {v i }, ( ) ( ) ( ) u i v i u i (x i x)(y i ȳ) SD x (8) ad the equality holds if ad oly if v i = ku i for all i, where k is a costat. You ca fid the proof of the iequality i, e.g., Wikipedia. It follows from the Cauchy-Schwarz iequality that the correlatio r xy, with r xy = if ad oly if x i ad y i fall exactly o a straight lie, i.e. ɛ i = 0 for all i. So the correlatio r xy measures how tightly the poits (x i, y i ) are clustered aroud a lie..3 Regressio Equatio To derive the regressio equatio, we first rewrite SSE i equatio (3) i terms of Z-scores. It follows from the defiitio of Z-scores that x i = x + SD x Z xi, y i = ȳ + Z yi ad the SSE becomes SSE = For simplicity, we deote β 0 = ȳ β 0 β x. The we have SSE = = [ȳ + Z yi β 0 β ( x + SD x Z xi )] ( Z yi β SD x Z xi + ȳ β 0 β x) ( Z yi β SD x Z xi + β 0 ) = SD y Zyi + βsd x Zxi + + β 0 Z yi β 0 β SD x v i β 0 β SD x Z xi Z yi Z xi (9) Recall that the Z-scores are costructed to have zero mea ad uit stadard deviatio. It follows that Z xi = Z yi = 0, Zxi = Zyi = i
3 ad from the defiitio of correlatio we have Thus the SSE i equatio (9) ca be writte as Z xi Z yi = r xy, SSE = (SDy + βsd x + β 0 β r xy SD x ) = [ β 0 + (SD x β r xy ) + ( rxy)sd y] = [(ȳ β 0 β x) + (SD x β r xy ) ] + ( rxy)sd y (0) Note that we have re-expressed β 0 i terms of β 0 ad β i the last step. Remember your goal is to fid β 0 ad β to miimize SSE. The quatity iside the square bracket i equatio (0) is a sum of two squares ad cotai the variables β 0 ad β, whereas ( r xy)sd y is a fixed quatity. Therefore, we coclude that ad the equality holds if ad oly if SSE ( r xy)sd y ȳ β 0 β x = 0 ad SD x β r xy = 0. Therefore, the values of β 0 ad β that miimize SSE is β = r xy SD x, β 0 = ȳ β x () ad the miimum SSE is SSE = ( r xy)sd y () How does equatios () related to equatios (5) derived from calculus? We are goig to prove that they are the same thig. Usig equatios () we have ɛ i = = = = = 0 (y i ŷ i ) (y i β 0 β x i ) If the last step is ot obvious to you, here s a simple proof: (y i ȳ + β x β x i ) (y i ȳ) β (x i x) x = x i x i = x = x (x i x) = 0 ad similarly (y i ȳ) = 0. 3
4 Now let s look at the secod equatio of (5): (x i x)(y i ȳ) = = (x i x)(y i ȳ + β x β x i ) (x i x)(y i ȳ) β (x i x) From the defiitio of correlatio ad stadard deviatio, we have r xy = (x i x)(y i ȳ) SD x (x i x)(y i ȳ) = r xy SD x Thus, SD x = (x i x) (x i x) = SDx (x i x)(y i ȳ) = r xy SD x β SDx = SDx ( ) r xy β = 0. SD x Therefore, we have proved that equatios (5) follows from equatios (). O the other had, equatios (5) are two liear equatios for β 0 ad β. We ca solve for β 0 ad β from these equatios, which we will do so i Sectio.7. The result is that β 0 ad β are give by equatios () ad so the two sets of equatios are equivalet..4 Iterpretatio We will try to give you a ituitio of the meaig of equatios (5) below. We ca rewrite the first equatio of (5) as ɛ = 0. (3) That is to say that the mea of ɛ i vaishes. The secod equatio of (5) meas that (x i x)ɛ i = (x i x)(ɛ i ɛ) = 0 SD x SD ɛ r xɛ = 0 r xɛ = 0. That is to say that x i ad ɛ i are ucorrelated. Thus the least square prescriptio is to fid β 0 ad β to make the residuals havig zero mea ad beig ucorrelated with x i. Let s illustrate how this ca be doe graphically. Suppose we have the followig data set of x i ad y i : 4
5 x y We wat to fit a straight lie y = β 0 + β x to the data poits. We kow the lie has to make the resuduals havig zero mea ad beig ucorrelated with x i. First, let s imagie choosig various values of β ad see what the plot y β x versus x look like. I geeral, we will have the plot similar to the oe above but with differet overall slope. If we pick the right value of β, we will see that the residual y β x vs x is flat: x y β x We kow have achieved the first goal: by choosig the right value of β, the residual y β x is ucorrelated with x. However, the residuals do ot have zero mea. So ext we wat to choose β 0 so that y β 0 β x has zero 5
6 mea. Whe the right value of β 0 is chose, the residuals become this: x ε = y β 0 β x We have accomplished our goal. The residuals ow have zero mea ad are ucorrelated with x i. The resultig regressio lie y = β 0 + β x is the straight lie that is best fit to the data poits. The graph below shows the regressio lie (blue), data poits (black) ad the residuals (red) x y We have outlied the idea of liear regressio. Next we itroduce the the cocept of a vector, which proves to be very useful for regressio with more tha oe variables. 6
7 .5 Vectors A -dimesioal vector V cotais real umbers v, v,, v, ofte writte i the form v v V =. v The umbers v, v,, v are called the compoets of V. Here we adopt the covetio to write vector variables i boldface to distiguish them from umbers. It is useful to defie a special vector X 0 whose compoets are all equal to. That is, X 0 = (4). The scalar product of two vectors U ad V is defied as U V = V U = u i v i, where v i ad u i are compoets of V ad U. Note that the result of the scalar product of two vectors is a umber, ot a vector. Two vectors U ad V are said to be orthogoal if ad oly if U V = 0. It follows from the defiitio that V V = vi 0, with V V = 0 if ad oly if all compoets of V are 0. It follows from the Cauchy-Schwarz iequality that (U V ) (U U)(V V ) with the equality holds if ad oly if U = kv for some costat k. Now that we have itroduced the basic cocept of a vector, we are ow ready to express the regressio equatios i vector form..6 Regressio Equatios i Vector Form Equatio () ca be writte as The sum of square of residuals SSE i (3) ca be writte as Y = Ŷ + ɛ = β 0 X 0 + β X + ɛ. (5) SSE = ɛ ɛ = (Y Ŷ ) (Y Ŷ ) = (Y β 0 X 0 β X) (Y β 0 X 0 β X) (6) To miimize SSE, we set the derivatives of SSE with respect to β 0 ad β to 0. The resultig equatios are or X 0 (Y β 0 X 0 β X) = 0, X (Y β 0 X 0 β X) = 0 X 0 ɛ = X ɛ = 0. (7) This is equatio (4) writte i vector form. We see that the least square prescriptio ca be iterpreted as fidig β 0 ad β to make the residual vector ɛ orthogoal to both X 0 ad X. 7
8 The mea ad stadard deviatio i (6) ca be expressed as ū = X 0 U, SD u = (U ūx 0) (U ūx 0 ) (8) ad the Z score i (7) becomes It follows that The correlatio r xy i (8) is proportioal to the scalar product of Z x ad Z y : The equatio r xɛ = 0 is equivalet to Z u = U ūx 0 SD u (9) Z u Z u = (0) r xy = Z x Z y () Z x ɛ = 0 () We see that the equatios i vector form are simpler ad more elegat. We are ow ready to solve the regressio equatios (5) ad (7) for β 0 ad β..7 Regressio Coefficiets I this sectio, we are goig to use the vector algebra to solve for β 0 ad β to re-derive equatios (). This is a useful exercise because it ca be easily geeralized to multiple regressio we cosider later. Start with equatio (5). Takig the scalar product of (5) with X 0, usig equatios (8), (7) ad X 0 X 0 =, we obtai ȳ = β 0 + β x ȳ = β 0 + β x, (3) Multiplyig both sides of the above equatio by X 0 gives Subtractig the above equatio from (5) gives ȳx 0 = β 0 X 0 + β xx 0 Y ȳx 0 = β (X xx 0 ) + ɛ It follows from equatio (9) that Y ȳx 0 = Z y ad X xx 0 = SD x Z x. Hece we have Z y = β SD x Z x + ɛ Takig the scalar product with Z x ad usig Z x ɛ = 0, we obtai Z x Z y = β SD x Z x Z x It follows from equatio () ad (0) that Z x Z y = r xy ad Z x Z x = ad so the above equatio reduces to r xy = β SD x β = r xy SD x. Combiig the above equatio with (3), we fially solve β 0 ad β : β = r xy SD x, β 0 = ȳ β x (4) The regressio lie is the straight lie with slope r xy /SD x passig through the poit ( x, ȳ). 8
9 .8 Root Mea Square Error Havig foud a straight lie that best fit the data poits, we ext wat to kow how good the fit is. Oe way to characterize the goodess of the fit is to calculate the mea square error RMSE, defied as SSE ɛ ɛ RMSE = = (5) It follows from equatio () that RMSE = rxy. Here we derive it agai usig the vector algebra sice it ca be easily geeralized to multiple regressio. We itroduce two quatities SST ad SSM. The total sum square SST is defied as SST = (y i ȳ) = (Y ȳx 0 ) (Y ȳx 0 ) (6) It follows from the defiitio of stadard deviatio [see equatio (6) or (8)] that SST = SD y (7) The sum square predicted by the liear model SSM is defied as SSM = (ŷ i ȳ) = (Ŷ ȳx 0 ) (Ŷ ȳx 0 ) (8) Usig Ŷ = β 0 X 0 + β X ad β 0 = ȳ β x, we have Ŷ ȳx 0 = (ȳ β x)x 0 + β X ȳx 0 = β (X xx 0 ) = β SD x Z x, (9) where we have used equatio (9) to obtai the last equality. Combiig (8), (9), (0) (4) ad (7), we obtai ( ) SSM = βsd x SD y = r xy SDx = r SD xysd y = rxysst (30) x Next we wat to prove a idetity that SST = SSM + SSE. To see that, we start with the idetity Y ȳx 0 = (Y Ŷ ) + (Ŷ ȳx 0 ) = ɛ + (Ŷ ȳx 0 ) We the square both sides by takig the scalar product with itself: (Y ȳx 0 ) (Y ȳx 0 ) = [ɛ + (Ŷ ȳx 0 )] [ɛ + (Ŷ ȳx 0 )] The left had side is SST, the right had side is the sum of SSE, SSM ad a cross term: SST = ɛ ɛ + (Ŷ ȳx 0 ) (Ŷ ȳx 0 ) + ɛ (Ŷ ȳx 0 ) = SSE + SSM + ɛ (Ŷ ȳx 0 ) It follows from (9) ad Z x ɛ = 0 that the cross term vaishes: ɛ (Ŷ ȳx 0 ) = β SD x Z x ɛ = 0 Aother way of seeig this is to ote that ɛ is orthogoal to both X 0 ad X, as required by the least square prescriptio (7). So ɛ is orthogoal to ay liear combiatio of X 0 ad X. Sice Ŷ ȳx 0 is a liear combiatio of X 0 ad X, it is orthogoal to ɛ. We have just proved that SST = SSM + SSE (3) We have calculated above that SSM = r xy ad SST = SD y. Usig the idetity SST = SSM + SSE, we obtai SSE = SST SSM = ( r xy)sst = ( r xy)sd y. Combiig this equatio with the defiitio of RMSE i (5), we fid RMSE = rxy (3) 9
10 Multiple Regressio Now we wat to geeralize the results of simple regressio to multiple regressio. We will first cosider the case with two variables i Sectio., because simple aalytic expressios exist i this case ad the derivatio is relatively straightforward. The we will tur to the more geeral case with more tha two variables i Sectio.. Fially, we will derive a geeral result for the RMSE calculatio i Sectio.4.. Two Variables Suppose we wat to fit the data poits {y i } with two variables {x i } ad {x i } by a liear model with y i = ŷ i + ɛ i, i =,,, ŷ i = β 0 + β x i + β x i. The least square prescriptio is agai to fid β 0, β ad β to miimize the sum square of the residuals: SSE = ɛ i = (y i ŷ i ) As i the case of simple regressio, it is more coveiet to rewrite the above equatios i vector form as follows: Y = Ŷ + ɛ = β 0 X 0 + β X + β X + ɛ (33) Ŷ = β 0 X 0 + β X + β X (34) SSE = ɛ ɛ = (Y β 0 X 0 β X β X ) (Y β 0 X 0 β X β X ) (35) To employ the least square prescriptio, we set the derivatives of SSE with respect to β 0, β ad β to 0. The resultig equatios are X 0 ɛ = X ɛ = X ɛ = 0. (36) That is to say that ɛ is orthogoal to X 0, X ad X. I other words, Ŷ is the vector Y projected oto the vector space spaed by X 0, X ad X. For coveiece, we deote Z ad Z as the Z-score vectors associated with X ad X, respectively. That is, Z = X x X 0, Z = X x X 0, (37) SD SD where SD ad SD are the stadard deviatio of {x i } ad {x i }, respectively. We see that Z is a liear combiatio of X 0 ad X ; Z is a liear combiatio of X 0 ad X. Sice ɛ to orthogoal to X 0, X ad X, it is orthogoal to Z ad Z as well: Z ɛ = Z ɛ = 0. (38) Recall that the mea of a vector U is ū = X 0 U/, ad the correlatio betwee U ad V is r uv = Z u Z v /. Thus the orthogoality coditios of ɛ mea that () ɛ has zero mea, ɛ = 0, ad () ɛ is ucorrelated with both X ad X : r ɛ = r ɛ = 0. To solve the regressio equatios (36), we take the scalar product of equatio (33) with X 0, resultig i the equatio ȳ = β 0 + β x + β x (39) This meas that the regressio lie passes through the average poit ( x, x, ȳ). Multiplyig equatio (39) by X 0 yields ȳx 0 = β 0 X 0 + β x X 0 + β x X 0 0
11 Subtractig the above equatio from (33) gives Y ȳx 0 = β (X x X 0 ) + β (X x X 0 ) + ɛ Usig the defiitio of the Z score vector, we ca write the above equatio as Takig the scalar product of the above equatio with Z gives Z y = β SD Z + β SD Z + ɛ (40) r y = β SD + β SD r, (4) where r y = Z y Z / is the correlatio betwee Y ad X, r = Z Z / is the correlatio betwee X ad X. Equatio (4) ca be writte as ( ) SD β = r y β r = β y β β, (4) SD SD where β y = r y /SD is the slope i the simple regressio for predictig Y from X, ad β = r SD /SD is the slope i the simple regressio for predictig X from X. Before derivig the fial expressio for β, let s poit out two thigs. First, if r = 0 (or equivaletly X X = 0) the β = β y. That is to say that if X ad X are ucorrelated (orthogoal), the slope β i the multiple regressio is exactly the same as the slope β y i the simple regressio for predictio Y from X. The additio of the X does ot chage the slope. Secod, if r > 0 ad β > 0, the β < β y. The slope β decreases if X ad X are positively correlated ad if the slope of X i the multiple regressio is positive. Let s go back to equatio (4). If r 0, the equatio of β ivolves β. So β ad β has to be solved together. Sice X ad X are symmetric, the equatio for β ca be obtaied by simply exchagig the idex betwee ad of the β equatio: β = β y β β, (43) where Substitutig β from (43) ito (4), we obtai β y = r y SD ad β = r SD SD. (44) β = β y (β y β β )β = β y β y β + β β β (45) Note that the product Thus equatio (45) becomes β β = ( ) ( ) SD SD r r = r SD SD. or Usig the expressios for β y, β y ad β, we fid β = β y β y β + r β ( r )β = β y β y β β = β y β y β r β = b SD, (46) where b = r y r y r r
12 may be iterpreted as the adjusted correlatio betwee Y ad X takig ito accout the presece of X. The expressio for β is obtaied by exchagig the idex betwee ad : β = b, b = r y r y r SD r. Gatherig all the results, we coclude that the regressio coefficiets are give by with β = b SD, β = b SD, β 0 = ȳ β x β x (47) b = r y r y r r, b = r y r y r r I Sectio.8, we see that the RMSE i simple regressio is related to by RMSE = multiple regressio, we will show i Sectio.4 that the formula is geeralized to where R is the correlatio betwee Y ad Ŷ : RMSE = R, R = Z y Zŷ. (48) r xy. I We will defer the calculatio of R to Sectio.3. I the case of multiple regressio with two variables cosidered here, R is give by ry R = + r y r yr y r r (49). More Tha Two Variables Suppose we ow wat to fit {y i } with p variables {x i, x i,, x pi }. The regressio equatio has p + parameters β 0, β, β,, β p. It ca be writte i the vector form as Y = Ŷ + ɛ = Ŷ = SSE = ɛ ɛ = Y β j X j + ɛ (50) j=0 β j X j (5) j=0 β j X j Y j=0 β j X j (5) To miimize SSE, we set the derivatives of SSE with respect to β j (j = 0,,, p) to 0. The resultig equatios ca be writte as X j ɛ = 0, j = 0,,, p. (53) This meas that ɛ is orthogoal all X 0, X,, X p, or equivaletly, the mea of ɛ is 0 ad ɛ is ucorrelated with ay of the variables we are tryig to fit. j=0
13 To fid the regressio coefficiets, we follow the same procedures as before. First, take the scalar product of equatio (50) with X 0. The result is ȳ = β 0 + β j x j (54) As before, this meas that the regressio lie passes through the poit of average. Next, we multiple the above equatio by X 0 : ȳx 0 = β 0 X 0 + β j x j X 0 ad the subtract it from equatio (50): Y ȳx 0 = β j (X j x j X 0 ) + ɛ. Usig the defiitio of the Z score vector (9), we ca write the above equatio as Dividig both sides by results i where Z y = Z y = β j SD j Z j + ɛ. Takig the scalar product of equatio (55) with Z i (i = 0,,,, p), we obtai b j Z j + ɛ, (55) b j = β j SD j. (56) r ij b j = r yi, i =,,, p. (57) This is a system of liear equatios for b, b,, b p. Writte them out, they look like b + r b + r 3 b r p b p = r y b r + b + r 3 b r p b p = r y b r 3 + b r 3 + b r 3p b p = r y3 b r p + b r p + b 3 r p3 + + b p = r yp If all of the variables X j are ucorrelated (i.e. r ij = 0 if i j), the solutio is b j = r yj ad the slopes are β j = r yj /SD j. This is exactly the same as the slopes i simple regressio for predictig Y from X j. There are o simple aalytic expressios for b j i geeral, but there are several well-kow procedures to obtai the solutio by successive algebraic operatios, but we will ot discuss the methods here. Suppose all the b s have bee solved usig oe of those procedures, the slopes are give by equatio (56) as ad the itercept β 0 is give by equatio (54) as... β j = b j SD j, j =,,, p (58) β 0 = ȳ 3 β j x j. (59)
14 Fially, we should metio that the term liear i liear regessio refers to a model beig liear i the fittig parameters β 0, β,, β p. For example, we ca fit y i by the model y i = β 0 + β x i + β xi + β 3 x i + β 4 l x i + β 5 + x i usig the techique of multiple liear regressio sice the model is liear i β 0, β,, β 5. We simply label x 3 i (60) x i = x i, x i = x i, x 3i = x i, x 4i = l x i, x 5i = x3 i + x i ad equatio (60) ca be writte i vector form as Y = β 0 X 0 + β X + β X + β 3 X 3 + β 4 X 4 + β 5 X 5, which is equatio (50) with p = 5. The key is to ote that the least square prescriptio is to miimize SSE by varyig the paremeters β s, ot the idepedet variables x s..3 Correlatio Betwee Y ad Ŷ To geeralize the RM SE expressio i Sectio.8 for multiple regressio, we cosider the quatity R defied as the correlatio betwee Y ad Ŷ : R = Z y Zŷ. (6) We first calculate the average of Ŷ : ŷ = Ŷ X 0 = β 0 X 0 + β j X j X 0 = β 0 + β j x j = ȳ, where we have used equatio (5) for Ŷ ad (54) for ȳ. From the defiitio of the Z score vector (9) ad ŷ = ȳ, we ca write (6) as R = (Y ȳx 0) (Ŷ ȳx 0 ) SDŷ = (Ŷ + ɛ ȳx 0 ) (Ŷ ȳx 0 ) SDŷ = (Ŷ ȳx 0 ) (Ŷ ȳx 0 ) SDŷ = SD ŷ (6) where we have used ɛ X 0 = 0 ad ɛ Ŷ = 0 (sice Ŷ is a liear combiatio of X 0, X p ad all are orthogoal to ɛ). We have also used the defiitio of the stadard deviatio (8) to obtai the last lie. Hece we have R = SD ŷ SD y = (ŷ i ȳ), (63) (y i ȳ) which is iterpreted as the fractio of the variace of Y explaied by the liear model. 4
15 To compute R, we use equatio (5) for Ŷ ad (59) for β 0, ad write SDŷ = (Ŷ ȳx 0 ) (Ŷ ȳx 0 ) [ ] = (β 0 ȳ)x 0 + β i X i (β 0 ȳ)x 0 + β j X j [ ] = β i (X i x i X 0 ) β j (X j x j X 0 ) ( ) = SD i β i Z i SD j β j Z j = = SD i SD j β i β j Z i Z j SD i SD j β i β j r ij R = SD ŷ SD y = b i b j r ij, where we have used the defiitio of b j i equatio (56). The sum ca be simplified by otig that b j satisfy equatio (57), ad thus we obtai R = b i r yi (64) Whe the solutio of b i is obtaied, R ca be calculated usig the above equatio. I the case of multiple regressio with two variables (p = ), b ad b are give by equatio (48). Pluggig them ito equatio (64), we obtai equatio (49)..4 Root Mea Square Error As i Sectio.8, we defie SST ad SSM as SST = (y i ȳ) = (Y ȳx 0 ) (Y ȳx 0 ) SSM = (ŷ i ȳ) = (Ŷ ȳx 0 ) (Ŷ ȳx 0 ) It follows from the defiitio of stadard deviatio that ad it follows from (63) that SST = SD y (65) SSM = R SST. (66) The idetity SST = SSM + SSE still holds i muiltiple liear regressio. The proof is almost exactly the same as i Sectio.8. 5
16 We start with the idetity Take the scalar product with itself: Y ȳx 0 = (Y Ŷ ) + (Ŷ ȳx 0 ) = ɛ + (Ŷ ȳx 0 ). (Y ȳx 0 ) (Y ȳx 0 ) = [ɛ + (Ŷ ȳx 0 )] [ɛ + (Ŷ ȳx 0 )] SST = SSE + SSM + ɛ (Ŷ ȳx 0 ) Sice Ŷ ȳx 0 is a liear combiatio of X 0, X,, X p ad ɛ is orthogoal to all these vectors, ɛ (Ŷ ȳx 0 ) = 0 ad so we have SST = SSM + SSE (67) Combiig equatio (65), (66) ad (67), we have SSE = ( R )SD y. It follows from the defiitio of RMSE = SSE/ that which is the geeralizatio of equatio (3). RMSE = R (68) 6
ECON 3150/4150, Spring term Lecture 3
Itroductio Fidig the best fit by regressio Residuals ad R-sq Regressio ad causality Summary ad ext step ECON 3150/4150, Sprig term 2014. Lecture 3 Ragar Nymoe Uiversity of Oslo 21 Jauary 2014 1 / 30 Itroductio
More informationSimple Linear Regression
Chapter 2 Simple Liear Regressio 2.1 Simple liear model The simple liear regressio model shows how oe kow depedet variable is determied by a sigle explaatory variable (regressor). Is is writte as: Y i
More informationMachine Learning for Data Science (CS 4786)
Machie Learig for Data Sciece CS 4786) Lecture & 3: Pricipal Compoet Aalysis The text i black outlies high level ideas. The text i blue provides simple mathematical details to derive or get to the algorithm
More informationAlgebra of Least Squares
October 19, 2018 Algebra of Least Squares Geometry of Least Squares Recall that out data is like a table [Y X] where Y collects observatios o the depedet variable Y ad X collects observatios o the k-dimesioal
More informationGeometry of LS. LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT
OCTOBER 7, 2016 LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT Geometry of LS We ca thik of y ad the colums of X as members of the -dimesioal Euclidea space R Oe ca
More informationSummary: CORRELATION & LINEAR REGRESSION. GC. Students are advised to refer to lecture notes for the GC operations to obtain scatter diagram.
Key Cocepts: 1) Sketchig of scatter diagram The scatter diagram of bivariate (i.e. cotaiig two variables) data ca be easily obtaied usig GC. Studets are advised to refer to lecture otes for the GC operatios
More informationU8L1: Sec Equations of Lines in R 2
MCVU U8L: Sec. 8.9. Equatios of Lies i R Review of Equatios of a Straight Lie (-D) Cosider the lie passig through A (-,) with slope, as show i the diagram below. I poit slope form, the equatio of the lie
More informationFirst, note that the LS residuals are orthogonal to the regressors. X Xb X y = 0 ( normal equations ; (k 1) ) So,
0 2. OLS Part II The OLS residuals are orthogoal to the regressors. If the model icludes a itercept, the orthogoality of the residuals ad regressors gives rise to three results, which have limited practical
More informationSNAP Centre Workshop. Basic Algebraic Manipulation
SNAP Cetre Workshop Basic Algebraic Maipulatio 8 Simplifyig Algebraic Expressios Whe a expressio is writte i the most compact maer possible, it is cosidered to be simplified. Not Simplified: x(x + 4x)
More informationInverse Matrix. A meaning that matrix B is an inverse of matrix A.
Iverse Matrix Two square matrices A ad B of dimesios are called iverses to oe aother if the followig holds, AB BA I (11) The otio is dual but we ofte write 1 B A meaig that matrix B is a iverse of matrix
More informationThe Binomial Theorem
The Biomial Theorem Robert Marti Itroductio The Biomial Theorem is used to expad biomials, that is, brackets cosistig of two distict terms The formula for the Biomial Theorem is as follows: (a + b ( k
More informationLinear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d
Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y
More informationEconomics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator
Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters
More information1 Inferential Methods for Correlation and Regression Analysis
1 Iferetial Methods for Correlatio ad Regressio Aalysis I the chapter o Correlatio ad Regressio Aalysis tools for describig bivariate cotiuous data were itroduced. The sample Pearso Correlatio Coefficiet
More informationCHAPTER 5. Theory and Solution Using Matrix Techniques
A SERIES OF CLASS NOTES FOR 2005-2006 TO INTRODUCE LINEAR AND NONLINEAR PROBLEMS TO ENGINEERS, SCIENTISTS, AND APPLIED MATHEMATICIANS DE CLASS NOTES 3 A COLLECTION OF HANDOUTS ON SYSTEMS OF ORDINARY DIFFERENTIAL
More informationDefinitions and Theorems. where x are the decision variables. c, b, and a are constant coefficients.
Defiitios ad Theorems Remember the scalar form of the liear programmig problem, Miimize, Subject to, f(x) = c i x i a 1i x i = b 1 a mi x i = b m x i 0 i = 1,2,, where x are the decisio variables. c, b,
More informationSimple Linear Regression
Simple Liear Regressio 1. Model ad Parameter Estimatio (a) Suppose our data cosist of a collectio of pairs (x i, y i ), where x i is a observed value of variable X ad y i is the correspodig observatio
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationThe Method of Least Squares. To understand least squares fitting of data.
The Method of Least Squares KEY WORDS Curve fittig, least square GOAL To uderstad least squares fittig of data To uderstad the least squares solutio of icosistet systems of liear equatios 1 Motivatio Curve
More informationKinetics of Complex Reactions
Kietics of Complex Reactios by Flick Colema Departmet of Chemistry Wellesley College Wellesley MA 28 wcolema@wellesley.edu Copyright Flick Colema 996. All rights reserved. You are welcome to use this documet
More informationMath 155 (Lecture 3)
Math 55 (Lecture 3) September 8, I this lecture, we ll cosider the aswer to oe of the most basic coutig problems i combiatorics Questio How may ways are there to choose a -elemet subset of the set {,,,
More informationCHAPTER I: Vector Spaces
CHAPTER I: Vector Spaces Sectio 1: Itroductio ad Examples This first chapter is largely a review of topics you probably saw i your liear algebra course. So why cover it? (1) Not everyoe remembers everythig
More informationCorrelation Regression
Correlatio Regressio While correlatio methods measure the stregth of a liear relatioship betwee two variables, we might wish to go a little further: How much does oe variable chage for a give chage i aother
More informationReview Problems 1. ICME and MS&E Refresher Course September 19, 2011 B = C = AB = A = A 2 = A 3... C 2 = C 3 = =
Review Problems ICME ad MS&E Refresher Course September 9, 0 Warm-up problems. For the followig matrices A = 0 B = C = AB = 0 fid all powers A,A 3,(which is A times A),... ad B,B 3,... ad C,C 3,... Solutio:
More informationZeros of Polynomials
Math 160 www.timetodare.com 4.5 4.6 Zeros of Polyomials I these sectios we will study polyomials algebraically. Most of our work will be cocered with fidig the solutios of polyomial equatios of ay degree
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationACCESS TO SCIENCE, ENGINEERING AND AGRICULTURE: MATHEMATICS 1 MATH00030 SEMESTER / Statistics
ACCESS TO SCIENCE, ENGINEERING AND AGRICULTURE: MATHEMATICS 1 MATH00030 SEMESTER 1 018/019 DR. ANTHONY BROWN 8. Statistics 8.1. Measures of Cetre: Mea, Media ad Mode. If we have a series of umbers the
More information(3) If you replace row i of A by its sum with a multiple of another row, then the determinant is unchanged! Expand across the i th row:
Math 5-4 Tue Feb 4 Cotiue with sectio 36 Determiats The effective way to compute determiats for larger-sized matrices without lots of zeroes is to ot use the defiitio, but rather to use the followig facts,
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationProperties and Hypothesis Testing
Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.
More informationDiscrete Mathematics for CS Spring 2007 Luca Trevisan Lecture 22
CS 70 Discrete Mathematics for CS Sprig 2007 Luca Trevisa Lecture 22 Aother Importat Distributio The Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first
More information(3) If you replace row i of A by its sum with a multiple of another row, then the determinant is unchanged! Expand across the i th row:
Math 50-004 Tue Feb 4 Cotiue with sectio 36 Determiats The effective way to compute determiats for larger-sized matrices without lots of zeroes is to ot use the defiitio, but rather to use the followig
More information3.2 Properties of Division 3.3 Zeros of Polynomials 3.4 Complex and Rational Zeros of Polynomials
Math 60 www.timetodare.com 3. Properties of Divisio 3.3 Zeros of Polyomials 3.4 Complex ad Ratioal Zeros of Polyomials I these sectios we will study polyomials algebraically. Most of our work will be cocered
More informationSeptember 2012 C1 Note. C1 Notes (Edexcel) Copyright - For AS, A2 notes and IGCSE / GCSE worksheets 1
September 0 s (Edecel) Copyright www.pgmaths.co.uk - For AS, A otes ad IGCSE / GCSE worksheets September 0 Copyright www.pgmaths.co.uk - For AS, A otes ad IGCSE / GCSE worksheets September 0 Copyright
More informationTheorem: Let A n n. In this case that A does reduce to I, we search for A 1 as the solution matrix X to the matrix equation A X = I i.e.
Theorem: Let A be a square matrix The A has a iverse matrix if ad oly if its reduced row echelo form is the idetity I this case the algorithm illustrated o the previous page will always yield the iverse
More informationChapter Vectors
Chapter 4. Vectors fter readig this chapter you should be able to:. defie a vector. add ad subtract vectors. fid liear combiatios of vectors ad their relatioship to a set of equatios 4. explai what it
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More informationWe are mainly going to be concerned with power series in x, such as. (x)} converges - that is, lims N n
Review of Power Series, Power Series Solutios A power series i x - a is a ifiite series of the form c (x a) =c +c (x a)+(x a) +... We also call this a power series cetered at a. Ex. (x+) is cetered at
More informationMachine Learning for Data Science (CS 4786)
Machie Learig for Data Sciece CS 4786) Lecture 9: Pricipal Compoet Aalysis The text i black outlies mai ideas to retai from the lecture. The text i blue give a deeper uderstadig of how we derive or get
More informationSome examples of vector spaces
Roberto s Notes o Liear Algebra Chapter 11: Vector spaces Sectio 2 Some examples of vector spaces What you eed to kow already: The te axioms eeded to idetify a vector space. What you ca lear here: Some
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationEcon 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.
Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio
More informationTHE KALMAN FILTER RAUL ROJAS
THE KALMAN FILTER RAUL ROJAS Abstract. This paper provides a getle itroductio to the Kalma filter, a umerical method that ca be used for sesor fusio or for calculatio of trajectories. First, we cosider
More informationLesson 10: Limits and Continuity
www.scimsacademy.com Lesso 10: Limits ad Cotiuity SCIMS Academy 1 Limit of a fuctio The cocept of limit of a fuctio is cetral to all other cocepts i calculus (like cotiuity, derivative, defiite itegrals
More informationAssignment 5: Solutions
McGill Uiversity Departmet of Mathematics ad Statistics MATH 54 Aalysis, Fall 05 Assigmet 5: Solutios. Let y be a ubouded sequece of positive umbers satisfyig y + > y for all N. Let x be aother sequece
More information2 Geometric interpretation of complex numbers
2 Geometric iterpretatio of complex umbers 2.1 Defiitio I will start fially with a precise defiitio, assumig that such mathematical object as vector space R 2 is well familiar to the studets. Recall that
More informationP.3 Polynomials and Special products
Precalc Fall 2016 Sectios P.3, 1.2, 1.3, P.4, 1.4, P.2 (radicals/ratioal expoets), 1.5, 1.6, 1.7, 1.8, 1.1, 2.1, 2.2 I Polyomial defiitio (p. 28) a x + a x +... + a x + a x 1 1 0 1 1 0 a x + a x +... +
More informationNUMERICAL METHODS COURSEWORK INFORMAL NOTES ON NUMERICAL INTEGRATION COURSEWORK
NUMERICAL METHODS COURSEWORK INFORMAL NOTES ON NUMERICAL INTEGRATION COURSEWORK For this piece of coursework studets must use the methods for umerical itegratio they meet i the Numerical Methods module
More informationGoodness-of-Fit Tests and Categorical Data Analysis (Devore Chapter Fourteen)
Goodess-of-Fit Tests ad Categorical Data Aalysis (Devore Chapter Fourtee) MATH-252-01: Probability ad Statistics II Sprig 2019 Cotets 1 Chi-Squared Tests with Kow Probabilities 1 1.1 Chi-Squared Testig................
More informationConfidence Intervals for the Population Proportion p
Cofidece Itervals for the Populatio Proportio p The cocept of cofidece itervals for the populatio proportio p is the same as the oe for, the samplig distributio of the mea, x. The structure is idetical:
More informationREVISION SHEET FP1 (MEI) ALGEBRA. Identities In mathematics, an identity is a statement which is true for all values of the variables it contains.
The mai ideas are: Idetities REVISION SHEET FP (MEI) ALGEBRA Before the exam you should kow: If a expressio is a idetity the it is true for all values of the variable it cotais The relatioships betwee
More informationPrinciple Of Superposition
ecture 5: PREIMINRY CONCEP O RUCUR NYI Priciple Of uperpositio Mathematically, the priciple of superpositio is stated as ( a ) G( a ) G( ) G a a or for a liear structural system, the respose at a give
More information10. Comparative Tests among Spatial Regression Models. Here we revisit the example in Section 8.1 of estimating the mean of a normal random
Part III. Areal Data Aalysis 0. Comparative Tests amog Spatial Regressio Models While the otio of relative likelihood values for differet models is somewhat difficult to iterpret directly (as metioed above),
More informationResponse Variable denoted by y it is the variable that is to be predicted measure of the outcome of an experiment also called the dependent variable
Statistics Chapter 4 Correlatio ad Regressio If we have two (or more) variables we are usually iterested i the relatioship betwee the variables. Associatio betwee Variables Two variables are associated
More information6.867 Machine learning, lecture 7 (Jaakkola) 1
6.867 Machie learig, lecture 7 (Jaakkola) 1 Lecture topics: Kerel form of liear regressio Kerels, examples, costructio, properties Liear regressio ad kerels Cosider a slightly simpler model where we omit
More informationComplex Numbers Solutions
Complex Numbers Solutios Joseph Zoller February 7, 06 Solutios. (009 AIME I Problem ) There is a complex umber with imagiary part 64 ad a positive iteger such that Fid. [Solutio: 697] 4i + + 4i. 4i 4i
More informationStatistics 511 Additional Materials
Cofidece Itervals o mu Statistics 511 Additioal Materials This topic officially moves us from probability to statistics. We begi to discuss makig ifereces about the populatio. Oe way to differetiate probability
More informationChapter 8: Estimating with Confidence
Chapter 8: Estimatig with Cofidece Sectio 8.2 The Practice of Statistics, 4 th editio For AP* STARNES, YATES, MOORE Chapter 8 Estimatig with Cofidece 8.1 Cofidece Itervals: The Basics 8.2 8.3 Estimatig
More informationApply change-of-basis formula to rewrite x as a linear combination of eigenvectors v j.
Eigevalue-Eigevector Istructor: Nam Su Wag eigemcd Ay vector i real Euclidea space of dimesio ca be uiquely epressed as a liear combiatio of liearly idepedet vectors (ie, basis) g j, j,,, α g α g α g α
More informationU8L1: Sec Equations of Lines in R 2
MCVU Thursda Ma, Review of Equatios of a Straight Lie (-D) U8L Sec. 8.9. Equatios of Lies i R Cosider the lie passig through A (-,) with slope, as show i the diagram below. I poit slope form, the equatio
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationMa 530 Introduction to Power Series
Ma 530 Itroductio to Power Series Please ote that there is material o power series at Visual Calculus. Some of this material was used as part of the presetatio of the topics that follow. What is a Power
More informationBIOSTATS 640 Intermediate Biostatistics Frequently Asked Questions Topic 1 FAQ 1 Review of BIOSTATS 540 Introductory Biostatistics
BIOTAT 640 Itermediate Biostatistics Frequetly Asked Questios Topic FAQ Review of BIOTAT 540 Itroductory Biostatistics. I m cofused about the jargo ad otatio, especially populatio versus sample. Could
More information11 Correlation and Regression
11 Correlatio Regressio 11.1 Multivariate Data Ofte we look at data where several variables are recorded for the same idividuals or samplig uits. For example, at a coastal weather statio, we might record
More information(all terms are scalars).the minimization is clearer in sum notation:
7 Multiple liear regressio: with predictors) Depedet data set: y i i = 1, oe predictad, predictors x i,k i = 1,, k = 1, ' The forecast equatio is ŷ i = b + Use matrix otatio: k =1 b k x ik Y = y 1 y 1
More informationEfficient GMM LECTURE 12 GMM II
DECEMBER 1 010 LECTURE 1 II Efficiet The estimator depeds o the choice of the weight matrix A. The efficiet estimator is the oe that has the smallest asymptotic variace amog all estimators defied by differet
More informationWe will conclude the chapter with the study a few methods and techniques which are useful
Chapter : Coordiate geometry: I this chapter we will lear about the mai priciples of graphig i a dimesioal (D) Cartesia system of coordiates. We will focus o drawig lies ad the characteristics of the graphs
More informationStatistical Properties of OLS estimators
1 Statistical Properties of OLS estimators Liear Model: Y i = β 0 + β 1 X i + u i OLS estimators: β 0 = Y β 1X β 1 = Best Liear Ubiased Estimator (BLUE) Liear Estimator: β 0 ad β 1 are liear fuctio of
More informationLecture 15: Learning Theory: Concentration Inequalities
STAT 425: Itroductio to Noparametric Statistics Witer 208 Lecture 5: Learig Theory: Cocetratio Iequalities Istructor: Ye-Chi Che 5. Itroductio Recall that i the lecture o classificatio, we have see that
More informationMath 113 Exam 3 Practice
Math Exam Practice Exam will cover.-.9. This sheet has three sectios. The first sectio will remid you about techiques ad formulas that you should kow. The secod gives a umber of practice questios for you
More informationMath 61CM - Solutions to homework 3
Math 6CM - Solutios to homework 3 Cédric De Groote October 2 th, 208 Problem : Let F be a field, m 0 a fixed oegative iteger ad let V = {a 0 + a x + + a m x m a 0,, a m F} be the vector space cosistig
More information10-701/ Machine Learning Mid-term Exam Solution
0-70/5-78 Machie Learig Mid-term Exam Solutio Your Name: Your Adrew ID: True or False (Give oe setece explaatio) (20%). (F) For a cotiuous radom variable x ad its probability distributio fuctio p(x), it
More informationIn algebra one spends much time finding common denominators and thus simplifying rational expressions. For example:
74 The Method of Partial Fractios I algebra oe speds much time fidig commo deomiators ad thus simplifyig ratioal epressios For eample: + + + 6 5 + = + = = + + + + + ( )( ) 5 It may the seem odd to be watig
More informationBeurling Integers: Part 2
Beurlig Itegers: Part 2 Isomorphisms Devi Platt July 11, 2015 1 Prime Factorizatio Sequeces I the last article we itroduced the Beurlig geeralized itegers, which ca be represeted as a sequece of real umbers
More informationCALCULATING FIBONACCI VECTORS
THE GENERALIZED BINET FORMULA FOR CALCULATING FIBONACCI VECTORS Stuart D Aderso Departmet of Physics, Ithaca College 953 Daby Road, Ithaca NY 14850, USA email: saderso@ithacaedu ad Dai Novak Departmet
More informationMATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4
MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.
More informationAPPENDIX F Complex Numbers
APPENDIX F Complex Numbers Operatios with Complex Numbers Complex Solutios of Quadratic Equatios Polar Form of a Complex Number Powers ad Roots of Complex Numbers Operatios with Complex Numbers Some equatios
More informationREVISION SHEET FP1 (MEI) ALGEBRA. Identities In mathematics, an identity is a statement which is true for all values of the variables it contains.
the Further Mathematics etwork wwwfmetworkorguk V 07 The mai ideas are: Idetities REVISION SHEET FP (MEI) ALGEBRA Before the exam you should kow: If a expressio is a idetity the it is true for all values
More informationMath 475, Problem Set #12: Answers
Math 475, Problem Set #12: Aswers A. Chapter 8, problem 12, parts (b) ad (d). (b) S # (, 2) = 2 2, sice, from amog the 2 ways of puttig elemets ito 2 distiguishable boxes, exactly 2 of them result i oe
More informationPaired Data and Linear Correlation
Paired Data ad Liear Correlatio Example. A group of calculus studets has take two quizzes. These are their scores: Studet st Quiz Score ( data) d Quiz Score ( data) 7 5 5 0 3 0 3 4 0 5 5 5 5 6 0 8 7 0
More informationEmpirical Process Theory and Oracle Inequalities
Stat 928: Statistical Learig Theory Lecture: 10 Empirical Process Theory ad Oracle Iequalities Istructor: Sham Kakade 1 Risk vs Risk See Lecture 0 for a discussio o termiology. 2 The Uio Boud / Boferoi
More information1 Generating functions for balls in boxes
Math 566 Fall 05 Some otes o geeratig fuctios Give a sequece a 0, a, a,..., a,..., a geeratig fuctio some way of represetig the sequece as a fuctio. There are may ways to do this, with the most commo ways
More informationOrthogonal transformations
Orthogoal trasformatios October 12, 2014 1 Defiig property The squared legth of a vector is give by takig the dot product of a vector with itself, v 2 v v g ij v i v j A orthogoal trasformatio is a liear
More informationb i u x i U a i j u x i u x j
M ath 5 2 7 Fall 2 0 0 9 L ecture 1 9 N ov. 1 6, 2 0 0 9 ) S ecod- Order Elliptic Equatios: Weak S olutios 1. Defiitios. I this ad the followig two lectures we will study the boudary value problem Here
More informationTEACHER CERTIFICATION STUDY GUIDE
COMPETENCY 1. ALGEBRA SKILL 1.1 1.1a. ALGEBRAIC STRUCTURES Kow why the real ad complex umbers are each a field, ad that particular rigs are ot fields (e.g., itegers, polyomial rigs, matrix rigs) Algebra
More information5.1 Review of Singular Value Decomposition (SVD)
MGMT 69000: Topics i High-dimesioal Data Aalysis Falll 06 Lecture 5: Spectral Clusterig: Overview (cotd) ad Aalysis Lecturer: Jiamig Xu Scribe: Adarsh Barik, Taotao He, September 3, 06 Outlie Review of
More information1 Approximating Integrals using Taylor Polynomials
Seughee Ye Ma 8: Week 7 Nov Week 7 Summary This week, we will lear how we ca approximate itegrals usig Taylor series ad umerical methods. Topics Page Approximatig Itegrals usig Taylor Polyomials. Defiitios................................................
More informationMachine Learning Theory Tübingen University, WS 2016/2017 Lecture 11
Machie Learig Theory Tübige Uiversity, WS 06/07 Lecture Tolstikhi Ilya Abstract We will itroduce the otio of reproducig kerels ad associated Reproducig Kerel Hilbert Spaces (RKHS). We will cosider couple
More informationMATH 112: HOMEWORK 6 SOLUTIONS. Problem 1: Rudin, Chapter 3, Problem s k < s k < 2 + s k+1
MATH 2: HOMEWORK 6 SOLUTIONS CA PRO JIRADILOK Problem. If s = 2, ad Problem : Rudi, Chapter 3, Problem 3. s + = 2 + s ( =, 2, 3,... ), prove that {s } coverges, ad that s < 2 for =, 2, 3,.... Proof. The
More informationR is a scalar defined as follows:
Math 8. Notes o Dot Product, Cross Product, Plaes, Area, ad Volumes This lecture focuses primarily o the dot product ad its may applicatios, especially i the measuremet of agles ad scalar projectio ad
More informationUniversity of California, Los Angeles Department of Statistics. Practice problems - simple regression 2 - solutions
Uiversity of Califoria, Los Ageles Departmet of Statistics Statistics 00C Istructor: Nicolas Christou EXERCISE Aswer the followig questios: Practice problems - simple regressio - solutios a Suppose y,
More information3. Z Transform. Recall that the Fourier transform (FT) of a DT signal xn [ ] is ( ) [ ] = In order for the FT to exist in the finite magnitude sense,
3. Z Trasform Referece: Etire Chapter 3 of text. Recall that the Fourier trasform (FT) of a DT sigal x [ ] is ω ( ) [ ] X e = j jω k = xe I order for the FT to exist i the fiite magitude sese, S = x [
More informationLet us consider the following problem to warm up towards a more general statement.
Lecture 4: Sequeces with repetitios, distributig idetical objects amog distict parties, the biomial theorem, ad some properties of biomial coefficiets Refereces: Relevat parts of chapter 15 of the Math
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More information[ 11 ] z of degree 2 as both degree 2 each. The degree of a polynomial in n variables is the maximum of the degrees of its terms.
[ 11 ] 1 1.1 Polyomial Fuctios 1 Algebra Ay fuctio f ( x) ax a1x... a1x a0 is a polyomial fuctio if ai ( i 0,1,,,..., ) is a costat which belogs to the set of real umbers ad the idices,, 1,...,1 are atural
More informationSection 14. Simple linear regression.
Sectio 14 Simple liear regressio. Let us look at the cigarette dataset from [1] (available to dowload from joural s website) ad []. The cigarette dataset cotais measuremets of tar, icotie, weight ad carbo
More informationMATH 304: MIDTERM EXAM SOLUTIONS
MATH 304: MIDTERM EXAM SOLUTIONS [The problems are each worth five poits, except for problem 8, which is worth 8 poits. Thus there are 43 possible poits.] 1. Use the Euclidea algorithm to fid the greatest
More informationRademacher Complexity
EECS 598: Statistical Learig Theory, Witer 204 Topic 0 Rademacher Complexity Lecturer: Clayto Scott Scribe: Ya Deg, Kevi Moo Disclaimer: These otes have ot bee subjected to the usual scrutiy reserved for
More informationNUMERICAL METHODS FOR SOLVING EQUATIONS
Mathematics Revisio Guides Numerical Methods for Solvig Equatios Page 1 of 11 M.K. HOME TUITION Mathematics Revisio Guides Level: GCSE Higher Tier NUMERICAL METHODS FOR SOLVING EQUATIONS Versio:. Date:
More informationmultiplies all measures of center and the standard deviation and range by k, while the variance is multiplied by k 2.
Lesso 3- Lesso 3- Scale Chages of Data Vocabulary scale chage of a data set scale factor scale image BIG IDEA Multiplyig every umber i a data set by k multiplies all measures of ceter ad the stadard deviatio
More information