3 The Scalar Algebra of Variances, Covariances, and Correlations

Transcription

1 3 The Scalar Algebra of Variaces, Covariaces, ad Correlatios I this chapter, we review the defiitios of some key statistical cocepts: meas, covariaces, ad correlatios. We show how the meas, variaces, covariaces, ad correlatios of variables are related whe the variables themselves are coected by oe or more liear equatios by developig the liear combiatio ad trasformatio rules. These rules, which hold both for lists of umbers ad radom variables, form the algebraic foudatios for regressio aalysis, factor aalysis, ad SEM. 3.1 Meas, Variaces, Covariaces, ad Correlatios I this sectio, we quickly review the basic defiitios of the statistical quatities at the heart of structural equatio modelig. Our approach is to keep the mathematical statistical cotet to a miimum while still coveyig the importat details Meas The mea of a statistical populatio is simply the arithmetic average of all the umbers i the populatio. Usually, we model statistical populatios as radom variables with a particular statistical distributio. Sice this is ot a course i probability theory, we will ot review the formal defiitio of a radom variable as a fuctio assigig umbers to outcomes, but rather use a very iformal otio of such a variable as a process that geerates umbers accordig to a specifiable system. The mea of the populatio ca be represeted as the expected value of the

2 42 structural equatio modelig radom variable. Defiitio (Expected Value of a Radom Variable) The expected value of a radom variable X, deoted E(X), is the log ru average of the umbers geerated by the radom variable X. This defiitio will usually prove adequate for our purposes i this text, although the vetera of a course or two i statistics udoubtedly kows the formal defiitios i terms of sums ad itegrals, ad recogizes that the formal defiitios are differet for discrete ad cotiuous radom variables. The arithmetic average of a sample of umbers take from some statistical populatio is referred to as the sample mea. This was defied already i sectio Deviatio Scores The deviatio score plays a importat role i statistical formulas. The defiitios are quite similar, for the sample ad the populatio. I each case, the deviatio score is simply the differece betwee the score ad the mea. Deviatio scores i a statistical populatio are obtaied by subtractig the populatio mea from every score i the populatio. If the populatio is modeled with a radom variable, the radom variable is coverted to deviatio score form by subtractig the populatio mea from each observatio, as described i the followig defiitio. Defiitio (Deviatio Score Radom Variable) The deviatio score form of a radom variable X is defied as [dx] = X E(X). Sample deviatio scores were defied aalogously i sectio

3 the scalar algebra of variaces, covariaces, ad correlatios Variaces ad Stadard Deviatios The variace of a statistical sample or populatio is a measure of how far the umbers are spread out aroud the mea. Sice the deviatio score reflects distace from the mea, perhaps the two most obvious measures of spread are the average absolute deviatio ad the average squared deviatio. For reasos of mathematical tractability, the latter is much preferred by iferetial statisticias. Defiitio (Populatio Variace) The variace of a statistical populatio, deoted Var(X) or σx 2, is the average squared deviatio score i that populatio. Hece, σx 2 = E (X E(X))2. A importat relatioship, which we give here without proof, is Defiitio (Populatio Variace Computatioal Formula) σx (X 2 = E 2) (E(X)) 2 (3.1) ( = E X 2) µ 2 (3.2) The populatio stadard deviatio relates to the variace aalogously to the correspodig sample quatities. Defiitio (The Populatio Stadard Deviatio) The populatio stadard deviatio is the square root of the populatio variace, i.e., σ X = σx 2 The atural trasfer of the otio of a populatio variace to a sample statistic would be to simply compute, for a sample of umbers, the average squared (sample)

4 44 structural equatio modelig deviatio score. Ufortuately, such a estimate teds to be, o average, lower tha the populatio variace, i.e., egatively biased. To correct this bias, we divide the sum of squared deviatios by 1, istead of. Defiitio (The Sample Variace) The sample variace for a set of scores o the variable X is defied as S 2 X = 1/( 1) (X i X ) 2 As a elemetary exercise i summatio algebra, it ca be show that the sample variace may also be computed via the followig equivalet, but more coveiet formula: ( S 2 X = 1/( 1) Xi 2 ( X i) 2 ) Defiitio (The Sample Stadard Deviatio) The sample stadard deviatio is the square root of the sample variace, i.e., S X = S 2 X 3.2 The Algebra of Meas The liear trasformatio is a importat cocept i statistics, because may elemetary statistical formulas ivolve liear trasformatios. Defiitio (Liear Trasformatio) Suppose we have two variables, X ad Y, represetig either radom variables or two lists of umbers. Suppose furthermore that the Y scores ca be expressed as liear fuctios of the X scores, that is, Y i = ax i + b for some costats a ad b. The we say that Y

5 the scalar algebra of variaces, covariaces, ad correlatios 45 is a liear trasformatio (or liear trasform ) of X. If the multiplicative costat a is positive, the the liear trasformatio is order-preservig The Liear Trasform Rule Both sample ad populatio meas behave i a predictable ad coveiet way if oe variable is a liear trasformatio of the other. Theorem (The Sample Mea of a Liear Trasform) Let Y ad X represet two lists of umbers. If, for each Y i, we have Y i = ax i + b, the Y = ax + b. Proof. For lists of umbers, we simply substitute defiitios ad employ elemetary summatio algebra. That is, Y = (1/) = (1/) = (1/) = a(1/) Y i = ax + b (ax i + b) ax i + (1/) X i + (1/)b b The secod lie above follows from the distributive rule of summatio algebra, give o page 23. The third lie uses the secod costat rule of summatio algebra give o page 22, for the left term, ad the first costat rule (page 20), for the right. The theorem actually icludes more fudametal rules about the sample mea as special cases. First, by lettig b = 0 ad rememberig that a ca represet either multiplicatio or divisio by a costat, we see that multiplyig or dividig a variable (or every umber i a sample) by a costat multiplies or divides its mea by the same costat. Secod, by lettig a = 1, ad recallig that b ca represet either additio or subtractio, we see that addig or subtractig a costat from a variable adds or subtracts that costat from the mea of that variable.

6 46 structural equatio modelig After quickly metioig two additioal results, we shall costruct a M-algebra or algebra of meas. Subsequetly, we ll discover that this simple set of rules allows us to derive may importat results. Theorem (Mea of a Costat) Suppose that, over a list of umbers X, X i ever varies. For example, suppose X i = b for all i. The X = b. The proof of this result is left to a exercise. Theorem (Mea of a Sum) Suppose two lists X ad Y of equal legth are added together to produce a ew list W. The W = X + Y. Proof W = 1 = 1 W i (X i + Y i ) = 1 ( X i + = 1 X i + 1 = X + Y Y i Y i ) (byequatio Costructig a M-Algebra I the precedig sectio, we derived 3 key results: (a) The mea of a liear trasformatio is the same trasformatio rule applied to the origial mea, i.e., if Y = ax + b, the Y = ax + b. (b) The mea of a variable that stays costat at b is equal to b.

7 the scalar algebra of variaces, covariaces, ad correlatios 47 (c) The mea of a sum of two variables is the sum of their meas. Now we ll simply restate these rules i a slightly differet form. I what follows, defie capital letters as variable ames ad lower-case letters as costats. Defie the mea operator M to represet the sample mea of whatever quatity it is applied to. For example, M(X) represets the sample mea of a variable X represetig a list of umbers X i. The the followig rules represet the algebra of sample meas. (a) M(a) = a Note that the third rule also implies that M(X Y) = M(X) M(Y) (b) M(aX + b) = am(x) + b (c) M(X + Y) = M(X) + M(Y) We ca use this algebra to prove may thigs about sample meas more compactly tha we ca with summatio algebra. The mai hurdle is simply gettig used to the otatio. Example (Mea of a Set of Deviatio Scores) A set of scores i the variable X ca be coverted to deviatio scores via the formula [dx] = X M(X). Now it is absolutely trivial to prove that the mea of a set of deviatio scores is zero. Here we go: The third lie combies the first rule with the fact that M(X) is itself a costat. M([dX]) = M(X M(X)) = M(X) M(M(X)) (Third Rule of M-algebra) = M(X) M(X) (First Rule of M-algebra) = The Algebra of Expected Values The algebra of sample meas we developed i the precedig sectio has a couterpart for radom variables. Recall that the mea of a radom variable X is called its expected value ad is deoted E(X). For radom variables X ad Y, ad costats a ad b, the three primary laws of expected value algebra are:

8 48 structural equatio modelig (a) E(a) = a (b) E(aX + b) = ae(x) + b (c) E(X + Y) = E(X) + E(Y) These rules for both discrete ad cotiuous radom variables are a straightforward cosequece of the formulas for expected value i the discrete ad cotiuous cases, ad we omit them here. However, the astute reader should ask the followig very serious questio: would ot ay reasoable measure of cetral tedecy have to follow these rules? 3.4 Liear Trasform Rules for Variaces ad Stadard Deviatios I this sectio, we examie the effect of a liear trasformatio o the variace ad stadard deviatio. Sice both the variace ad stadard deviatio are measures o deviatio scores, we begi by ivestigatig the effect of a liear trasformatio of raw scores o the correspodig deviatio scores. We fid that the multiplicative costat comes straight through i the deviatio scores, while a additive or subtractive costat has o effect o deviatio scores. Theorem (Liear Trasforms ad Deviatio Scores) Suppose a variable X is is trasformed liearly ito a variable Y, i.e., Y = ax + b. The the Y deviatio scores must relate to the X deviatio scores via the equatio [dy] = a[dx]. Proof First, cosider the case of sets of scores. If Y i = ax i + b, the [dy] i = Y i Y = (ax i + b) Y = (ax i + b) (ax + b) [Theorem 3.2.1] = ax i + b ax b = a(x i X ) = a[dx] i

9 the scalar algebra of variaces, covariaces, ad correlatios 49 Next, cosider the case of radom variables. We have [dy] = Y E(Y) = ax + b ae(x) + b [Theorem 3.2.1] = a(x E(X)) Oe may also prove the result for samples of size usig the M-algebra. The proof is essetially the same as the proof for radom variables, with M substituted for E throughouth. This completes the proof. = a[dx] A simple umerical example will help cocretize these results. Cosider the origial set of scores show i Table 3.1. Their deviatio score equivalets are 1, 0, ad +1, as show. Suppose we geerate a ew set of Y scores usig the equatio Y = 2X + 5. The ew deviatio scores are twice the old set of deviatio scores. The multiplicative costat 2 has a effect o the deviatio scores, but the additive costat 5 does ot. From the precedig results, it is easy to deduce the effect of a liear trasformatio o the variace ad stadard deviatio. Sice additive costats have o effect o deviatio scores, they caot affect either the variace or the stadard deviatio. O the other had, multiplicative costats costats come straight through i the deviatio scores, ad this is reflected i the stadard deviatio ad the variace. [dx] X Y = 2X + 5 [dy] Table 3.1: Effect of a Liear Trasform o Deviatio Scores Theorem (Effect of a Liear Trasform o the Variace ad SD) Suppose a variable X is trasformed ito Y via the liear trasform Y = ax + b. The, for radom variables, the followig results hold. σ Y = a σ X σ 2 Y = a2 σ 2 X Similar results hold for samples of umbers, i.e., S Y = a S X S 2 Y = a2 S 2 X

10 50 structural equatio modelig Proof Cosider the case of the variace of sample scores. By defiitio, S 2 Y = 1/( 1) = 1/( 1) = 1/( 1) N = a 2 (1/( 1) = a 2 S 2 X For radom variables, we have σ 2 Y = E(Y E(Y)) 2 [dy] 2 i (a[dy] i ) 2 (from Theorem 3.4.1) a 2 [dx] 2 i [dx] 2 i = E(aX + b E(aX + b)) 2 = E(aX + b (ae(x) + b)) 2 (from Theorem 3.2.1) = E(aX + b ae(x) b) 2 = E(aX ae(x)) 2 = E(a 2 (X E(X)) 2 ) = a 2 E(X E(X)) 2 (from 2d rule, Sectio 3.3) = a 2 σ 2 X ) 3.5 Stadard Scores Havig deduced the effect of a liear trasformatio o the mea ad variace of a radom variable or a set of sample scores, we are ow better equipped to uderstad the otio of stadard scores. Stadard scores are scores that have a particular desired mea ad stadard deviatio. So log as a set of scores are ot all equal, it

11 the scalar algebra of variaces, covariaces, ad correlatios 51 is always possible to trasform them, by a liear trasformatio, so that they have a particular desired mea ad stadard deviatio. Theorem (Liear Stadardizatio Rules) Suppose we restrict ourselves to the class of order-preservig liear trasformatios, of the form Y = ax + b, with a > 0. If a set of scores curretly has mea X ad stadard deviatio S X, ad we wish the scores to have a mea Y ad stadard deviatio S Y, we ca achieve the desired result with a liear trasformatio with a = S Y S X ad b = Y ax Similar rules hold for radom variables, i.e., a radom variable X with mea µ X ad stadard deviatio σ X may be liearly trasformed ito a radom variable Y with mea µ Y ad stadard deviatio σ Y by usig ad a = σ Y σ X b = µ Y aµ X Example (Rescalig Grades) Result is used frequetly to rescale grades i uiversity courses. Suppose, for example, the origial grades have a mea of 70 ad a stadard deviatio of 8, but the typical scale for grades is a mea of 68 ad a stadard deviatio of 12. I that case, we have a = S Y S X = 12 8 = 1.5 b = Y ax = 68 (1.5)(70) = 37

12 52 structural equatio modelig Suppose that there is a particular desired metric (i.e., desired mea ad stadard deviatio) that scores are to be reported i, ad that you were seekig a formula for trasformig ay set of data ito this desired metric. For example, you seek a formula that will trasform ay set of X i ito Y i with a mea of 500 ad a stadard deviatio of 100. Result might make it seem at first glace that such a formula does ot exist, i.e., that the formula would chage with each ew data set. This is, of course, true i oe sese. However, i aother sese, we ca write a sigle formula that will do the job o ay data set. We begi by defiig Z-scores. Defiitio (Z-Scores) Z-scores are scores that have a mea of 0 ad a stadard deviatio of 1. Ay set of scores x i that are ot all equal ca be coverted ito Z-scores by the followig trasformatio rule z i = x i x S X We say that a radom variable is i Z-score form if it has a mea of 0 ad a stadard deviatio of 1. Ay radom variable X havig expected value µ ad stadard deviatio σ may be coverted to Z-score form via the trasformatio Z = X µ σ That the Z-score trasformatio rule accomplishes its desired purpose is derived easily from the liear trasformatio rules. Specifically, whe we subtract the curret mea from a radom variable (or a list of umbers), we do ot affect the stadard deviatio, while we reduce the mea to zero. At that poit, we have a list of umbers with a mea of 0 ad the origial, uchaged stadard deviatio. Whe we ext divide by the stadard deviatio, the mea remais at zero, while the stadard deviatio chages to 1. Oce the scores are i Z-score form, the liear trasformatio rules reveal that it is very easy to trasform them ito ay other desired metric. Suppose, for example, we wished to trasform the scores to have a mea of 500 ad a stadard deviatio of 100. Sice the mea is 0 ad the stadard deviatio 1, multiplyig by 100 will chage the stadard deviatio to 100 while leavig the mea at zero. If we the

13 the scalar algebra of variaces, covariaces, ad correlatios 53 add 500 to every value, the resultig scores will have a mea of 500 ad a stadard deviatio of 100. Cosequetly, the followig trasformatio rule will chage ay set of X scores ito Y scores with a mea of 500 ad a stadard deviatio of 100, so log as the origial scores are ot all equal. ( ) xi x y i = (3.3) S X The part of Equatio 3.3 i paretheses trasforms the scores ito Z-score form. Multiplyig by 100 moves the stadard deviatio to 100 while leavig the mea at 0. Addig 500 the moves the mea to 500 while leavig the stadard deviatio at 100. Z-scores also have a importat theoretical property, i.e., their ivariace uder liear trasformatio of the raw scores. Theorem (Ivariace Property of Z-Scores) Suppose you have two variables X ad Y, ad Y is a order-preservig liear trasformatio of X. Cosider X ad Y trasformed to Z-score variables [zx] ad [zy]. The [zx] = [zy]. The result holds for sets of sample scores, or for radom variables. Proof Suppose Y = ax + b, ad assume a > 0. The, by Theorems?? ad??, E(Y) = ae(x) + b, ad σ Y = aσ X. We the have [zy] = Y E(Y) σ Y ax + b (ae(x) + b) = aσ X = a(x E(X)) + (b b) aσ X = X E(X) σx = [zx] To costruct a simple example that dramatizes this result, we will employ a well-kow fudametal result i statistics.

14 54 structural equatio modelig Example (Properties of 3 Evely Spaced Numbers) Cosider a set of 3 evely spaced umbers, with middle value M ad spacig D, the differece betwee adjacet umbers. For example, the umbers 3,6,9 have M = 6 ad D = 3. Prove that, for ay such set of umbers, the sample mea X is equal to the middle value M, ad the sample stadard deviatio S X is equal to D, the spacig. Solutio The proof is straightforward if you set up the otatio properly. We may represet the ordered list of 3 umbers as X 1 = M D, X 2 = M, ad X 3 = M + D. We the have X = X i = 1 3 (X 1 + X 2 + X 3 ) = 1 (M D + M + M + D) 3 = 1 3 (3M) = M We the have [dx] 1 = D, [dx] 2 = 0, ad [dx] 3 = D, whece S 2 X = [dx] 2 i = 1 2 ([dx]2 1 + [dx]2 2 + [dx]2 3 ) = 1 2 (D D 2 ) = 1 2 (2D2 ) = D 2

15 the scalar algebra of variaces, covariaces, ad correlatios 55 [zx] X Y = 2X 5 [zy] Table 3.2: Ivariace of Z-Scores Uder Liear Trasformatio ad so S X = D. This result makes it easy to costruct simple data sets with a kow mea ad stadard deviatio. It also makes it easy to compute the mea ad variace of 3 evely spaced umbers by ispectio. Example (Ivariace of Z-Scores Uder Liear Trasformatio) Cosider the data i Table 3.2. The X raw scores are evely spaced, so it is easy to see they have a mea of 100 ad a stadard deviatio of 15. The [zx] scores are +1, 0, ad 1 respectively. Now, suppose we liearly trasform the X raw scores ito Y scores with the equatio Y = 2X 5. The ew scores, as ca be predicted from the liear trasformatio rules, will have a mea give by Y = 2x 5 = 2(100) 5 = 195, ad a stadard deviatio of S Y = 2S X = 2(15) = 30 Sice the Y raw scores were obtaied from the X scores by liear trasformatio, we ca tell, without computig them, that the [zy] scores must also be +1, 0, ad 1. This is easily verified by ispectio. A importat implicatio of Result is that ay statistic that is solely a fuctio of Z-scores must be ivariat uder liear trasformatios of the observed variables.

16 56 structural equatio modelig 3.6 Covariaces ad Correlatios Multivariate aalysis deals with the study of sets of variables, observed o oe or more populatios, ad the way these variables covary, or behave joitly. A atural measure of how variables vary together is the covariace, based o the followig simple idea. Suppose there is a depedecy betwee two variables X ad Y, reflected i the fact that high values of X ted to be associated with high values of Y, ad low values of X ted to be associated with low values of Y. We might refer to this as a direct (positive) relatioship betwee X ad Y. If such a relatioship exists, ad we covert X ad Y to deviatio score form, the cross-product of the deviatio scores should ted to be positive, because whe a X score is above its mea, a Y score will ted to be above its mea (i which case both deviatio scores will be positive), ad whe X is below its mea, Y will ted to be below its mea (i which case both deviatio scores will be egative, ad their product positive). Covariace is simply the average cross-product of deviatio scores. Defiitio (Populatio Covariace) The covariace of two radom variables, X ad Y, deoted σ XY, is the average cross-product of deviatio scores, computed as σ XY = E [(X E(X))(Y E(Y))] (3.4) = E(XY) E(X)E(Y) (3.5) Provig the equality of Equatios 3.4 ad 3.5 is a elemetary exercise i expected value algebra that is give i may mathematical statistics texts. Equatio 3.5 is geerally more coveiet i simple proofs. Whe defiig a sample covariace, we average by 1 istead of, to maitai cosistecy with the formulas for the sample variace ad stadard deviatio.

17 the scalar algebra of variaces, covariaces, ad correlatios 57 Defiitio (Sample Covariace) The sample covariace of X ad Y is defied as S XY = 1/( 1) = 1/( 1) = 1/( 1) [dx] i [dy] i (3.6) ( Y i Y i X i Y i (X i X )(Y i Y ) (3.7) ) (3.8) Note that the variace of a variable may be thought of as its covariace with itself. Hece, i some mathematical discussios, the otatio σ XX is used (i.e., the covariace of X with itself) is used to sigify the variace of X. The covariace, beig a measure o deviatio scores, is ivariat uder additio or subtractio of a costat. O the other had, multiplicatio or divisio of a costat affects the covariace, as described i the followig result. Theorem (Effect of a Liear Trasform o Covariace) Suppose the variables X ad Y are trasformed ito the variables W ad M via the liear trasformatios W = ax + b ad M = cy + d. The the covariace betwee the ew variables W ad M relates to the origial covariace i the followig way. For radom variables, we have For samples of scores, we have σ WM = acσ XY S WM = acs XY

18 58 structural equatio modelig Proof By Theorem 3.4.1, we kow that [dw] = a[dx] ad [dm] = c[dy]. So, for sample data, we have S WM = For radom variables, we have = = ac 1 = acs XY [dw] i [dm] i a[dx] i c[dy] i σ WM = E[dW][dM] [dx] i [dy] i = E(a[dX]c[dY]) = ace([dx][dy]) = acσ XY Covariaces are importat i statistical theory. (If they were ot, why would we be studyig the aalysis of covariace structures?) However, the covariace betwee two variables does ot have a stable iterpretatio if the variables chage scale. For example, the covariace betwee height ad weight chages if you measure height i iches as opposed to cetimeters. The value will be 2.54 times as large if you measure height i cetimeters. So, for example, if someoe says The covariace betwee height ad weight is 65, it is impossible to tell what this implies without kowig the measuremet scales for height ad weight The Populatio Correlatio As we oted i the precedig sectio, the covariace is ot a scale-free measure of covariatio. To stabilize the covariace across chages i scale, oe eed oly divide it by the product of the stadard deviatios of the two variables. The resultig statistic, ρ xy, the Pearso Product Momet Correlatio Coefficiet, is defied as

19 the scalar algebra of variaces, covariaces, ad correlatios 59 the average cross-product of Z scores. From Result 3.5.3, we ca deduce without effort oe importat characteristic of the Pearso correlatio, i.e., it is ivariat uder chages of scale of the variables. So, for example, the correlatio betwee height ad weight will be the same whether you measure height i iches or cetimeters, weight i pouds or kilograms. Defiitio (The Populatio Correlatio ρ xy ) For two radom variables X ad Y the Pearso Product Momet Correlatio Coefficiet, is defied as Alteratively, oe may defie the correlatio as ρ XY = E([zX][zY]) (3.9) ρ XY = σ XY σ X σ Y (3.10) The Sample Correlatio The sample correlatio coefficiet is defied aalogously to the populatio quatity. We have Defiitio (The Sample Correlatio r xy ) For pairs of scores X i ad Y i the Pearso Product Momet Correlatio Coefficiet, is defied as r XY = 1/( 1) [zx] i [zy] i (3.11) There are a umber of alterative computatioal formulas, such as r XY = S XY S X S Y, (3.12)

20 60 structural equatio modelig or its fully explicit expasio r XY = X i Y i X i Y i ( X 2 i ( X i ) 2) ( Y 2 i ( Y i ) 2) (3.13) Sice all summatios i Equatio 3.13 ru from 1 to, I have simplified the otatio by elimiatig them. With moder statistical software, actual had computatio of the correlatio coefficiet is a rare evet. 3.7 Liear Combiatio Rules The liear combiatio (or LC) is a key idea i statistics. While studyig factor aalysis, structural equatio modelig, ad related methods, we will ecouter liear combiatios repeatedly. Uderstadig how they behave is importat to the study of SEM. I this sectio, we defie liear combiatios, ad describe their statistical behavior Defiitio of a Liear Combiatio Defiitio (Liear Combiatio) A liear combiatio (or liear composite) L of J radom variables X j is ay weighted sum of those variables, i.e.., L = J c j X j (3.14) j=1 For sample data, i a data matrix X with rows ad J colums, represetig scores o each of J variables, a liear combiatio score for the ith set of observatios is ay expressio of the form L i = J c j X ij (3.15) j=1 The c j i Equatios 3.14 ad 3.15 are called liear weights.

21 the scalar algebra of variaces, covariaces, ad correlatios 61 It is importat to be able to examie a statistical formula, decide whether or ot it cotais or represets a LC, ad idetify the liear weights, with their correct sig. A simple example is give below. Example (Course Grades as a Liear Combiatio) Suppose a istructor aouces at the begiig of a course that the course grade will be produced by takig the mid-term exam grade, addig twice the fial exam grade, ad dividig by 3. If the fial grade is G, the mid-term is X, ad the fial exam Y, the the grades are computed usig the formula G = X + 2Y 3 = 1 3 X Y (3.16) A example of such a situatio is show i Table 3.3. I this case, the course grade is a liear combiatio, ad the liear weights are +1/3 ad +2/3. Note: Table 3.3 gives the meas ad stadard deviatios for the mid-term exam (X), the fial exam (Y), ad the course grade (G). I subsequet sectios of this chapter, we show how the mea ad stadard deviatio for G ca be calculated from a kowledge of the mea ad stadard deviatio of X ad Y, the liear weights that produced G, ad the covariace betwee X ad Y Mea Of A Liear Combiatio Whe several variables are liearly combied, the resultig LC has a mea that ca be expressed as a liear fuctio of the meas of origial variables. This result, give here without proof, is the foudatio of a umber of importat statistical methods. Studet Mid-term Fial Grade X Y G Judi Fred Albert Mea SD Table 3.3: Course Grades as a Liear Combiatio Theorem (Mea of a Liear Combiatio) Give J radom variables X j havig meas µ j, the mea of a liear combiatio L = J c j X j j=1

22 62 structural equatio modelig is µ L = J c j µ j j=1 For a set of N sample scores o J variables, with liear combiatio scores computed as J l i = c j X ij, j=1 the mea of the liear combiatio scores is give by l = J c j X j j=1 This simple, but very useful result allows oe to compute the mea of a LC without actually calculatig LC scores. Example (Computig the Mea of a Liear Combiatio) Cosider agai the data i Table 3.3. Oe may calculate the mea of the fial grades by calculatig the idividual grades, summig them, ad dividig by. A alterative way is to compute the mea from the meas of X ad Y. So, for example, the mea of the fial grades must be G = 1 3 X Y ( ) 1 = = = ( 2 3 ) Variace ad Covariace Of Liear Combiatios The variace of a LC also plays a very importat role i statistical theory.

23 the scalar algebra of variaces, covariaces, ad correlatios 63 Theorem (Variace of a Liear Combiatio) Give J radom variables X j havig variaces σj 2 = σ jj ad covariaces σ jk (betwee variables X j ad X k ), the variace of a liear combiatio is L = J c j X j j=1 σ 2 L = = J J j=1 k=1 J j=1 c j c k σ jk (3.17) c 2 j σ2 j + 2 J J 1 j=2 k=1 c j c k σ jk (3.18) For a set of sample scores o J variables, with liear combiatio scores computed as J L i = c j x ij, j=1 the variace of the liear combiatio scores is give by S 2 L = = J J j=1 k=1 J j=1 c j c k S jk (3.19) c 2 j S2 j + 2 J J 1 j=2 k=1 c j c k S jk (3.20) It is, of course, possible to defie more tha oe LC o the same set of variables. For example, cosider a persoality questioaire that obtais resposes from idividuals o a large umber of items. Several differet persoality scales might be calculated as liear combiatios of the items. (Items ot icluded i a scale have a liear weight of zero.) I that case, oe may be cocered with the covariace or correlatio betwee the two LCs. The followig theorem describes how the

24 64 structural equatio modelig covariace of two LCs may be calculated. Theorem (Covariace of Two Liear Combiatios) Cosider two liear combiatios L ad M of a set of J radom variables X j, havig variaces ad covariaces σ ij. (If i = j, a variace of variable i is beig referred to, otherwise a covariace betwee variables i ad j.) The two LCs are, ad L = M = J c j X j, j=1 J d j X j, j=1 The covariace σ LM betwee the two variables L ad M is give by σ LM = J J j=1 k=1 c j d k σ jk (3.21) The theorem geeralizes to liear combiatios of sample scores i the obvious way. For a set of sample scores o J variables, with liear combiatio scores computed as ad L i = M i = J c j X ij j=1 J d j X ij j=1 the covariace of the liear combiatio scores is give by S LM = J J j=1 k=1 c j d k S jk (3.22)

25 the scalar algebra of variaces, covariaces, ad correlatios 65 Some commets are i order. First, ote that Theorem icludes the result of Theorem as a special case (simply let L = M, recallig that the variace of a variable is its covariace with itself). Secod, the results of both theorems ca be expressed as a heuristic rule that is easy to apply i simple cases A Geeral Heuristic Rule for Liear Trasoforms ad Liear Combiatios We ca combie our previous rules for liear trasformatios ad combiatios ito a sigle heuristic. To compute a variace of a liear combiatio or trasformatio, for example L = X + Y, perform the followig steps: (a) Express the liear combiatio or trasformatio i algebraic form; (b) Square the expressio; X + Y (X + Y) 2 = X 2 + Y 2 + 2XY (c) Trasform the result with the followig coversio rules: (a) Wherever a squared variable appears, replace it with the variace of the variable. (b) Wherever the product of two variables appears, replace it with the covariace of the two variables. (c) Ay expressio that does ot cotai either the square of a variable or the product of two variables is elimiated. X 2 + Y 2 + 2XY σ 2 X + σ2 Y + 2σ XY To calculate the covariace of two liear combiatios, replace the first two steps i the heuristic rules as follows: (a) Write the two liear combiatios side-by-side, for example; (X + Y) (X 2Y)

26 66 structural equatio modelig (b) Compute their algebraic product; X 2 XY 2Y 2 The coversio rule described i step 3 above is the applied, yieldig, with the preset example σ 2 X 2σ XY 2σ 2 Y The above result applies idetically to radom variables or sets of sample scores. It provides a coveiet method for derivig a umber of classic results i statistics. A well kow example is the differece betwee two variables, or, i the sample case, the differece betwee two colums of umbers. Example (Variace of Differece Scores) Suppose you gather pairs of scores X i ad Y i for people, ad compute, for each perso, a differece score as D i = X i Y i Express the sample variace S 2 D of the differece scores as a fuctio of S2 X, S 2 Y, ad S XY. Solutio. First apply the heuristic rule. Squarig X Y, we obtai X 2 + Y 2 2XY. Apply the coversio rules, ad the result is S 2 D = S2 X + S2 Y 2S XY Example (Variace of Course Grades) The data i Table 3.3 give the variaces of the X ad Y tests. Sice the deviatio scores are easily calculated, we ca calculate the covariace S XY = 1/( 1) [dx][dy] betwee the two exams as 20. Sice the fial grades are a liear combiatio of the exam grades, we ca geerate a theoretical formula for the variace of the fial

27 the scalar algebra of variaces, covariaces, ad correlatios 67 grades. Sice G = 1/3X + 2/3Y, applicatio of the heuristic rule shows that S 2 G = 1/9S2 X + 4/9S2 Y + 4/9S XY = 1/9(100) + 4/9(16) + 4/9( 20) = 84/9 = 9.33 The stadard deviatio of the fial grades is thus 9.33 = Oe aspect of the data i Table 3.3 is uusual, i.e., the scores o the two exams are egatively correlated. I practice, of course, this is ulikely to happe uless the class size is very small ad the exams uusual. Normally, oe observes a substatially positive correlatio betwee grades o several exams i the same course. I the followig example, we use LC theory to explai a pheomeo, which we might call variace shrikage, that has caught may a teacher by surprise whe computig stadardized course grades. Example (Variace Shrikage i Course Grades) Suppose a istructor is teachig a course i which the grades are computed simply by averagig the marks from two exams. The istructor is told by the uiversity admiistratio that, i this course, fial grades should be stadardized to have a mea of 70 ad a stadard deviatio of 10. The istructor attempts to comply with this request. She stadardizes the grades o each exam to have a mea of 70 ad a stadard deviatio of 10. She the computes the fial grades by averagig the stadardized marks o the two exams, which, icidetally, had a correlatio of She seds the grades o to the admiistratio, but receives a memo several days later to the effect that, although her fial grades had a mea of 70, their stadard deviatio was icorrect. It was supposed to be 10, but it was oly What wet wrog? Solutio We ca explai this pheomeo usig liear combiatio theory. Sice the fial grades are obtaied averagig the grades o the two mid-terms, the grades are computed as a liear combiatio, i.e., G = (1/2)X + (1/2)Y

28 68 structural equatio modelig Usig the heuristic rule, we establish that the variace of the grades must follow the formula S 2 G = (1/4)S2 X + (1/4)S2 Y + (1/2)S XY However, sice S 2 X = S2 Y, ad S XY = S X S Y r XY = S 2 X r xy, the formula reduces to S 2 G = 1 + r xy S2 X 2 If r xy =.5, the S 2 G = (3/4)S2 X. We have show that, if the two tests correlate.5, ad each test is stadardized to yield a mea of 70 ad a stadard deviatio of 10, it is a ievitable cosequece of the laws of LC theory that the fial grades, calculated as a average of the two tests, will have a variace that is 75, ad a stadard deviatio equal to 75 = More geerally, we have demostrated that i similar situatios, if the idividual tests are stadardized to a particular mea ad variace, the average of the two tests will have a variace that shriks by a multiplicative factor of (1 + 2r xy )/2. O the other had, it is easy to verify that the mea of the fial grades will be the desired value of 70. Example (The Rich Get Richer Pheomeo) Ultimately, the hypothetical istructor i Example will have to do somethig to offset the variace shrikage. What should she do? Solutio The aswer should be apparet from our study of liear trasformatio theory. There are several ways to approach the problem, some more formal tha others. The problem is that the variace of the scores is too low, so they eed to be stretched out alog the umber lie. Ituitively, we should realize that if we subtract 70 from each studet s average grade, we will have a set of scores with a mea of 0 ad a stadard deviatio of If we the multiply each score by 10/8.66, we will have scores with a mea of 0 ad a stadard deviatio of 10. Addig back 70 to the umbers will the restore the proper mea of 70, while leavig the stadard deviatio of 10. This trasformatio

29 the scalar algebra of variaces, covariaces, ad correlatios 69 ca be writte literally as M i = (G i 70)(10/8.66) + 70 (3.23) There are, of course, alterative approaches. Oe is to trasform each studet s mark to a Z-score, the multiply by 10 ad add 70. This trasformatio ca be writte M i = 10 G i With a small amout of maipulatio, you ca see that these two trasformatio rules are equivalet. Study Equatio 3.23 briefly, ad see how it operates. Notice that it expads the idividual s deviatio score (his/her differece from the mea of 70), makig it larger tha it was before. I this case, the multiplyig factor is 10/8.66, so, for example, a grade of would be trasformed ito a grade of 80. It is easy to see that, if the idividual exam grades ad the fial mark are scaled to have the same stadard deviatio, ay studet with a exam average above the mea will receive a fial grade higher tha their exam average. O the other had, ay studet with a exam grade below average will receive a mark lower tha their exam average. For example, a perso with a grade of 61.34, ad a deviatio score of 8.66, would have the deviatio score trasformed to 10, ad the grade chaged to 60. I my udergraduate courses, I refer to this pheomeo as The Rich Get Richer, The Poor Get Poorer. This pheomeo ca be the source of cosiderable costeratio for a studet whose grades have bee below the mea, as the studet s fial grade may be lower tha ay of his/her grades o the exams. Ideed, the lower the studet s mark, the more it is reduced! O the other had, studets with cosistetly above-average performace may get a pleasat surprise if they are uaware of this effect. Liear combiatio theory ca be used to prove a umber of surprisig results i statistics. The followig example shows how to take two colums of umbers ad compute two liear combiatios o them that are guarateed to have a zero correlatio.

30 70 structural equatio modelig Example (Costructig Sets of Ucorrelated Scores) Statistics istructors sometimes costruct sample data sets with certai characteristics. Here is a simple way to costruct two colums of umbers that have a exactly zero correlatio. I will describe the method first, the we will use liear combiatio theory to explai why the scores must have a zero correlatio. The method is as follows: First, simply write dow a list of N differet umbers i a colum. Call these umbers X. Next to the X colum, write the exact same umbers, but i a differet order. Call these Y. Now, create two additioal colums of umbers, W ad M. W is the sum of X ad Y, M the differece betwee X ad Y. W ad M will have a zero correlatio. A example of the method is show i Table Usig ay geeral purpose statistical program, you ca verify that the umbers ideed have a precisely zero correlatio. Ca you explai why? X Y W = X + Y M = X Y Table 3.4: Costructig Numbers with Zero Correlatio Solutio We ca develop a geeral expressio for the correlatio betwee two colums of umbers, usig our heuristic rules for liear combiatios. Let s begi by computig the covariace betwee W ad M. First, we take the algebraic product of X + Y ad X Y. We obtai (X + Y)(X Y) = X 2 Y 2 Applyig the coversio rule, we fid that S X+Y,X Y = S 2 X S2 Y Note that the covariace betwee W ad M is urelated to the covariace betwee the origial colums of umbers, X ad Y! Moreover, if X ad Y have the same variace, the W ad M will have a covariace of exactly zero. Sice the correlatio is the covariace divided by the product of the stadard deviatios, it immediately follows that W ad M will have a correlatio of zero if ad oly if X ad Y have the same variace. If X ad Y cotai the same umbers (i permuted order), the they have the same variace, ad W ad M will have zero correlatio.