Mean Vector Inferences

Mean Vector Inferences Lecture 5 September 21, 2005 Multivariate Analysis Lecture #5-9/21/2005 Slide 1 of 34

Today s Lecture Inferences about a Mean Vector (Chapter 5). Univariate versions of mean vector inferences. Hypothesis tests. Confidence intervals. Inferences about multivariate means (mean vector). Today s Lecture Multivariate hypothesis tests. Confidence ellipsoids. Note: Homework # 3 due in my mailbox on Thursday (by 5pm). Lecture #5-9/21/2005 Slide 2 of 34

/Recurring Themes Inference - reaching valid conclusions concerning a population on the basis of information obtained from a sample. Univariate statistics perform inferences for a single variable at a time. Multivariate statistics perform inferences for a set of variables simultaneously. A recurring theme of multivariate statistics is that correlated variables (for instance, p of them) should be analyzed jointly. Data Collection You will discover why as today s lecture progresses. Lecture #5-9/21/2005 Slide 3 of 34

Data Collection Please pull out a piece of paper and write down: Your height in inches. An estimate of the number of inches from your elbow to your wrist. Data Collection Please omit any identifying remarks. I am compelled to mention that you can choose to not participate and yet still get credit for this course. I am also compelled to mention that this example may end up being very bad... Lecture #5-9/21/2005 Slide 4 of 34

A Wager For whatever reason, you believe the average height of graduate students at KU is 68 inches. Your friend, the contrarian, thinks otherwise and makes you a bet that the average height of graduate students is not 68 inches. Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals You collect data on a single variable: a person s height. How can you tell who wins the bet? Lecture #5-9/21/2005 Slide 5 of 34

Univariate Hypothesis Testing Imagine that you recall hypothesis testing and formulate your wager so as to gain statistical evidence either in support for or against your conjecture that graduate students at KU are, on average, 68 inches tall. The statistical null hypothesis would look a little something like: Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals H 0 : µ = 68 inches And, because your friend said that, on average, KU grad students were not 68 inches (as opposed to saying they were taller than 68 inches or shorter than 68 inches), the statistical null hypothesis test would look like: H 1 : µ 68 inches Lecture #5-9/21/2005 Slide 6 of 34

Univariate Hypothesis Testing More formally, hypothesis tests consist of a null hypothesis which is compared with an alternative hypothesis: H 0 : µ = µ 0 Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals H 1 : µ µ 0 The alternative hypothesis here is for a two-sided test (stemming from the ). One-sided hypothesis tests are equally valid given sound substantive reason for using such tests. Lecture #5-9/21/2005 Slide 7 of 34

Hypothesis Evaluation To evaluate the hypothesis, a test statistic must be created, and then compared with the distribution assumed under the null hypothesis. If the sample X 1, X 2,, X n come from a normal population, then the appropriate test statistic is: Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals where: t = ( X µ 0 ) s/ n X = 1 n s 2 = 1 n 1 n j=1 X j n ( Xj X ) 2 j=1 Lecture #5-9/21/2005 Slide 8 of 34

Hypothesis Evaluation The t statistic has a t-distribution with n 1 degrees of freedom (df). H 0 is rejected if t is greater than a specified percentage point of a t-distribution with n 1 df. Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals For instance, lets look at the values for the means of our data. SAS Example #1... Lecture #5-9/21/2005 Slide 9 of 34

Hypothesis Evaluation Note that rejecting H 0 when t is greater than t n 1 (α/2) is equivalent to rejecting H 0 when t 2 is greater than t 2 n 1(α/2): t 2 = ( X µ 0 ) 2 s 2 /n = n( X µ 0 )(s 2 ) 1 ( X µ 0 ) Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals t 2 is the square of the distance from the sample mean X to the test value µ 0, expressed in standard deviations of X (s/ n). H 0 is rejected at significance level α if: n( X µ 0 )(s 2 ) 1 ( X µ 0 ) > t 2 n 1(α/2) Lecture #5-9/21/2005 Slide 10 of 34

H 0 Not Rejected Imagine, for instance, that for your wager you conclude that H 0 is not rejected, or that there is not enough statistical evidence to conclude that the average height of KU grad students is 68 inches. Does this mean that you were correct, that 68 inches is the average height of KU grad students? Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals Are there other values of µ that would be also be consistent with the data? Could you pick another µ 0 such that you would fail to reject H 0? Lecture #5-9/21/2005 Slide 11 of 34

Confidence Intervals For the hypothesis test you just constructed, the reality is that there could be a range of values µ where you could not detect differences from your sample mean. This region is at the heart of the concept of confidence intervals. Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals Recall that a confidence interval gives us a range around the sample mean ( X) where the population mean (µ) lies with a given probability 1 α. Specifically, we say that with 100 (1 α)% confidence that µ 0 lies within: x ± t n 1 (α/2) s n Lecture #5-9/21/2005 Slide 12 of 34

Confidence Interval Example In reality, what does statistical confidence translate into? The statistical confidence level relays the probability that the mean µ falls within a specified interval. This is akin to having α% of confidence intervals not contain the true population mean. Imagine you set α = 0.05, then take a set of 100 random samples from a population. About 5 of those 100 samples would have confidence intervals where µ is not contained in the interval. Hypothesis Testing Hypothesis Evaluation H 0 Not Rejected Confidence Intervals In class example... Lecture #5-9/21/2005 Slide 13 of 34

In univariate statistics we were concerned......with making hypotheses based on a single mean µ 0....determining the location of the mean on a single interval. Multivariate statistics, however, shift the focus from a single mean (or single interval) to making inferences about multiple means (a mean vector). The problem is phrased as determining the whether a p 1 vector µ 0 is a plausible value for the true mean of a multivariate distribution. Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information Lecture #5-9/21/2005 Slide 14 of 34

A Wager, Part II Imagine, for instance, that you had a feeling that: The average height of KU grad students was 68 inches. -and- The average distance from the elbow to the wrist for KU grad students was 10 inches. Your other friend, another contrarian, thinks otherwise and makes you a bet that the average height of graduate students is not 68 inches and that the average forearm distance is not 10 inches. Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information You collect data on a both variables. How can you tell who wins the bet? Lecture #5-9/21/2005 Slide 15 of 34

Multivariate Hypotheses As with the univariate analog of the wager, a multivariate hypothesis test can be constructed. In multivariate, however, each hypothesis would be phrased in terms of mean vectors rather than a single mean value (scalar) as was done with univariate statistics: Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information H 0 : µ = µ 0 H 1 : µ µ 0 Or, phrased in terms of your wager: H 0 : µ = H 1 : µ [ [ 68 10 68 10 ] ] Lecture #5-9/21/2005 Slide 16 of 34

Why Use Multivariate? But why would we want to do use a multivariate test for a mean when we already know how to test each variable individually Controls for Type I error Univariate tests ignore associations between the variables (i.e., ignore the multivariate structure). Multivariate tests can be more powerful (i.e. there may be cases where each univariate test would not be significant, BUT the multivariate test is). Multivariate tests can reveal more about the variables (i.e., provide insight about why we rejected H 0 ). Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information Lecture #5-9/21/2005 Slide 17 of 34

Hotelling s T 2 To evaluate the multivariate hypothesis test, a test statistic must be constructed. This test statistic is called Hotelling s T 2 : where: T 2 = n ( X µ0 ) S 1 ( X µ0 ) S p p = 1 n 1 X p 1 = 1 n µ 0 = n i=1 X i n (X i X) (X i X) i=1 µ 10 µ 20. Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information µ p0 Lecture #5-9/21/2005 Slide 18 of 34

Multivariate Hypothesis Evaluation The T 2 statistic has a scaled F-distribution: (n 1)p (n p) F p,n p H 0 is rejected if T 2 is greater than a specified percentage point of the F distribution with p df numerator and n p degrees of freedom denominator. For instance, lets look at the values for the means of our data. SAS Example #2... Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information Lecture #5-9/21/2005 Slide 19 of 34

Invariance An appealing property of the T 2 statistic is that it is invariant (unchanged) under changes in the units of measurement for X of the form: Y p 1 = C p p X p 1 + d p 1 Multivariate Hypotheses Why Multivariate? Hotelling s T 2 MV Hypothesis Evaluation T 2 Information In our example, this means that even if we changed our units of measure from inches to meters (1 IN = 0.0254 M), our T 2 statistic will be unchanged. SAS Example #3... Lecture #5-9/21/2005 Slide 20 of 34

Note: This section is developed only for purposes of introducing new statistical terminology. A common method for conducting hypothesis tests in multivariate statistics is by use of likelihood ratios. A likelihood ratio is formed by comparing the likelihood of the data under the the null hypothesis with the likelihood of the data overall. For large samples (with frequently used distributions), LR tests have some very appealing statistical properties (known distributions, most powerful tests). T 2 LR The hypothesis test for the multivariate mean vector can be expressed in terms of a likelihood ratio. As we generalize from a mean vector to multiple mean vectors (think going from t-tests to ANOVA), LR tests will become our predominant way of evaluating multivariate hypotheses. Lecture #5-9/21/2005 Slide 21 of 34

The LR for comparing multivariate means is computed by: where Λ = max Σ L(µ 0,Σ) max µ,σ L(µ,Σ) = ˆΣ = 1 n ˆΣ 0 = 1 n ( ˆΣ ˆΣ 0 n (x i x) (x i x) i=1 n (x i µ 0 ) (x i µ 0 ) i=1 Λ can be expressed as a function of T 2 : ) (n/2) T 2 LR Λ 2/n = (1 + T 2 (n 1) ) 1 Lecture #5-9/21/2005 Slide 22 of 34

Typically, when the sample size is large, the distribution of a function Λ ( 2 lnλ) under the null hypothesis is approximately χ 2 where the degrees of freedom are reflected by the dimensionality of the hypotheses under study. 2 lnλ is something you may get used to seeing when using multivariate statistics. T 2 LR Lecture #5-9/21/2005 Slide 23 of 34

Just as with univariate statistics, we can construct confidence intervals for the mean vector for multivariate inference. These intervals are no longer for a single number, but for a set of numbers contained by the mean vector. The term Confidence Region is used to describe the multivariate confidence intervals. In general, a 100 (1 α)% confidence region for the mean vector of a p-dimensional normal distribution is the ellipsoid determined by all µ such that: Building CRs n ( X µ ) S 1 ( X µ ) = p(n 1) (n p) F p,n p(α) Lecture #5-9/21/2005 Slide 24 of 34

Building CRs - Population To build confidence regions, recall our last lecture about the multivariate normal distribution... Specifically: (x µ)σ 1 (x µ) = χ 2 p (α) Building CRs provides the confidence region containing 1 α of the probability mass of the MVN distribution. We then calculated the axes of the ellipsoid by computing the eigenvalues and eigenvectors of the covariance matrix Σ: Specifically: (x µ)σ 1 (x µ) = c 2 has ellipsoids centered at µ, and has axes ±c λ i e i. Lecture #5-9/21/2005 Slide 25 of 34

Building CRs - Sample A similar function is used to develop the confidence region for the multivariate mean vector based on the sample mean ( x) and covariance matrix ( S). Note that because we are taking a sample rather than the population, the distribution of the squared statistical distance is no longer χ 2 p(α) but rather p(n 1) (n p) F p,n p(α) Building CRs This means that the confidence region is centered at ( x), and has axes ± λ i p(n 1) (n p) F p,n p(α)e i. SAS Example #4... Lecture #5-9/21/2005 Slide 26 of 34

An advantage to constructing confidence regions lies in the reduction (projections) of such regions to marginal (univariate) variables. For instance, in our wager example, we already know what the confidence region would look like for both height and forearm length, but is there any way we could use the multivariate power to produce a set of univariate confidence intervals for height and forearm length that controls the overall error rate? The answer, of course, is yes (done in concert with both variables)... The method lies in creating linear combinations of the multivariate variables. Lecture #5-9/21/2005 Slide 27 of 34

Imagine you collect a multivariate random sample: x 1, x 2,...,x p (from N p (µ,σ). Imagine then that you combine all multivariate variables into a new composite variable z j : z j = a 1 x j1 + a 2 x j2 +... + a p x jp Using what we learned about linear combinations of variables that were jointly distributed multivariate normal, we know that Z has a normal distribution with: z = a x s 2 z = a S x a Lecture #5-9/21/2005 Slide 28 of 34

The 100 (1 α)% confidence interval for µ Z = aµ x is given by: a x ± p(n 2) (n p) F p,n p(α)a S x a Lecture #5-9/21/2005 Slide 29 of 34

Reasonable Combinations? What are the coefficients of a? Consider: A = 1 0 0 0 0 1 0 0........ 0 0 0 1 Here we are constructing simultaneous univariate confidence intervals for each variable p in our data set. These will capitalize on the multivariate distribution of the variables (if correlated). SAS Example # 5... Lecture #5-9/21/2005 Slide 30 of 34

Large Sample Inferences Recall that the t statistic approaches normality as n gets large. Similarly, as n gets large, the distribution of T 2 approximates the χ 2 formulated by the exponent of the multivariate normal distribution. Practically speaking, the formula for the value of F will approximate the χ 2 distribution value, although you can change the critical value if you feel the need to. Large Samples For more information see Section 5.5. Lecture #5-9/21/2005 Slide 31 of 34

Multivariate quality control charts are discussed in Section 5.6. The methods used to inspect such charts are based on T 2 and linear combinations of data. Inferences about the mean vector when some observations are missing are discussed in Section 5.7. This discussion quickly moves past the scope of this course, and for purposes of practicality, I am omitting it from lecture. Large Samples Corrections for serially correlated observations are discussed in Section 5.8. Lecture #5-9/21/2005 Slide 32 of 34

Final Thought We have begun our discussion of multivariate inferential statistics with methods analogous to the univariate t-test. Today we only discussed testing the mean vector versus some stable quantity. Final Thought Next Class As we move forward, recall how your univariate courses moved from t-tests to ANOVA. Such transitions will become apparent in the next two weeks. Lecture #5-9/21/2005 Slide 33 of 34

Next Time Inferences for Multiple Mean Vectors. Paired comparisons. MANOVA. Profile analysis. Growth Curves. More simultaneous confidence intervals. Final Thought Next Class Lecture #5-9/21/2005 Slide 34 of 34