I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN

Transcription

1 Introduction Edps/Psych/Stat/ 584 Applied Multivariate Statistics Carolyn J Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN c Board of Trustees, University of Illinois Linear Algebra Slide 1 of 30

2 Outline of Multivariate Analysis covered Statistics review (notation used in class) Graphical techniques Reading: Johnson & Wichern, Chapter 1 Linear Algebra Slide 2 of 30

3 More Specific Multivariate Data: When studying complex phenomenon, we need to collect measurements (observations) on many different variables Multivariate techniques: methods to elicit information from multivariate data Most are statistical Dependent variables are numerical, metric, continuous Linear Algebra Slide 3 of 30

4 More Specific More Specific Data reduction or structural simplification: represent phenomenon as simply as possible without loosing too much information Sorting and grouping: create groups of similar objects or classify objects into well-defined groups Investigation of the dependence among variables Prediction Hypothesis testing validate substantive theory Linear Algebra Slide 4 of 30

5 Covered Linear Algebra: Useful for summarizing (representing) data and performing manipulations Covered A Few uses of Linear Combinations Back to More that We May Not Cover Geometry: Observed cases ( sampling units, individuals, etc) can be viewed as points in high dimensional spaces and multivariate techniques are designed to study the point clouds Random Sampling: Typically, we ll assume that we have sampled cases from a multivariate normal (Gaussian) distribution or statistics follow a multivariate normal Linear Transformations: Very important! When you have multiple observed measures on individuals within a sample and you want a linear combination(s) of the measures to have certain properties Linear Algebra Slide 5 of 30

6 A Few uses of Linear Combinations Covered A Few uses of Linear Combinations Back to More that We May Not Cover Simple example: X 1 = baseline measure X 2 = post intervention Interest in change: D = X 2 X 1, what is µ D? and σ 2 D? More complex: 10 variables measured on individuals who are classified as normal and abnormal where the goal is to created a linear combination of the 10 measures such that those from different groups are as different as possible on this composite You have 10 variables measured on individuals and you want to combine them into a small number of composite variables that represent most of the information in the data Linear Algebra Slide 6 of 30

7 Back to Multivariate significance tests for inference about Covered A Few uses of Linear Combinations Back to More that We May Not Cover Means and confidence regions for them Comparisons of means across samples from different populations or groups Variances, covariances (including between sets of variables) Multivariate Analysis of Variance (MANOVA): Extension of multivariate significance tests to more complex experimental designs Extension of univariate ANOVA to multiple dependent variables Linear Algebra Slide 7 of 30

8 More Covered A Few uses of Linear Combinations Back to More that We May Not Cover Principal Components Analysis A data reduction method that focuses on studying the relationship between a set of variables by creating linear combinations of the variables The linear combination (composite measure) has the largest possible variance possible Best lower dimensional representation of the data Discrimination Analysis Related to MANOVA & significance tests of means Define a linear combination of variables that maximally accounts for differences between groups Classification Analysis Allocation of individuals into groups (populations) defined on the basis of observations on multiple variables Canonical Correlation Analysis Study the relationship between two sets or batteries of variables Factor Analysis A latent variable model that hypothesizes that the dependencies between observed values on variables are due to unobserved variables Linear Algebra Slide 8 of 30

9 that We May Not Cover Structural Equation modeling: factor analysis is a special case Covered A Few uses of Linear Combinations Back to More that We May Not Cover Optimal Scaling or Non-linear Multivariate Analysis This includes correspondence analysis, homogeneity analysis and others Multidimensional Scaling, which is related to optimal scaling Cluster Analysis, which include K means and hierarchical cluster analysis Multiple (Multivariate) Regression which is multiple regression with more than one dependent variable Linear Algebra Slide 9 of 30

10 Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize Getting use to the notation we ll use Data: measurements on p variables (attributes, characteristics, etc) for each of n cases (individuals, experimental units, etc) Notation: x jk = measurement of the kth variable on the jth case j = 1,,n where n = the number of cases k = 1,,p where p = the number of variables NOTE: notation in older editions (around 4th edition I think) of J&W used the reverse notation Linear Algebra Slide 10 of 30

11 Data as an Array Data are arranged as an array or matrix of cases by variables: Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize Variable Variable Variable Variable 1 2 k p case 1 x 11 x 12 x 1k x 1p case 2 x 21 x 22 x 2k x 2p case j x j1 x j2 x jk x jp case n x n1 x n2 x nk x np This is an (n p) rectangular matrix X All matrices will be denoted by capital bold faced letters, but more on this later Linear Algebra Slide 11 of 30

12 Descriptive Statistics Suppose we have n observation on one variable: Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize x 11,x 21,,x n1 (1st column of X) Mean (arithmetic average) measure of central tendency x 1 = 1 n n j=1 If the n observations are a sample from a larger population of possible measurements, then x 1 is the sample mean In general, sample means x j1 x k = 1 n n j=1 x jk for k = 1,,p This is the mean of n observations in the kth column of X Linear Algebra Slide 12 of 30

13 Variance & Standard Deviation Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize A sample of n observation on one variable from some population: x 11,x 21,,x n1 The Sample Variance for the first variable is s 2 1 = 1 n n (x j1 x 1 ) 2 j=1 For now we ll use the MLE and divide by n In general, where k = 1,,p s kk s 2 k = 1 n n (x jk x k ) 2 j=1 Sample standard deviation is s kk for k = 1,,p, which is in the same units (scale) as the original measurements Linear Algebra Slide 13 of 30

14 Covariance Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize A sample of n observation on two variables from some population: ( ) ( ) ( ) x 11 x 12 }{{}, x 21 x 22 }{{},, x n1 x n2 }{{} 1 st case 2 nd case n th case Sample Covariance s 12 = 1 n n (x j1 x 1 )(x j2 x 2 ) j=1 1 st column of X Average (mean) product of differences (distances) of variables 1 and 2 from their respective means Properties: s 12 > 0 large values tend to occur together s 12 < 0 larger values tend to occur with small values s 12 = 0 variables are not linearly associated Linear Algebra Slide 14 of 30

15 Covariance (continued) Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize In general, s ik = 1 n n (x ji x i )(x jk x k ) j=1 When i = k, s ii = s 2 i = ith sample variance Covariances are symmetric: s ik = s ki An array of variances and covariances: Variable Variable Variable 1 2 p Variable 1 s 11 s 12 s 1p Variable 2 s 12 s 22 s 2p Variable p s 1p s 2p s pp Linear Algebra Slide 15 of 30

16 Sample Correlation Coefficients r ik = s ik sii skk = 1 n 1 n n j=1 (x ji x i )(x jk x k ) n j=1 (x 1 ji x i ) 2 n n j=1 (x jk x k ) 2 Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize for i = 1,,p and k = 1,,p r ik is a standardized version of the sample covariance To see this, form z-scores: { z ji = x ji x i z i = 0 sii s ii = 1 z jk = x jk x k skk { z k = 0 s kk = 1 Replace x ji and x jk by z ji and z jk into the formula for r ik and you ll get r ik = s ik Linear Algebra Slide 16 of 30

17 Sample Correlation Coefficients (cont) Properties 1 r ik 1 (it s dimensionless) Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize r ik measures the strength of linear relationship r ik = 0 no linear association r ik < 0 negative linear relationship r ik > 0 positive linear relationship r ik is invariant if you change the location (mean) &/or re-scale (change variance), ie, If y ji = ax ji +b and y jk = cx jk +d, then r ik = corr(x ji,x jk ) = corr(y ji,y jk ) provided that a > 0 and c > 0 or a < 0 and c < 0 Linear Algebra Slide 17 of 30

18 Sum of Squared Deviations and sum of cross-product deviations are used in multivariate, so we ll have a symbol for them Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize Define w ik = w ii = n j=1 (x ji x i ) }{{} (x jk x k ) }{{} deviation of scores from their means n j=1 (x ji x i ) 2 }{{} sums of squares for i = 1,,p and k = 1,p Linear Algebra Slide 18 of 30

19 Data as an Array Descriptive Statistics Variance & Standard Deviation Covariance Covariance (continued) Coefficients Coefficients (cont) Sum of Squared Deviations To Summarize To Summarize Our descriptive statistics can all be organized into arrays (matrices): s 11 s 12 s 1p x 1 x = s 12 s 22 s 2p S n = x p s 1p s 2p s pp R = r 11 r 12 r 1p r 12 r 22 r 2p W = w 11 w 12 w 1p w 12 w 22 w 2p r 1p r 2p r pp w 1p w 2p w pp The arrays (matrices) S, W and R are symmetric Linear Algebra Slide 19 of 30

20 Important and usefulalways Look at your Data! Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays One variables: 1 dimensional plot dot diagram (see J&W) stem-n-left histogram box-plot eg, National Parks (table 111 in J&W): Size of Park: Stem Leaf # Boxplot *-----* Multiply StemLeaf by 10**+3 Linear Algebra Slide 20 of 30

21 Histograms of National Parks Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linear Algebra Slide 21 of 30

22 Two Variables (relationship between) Scatter plots: look for patterns of association eg, Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linear Algebra Slide 22 of 30

23 Two Numerical and One discrete Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linear Algebra Slide 23 of 30

24 New Data Set From Rencher (2002) who got it from Beall (1945) Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Description: 32 males and 32 females had measures on four psychological tests The tests were x 1 = pictorial inconsistencies x 2 = paper form board x 3 = tool recognition x 4 = vocabulary Simple descriptive statistics: Variable n x s Minimum Maximum Test Test Test Test What do we learn from this? Linear Algebra Slide 24 of 30

25 Covariances and Correlations What do we learn from these? (What don t we learn?) Covariances Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Test1 Test2 Test3 Test4 Test Test Test Test Correlations: Test1 Test2 Test3 Test4 Test Test Test Test Linear Algebra Slide 25 of 30

26 All Bivariate and Univariate Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linear Algebra Slide 26 of 30

27 Three Numerical Could do a three dimensional graph: Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linear Algebra Slide 27 of 30

28 Three Numerical & One Discrete Could do a three dimensional graph and identify points for males and females: Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linear Algebra Slide 28 of 30

29 Two Types of Scatter Plots Variable Space: n points in p-dimensional space Each row (individual, case) of X is a distinct point, Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays (x j1,x j2,,x jp ) We looked at all pairwise for the four psychological tests We looked at 3-way plot If we could view p-dimensional space, we maybe able to detect patterns, clusters, similarities and/or differences between cases Subject Space or Observation Space: View the data as p points in n-dimensional space Each column is a distinct point, (x 1k,x 2k,,x nk ) A case (individual) defines an axis (more on these later) Linear Algebra Slide 29 of 30

30 Other Graphical Displays What a neat graph! versus What an interesting story! Histograms of National Parks Two Variables (relationship between) Two Numerical and One discrete New Data Set Covariances and Correlations All Bivariate and Univariate Three Numerical Three Numerical & One Discrete Two Types of Scatter Plots Other Graphical Displays Linking multiple 2-dimensional scatter plots ( brushing, highlighting particular points, size of point convey frequency, others) Stars for p 2 dimensional graphical displays Chernoff faces Interactive real-time graphical software (SAS demo) Be creative (eg, Knowing the World powerpoint) Linear Algebra Slide 30 of 30