Outline Week 1 PCA Challenge. Introduction. Multivariate Statistical Analysis. Hung Chen

Transcription

1 Introduction Multivariate Statistical Analysis Hung Chen Department of Mathematics Old Math

2 1 Outline 2 Week 1 3 PCA multivariate normal distribution typical heads Principal Components Analysis some matrix language interpretation applications 4 Challenge

3 Objective Build knowledge on the key elements in statistical analysis of multidimensional data. Develop the theory and methods, and apply to real data sets.

4 Course Outline Introduction

5 Course Outline Introduction Example 1.2: PCA

6 Course Outline Introduction Example 1.2: PCA Matrix Algebra

7 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification

8 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution

9 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions

10 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions Principal Component Analysis (PCA)

11 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions Principal Component Analysis (PCA) Factor Analysis (FA)

12 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions Principal Component Analysis (PCA) Factor Analysis (FA) Discriminant Analysis (DA): (or classification analysis)

13 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions Principal Component Analysis (PCA) Factor Analysis (FA) Discriminant Analysis (DA): (or classification analysis) Cluster Analysis (CA)

14 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions Principal Component Analysis (PCA) Factor Analysis (FA) Discriminant Analysis (DA): (or classification analysis) Cluster Analysis (CA) Multivariate Analysis of Variance (MANOVA)

15 Course Outline Introduction Example 1.2: PCA Matrix Algebra Example 1.3: Classification Random Vectors and Multivariate Normal Distribution Statistical Inferences for Multivariate Distributions Principal Component Analysis (PCA) Factor Analysis (FA) Discriminant Analysis (DA): (or classification analysis) Cluster Analysis (CA) Multivariate Analysis of Variance (MANOVA) Canonical Correlation Analysis (CCA) and Multivariate Regression Analyse (MRA)

16 References and Grading (textbook) Johnson, R.A. and Wichern, D.W. (2007) Applied Multivariate Statistical Analysis. Pearson Prentice Hall. Flury, B. (1997) A First Course In Multivariate Statistics. Springer. Srivastava, M.S. (2002) Methods of Multivariate Statistics. Wiley. Grading scheme Homework (30%) Quiz (10%), Midterm (30%), Final (30%)

17 Office Hours Instructor s office hour: Tuesday 10:00-11:00; Friday 10:10-11:10 TA r @ntu.edu.tw (grade homework and provide homework solution) Class links: Meet on Wednesday 8:10 to 10:00 and Friday 9:10 to 10:00 at 403 Freshman building.

18 Outline Introduction Explore interrelationships among multiple random variables. Old stuff: Regression and ANOVA Example 1.2 Principal Components Analysis Example 1.1 Classification of Midge Discriminant Analysis (DA): (or classification analysis) Logistic Regression Example 1.3 Wing length of Water Pipits Discriminant Analysis (DA): (or classification analysis) Logistic Regression Normal Mixtures Issues on computation

19 Introduction Multivariate refers to the presence of multiple random variables. Multivariate data are data that are thought of as the realizations of several random variables. Multivariate Analysis can be defined broadly as an inquiry into the structure of interrelationships among multiple random variables. Regression Simple: Explain variability in Y from X. Multiple: Explain variability in Y from X 1, X 2,..., X r. Anova One-way: Explain variability in Y based on X, where X is a factor.

20 (cont) Introduction The above two examples are not usually considered to be multivariate analysis. Why? We distinguish between the Response Variable (Y ) and Explanatory Variables (X s) and in the analysis we treat the X s as fixed. Only 1 random variable in each situation. (multiple regression multivariate regression) One type of analysis that is a good example of a multivariate method is correlation analysis. Correlation Analysis: measure linear association between X and Y without distinguishing response versus explanatory.

21 Data Setting: We have n objects which are often denoted by O 1,..., O n and the variables associated with the jth object are denoted by x j = (x j1,, x jk,, x jp ). Then X n p = x 1 x 2. x n = x 11 x 12 x 1p x 21 x 22 x 2p... x j1 x j2 x jp... x n1 x n2 x np

22 multivariate normal distribution Example 1.2 (Flury): Head dimensions of 200 young men n = 200 For the 1st object, the measures are (113.2, 111.7, 119.6, 53.9, 127.4, 143.6). p = 6, (x 11, x 12,, x 1p ) Can we use multivariate normal distribution to describe those six measurements? QQ plot: Check on distributional assumption. Mahalanobis distance: Formally, the Mahalanobis distance from a group of values with mean µ = (µ 1, µ 2,..., µ p ) T and covariance matrix S for a multivariate vector x = (x 1, x 2,..., x p ) T is defined as: [ (x µ) T S 1 (x µ) ] 1/2. χ 2 6 distribution

23 multivariate normal distribution Example 1.2 Characteristics of Head Dimension Data sources: Flury, B.D. and H. Riedwyl (1988) Multivariate Statistics; A Practical Approach London: Chapman and Hall. These data concern head measurements of members of the Swiss Army. The purpose of this project was to give an empirical basis to the construction of new gas masks. After having completed the data collection, the investigators were to determine k typical heads after which to model the new masks. k is a number between two and six. What are typical heads?

24 multivariate normal distribution central tendency If k = 1, what will be your choice on typical heads? How do we measure central tendency? Define loss function. (Consider one-dimensional problem first.) L 1 norm min c E X c L 2 norm min c E(X c) 2 How do you convince others that k 2 is necessary? How do we measure spread out? Variance versus Probability

25 typical heads measure variability of multivariate data Multivariate normal distribution is determined by mean vector and covariance matrix only. p sample mean: x k = n 1 n j=1 x jk p sample variance: s kk = s 2 k = n 1 n j=1 (x jk x k ) 2 p(p 1)/2 sample covariance: s ik = 1 n n (x ji x i )(x jk x k ), 1 i, j p. j=1 Sample covariance matrix: p p matrix symmetric non-negative definite

26 typical heads cont. sample correlation: r ik = s ik sii skk, 1 i, j p. Remarks: 1. 1 r ik 1 2. r ik measures the strength of linear association 3. r ik is scale invariant We would probably want the typical heads to represent the dominant directions of variability in the data. Draw a two-dimensional contour plot.

27 typical heads Example 1.2: Principal Components Analysis Swiss heads data Six readings on the dimensions of the heads of 200 twenty years old soldiers A data with 200 observations on the following 6 variables. MFB: a numeric vector, minimum frontal breadth BAM: a numeric vector, breadth of angulus mandibulae TFH: a numeric vector, true facial height LGAN: a numeric vector, length from glabella to apex nasi LTN: a numeric vector, length from tragion to nasion LTG: a numeric vector, length from tragion to gnathion

28 typical heads cont. PURPOSE: Study the variability in size and shape of young men in order to help to design a new protection mask Data: Type flury-package in the search engine google. The R project for statistical computing: cran.r-project.org data(swiss.heads) How do we describe a distribution on six-dimension random vectors?

29 Principal Components Analysis Example 1.2 Summary statistics low-dimension information: histograms and pairwise scatterplots pairs(swiss.heads) apply(swiss.heads,2,hist) means, standard deviations, and correlations Why do we pay attention to the relationship between LTG and LTN? plot(swiss.heads$ltg,swiss.heads$ltn) Objective of Principal Component Analysis (PCA): Look for a few linear combinations, which can be used to summarize the data and loses in data as little information as possible. Parsimonious summarization

30 Principal Components Analysis projection and orthogonal least squares How do we carry it out? Play with the coordinate system by shifting it around and rotating it. If we are lucky, we may find some rotated version of the coordinate system in which the data exhibits no or almost no variability in some of the coordinate directions. Then we might claim that the data is well-represented by fewer than p coordinate directions, and thus, approximate the p-dimensional data in a subspace of lower dimension.

31 Principal Components Analysis population version Let X denote a p-variate random vector with EX = µ Cov(X) = Σ, and let Y denote an orthogonal projection of X on a line in R p. i.e., Y = x 0 + bb T (X x 0 ) for some point x 0 R p and some b R p, b = 1. Then MSE(Y; X) is minimal for Y = Y (1) = µ + β 1 β T 1 (X µ), where β 1 is a normalized eigenvector associated with the largest eigenvalue λ 1 of Σ, and MSE(Y (1) ; X) = tr(σ) λ 1. How to determine x 0 and b so as to minimize MSE(Y; X)?

32 Principal Components Analysis cont. Write P = bb T. (projection matrix) Then MSE(Y; X) = E[ Y X 2 ] How to maximize tr(pσ)? Note that = E[ (I p P)(X x 0 ) 2 ] E[ (I p P)(X µ) 2 ] = tr {Cov[(I p P)X]} = tr[(i p P)Σ(I p P)] = tr[(i p P) 2 Σ] = tr(σ) tr(pσ). tr(pσ) = tr(bb T Σ) = b T Σb. We have to maximize Var(bX) over all b of unit length.

33 Principal Components Analysis constrained minimization Let h(b) = b T σb and h (b) = h(b) λ(b T b 1) where λ is a Language multiplier. The method of Language multiplier leads to h b = 2Σb 2λb. eigenvalue-eigenvector problem Σb = λb. Critical points are given by the eigenvectors β i and associated eigenvalues λ i of Σ.

34 Principal Components Analysis cont. Since Var(β T i X) = β T i Σβ i = β T i λβ i = λ i, the variance of b T X is maximized by choosing b = β 1, a normalized eigenvector associated with the largest root. We conclude that MSE(Y (1) ; X) = tr(σ) λ 1.

35 Principal Components Analysis data version Pearson s original approach: Pearson, K. (1901). On Lines and Planes of Closest Fit to Systems of Points in Space. Philosophical Magazine 2(6): 559V Let x 1,..., x n denote n data points in R p. Let x = 1 n i x i S = 1 (x i x)(x i x) T. n Let y i denote the orthogonal projections of the x i on the straight line. Find the solution in terms of empirical cdf. i

36 some matrix language Principal Components Analysis: Reduce the dimension of a data set by examining a small number of linear combinations of the original variables that explain most of the variability among the original variables. Rotations and Orthogonal Projections: A rotation matrix is a real square matrix whose transpose is its inverse and whose determinant is 1. (It geometrically corresponds to a linear map that sends vectors to a corresponding vector rotated about the origin by a fixed angle.) A p p matrix Γ is orthogonal if Γ t Γ = ΓΓ t = I p. In Euclidean geometry, a rotation is an example of an isometry, a transformation that moves points without changing the distances between them. Think of the data matrix X being the set of n points in p-dimensional space.

37 some matrix language cont. For orthogonal matrix Γ, what does the set of points W = XΓ look like? It looks exactly like X, but rotated or flipped. In particular, WW t = XΓΓ t X t = XX t. So each point remains the same distance from 0. The distance between any two points in X is the same as the distance between the corresponding points in W. The distance between points b and c is b c, so if x i is the ith row of X, and w i the ith row of W, then x i x j = w i w j. What is the objective of rotating point clouds described?

38 interpretation cont. Principal components are the orthonormal combinations that maximize the variance. The idea behind them is that variation is information, so if one has several variables, one wishes the linear combinations that capture as much of the variation in the data as possible. For p variables, there are p principal components: The first has the maximal variance any one linear combination (with norm 1) can have, the first two have the maximal total variance any two linear combinations can have, etc. For n p data matrix X, the first principal component is the p 1 vector g 1 with g 1 = 1 that maximizes the sample variance of Xg over g = 1. For a given linear combination a, the mean and variance of the elements in the vector Z = Xa are easily obtained from the mean and covariance matrix of X:

39 interpretation cont. t t z = 1 = 1 nz 1 Xa = xa, 1 n 1 n = (1, 1,..., 1) t, ( 1 X 1 n x = X 1 n n 1t nx = I n 1 ) n 1 n1 t n X = H n X, s zz = 1 n Zt H n Z = 1 n at X t H n Xa = a t Sa, where S is the covariance matrix of X and 1 1/n 1/n 1/n 1/n 1 1/n 1/n 1/n H n = /n 1/n 1/n 1 1/n

40 interpretation cont. The first principal component is then the g 1 that maximizes g t Sg over g = 1. The maximum does exists, but it may not be unique.

41 applications PCA in Computer Vision Feature extraction and data compression are closely related problems that can be attacked using PCA. Reference: facial recognition Face Recognition: Eigenface, Elastic Matching, and Neural Nets, Jun Zhang et al. Proceedings of the IEEE, Vol. 85, No. 9, September Representation: Consider a square, N by N image can be expressed as an N 2 -dimensional vector where the rows of pixels in the image are placed one after the other to form a one-dimensional image. For example, the first N elements will be the first row of the image, the next N elements are the next row, and so on. The values in the vector are the intensity values of the image, possibly a single greyscale value.

42 applications (cont.) Use PCA to find patterns Say we have 20 images and each image is N pixels high by N pixels wide. Put all the images together in one big 20 N 2 image-matrix. Perform PCA analysis on the ImagesMatrix. Suppose we want to do facial recognition, and our original images were of peoples faces. Then, the problem is, given a new image, whose face from the original set is it? (Note that the new image is not one of the 20 we started with.) In computer vision with PCA, it measures the difference between the new image and the original images, but not along the original axes, along the new axes derived from the PCA analysis.

43 applications PCA for image compression m bit data is converted to n bit data where n < m. Using PCA for image compression also know as the Hotelling, or Karhunen-Leove (KL), transform. If we have 20 images, each with N 2 pixels, we can form N 2 vectors, each with 20 dimensions. Each vector consists of all the intensity values from the same pixel from each picture. This is different from the previous example because before we had a vector for image, and each item in that vector was a different pixel, whereas now we have a vector for each pixel, and each item in the vector is from a different image.

44 applications (cont.) Perform the PCA on this set of data, Get 20 eigenvectors because each vector is 20-dimensional. To compress the data, we can then choose to transform the data only using, say 15 of the eigenvectors. This gives us a final data set with only 15 dimensions, which has saved us 1/4 of the space. However, when the original data is reproduced, the images have lost some of the information.

45 applications Karhunen-Leove (KL), transform The Wiener process W t is characterized by three facts: 1. W 0 = 0 2. W t is almost surely continuous 3. W t has independent increments with distribution (for 0 s < t). Note that the covariance function is Cov(W t, W s ) = min(s, t). The eigenvectors of the covariance kernel are e k (t) = ( 2 sin k 1 ) πt 2 and the corresponding eigenvalues are λ k = 4 (2k 1) 2 π 2.

46 applications This gives the following representation: Theorem. There is a sequence {W i } of independent standard normal random variables such that W t = sin ( k 1 ) 2 πt 2 W k ( ). k 1 2 π k=1

47 Swiss bank notes Six variables are measurements of the size of the bank notes length of bank note near the top left-hand height of bank note right-hand height of bank note distance from bottom of bank note to beginning of patterned border distance from top of bank note to beginning of patterned border diagonal distance 200 Swiss bank notes, 100 of which are genuine and 100 are forged Some complications: Some of the notes in either group may have been misclassified. Forged notes may not form a homogeneous group. We expect a higher variability for forged notes.

48 Issues of Swiss bank notes Outliers: Some of the notes in either group may have been misclassified. Forged notes may not form a homogeneous group. more than one forger at work A single forger may have short print runs before repeatedly moving premises in order to avoid detection. discriminant analysis versus cluster analysis Reference: Exploring multivariate data with the forward search