Hidden Predictors: A Factor Analysis Primer

Transcription

1 Hidden Predictors: A Factor Analysis Primer Ryan C Sanchez Western Washington University Factor Analysis is a owerful statistical method in the modern research sychologist s toolbag When used roerly, factor analysis discovers a minimal set of hidden features necessary to redict observed measurements This aer introduces factor analysis from a non-standard sychometrical viewoint; building models and solutions from the "ground u" Keywords: Factor Analysis, PCA, Orthogonal Rotation, Oblique Rotation, Particle Swarm Otimization, K-means Factor Analysis was develoed in the early 1900 s by researchers interested in quantifying human intelligence?? Over the ast century, these techniques were refined, imroved and extended; leading to general accetance and adotion within the sychometric community Today, modern statistical software ackages (SPSS, R, ) allow researchers to maniulate large datasets at the cost of obfuscating underlying rinciles These rinciles are exlored with incomlete knowledge of the factor analysis literature with the hoe to minimize biases and, erhas, rovide an alternative route to understanding Single Factor Model Models A reasonable assumtion in factor analysis is a linear relationshi between the observed measurements and the underlying unknown contributor 1 For a hyothetical single factor measurement, this is written as: s = λ f (1) where s, f, and λ are the observed measurement, unknown factor, and relationshi between them resectively As in all exeriments, measurement error arises from both known and unknown sources For examle, in an assessment questionnaire, individual interretations of the question may lead to measurement variability not owing to the factor Accounting for these externalities, a comlete single factor model is: s = λ f + ɛ (2) where ɛ is an "error" term not associated with the factor (referred within the sychometric lexicon as a "unique factor") It is imortant to note that in factor analysis, the only known quantity is the observed measurement, s Multi-Factor Model Tyically, the number of underlying factors contributing to the observed measurement is not know beforehand The multi-factor model allows for the hyothesis of multile contributors and can be written as the sum of N single factors m = s j = (λ j f j + ɛ j ) (3) This generalizes the observed measurement as a weighted linear combination of N unknown factors and the average error, ψ N ɛ Within the sychometric lexicon, λ j is called "factor loading": it is the contribution of the j-th factor on the measurement, m The multi-factor model is written, then, as: m = ψ + λ 1 f 1 + λ 2 f λ N f N (4) Again, it is worth ointing out only one measurement is recorded, m, and the unique determination of unknown arameters, λ j, f j, and ψ, is generally not ossible This multifactor model is assumed in the duration of this aer Multile Measurement Multi-Factor Model It is often the case that research exeriments use multile measurements to imrove the robustness and detectability of the unknown factors 2 It is assumed (or at least hoed) the introduction of multile measurements does not roliferate the number of unknown factors, ie all M measurements are a unique combination of the N unknown factors Personality questionnaires are an examle of using the multile-measurement technique to collect redundant information about the unknown factors Incororating multile measurements in the multi-factor model, the i-th measurement i s: 1 Using the same arguments used for linear regression 2 An excellent introduction to mathematical frames is Morgenshtern et al, 2011

2 2 SANCHEZ m i = ψ i + λ i j f j (5) Dislaying all the measurements together, a relationshi between the unknown factors, average errors, and factor loadings can be examined analytically m 1 m 2 m M = ψ 1 + λ 11 f 1 + λ 12 f λ 1N f N ψ 2 + λ 21 f 1 + λ 22 f λ 2N f N ψ M + λ M1 f 1 + λ M2 f λ MN f N Linear algebra simlifies the tabulation of measurements, aids in interretation, as well as foreshadowing otential solutions to the roblem m 1 m 2 m M λ 11 λ 12 λ 1N λ 21 λ 22 λ 2N = λ M1 λ M2 λ MN f 1 f 2 f N + ψ 1 ψ 2 Multi-Measurement Assumtions Equation 7 needs some unacking The observed measurements, m i, are assumed to adhere to the same multi-factor model In this aradigm, the factors remain unchanged and the variation in measurements is due entirely to different factor loadings, λ i j, and errors, ψ i An illustrative examle is the big five ersonality trait test The factors, f j, are the traits (oenness, agreeableness, ) unique to the articiant These traits remain unchanged over the duration of the test, and the measurements (answers to the questionnaire), m i, are deendent on the comosition, λ i j, and interretation, ψ i, of the questions Factor Analysis Model The last iece is the inclusion of multile articiants, or observations, in the multi-measurment multi-factor model This amounts to an addendum to equation 5 The m i measurement is: m i = ψ i + λ i j f j (8) or, written in linear algebraic notation: ψ M (6) (7) Using concise matrix notation, the final model used in factor analysis: M = ΛF + Ψ (9) Matrix Synosis Outlined below is a reference for the factor analysis model matrices Measurement Matrix The measurement matrix, M, reresents the only measured quantity in factor analysis The rows (horizontal elements) reresent the answer for a single question from every articiant and the columns (vertical elements) reresent the collection of all measurements for a single articiant Loadings Matrix The factor loadings matrix, Λ, describe the necessary weighting for each of the unknown factors Rows reresent the unique factor recie comrising each measurable, and columns are the contribution of a articular factor on each question While this matrix is not generally known beforehand, insight into the uniqueness (linear indeendence) of the rows and columns ought to be considered Factor Matrix The factor matrix, F, reresents the scores of underlying factors for each of the articiants Rows corresond to how a articular factor is distributed among all the articiants, and columns are the unique makeu of factors for one articiant This matrix is usually of great interest, however, since it is not directly observable aroximation methods have been devised and are addressed later Error Matrix The error matrix, Ψ, is the collection of average errors Rows reresent the average error of each articiant for each questions Columns reresent the average error for all the questions for a given articiant Again, this matrix is unobserved but, theoretically, rovides insight into the quality of the exeriment A highly variable row suggests a oorly constructed question, while a highly variable column suggests an atyical articiant, and ossible means of exclusion This matrix is the differentiator between rincial comonent analysis (PCA) and factor analysis (FA) Emloying data reduction techniques excluding this matrix results in PCA; including this matrix results in FA m 11 m 1P λ 11 λ 12 λ 1N f 11 f 1P ψ 11 ψ 1P m 21 m 2P λ 21 λ 22 λ 2N f 21 f 2P ψ 21 ψ 2P = + m M1 m MP λ M1 λ M2 λ MN f N1 f NP ψ M1 ψ MP } {{ }} {{ }} {{ }} {{ } M Λ F Ψ

3 FACTOR ANALYSIS 3 Solutions In the revious section, there are multile reminders about what is measured (the M matrix) and what is not measured (everything else) A natural question arising is how to determine the unknown matrices (Λ, F, and Ψ)? As reviously mentioned, the sychometric community distinguishes between inclusion and exclusion of the Ψ matrix as distinct methods in analysis Reduced Measurement Matrix, R In determining otential solutions, dimensionality of the measurement matrix is reduced by integrating over articiants Mathematically, this is equivalent to the inner roduct over all ossible row sace combinations of M: or R MM T (10) R uv = m u m v (11) The reduced measurement matrix has the roerty of symmetry, R R T = 0, with real eigenvalues and orthonormal eigenvectors These roerties are exloited when solving for solutions To reduce the condition number of the R matrix, the measured observables are often scaled and shifted While, scaling and shifting coefficients are not unique, selecting aroriate values will aide in downstream interretation of the results The general equation is: m m = m m γ (12) the two most commonly used coefficients are outlined below Covariance: Integral constraints on the measured observables define the coefficients, secifically: m m = 0 and m 2 m = σ 2 After some algebra, this leads to the following equations: = µ m = 1 P γ = P 1 m m the reduced measurement matrix is equivalent to the covariance matrix associated with M This can readily be seen by inserting the above into equation 12: R uv = 1 P 1 (m u µ u )(m v µ v ) (13) which is exactly the covariance definition [insert reference] Correlation: Correlation in this context refers to the normalized correlation coefficient, or Pearson s correlation coefficient Using similar constraints as covariance, but normalizing the energy, m m = 0 and m 2 m = 1, the scale and shift coefficients take on the following values: = µ m = 1 m m P γ = (m m µ m ) 2 (14) The reduced measurement matrix has "1s" on the main diagonal and all the entries are on the range [-1, 1] 3 Discussion Throughout factor analysis literature, the reduced measurement matrix corresonding to correlation is emloyed most often Unfortunately, exlanations detailing sufficient reasons are wanting Heuristic arguments are numerous, with the most used reason for or against being to buttress the hyothesis after the fact When viewed as an inner roduct sace, differences between the two reduced measurement matrices are discernible and a b = a b cos θ â ˆb = cos θ (Covariance) (Correlation) Covariance catures information about the magnitude in addition to the angle This inclusion unbounds the range, (, ), and imacts interretation of scalar relationshis Scaled versions of measurements are allowed to co-vary with unlimited range b = ɛ a a b = ɛ a 2 Correlation measures the similarity between measurements It has a bounded range equivalent to the cosine function, [-1, 1] This gives correlation a referential frame for comarison Scaled versions of measurements are identified as similar, 1, or anti-similar, -1 The basis for comarison and ease of interretation give the correlation reduced measurement matrix an edge and is likely the reason for its reference Emloying the covariance model is useful when equivalence under scalar multiles is not desired, or when using model based techniques ie maximum likelihood factor analysis Moving forward the correlation reduced matrix is used, though the techniques discussed are valid for any set of coefficients, and γ 3 This is essentially a direction cosine matrix

4 4 SANCHEZ General Solution Equation The imlications for using the reduced measurement matrix, R, are roagated through by combining Eq 10 and Eq 11 The literature makes reasonable assumtions about the correlative nature between the error and factor matrices, namely, there is no correlation If this assumtion is violated, then the recovered factor matrix is contaminated with unique errors (an examle is a systemic bias resent in the exeriment) The resulting equation rovides context to otential solutions R MM T = (ΛF + Ψ)(ΛF + Ψ) T = (ΛF + Ψ)(Ψ T + F T Λ T ) = ΛFF T Λ T + ΨΨ T (15) the correlative assumtions imly the following, 0 is a matrix of all zeros: ΛFΨ T = 0 Moving forward into solution techniques, Equation 16 reresents the general solution equation used by all solvers Princial Comonent Analysis Recalling that the reduced measurement matrix is symmetric, it is also diagonalizable Mathematically, a diagonalizable matrix takes the form: R = BDB T (16) The matrix B is orthonormal and the matrix D is diagonal The entries in D reresent the eigenvalues of R and columns of B are the eigenvectors Equation 17 has many names: Eigenvalue Decomosition, Sectral Decomosition, Singular Value Decomosition, etc Here it is named Princial Comonent Analysis (PCA) with the eigenvalues (D matrix) sorted in descending order Equating terms between Eq 16 and Eq 17 leads to one ossible solution which suggests: R = ΛFF T Λ T + ΨΨ T = BDB T B = Λ D = FF T (17) Ψ = 0 Discussion Recovery of the factor matrix, F, from the diagonal eigenvalue matrix, D, is ambiguous and, in general, not ossible 4 Equation 18 allows a ossible interretation of D as a covarariance-like measure of hidden factors among all articiants The dimensionality of D is MxM and the desire is to reduce the number of factors to N This is done by selecting the first N diagonal elements of D and discarding the rest 5 The culling of eigenvalues from D converts Equation 17 from an equality to an aroximation The subscrit denotes taking the aroriate N matrix terms R B N D N B T N = R (18) Using a covariance-like interretation of D, columns of B reresent the comonent of covariance for a given measurement These are combined into one matrix Φ that associates measurements with factors, and is ofter read as: "Measurement M loads on Factor F" with the following equivalence noted: Φ B N D 1 2 N (19) R = ΦΦ T The ideal Φ matrix is one said to have "simle structure" Simle structure is defined as reducing each measurement s loading on as few factors as ossible The common ractice of thresholding matrix entries further simlifies the analysis Table 7 is an examle of a matrix without simle structure Table 10 is an examle of a matrix exhibiting simle structure, a threshold can be alied to further distinguish the grouings 0 φ i j τ φ i j = (Threshold Function) φ i j > τ φ i j PCA Examle A toy roblem illustrating the stes in rincial comonent analysis is demonstrated below The roblem consists of four hyothetical questions with eight indeendent articiants The contrived measurement matrix, M, to three significant figures is Table 1 Table 1 Measurement Matrix, M m m m m The dimensionality of F would be NxP, whereas the dimensionality of D is MxM Hoefully, the exeriment has both M >> N and P >> M 5 Methods for determining the aroriate number, N, to select are discussed later

5 FACTOR ANALYSIS 5 The reduced measurement matrix, using Equations 11, 13, and 15 is listed in Table 2 A quick insection readily shows a high level of correlation between m 1 and m 2 as well as between m 3 and m 4 Stated alternatively, m 1 and m 2 are redundant measurements of a common hidden variable, likewise for m 3 and m 4 Table 5 Culled Diagonal Matrix, D N n n Table 2 Reduced Measurement Matrix, R m 1 m 2 m 3 m 4 m m m m Next, the R matrix is diagonalized Standard linear algebra ackages have singular value decomositions or eigenvalue decomosition functions that are used in this ste For comleteness the full B and D matrices from Equation 17 are listed, δ reresents an extremely small number Table 3 Diagonal Matrix, D n 3 n 4 n n n δ 0 n δ Table 6 Culled Psuedo-Loadings Matrix, B N m m m m The final ste is to combine the factor variance with the loadings, defined in Equation 19 At this oint, the matrix Φ is examined for grouings At first glance, Table 7 suggests the measurements load on every factor, which does not exhibit simle structure However, a lot of the factors, Figure 1, with the n i -th factor maed onto the cardinal axes, n 1, n 2 x, y, yields further insight Table 7 Factor Values Matrix, Φ m m m m Table 4 Psuedo-Loadings Matrix, B n 3 n 4 m m m m Letting N = 2, only two factors are retained in this examle Again, this reresents a contrived examle and in real exeriments N is estimated via other methods The reduced matrices, as defined in Equation 19, take the following forms Rotations Figure 1 correctly demonstrates the two grouings of the toy examle Ideally, the factor values matrix (Table 7), Φ, would yield similar interretation uon insection This is accomlished by rotating the factor values matrix Rotations aid only in the interretation of ossible grouings and do not imrove validity of the factors 6 For rotations to be valid, the aroximate reduced measurement matrix, R, must not change This constraint is met by dividing and multilying Φ by the same matrix 7, A: 6 An examle is relacing the geograhical directions, North, East, South and West with North-West, North-East, South-East, and South-West The distance between two cities remains unchanged, but the verbal directions between them do change 7 This requires A 1 exists

6 6 SANCHEZ Figure 1 Unrotated Factors Figure 2 Orthogonal Rotation Figure 3 Oblique Rotation m m m m m 1 m 2 m m m m m m m 1 m 2 m m m m m m m 1 m 2 m m Factor Values Matrices, Φ R, (left) and Basis Correlation Matrix, C, (right) for different rotations Notice the measurements become rogressively easier to identify common grouings from left to right Oblique rotations exhibit redundancy between factors; in this examle aroximately 21% or 10 off orthogonal Φ = Φ A 1 A }{{} I Plugging this into the definition of R yields a general solution for transforming the factor value matrix: R = ΦA 1 (AA T )A T Φ T (20) Insecting Equation 21, two new variables are defined to extract the necessary information: Φ R = ΦA 1 (21) C = AA T (22) Φ R reresents the rotated factor value matrix 8 After each rotation, this matrix is insected until the desired searation is achieved C reresents a factor basis correlation matrix and measures redundancies in measurement grouings Orthogonal Rotations A subset of rotations, orthogonal 9 rotations, have the secial roerty that the inverse is equivalent to the transose, A T = A 1 Under orthogonal rotations, angles and lengths remain unchanged Intuitively, this reresents a rotation that is easy to visualize 10 and often what is meant by the lain term "rotation" In two dimensions, the orthogonal rotation matrix takes the following form: cos(θ) sin(θ) A(θ) = sin(θ) cos(θ) (23) Alying this to the toy PCA examle yields a factor value matrix that shows the grouings by visual insection and allows for ossible thresholding, Figure 2 The factor basis correlation matrix, C, for orthogonal rotations is equal to the identity matrix; in other words, these rotations will not measure any redundancies between factors Oblique Rotations A general aroach to rotations is given in Equation 20 Within the sychometric lexicon, this is referred as oblique rotations Further constraints are imarted on Equation 23, namely the diagonal of the C matrix is all 1s 11 The two-dimensional oblique rotation matrix takes the form below, it is seen that the orthogonal rotation matrix is a degenerate case where θ 2 = θ , A(θ 1, θ 2 ) = cos(θ 1) sin(θ 1 ) cos(θ 2 ) sin(θ 2 ) (24) Alying to the toy PCA examle, measurements load almost entirely on unique factors However, the factors themselves correlate, as seen below Figure 3 8 A coefficient matrix defining movement along the bases defined in the row sace of A 9 The underlying basis vectors are at right angles, or 90 relative to each other 10 Imagine a iece of aer on a table, lacing your finger in the middle and sinning the aer 11 Conservation of Energy or Variance

7 FACTOR ANALYSIS 7 Factor Analysis PCA reresents a "to-down" solution to Equation 15 The assumtion of no error, Ψ = 0, allows sychometricians to borrow data reduction techniques originally develoed in other discilines True data always exhibits some level of uncertainty that breaks the PCA assumtion Unfortunately, the level and nature of errors is not known beforehand 12 Factor Analysis does not assume the absence of error, rather that errors between measurements are uncorrelated This is modeled by setting ΨΨ T equal to a diagonal matrix, D ψ, which has the form: a a 22 0 D ψ = (25) 0 0 a MM Using this information, Equations 15, and 17, Factor Analysis solutions extract factors from the following: R = ΛDΛ T + D ψ (26) The above equation has no analytical solution, instead various numerical methods are used to estimate Λ, D, and D ψ Numerical methods are not guaranteed to find a solution or to find the global solution; when in doubt, PCA is fallback to rovide at least some insight into the hidden factors Three of most oular extraction methods are outlined below, Princial Axis Factor (PAF), Maximum Likelihood (ML), and Least-Squares Secial consideration must be given to solutions for each method to guard against results that have no hysical interretation, known as Heywood cases Princial Axis Factoring Princial Axis Factoring is the simlest extraction method and shares many similarities with PCA Fixed-oint iteration is used to udate D ψ and the PCA algorithm udates both Λ and D; in fact, the first iteration of PAF is exactly PCA The notation d ψ reresents the diagonal vector of D ψ with udates to d ψ also udating D ψ The fixed oint iterate takes the form: d ψ,k+1 = f (d ψ,k ) (PAF fixed-oint iterate) The reduced measurement matrix, R, is erturbed by D ψ, so that R f (d ψ ) = R D ψ Re-writing Equation 26, a familiar form is resented again (see Equations 16 and 17) R f (d ψ ) = ΛDΛ T (27) Several oints worth noting: since both R and D ψ are symmetric, R f is as well, and R f is no longer semi-ositive definite The first oint allows the use of PCA, but the second oint may result in negative eigenvalues Given the covariance-like interretation used in revious analysis, negative eigenvalues have no interretable meaning The fixed-oint iterate function, f, is comuted as the difference between reduced correlation matrix, R and aroximation, R f The final ste nulls the off diagonal comonents d ψ,k+1 = diag(r R f (d ψ,k )) (28) The culled diagonal matrix needs to be insected at each iteration in R f for negative entries before continuing The resence of negative entries may suggest too many factors, N, retained The iterate continues until a sufficient stoing criteria is achieved, usually something similar to: d ψ,k+1 d ψ,k < τ (PAF stoing criteria) While the seeding of d ψ,k=0 is arbitrary, convergence roerties are affected by this initial guess Letting the first iteration equal 0 is a reasonable choice and leads to the equivalence between PCA and the first iteration of PAF After convergence or a sufficient number of iterations, Λ, D, and D ψ are reorted with the same interretation as before Rotations are erformed to maximize interretation if required Maximum Likelihood Maximum Likelihood reresents a model-based solution to Equation 26 An underlying robability distribution function is osited, usually gaussian, and the unknown arameters are modified until the robability of observing the measured variables is maximized 13 The benefits of model-based solutions are well known: measures of goodness-of-fit, quantiles, and robustness in the resence of noise to name a few However, if data deviates from the model, erroneous conclusions drawn from the results is likely A multi-variate gaussian distribution takes the form: f (x, µ, Σ) = 2πΣ 1/2 ex{ 1 2 (x µ)t Σ 1 (x µ)} This is a well studied roblem, with the resulting costfunction for minimization taking the form: c(σ) = ln Σ + Tr[Σ 1 V] (29) where V is the covariance version of the R matrix (γ = P 1) Substituting an estimate of Equation 26, cf Equation 18, for Σ, leads to a function where Λ N, D N, and D ψ are estimated via numerical methods c(λ N, D N, D ψ ) = ln Λ N D N Λ T N +D ψ +Tr[(Λ N D N Λ T N +D ψ) 1 V] (30) 12 If it is known, then the exeriment should remove it 13 This reresents a suedo-matched filter