Number of cases (objects) Number of variables Number of dimensions. n-vector with categorical observations
|
|
- Domenic Rose
- 5 years ago
- Views:
Transcription
1 PRINCALS Notation The PRINCALS algorithm was first described in Van Rickevorsel and De Leeuw (1979) and De Leeuw and Van Rickevorsel (1980); also see Gifi (1981, 1985). Characteristic features of PRINCALS are the ability to specify any of a number of measurement levels for each variable separately and the treatment of missing values by setting weights in the loss function equal to 0. The following notation is used throughout this chapter unless otherwise noted: n m p Number of cases (obects) Number of variables Number of dimensions For variable, h = 1, K, m n-vector with categorical observations k Number of categories (distinct values) of variable G Indicator matrix for variable, of order n k The elements of G are defined as i = 1, K, n; r = 1, K, k 16 = % & ' 1 g ir 0 when the ith obect is in the rth category of variable when the ith obect is not in the rth category of variable M Binary, diagonal n n matrix The diagonal elements of M are i k m16 % 1 when the th observation is within the range [ 1, ] ii= & ' 0 when the ith observation is outside the range [ 1, k ] D Diagonal matrix containing the univariate marginals; that is, the column sums of G 1
2 2 PRINCALS The quantification matrices and parameter vectors are: X Obect scores, of order n p Y Multiple category quantifications, of order k p y a Single category quantifications, of order k Variable weights (equal to component loadings) of order p Q Transformed data matrix of order n m with columns q = G y Y Collection of multiple and single category quantifications Note: The matrices G, M, and D are exclusively notational devices; they are stored in reduced form, and the program fully profits from their sparseness by replacing matrix multiplications with selective accumulation. Obective Function Optimization The PRINCALS obective is to find obect scores X and a set of Y (for = 1, K, m) the underlining indicates that they are possibly restricted so that the function σ XY ; = 1 m tr X G Y M X GY is minimal, under the normalization restriction XM X = M = M mn I, where the matrix, and I is the p p identity matrix. The inclusion of M in σ1xy ; 6 ensures that there is no influence of data values outside the range [ 1, k ], which may be really missing or merely regarded as such; M contains the number of active data values for each obect. The obect scores are also centered; that is, they satisfy um X = 0, with u denoting an n-vector with ones.
3 PRINCALS 3 The following measurement levels are distinguished in PRINCALS: Multiple Nominal Y = Y (unrestricted) Single Nominal Y = y a (rank-one restrictions only) (Single) Ordinal Y = y a and y C (rank-one and montonicity restrictions) (Single) Numerical Y = y a and y L (rank-one and linearity restrictions) For each variable, these levels can be chosen independently. The general requirement in the single options is Y = y a ; that is, Y is of rank one; for identification purposes, y is always normalized so that y Dy = n, which implies that the variance of the quantified variable q = G y is 1. In the ordinal case, the additional restriction y C means that y must be located in the convex cone of all k -vectors with nondecreasing elements. In the numerical case, the additional restriction y L means that y must be located in the subspace of all k -vectors that are a linear transformation of the vector consisting of k successive integers. Optimization is achieved through the following iteration scheme twice: (1) Initialization (a) or (b) (2) Update obect scores
4 4 PRINCALS (3) Orthonormalization (4) Update category quantifications (5) Convergence test: repeat (2) (4) or continue (6) Rotation The first time (for the initial configuration) initialization (a) described below is used and all single variables are temporarily treated as single numerical. Multiple nominal variables are treated as multiple nominal. The second time (for the final configuration) initialization (b) is used. Steps (1) through (6) are explained below. (1) Initialization (a) The obect scores X are initialized with random numbers, which are normalized so that um X = 0 and XM X = mn I, yielding X ~. For multiple variables, the initial category quantifications are obtained as ~ 1 Y D G X ~ =. For single variables, the initial category quantifications are defined as the first k successive integers, normalized so that ud ~ y = 0 and ~ ydy ~ = n, and the initial variable weights are calculated as the vector ~ ~ a X G ~ = y, rescaled to unit length. (b) All relevant quantities are copied from the results of the first cycle. (2) Update obect scores First, the auxiliary score matrix Z is computed as Z ~ M G Y and centered with respect to M : ~ Z M M uu M u M u Z 1 6 < A. These two steps yield locally the best updates when there are no orthogonality constraints. (3) Orthonormalization The orthonormalization problem is to find an M -orthonormal X + that is closest to Z ~ in the least squares sense. In PRINCALS, this is done by setting + 12 X M 12 m M 12~ GRAM Z 4 9 which is equal to the genuine least squares estimate up to a rotation see (6). The notation GRAM( ) is used to denote the Gram-Schmidt transformation (Börk and Golub, 1973).
5 PRINCALS 5 (4) Update category quantifications; loop across variable = 1, K, m (a) For multiple nominal variables, the new category quantifications are computed as: + 1 ~ Y = D G X (b) For single variables, first an unconstrained update is computed in the same way: ~ 1 + Y = D G X Next, one cycle of an ALS algorithm (De Leeuw et al., 1976) is executed for computing a rank-one decomposition of Y ~, with restrictions on the left-hand vector. This cycle starts from the previous single quantifications ~ y with a + ~ = Y D ~ y When the current variable is numerical, we are ready; otherwise we compute y = Y ~ a + + Now, when the current variable is single nominal you can simply obtain y by normalizing y in the way indicated below; otherwise the variable must be ordinal, and you have to insert the weighted monotonic regression process y WMON4y 9 which make y monotonically increasing. The weights used are the diagonal elements of D, and the subalgorithm used is the up-and-down-blocks minimum violators algorithm (Kruskal, 1964; Barlow et al., 1972). The result is normalized: y = n y y D y Finally, we set Y + = y + a +
6 6 PRINCALS (5) Convergence test The difference between consecutive values of the quantity 16 16! TFIT = 1 m y sd y s+ a a s J J where y16 s denotes the sth column of Y and where J is an index set recording which variables are multiple, is compared with the user-specified convergence criterion e a small positive number. It can be shown that TFIT = p σ 1X; Y6. Steps (2) through (4) are repeated as long as the loss difference exceeds ε. (6) Rotation As remarked in (3), during iteration the orientation of X and Y with respect to the coordinate system is not necessarily correct; this also reflects that σ1xy ; 6 is invariant under simultaneous rotations of X and Y. From the theory of principal components, it is known that if all variables would be single, the matrix A which can be formed by stacking the row vectors a has the property that AA is diagonal. Therefore you can rotate so that the matrix 1 m AA = 1 m a a = 1 m YD Y " # $ # Diagnostics becomes diagonal. The corresponding eigenvalues are printed after the convergence message of the program. The calculation involves tridiagonalization with Householder transformations followed by the implicit QL algorithm (Wilkinson, 1965). Maximum Rank (may be issued as a warning when exceeded) The maximum rank p max indicates the maximum number of dimensions that can be computed for any data set. In general
7 PRINCALS 7 % ( &K J )K, 'K *K pmax = min n 1, k + m max m, max, m 0 1 where m 1 is the number of multiple variables with no missing values, m 2 is the number of single variables, and J is an index set recording which variables are multiple. Although the number of nontrivial dimensions may be less than p max when m = 2, PRINCALS does allow dimensionalities all the way up to p max. When, due to empty categories in the actual data, the rank deteriorates below the specified dimensionality, the program stops. Marginal Frequencies The frequencies table gives the univariate marginals and the number of missing values (that is, values that are regarded as out of range for the current analysis) for each variable. These are computed as the column sums of D and the total sum of M. Fit and Loss Measures When the HISTORY option is in effect, the following fit and loss measures are reported: (a) Total fit. This is the quantity TFIT defined in (5). (b) Total loss. This is σ XY ; loss defined below. (c) Multiple loss. This measure is computed as TMLOSS = p 1 m tr YD Y 1 6, computed as the sum of multiple loss and single (d) Single loss. This measure is computed only when some of the variables are single: SLOSS = 1 m tr YD Y + a a J J
8 8 PRINCALS Eigenvalues and Correlations between Optimally Scaled Variables References If there are no missing data, the eigenvalues printed by PRINCALS are those of 1 mr1q6, where RQ 1 6 denotes the matrix of correlations between the optimally scaled variables in the columns of Q. For multiple variables, q is defined here as 1 6 itself is also printed. G y161. When all variables are single or when p = 1, RQ If there are missing data, then the eigenvalues are those of the matrix with elements qm 1 q 1, which is not necessarily a correlation matrix, although it is positive semidefinite. Barlow, R. E., Bartholomew, D. J., Bremner, J. M., and Brunk, H. D Statistical inference under order restrictions. New York: John Wiley & Sons, Inc. Börk, A., and Golub, G. H Numerical methods for computing angles between linear subspaces. Mathematics of Computation, 27: De Leeuw, J., and Van Rickevorsel, J HOMALS and PRINCALS Some generalizations of principal components analysis. In: Data Analysis and Informatics, E. Diday et al, eds. Amsterdam: North-Holland. De Leeuw, J., Young, F. W., and Takane, Y Additive structure in qualitative data: an alternating least squares method with optimal scaling features. Psychometrika, 31: Gifi, A Nonlinear multivariate analysis. Leiden: Department of Data Theory. Gifi, A PRINCALS. Leiden: Department of Data Theory, Internal Report UG Kruskal, J. B Nonmetric multidimensional scaling: a numerical method. Psychometrika, 29: Van Rickevorsel, J., and De Leeuw, J An outline of PRINCALS. Leiden: Internal Report RB , Department of Data Theory.
9 PRINCALS 9 Wilkinson, J. H The algebraic eigenvalue problem. Oxford: Clarendon Press.
Number of analysis cases (objects) n n w Weighted number of analysis cases: matrix, with wi
CATREG Notation CATREG (Categorical regression with optimal scaling using alternating least squares) quantifies categorical variables using optimal scaling, resulting in an optimal linear regression equation
More informationNumber of analysis cases (objects) n n w Weighted number of analysis cases: matrix, with wi
CATREG Notation CATREG (Categorical regression with optimal scaling using alternating least squares) quantifies categorical variables using optimal scaling, resulting in an optimal linear regression equation
More informationChapter 2 Nonlinear Principal Component Analysis
Chapter 2 Nonlinear Principal Component Analysis Abstract Principal components analysis (PCA) is a commonly used descriptive multivariate method for handling quantitative data and can be extended to deal
More informationover the parameters θ. In both cases, consequently, we select the minimizing
MONOTONE REGRESSION JAN DE LEEUW Abstract. This is an entry for The Encyclopedia of Statistics in Behavioral Science, to be published by Wiley in 200. In linear regression we fit a linear function y =
More informationNONLINEAR PRINCIPAL COMPONENT ANALYSIS
NONLINEAR PRINCIPAL COMPONENT ANALYSIS JAN DE LEEUW ABSTRACT. Two quite different forms of nonlinear principal component analysis have been proposed in the literature. The first one is associated with
More informationPrincipal Component Analysis for Mixed Quantitative and Qualitative Data
1 Principal Component Analysis for Mixed Quantitative and Qualitative Data Susana Agudelo-Jaramillo sagudel9@eafit.edu.co Manuela Ochoa-Muñoz mochoam@eafit.edu.co Francisco Iván Zuluaga-Díaz fzuluag2@eafit.edu.co
More informationEvaluating Goodness of Fit in
Evaluating Goodness of Fit in Nonmetric Multidimensional Scaling by ALSCAL Robert MacCallum The Ohio State University Two types of information are provided to aid users of ALSCAL in evaluating goodness
More informationTwo-stage acceleration for non-linear PCA
Two-stage acceleration for non-linear PCA Masahiro Kuroda, Okayama University of Science, kuroda@soci.ous.ac.jp Michio Sakakihara, Okayama University of Science, sakaki@mis.ous.ac.jp Yuichi Mori, Okayama
More informationLeast squares multidimensional scaling with transformed distances
Least squares multidimensional scaling with transformed distances Patrick J.F. Groenen 1, Jan de Leeuw 2 and Rudolf Mathar 3 1 Department of Data Theory, University of Leiden P.O. Box 9555, 2300 RB Leiden,
More informationMath 3191 Applied Linear Algebra
Math 9 Applied Linear Algebra Lecture : Orthogonal Projections, Gram-Schmidt Stephen Billups University of Colorado at Denver Math 9Applied Linear Algebra p./ Orthonormal Sets A set of vectors {u, u,...,
More informationA Goodness-of-Fit Measure for the Mokken Double Monotonicity Model that Takes into Account the Size of Deviations
Methods of Psychological Research Online 2003, Vol.8, No.1, pp. 81-101 Department of Psychology Internet: http://www.mpr-online.de 2003 University of Koblenz-Landau A Goodness-of-Fit Measure for the Mokken
More informationTHE GIFI SYSTEM OF DESCRIPTIVE MULTIVARIATE ANALYSIS
THE GIFI SYSTEM OF DESCRIPTIVE MULTIVARIATE ANALYSIS GEORGE MICHAILIDIS AND JAN DE LEEUW ABSTRACT. The Gifi system of analyzing categorical data through nonlinear varieties of classical multivariate analysis
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector
More informationAn Introduction to Multivariate Statistical Analysis
An Introduction to Multivariate Statistical Analysis Third Edition T. W. ANDERSON Stanford University Department of Statistics Stanford, CA WILEY- INTERSCIENCE A JOHN WILEY & SONS, INC., PUBLICATION Contents
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationFACTOR ANALYSIS AS MATRIX DECOMPOSITION 1. INTRODUCTION
FACTOR ANALYSIS AS MATRIX DECOMPOSITION JAN DE LEEUW ABSTRACT. Meet the abstract. This is the abstract. 1. INTRODUCTION Suppose we have n measurements on each of taking m variables. Collect these measurements
More informationCorrespondence Analysis of Longitudinal Data
Correspondence Analysis of Longitudinal Data Mark de Rooij* LEIDEN UNIVERSITY, LEIDEN, NETHERLANDS Peter van der G. M. Heijden UTRECHT UNIVERSITY, UTRECHT, NETHERLANDS *Corresponding author (rooijm@fsw.leidenuniv.nl)
More informationPrincipal components based on a subset of qualitative variables and its accelerated computational algorithm
Int. Statistical Inst.: Proc. 58th World Statistical Congress, 2011, Dublin (Session IPS042) p.774 Principal components based on a subset of qualitative variables and its accelerated computational algorithm
More informationChapter 7. Least Squares Optimal Scaling of Partially Observed Linear Systems. Jan DeLeeuw
Chapter 7 Least Squares Optimal Scaling of Partially Observed Linear Systems Jan DeLeeuw University of California at Los Angeles, Department of Statistics. Los Angeles, USA 7.1 Introduction 7.1.1 Problem
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions
More informationMAXIMUM LIKELIHOOD IN GENERALIZED FIXED SCORE FACTOR ANALYSIS 1. INTRODUCTION
MAXIMUM LIKELIHOOD IN GENERALIZED FIXED SCORE FACTOR ANALYSIS JAN DE LEEUW ABSTRACT. We study the weighted least squares fixed rank approximation problem in which the weight matrices depend on unknown
More informationPrincipal Component Analysis-I Geog 210C Introduction to Spatial Data Analysis. Chris Funk. Lecture 17
Principal Component Analysis-I Geog 210C Introduction to Spatial Data Analysis Chris Funk Lecture 17 Outline Filters and Rotations Generating co-varying random fields Translating co-varying fields into
More informationA GENERALIZATION OF TAKANE'S ALGORITHM FOR DEDICOM. Yosmo TAKANE
PSYCHOMETR1KA--VOL. 55, NO. 1, 151--158 MARCH 1990 A GENERALIZATION OF TAKANE'S ALGORITHM FOR DEDICOM HENK A. L. KIERS AND JOS M. F. TEN BERGE UNIVERSITY OF GRONINGEN Yosmo TAKANE MCGILL UNIVERSITY JAN
More informationREGRESSION, DISCRIMINANT ANALYSIS, AND CANONICAL CORRELATION ANALYSIS WITH HOMALS 1. MORALS
REGRESSION, DISCRIMINANT ANALYSIS, AND CANONICAL CORRELATION ANALYSIS WITH HOMALS JAN DE LEEUW ABSTRACT. It is shown that the homals package in R can be used for multiple regression, multi-group discriminant
More informationSection 4.5 Eigenvalues of Symmetric Tridiagonal Matrices
Section 4.5 Eigenvalues of Symmetric Tridiagonal Matrices Key Terms Symmetric matrix Tridiagonal matrix Orthogonal matrix QR-factorization Rotation matrices (plane rotations) Eigenvalues We will now complete
More informationSolving large scale eigenvalue problems
arge scale eigenvalue problems, Lecture 4, March 14, 2018 1/41 Lecture 4, March 14, 2018: The QR algorithm http://people.inf.ethz.ch/arbenz/ewp/ Peter Arbenz Computer Science Department, ETH Zürich E-mail:
More informationALGORITHM CONSTRUCTION BY DECOMPOSITION 1. INTRODUCTION. The following theorem is so simple it s almost embarassing. Nevertheless
ALGORITHM CONSTRUCTION BY DECOMPOSITION JAN DE LEEUW ABSTRACT. We discuss and illustrate a general technique, related to augmentation, in which a complicated optimization problem is replaced by a simpler
More information5 Linear Algebra and Inverse Problem
5 Linear Algebra and Inverse Problem 5.1 Introduction Direct problem ( Forward problem) is to find field quantities satisfying Governing equations, Boundary conditions, Initial conditions. The direct problem
More informationSingular Value Decomposition
Chapter 6 Singular Value Decomposition In Chapter 5, we derived a number of algorithms for computing the eigenvalues and eigenvectors of matrices A R n n. Having developed this machinery, we complete our
More informationMATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)
MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m
More information4.8 Arnoldi Iteration, Krylov Subspaces and GMRES
48 Arnoldi Iteration, Krylov Subspaces and GMRES We start with the problem of using a similarity transformation to convert an n n matrix A to upper Hessenberg form H, ie, A = QHQ, (30) with an appropriate
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationNo books, no notes, no calculators. You must show work, unless the question is a true/false, yes/no, or fill-in-the-blank question.
Math 304 Final Exam (May 8) Spring 206 No books, no notes, no calculators. You must show work, unless the question is a true/false, yes/no, or fill-in-the-blank question. Name: Section: Question Points
More information1 Cricket chirps: an example
Notes for 2016-09-26 1 Cricket chirps: an example Did you know that you can estimate the temperature by listening to the rate of chirps? The data set in Table 1 1. represents measurements of the number
More informationA User's Guide To Principal Components
A User's Guide To Principal Components J. EDWARD JACKSON A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York Chichester Brisbane Toronto Singapore Contents Preface Introduction 1. Getting
More informationThe Singular Value Decomposition and Least Squares Problems
The Singular Value Decomposition and Least Squares Problems Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 27, 2009 Applications of SVD solving
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More information7. Symmetric Matrices and Quadratic Forms
Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9
STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9 1. qr and complete orthogonal factorization poor man s svd can solve many problems on the svd list using either of these factorizations but they
More informationWHEN MODIFIED GRAM-SCHMIDT GENERATES A WELL-CONDITIONED SET OF VECTORS
IMA Journal of Numerical Analysis (2002) 22, 1-8 WHEN MODIFIED GRAM-SCHMIDT GENERATES A WELL-CONDITIONED SET OF VECTORS L. Giraud and J. Langou Cerfacs, 42 Avenue Gaspard Coriolis, 31057 Toulouse Cedex
More informationPrincipal Component Analysis (PCA) Our starting point consists of T observations from N variables, which will be arranged in an T N matrix R,
Principal Component Analysis (PCA) PCA is a widely used statistical tool for dimension reduction. The objective of PCA is to find common factors, the so called principal components, in form of linear combinations
More informationMore Powerful Tests for Homogeneity of Multivariate Normal Mean Vectors under an Order Restriction
Sankhyā : The Indian Journal of Statistics 2007, Volume 69, Part 4, pp. 700-716 c 2007, Indian Statistical Institute More Powerful Tests for Homogeneity of Multivariate Normal Mean Vectors under an Order
More informationProblem # Max points possible Actual score Total 120
FINAL EXAMINATION - MATH 2121, FALL 2017. Name: ID#: Email: Lecture & Tutorial: Problem # Max points possible Actual score 1 15 2 15 3 10 4 15 5 15 6 15 7 10 8 10 9 15 Total 120 You have 180 minutes to
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 12 1 / 18 Overview
More informationWe will discuss matrix diagonalization algorithms in Numerical Recipes in the context of the eigenvalue problem in quantum mechanics, m A n = λ m
Eigensystems We will discuss matrix diagonalization algorithms in umerical Recipes in the context of the eigenvalue problem in quantum mechanics, A n = λ n n, (1) where A is a real, symmetric Hamiltonian
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 7: More on Householder Reflectors; Least Squares Problems Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 15 Outline
More informationLinear Models 1. Isfahan University of Technology Fall Semester, 2014
Linear Models 1 Isfahan University of Technology Fall Semester, 2014 References: [1] G. A. F., Seber and A. J. Lee (2003). Linear Regression Analysis (2nd ed.). Hoboken, NJ: Wiley. [2] A. C. Rencher and
More informationRecall the convention that, for us, all vectors are column vectors.
Some linear algebra Recall the convention that, for us, all vectors are column vectors. 1. Symmetric matrices Let A be a real matrix. Recall that a complex number λ is an eigenvalue of A if there exists
More informationLecture 5 Singular value decomposition
Lecture 5 Singular value decomposition Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn
More informationFACTOR ANALYSIS AND MULTIDIMENSIONAL SCALING
FACTOR ANALYSIS AND MULTIDIMENSIONAL SCALING Vishwanath Mantha Department for Electrical and Computer Engineering Mississippi State University, Mississippi State, MS 39762 mantha@isip.msstate.edu ABSTRACT
More information1 Singular Value Decomposition and Principal Component
Singular Value Decomposition and Principal Component Analysis In these lectures we discuss the SVD and the PCA, two of the most widely used tools in machine learning. Principal Component Analysis (PCA)
More informationBare minimum on matrix algebra. Psychology 588: Covariance structure and factor models
Bare minimum on matrix algebra Psychology 588: Covariance structure and factor models Matrix multiplication 2 Consider three notations for linear combinations y11 y1 m x11 x 1p b11 b 1m y y x x b b n1
More informationFactor Analysis of Data Matrices
Factor Analysis of Data Matrices PAUL HORST University of Washington HOLT, RINEHART AND WINSTON, INC. New York Chicago San Francisco Toronto London Contents Preface PART I. Introductory Background 1. The
More informationProcedia - Social and Behavioral Sciences 109 ( 2014 )
Available online at www.sciencedirect.com ScienceDirect Procedia - Social and Behavioral Sciences 09 ( 04 ) 730 736 nd World Conference On Business, Economics And Management - WCBEM 03 Categorical Principal
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 26, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 55 High dimensional
More informationQR Decomposition. When solving an overdetermined system by projection (or a least squares solution) often the following method is used:
(In practice not Gram-Schmidt, but another process Householder Transformations are used.) QR Decomposition When solving an overdetermined system by projection (or a least squares solution) often the following
More informationIDL Advanced Math & Stats Module
IDL Advanced Math & Stats Module Regression List of Routines and Functions Multiple Linear Regression IMSL_REGRESSORS IMSL_MULTIREGRESS IMSL_MULTIPREDICT Generates regressors for a general linear model.
More informationAssignment #9: Orthogonal Projections, Gram-Schmidt, and Least Squares. Name:
Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due date: Friday, April 0, 08 (:pm) Name: Section Number Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due
More informationExercise Set 7.2. Skills
Orthogonally diagonalizable matrix Spectral decomposition (or eigenvalue decomposition) Schur decomposition Subdiagonal Upper Hessenburg form Upper Hessenburg decomposition Skills Be able to recognize
More informationData Mining Lecture 4: Covariance, EVD, PCA & SVD
Data Mining Lecture 4: Covariance, EVD, PCA & SVD Jo Houghton ECS Southampton February 25, 2019 1 / 28 Variance and Covariance - Expectation A random variable takes on different values due to chance The
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More informationPreprocessing & dimensionality reduction
Introduction to Data Mining Preprocessing & dimensionality reduction CPSC/AMTH 445a/545a Guy Wolf guy.wolf@yale.edu Yale University Fall 2016 CPSC 445 (Guy Wolf) Dimensionality reduction Yale - Fall 2016
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationNonlinear Support Vector Machines through Iterative Majorization and I-Splines
Nonlinear Support Vector Machines through Iterative Majorization and I-Splines P.J.F. Groenen G. Nalbantov J.C. Bioch July 9, 26 Econometric Institute Report EI 26-25 Abstract To minimize the primal support
More informationUNIT 6: The singular value decomposition.
UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T
More informationPRACTICE FINAL EXAM. why. If they are dependent, exhibit a linear dependence relation among them.
Prof A Suciu MTH U37 LINEAR ALGEBRA Spring 2005 PRACTICE FINAL EXAM Are the following vectors independent or dependent? If they are independent, say why If they are dependent, exhibit a linear dependence
More informationMethods for sparse analysis of high-dimensional data, II
Methods for sparse analysis of high-dimensional data, II Rachel Ward May 23, 2011 High dimensional data with low-dimensional structure 300 by 300 pixel images = 90, 000 dimensions 2 / 47 High dimensional
More information7 Principal Component Analysis
7 Principal Component Analysis This topic will build a series of techniques to deal with high-dimensional data. Unlike regression problems, our goal is not to predict a value (the y-coordinate), it is
More informationSparse orthogonal factor analysis
Sparse orthogonal factor analysis Kohei Adachi and Nickolay T. Trendafilov Abstract A sparse orthogonal factor analysis procedure is proposed for estimating the optimal solution with sparse loadings. In
More informationDimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas
Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx
More informationCOMP6237 Data Mining Covariance, EVD, PCA & SVD. Jonathon Hare
COMP6237 Data Mining Covariance, EVD, PCA & SVD Jonathon Hare jsh2@ecs.soton.ac.uk Variance and Covariance Random Variables and Expected Values Mathematicians talk variance (and covariance) in terms of
More informationQUADRATIC MAJORIZATION 1. INTRODUCTION
QUADRATIC MAJORIZATION JAN DE LEEUW 1. INTRODUCTION Majorization methods are used extensively to solve complicated multivariate optimizaton problems. We refer to de Leeuw [1994]; Heiser [1995]; Lange et
More informationHOMOGENEITY ANALYSIS USING EUCLIDEAN MINIMUM SPANNING TREES 1. INTRODUCTION
HOMOGENEITY ANALYSIS USING EUCLIDEAN MINIMUM SPANNING TREES JAN DE LEEUW 1. INTRODUCTION In homogeneity analysis the data are a system of m subsets of a finite set of n objects. The purpose of the technique
More informationMULTILEVEL IMPUTATION 1
MULTILEVEL IMPUTATION 1 Supplement B: MCMC Sampling Steps and Distributions for Two-Level Imputation This document gives technical details of the full conditional distributions used to draw regression
More information1 Linear Algebra Problems
Linear Algebra Problems. Let A be the conjugate transpose of the complex matrix A; i.e., A = A t : A is said to be Hermitian if A = A; real symmetric if A is real and A t = A; skew-hermitian if A = A and
More informationRobustness of Principal Components
PCA for Clustering An objective of principal components analysis is to identify linear combinations of the original variables that are useful in accounting for the variation in those original variables.
More informationFinding normalized and modularity cuts by spectral clustering. Ljubjana 2010, October
Finding normalized and modularity cuts by spectral clustering Marianna Bolla Institute of Mathematics Budapest University of Technology and Economics marib@math.bme.hu Ljubjana 2010, October Outline Find
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More informationSection 6.4. The Gram Schmidt Process
Section 6.4 The Gram Schmidt Process Motivation The procedures in 6 start with an orthogonal basis {u, u,..., u m}. Find the B-coordinates of a vector x using dot products: x = m i= x u i u i u i u i Find
More informationAssessing Sample Variability in the Visualization Techniques related to Principal Component Analysis : Bootstrap and Alternative Simulation Methods.
Draft from : COMPSTAT, Physica Verlag, Heidelberg, 1996, Alberto Prats, Editor, p 205 210. Assessing Sample Variability in the Visualization Techniques related to Principal Component Analysis : Bootstrap
More informationLinear Models Review
Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign
More informationAssignment 1 Math 5341 Linear Algebra Review. Give complete answers to each of the following questions. Show all of your work.
Assignment 1 Math 5341 Linear Algebra Review Give complete answers to each of the following questions Show all of your work Note: You might struggle with some of these questions, either because it has
More informationSeminar on Linear Algebra
Supplement Seminar on Linear Algebra Projection, Singular Value Decomposition, Pseudoinverse Kenichi Kanatani Kyoritsu Shuppan Co., Ltd. Contents 1 Linear Space and Projection 1 1.1 Expression of Linear
More informationMatrix Differential Calculus with Applications in Statistics and Econometrics
Matrix Differential Calculus with Applications in Statistics and Econometrics Revised Edition JAN. R. MAGNUS CentERjor Economic Research, Tilburg University and HEINZ NEUDECKER Cesaro, Schagen JOHN WILEY
More informationMath 2331 Linear Algebra
6. Orthogonal Projections Math 2 Linear Algebra 6. Orthogonal Projections Jiwen He Department of Mathematics, University of Houston jiwenhe@math.uh.edu math.uh.edu/ jiwenhe/math2 Jiwen He, University of
More informationOrthogonalization and least squares methods
Chapter 3 Orthogonalization and least squares methods 31 QR-factorization (QR-decomposition) 311 Householder transformation Definition 311 A complex m n-matrix R = [r ij is called an upper (lower) triangular
More informationKrylov Subspaces. Lab 1. The Arnoldi Iteration
Lab 1 Krylov Subspaces Lab Objective: Discuss simple Krylov Subspace Methods for finding eigenvalues and show some interesting applications. One of the biggest difficulties in computational linear algebra
More informationContraction Methods for Convex Optimization and Monotone Variational Inequalities No.16
XVI - 1 Contraction Methods for Convex Optimization and Monotone Variational Inequalities No.16 A slightly changed ADMM for convex optimization with three separable operators Bingsheng He Department of
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationSTATISTICAL LEARNING SYSTEMS
STATISTICAL LEARNING SYSTEMS LECTURE 8: UNSUPERVISED LEARNING: FINDING STRUCTURE IN DATA Institute of Computer Science, Polish Academy of Sciences Ph. D. Program 2013/2014 Principal Component Analysis
More informationSection 6.2, 6.3 Orthogonal Sets, Orthogonal Projections
Section 6. 6. Orthogonal Sets Orthogonal Projections Main Ideas in these sections: Orthogonal set = A set of mutually orthogonal vectors. OG LI. Orthogonal Projection of y onto u or onto an OG set {u u
More informationNONLINEAR PRINCIPAL COMPONENT ANALYSIS AND RELATED TECHNIQUES. Principal Component Analysis (PCA from now on) is a multivariate data
NONLINEAR PRINCIPAL COMPONENT ANALYSIS AND RELATED TECHNIQUES JAN DE LEEUW 1. INTRODUCTION Principal Component Analysis (PCA from now on) is a multivariate data analysis technique used for many different
More informationProblem Set (T) If A is an m n matrix, B is an n p matrix and D is a p s matrix, then show
MTH 0: Linear Algebra Department of Mathematics and Statistics Indian Institute of Technology - Kanpur Problem Set Problems marked (T) are for discussions in Tutorial sessions (T) If A is an m n matrix,
More informationMATH 31 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL
MATH 3 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL MAIN TOPICS FOR THE FINAL EXAM:. Vectors. Dot product. Cross product. Geometric applications. 2. Row reduction. Null space, column space, row space, left
More informationLast Time. Social Network Graphs Betweenness. Graph Laplacian. Girvan-Newman Algorithm. Spectral Bisection
Eigenvalue Problems Last Time Social Network Graphs Betweenness Girvan-Newman Algorithm Graph Laplacian Spectral Bisection λ 2, w 2 Today Small deviation into eigenvalue problems Formulation Standard eigenvalue
More informationLinear Algebra problems
Linear Algebra problems 1. Show that the set F = ({1, 0}, +,.) is a field where + and. are defined as 1+1=0, 0+0=0, 0+1=1+0=1, 0.0=0.1=1.0=0, 1.1=1.. Let X be a non-empty set and F be any field. Let X
More informationLecture Notes 1: Vector spaces
Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More informationEigenvalues and diagonalization
Eigenvalues and diagonalization Patrick Breheny November 15 Patrick Breheny BST 764: Applied Statistical Modeling 1/20 Introduction The next topic in our course, principal components analysis, revolves
More informationNumerically Stable Cointegration Analysis
Numerically Stable Cointegration Analysis Jurgen A. Doornik Nuffield College, University of Oxford, Oxford OX1 1NF R.J. O Brien Department of Economics University of Southampton November 3, 2001 Abstract
More information