Linear Models in Matrix Form

Transcription

1 Linear Models in Matrix Form

2

3 Jonathon D. Brown Linear Models in Matrix Form A Hands-On Approach for the Behavioral Sciences

4 Jonathon D. Brown Department of Psychology University of Washington Seattle, WA, USA ISBN ISBN (ebook) DOI / Springer Cham Heidelberg New York Dordrecht London Library of Congress Control Number: Springer International Publishing Switzerland 2014 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (

5 To Noah and Avi. Two (very) special additions!

6

7 Linear Models in Matrix Form: A Hands-On Approach for the Behavioral Sciences Technological advances inevitably shape the teaching of statistics. For many years, the matrix calculations required in order to do most statistical analyses were too laborious to be performed by hand, so simple algebra was used. The formula for computing an unstandardized regression coefficient in a two-variable, multiple regression analysis provides one example of this first generation approach. b 1 ¼ s 1y s 2 2 s12 s 2y s 2 1 s2 2 ð s12 Þ 2 The availability of handheld calculators in the 1970s made these calculations less tedious, but the pedagogical approach to teaching statistics remained unchanged until statistical packages designed to take advantage of high-speed computers ushered in a new, click and run approach in the mid-1980s. Now, all the calculations took place behind the scenes, and formulas were replaced by a more practical method that emphasized the interpretation of output rather than its production. Unfortunately, the click and run approach has produced a new generation of students who know how to use computer software to perform statistical analyses without knowing much about the computations that produce the output they instantly and (apparently) magically receive. I think this lack of understanding is lamentable; I also think it is unnecessary. Instead of leading us to ignore the calculations that comprise statistical analyses, the ready accessibility of highspeed computers should have led us to reveal the matrix algebra that produces the output: b ¼ 1X X 0 0 X y or b ¼ R 1 Q 0 y That way, students would gain valuable insight into the mathematics behind the results. vii

8 viii Linear Models in Matrix Form: A Hands-On Approach for the Behavioral Sciences You Do the Math! I wrote this book to provide that balance, creating an easy-to-understand, hands-on approach to statistical analysis using matrix algebra. Although many of the topics are advanced (e.g., maximum-likelihood estimation, matrix decompositions, nonparametric smoothers, and penalized cubic splines), all the analyses were performed on an open-source spreadsheet using a few built-in functions (e.g., transpose, inverse). 1 Moreover, every statistical analysis is illustrated using a (fictitious) data set comprised of only 12 observations. The sample size is kept purposefully small so that readers will not be dissuaded from performing the calculations themselves, and each data set tells a coherent story based on statistical significance and confidence intervals. In my experience, statistical analyses come alive when students know how the numbers were generated and how they can be used to make cogent arguments about everyday matters. Intended Audience This book is intended for use in an upper-level undergraduate course or an entrylevel graduate course. I assume some knowledge of basic statistical principles (e.g., measures of central tendencies, hypothesis testing), but familiarity with matrix algebra is not needed. The first chapter introduces students to linear equations, and then covers matrix algebra, focusing on three essential operations: the sum of squares, the determinant, and the inverse. These operations are explained in common everyday language, and their calculations are demonstrated using concrete examples. The remaining chapters build on these operations, progressing from simple linear regression to mediational models with bootstrapped standard errors. This book is also appropriate for intellectually curious researchers. I hope to evoke an aha experience among those who learned only the click and run approach and a so that s all there is to it? reaction among those with only a limited background in mathematics. 1 The spreadsheets accompanying each chapter are available as supplementary material with the book on SpringerLink, and can be downloaded directly at social+sciences+%26+law/book/

9 Linear Models in Matrix Form: A Hands-On Approach for the Behavioral Sciences ix Using R Although this book emphasizes performing calculations with a spreadsheet rather than a statistical package, I do not expect readers to ordinarily use a spreadsheet to analyze their data once they have learned the calculations. 2 Toward that end, each chapter includes code for running all analyses in R, a free software programming language and software environment for statistical computing and graphics that can be downloaded at Whenever possible, I have used functions from the base suite of tools or have provided simple code for ones that are not available elsewhere. Occasionally, I have relied on functions associated with packages that must be installed and attached. To the Reader It has been said that there are three kinds of people those who can do math and those who cannot. If you get the joke, you should be able to solve all the problems in this textbook. Stripped of the jargon and the arcane formulas, most statistical analyses are not all that complicated. We have a variable we wish to predict and a set of variables we use to fashion that prediction. There is really nothing more to it than that. In the end, then, there really is only one kind of person: a person who can do math provided it is explained clearly and illustrated with examples. 2 The results of a great many calculations are presented in the text to illustrate various operations. Due to rounding error, the results obtained using the printed values do not always match the ones obtained using a spreadsheet or statistical package. Rather than confuse readers with two different values, I report the more accurate values one would get using computer software.

10

11 Contents 1 Matrix Properties and Operations Linear Equations and Matrix Algebra What Is a Matrix? Matrix Operations Matrix Rules of Multiplication Using a Spreadsheet to Perform Matrix Operations Matrix Transpose and Sum of Squares Transposition Premultiplying a Matrix by Its Transpose Covariance Matrix Correlation Matrix Diagonal Matrix Summary R Code: Matrix Multiplication Matrix Determinants Visualizing the Determinant Using the Determinant to Solve Linear Equations Linear Dependencies and Singular Matrices Calculating the Determinant with Large Matrices R Code: Determinants Matrix Inverse Matrix Multiplication and Matrix Inverse Calculating the Matrix Inverse Using the Inverse to Solve Linear Equations R Code: Matrix Inverse Chapter Summary Appendix xi

12 xii Contents 2 Simple Linear Regression Mathematical Models What Is a Model? What Is a Regression Model? What Is a Linear Regression Model? Simple Linear Regression Preliminary Analysis Without an Intercept Complete Analysis: Adding an Intercept Understanding Linear Regression Standardized Regression Coefficients Correlation Coefficient R Code: Simple Linear Regression Population Estimation and Statistical Significance The Logic Behind Null Hypothesis Testing Testing the Regression Model Testing the Regression Coefficients Parameter Covariance Matrix (C) R Code: Hypothesis Testing Forecasting Average Expected Values Single Predicted Values Forecasting with Caution R Code: Forecasting Chapter Summary Maximum-Likelihood Estimation Probability and Likelihood in a Normal Distribution Likelihood Function Log-Likelihood Function Using the Grid-Search Method to Find the Maximum-Likelihood Estimate R Code: Maximum-Likelihood Estimation with Normal Distribution Differential Calculus Differentiating a Function to Find Its Derivative Differentiation and Maximum-Likelihood Estimation Computing the Standard Errors R Code: Derivatives and Standard Errors Maximum-Likelihood Estimation in Regression Differentiating the Function Standard Errors Maximum-Likelihood Estimation: Numerical Methods Newton Raphson and Fisher s Method of Scoring Illustration Using Fisher s Method of Scoring

13 Contents xiii R Code: Maximum-Likelihood Estimation with Fisher s Method of Scoring Chapter Summary Appendix Multiple Regression Multiple Regression Correlations Unstandardized Regression Coefficients Fitted Values and Residuals Testing the Regression Model R Code: Multiple Regression Interpreting and Testing Regression Coefficients Comparing Regression Coefficients and Correlations Interpreting the Numerical Value of a Regression Coefficient Calculating Regression Coefficients Testing the Significance of Regression Coefficients Forecasting Comparing the Predictors R Code: Testing Regression Coefficients Partitioning the Variance Semipartial Correlation Partial Correlation Are Regression Coefficients Semipartial Coefficients or Partial Coefficients? R Code: Partitioning the Variance Calculating Regression Coefficients Using Cofactors Complete Sum of Squares Residual Sum of Squares and Coefficient of Determination Regression Coefficients Computing the Remaining Coefficients and Correlations Summary R Code: Regression Coefficients as Cofactors Chapter Summary Matrix Decompositions Eigen Decomposition Matrix Multiplication with an Ordinary Vector Matrix Multiplication with an Eigenvector Calculating Eigenvalues

14 xiv Contents Calculating Eigenvectors Eigenvalues and Variance Consolidation Eigen Decomposition and Matrix Recomposition R Code: Eigen Decomposition QR Decomposition Computations with Householder Transformations Linear Regression QR Algorithm for Finding the Eigenpairs R Code: QR Decomposition Singular Value Decomposition Preliminary Calculations Reconstructing X Regression Coefficients Standard Errors R Code: Singular Value Decomposition Cholesky Decomposition Calculations Calculating the Determinant and the Inverse Least Squares Regression Using the Cholesky Decomposition to Find the Eigenvalues R Code: Cholesky Decomposition Comparing the Decompositions Chapter Summary Problematic Observations Influential Observations Discrepant Observations Illustrating Undue Influence Leverage and the Hat Matrix Residuals and Outliers Variance of Fitted Values and Residuals Quantifying Influence Commentary R Code: Regression Diagnostics Departures from Normality Reviewing the Normality Assumption Assessing Normality Correcting Violations of Normality R Code: Departures from Normality Collinearity Problems with Overly Redundant Predictors Matrices with a Near-Linear Dependence are Ill Conditioned Demonstrating Collinearity

15 Contents xv Quantifying Collinearity with the Variance Inflation Factor Condition Index and Variance Proportion Decomposition Summary R Code: Collinearity Chapter Summary Errors and Residuals Errors and Their Assumed Distribution Why It Matters Errors and Residuals Generalized Least Squares Estimation Heteroscedasticity Small Sample Example Detecting Heteroscedasticity Weighted Least Squares Estimation Heteroscedasticity-Consistent Covariance Matrix Summary R Code: Heteroscedasticity Autocorrelations Mathematical Representation Detecting Autocorrelations Generalized Least Squares Estimation for Managing Autocorrelation Autocorrelation-Consistent Covariance Matrix R Code: Autocorrelations Chapter Summary Linearizing Transformations and Nonparametric Smoothers Understanding Linearity Partial Derivatives and Linear Functions Assessing Linear Relations Options for Analyzing Nonlinear Relations R Code: Assessing Nonlinearity Transformations to Linearity Understanding Transformations Logarithmic Model Exponential Model Power Function Box-Tidwell Transformation Summary R Code: Linear Transformations Nonparametric Smoothers Understanding Nonparametric Regression Running Average

16 xvi Contents Running Line Kernel Regression Locally Weighted Regression Extensions and Applications R Code: Nonparametric Smoothers Chapter Summary Cross-Product Terms and Interactions Understanding Interactions Depicting an Interaction Modeling Interactions with Cross-Product Terms Testing Cross-Product Terms R Code: Testing a Cross-Product Term Probing an Interaction Calculating Predicted Values Plotting Predicted Values Testing Simple Slopes Characterizing an Interaction R Code: Predicted Values and Simple Slopes Johnson-Neyman Technique R Code: Johnson-Neyman Regions of Significance Higher-Order Interactions Testing the Regression Equation Probing a Three-Variable Interaction R Code: Three-Way Interaction Recentering Variables to Calculate Simple Slopes R Code: Three-Way Interaction Using Recentering Effect Size and Statistical Power Effect Size Statistical Power R Code: Effect Size of Three-Way Cross-Product Term Chapter Summary Polynomial Regression Simple Polynomial Regression Testing the Linear Component Adding a Quadratic Term Testing Other Polynomials R Code: Cubic Polynomial Polynomial Interactions Regression Equations Testing the Regression Coefficients Probing a Polynomial Interaction R Code: Polynomial Interaction

17 Contents xvii 10.3 Piecewise Polynomials Regression Splines Natural Cubic Splines R Code: Unpenalized Regression Splines Penalized Natural Cubic Spline R Code: Penalized Natural Cubic Splines Chapter Summary Categorical Predictors Coding Schemes Analysis of Variance Overview of Coding Schemes Orthogonal Contrast Codes Dummy Codes Effect Codes Summary R Code: Coding Schemes Creating Orthogonal Contrast Codes Helmert Contrasts Gram-Schmidt Orthogonalization Polynomial Terms in a Trend Analysis R Code: Creating Orthogonal Contrasts Contrast Codes with Unbalanced Designs Analysis with Unweighted Means Weighted Means Analysis R Code: Unbalanced Designs Chapter Summary Factorial Designs Basics of Factorial Designs Regression Analysis of a One-Way Design Recasting the Data as a Factorial Design Properties of a Factorial Design Sources of Variance in a Balanced Factorial Design Probing an Interaction R Code: Factorial Design Unbalanced Factorial Designs Unweighted Means Weighted Means R Code: Unbalanced Factorial Design Multilevel Designs Coding Scheme Regression Analysis ANOVA Table R Code: Multilevel Design

18 xviii Contents Probing the Interaction Higher-Order Designs R Code: Simple Effects in Multilevel Design Chapter Summary Analysis of Covariance Introduction to ANCOVA Mechanics Preliminary Analyses Main Analysis R Code: ANCOVA Adjusted Means and Simple Effects R Code: Adjusted Means and Simple Effects Extensions to More Complex Designs Preliminary Analyses Main Analysis Adjusted Means Augmented Matrix and Multiple Comparisons R Code: ANCOVA with Multiple Covariates Uses (and Misuses) of ANCOVA A Residualized Criterion Association with the Predictor Chapter Summary Moderation Moderated Regression Illustration Implementation Regression Coefficients Plotting Predicted Values Crossing Point Testing the Simple Slopes R Code: Moderation Simple Slopes Simple Effects Augmented b Vector and C Matrix S Matrix Specific Tests of Interest R Code: Moderation Simple Effects Regions of Significance Reviewing the Johnson-Neyman Method Extending the Johnson-Neyman Method Illustration R Code: Regions of Significance Chapter Summary

19 Contents xix 15 Mediation Simple Mediation Analytic Strategy Assessing the Importance of the Mediated Effect Effect Sizes Contrasts Summary R Code: Simple Mediation Higher-Order Designs Mediation with Three Groups R Code: Mediation with Three Groups Multiple Mediators R Code: Multiple Mediators Mediation and Moderation R Code: Mediation and Moderation Mediation and Causal Inference Chapter Summary References Index