Linear Models 1. Isfahan University of Technology Fall Semester, 2014

Size: px
Start display at page:

Download "Linear Models 1. Isfahan University of Technology Fall Semester, 2014"

Transcription

1 Linear Models 1 Isfahan University of Technology Fall Semester, 2014 References: [1] G. A. F., Seber and A. J. Lee (2003). Linear Regression Analysis (2nd ed.). Hoboken, NJ: Wiley. [2] A. C. Rencher and G. B. Schaalje (2008). Linear Models in Statistics (2nd ed.). John Wiley & Sons, Inc. [3] R. B., Bapat (2000), Linear Algebra and Linear Models, (2 nd ed.). Springer-Verlag. [4] S. R., Searle (1971). Linear Models. New York: Wiley. The PDF versions of references are available in the link Prerequisites:. Linear Algebra Basic Statistical Inference. Grading Policy: Assignments & Quizes: 3-5 points. Midterm Exam: 5-7 points. Final Exam: 10 points.

2

3 Contents Ch. 2. Matrix Algebra. ( Rencher and Schaalje )l Preface xv 1 Vectors of Random Variables Notation Statistical Models Linear Regression Models Expectation and Covariance Operators 5 Exercises la Mean and Variance of Quadratic Forms 9 Exercises 1 b Moment Generating Functions and Independence 13 Exercises lc 15 Miscellaneous Exercises Multivariate Normal Distribution Density Function 17 Exercises 2a Moment Generating Functions 20 Exercises 2b Statistical Independence 24 v

4 VI CONTENTS Exercises 2c Distribution of Quadratic Forms 27 Exercises 2d 31 Miscellaneous Exercises Linear Regression: Estimation and Distribution Theory Least Squares Estimation 35 Exercises 3a Properties of Least Squares Estimates 42 Exercises 3b Unbiased Estimation of (72 44 Exercises 3c Distribution Theory 47 Exercises 3d Maximum Likelihood Estimation Orthogonal Columns in the Regression Matrix 51 Exercises 3e Introducing Further Explanatory Variables General Theory One Extra Variable 57 Exercises 3f Estimation with Linear Restrictions Method of Lagrange Multipliers Method of Orthogonal Projections 61 Exercises 3g Design Matrix of Less Than Full Rank Least Squares Estimation 62 Exercises 3h Estimable Functions 64 Exercises 3i Introducing Further Explanatory Variables Introducing Linear Restrictions 65 Exercises 3j Generalized Least Squares 66 Exercises 3k Centering and Scaling the Explanatory Variables Centering Scaling 71

5 CONTENTS VII Exercises Bayesian Estimation 73 Exercises 3m Robust Regression M-Estimates Estimates Based on Robust Location and Scale Measures Measuring Robustness Other Robust Estimates Exercises 3n Miscellaneous Exercises Hypothesis Testing 4.1 Introduction 4.2 Likelihood Ratio Test 4.3 F-Test Motivation Derivation Exercises 4a Some Examples The Straight Line Exercises 4b 4.4 Multiple Correlation Coefficient Exercises 4c 4.5 Canonical Form for H Exercises 4d 4.6 Goodness-of-Fit Test 4.7 F-Test and Projection Matrices Miscellaneous Exercises 4 5 Confidence Intervals and Regions 5.1 Simultaneous Interval Estimation Simultaneous Inferences Comparison of Methods Confidence Regions Hypothesis Testing and Confidence Intervals 5.2 Confidence Bands for the Regression Surface Confidence Intervals Confidence Bands

6 VIII CONTENTS 5.3 Prediction Intervals and Bands for the Response Prediction Intervals Simultaneous Prediction Bands 5.4 Enlarging the Regression Matrix Miscellaneous Exercises Straight-Line Regression The Straight Line Confidence Intervals for the Slope and Intercept Confidence Interval for the x-intercept Prediction Intervals and Bands Prediction Intervals for the Response Inverse Prediction (Calibration) Exercises 6a 6.2 Straight Line through the Origin 6.3 Weighted Least Squares for the Straight Line Known Weights Unknown Weights Exercises 6b 6.4 Comparing Straight Lines General Model Use of Dummy Explanatory Variables Exercises 6c 6.5 Two-Phase Linear Regression 6.6 Local Linear Regression Miscellaneous Exercises Polynomial Regression 7.1 Polynomials in One Variable Problem of Ill-Conditioning Using Orthogonal Polynomials Controlled Calibration 7.2 Piecewise Polynomial Fitting Unsatisfactory Fit Spline Functions Smoothing Splines 7.3 Polynomial Regression in Several Variables Response Surfaces

7 CONTENTS IX Multidimensional Smoothing Miscellaneous Exercises Analysis of Variance 8.1 Introduction 8.2 One-Way Classification General Theory Confidence Intervals Underlying Assumptions Exercises 8a 8.3 Two-Way Classification (Unbalanced) Representation as a Regression Model Hypothesis Testing Procedures for Testing the Hypotheses Confidence Intervals Exercises 8b 8.4 Two-Way Classification (Balanced) Exercises 8c 8.5 Two-Way Classification (One Observation per Mean) Underlying Assumptions 8.6 Higher-Way Classifications with Equal Numbers per Mean Definition of Interactions Hypothesis Testing Missing Observations Exercises 8d 8.7 Designs with Simple Block Structure 8.8 Analysis of Covariance Exercises 8e Miscellaneous Exercises Departures from Underlying Assumptions 9.1 Introduction 9.2 Bias Bias Due to Underfitting Bias Due to Overfitting Exercises 9a 9.3 Incorrect Variance Matrix Exercises 9b

8 x CONTENTS 9.4 Effect of Outliers Robustness of the F-Test to Nonnormality Effect of the Regressor Variables Quadratically Balanced F-Tests 236 Exercises 9c Effect of Random Explanatory Variables Random Explanatory Variables Measured without Error Fixed Explanatory Variables Measured with Error Round-off Errors Some Working Rules Random Explanatory Variables Measured with Error Controlled Variables Model Collinearity Effect on the Variances of the Estimated Coefficients Variance Inflation Factors Variances and Eigenvalues Perturbation Theory Collinearity and Prediction 261 Exercises 9d 261 Miscellaneous Exercises Departures from Assumptions: Diagnosis and Remedies Introduction Residuals and Hat Matrix Diagonals 266 Exercises loa Dealing with Curvature Visualizing Regression Surfaces Transforming to Remove Curvature Adding and Deleting Variables 277 Exercises lob Nonconstant Variance and Serial Correlation Detecting Nonconstant Variance Estimating Variance Functions Transforming to Equalize Variances Serial Correlation and the Durbin-Watson Test 292 Exercises 10c Departures from Normality Normal Plotting 295

9 CONTENTS XI Transforming the Response Transforming Both Sides Exercises 10d 10.6 Detecting and Dealing with Outliers Types of Outliers Identifying High-Leverage Points Leave-One-Out Case Diagnostics Test for Outliers Other Methods Exercises loe 10.7 Diagnosing Collinearity Drawbacks of Centering Detection of Points Influencing Collinearity Remedies for Collinearity Exercises 10f Miscellaneous Exercises Computational Algorithms for Fitting a Regression 11.1 Introduction Basic Methods 11.2 Direct Solution of the Normal Equations Calculation of the Matrix XIX Solving the Normal Equations Exercises 11 a 11.3 QR Decomposition Calculation of Regression Quantities Algorithms for the QR and WU Decompositions Exercises 11 b 11.4 Singular Value Decomposition!l.4.1 Regression Calculations Using the SVD Computing the SVD 11.5 Weighted Least Squares 11.6 Adding and Deleting Cases and Variables Updating Formulas Connection with the Sweep Operator Adding and Deleting Cases and Variables Using QR 11.7 Centering the Data 11.8 Comparing Methods

10 xii CONTENTS Resources Efficiency Accuracy Two Examples Summary Exercises 11c 11.9 Rank-Deficient Case Modifying the QR Decomposition Solving the Least Squares Problem Calculating Rank in the Presence of Round-off Error Using the Singular Value Decomposition Computing the Hat Matrix Diagonals Using the Cholesky Factorization Using the Thin QR Decomposition Calculating Test Statistics Robust Regression Calculations Algorithms for L1 Regression Algorithms for M- and GM-Estimation Elemental Regressions Algorithms for High-Breakdown Methods 385 Exercises 11d 388 Miscellaneous Exercises Prediction and Model Selection Introduction Why Select? 393 Exercises 12a Choosing the Best Subset Goodness-of-Fit Criteria Criteria Based on Prediction Error Estimating Distributional Discrepancies Approximating Posterior Probabilities 410 Exercises 12b Stepwise Methods Forward Selection Backward Elimination Stepwise Regression 418 Exercises 12c 420

11 CONTENTS xiii 12.5 Shrinkage Methods Stein Shrinkage Ridge Regression Garrote and Lasso Estimates 425 Exercises 12d Bayesian Methods Predictive Densities Bayesian Prediction Bayesian Model Averaging 433 Exercises 12e Effect of Model Selection on Inference Conditional and Unconditional Distributions Bias Conditional Means and Variances Estimating Coefficients Using Conditional Likelihood Other Effects of Model Selection 438 Exercises 12f Computational Considerations Methods for All Possible Subsets Generating the Best Regressions All Possible Regressions Using QR Decompositions 446 Exercises 12g 12.9 Comparison of Methods Identifying the Correct Subset Using Prediction Error as a Criterion Exercises 12h Miscellaneous Exercises Appendix A Some Matrix Algebra A.l Trace and Eigenvalues A.2 Rank A.3 Positive-Semidefinite Matrices A.4 Positive-Definite Matrices A.5 Permutation Matrices A.6 Idempotent Matrices A.7 Eigenvalue Applications A.8 Vector Differentiation A.9 Patterned Matrices

12 xiv COIIJTENTS A.lO Generalized bversc 469 A.l1 Some Useful Results 471 A.12 Singular Value Decomposition 471 A.13 Some Miscellaneous Statistical Results 472 A.14 Fisher Scoring 473 Appendix B Orthogonal Projections 475 B.1 Orthogonal Decomposition of Vectors 475 B.2 Orthogonal Complements 477 B.3 Projections on Subs paces 477 Appendix C Tables 479 C.1 Percentage Points of the Bonferroni t-statistic 480 C.2 Distribution of the Largest Absolute Value of k Student t Variables 482 C.3 Working-Hotelling Confidence Bands for Finite Intervals 489 Outline Solutions to Selected Exercises 491 References 531 Index 549

13 Why statistical models? It is in human nature to try and understand the physical and natural phenomena that occur around us. When observations on a phenomenon can be quantified, such an attempt at understanding often takes the form of building a mathematical model, even if it is only a simplistic attempt to capture the essentials. Either because of our ignorance or in order to keep it simple, many relevant factors may be left out. Also models need to be validated through measurement, and such measurements often come with error. In order to account for the measurement or observational errors as well as the factors that may have been left out, one needs a statistical model which incorporates some amount of uncertainty. Why a linear model? Some of the reasons why we undertake a detailed study of the linear model are as follows. (a) Because of its simplicity, the linear model is better understood and easier to interpret than most of the other competing models, and the methods of analysis and inference are better developed Therefore, if there is no particular reason to presuppose another model, the linear model may be used at least as a first step. (b) The linear model formulation is useful even for certain nonlinear models which can be reduced to the linear form by means of a transformation. (c) Results obtained for the linear model serve as a stepping stone for the analysis of a much wider class of related models such as mixed effects model, state-space and other time series models. (d) Suppose that the response is modelled as a nonlinear function of the explanatory variables plus error. In many practical situations only a part of the domain of this function is of interest. For example, in a manufacturing process, one is interested in a narrow region centered around the operating point. If the above function is reasonably smooth in this region, a linear model serves as a good first order approximation to what is globally a nonlinear model.

14 Many statistical concepts can be viewed in the framework of linear models suppose that we wish to compare the means of two populations, say, i = E [U i ] (i = 1; 2). Then we can combine the data into the single model E (Y ) = 1 + ( 2 1 ) x = x; where x = 0 when Y is a U 1 observation and x = 1 when Y is a U 2 observation. Here 1 = 0 and 2 = 0 + 1, the di erence being 1. We can extend this idea to the case of comparing m means using m -1 dummy variables. In a similar fashion we can combine two straight lines, U j = j + j x 1 (j = 1; 2); and 1 otherwise. The combined model is E (Y ) = x 1 + ( 2 1 ) x 2 + ( 2 1 ) x 1 x 2 = x x x 3; using a dummy x 2 variable which takes the value 0 if the observation is from the rst line, 2

Wiley. Methods and Applications of Linear Models. Regression and the Analysis. of Variance. Third Edition. Ishpeming, Michigan RONALD R.

Wiley. Methods and Applications of Linear Models. Regression and the Analysis. of Variance. Third Edition. Ishpeming, Michigan RONALD R. Methods and Applications of Linear Models Regression and the Analysis of Variance Third Edition RONALD R. HOCKING PenHock Statistical Consultants Ishpeming, Michigan Wiley Contents Preface to the Third

More information

Linear Models in Statistics

Linear Models in Statistics Linear Models in Statistics ALVIN C. RENCHER Department of Statistics Brigham Young University Provo, Utah A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York Chichester Weinheim Brisbane

More information

An Introduction to Multivariate Statistical Analysis

An Introduction to Multivariate Statistical Analysis An Introduction to Multivariate Statistical Analysis Third Edition T. W. ANDERSON Stanford University Department of Statistics Stanford, CA WILEY- INTERSCIENCE A JOHN WILEY & SONS, INC., PUBLICATION Contents

More information

G. S. Maddala Kajal Lahiri. WILEY A John Wiley and Sons, Ltd., Publication

G. S. Maddala Kajal Lahiri. WILEY A John Wiley and Sons, Ltd., Publication G. S. Maddala Kajal Lahiri WILEY A John Wiley and Sons, Ltd., Publication TEMT Foreword Preface to the Fourth Edition xvii xix Part I Introduction and the Linear Regression Model 1 CHAPTER 1 What is Econometrics?

More information

New Introduction to Multiple Time Series Analysis

New Introduction to Multiple Time Series Analysis Helmut Lütkepohl New Introduction to Multiple Time Series Analysis With 49 Figures and 36 Tables Springer Contents 1 Introduction 1 1.1 Objectives of Analyzing Multiple Time Series 1 1.2 Some Basics 2

More information

Regression, Ridge Regression, Lasso

Regression, Ridge Regression, Lasso Regression, Ridge Regression, Lasso Fabio G. Cozman - fgcozman@usp.br October 2, 2018 A general definition Regression studies the relationship between a response variable Y and covariates X 1,..., X n.

More information

Regression Analysis By Example

Regression Analysis By Example Regression Analysis By Example Third Edition SAMPRIT CHATTERJEE New York University ALI S. HADI Cornell University BERTRAM PRICE Price Associates, Inc. A Wiley-Interscience Publication JOHN WILEY & SONS,

More information

Response Surface Methodology

Response Surface Methodology Response Surface Methodology Process and Product Optimization Using Designed Experiments Second Edition RAYMOND H. MYERS Virginia Polytechnic Institute and State University DOUGLAS C. MONTGOMERY Arizona

More information

Contents. Acknowledgments. xix

Contents. Acknowledgments. xix Table of Preface Acknowledgments page xv xix 1 Introduction 1 The Role of the Computer in Data Analysis 1 Statistics: Descriptive and Inferential 2 Variables and Constants 3 The Measurement of Variables

More information

TIME SERIES ANALYSIS. Forecasting and Control. Wiley. Fifth Edition GWILYM M. JENKINS GEORGE E. P. BOX GREGORY C. REINSEL GRETA M.

TIME SERIES ANALYSIS. Forecasting and Control. Wiley. Fifth Edition GWILYM M. JENKINS GEORGE E. P. BOX GREGORY C. REINSEL GRETA M. TIME SERIES ANALYSIS Forecasting and Control Fifth Edition GEORGE E. P. BOX GWILYM M. JENKINS GREGORY C. REINSEL GRETA M. LJUNG Wiley CONTENTS PREFACE TO THE FIFTH EDITION PREFACE TO THE FOURTH EDITION

More information

Machine Learning Linear Regression. Prof. Matteo Matteucci

Machine Learning Linear Regression. Prof. Matteo Matteucci Machine Learning Linear Regression Prof. Matteo Matteucci Outline 2 o Simple Linear Regression Model Least Squares Fit Measures of Fit Inference in Regression o Multi Variate Regession Model Least Squares

More information

Chapter 1 Statistical Inference

Chapter 1 Statistical Inference Chapter 1 Statistical Inference causal inference To infer causality, you need a randomized experiment (or a huge observational study and lots of outside information). inference to populations Generalizations

More information

Applied Regression Modeling

Applied Regression Modeling Applied Regression Modeling A Business Approach Iain Pardoe University of Oregon Charles H. Lundquist College of Business Eugene, Oregon WILEY- INTERSCIENCE A JOHN WILEY & SONS, INC., PUBLICATION CONTENTS

More information

Math 423/533: The Main Theoretical Topics

Math 423/533: The Main Theoretical Topics Math 423/533: The Main Theoretical Topics Notation sample size n, data index i number of predictors, p (p = 2 for simple linear regression) y i : response for individual i x i = (x i1,..., x ip ) (1 p)

More information

Statistical Methods for Forecasting

Statistical Methods for Forecasting Statistical Methods for Forecasting BOVAS ABRAHAM University of Waterloo JOHANNES LEDOLTER University of Iowa John Wiley & Sons New York Chichester Brisbane Toronto Singapore Contents 1 INTRODUCTION AND

More information

Experimental Design and Data Analysis for Biologists

Experimental Design and Data Analysis for Biologists Experimental Design and Data Analysis for Biologists Gerry P. Quinn Monash University Michael J. Keough University of Melbourne CAMBRIDGE UNIVERSITY PRESS Contents Preface page xv I I Introduction 1 1.1

More information

Introduction to Eco n o m et rics

Introduction to Eco n o m et rics 2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Introduction to Eco n o m et rics Third Edition G.S. Maddala Formerly

More information

Linear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,

Linear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,

More information

Stat 5101 Lecture Notes

Stat 5101 Lecture Notes Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random

More information

DESIGN AND ANALYSIS OF EXPERIMENTS Third Edition

DESIGN AND ANALYSIS OF EXPERIMENTS Third Edition DESIGN AND ANALYSIS OF EXPERIMENTS Third Edition Douglas C. Montgomery ARIZONA STATE UNIVERSITY JOHN WILEY & SONS New York Chichester Brisbane Toronto Singapore Contents Chapter 1. Introduction 1-1 What

More information

Final Review. Yang Feng. Yang Feng (Columbia University) Final Review 1 / 58

Final Review. Yang Feng.   Yang Feng (Columbia University) Final Review 1 / 58 Final Review Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Final Review 1 / 58 Outline 1 Multiple Linear Regression (Estimation, Inference) 2 Special Topics for Multiple

More information

For Bonnie and Jesse (again)

For Bonnie and Jesse (again) SECOND EDITION A P P L I E D R E G R E S S I O N A N A L Y S I S a n d G E N E R A L I Z E D L I N E A R M O D E L S For Bonnie and Jesse (again) SECOND EDITION A P P L I E D R E G R E S S I O N A N A

More information

Econometric Analysis of Cross Section and Panel Data

Econometric Analysis of Cross Section and Panel Data Econometric Analysis of Cross Section and Panel Data Jeffrey M. Wooldridge / The MIT Press Cambridge, Massachusetts London, England Contents Preface Acknowledgments xvii xxiii I INTRODUCTION AND BACKGROUND

More information

A Guide to Modern Econometric:

A Guide to Modern Econometric: A Guide to Modern Econometric: 4th edition Marno Verbeek Rotterdam School of Management, Erasmus University, Rotterdam B 379887 )WILEY A John Wiley & Sons, Ltd., Publication Contents Preface xiii 1 Introduction

More information

DETAILED CONTENTS PART I INTRODUCTION AND DESCRIPTIVE STATISTICS. 1. Introduction to Statistics

DETAILED CONTENTS PART I INTRODUCTION AND DESCRIPTIVE STATISTICS. 1. Introduction to Statistics DETAILED CONTENTS About the Author Preface to the Instructor To the Student How to Use SPSS With This Book PART I INTRODUCTION AND DESCRIPTIVE STATISTICS 1. Introduction to Statistics 1.1 Descriptive and

More information

Elements of Multivariate Time Series Analysis

Elements of Multivariate Time Series Analysis Gregory C. Reinsel Elements of Multivariate Time Series Analysis Second Edition With 14 Figures Springer Contents Preface to the Second Edition Preface to the First Edition vii ix 1. Vector Time Series

More information

Response Surface Methodology:

Response Surface Methodology: Response Surface Methodology: Process and Product Optimization Using Designed Experiments RAYMOND H. MYERS Virginia Polytechnic Institute and State University DOUGLAS C. MONTGOMERY Arizona State University

More information

ISyE 691 Data mining and analytics

ISyE 691 Data mining and analytics ISyE 691 Data mining and analytics Regression Instructor: Prof. Kaibo Liu Department of Industrial and Systems Engineering UW-Madison Email: kliu8@wisc.edu Office: Room 3017 (Mechanical Engineering Building)

More information

Preface to Second Edition... vii. Preface to First Edition...

Preface to Second Edition... vii. Preface to First Edition... Contents Preface to Second Edition..................................... vii Preface to First Edition....................................... ix Part I Linear Algebra 1 Basic Vector/Matrix Structure and

More information

Principal Component Analysis

Principal Component Analysis I.T. Jolliffe Principal Component Analysis Second Edition With 28 Illustrations Springer Contents Preface to the Second Edition Preface to the First Edition Acknowledgments List of Figures List of Tables

More information

Generalized, Linear, and Mixed Models

Generalized, Linear, and Mixed Models Generalized, Linear, and Mixed Models CHARLES E. McCULLOCH SHAYLER.SEARLE Departments of Statistical Science and Biometrics Cornell University A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS, INC. New

More information

Testing Statistical Hypotheses

Testing Statistical Hypotheses E.L. Lehmann Joseph P. Romano Testing Statistical Hypotheses Third Edition 4y Springer Preface vii I Small-Sample Theory 1 1 The General Decision Problem 3 1.1 Statistical Inference and Statistical Decisions

More information

HANDBOOK OF APPLICABLE MATHEMATICS

HANDBOOK OF APPLICABLE MATHEMATICS HANDBOOK OF APPLICABLE MATHEMATICS Chief Editor: Walter Ledermann Volume VI: Statistics PART A Edited by Emlyn Lloyd University of Lancaster A Wiley-Interscience Publication JOHN WILEY & SONS Chichester

More information

A User's Guide To Principal Components

A User's Guide To Principal Components A User's Guide To Principal Components J. EDWARD JACKSON A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York Chichester Brisbane Toronto Singapore Contents Preface Introduction 1. Getting

More information

Parameter Estimation and Hypothesis Testing in Linear Models

Parameter Estimation and Hypothesis Testing in Linear Models Parameter Estimation and Hypothesis Testing in Linear Models Springer-Verlag Berlin Heidelberg GmbH Karl-Rudolf Koch Parameter Estimation and Hypothesis Testing in Linear Models Second, updated and enlarged

More information

MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2

MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2 MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2 1 Ridge Regression Ridge regression and the Lasso are two forms of regularized

More information

Christopher Dougherty London School of Economics and Political Science

Christopher Dougherty London School of Economics and Political Science Introduction to Econometrics FIFTH EDITION Christopher Dougherty London School of Economics and Political Science OXFORD UNIVERSITY PRESS Contents INTRODU CTION 1 Why study econometrics? 1 Aim of this

More information

Hands-on Matrix Algebra Using R

Hands-on Matrix Algebra Using R Preface vii 1. R Preliminaries 1 1.1 Matrix Defined, Deeper Understanding Using Software.. 1 1.2 Introduction, Why R?.................... 2 1.3 Obtaining R.......................... 4 1.4 Reference Manuals

More information

Preface. Figures Figures appearing in the text were prepared using MATLAB R. For product information, please contact:

Preface. Figures Figures appearing in the text were prepared using MATLAB R. For product information, please contact: Linear algebra forms the basis for much of modern mathematics theoretical, applied, and computational. The purpose of this book is to provide a broad and solid foundation for the study of advanced mathematics.

More information

Linear Statistical Models

Linear Statistical Models Linear Statistical Models JAMES H. STAPLETON Michigan State University A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York 0 Chichester 0 Brisbane 0 Toronto 0 Singapore This Page Intentionally

More information

Generalized Linear. Mixed Models. Methods and Applications. Modern Concepts, Walter W. Stroup. Texts in Statistical Science.

Generalized Linear. Mixed Models. Methods and Applications. Modern Concepts, Walter W. Stroup. Texts in Statistical Science. Texts in Statistical Science Generalized Linear Mixed Models Modern Concepts, Methods and Applications Walter W. Stroup CRC Press Taylor & Francis Croup Boca Raton London New York CRC Press is an imprint

More information

1 Cricket chirps: an example

1 Cricket chirps: an example Notes for 2016-09-26 1 Cricket chirps: an example Did you know that you can estimate the temperature by listening to the rate of chirps? The data set in Table 1 1. represents measurements of the number

More information

Contest Quiz 3. Question Sheet. In this quiz we will review concepts of linear regression covered in lecture 2.

Contest Quiz 3. Question Sheet. In this quiz we will review concepts of linear regression covered in lecture 2. Updated: November 17, 2011 Lecturer: Thilo Klein Contact: tk375@cam.ac.uk Contest Quiz 3 Question Sheet In this quiz we will review concepts of linear regression covered in lecture 2. NOTE: Please round

More information

INTRODUCTORY REGRESSION ANALYSIS

INTRODUCTORY REGRESSION ANALYSIS ;»»>? INTRODUCTORY REGRESSION ANALYSIS With Computer Application for Business and Economics Allen Webster Routledge Taylor & Francis Croup NEW YORK AND LONDON TABLE OF CONTENT IN DETAIL INTRODUCTORY REGRESSION

More information

Time Series Analysis. James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY

Time Series Analysis. James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY Time Series Analysis James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY & Contents PREFACE xiii 1 1.1. 1.2. Difference Equations First-Order Difference Equations 1 /?th-order Difference

More information

Pattern Recognition and Machine Learning

Pattern Recognition and Machine Learning Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability

More information

TIME SERIES DATA ANALYSIS USING EVIEWS

TIME SERIES DATA ANALYSIS USING EVIEWS TIME SERIES DATA ANALYSIS USING EVIEWS I Gusti Ngurah Agung Graduate School Of Management Faculty Of Economics University Of Indonesia Ph.D. in Biostatistics and MSc. in Mathematical Statistics from University

More information

PART I INTRODUCTION The meaning of probability Basic definitions for frequentist statistics and Bayesian inference Bayesian inference Combinatorics

PART I INTRODUCTION The meaning of probability Basic definitions for frequentist statistics and Bayesian inference Bayesian inference Combinatorics Table of Preface page xi PART I INTRODUCTION 1 1 The meaning of probability 3 1.1 Classical definition of probability 3 1.2 Statistical definition of probability 9 1.3 Bayesian understanding of probability

More information

Statistics 262: Intermediate Biostatistics Model selection

Statistics 262: Intermediate Biostatistics Model selection Statistics 262: Intermediate Biostatistics Model selection Jonathan Taylor & Kristin Cobb Statistics 262: Intermediate Biostatistics p.1/?? Today s class Model selection. Strategies for model selection.

More information

Continuous Univariate Distributions

Continuous Univariate Distributions Continuous Univariate Distributions Volume 2 Second Edition NORMAN L. JOHNSON University of North Carolina Chapel Hill, North Carolina SAMUEL KOTZ University of Maryland College Park, Maryland N. BALAKRISHNAN

More information

Prediction of Bike Rental using Model Reuse Strategy

Prediction of Bike Rental using Model Reuse Strategy Prediction of Bike Rental using Model Reuse Strategy Arun Bala Subramaniyan and Rong Pan School of Computing, Informatics, Decision Systems Engineering, Arizona State University, Tempe, USA. {bsarun, rong.pan}@asu.edu

More information

Finite Population Sampling and Inference

Finite Population Sampling and Inference Finite Population Sampling and Inference A Prediction Approach RICHARD VALLIANT ALAN H. DORFMAN RICHARD M. ROYALL A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York Chichester Weinheim Brisbane

More information

Contents. Part I: Fundamentals of Bayesian Inference 1

Contents. Part I: Fundamentals of Bayesian Inference 1 Contents Preface xiii Part I: Fundamentals of Bayesian Inference 1 1 Probability and inference 3 1.1 The three steps of Bayesian data analysis 3 1.2 General notation for statistical inference 4 1.3 Bayesian

More information

Course in Data Science

Course in Data Science Course in Data Science About the Course: In this course you will get an introduction to the main tools and ideas which are required for Data Scientist/Business Analyst/Data Analyst. The course gives an

More information

FINITE-DIMENSIONAL LINEAR ALGEBRA

FINITE-DIMENSIONAL LINEAR ALGEBRA DISCRETE MATHEMATICS AND ITS APPLICATIONS Series Editor KENNETH H ROSEN FINITE-DIMENSIONAL LINEAR ALGEBRA Mark S Gockenbach Michigan Technological University Houghton, USA CRC Press Taylor & Francis Croup

More information

Regression Model Specification in R/Splus and Model Diagnostics. Daniel B. Carr

Regression Model Specification in R/Splus and Model Diagnostics. Daniel B. Carr Regression Model Specification in R/Splus and Model Diagnostics By Daniel B. Carr Note 1: See 10 for a summary of diagnostics 2: Books have been written on model diagnostics. These discuss diagnostics

More information

Contents. Preface for the Instructor. Preface for the Student. xvii. Acknowledgments. 1 Vector Spaces 1 1.A R n and C n 2

Contents. Preface for the Instructor. Preface for the Student. xvii. Acknowledgments. 1 Vector Spaces 1 1.A R n and C n 2 Contents Preface for the Instructor xi Preface for the Student xv Acknowledgments xvii 1 Vector Spaces 1 1.A R n and C n 2 Complex Numbers 2 Lists 5 F n 6 Digression on Fields 10 Exercises 1.A 11 1.B Definition

More information

STATISTICS; An Introductory Analysis. 2nd hidition TARO YAMANE NEW YORK UNIVERSITY A HARPER INTERNATIONAL EDITION

STATISTICS; An Introductory Analysis. 2nd hidition TARO YAMANE NEW YORK UNIVERSITY A HARPER INTERNATIONAL EDITION 2nd hidition TARO YAMANE NEW YORK UNIVERSITY STATISTICS; An Introductory Analysis A HARPER INTERNATIONAL EDITION jointly published by HARPER & ROW, NEW YORK, EVANSTON & LONDON AND JOHN WEATHERHILL, INC.,

More information

COPYRIGHTED MATERIAL CONTENTS. Preface Preface to the First Edition

COPYRIGHTED MATERIAL CONTENTS. Preface Preface to the First Edition Preface Preface to the First Edition xi xiii 1 Basic Probability Theory 1 1.1 Introduction 1 1.2 Sample Spaces and Events 3 1.3 The Axioms of Probability 7 1.4 Finite Sample Spaces and Combinatorics 15

More information

Lessons in Estimation Theory for Signal Processing, Communications, and Control

Lessons in Estimation Theory for Signal Processing, Communications, and Control Lessons in Estimation Theory for Signal Processing, Communications, and Control Jerry M. Mendel Department of Electrical Engineering University of Southern California Los Angeles, California PRENTICE HALL

More information

MS-C1620 Statistical inference

MS-C1620 Statistical inference MS-C1620 Statistical inference 10 Linear regression III Joni Virta Department of Mathematics and Systems Analysis School of Science Aalto University Academic year 2018 2019 Period III - IV 1 / 32 Contents

More information

Linear and Nonlinear Models

Linear and Nonlinear Models Erik W. Grafarend Linear and Nonlinear Models Fixed Effects, Random Effects, and Mixed Models magic triangle 1 fixed effects 2 random effects 3 crror-in-variables model W DE G Walter de Gruyter Berlin

More information

Testing Statistical Hypotheses

Testing Statistical Hypotheses E.L. Lehmann Joseph P. Romano, 02LEu1 ttd ~Lt~S Testing Statistical Hypotheses Third Edition With 6 Illustrations ~Springer 2 The Probability Background 28 2.1 Probability and Measure 28 2.2 Integration.........

More information

Appendix A Summary of Tasks. Appendix Table of Contents

Appendix A Summary of Tasks. Appendix Table of Contents Appendix A Summary of Tasks Appendix Table of Contents Reporting Tasks...357 ListData...357 Tables...358 Graphical Tasks...358 BarChart...358 PieChart...359 Histogram...359 BoxPlot...360 Probability Plot...360

More information

ECE521 week 3: 23/26 January 2017

ECE521 week 3: 23/26 January 2017 ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear

More information

Numerical Methods in Matrix Computations

Numerical Methods in Matrix Computations Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices

More information

Statistical Methods. for Forecasting

Statistical Methods. for Forecasting Statistical Methods for Forecasting Statistical Methods for Forecasting BOVAS ABRAHAM JOHANNES LEDOLTER WILEY- INTERSCI ENCE A JOHN WILEY & SONS, INC., PUBLICA'TION Copyright 0 1983.2005 by John Wiley

More information

Regression III Lecture 1: Preliminary

Regression III Lecture 1: Preliminary Regression III Lecture 1: Preliminary Dave Armstrong University of Western Ontario Department of Political Science Department of Statistics and Actuarial Science (by courtesy) e: dave.armstrong@uwo.ca

More information

Independent Component Analysis. Contents

Independent Component Analysis. Contents Contents Preface xvii 1 Introduction 1 1.1 Linear representation of multivariate data 1 1.1.1 The general statistical setting 1 1.1.2 Dimension reduction methods 2 1.1.3 Independence as a guiding principle

More information

Linear Regression Models

Linear Regression Models Linear Regression Models Model Description and Model Parameters Modelling is a central theme in these notes. The idea is to develop and continuously improve a library of predictive models for hazards,

More information

A Second Course in Statistics: Regression Analysis

A Second Course in Statistics: Regression Analysis FIFTH E D I T I 0 N A Second Course in Statistics: Regression Analysis WILLIAM MENDENHALL University of Florida TERRY SINCICH University of South Florida PRENTICE HALL Upper Saddle River, New Jersey 07458

More information

Machine Learning Linear Classification. Prof. Matteo Matteucci

Machine Learning Linear Classification. Prof. Matteo Matteucci Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)

More information

Subject CS1 Actuarial Statistics 1 Core Principles

Subject CS1 Actuarial Statistics 1 Core Principles Institute of Actuaries of India Subject CS1 Actuarial Statistics 1 Core Principles For 2019 Examinations Aim The aim of the Actuarial Statistics 1 subject is to provide a grounding in mathematical and

More information

FORECASTING METHODS AND APPLICATIONS SPYROS MAKRIDAKIS STEVEN С WHEELWRIGHT. European Institute of Business Administration. Harvard Business School

FORECASTING METHODS AND APPLICATIONS SPYROS MAKRIDAKIS STEVEN С WHEELWRIGHT. European Institute of Business Administration. Harvard Business School FORECASTING METHODS AND APPLICATIONS SPYROS MAKRIDAKIS European Institute of Business Administration (INSEAD) STEVEN С WHEELWRIGHT Harvard Business School. JOHN WILEY & SONS SANTA BARBARA NEW YORK CHICHESTER

More information

Linear Models in Matrix Form

Linear Models in Matrix Form Linear Models in Matrix Form Jonathon D. Brown Linear Models in Matrix Form A Hands-On Approach for the Behavioral Sciences Jonathon D. Brown Department of Psychology University of Washington Seattle,

More information

Dr. Maddah ENMG 617 EM Statistics 11/28/12. Multiple Regression (3) (Chapter 15, Hines)

Dr. Maddah ENMG 617 EM Statistics 11/28/12. Multiple Regression (3) (Chapter 15, Hines) Dr. Maddah ENMG 617 EM Statistics 11/28/12 Multiple Regression (3) (Chapter 15, Hines) Problems in multiple regression: Multicollinearity This arises when the independent variables x 1, x 2,, x k, are

More information

PRINCIPLES OF STATISTICAL INFERENCE

PRINCIPLES OF STATISTICAL INFERENCE Advanced Series on Statistical Science & Applied Probability PRINCIPLES OF STATISTICAL INFERENCE from a Neo-Fisherian Perspective Luigi Pace Department of Statistics University ofudine, Italy Alessandra

More information

Review of Classical Least Squares. James L. Powell Department of Economics University of California, Berkeley

Review of Classical Least Squares. James L. Powell Department of Economics University of California, Berkeley Review of Classical Least Squares James L. Powell Department of Economics University of California, Berkeley The Classical Linear Model The object of least squares regression methods is to model and estimate

More information

DESIGNING EXPERIMENTS AND ANALYZING DATA A Model Comparison Perspective

DESIGNING EXPERIMENTS AND ANALYZING DATA A Model Comparison Perspective DESIGNING EXPERIMENTS AND ANALYZING DATA A Model Comparison Perspective Second Edition Scott E. Maxwell Uniuersity of Notre Dame Harold D. Delaney Uniuersity of New Mexico J,t{,.?; LAWRENCE ERLBAUM ASSOCIATES,

More information

Linear Regression. September 27, Chapter 3. Chapter 3 September 27, / 77

Linear Regression. September 27, Chapter 3. Chapter 3 September 27, / 77 Linear Regression Chapter 3 September 27, 2016 Chapter 3 September 27, 2016 1 / 77 1 3.1. Simple linear regression 2 3.2 Multiple linear regression 3 3.3. The least squares estimation 4 3.4. The statistical

More information

Introduction to Econometrics

Introduction to Econometrics Introduction to Econometrics T H I R D E D I T I O N Global Edition James H. Stock Harvard University Mark W. Watson Princeton University Boston Columbus Indianapolis New York San Francisco Upper Saddle

More information

Contents. Preface to Second Edition Preface to First Edition Abbreviations PART I PRINCIPLES OF STATISTICAL THINKING AND ANALYSIS 1

Contents. Preface to Second Edition Preface to First Edition Abbreviations PART I PRINCIPLES OF STATISTICAL THINKING AND ANALYSIS 1 Contents Preface to Second Edition Preface to First Edition Abbreviations xv xvii xix PART I PRINCIPLES OF STATISTICAL THINKING AND ANALYSIS 1 1 The Role of Statistical Methods in Modern Industry and Services

More information

Review: Second Half of Course Stat 704: Data Analysis I, Fall 2014

Review: Second Half of Course Stat 704: Data Analysis I, Fall 2014 Review: Second Half of Course Stat 704: Data Analysis I, Fall 2014 Tim Hanson, Ph.D. University of South Carolina T. Hanson (USC) Stat 704: Data Analysis I, Fall 2014 1 / 13 Chapter 8: Polynomials & Interactions

More information

Linear Regression In God we trust, all others bring data. William Edwards Deming

Linear Regression In God we trust, all others bring data. William Edwards Deming Linear Regression ddebarr@uw.edu 2017-01-19 In God we trust, all others bring data. William Edwards Deming Course Outline 1. Introduction to Statistical Learning 2. Linear Regression 3. Classification

More information

Local Polynomial Modelling and Its Applications

Local Polynomial Modelling and Its Applications Local Polynomial Modelling and Its Applications J. Fan Department of Statistics University of North Carolina Chapel Hill, USA and I. Gijbels Institute of Statistics Catholic University oflouvain Louvain-la-Neuve,

More information

Any of 27 linear and nonlinear models may be fit. The output parallels that of the Simple Regression procedure.

Any of 27 linear and nonlinear models may be fit. The output parallels that of the Simple Regression procedure. STATGRAPHICS Rev. 9/13/213 Calibration Models Summary... 1 Data Input... 3 Analysis Summary... 5 Analysis Options... 7 Plot of Fitted Model... 9 Predicted Values... 1 Confidence Intervals... 11 Observed

More information

New Statistical Methods That Improve on MLE and GLM Including for Reserve Modeling GARY G VENTER

New Statistical Methods That Improve on MLE and GLM Including for Reserve Modeling GARY G VENTER New Statistical Methods That Improve on MLE and GLM Including for Reserve Modeling GARY G VENTER MLE Going the Way of the Buggy Whip Used to be gold standard of statistical estimation Minimum variance

More information

Institute of Actuaries of India

Institute of Actuaries of India Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics For 2018 Examinations Subject CT3 Probability and Mathematical Statistics Core Technical Syllabus 1 June 2017 Aim The

More information

IDL Advanced Math & Stats Module

IDL Advanced Math & Stats Module IDL Advanced Math & Stats Module Regression List of Routines and Functions Multiple Linear Regression IMSL_REGRESSORS IMSL_MULTIREGRESS IMSL_MULTIPREDICT Generates regressors for a general linear model.

More information

Machine Learning for OR & FE

Machine Learning for OR & FE Machine Learning for OR & FE Regression II: Regularization and Shrinkage Methods Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com

More information

TABLE OF CONTENTS CHAPTER 1 COMBINATORIAL PROBABILITY 1

TABLE OF CONTENTS CHAPTER 1 COMBINATORIAL PROBABILITY 1 TABLE OF CONTENTS CHAPTER 1 COMBINATORIAL PROBABILITY 1 1.1 The Probability Model...1 1.2 Finite Discrete Models with Equally Likely Outcomes...5 1.2.1 Tree Diagrams...6 1.2.2 The Multiplication Principle...8

More information

Time Series Analysis. James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY

Time Series Analysis. James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY Time Series Analysis James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY PREFACE xiii 1 Difference Equations 1.1. First-Order Difference Equations 1 1.2. pth-order Difference Equations 7

More information

Data Analysis. with Excel. An introduction for Physical scientists. LesKirkup university of Technology, Sydney CAMBRIDGE UNIVERSITY PRESS

Data Analysis. with Excel. An introduction for Physical scientists. LesKirkup university of Technology, Sydney CAMBRIDGE UNIVERSITY PRESS Data Analysis with Excel An introduction for Physical scientists LesKirkup university of Technology, Sydney CAMBRIDGE UNIVERSITY PRESS Contents Preface xv 1 Introduction to scientific data analysis 1 1.1

More information

A Modern Look at Classical Multivariate Techniques

A Modern Look at Classical Multivariate Techniques A Modern Look at Classical Multivariate Techniques Yoonkyung Lee Department of Statistics The Ohio State University March 16-20, 2015 The 13th School of Probability and Statistics CIMAT, Guanajuato, Mexico

More information

* Tuesday 17 January :30-16:30 (2 hours) Recored on ESSE3 General introduction to the course.

* Tuesday 17 January :30-16:30 (2 hours) Recored on ESSE3 General introduction to the course. Name of the course Statistical methods and data analysis Audience The course is intended for students of the first or second year of the Graduate School in Materials Engineering. The aim of the course

More information

3 Joint Distributions 71

3 Joint Distributions 71 2.2.3 The Normal Distribution 54 2.2.4 The Beta Density 58 2.3 Functions of a Random Variable 58 2.4 Concluding Remarks 64 2.5 Problems 64 3 Joint Distributions 71 3.1 Introduction 71 3.2 Discrete Random

More information

Data Analysis and Machine Learning Lecture 12: Multicollinearity, Bias-Variance Trade-off, Cross-validation and Shrinkage Methods.

Data Analysis and Machine Learning Lecture 12: Multicollinearity, Bias-Variance Trade-off, Cross-validation and Shrinkage Methods. TheThalesians Itiseasyforphilosopherstoberichiftheychoose Data Analysis and Machine Learning Lecture 12: Multicollinearity, Bias-Variance Trade-off, Cross-validation and Shrinkage Methods Ivan Zhdankin

More information

Unit 10: Simple Linear Regression and Correlation

Unit 10: Simple Linear Regression and Correlation Unit 10: Simple Linear Regression and Correlation Statistics 571: Statistical Methods Ramón V. León 6/28/2004 Unit 10 - Stat 571 - Ramón V. León 1 Introductory Remarks Regression analysis is a method for

More information

Part 6: Multivariate Normal and Linear Models

Part 6: Multivariate Normal and Linear Models Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of

More information

LECTURE 11. Introduction to Econometrics. Autocorrelation

LECTURE 11. Introduction to Econometrics. Autocorrelation LECTURE 11 Introduction to Econometrics Autocorrelation November 29, 2016 1 / 24 ON PREVIOUS LECTURES We discussed the specification of a regression equation Specification consists of choosing: 1. correct

More information