Advising on Research Methods: A consultant's companion. Herman J. Ader Gideon J. Mellenbergh with contributions by David J. Hand

Transcription

1 Advising on Research Methods: A consultant's companion Herman J. Ader Gideon J. Mellenbergh with contributions by David J. Hand

2 Contents Preface 13 I Preliminaries 19 1 Giving advice on research methods 21 (by David J. Hand) The consulting environment Communication Problem formulation The consultant's tools Training consultants Conclusion Further reading 38 2 Methodological consultancy problems 39 (by Gideon J. Mellenbergh) Design questions Questions on measurement instruments Data analysis questions Questions concerning reporting 46 3 Methodological quality 49 (by Herman J. Ader) Research question and research plan Literature assessment by systematic reviewing Study design Quality of the measurement instruments 57

3 4 CONTENTS 3.3 Implementation and data collection Data collection and data quality Distributed data collection Data quality Data quality and data analysis : Data quality and consultation Data quality in data mining The interpretation of the analysis results Methodological Quality in practical consultation Further reading 67 Exercises 70 4 Dissemination and publishing 71 (by David J. Hand) 71 Further reading 77 Topics for classroom discussion 78 II Sampling and Design 81 5 Sampling 83 (by David J. Hand) Background fundamentals Sampling methods Simple random sampling Stratified sampling Cluster sampling Choice of sampling frame Other issues Further reading 104 Exercises General issues of research design 107 (by Gideon J. Mellenbergh) Scientific problems Constraints Elements of the systematic plan Ill Units Ill Variables Ill

4 CONTENTS Measurement occasions Precision Generalization Typical topics for consultation 125 Exercises Research designs: Description, exploration, prediction 129 (by Gideon J. Mellenbergh)... ~ State description Process description State exploration Process exploration State prediction Process prediction 140 Exercises Research designs: Testing of research hypotheses 143 (by Gideon J. Mellenbergh) State hypothesis-testing problems Independent variable types Design types Data collection for hypothesis-testing designs Multiple substantive independent variables Background variables Null hypothesis testing Alternative explanations Prevention of alternative explanations in experimental designs Prevention of alternative explanations in quasi-experimental designs Prevention of alternative explanations in correlational designs Combining quantitative and qualitative studies for causal interpretations Giving advice on State hypothesis-testing problems Process hypothesis-testing problems Discussion topics (Process hypothesis-testing problems) Further reading 179 Exercises 179

5 6 CONTENTS III Measurement Surveys 183 (by Gideon J. Mellenbergh) Blueprint Questions and response formats Questions Response formats Administration modes Question ordering Pretests Experts Respondents and interviewers Coders Pilot Nonresponse reduction Unit nonresponse prevention Item nonresponse prevention Data processing Discussion topics Further Reading 208 Exercises Tests and questionnaires: Construction and administration 211 (by Gideon J. Mellenbergh) Blueprint Construct Method Facet Design Method Item production Test items Questionnaire items Pilot studies First draft of the instrument Test assembly Questionnaire assembly Administration modes Try-out 234 Exercises 234

6 CONTENTS 7 11 Tests and questionnaires: Analysis 235 (by Gideon J. Mellenbergh) Data processing Observed score analysis Classical observed score analysis Classical item analysis Item response analysis : Linear item response functions Logistic item response functions Monotonically nondecreasing item response functions Measurement invariance Validation Compound scores The gain score The product score Consultation topics Software 266 Exercises 267 IV Data analysis: Basics Modelling 271 (by Herman J. Ader) Hypothesis testing, estimation, prediction and modelling Statistical hypothesis testing Estimation Prediction Modelling Statistical modelling Methodological modelling Justification versus discovery Modelling in the context of justification Modelling in the context of discovery Methodological modelling and the contexts of justification and discovery Fitting Stepwise approaches User control Model search 293

7 8 CONTENTS Once more: User control Practical issues Further reading 302 Exercises Missing and biasing information 305 (by Herman J. Ader) Bias caused by defects in sampling frame Missing data A Methodological versus a statistical view of the missing data problem Imputation Handling outlying observations Hidden and overt bias Randomization Propensity scores Estimating propensity scores Confounding Software Further reading 329 Exercises Phases and initial steps in data analysis 333 (by Herman J. Ader) Data maintenance The initial data analysis (IDA) phase Analyses to assess data quality Analyses to assess measurement quality Initial transformations Has the implementation of the research design been successful? Analyses aimed at establishing the structure of the sample 'Corrective actions' and adapting the plans for the main analysis phase Techniques used during IDA Nominal and ordinal variables Continuous variables Costs. A special kind of continuous variable Exercises 355

8 CONTENTS 9 15 The main analysis phase. 357 (by Herman J. Ader) Documentation Comparability of measurements Exploratory versus confirmatory approaches Once more: Statistical modelling The main and final analysis phase. v Controlling and checking the stability of the results Crossvalidation: analyses on strictly different cases Sensitivity analysis: analyses on data of the same cases Methodological aspects of bootstrapping Situations in which bootstrap procedures are useful Disadvantages and objections Cross-sectional data analysis Ordinal variables Exploratory cross-sectional data analysis *Confirmatory cross-sectional data analysis: the use of multilevel analysis Effect studies Analyzing change and the development over time Further reading 383 Exercises 384 V Data analysis: Techniques The analysis of longitudinal designs 389 (by David J. Hand) Preliminaries Methods based on analysis of variance The univariate analysis of variance approach to repeated measures analysis The multivariate analysis of variance approach to repeated measures analysis General regression models Regression approach with arbitrary covariance matrices Multistage models Non-normal data 404

9 10 CONTENTS 16.5 Further issues Survival analysis The literature Data analysis and software Conclusion 412 Exercises Regression analysis and beyond 415 (by Herman J. Ader) Regression analysis, analysis of covariance and multilevel analysis Regression analysis Regression diagnostics Special topics in regression analysis Univariate analysis of covariance Reducing from complex to simple Checks on assumptions at the end of the analysis cycle Once more: Contrasts Analysis of covariance Covariates in the analysis of repeated measurements When to use what? Multivariate analysis of variance models Multilevel analysis Some technical remarks The analysis of longitudinal data Repeated measures analysis of variance Repeated measures multilevel modelling and generalized estimating equations Assessment of interobserver agreement Software Further reading 456 Exercises Principal component and factor analysis 463 (by Herman J. Ader) Exploratory and confirmative factor analysis Principal component analysis Confirmatory factor analysis 478

10 CONTENTS Mokken's scale analysis Special topics Further reading Software 486 Exercises Epilogue: Structural equation modelling 489 (by Herman J. Ader) ".' Structural equation modelling Graphical modelling Further reading Software 499 Exercises 499 Appendices 503 A A submission process 503 B Functional notation 519 C Imputation: Theoretical background 525 D Standard software used in this book 527 Index 530 List of authors 546 References 553