Internal Validity & Research Design. Internal Validity: Assuming that there is a relationship in this study, is the relationship a causal one?

Transcription

1 Internal Validity & Research Design Internal Validity: Assuming that there is a relationship in this study, is the relationship a causal one?

2

3 Problem: How to increase pet adoptions from the center? Solution: Establish a Facebook presence!

4

5 Results (note this is a fictitious example): Dogs Cats TTL May adoptions (no Facebook page) Facebook page created June 1 June adoptions Did the additional communication from the Facebook page cause the increase in pet adoptions?

6 How do we Establish a Cause-Effect Relationship? 3 criteria Temporal Precedence Covariation of the Cause and Effect No Plausible Alternative Explanations

7 How do we Establish a Cause-Effect Relationship? Temporal Precedence Covariation of the Cause and Effect No Plausible Alternative Explanations

8 Research Design R O X O R O O Notation: O = Observations / Measures O subscript = measure taken on that occasion X = Treatments / Programs Rows = Groups R = Random assignment to group N = Nonequivalent groups C = Assignment by cutoff Time = moves left to right

9

10 Types of Designs

11 Types of Designs

12 Single Group Threats to Internal Validity The Single Group Case Posttest only Pretest-Posttest X O O X O History Threat Maturation Threat History Threat Maturation Threat Testing Threat (primed) Instrumentation Threat (change test) Mortality Threat Regression Threat (regression to the mean)

13 Regression to the Mean a statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated. It is a statistical phenomenon. It is a group phenomenon. It happens between any two variables. It is a relative phenomenon. You can have regression up or down. The more extreme the sample group, the greater the regression to the mean. The less correlated the two variables, the greater the regression to the mean.

14 A real example: Exam 1 Exam 2 Pop. Mean Standard Dev Correlation r Exam1 Exam2 = 0.44 Exam 1 Bottom 10% Mean Distance (Std. Devs.) Exam 1 Top 10% Mean Distance (Std. Devs.)

15 A real example: Exam 1 Exam 2 Pop. Mean Standard Dev Correlation r Exam1 Exam2 = 0.44 Exam 1 Bottom 10% Mean Distance (Std. Devs.) Exam 1 Top 10% Mean Distance (Std. Devs.)

16 How do we deal with single group threats to internal validity? Most common: change the research design Most common change: add a control group (Of course, when you add a control group, you no longer have a single group design.)

17 Multiple Group Threats to Internal Validity There is one multiple group threat to internal validity: that the groups were not comparable before the study. A selection bias or selection threat: any factor other than the program that leads to posttest differences between groups. Pretest-Posttest with Control O X O O O Selection-History Threat Selection-Maturation Threat Selection-Testing Threat (primed) Selection-Instrumentation Threat (change test) Selection-Mortality Threat Selection-Regression Threat (regression to the mean)

18 Experimental Design probably the strongest design with respect to internal validity. key to the success of the experiment is in the random assignment (achieves probabilistic equivalence).

19 Probabilistic Equivalence

20 Random Selection Sampling External Validity vs. Random Assignment Design Internal Validity

21 Social Interaction Threats to Internal Validity Social pressures in the research context that can lead to posttest differences Groups Research Admin. Diffusion or Imitation of Treatment Compensatory Rivalry Resentful Demoralization Compensatory Equalization of Treatment

22 The simplest of all experimental designs: two-group posttestonly randomized experiment.

23 Remember this?

24 Remember this?

25 This is an Extremely Simple Design Only One Independent Variable (Factor): Advertising Exposure

26 What about a design with more than one independent variable? Independent Variable 1 (Factor 1): Ad Color Two Levels: B&W (1) / CMYK (2) Independent Variable 2 (Factor 2): Humor in Ad Two Levels: Not Humorous (1) / Humorous (2) Research Design Notation (Post-Test Only) = 4 Groups

27 R X 11 O R X 12 O R X 21 O R X 22 O B&W / Non-humorous B&W / Humorous CMYK / Non-humorous CMYK / Humorous

28 2 x 2 Factoral Design Color B&W Non-humorous X 11 CMYK Non-humorous X 21 Humor B&W Humorous X 12 CMYK Humorous X 22

29 Understanding Factoral Design Numbering Notation Design Factors Levels Groups 2 x / x / x / x 2 x / 2 / x 2 x / 2 / 2 12

30 How Factoral Designs Work Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level 1 Group 1 Mean Group 2 Mean Mean Across Factor 2 Level 2 Group 3 Mean Group 4 Mean Mean Across Mean Down Mean Down

31 No Effects (Null Outcome) Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level Factor 2 Level

32 Main Effect of Factor 1 Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level Factor 2 Level

33 Main Effect of Factor 2 Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level Factor 2 Level

34 Main Effect of Each Factor Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level Factor 2 Level

35 Interaction Effect (one group differs from all others) Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level Factor 2 Level

36 Interaction Effect (Crossover) Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level Factor 2 Level

37 Adding a Control Group R X 11 O B&W / Non-humorous R X 12 O B&W / Humorous R X 21 O CMYK / Non-humorous R X 22 O CMYK / Humorous R O Control 2 x 2 Factoral Design with Control (not a fully-crossed design)

38 Incomplete Factoral Design Factor 1 Level 1 Factor 1 Level 2 Factor 2 Level 1 Group 1 Mean Group 2 Mean Mean Across Factor 2 Level 2 Group 3 Mean Group 4 Mean Mean Across Mean Down Mean Down Control Mean