Using Decision Trees to Evaluate the Impact of Title 1 Programs in the Windham School District

Transcription

1 Using Decision Trees to Evaluate the Impact of Programs in the Windham School District Box Lubbock, Texas Phone: (806) Stats Camp offers week long sessions for academic and industry professionals. For more information and to view all current courses offered, go to: statscamp.org. Eugene W. Wang, Ph.D. Steven R. Chesnut, M.Ed.

2 WINDHAM SCHOOL DISTRICT USING DECISION TREES TO EVALUATE THE IMPACT OF TITLE 1 PROGRAMS IN THE WINDHAM SCHOOL DISTRICT INTRODUCTION The purpose of this evaluation was to identify the impact of Windham School District (WSD) services on academic outcomes, specifically GED attainment (and time to GED), and gains on the Composite, Reading, Language, and Math subtests of the Test of Adult Basic Education (TABE). Gains were computed as the difference between the initial TABE and the highest TABE; if the initial TABE was the highest, gain was computed as 0. The primary predictors of interest were: Total number of hours of o services o ESL services o Special Education services o o Services o Cognitive Intervention Program (CIP) o programming Offender Type (ID/State Jail/SAFP/IS) Offender gender, race, age at first incarceration and at beginning of the school year, sentence length, and offense type Initial TABE scores HIGHLIGHTS: The best predictors of GED attainment were number of vocational hours, race, offender type, and hours (in that order) The best predictors of Composite TABE gains were offender type, computer lab hours/ hours, and vocational hours/ hours, and gender The best predictors of Reading TABE gains were offender type, computer lab/ hours, and age/vocational hours/ hours The best predictors of Language TABE gains were computer lab hours, offender type, and age/title 1 hours/vocational hours The best predictors of Math TABE gains were offender type, computer lab/ hours, and gender/ hours CAVEATS: Not all hours increased academic success (in some cases academic outcomes DECREASED) The dffectiveness of different programs/ strategies was different for different offender subtypes this is most evident in the decision tree graphs 1

3 ABOUT THE SAMPLE Deidentified data were provided to TTU IMMAP from WSD. The sample included all offenders in School Year (SY) 2011, 2012, and 2013 WSD accountability data that were 22 years old or less as of 8/31 of each respective year. The be included in the WSD accountability data an offender must have been an academic participant (Regular academic, ESL, SPED, and/or ), and had two TABE tests during the school year, or a baseline test from a previous school year and a subsequent test during the SY being reviewed. An offender may or may not have met the accountability criteria for three (3) consecutive school years. Data were included for the year(s) the offender met the accountability criteria. Below are descriptive data for the entire sample of 14,196 offenders: Outcomes o Received GED During/After 3-year cohort! Yes = 4,239 (30%)! No = 9,957 (70%) o Days to GED! N = 3,381, Mean = 87.8 days, median = 61.7 days, range = days o Composite TABE Gains (n = 14,195)! mean = 1.7 (grade levels), median = 1.3, range = o Reading TABE Gains! mean = 1.5, median = 0.7, range = o Language TABE Gains! mean = 2.0, median = 1.2, range = o Math TABE Gains! mean = 1.8, median = 1.2, range = Predictors o Offender Type! ID (n = 10,688, 75%)! (n = 2,840, 20%)! SAFP (n = 602, 4.2%)! IS (n = 66, 0.5%) o Gender! Male (n = 12,742, 90%)! Female (n = 1,290, 9.1%)! Missing (n = 164, 1.2%) o Race! White (n = 2,420, 17%)! Black (n = 5,678, 40%)! Hispanic (n = 6,044, 43%)! Asian (n = 39, 0.3%) 2

4 ! Other (n = 5, 0%)! Missing (n = 10, 0.1%) o Age at incarceration, n = 14,195, mean = 20.5, median = 21, range = o Sentence length, n = 14,125, mean = 2,204, median = 1,461, range = 0 36,159 o Number of incarcerations [most data were missing] o Exposure to multiple academic programs (n = 14,196)! Hours: mean = 26.4, median = 0, range = 0 1,446! Hours: mean = 39.0, median = 0, range = 0 1,319! ESL Hours: mean = 1.84, median = 0, range = 0 1,225! SPED Hours: mean = 3.69, median = 0, range = 0 1,302! Hours: mean = 17.1, median = 0, range = 0 1,401! Cognitive Intervention Program (CIP) Hours: mean = 16.6, median = 0, range = 0 489! Program Hours: mean = 31, median = 0, range = o Initial TABE Scores (n = 14,044)! Composite: mean = 6.4, median = 5.9, range = ! Reading: mean = 7.3, median = 6.8, range = ! Language: mean = 5.9, median = 5.3, range = ! Math: mean = 6.3, median = 5.8, range =

5 ANALYTIC METHODS Because it was determined that several variables were confounding the analysis, these variables were not used in the subsequent analyses: Days to GED: this was confounded with number of hours of academic services, and thus any correlations were deemed to be because of a selection/opportunity bias. Total academic hours: rather, specific hours in each type of program or service were used) Age at incarceration was used rather than age at present school year Number of incarcerations had too many missing values CLASSIFICATION & REGRESSION TREES ( DECISION TREES ) Classification tree analysis predicts values of a categorical outcome variable from a number of predictor variables and offers a number of advantages over more commonly used statistical techniques (Horner, Fireman, & Wang, 2010). For instance, classification trees are both nonparametric and nonlinear. These features mean that missing data are not a problem and that typical assumptions regarding normality and linear relationship between variables is neither assumed nor necessary. In the present study, for example, a lack of normality and linear relations may mean that hours of services are related to academic outcomes only above a certain number of hours of service. In this initial exploratory stage with many potential predictor variables and without specific a priori hypotheses about how the predictor variables are related to each other and to academic outcomes this approach is potentially useful in discovering relations between variables that may have otherwise been missed. 4

6 CAVEATS For all the TABE scores (Composite, Reading, Language, and Math) there was an interesting relationship between hours and TABE gains: a small number of hours was actually worse than no hours, but a large number of hours was effective. In statistical terms, this is called a curvilinear relationship between the predictor ( hours) and the outcome (TABE gains). In applied practice, it suggests that if a person is going to receive services, then WSD should ensure that the person has a large enough dose (at least 73 hours) to get the most efficient and effective outcome. 5

7 Receipt of GED Yes = 1,123 (26%) N = 4,398 ID; SAFP Yes = 1,230 (23%) N = 5,401 ; IS Yes = 107 (11%) N = 1,003 Yes = 4,239 (30%) N = 14,196 Hours =0 Yes = 3,702 (28%) N = 13,349 Black, Other Hispanic White Asian Yes = 1,567 (28%) N = 5,681 Yes = 905 (40%) N = 2,267 ID; SAFP ; IS ID; SAFP ; IS Yes = 1,347 (30%) N = 4,432 Yes = 220 (18%) N = 1,249 Yes = 744 (44%) N = 1,683 Yes = 537 (63%) N = 847 Black Hispanic Asian White Yes = 159 (55%) N = 292 Yes = 231 (63%) N = 366 Hours < 73 Yes = 193 (60%) N = 321 Yes = 161 (28%) N = 584 Yes = 147 (78%) N = 189 Yes = 38 (84%) N = 45 Participation in Hours: Students that participated in vocational hours were more than twice as likely to receive their GED than students that did not. Inclusion of Hours: Students that participated in vocational hours and were Hispanic / Latino or Asian benefitted greatly from hours with a 40% greater rate of GED reception. 6

8 Mean = 1.7 N = 10,688 Hours < 48 Hours Hours > 139 Mean = 1.6 N = 8,265 Mean = 1.9 N = 1,1,39 N = 1,052 Mean = 2.8 N = 87 ID Mean = 2.6 N = 1,284 Mean = 2.8 N = 876 Mean = 1.5 N = 14,195 SAFP; IS Mean = 1.3 N = 667 Mean = 1.2 N = 638 Mean = 2.2 N = 29 Mean = 2.2 N = 408 Mean = 0.7 N = 1,986 Mean = 0.8 N = 2,840 Hours < 176 Hours > 176 Mean = 0.7 N = 2,651 Mean = 1.6 N = 189 Mean = 0.5 N = 421 Mean = 1.3 N = 244 Changes Hours: For students in an facility, hours (>176) increased TABE reading gains by nearly a grade level. Hours after : hours were not linearly related. Students that received no hours had higher TABE reading gains than those taking 1 to 73 hours. However, students receiving more than 73 hours of instruction exhibited nearly a half grade level increase in TABE reading gains over their peers. : Students in SAFP and IS facilities demonstrated nearly a 1 grade level increased in TABE reading when given computer lab time. and hours: For students in an ID facility, TABE reading gains showed a linear relationship with computer hours. When large amounts of computer hours cannot be offered, vocational hours can help to bridge that gap and increase TABE reading gains by nearly 1 grade level. 7

9 Mean = 2.1 N = 8,265 Mean = 1.0 N = 1,899 Hours < 48 N = 11,360 ID SAFP; IS Mean = 1.0 N = 2,445 Mean = 1.6 N = 650 Mean = 0.7 N = 366 Mean = 1.9 N = 180 Mean = 2.0 N = 14,195 Hours Hours > 139 Mean = 2.4 N = 1,412 ID; SAFP; IS Mean = 2.7 N = 1,152 Mean = 1.3 N = 260 Mean = 2.6 N = 1,065 Mean = 4.1 N = 87 Mean = 3.4 N = 1,423 ID; SAFP; IS Mean = 3.5 N = 1,288 Mean = 2.1 N = 135 Mean = 3.4 N = 1,171 Mean = 4.5 N = 117 Hours: Students that received more than 139 computer lab hours exhibited nearly 2 grade level gains in language above those that had fewer hours (<48). Hours: Students in ID, SAFP, and IS facilities benefited greatly from vocational hours by nearly 1 grade level above those without vocational hours. influence with Fewer Lab Hours: Students in facilities that received less than 48 computer lab hours benefited greatly from more than 73 hours of instruction. Hours with Lab Hours: Students in ID, SAFP, and IS facilities that received moderate amount of computer lab time demonstrated nearly a grade and a half increase in TABE language gains over those without. 8

10 Mean = 2.0 N = 10,688 Hours < 48 Hours Hours > 139 N = 8,265 Mean = 2.2 N = 1,139 Mean = 2.1 N = 899 Mean = 1.5 N = 68 N = 6,971 Mean = 1.3 N = 558 Mean = 2.8 N = 736 N = 14,195 ID SAFP; IS Mean = 1.5 N = 667 Mean = 3.0 N = 1,284 Mean = 1.4 N = 638 Mean = 2.5 N = 29 Male Female Mean = 3.0 N = 172 Male Female Mean = 1.3 N = 491 N = 147 Mean = 2.9 N = 1,237 Mean = 4.5 N = 47 Mean = 0.9 N = 2,840 Mean = 0.9 N = 2,150 Mean = 0.6 N = 430 Hours < 176 Hours > 176 Mean = 0.8 N = 1,986 Mean = 1.5 N = 164 Mean = 1.7 N = 260 Hours: For students in facilities, hours maintained a non-linear relationship with TABE math gains. Those receiving over 73 hours averaged 1 grade gain more than those that received fewer hours., Lacking Hours: For students in facilities that did not receive hours, hours in excess of 176 hours promoted nearly 1 grade level more in TABE math gains than in those that received less. Hours to Supplement Lab Hours: Students in ID facilities that received few computer lab hours exhibited nearly a 1 grade increase in TABE math gains after receiving more than 73 hours of instruction over their peers. Additionally, those that received a moderate amount of computer lab hours also saw nearly a 1 grade increase in TABE math gains when they additionally received more than 73 hours of instruction. 9

11 Mean = 1.9 N = 10,688 Computer Hours < 48 Computer Hours Computer Hours > 139 N = 8,265 Mean = 2.2 N = 1,139 Mean = 2.1 N = 1,052 Mean = 1.7 N = 6,971 Mean = 1.2 N = 558 Mean = 2.5 N = 736 ID Mean = 2.9 N = 1,284 Mean = 3.4 N = 87 Mean = 2.8 N = 1,167 Mean = 1.7 N = 14,195 SAFP; IS Mean = 1.4 N = 667 Mean = 1.4 N = 638 Mean = 2.5 N = 29 Mean = 3.7 N = 117 Mean = 0.9 N = 2,840 Mean = 0.9 N = 2,150 Mean = 0.6 N = 430 Mean = 1.5 N = 260 Hours < 176 Hours > 176 Male Female Mean = 0.8 N = 1,986 Mean = 1.6 N = 164 Mean = 1.7 N = 221 Mean = 0.9 N = 39 Hours and : Students in facilities that received more than 73 hours of instruction exhibited nearly half a grade more in TABE composite gains than those that received none. Additionally, they gained nearly 1 grade more in composite gains than those that received less than 73 hours of instruction. Students that received no hours exhibited nearly a grade increase in TABE composite gains over their peers when they received greater than 176 hours of instruction. Hours: For students participating in few computer lab hours, those that received over 73 hours of instruction exhibited nearly a grade level increase in TABE composite gains over their peers receiving no hours and over a grade compared to those receiving 1-73 hours. Hours: For students receiving moderate to large amounts of computer lab time, those engaging in vocational hours exhibited nearly a grade level increase in TABE composite gains over their peers. 10