Predicting Retention Rates from Placement Exam Scores

Transcription

1 Predicting Retention Rates from Placement Exam Scores Dr. Michael S. Pilant, Dept. of Mathematics, Texas A&M University Dr. Robert Hall, Dept. of Ed. Psychology, Texas A&M University Amy Austin, Senior Lecturer, Dept. of Mathematics, Texas A&M University Marcia Drost, Senior Lecturer (retired), Dept. of Mathematics, Texas A&M University March 18, 2012 Abstract During the past 5 years, over 15,000 students have taken a math placement exam at Texas A&M University to determine their readiness for Calculus. By tracking the performance of those students who subsequently enrolled in Calculus, a very accurate model has been developed which predicts the probability of success (retention rate) in Calculus based on precalculus math placement scores. 1 Introduction The goal of this paper is to be able to accurately compute the retention rate R(s) where s is the score on the math placement exam (MPE), and 0 s S max. The retention rate R(S) is defined as the percentage of students in a group Ω who pass the course, and whose MPE scores exceed S. We begin by considering P (S), where P (S) is the number of students with scores s S who pass the course. This function is guaranteed to be monotone increasing, and typically has the shape of an S-curve, as shown in Figure 1. This figure was obtained, using historical data from thousands of students. By historical we mean data for students who have already taken the Math Placement Exam and taken Calculus I and received a grade. 1

2 Figure 1: Data vs Logistic Model We model it as a two parameter logistic (2-PL) function of the following form P (S) N P = e a P b P S (1) where N P is the total number who pass (N P = P (S max )), and the parameters a P, b P define the two parameter logistic function. Similarly, let F (S) denote the number of students with scores s S who fail the course. This function has the same general shape, and can also be modeled by a two parameter logistic function F (S) N F = e a F b F S (2) where N F is the total number who fail (N F = F (S max )), and the parameters a F, b F define the two parameter logistic function. Clearly N(S) = P (S) + F (S) where N(S) is the total number of students with MPE scores s S. We approximately the probability density functions by differentiating the cumulative density functions

3 p(s) = P (S) N P = b P e ap bp S (1 + e a P b P S ) 2 (3) The function p(s) has a maximum at s = a P /b P and a maximum height of b P 4. The mean for this distribution is µ P = a P /b P and the standard deviation is σ P = Likewise, π b P 3 f(s) = F (S) N F = b F e af bf S (1 + e a F b F S ) 2 (4) The function f(s) has a maximum at s = a F /b F and a maximum height of b F 4. The mean for this distribution is µ F = a F /b F and the standard deviation is σ F = We can solve these equations for a, b in terms if µ, σ by π b F 3 b P = π σ P 3 (5) a P = b P µ P (6) b F = π σ F 3 (7) a F = b F µ F (8)

4 2 Fall 2010 Data set The actual numbers from Fall 2010 are given in the table below s p(s) f(s) P (s) F (s) N(s) Table 1: Fall 2010 pass/fail data 3 Comparison with Normal Distributions The cumulative density functions are of the form: 1 Π(S) = 1 + e a bs (9) The probability density functions are given by π(s) = Π (s) = be a bs (1 + e a bs ) 2 (10)

5 Although the optimal choice of model parameters for logistical regression is determined by maximal likelihood, we approximate this by equating the mean and standard deviation of the logistic function to that of the sample population µ = a b σ = π b 3 In terms of µ, σ we may write (10) as π(s) = π 3σ e π(s µ)/( 3σ) (1 + e π(s µ)/( 3σ ) 2 (11) The equivalent normal distribution (with mean µ and standard deviation σ) is given by ν(s) = 1 2πσ e (s µ)2 /(2σ 2 ) These differ slightly at the maximum probability s = µ, as shown in Figure 2 below: π(µ) = π 4 3σ = σ 1 ν(µ) = 1 2πσ = σ 1 Figure 2: Probability Density Functions The logistic distribution has the same first and second moment as the normal distribution. The logistic function is not symmetric, so the third moment differs from zero. The fourth moment

6 defines the excess kurtosis, which is 1.2 for the logistic and zero for the normal distribution. This indicates that the logistic distribution has a sharper peak, and longer flatter tails, which is the case. This appears to make it more suitable for distribution of test scores which have flatter tails than those given by a normal distribution. The cumulative density functions are almost indistinguishable, with the slope of the logistic cumulative density function slightly steeper at the median. The primary advantage of the logistic function is that it has an explicit cumulative density function, as opposed to the normal distribution s error function. Figure 3: Cumulative Density Functions We now have a two-parameter family of functions which describe the probability of passing given a particular placement score. The distributions are defined in terms of the parameters a, b or equivalently µ, σ. To complete the mathematical model, we examine how the parameters of the passing and failing distributions are related, through the statistics of the MPE. 4 MPE statistical parameters The math placement exam consists of 33 multiple choice questions, and was written by Amy Austin and Marcia Drost - senior lecturers with extensive experience in both precalculus and calculus and Texas A&M University. Each question is chosen from a test bank of 15 questions, and the stems are randomly permuted. The MPE exam is timed (90 minutes), un proctored, and calculators are not allowed. The test may be taken any time after a student is admitted to Texas A&M, and can be repeated after 30 days have

7 passed. Beginning in Fall 2011, a score of 22 or higher out of 33 is required to enroll in Calculus I. Historically, each question has several parameters associated with it, given by the ratio of correct responses to the total number of responses, for each of the two groups of passing and failing students: µ k = #correct N (12) µ k,p = #correct N p (13) µ k,f = #correct N f (14) The individual test item means µ are an indication of the overall difficulty of the question, and the difference of the means measures the ability of a particular question to discriminate between passing and failing students. δ k = µ k,p µ k,f (15) for each question. The historical difference in means, for a given set of questions, is therefore = k δ k = µ P µ F (16) The MPE test mean is related through µ = N F N µ F + N P N µ P = (1 R)µ F + Rµ P (17) where R is the historical probability of being in the passing group (that is, the overall retention rate). We can solve (16) and (17) to obtain µ F = µ R (18) µ P = µ + (1 R) (19) This allows us to estimate the scores on the MPE exam from the overall mean, and the historical passing rates for Calculus.

8 The standard deviations are related through a weighted average We can define a parameter κ by the following σ 2 = Rσ 2 P + (1 R)σ 2 F + R(1 R) 2 (20) σ F σ P = 1 + κ (21) (kappa measures the relative size of the passing and failing standard deviations.) (18) and (19) can be solved to obtain σ σ P = 2 R(1 R) 2 R + (1 R)(1 + κ) 2 (22) σ F = (1 + κ) σ P (23) Given a set of N students, and calculating their MPE scores based on a set of k 33 questions, we now have enough information to calculate the probability and cumulative density functions. 5 Computational Algorithm 1. Compute the overall historical passing rate for Calculus for previous semesters. Passing is defined as achieving a grade of A, B, or C. This number R is quite stable given the large number of students taking Calculus each year. It is slowly increasing, but we can use the previous years retention rate as a good estimate of R. Also, compute the mean of the MPE µ and the standard deviation σ, and the difference between the means of those who pass and those who fail, = µ P µ F. 2. Choose a sample of N students that have taken the MPE exam and are enrolled in (or have taken) Calculus. We now have the parameters {N, µ, σ, R, } available. 3. The total number of students expected to pass is therefore given by N P = RN and the total number expected to fail is given by N F = (1 R)N

9 4. The mean MPE scores for the two groups (pass, fail) is given by µ P = µ + (1 R) and µ F = µ R 5. Given the heuristic constant κ = 0.05, we can calculate the standard deviations by σ σ P = 2 R(1 R) 2 R + (1 R)(1 + κ) 2 and σ F = (1 + κ) σ P 6. The Logistic parameters are then estimated by (a) b F = π/( 3σ F ), a F = b F.µ F (b) b P = π/( 3σ P ), a P = b P.µ P 7. The Logistic CDFs are therefore given by (a) P (S) = (b) F (S) = N P 1+e a P b P S N F 1+e a F b F S We now have enough information to estimate the following 1. False Negatives: Number of individuals who pass with scores s S P (S) 2. Positives: Number of individuals who pass with scores s > S N P P (S) 3. Negatives: Number of people who fail with scores s S F (S) 4. False Positives: Number of people who fail with scores s > S N F F (S) and compare it to actual data. For example, one can compute the value of the cutoff score S C such that the number of false positives equals the number of false negatives

10 6 Retention Rate vs MPE scores We can now compute the theoretical retention rate function R(S) defined as the percentage of students with scores s S who pass. R(S) = N P P (S) N (P (S) + F (S)) We can compare it to historical data, as shown in the table below. We can also compute a cutoff score S C defined by the requirement that R(S C ) = Correlation between MPE scores and grades If one calculates the Pearson correlation between the MPE scores and the pass/fail grades (pass=1, fail=0) for the Fall 2010 data set, we get the following R 2 =... This is not very good, indicating that only a third of the variance of the outcome is explained by the MPE scores. This is the reason why we went to an odds or probability based outcome measure. 8 Summary Based on a large sample of students who have taken the Math Placement Exam (MPE), and who have subsequently enrolled in Calculus, we have developed a model that estimates the likelihood that a student will pass Calculus, given a particular score on the MPE. Because of the binary nature of the result (pass=1, fail=0) we can only estimate the probability that a student with a particular score will succeed in calculus. However, due to the large sample size, the logistical models predictions are quite accurate. The existence of a highly accurate parameterized model relating probability of success to a small number of parameters allows one to apply various optimization techniques, perturbation analysis and other mathematical tools to help understand the relationship between the individual MPE questions, the MPE exam, and probability of success.