Presented by. Philip Holmes-Smith School Research Evaluation and Measurement Services

Transcription

1 The Statistical Analysis of Student Performance Data Presented by Philip Holmes-Smith School Research Evaluation and Measurement Services

2 Outline of session This session will address four topics, namely:. Item Response Theory (IRT) -the statistical theory underpinning the scale used to report the National Assessment Program -Literacy and Numeracy (NAPLAN) results. 2. Understanding the NAPLAN Scale Score. 3. An overview of the statistics available through the MySchool website. 4. An overview of the web-based Student Performance Analyser (SPA) program used to interpret NAPLAN and other testing data.

3 Item Response Theory (IRT)

4 Year 7 NAPLAN Numeracy Test

5 Number Test Results (8 items) StudID N320 N307 N32 N30 N323 N322 N308 N35 N302 N324 N309 N37 N33 N304 N325 N303 N39 N36 Total Score Correct Mean % 93.9% 88.4% 8.3% 79.4% 76.4% 70.4% 68.% 62.7% 6.4% 58.9% 54.9% 48.9% 48.4% 40.% 40.0% 29.8% 26.7% 24.5% 0.54

6 Test Score Frequencies Test Score N Freq % 508.3% % 3, % 4, % 5, % 6 2,39 5.3% 7 2, % 8 2,67 6.6% 9 2,87 7.0% 0 3,40 7.8% 3,22 7.9% 2 3, % 3 3, % 4 3, % 5 2,869 7.% 6 2, % 7, % 8,3 2.8% Total Students 40, % Mean 0.54 Frequency 0.0% 9.0% 8.0% 7.0% 6.0% 5.0% 4.0% 3.0% 2.0%.0% 0.0% Distribution of test scores Test Score

7 Test Score Frequencies Cummulative Total Test Score N Cum Tot Cum Freq % % % 3, % 4, % 5, % 6 2, % 7 2, % 8 2, % 9 2, % 0 3, % 3, % 2 3, % 3 3, % 4 3, % 5 2, % 6 2, % 7, % 8, % Cummulative Freque ency 00.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 0.0% 0.0% Total Test Score

8 Proportion of students answering correctly by total test score Item No. Test Score N N320 N307 N32 N30 N323 N322 N308 N35 N302 N324 N309 N37 N33 N304 N325 N303 N39 N % 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% % 2.% 4.3% 8.9% 5.9% 2.2% 8.% 4.5% 3.0% 0.2% 0.6% 0.6%.0%.0% 0.2% 0.4% 0.0% 0.4% % 37.3% 7.8% 20.4% 4.6% 6.% 4.3% 9.4% 4.8% 3.2%.% 0.9% 3.2% 4.9% 0.5% 0.8% 0.4%.% 3, % 53.0% 33.8% 28.6% 28.2% 2.% 20.2% 5.6% 8.0% 5.2% 3.7%.9% 5.0% 4.7% 2.4%.4% 0.8%.5% 4, % 63.2% 47.0% 39.% 42.7% 8.4% 27.9% 24.5% 2.5% 0.4% 5.6% 4.% 7.0% 4.6% 3.8% 2.0%.2% 2.% 5, % 72.3% 6.0% 49.5% 50.6% 25.3% 36.3% 32.3% 9.4% 6.8% 9.9% 6.4% 9.3% 8.% 5.6% 2.3% 2.% 3.4% 6 2,39 9.9% 79.3% 67.9% 64.9% 60.7% 36.6% 43.% 38.4% 27.% 22.3% 3.6% 9.4% 5.% 0.8% 7.9% 4.0% 2.7% 4.4% 7 2, % 85.2% 74.4% 69.9% 69.8% 45.0% 50.7% 46.7% 36.4% 33.0% 20.4% 4.4% 8.% 3.4% 2.8% 5.8% 4.3% 6.3% 8 2, % 89.3% 79.3% 77.2% 70.6% 56.2% 59.6% 49.9% 46.7% 39.0% 29.5% 22.9% 27.9% 8.% 5.9% 8.0% 6.7% 7.7% 9 2,87 96.% 9.6% 82.7% 82.3% 78.2% 66.4% 63.7% 56.4% 55.% 48.2% 39.9% 30.4% 34.2% 2.2% 2.4% 2.5% 9.3% 0.4% 0 3, % 92.7% 87.0% 84.8% 79.4% 76.3% 70.4% 62.0% 6.8% 58.5% 49.5% 4.4% 40.7% 26.3% 29.4% 5.9% 3.3% 2.8% 3, % 95.6% 88.7% 88.2% 83.0% 83.3% 75.2% 66.9% 68.6% 64.4% 60.8% 52.9% 49.8% 35.0% 34.2% 2.2% 7.3% 7.4% 2 3, % 96.% 90.7% 90.6% 86.2% 88.0% 80.5% 70.9% 75.0% 73.5% 70.8% 60.3% 57.0% 43.8% 43.9% 28.5% 23.2% 22.5% 3 3, % 97.4% 92.8% 92.2% 87.9% 92.% 82.8% 76.3% 8.3% 78.8% 79.3% 69.% 66.5% 52.5% 54.3% 37.5% 3.9% 28.4% 4 3, % 98.4% 95.% 93.6% 90.5% 94.9% 86.3% 80.9% 84.2% 86.0% 86.4% 80.0% 73.3% 62.8% 64.3% 48.0% 42.% 33.9% 5 2, % 99.% 96.7% 95.5% 93.7% 96.5% 89.2% 86.4% 90.% 90.4% 90.5% 86.4% 80.8% 73.8% 73.9% 59.6% 53.5% 44.3% 6 2, % 99.2% 97.2% 96.6% 95.5% 98.2% 92.0% 9.8% 93.0% 94.2% 95.4% 90.9% 88.5% 83.9% 84.8% 7.5% 69.0% 58.7% 7, % 99.8% 98.6% 98.2% 97.8% 99.3% 95.3% 96.2% 96.7% 97.7% 98.% 96.6% 93.4% 93.6% 93.4% 86.4% 83.6% 75.5% 8,3 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0% 00.0%

9 Proportion of students answering N309 correctly by total test score Test Score N N % % % 3, % 4, % 5, % 6 2,39 3.6% 7 2, % 8 2, % 9 2, % 0 3, % 3, % 2 3, % 3 3, % 4 3, % 5 2, % 6 2, % 7, % 8,3 00.0% Proportion of students answering ite em correctly 00.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 0.0% 0.0% "Number" Question N Total "Number" Test Score

10 Item Characteristic Curve 0.9 Item Characteristic Curve Probability of Answering Item Correctly Probability of answering correctly = 50% Students with ability less than - are more likely than not to answer this item incorrectly whereas students with ability of more than - are more likely than not to answer this item correctly. The ability level of students at which it becomes more likely that they answer the item correctly than incorrectly is called the Item Difficulty parameter. 0. Item Difficulty = Ability of Students

11 A variety of Item Characteristic Curves Probability of Answering Item Correctly ZZZZZZZZ Each Item has its own Item Characteristic Curve. The curves can differ in terms of their: - slope (discrimination), - location (difficulty), and - intercept (degree of guessing) Ability of Students

12 Item Characteristic Curves Normal Ogive P(θ), the probability of a person with ability θ answering an item correctly, can be expressed as: 2 a( θ b) t P( θ ) = c + ( c) 2 e dt 2 π where: θ = the ability level, a = the slope of the curve at the point of inflection, b = difficulty of the item (i.e. the point it becomes more likely than not that the student answers correctly), c = probability of someone who completely lacks any ability getting the item correct.

13 The Logistic Function Three-Parameter Model P(θ), the probability of a person with ability θ answering an item correctly, can be expressed as: e + e.7a( θ b) P ( θ ) = c + ( + c).7 a( θ b) where: θ = the ability level, a = the slope of the curve at the point of inflection, b = difficulty of the item (i.e. the point it becomes more likely than not that the student answers correctly), c = probability of someone who completely lacks any ability getting the item correct. Note, we call: a = discrimination parameter b = difficulty parameter c = guessing parameter

14 Other Models Two-Parameter Model P(θ), the probability of a person with ability θ answering an item correctly, can be expressed as: P( θ ) = e + e.7a ( θ b) P ( =.7a ( θ b) where: θ = the ability level, a = the slope of the curve at the point of inflection, b = difficulty of the item (i.e. the point it becomes more likely than not that the student answers correctly).

15 Other Models (cont.) One-Parameter Model (Rasch Model) P(θ), the probability of a person with ability θ answering an item correctly, can be expressed as: ( θ b ) P ( θ ) = e ( θ b) + e where: θ = the ability level, b = difficulty of the item (i.e. the point it becomes more likely than not that the student answers correctly),

16 RaschModel Probability of Answering Correctly In Australia, most testing programs have adopted a "one parameter" (Rasch) model. (i.e. The only differences between items are their item difficulties.) Ability

17 Is the Raschassumption assumption valid? Proportion Correct N320 N307 N32 N30 N323 N322 N308 N35 N302 N324 N309 N37 N33 N304 N325 N303 N39 N Score

18 Raschexample Using the One-Parameter (Rasch) Model: ( θ e ( + e b) P ( θ ) = θ b) then the likelihood of a person with ability θ = 0 correctly answering an item of difficulty b = -0.5 can be calculated as follows: (0 e + e 0.5) P ( 0) = (0 0.5) P(0) = 0.622

19 Consider five items of known difficulty (-.0, -0.5, 0.0,.0 and.5 respectively) Probability of answering correctly P(θ) Item Item 2 Item 3 Item 4 Item Item Difficulty/ Student Ability (θ)

20 Likelihood of an Item Responses pattern Items to 5 are marked as either correct or incorrect and are coded as: (correct) or 0 (incorrect). For any given student let: µ be their response to item µ 2 be their response to item 2 µ 3 be their response to item 3 µ 4 be their response to item 4 µ 5 be their response to item 5 Suppose a student with ability θ had the following responses: Item : (correct) Item 2: (correct) Item 3: 0 (incorrect) Item 4: 0 (incorrect) Item 5: 0 (incorrect) The likelihood of this response pattern (assuming the items are independent) is the product of the individual likelihoods for each item. L(µ, µ 2,... µ 5 θ) = n j= µ j ( µ j ) j P j Q where the j th j P µ j is the likelihood of answering item correctly and ( j ) Q µ j is the likelihood of answering the j th item j incorrectly (i.e. - P µ ) j For the above response pattern: L(µ, µ 2,... µ 5 θ) = P P 2 Q 3 Q 4 Q 5

21 Likelihood of the Item Response pattern for a person of ability θ= = 0 Question Response Difficulty (i) (u i ) (b i ) Likelihood of correct answer (P) 0.00 Likelihood of incorrect answer (Q) -.0 P ( θ m )= ( θ m b ) + e P 2 ( θ m )= ( θ m b2 ) + e P 3 ( θ m )= ( θ m b3 ) + e P 4 ( θ m )= ( θ m b4 ) + e P 5 ( θ m )= ( θ m b5 ) + e θ 0 = L(µ, µ 2,... µ 5 θ) = P P 2 Q 3 Q 4 Q 5 L(µ, µ 2,... µ 5 θ) = 0.73 * 0.62 * 0.50 * 0.73 * 0.83 = 0.35

22 Likelihood of answering each of the five items correctly for a person of ability θ= = Probability of answering correctly P(θ) Item Item 2 Item 3 Item 4 Item Item Difficulty/ Student Ability (θ)

23 Likelihood of response patterns for three examinees across a range of possible ability levels Question Response Difficulty (i) (u i) (b i) P ( θ m )= P 2 ( θ m )= P 3 ( θ m )= P 4 ( θ m )= P 5 ( θ m )= Question Response Difficulty (i) (u i) (b i) P ( θ m )= P 2 ( θ m )= P 3 ( θ m )= P 4 ( θ m )= P 5 ( θ m )= Question Response Difficulty (i) (u i) (b i) P ( θ m )= P 2 ( θ m )= P 3 ( θ m )= P 4 ( θ m )= P 5 ( θ m )=

24 Likelihood of response patterns for three examinees across a range of possible ability levels Log Likeliho ood Examinee Examinee 2 Examinee Ability

25 Estimating Ability initial estimate st iteration 2nd iteration 3rd iteration m=0 m= m=2 m=3 Question Response Difficulty θ 0 = θ 0 = θ = θ 2 = θ 3 = (i) (u i ) (b i ) Ln(#right/(n - #right)) P ( θ m )= ( θ m b ) + e P 2 ( θ m )= ( θ m b2 ) + e P 3 ( θ m )= ( θ m b3 ) + e P 4 ( θ m )= ( θ m b4 ) + e P 5 ( θ m )= ( θ m b5 ) + e =correct 0=incorrect n ( u i P i ( θm)) i= n 2 P i ( θ m )( Pi ( θ m )) i= h = / θ m+ = θ m - h

26 Estimating Item Difficulty initial estimate st iteration 2nd iteration 3rd iteration m=0 m= m=2 m=3 Examinee Response Ability b 0 = b 0 = b = b 2 = b 3 = (i) (u i ) (θ i ) Ln((n - #right)/#right) P ( θ m )= ( θ m b + e ) e + e + e + e P 2 ( θ m )= θ ( m b2 ) P 3 ( θ m )= P 4 ( θ m )= P 5 ( θ m )= =correct 0=incorrect n ( u i P i ( θm)) i= n 2 P i ( θ m )( Pi ( θ m )) i= ( θ b3 ) m ( θ m b4 ( θ m b5 ) ) h = / θ m+ = θ m + h

27 Item Characteristic Curves for the Partial Credit Model Partial Credit Model Score=0/3 Score=/3 Score=2/3 Score=3/3 Score = 0 Score = Score = 2 Score = 3 δ3 δ δ Ability(θ) Prob( -3.5 )

28 Item Characteristic Curves for the Partial Credit Model Partial Credit Model Prob(η) Score = 0 Score = 2 Score = 3 Score = Score = 4 Score = 5 Score = Ability(θ)

29 Understanding the NAPLAN Scale Score

30 Understanding the National Scale The National Scale is an arbitrary scale at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future.

31 Understanding the National Scale STD 2008 National Mean - STD The National Scale is an arbitrary scale at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future. The National Scale was fixed in 2008 as follows: Range: Mean: 500 Standard Deviation: 00 (i.e. 68% students between )

32 National Averages (2008) My School Website (2009) School s Average National Average Average for school s with similar ICSEA Only 20.% of Australian school communities are more advantaged

33 Understanding the National Scale The National Scale is an arbitrary scale at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future. The National Scale was fixed in 2008 as follows: Range: Mean: 500 Standard Deviation: 00 (i.e. 68% students between ) The Scale has been divided into ten Bandswhich are used for reporting to parents. Band covers all scores equal to or less than 270. Bands 2 9 increment by 52 score points each Band Band 0 covers all scores above 686.

34 Parent Reports

35 Parent Reports (NB. Bands do not equate to Year Level means) Year 9 Year 7 Band 0 Year 5 Band 9 Band 9 Year 3 Band 6 Band 8 Band 7 Band 6 Yr 5 Means Band 8 Band 7 Band 6 Yr 7 Means Band 8 Band 7 Band 6 Yr 9 Means Band 5 Band 4 Yr 3 Means Band 5 Band 4 Band 5 Band 4 Band 5 Band 3 Band 2 Band Band 3 Above National Minimum Standard At National Minimum Standard Below National Minimum Standard

36 Understanding the National Scale The National Scale is an arbitrary scale at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future. Yr 9 Yr 7 Yr 5 Yr 3 The National Scale was fixed in 2008 as follows: Range: Mean: 500 Standard Deviation: 00 (i.e. 68% students between ) The Scale has been divided into ten Bandswhich are used for reporting to parents. Band covers all scores equal to or less than 270. Bands 2 9 increment by 52 score points each Band Band 0 covers all scores above 686. At this stage Bands have no explicit curriculum meaning but results show that for Victorian students in 200: A typical Yr3 level of performance is at the bottom of Band 5 A typical Yr5 level of performance is almost halfway into Band 6 A typical Yr7 level of performance is a third into Band 7 A typical Yr9 level of performance is at the bottom of Band 8

37 Band 0 Band 9 Band 8 Band 7 Band 6 Band 5 Band 4 Band 3 Band 2 Band Victorian State Averages -200 (By Year Level and Dimension) Year Year Year Year 3 Reading Writing Spelling Gram. & Punct. Numeracy

38 Comparing National Scale Scores to Estimated VELS Equivalent scores The Victorian means for Year 3 and Year 9 Reading and Numeracy on the National scale are compared to the estimated VELS equivalent scores below: Dimension Year 3 Victoria Means National Scale VELS Equivalent Year 9 Victoria Means National Scale VELS Equivalent Reading Numeracy Compared to our expected curriculum outcomes for Year 3 students (2.75), the State Reading mean is about ½ term ahead of where we expect a typical Year 3 student to be. However, the State Numeracy mean is about 2½ terms below where we expect a typical Year 3 student to be. Compared to our expected curriculum outcomes for Year 9 students (5.75), the State Reading mean is just above where we expect a typical Year 9 student to be. However, the State Numeracy mean is about 2½ terms below where we expect a typical Year 9 student to be.

39 Cautionary Note Equal scores amongst different dimensions (on the National Scale) do not equate to equal levels of performance in terms of expected VELS levels. e.g. A National Yr9 Reading score of is equivalent to a VELS score of 5.22, but A higher National Yr9 Numeracy score of is equivalent to a lower VELS score of 4.84.

40 The MySchool website

41 Go to

42 The SPA website

43 Go to logon: VICTORIA password: demo