(Debriefing Seminar) Test

Size: px

Start display at page:

Download "(Debriefing Seminar) Test"

Delphia Bishop
6 years ago
Views:

1 HKASME Science Assessment Test 2017 (Debriefing Seminar) DATE: 6 JULY 2017 HKASME Science Assessment Test Purpose: To develop and to implement a science assessment test for S2&3 students in Hong Kong To analyse the test results and to provide feedback to schools/the administration on the strengths and weaknesses of students in learning science 1

2 HKASME Science Assessment Test Duration: 1 hour Paper Structure: Section A: 24 MCQs Section B: 2 Short-response questions Coverage: Science knowledge in topics 1 to 11 of CDC Science Curriculum S1 to S Science Assessment Test Bronze Award based on a criteria-based model Introducing a Diamond Award (top 5%) Expert judgment for Gold, Silver and Bronze Award QP and students performance on individual questions available to participating schools More informative school reports 2

3 Participations in 2017 SAT Participants S1 S2 S3 Total Boys Girls Total No. of Participating Schools: 88 Science Process Skills Observing, measuring and recording data (SP1) Comparing and classifying (SP2) Planning and designing (SP3) Experimenting (SP4) Interpreting data / Inferring (SP5) Communicating (SP6) 3

4 Weaknesses of Participants as Revealed from Their Performance Section A Discrimination index (DI) describes the ability of an item to distinguish between high and low scorers ranges between and expected that the high-performing students select the correct answer for each item more often than the low-performing students DI = (H-L) / ½ N 4

5 items with p-value between and DI > 0.24 appropriate Understanding and Interpreting Graphical Information Q.12 The graph below shows the variation of solubility of a salt X with temperature: 5

6 In an experiment, 40 g of X is added to 100 g of water kept at 60 o C and the mixture is stirred until there is no further change. Based on the above information, which of the following statements about the resulting mixture is correct? *A. It is an unsaturated solution. (42%) B. It is a saturated solution. (41%) C. It contains undissolved X. (13%) D. It has a mass of 100 g. (4%) Discrimination index = 0.56 Less than half of the participants were able to choose the key showing that they were weak in the interpretation of graphs. Q.19 An experiment using the set-up shown on the right can be used to study the pressure-volume relationship of a fixed mass of air at room temperature. Pressures (P) of different magnitudes are applied onto the plunger and the corresponding volumes of the air inside the syringe (V) are measured. Which of the following graphs correctly represents the experimental results obtained? 6

7 V V P P V V P P Discrimination index = 0.22 A majority of the participants knew that, under constant temperature, the volume of a fixed mass of gas decreases with its pressure. However, most of them did not know that the volumepressure relationship is non-linear. Understanding of Abstract Science Concepts Q.7 Toby throws a ball in air at an angle. The diagram below shows the path of the ball and four positions, P, Q, R and S, of the ball along the path: In which of these positions, does the arrow represent the direction of the force acting on the ball? (Assume that there is no air resistance.) 7

8 A. P (17%) B. Q (54%) C. R (12%) *D. S (17%) Discrimination index = 0.58 More than half of the participants wrongly thought that the direction of the force is tangential to the trajectory of the ball. They did not realize that when the ball had left Toby s hand, the pushing force (a contact force) would cease to act on it, and the only force acting on the ball was the gravitational attraction (a non-contact force). Q.15 Peter used the set-up shown below to study the image formed by a convex lens: He moved the lens to and fro until a sharp inverted image was formed on the screen. If Peter repeats the experiment using a thicker convex lens, which of the following combinations correctly describes the position of the lens and the appearance of the sharp image formed? 8

9 Position of the lens Appearance of the image *A. closer to the screen remains inverted (42%) B. closer to the screen becomes upright (13%) C. D. further away from the screen further away from the screen remains inverted (36%) becomes upright (9%) Discrimination index = 0.37 Nearly 80% of the participants knew that, under the experimental conditions, an inverted image would be formed. However, quite a number of them failed to realize that a thicker convex lens bends a beam of light to a greater extent and forms an image closer to the screen. Classification of Animals Q.14 Consider the animals shown below: Which of the following features of these animals can be used to classify them into two groups? (1) whether or not they live in water (2) whether or not they have antennae (3) whether or not they have wings 9

10 A. (1) only (12%) B. (2) only (5%) C. (1) and (3) only (56%) *D. (2) and (3) only (27%) Discrimination index = 0.35 A majority of the participants recognized that ducks and flies have wings while elephants and frogs do not. However, quite a number of them did not know that frogs are amphibians, and some failed to realize that flies do have antennae. Weaknesses of Participants as Revealed from Their Performance Section B 10

11 Q.1 - to test participants understanding of experimental procedures, communicative and graph-plotting skills, and interpretation of experimental data Q.2 to test participants understanding of scientific methods, the design of fair tests and the various factors that affect a fair comparison Question No. 1 2 Overall Mean 3.0 (30%) 4.6 (58%) 7.7 (43%) S.D (21.3%) 1.79 (22.4%) 3.28 (18.2%) B1 Question on Data-handling 11

12 (b) Word equation: chips + oxygen carbon dioxide + water + energy No. of response % s 10 All correct Replacing oxygen with air Replacing energy with heat ( 熱 / 熱能 ) Replacing oxygen with air and energy with heat Failure to give both (answers other than oxygen/air & energy/heat) Giving oxygen/air but failing to give energy/heat Failing to give oxygen/air but correctly giving energy/heat Using O 2 for oxygen & E for energy Nil response

13 90 92 (c)(i) Plot a graph to show the relationship between the mass of potato chip burnt and the temperature rise of the water in the boiling tube. Three aspects were looked for in marking the graph: correct labeling of axes (code: A1x, A9x); appropriate choice of scale (code: S1x, S9x); and joining the points with a straight line (code: G1x, G9x). 13

14 No. of responses % A10 Plotting temperature rise of water against mass of potato chips burnt, and both axes are correctly labeled with correct units Plotting temperature rise of water against mass of A11 potato chips burnt, and both axes are correctly labeled with some units (g/ o C) missing Plotting mass of potato chips burnt against A12 temperature rise of water, and both axes are correctly labeled with correct units Plotting mass of potato chips burnt against A13 temperature rise of water, and both axes are correctly labeled with some units (g/ o C) missing A91 Plotting temperature rise of water against mass of potato chips burnt, and both axes are correctly labeled with no/incorrect units (e.g. writing kg instead of g) Plotting mass of potato chips burnt against A92 temperature rise of water, and both axes are correctly labeled with no/incorrect units A93 Plotting final temperature of water against mass of potato chips burnt or vice versa A94 Graph with one of the axes unlabeled A98 Other mistakes related to axes (e.g. double axis) A99 Graph without showing labelled axes/ Nil response No. of responses % S10 Graph with appropriate scale in both x- and y-axes S91 S92 Graph with the scale in one of the axes not equally spaced Graph with a poor scale (e.g. the graph is highly compressed or falls out of the graph paper) in either axis S93 Scale NOT at origin S94 Scale using direct entry in either axis S98 Other mistakes related to scale S99 Graph without scale in both axes/ Nil response

15 (Code: A94, S92, G13) (Code: A93, S94, G91) (Code: A93, S93, G11) (Code: A94, S94, G92) 15

16 G10 G11 G12 G13 A straight line graph passing through all points and the origin A straight line graph passing through all points but not the origin A graph with line segments passing though all points and the origin A graph with line segments passing though all points but not the origin No. of responses % G90 No line shown on the data points G91 All or some of the points are incorrectly plotted G92 A graph showing a histogram/bar chart/pie chart instead of a straight line G98 Other mistakes related to graph (e.g. two lines/curves) G99 Nil response Mean score for plotting graph = 1.1 (out of 3) Some participants were able to estimate the temperature rise to be 41 o C, but failed to give the final temperature as 66 o C. No. of responses 20 No. The heat absorbed by the water in the boiling tube depends on its temperature rise It is not related to the initial temperature of the water. 10 No. It is not possible to set all initial temperature at 25.0 o C Yes. To ensure the experiment is a fair test, the initial temperature should be kept the same Yes. The initial temperature is the controlled variable Yes. Different initial temperatures will affect experimental results/ lead to inaccurate results Yes/No. Other irrelevant reasons (e.g. easier to compare) Nil response % 16

91 92 93 No. of responses 2 Two correct reasons given 89 3.7 1 One correct reason given 744 31.0 0 Incorrect reasons/ nil response 1569 65.

17 No. of responses 2 Two correct reasons given One correct reason given Incorrect reasons/ nil response % About 65% of the participants failed to give the following two reasons for the difference in energy content: heat loss to surrounding, and incomplete burning of potato chips. Some strange reasons such as energy given out by digestion/body consumption is different from the energy measured by the burning of potato chips, and other substances present in the package may increase the energy of potato chips were given. 17

18 1 correct reason given: Incorrect responses: Q2 18

19 Q.2(a)(i) Marking Scheme (Double-digit Coding) Mean mark = 0.5 (out of 1) Q.2(a) (i) Samples of Participants Work

20 Q.2(a)(ii) Marking Scheme and Samples (Similar marking scheme for dependent and controlled variables) Mean mark for Q.2(a)(ii) = 2.4 (out of 3) Q.2(a)(iii) Marking Scheme Mean mark = 0.3 (out of 2) 20

21 Q.2(a)(iii)Samples of Participants Work Q.2(b) Marking Scheme Mean mark = 1.4 (out of 2) 21

22 Q.2(b) Samples of Participants Work Analysis of 2017 SAT Results Mean SD Section A 14.7 (61%) 3.47 (14.4%) B1 3.0 (30%) 2.13 (21.3%) B2 4.6 (58%) 1.79 (22.4%) Section B 7.7 (43%) 3.28 (18.2%) 22

23 Comparison of Performance of S2 and S3 Students Section A Section B S2 S3 Mean S.D. Mean S.D (59%) 15.3 (64%) 3.33 (13.9%) 3.49 (14.5%) 7.1 (39%) 8.2 (46%) 3.04 (16.9%) 3.42 (19.0%) Comparison of the Performance of Boys and Girls in Section A 15 correct S2 S3 Whole group Boys 53% 64% 59% Girls 38% 45% 47% Median (correct to the unit digit) = 15 23

24 Performance of S2 Students in Section A Performance of S3 Students in Section A 24

25 Performance of Whole Group in Section A 2017 SAT Award Scheme HKASME set up an expert panel to determine the cut-off based on the performance of the participants. Paper score / marks Section A score / MCQs Section B score / marks Bronze Silver Gold Diamond >42.0 > 10 > 10 > 10 > 10 > 4 > 4 > 4 > 4 (Paper score = 1.5 Mark in Section A + Mark in Section B) 25

26 2017 SAT Awards Award type Diamond Gold Silver Bronze Appreciation Number 120 (5%) 351 (15%) 492 (20%) 873 (36%) 566 (24%) School-based Report Appendix I Analysis of Participants Responses to the Multiple-choice Questions (MCQs) Q. No Skills assessed Key WG % correct School Strength / weakness 1 Comparing & Classifying; Understanding B Laboratory safety; Understanding B W 3 Inferring C S 4 Planning & Designing; Fair tests C Inferring; Understanding D S 6 Applying; Mathematics skills D Understanding; Inferring D D 8 Measuring; Choice of apparatus C S 9 Understanding; Predicting B Inferring; Understanding A

27 Strength in a question (indicated by S ) means that the school got a higher correct % by 15% or more on that question compared to the whole group in 2016 SAT. Weakness in a question (indicated by W ) means that the school got a lower correct % by 15% or more on that question compared to the whole group in 2016 SAT. Questions that are poorly answered by the whole group ( 1/3 correct) are represented by D Question(s) showing weaker performance as compared to WG WG Q. No. Key Most Popular Distractor Key School Most Popular Distractor 9 B (64.1%) C (26.0%) B (46.7%) C (36.7%) 12 A (42.0%) B (40.5%) A (26.7%) B (40.0%) 20 D (54.0%) B (27.5%) D (33.3%) B (40.0%) 27

28 Analyzing student responses & identifying their problems Q. No. A B C D % 4.75 % % % Right Ans. D Correct % % Wrong % Diff. Index Disc. Index 72.44% 32.45% 0.35 Analysis of Participants Performance in Short-response Questions Appendix II Mean and standard deviation Question 1 (out of 10) Whole Group (WG) School Mean SD Mean SD 3.0 (30%) 2.13 (21.3%) 4.6 (46%) 1.82 (18.2%) Question 2 (out of 8) 4.6 (58%) 1.79 (22.4%) 5.4 (68%) 1.06 (13.3%) Section B (out of 18) 7.7 (43%) 3.28 (18.2%) 10.0 (56%) 2.08 (11.6%) 28

29 Double-digit coding system for Q1(c)(i) CODE Plot a graph to show the relationship between the mass of potato chip burnt and the temperature rise of the water in the boiling tube ITEM Correct Responses : S10 Graph with appropriate scale in both x- and y- axes Incorrect Responses/Nil Response S91 S92 S93 S94 S98 S99 Graph with the scale in one of the axes not equally spaced Graph with a poor scale (e.g. the graph is highly compressed or falls out of the graph paper) in either axis Scale NOT at origin Scale using direct entry in either axis Other mistakes related to scale Graph without scale in both axes/ Nil response Number of Response WG School (34.6%) (78.8%) (34.6%) (78.8%) (65.4%) (21.2%) (8.9%) (3.0%) 456 (19.0%) 128 (5.3%) 502 (20.9%) 94 (3.9%) 178 (7.4%) 3 (9.1%) 0 (0.0%) 3 (9.1%) 0 (0.0%) 0 (0.0%) School-based Report Appendix IV (1) Diamond, Gold, Silver and Bronze Awards in 2017 Award type Number Diamond 120 (5%) Gold 351 (15%) Silver 492 (20%) Bronze 873 (36%) Appreciation 566 (24%) (2) Overall Performance of School Mean score: (out of 54) Whole Group (WG) School 29.7 (55%) 34.7 (64%) Standard deviation: 7.61 (14.1%) 6.92 (12.8%) 29

30 Student Performance Report (Sample) Professional Learning Community on Science Assessment Collaborative culture in the development of quality assessment items/tasks among science teachers MCQs, short-response questions, double digit coding marking Talks, group discussion, moderation exercise and assignments TWO 3-hour Saturday meetings (19/8, 2/9) Meeting fee: $100 (Full refund if all 2 meetings are attended and assignments completed.) 30

A-LEVEL PHYSICS A. PHA6T Investigative and Practical Skills in A2 Physics (ISA) Report on the Examination June Version: 0.

A-LEVEL PHYSICS A. PHA6T Investigative and Practical Skills in A2 Physics (ISA) Report on the Examination June Version: 0. A-LEVEL PHYSICS A PHA6T Investigative and Practical Skills in A2 Physics (ISA) Report on the Examination 2450 June 2014 Version: 0.1 Further copies of this Report are available from aqa.org.uk Copyright