Toward High School Biology: Evaluating student understanding

Transcription

1 Toward High School Biology: Evaluating student understanding Cari F. Herrmann Abell, Jean C. Flanagan, and Jo Ellen Roseman AAAS / Project 2061 BCCE University Park, PA July 29, 2012 This work is funded by the Department of Education (Grant R305A100714)

2 Symposium Agenda Paper Defining the targeted knowledge Speaker Jo Ellen Roseman, AAAS Curriculum development Elaine Howes, BSCS Teacher support and professional development Rebecca Kruse, BSCS Evaluating student understanding Cari Herrmann-Abell, AAAS Evaluating teacher knowledge 2 Jean Flanagan, AAAS

3 Implementation l Unit was implemented in the Spring of l 8 teachers from 6 schools in 4 states participated. l Unit was too long for the time allotted. l Curriculum developers made suggestions as to activities that could be cut without affecting the coherence of the content storyline. l Teachers made some additional cuts due to unforeseen time losses. l Some teachers did not reach the end of the unit. l 1 teacher did not reach the biology lessons. l 2 teachers did not reach the lessons on photosynthesis and plant growth. 3

4 Determining the unit s promise l Students were given a test before and after participating in the unit Pre- and s l 4 versions with linking items l Multiple choice items l First 3 items required students to write an explanation for why they chose their answer choice and why they didn t choose the other choices l Each item aligned to 1 or 2 ideas l Distractors probe for common student misconceptions l Each student received items l A total of 54 items were included 4

5 Participants l 6 schools in 4 states (1 West, 1 Northeast, & 2 Southeast) l 677 8th grade students participated l 601 students completed both pre and post tests Gender % Male 48% Female 52% Language 5 % Ethnicity % White 44% Black 20% Asian 15% Hispanic 13% English is primary language 91% Other 3% English is not primary language 9% 2 or more 5%

6 Chemistry Ideas: l Chemical Reactions: Many substances react chemically with other substances to form new substances with different characteristic properties. l Chemical Reactions: When substances interact to form new substances, the atoms that make up the molecules of the starting substances rearrange to form the molecules of the ending substances. l Conservation: If all of the starting substances and ending substances are accounted for, the mass of the ending substances is the same as the mass of the starting substances. l Conservation: Mass is conserved during chemical reactions because the total number of each type of atom making up the molecules of the reactants is the same as the total number of each type of atom making up the molecules of the products. 6

7 Biology Ideas: l Animal Growth: Animals use polymer molecules from food to make a variety of other polymer molecules that become part of their body structures. l Photosynthesis: Plants make the glucose molecules they need for growth from carbon dioxide molecules and water molecules during a chemical reaction that also produces oxygen. l Plant Growth: Plants use glucose molecules to make a variety of larger polymer molecules that become part of their body structures. 7

8 Rasch Modeling l The dichotomous Rasch model was used to estimate the students abilities and the item difficulties. l The probability that a student will respond to an item correctly is determined by the difference in the student s ability and the difficulty of the item. l Student and item measures are expressed on the same interval scale and are mutually independent, which is not the case for percent correct statistics. 8

9 Measuring change over time using Rasch modeling l Stacking data: study the effect of the unit on students understanding of chemistry and biology. l Racking data: study the effect of the unit on the items difficulty level. 9 (Wright, 2003)

10 Stacked method: Changes in student understanding 10

11 Wright Map from Stacked Analysis 4 Items # ### # # #### ### #### #### ### ### ### ### # #### #### ### ## ## # # ## # TT S S M M S S T T XXXX XX X T S Abilities XXXXX XXX XXXXXX X XXXXXXXXX XXXXXX XXXXXXXXXXXX XXXXXXXXXXX XXXXXXXXX XXXXXXXXX XXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXX XXXXXXXXXXXXXXXX XXXXX XXXXXX XX XXX XXXXXXX XX XX XXXXX XXXXXXXX XXXXXXX XXXXXXXX Abilities XXXXXXXXXXXXXX XXXXXXXXXXX XXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXX XXXXXXXXXXXXXXXX XX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXX M XXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX S XXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXX X XXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXX X XXXX X T X XX abilities are higher than pretest ability Item difficulty matches students ability range during pretest No items that match students on upper end of ability range

12 Stacked method: Changes in student understanding Total (N=601) aeffect Mean SD Gain 1.19 Effect sizea 1.25 size calculated by dividing the difference of the means by the pretest standard deviation l Overall, posttest abilities were significantly higher than pretest abilities (t = , p<.001). l No significant differences observed between gains of males and females (t = -0.29, p>.05). l No significant differences observed between gains of students grouped by primary language (t = 0.33, p>.05). l No significant differences observed between gains of students grouped by ethnicity (t = 1.89, p>.05). 12

13 Racked method: Changes in item difficulties 13

14 Wright Map from Racked Analysis 3..#..# #. ## #.# ###.#.###.##### #####.##### ########.######.######### #########.#########.#######.##########.###########.#########.############.###########.#######.######## #########.###### ######.######.####.###.## #.#. # Students TT SS M M S S T T difficulties are lower than pretest difficulties Item Difficulties SC754 SB042 SB033 SB371 SB072 SB111 SB062 SB261 SB461 SB471 SB221 SB272 SB142 SB391 SB312 SB362 SC354 SB152 SB322 SC765 SC505 SB342 SB012 SB022 SB051 SB081 SB181 SB191 SC455 SB172 SB231 SB381 SC704 SC665 SC594 SB082 SB431 SC697 SB212 SB451 SB092 SB481 SC944 SC1013 SC1022 SB411 SB491 AM245 SB441 AM235 AM592 SB421 SC435 SC673 T S M S T SB342 SB221 SC754 SB272 SB042 SB081 SB152 SB033 SB062 SB012 SB371 SB312 SB051 SC594 SB481 SB231 SB092 SB142 SC455 SB451 AM245 AM235 SB172 SB322 SB072 SC673 SB381 SB261 SB411 SC435 SB022 SB391 SB461 SC765 SB082 SB362 SB491 SC1013 SB111 SB441 SB431 SC1022 SC665 SC704 SC944 SC697 SB181 AM592 SB421 SB191 SB471 SC505 SC354 SB212

15 Summary of item difficulties by idea Total (N=54) Mean SD Difference l For most items, the difficulty decreased from pre to post, suggesting that the unit was successful in teaching students the targeted ideas. l Items with the largest decrease in difficulty were targeting Animal growth and Conservation (atomic level). l Items with little to no decrease in difficulty were targeting Chemical reactions (substance level). l This idea was the focus of an early lesson but the unit quickly switches over to atomic level idea for the remainder of the lessons 15

16 Previous work on assessing students understanding of chemistry revealed common misconceptions Misconceptions Grade 6-8 Grade The atoms of the reactants are transformed into other atoms during a chemical reaction (Andersson, 1986). 45% 35% 2. When mold grows in a closed system, the mass of the system must have increased. 56% 50% 3. New atoms are created during chemical reactions. 32% 22% 4. Atoms are destroyed during chemical reactions. 24% 20% 5. The total number of atoms and, therefore, the total mass remain the same because some atoms are destroyed as other new ones are created. 17% 12% 16

17 Previous work on assessing students understanding of biology revealed common misconceptions Misconceptions Grade 6-8 Grade Food is either used for energy or eliminated as waste, ignoring the idea that some of the food is used to build/repair body parts (Smith & Anderson, 1986). 60% 69% 2. Most of a plant's mass comes from minerals that the plant takes in from the soil (Vaz et al, 1997). 54% 17% l The activities in the unit provided students with evidence that contradicted these misconceptions and supported the correct science ideas. l Distractors on tests probe for student misconceptions, so we can study the students response patterns to determine the unit s effect on students ideas and misconceptions. 17

18 Transmutation vs. rearrangement or atoms during chemical reactions National Misconception HS 35% HS 48% The atoms of the reactants are transformed into other atoms during a chemical reaction (Andersson, 1986). Correct Idea The atoms of the reactants rearrange to form new molecules during a chemical reaction. χ2 Sig. 42% 15% 112 <.001 χ2 Sig. 40% 60% 232 <.001 l 4 items included distractors aligned to this misconception. 18

19 When mold grows in a closed system, does the weight increase or stay the same? National HS Misconception 50% Weight increased National HS Correct Idea 27% Weighed stayed the same χ2 Sig. 69% 50% 21.7 <.001 χ2 Sig. 17% 41% 42.9 <.001 l Additionally, on the posttest, some students volunteered atomic level explanations for why the weight would stay the same. 19 l 14% of the students mentioned atoms in their explanations on the posttest, while only 0.3% mentioned atoms on the pretest.

20 Sample substance level explanations 1. The mold weighs more as it grows so more mold means more weight. 2. The mold grows on the bread, adding a little more weight. 20 The bag is closed and nothing can get in or out, the mass does not change. When the mold is made, it must use something else in the bag that loses the same mass that the mold gains. Parts of the bread and moisture in the bag were used to create the mold so there is still the same amount of mass in the bag because it's sealed so no gases can get in or out.

21 Sample atomic level explanations 3. Since the mold grew it added weight onto the piece of bread so the bag and the content must of gotten heavier after two weeks because the mold added weight. 4. Because of the mold's growth, its weight increased. 21 Part of this question stated that the bag was sealed so nothing could get in or out which means this bag became a closed system trapping all of the molecules and atoms in the bag. Therefore, if the mold grew on the bread the mold would only be made up of the atoms and molecules that were inside the bag while the bag was being sealed tightly. So the mass would stay the same since no molecules were added or removed from the bag. In a sealed container, nothing can get in or out. So the atoms of the ending substance stays the same as the starting substance since no atom can be created nor destroyed. When there are same amounts of atoms of specific type, the mass is the same, and when the mass is the same, the weight is the same.

22 Creation and destruction of atoms vs. # atoms remaining the same National HS Answer Choice χ2 Sig. 22% Mass because atoms are created 36% 29% 3.53 n.s. 20% Mass because atoms are destroyed 20% 5% 26.7 < % Mass same because atoms are created & destroyed 7% 4% 1.50 n.s. 46% Mass same because # of atoms stays the same 31% 57% 34.8 <.001 l Significant decrease in destruction misconception. l Significant increase in correct answer. l No significant change in the creation misconception. 22

23 Conservation of mass during growth of living things Misconception about destruction of matter # items 1 14% 4% # items When organisms grow, matter is created because new atoms are created. 3 30% 19% Living organisms grow by creating new matter through cell division, no additional material/ atoms are needed for growth. 5 28% 10% Plants use up glucose when they grow, destroying matter in the process. Misconceptions about creation of matter 23 Photos from video by JAMagicFilms Oak sapling. Photo (CC) by seth_k on Flickr.

24 Food becoming all waste vs. becoming part of the body Misconception All of the food an animal eats goes through the digestive system and ends up as waste. Correct Idea Some of the food goes through chemical reactions and becomes part of the animal s body. χ2 Sig. 42% 13% 160 <.001 χ2 Sig. 50% 82% 175 <.001 l 3 items included distractors aligned to this misconception. l After participating in the unit, significantly fewer students chose the misconception, and significantly more students chose the correct answer. 24

25 Sample written explanations 5. The rabbit is a consumer, so it eats, digests the food, then it puts it out as waste. The rabbit uses [the grass] as energy then it creates a waste product. 6. Grass is food, and food is digested and turned into waste. When we eat vegetables not unlike grass, it doesn't just magically become part of our leg or something. 25 When the rabbit eats the plant it goes to the stomach and gets separated through a chemical reaction in the stomach. It separates it from protein to waste. Then the protein polymers get broken down further in monomers which can then be rearranged to be used in the body. In some THSB lessons, we learned that food that organisms consume goes through chemical reactions in their stomachs and is turned into proteins that help the organism live, grow, and develop. Photo (CC) by Larry D. Moore

26 Most of a plants mass comes from Minerals vs. CO2 Most of a plant s mass comes from Misconception χ2 Sig. Minerals that the plant takes in from the 49% 29% 18.2 <.001 soil. Sig. Correct Idea χ2 Carbon dioxide that the plant takes in 30% 53% 31.0 <.001 from the air. l Significantly fewer students chose minerals in the soil l Significantly more students chose the correct answer. 26 Photo (CC) by Lucas Maystre

27 Conclusions l Rasch modeling showed that l students understanding of most of the targeted chemistry and biology ideas improved significantly and l most of the items got easier from pretest to posttest. l Distractor analysis revealed a decrease from pretest to posttest in the popularity of several misconceptions. l An analysis of students written explanations showed an increase in understanding the targeted ideas. 27