On ASSESSMENT EVIDENCE in CBME: Assessment, Strategies and Pitfalls

Transcription

1 On ASSESSMENT EVIDENCE in CBME: Assessment, Strategies and Pitfalls Inf e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE LATENT ATTRIBUTES Inf e r e n t i a l L e a p Walter Tavares, PhD Scientist Assistant Professor The Wilson Centre & Post MD Education [PGME & CPD] Department of Medicine University of Toronto Sim Educator Royal College of Physicians and Surgeons walter.tavares@utoronto.ca

2 Disclosures: None

3 C O M P E T E N C E Complex Abstract Inferred Kane 1999

4 V A L I D I T Y How Good Those Inferences Are Kane 2003

5 PROGRAMMATIC ASSESSMENT

6 I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p

7 CBME Moving from OUTCOMES to Assessment Plans

8 CBME ACTIVITIES F I X E D I N T E N D E D E N A C T E D CURRICULUM MAP PRACTICE OUTCOME MAP EXPERIENCE MAP Trainee Cohort Programs ASSESSMENT MAP Represents translational processes and direction of translation Represents feedback opportunities

9 A S S U M P T I O N # 1 AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT

10 A S S U M P T I O N # 2 Outcome Frameworks CAN ALSO BE USED as Assessment Frameworks

11 A S S U M P T I O N # 3 Faculty ACCURATELY TRANSLATE Observations to Rating Tools

12 A S S U M P T I O N # 4 The COLLECTION OF EPAs = Competence

13 A S S U M P T I O N # 1 AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT NOT ALWAYS OPPORTUNITIES MAY BE ABSENT SURROGATES ARE USED

14 AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT Surgery: important cases < 4 / year; some not at all Internal Medicine: 17% of learners encounter most common of diagnoses Emergency Medicine, Peads, Medical Students etc.

15 AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT Others / ACGME narrowing of: critical illnesses case mix / amount opportunity for procedural skills Access to competencies / feedback Insufficient, some as lows as 0

16 I N T E N D E D E N A C T E D ASSESSMENT PLANS DEGREE OF VALIDITY THREAT D ASSESSMENT ACTIVITIES A C B CASE TYPE CASE COMPLEXITY CONTEXT CONSTRUCT

17 I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p

18 A S S U M P T I O N # 2 Outcome Frameworks CAN ALSO BE USED as Assessment Frameworks NOT EXACTLY DIFFERENT PURPOSE INTENDED VS. ENACTED

19 No evidence that current measurement tools can assess the competencies independently. Use competencies to guide specific evaluation efforts.

20 Collaborator, Manager, Health Advocate, Scholar < 25%...largest sources of variance was rater bias factor analysis revealed a single factor.

21 I N T E N D E D E N A C T E D ASSESSMENT PLANS DEGREE OF VALIDITY THREAT D ASSESSMENT ACTIVITIES A C B

22 I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p

23 A S S U M P T I O N # 3 Faculty ACCURATELY TRANSLATE Observations to Rating Tools

24 The Act of RATER Based Assessments Fallible and subject to difficulties: observational inaccuracy common rating biases difficulty discriminating raters as significant source of error Herbers et al. 1989; Thompson et al. 1990; Kalet et al. 1992; Noel et al. 1992; Haber and Avins 1994; LaMantia et al. 1999; Williams et al. 2003; Lurie et al. 2009; Downing 2005; Margolis et al. 2006; Cook et al. 2010

25 Expectations E.g., - Assess what we want them to; hold shared views; objective measurement devices Behaviors E.g., - Impact on staff and disposition; personal qualities; professional and relational features Ginsburg 2010; Bogo 2009

26 A TOUR Through Rater Cognition Research Rater are not objective measurement devices Cognitive activities can be compartmentalized Raters behave (cognitively) in ways we didn t expect Raters are inherently and unavoidably idiosyncratic

27 the most remarkable observation might be in how irrational we have been to date with work based assessment instruments and processes.

28 I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p

29 A S S U M P T I O N # 4 The COLLECTION OF EPAs = Competence Descriptions of Work Physicianship Same as Competence

30 CBME Moving from OUTCOMES to Assessment Plans

31 W H A T T O D O? How Good Those Inferences Are

32 A S S E S S M E N T 1. Have a flexible assessment plan that is responsive to experiences ; 2. Faculty are the measurement devices; 3. Translate outcome frameworks by starting with faculty; 4. Faculty as independent contributors; 5. Faculty provide idiosyncratic and chunks of information; 6. Explore validity of: EPAs = Competence 7. Sample wisely.

33 CBME ACTIVITIES F I X E D I N T E N D E D E N A C T E D CURRICULUM MAP PRACTICE OUTCOME MAP EXPERIENCE MAP Trainee Cohort Programs ASSESSMENT MAP Represents translational processes and direction of translation Represents feedback opportunities

34 I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p

35 THANK YOU! Inf e r e n t i a l L e a p Walter Tavares, PhD walter.tavares@utoronto.ca ASSESSMENT CONTEXT PROGRAMMATIC ASSESSMENT FUTURE CLINICAL PERFORMANCE LATENT ATTRIBUTES Follow me on Inf e r e n t i a l L e a p

36

37 F A C U L T Y How Good Those Inferences Are