On ASSESSMENT EVIDENCE in CBME: Assessment, Strategies and Pitfalls Inf e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE LATENT ATTRIBUTES Inf e r e n t i a l L e a p Walter Tavares, PhD Scientist Assistant Professor The Wilson Centre & Post MD Education [PGME & CPD] Department of Medicine University of Toronto Sim Educator Royal College of Physicians and Surgeons walter.tavares@utoronto.ca
Disclosures: None
C O M P E T E N C E Complex Abstract Inferred Kane 1999
V A L I D I T Y How Good Those Inferences Are Kane 2003
PROGRAMMATIC ASSESSMENT
I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p
CBME Moving from OUTCOMES to Assessment Plans
CBME ACTIVITIES F I X E D I N T E N D E D E N A C T E D CURRICULUM MAP PRACTICE OUTCOME MAP EXPERIENCE MAP Trainee Cohort Programs ASSESSMENT MAP Represents translational processes and direction of translation Represents feedback opportunities
A S S U M P T I O N # 1 AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT
A S S U M P T I O N # 2 Outcome Frameworks CAN ALSO BE USED as Assessment Frameworks
A S S U M P T I O N # 3 Faculty ACCURATELY TRANSLATE Observations to Rating Tools
A S S U M P T I O N # 4 The COLLECTION OF EPAs = Competence
A S S U M P T I O N # 1 AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT NOT ALWAYS OPPORTUNITIES MAY BE ABSENT SURROGATES ARE USED
AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT Surgery: important cases < 4 / year; some not at all Internal Medicine: 17% of learners encounter most common of diagnoses Emergency Medicine, Peads, Medical Students etc.
AUTHENTIC WORKPLACE SETTINGS ARE FULLSOME FOR ASSESSMENT Others / ACGME narrowing of: critical illnesses case mix / amount opportunity for procedural skills Access to competencies / feedback Insufficient, some as lows as 0
I N T E N D E D E N A C T E D ASSESSMENT PLANS DEGREE OF VALIDITY THREAT D ASSESSMENT ACTIVITIES A C B CASE TYPE CASE COMPLEXITY CONTEXT CONSTRUCT
I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p
A S S U M P T I O N # 2 Outcome Frameworks CAN ALSO BE USED as Assessment Frameworks NOT EXACTLY DIFFERENT PURPOSE INTENDED VS. ENACTED
No evidence that current measurement tools can assess the competencies independently. Use competencies to guide specific evaluation efforts.
Collaborator, Manager, Health Advocate, Scholar < 25%...largest sources of variance was rater bias factor analysis revealed a single factor.
I N T E N D E D E N A C T E D ASSESSMENT PLANS DEGREE OF VALIDITY THREAT D ASSESSMENT ACTIVITIES A C B
I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p
A S S U M P T I O N # 3 Faculty ACCURATELY TRANSLATE Observations to Rating Tools
The Act of RATER Based Assessments Fallible and subject to difficulties: observational inaccuracy common rating biases difficulty discriminating raters as significant source of error Herbers et al. 1989; Thompson et al. 1990; Kalet et al. 1992; Noel et al. 1992; Haber and Avins 1994; LaMantia et al. 1999; Williams et al. 2003; Lurie et al. 2009; Downing 2005; Margolis et al. 2006; Cook et al. 2010
Expectations E.g., - Assess what we want them to; hold shared views; objective measurement devices Behaviors E.g., - Impact on staff and disposition; personal qualities; professional and relational features Ginsburg 2010; Bogo 2009
A TOUR Through Rater Cognition Research Rater are not objective measurement devices Cognitive activities can be compartmentalized Raters behave (cognitively) in ways we didn t expect Raters are inherently and unavoidably idiosyncratic
the most remarkable observation might be in how irrational we have been to date with work based assessment instruments and processes.
I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p
A S S U M P T I O N # 4 The COLLECTION OF EPAs = Competence Descriptions of Work Physicianship Same as Competence
CBME Moving from OUTCOMES to Assessment Plans
W H A T T O D O? How Good Those Inferences Are
A S S E S S M E N T 1. Have a flexible assessment plan that is responsive to experiences ; 2. Faculty are the measurement devices; 3. Translate outcome frameworks by starting with faculty; 4. Faculty as independent contributors; 5. Faculty provide idiosyncratic and chunks of information; 6. Explore validity of: EPAs = Competence 7. Sample wisely.
CBME ACTIVITIES F I X E D I N T E N D E D E N A C T E D CURRICULUM MAP PRACTICE OUTCOME MAP EXPERIENCE MAP Trainee Cohort Programs ASSESSMENT MAP Represents translational processes and direction of translation Represents feedback opportunities
I n f e r e n t i a l L e a p ASSESSMENT CONTEXT FUTURE CLINICAL PERFORMANCE PROGRAMMATIC ASSESSMENT LATENT ATTRIBUTES I n f e r e n t i a l L e a p
THANK YOU! Inf e r e n t i a l L e a p Walter Tavares, PhD walter.tavares@utoronto.ca ASSESSMENT CONTEXT PROGRAMMATIC ASSESSMENT FUTURE CLINICAL PERFORMANCE LATENT ATTRIBUTES Follow me on twitter @WalterTava Inf e r e n t i a l L e a p
F A C U L T Y How Good Those Inferences Are