TY - JOUR
T1 - Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities
AU - Schumacher, Daniel J.
AU - West, Daniel C.
AU - Schwartz, Alan
AU - Li, Su Ting
AU - Millstein, Leah
AU - Griego, Elena C.
AU - Turner, Teri
AU - Herman, Bruce E.
AU - Englander, Robert
AU - Hemond, Joni
AU - Hudson, Valera
AU - Newhall, Lauren
AU - McNeal Trice, Kenya
AU - Baughn, Julie
AU - Giudice, Erin
AU - Famiglietti, Hannah
AU - Tolentino, Jonathan
AU - Gifford, Kimberly
AU - Carraccio, Carol
N1 - Publisher Copyright:
© 2020 American Institute of Physics Inc.. All rights reserved.
PY - 2020/1/15
Y1 - 2020/1/15
N2 - Importance: Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective: To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants: Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions: Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures: Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results: Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance: This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence..
AB - Importance: Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective: To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants: Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions: Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures: Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results: Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance: This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence..
UR - http://www.scopus.com/inward/record.url?scp=85077884547&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85077884547&partnerID=8YFLogxK
U2 - 10.1001/jamanetworkopen.2019.19316
DO - 10.1001/jamanetworkopen.2019.19316
M3 - Article
C2 - 31940042
AN - SCOPUS:85077884547
SN - 2574-3805
VL - 3
JO - JAMA network open
JF - JAMA network open
IS - 1
M1 - 19316
ER -