Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy

Christopher C. Stahl, Sarah A. Jung, Alexandra A. Rosser, Aaron S. Kraut, Benjamin H. Schnapp, Mary Westergaard, Azita G. Hamedani, Rebecca M. Minter, Jacob A. Greenberg

Research output: Contribution to journalArticlepeer-review

7 Scopus citations


Background: Entrustable Professional Activities (EPAs) contain narrative ‘entrustment roadmaps’ designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. Methods: All text comments associated with EPA microassessments at a single institution were combined. EPA—entrustment level pairs (e.g. Gallbladder Disease—Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. Results: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). Conclusions: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Original languageEnglish (US)
Pages (from-to)369-375
Number of pages7
JournalAmerican Journal of Surgery
Issue number2
StatePublished - Feb 2021
Externally publishedYes


  • Assessment
  • Entrustable professional activities
  • Feedback
  • Natural language processing
  • Surgery education

ASJC Scopus subject areas

  • Surgery


Dive into the research topics of 'Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy'. Together they form a unique fingerprint.

Cite this