Can Residents Assess Other Providers' Infant Lumbar Puncture Skills? Validity Evidence for a Global Rating Scale and Subcomponent Skills Checklist

Objectives The aims of this study were to provide validity evidence for infant lumbar puncture (ILP) checklist and global rating scale (GRS) instruments when used by residents to assess simulated ILP performances and to compare these metrics to previously obtained attending rater data. Methods In 2009, the International Network for Simulation-based Pediatric Innovation, Research, and Education (INSPIRE) developed checklist and GRS scoring instruments, which were previously validated among attending raters when used to assess simulated ILP performances. Video recordings of 60 subjects performing an LP on an infant simulator were collected; 20 performed by subjects in 3 categories (beginner, intermediate, and expert). Six blinded pediatric residents independently scored each performance (3 via the GRS, 3 via the checklist). Four of the 5 domains of validity evidence were collected: content, response process, internal structure (reliability and discriminant validity), and relations to other variables. Results Evidence for content and response process validity is presented. When used by residents, the checklist performed similarly to what was found for attending raters demonstrating good internal consistency (Cronbach α = 0.77) and moderate interrater agreement (intraclass correlation coefficient = 0.47). Residents successfully discerned beginners (P < 0.01, effect size = 2.1) but failed to discriminate between expert and intermediate subjects (P = 0.68, effect size = 0.34). Residents, however, gave significantly higher GRS scores than attending raters across all subject groups (P < 0.001). Moderate correlation was found between GRS and total checklist scores (P = 0.49, P < 0.01). Conclusions This study provides validity evidence for the checklist instrument when used by pediatric residents to assess ILP performances. Compared with attending raters, residents appeared to over-score subjects on the GRS instrument.