TY - JOUR
T1 - Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests
AU - Juster, Fern R.
AU - Baum, Robin Camhi
AU - Zou, Christopher
AU - Risucci, Don
AU - Ly, Anhphan
AU - Reiter, Harold
AU - Miller, D. Douglas
AU - Dore, Kelly L.
N1 - Funding Information:
The authors gratefully acknowledge the gracious support of the Stemmler Fund of the National Board of Medical Examiners (Grant 1011 - 039, entitled "Ensuring diversity and test security: An examination of the reliability, validity, feasibility and acceptability of the Computer-based Assessment for Sampling Personal Characteristics (CASPer) in diverse populations of medical school applicants").
Publisher Copyright:
Copyright © 2019 The Author(s).
PY - 2019
Y1 - 2019
N2 - Purpose To examine the magnitudes of score differences across different demographic groups for three academic (grade point average [GPA], old Medical College Admission Test [MCAT], and MCAT 2015) and one nonacademic (situational judgment test [SJT]) screening measures and one nonacademic (multiple miniinterview [MMI]) interview measure (analysis 1), and the demographic implications of including an SJT in the screening stage for the pool of applicants who are invited to interview (analysis 2). Method The authors ran the analyses using data from New York Medical College School of Medicine applicants from the 2015- 2016 admissions cycle. For analysis 1, effect sizes (Cohen d) were calculated for GPA, old MCAT, MCAT 2015, CASPer (an online SJT), and MMI. Comparisons were made across gender, race, ethnicity (African American, Hispanic/Latino), and socioeconomic status (SES). For analysis 2, a series of simulations were conducted to estimate the number of underrepresented in medicine (UIM) applicants who would have been invited to interview with different weightings of GPA, MCAT, and CASPer scores. Results A total of 9, 096 applicants were included in analysis 1. Group differences were significantly smaller or reversed for CASPer and MMI compared with the academic assessments (MCAT, GPA) across nearly all demographic variables/indicators. The simulations suggested that a higher weighting of CASPer may help increase gender, racial, and ethnic diversity in the interview pool; results for low-SES applicants were mixed. Conclusions The inclusion of an SJT in the admissions process has the potential to widen access to medical education for a number of UIM groups.
AB - Purpose To examine the magnitudes of score differences across different demographic groups for three academic (grade point average [GPA], old Medical College Admission Test [MCAT], and MCAT 2015) and one nonacademic (situational judgment test [SJT]) screening measures and one nonacademic (multiple miniinterview [MMI]) interview measure (analysis 1), and the demographic implications of including an SJT in the screening stage for the pool of applicants who are invited to interview (analysis 2). Method The authors ran the analyses using data from New York Medical College School of Medicine applicants from the 2015- 2016 admissions cycle. For analysis 1, effect sizes (Cohen d) were calculated for GPA, old MCAT, MCAT 2015, CASPer (an online SJT), and MMI. Comparisons were made across gender, race, ethnicity (African American, Hispanic/Latino), and socioeconomic status (SES). For analysis 2, a series of simulations were conducted to estimate the number of underrepresented in medicine (UIM) applicants who would have been invited to interview with different weightings of GPA, MCAT, and CASPer scores. Results A total of 9, 096 applicants were included in analysis 1. Group differences were significantly smaller or reversed for CASPer and MMI compared with the academic assessments (MCAT, GPA) across nearly all demographic variables/indicators. The simulations suggested that a higher weighting of CASPer may help increase gender, racial, and ethnic diversity in the interview pool; results for low-SES applicants were mixed. Conclusions The inclusion of an SJT in the admissions process has the potential to widen access to medical education for a number of UIM groups.
UR - http://www.scopus.com/inward/record.url?scp=85070539290&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85070539290&partnerID=8YFLogxK
U2 - 10.1097/ACM.0000000000002769
DO - 10.1097/ACM.0000000000002769
M3 - Article
C2 - 31033603
AN - SCOPUS:85070539290
SN - 1040-2446
VL - 94
SP - 1197
EP - 1203
JO - Academic Medicine
JF - Academic Medicine
IS - 8
ER -