Interobserver agreement among endosonographers for endoscopic ultrasound (EUS) evaluation of submucosal masses

INTRODUCTION: EUS has been reported to be an accurate means of evaluating and diagnosing submucosal lesions of the GI tract. AIMS OF STUDY: To determine endosonographer interobserver reliability for classifying submucosal masses by EUS. METHODS: Twenty patients with submucosal mass lesions diagnosed at upper endoscopy underwent EUS evaluation. Surgical findings or FNA cytology were available for 16 patients. In 4 patients with obvious cystic/vascular structures (ie, varices) no surgical correlation was necessary. A blinded observer developed a study videotape including critical endoscopic and EUS data for each lesion. The videotape was then distributed to 10 endosonographers with at least one year of experience who then independently reviewed the tape and recorded their diagnosis based upon EUS features. These endosonographers used previously agreed upon standardized EUS diagnostic criteria for each category of lesion (extrinsic compression, leiomyoma, vascular, lipoma, cyst, other submucosal lesion ie; carcinoid) Each endosonographer independently categorized the lesions. A multiple observer kappa (k) statistic for agreement was calculated for each lesion category and an overall k calculated. Kappa accounts for agreement due to chance alone and standardized ranges are available. Log transformation was performed for # cases and kappa and regression analysis used to examine the effect of experience on agreement. RESULTS: We found agreement was excellent for extrinsic compressions, vascular structures and lipomas (k=0.94), good for leiomyoma other submucosal lesions (k=0.67) and fair for cystic lesions. Overall agreement among observers was good (k=0.63) and highly statistically significant (z=21).The lowest kappa scores occurred with the most inexperienced endosonographers and the highest among the most experienced. The association between kappa and experience did not reach statistical significance. CONCLUSION: Interobserver agreement is good for characterizing submucosal masses by EUS. However, it appears to be better for some lesion types than others. Experience may play a role in the successful use of this modality to evaluate submucosal lesions.