Interobserver Variability and Diagnostic Performance with the Fifth Edition of the ACR BI-RADS Lexicon for Ultrasound; Validation from Multi-institutions

Abstract

Objective: To evaluate the interobserver agreement of radiologists from multiple institutions when breast masses obtained from multi-institutions were characterized using the revised Breast Imaging Reporting and Data System (BI-RADS) lexicon and to investigate whether agreement levels and diagnostic performances differed according to years of experience.

Materials and Methods: US images of 80 breast masses in 74 women were obtained from five institutions (51 benign and 29 malignant masses). Five readers (two from the same institution and 3 from different academic institutions; less than 10 years of experience in two readers and more than 10 years in the remaining 3) independently reviewed the two orthogonal images of each case. Each reader described and recorded the findings of the breast masses, and assigned a final category according to the revised BI-RADS lexicon. Interobserver agreement was determined using Cohen’s kappa and diagnostic performances were compared according to the years of experience in breast imaging.

Results: Overall agreement for the five radiologists varied from fair to substantial with the following kappa values: orientation (κ= 0.63), shape (κ= 0.54), final assessment (κ= 0.40), posterior features (κ= 0.30), margin (κ= 0.29) and echo pattern (κ= 0.28). There were no significant differences for US descriptors except for posterior features according to institution or experience level. The overall sensitivity, specificity, positive predictive value, negative predictive value and accuracy were 96.4% (106/110), 52.8% (153/290), 43.6% (106/243), 97.5% (153/157), 64.7% (259/400) without significant difference according to years of experience.

Conclusion: The overall interobserver agreement using multi-institutional cases was good, which validates the fifth edition of the BI-RADS lexicon, regardless of differences in years of experience. The overall diagnostic performances of reviewers from multi-institutions were excellent without significant difference according to years of experience.