Bethesda Interobserver Reproducibility Study-2 (BIRST-2): Bethesda System 2014

In concert with the 2014 update to the Bethesda System for Reporting Cervical Cytology, a Web-based image interobserver study was performed to evaluate concordance with the "expert panel" interpretation, as was done during the Bethesda 2001 update. The aim was to identify cytomorphologic features and Bethesda reporting categories that represent sources of poor interobserver agreement and see how the trends compared to the first Bethesda Interobserver Reproducibility Study (BIRST).

Participants were recruited online through national and international cytopathology professional societies. Study participants evaluated 84 previously unpublished web images chosen from the third Bethesda Atlas image set, prior to the release of the atlas. These images spanned all reporting categories and included typical and borderline cytomorphology. Demographic information was collected on level of training, practice patterns, and experience of the participants. Participation was restricted to those correctly answering 2 basic cytopathology questions, ensuring minimal knowledge of gynecologic cytopathology.

A total of 1290 unique individuals attempted access to this Web-based study and 833 correctly answered the two qualifying questions. Of these, 518 respondents completed the survey. Participant origin included: 59% United States, 41% international; 48% cytotechnologists, 41% pathologists, 5% fellows, and 6% other. Practice types were: 39% academic institutions, 29% private hospitals, and 16% commercial laboratories. Overall, the mean participant agreement with the exact Bethesda panel interpretation was 62.8%. The best agreement was found for negative for intraepithelial lesion or malignancy (NILM; 74%) and low-grade squamous intraepithelial lesion (LSIL; 86%) categories. Squamous cell carcinoma (SCC) (63%), high-grade squamous intraepithelial lesion (HSIL; 60%), atypical squamous cells of undetermined significance (ASC-US; 62%) and atypical squamous cells, cannot exclude high-grade squamous intraepithelial lesion (ASC-H; 60%) showed slightly lower concordance with the panel interpretations. Cervical glandular lesions were more problematic (33%). Anal samples performed similarly to their gynecologic counterparts. There was similar diagnostic agreement across participant certifications and practice type (academic versus non-academic). Performance was higher for United States and other North America-based participants (P = 0.0104). This significance may be attributed to a language bias, as the survey was only offered in English.

Similar to the BIRST-1 study conducted in 2001, the most important factor for diagnostic agreement by cytotechnologists, pathologists, and trainees was the a priori difficulty of an image rather than participant training, certification, or experience. Participants showed better general diagnostic agreement with the expert panel interpretations of the material in BIRST-2 than in BIRST-1. Agreement was highest for Bethesda categories of NILM, LSIL, HSIL, and SCC. Concordance for even the borderline ASC-US and ASC-H categories exhibited remarkable improvement in the BIRST-2.