Accuracy, Relevance, and Redundancy of ChatGPT-5.3 References in Open Fracture Management: A Methodological Analysis

This study aimed to evaluate the bibliographic accuracy of the references that ChatGPT-5.3 (OpenAI, San Francisco, CA) presented as PubMed-indexed sources for open fracture management and determine whether the verifiable references were relevant to the topics they were meant to support.

In a single session in March 2026, ChatGPT-5.3 (Plus subscription), with the web search function enabled, was asked to systematically describe the management of open fractures. The model was then prompted to provide three PubMed-indexed references for each topic it had generated. ChatGPT-5.3 structured open fracture management into 15 topic categories and produced a total of 45 references. Two independent reviewers assessed each reference for bibliographic accuracy and content relevance. Bibliographic accuracy was scored from 0 to 2, and content relevance from 0 to 1. Final scores were determined by consensus. Inter-rater agreement was assessed using Cohen's kappa coefficient.

Of the 45 references, 11 (24.4%) were fully bibliographically accurate. Nineteen references (42.2%) existed but contained errors in the PMID, year, journal, title, or other bibliographic details. Fifteen references (33.3%) could not be verified as valid PubMed-indexed articles. The overall bibliographic accuracy rate was 11 (45.6%), the content relevance rate was 30 (66.7%), and the overall performance score was 52.6%. All bibliographically verifiable references were content-relevant to their assigned open fracture management topics, with 30 (66.7%) references being content-relevant. The 45 references corresponded to only 36 unique sources; a single review article was reproduced under 8 (53.3%) of the 15 topic categories and accounted for 8 (72.7%) of the 11 fully accurate references. In a sensitivity analysis restricted to unique sources, the bibliographic accuracy rate decreased to 13 (37.5%). Inter-rater agreement was high, with Cohen's kappa values of 0.966 for bibliographic accuracy and 0.951 for content relevance.

Although ChatGPT-5.3 generated a clinically reasonable and systematic framework for open fracture management, the references provided to support this response showed substantial bibliographic reliability problems. Because all verifiable references were content-relevant, the model's principal limitation appeared to lie not in the selection of irrelevant sources, but in the presentation of seemingly appropriate sources with inaccurate or unverifiable bibliographic information. References generated by ChatGPT should not be used for academic or clinical purposes without independent verification.