Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Inter-rater agreement in physician-coded problem lists.

Adam S Rothschild1, Harold P Lehmann, George Hripcsak

  • 1Department of Biomedical Informatics, Columbia University, New York, NY, USA.

AMIA ... Annual Symposium Proceedings. AMIA Symposium
|June 17, 2006
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

An examination of the availability and characteristics of social needs data in the electronic health records: a path to social data harmonization and standardization at Johns Hopkins medicine.

JAMIA open·2026
Same author

Neuroretinal Layer Thinning on OCT Imaging and Hemoglobin A1c in Youth With Type 1 Diabetes.

JAMA ophthalmology·2026
Same author

A comparison of Fast Healthcare Interoperability Resources and Observational Medical Mutcomes Partnership electronic health record data within the All of Us Research Program.

Journal of the American Medical Informatics Association : JAMIA·2026
Same author

Heterogeneity of Treatment Effects Across Nine Glucose-Lowering Drug Classes in Type 2 Diabetes: Extension of the LEGEND-T2DM Network Study.

Diabetes, obesity & metabolism·2026
Same author

Semaglutide and Neovascular Age-Related Macular Degeneration Among Adults with Type 2 Diabetes: An OHDSI Network Study.

Ophthalmology·2026
Same author

Comparative Cardiovascular Effectiveness of Glucagon-Like Peptide 1 Receptor Agonists and Sodium-Glucose Cotransporter 2 Inhibitors in Diabetes Mellitus.

Journal of the American College of Cardiology·2026
Same journal

Sensitivity Analyses of a Scoring System for a Contraception Decision Aid.

AMIA ... Annual Symposium proceedings. AMIA Symposium·2026
Same journal

Improving electronic health record processing of large language models via retrieval-augmented generation: A case study on dietary supplements.

AMIA ... Annual Symposium proceedings. AMIA Symposium·2026
Same journal

Developing a User-Centered Mobile Application Prototype: Bridging Lower-Limb Fracture Care from Skilled Nursing Facility and Back to the Community.

AMIA ... Annual Symposium proceedings. AMIA Symposium·2026
Same journal

KERAP: A Knowledge-Enhanced Reasoning Approach for Accurate Zero-shot Diagnosis Prediction Using Multi-agent LLMs.

AMIA ... Annual Symposium proceedings. AMIA Symposium·2026
Same journal

Automating Adjudication of Cardiovascular Events Using Large Language Models.

AMIA ... Annual Symposium proceedings. AMIA Symposium·2026
Same journal

Predictive Factors and State-Level Barriers to Postpartum Birth Control Usage in the United States: Insights from PRAMS Phase 8.

AMIA ... Annual Symposium proceedings. AMIA Symposium·2026
See all related articles

Clinician consistency in creating coded problem lists is moderate. Standardizing terms improves agreement, but variations persist due to clinical style and diagnostic ambiguity.

Area of Science:

  • Health Informatics
  • Clinical Documentation
  • Medical Terminology

Background:

  • Coded problem lists are crucial for various healthcare applications.
  • Their utility hinges on consistent problem enumeration and terminology selection by clinicians.
  • Variability in these processes can limit the effectiveness of coded problem lists.

Purpose of the Study:

  • To assess inter-rater agreement among physicians creating coded problem lists.
  • To evaluate the impact of standardization on the consistency of coded problem lists.
  • To identify sources of variability in clinical problem representation.

Main Methods:

  • Ten physicians independently generated coded problem lists for five clinical cases.
  • The Unified Medical Language System (UMLS) was used as the controlled terminology.

Related Experiment Videos

  • Inter-rater agreement was measured using average pair-wise positive specific agreement.
  • Problems were standardized to common terms, accounting for synonymy, granularity, and concept representation.

    • Main Results:

    • Unstandardized coded problem lists demonstrated only moderate inter-rater agreement.
    • Standardization of terms significantly improved agreement among physicians.
    • Substantial variability remained, attributed to differences in clinical style and diagnostic ambiguity.

    Conclusions:

    • Inter-rater reliability for coded problem lists is currently limited.
    • Standardization methods enhance consistency but do not fully resolve variability.
    • Addressing clinical style and inherent diagnostic fuzziness is necessary for optimal coded problem list utility.