大型言語モデルの医療応用のテストと評価:体系的なレビュー
Suhana Bedi1, Yutong Liu2, Lucy Orr-Ewing2
1Department of Biomedical Data Science, Stanford School of Medicine, Stanford, California.
JAMA
|October 15, 2024
まとめ
医療における大型言語モデル (LLM) の現在の評価は,主に試験の正確性を評価し,実際の患者データや公平性などの重要な側面を無視しています. 将来の研究は,より広い範囲と臨床的関連性が必要です.
科学分野:
- 医療における人工知能
- 自然言語処理 (NLP) アプリケーション
- 医療情報工学
背景:
- 大型言語モデル (LLM) は医療応用の可能性を示しています.
- 既存の評価方法は,臨床環境におけるLLMの有用性を完全に捉えることはできません.
- 将来の研究を導くために,現在のLLM評価の包括的な理解が必要です.
研究 の 目的:
- 医療におけるLLMに関する既存の評価を体系的に検討し,分類する.
- データタイプ,ヘルスケアタスク,NLP/NLUタスク,評価次元,医療専門分野に基づいて評価を分析する.
- 医学分野における現在のLLM評価戦略のギャップと限界を特定する.
主な方法:
- PubMedとWeb of Scienceの体系的な文献検索 (2022年1月から2024年2月まで)
- 医療における1つ以上のLLMを評価した研究を含める.
- 独立した審査員による519件の研究を5つの主要要素に分類する
主要な成果:
- 実際の患者ケアデータを利用した研究は5%のみで,ほとんどが医学知識評価 (例えば,免許試験) と診断に焦点を当てていた.
- 質問に答えるのがNLP/NLUの主要なタスク (84.2%) で,要約や対話に限られた重点を置いた.
- 精度が主な評価指標 (95. 4%) であったが,公平性,バイアス,毒性,導入の考慮事項はまれに評価された.
結論:
- 現在の医療におけるLLMの評価は 標準化されたテストの精度に大きく偏っており 現実世界の臨床データと 多様な課題の考慮が欠けている.
- 公平性や偏見 配備の準備といった重要な側面は 評価されていません
- 将来の研究は,標準化されたメトリック,臨床データ利用,そしてより幅広いヘルスケアタスクと専門分野をLLM評価に優先すべきです.
関連する概念動画
Issues And Trends In Healthcare Delivery System
5.6K
The issues and trends in healthcare delivery are constantly changing. The COVID-19 pandemic is one recent issue that wreaked havoc on healthcare systems, causing a shortage of healthcare workers, high demand for medicines and supplies, and increased medical expenditure due to a lack of insurance. Other issues include rising healthcare costs and care fragmentation.
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
5.6K
Methods of Documentation VI: Case Management Model
559
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
559
Health Information Technology and Healthcare Information System
806
Health Information Technology (HIT)
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
806
Documentation in Long-Term and Home Healthcare Setting
873
Documentation in long-term care facilities and home healthcare settings is crucial for ensuring continuous, coordinated, and comprehensive care for patients. Each setting has its specific documentation processes and tools:
Long-Term Care Facilities
Long-Term Care Facilities
873
Statistical Software for Data Analysis and Clinical Trials
502
Statistical software is pivotal in data analysis and clinical trials by providing tools to analyze data, draw conclusions, and make predictions. These software packages range from simple data management applications to complex analytical platforms, supporting various statistical tests, models, and simulation techniques. Their significance lies in their ability to handle vast amounts of data with precision and efficiency, enabling researchers to validate hypotheses, identify trends, and make...
502
Purpose of Health Records I
1.2K
The vital purpose of health records is to provide a complete and accurate account of a patient's medical history, including communication, diagnostic and therapeutic orders, care planning, research, and quality review.
Here's a breakdown of how health records serve these purposes:
Here's a breakdown of how health records serve these purposes:
1.2K


