Related Experiment Video
Updated: Jul 19, 2026

A Study on an Intelligent Diagnosis and Treatment Assistant System for Acupuncture in Diminished Ovarian Reserve Based on a Knowledge Graph
Published on: May 29, 2026
Can concept sorting provide a reliable, valid and sensitive measure of medical knowledge structure?
Kevin McLaughlin1, Sylvain Coderre, Garth Mortis
1Division of Nephrology, Foothills Hospital, 1403 29th Street NW, T2N 2T9, Calgary, Alberta, Canada. kevin.mclaughlin@calgaryhealthregion.ca
Context:
Evolution from novice to expert is associated with the development of expert-type knowledge structure. The objectives of this study were to examine reliability and validity of concept sorting (ConSort) as a measure of static knowledge structure and to determine the relationship between concepts in static knowledge structure and concepts used during diagnostic reasoning.
Method:
ConSort was used to identify static knowledge concepts and analysis of think-aloud protocols was used to identify dynamic knowledge concepts (used during diagnostic reasoning). Intra- and inter-rater reliability, and correlation across cases, were evaluated. Construct validity was evaluated by comparing proportions of nephrologists and students with expert-type knowledge structure. Sensitivity and specificity of static knowledge concepts as a predictor of dynamic knowledge concepts were estimated.
Results:
Thirteen first-year medical students and 19 nephrologists participated. Intra- and inter-rater agreement for determination of static knowledge concepts were 1.0 and 0.90, respectively. Reliability across cases was 0.45. The proportions of nephrologists and students identified as having expert-type knowledge structure were 82.9% and 55.8%, respectively (p=0.001). Sensitivity and specificity of ConSort((c)) in predicting concepts that were used during diagnostic reasoning were 96.8% and 27.8% for nephrologists and 87.2% and 55.1% for students.
Conclusions:
ConSort is a reliable, valid and sensitive tool for studying static knowledge structure. The applicability of tools that evaluate static knowledge structure should be explored as an addition to existing tools that evaluate dynamic tasks such as diagnostic reasoning.
More Related Videos
07:35A Knowledge Graph Approach to Elucidate the Role of Organellar Pathways in Disease via Biomedical Reports
Published on: October 13, 2023
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
Related Concept Videos
Concepts of Health and Illness
Concepts and Prototypes
The brain organizes this information using concepts, which are mental categories grouping linguistic data,...
Natural and Artificial Concepts
Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Sensitivity, Specificity, and Predicted Value
Sensitivity is the...