Related Experiment Videos
The hidden patterns of alignment: exploring variability between generative AI and human
Henry Moon1, Ahmet Guven2, Kuang-Drew Li3
1Educational Innovation Institute and Department of Pediatrics, Medical College of Georgia, Augusta University, Augusta, GA, USA.
Medical Education Online
|August 5, 2026
Summary
Generative artificial intelligence (GenAI) models offer consistent and accurate curriculum alignment for medical education, outperforming human experts in reliability. A Curricular Human-AI Teaming model could leverage AI efficiency with human expertise for improved outcomes.
Area of Science:
- Medical Education
- Artificial Intelligence in Education
Background:
- Aligning medical curricula with standardized assessments like the United States Medical Licensing Examination (USMLE) Step 1 is crucial for competency-based education.
- Generative artificial intelligence (GenAI) presents a potential tool for this alignment, but its effectiveness compared to human experts is not well-established.
Purpose of the Study:
- To investigate the consistency, accuracy, and strategies of GenAI models versus human subject matter experts in aligning pharmacological topics with USMLE Step 1 content.
- To compare the inter-rater reliability and consensus between AI-generated and human-generated curricular alignments.
Main Methods:
- A comparative case study design was employed, analyzing alignments generated by GenAI models and human experts.
- Quantitative metrics including exact agreement percentages and pairwise correlation coefficients were used to assess consistency and consensus.
- Qualitative examination of challenges, strategies, and the impact of time constraints was also conducted.
Main Results:
- GenAI models demonstrated higher within-group inter-rater reliability (60-79% exact agreement) and stronger correlations compared to human raters (14-66% exact agreement).
- GenAI alignments were based on contextual analysis and hierarchical mapping, while human alignments involved iterative refinement and reflection.
- Time constraints minimally impacted GenAI performance, unlike human experts.
Conclusions:
- GenAI models provide efficient, consistent, and accurate curricular alignments but lack the nuanced decision-making of human experts.
- Human experts offer contextually informed decisions despite inherent variability.
- A Curricular Human-AI Teaming (Curricular HAT) model is proposed to integrate AI efficiency with human judgment for enhanced alignment processes.
Related Concept Videos
01:16Non-equilibrium in the Cell
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
01:28Automatic Processing and Automatic Social Behavior
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...
01:29Impression Management Techniques III: Aligning Actions
Aligning actions are communicative strategies individuals employ to maintain social harmony and preserve personal identity in the face of potential disruptions to social norms. These actions are particularly important in managing social impressions when one's behavior might be seen as inappropriate, incompetent, or morally questionable.Types of Aligning ActionsThe three principal types of aligning actions are disclaimers, accounts, and apologies.DisclaimersDisclaimers are preventive; they are...
01:11Variability: Analysis
Measures of variability are statistical metrics that reveal the dispersion pattern within a dataset. They are pivotal in biostatistics, providing insights into the heterogeneity within health and biological data. Variability signifies the degree to which data points diverge from one another, helping researchers understand the potential range of values and associated uncertainty within the data.
The range is a simple measure of variability, indicating the difference between the highest and...
The range is a simple measure of variability, indicating the difference between the highest and...
01:24Sequence Networks of Rotating Machines
A Y-connected synchronous generator, grounded through a neutral impedance, is designed to produce balanced internal phase voltages with only positive-sequence components. The generator's sequence networks include a source voltage that is exclusively in the positive-sequence network. The sequence components of line-to-ground voltages at the generator terminals illustrate this configuration.
Zero-sequence current induces a voltage drop across the generator's neutral impedance and other...
Zero-sequence current induces a voltage drop across the generator's neutral impedance and other...