AI-Assisted Rapid Quality Analysis in Implementation Science: Methodological Study

Area of Science:

• Implementation Science
• Artificial Intelligence
• Qualitative Analysis

Background:

• Translating evidence-based therapies into practice is challenging, with implementation science (IS) experts playing a crucial role.
• Manual qualitative coding is time-consuming and costly; AI offers a potential solution, but concerns about quality, validity, and ethics persist.
• A novel method for AI-assisted rapid qualitative analysis has been developed to address these concerns.

Purpose of the Study:

• To develop an open-source, encoder-based small language model (SLM) for AI-assisted rapid qualitative analysis in IS.
• To evaluate the accuracy and generalizability of DistilBERT and ELECTRA models in replicating expert coding.
• To enhance accessibility for non-technical experts through user-friendly tools.

Main Methods:

• Two previously coded IS datasets were used to train and fine-tune DistilBERT and ELECTRA models.
• Performance was measured using area under the precision-recall curve and Cohen κ, comparing model output to expert coding.
• An open-source Python package (pytranscripts) and a Streamlit web application were developed for user-friendly transcript processing, coding, and evaluation.

Main Results:

• SLMs significantly accelerate qualitative analysis with high accuracy, showing strong agreement with human annotators (DistilBERT Cohen κ=0.95; ELECTRA Cohen κ=0.71 on the original dataset).
• Model performance decreased on a second, more ambiguous dataset (DistilBERT Cohen κ=0.48; ELECTRA Cohen κ=0.39), highlighting the impact of coding approach.
• Performance is influenced by the number of codes and whether multiple codes are applied per data segment.

Conclusions:

• Small language models can effectively assist qualitative researchers with coding tasks, provided careful attention is given to data preparation for training.
• This AI-assisted approach is particularly valuable in settings where large language models are impractical or undesirable.
• The developed tools enhance the accessibility of AI for qualitative analysis in implementation science.