Usability and feasibility of the longitudinal implementation strategy tracking system: a think-aloud study with...

Tracking implementation strategies and their adaptations over time remains a persistent methodological challenge in health services research. The Longitudinal Implementation Strategy Tracking System (LISTS) was developed to support structured, longitudinal documentation of strategies, modifications, and discontinuations aligned with established implementation science frameworks using timeline follow-back procedures and an open-access web-based platform. The study objective was to evaluate the usability, feasibility, and methodological fit of the open-access web-based LISTS platform using a structured think-aloud protocol with implementation researchers.

Nine implementation researchers completed structured think-aloud usability sessions while performing core LISTS tasks, including project setup, strategy entry, bundling, modification logging, discontinuation tracking, and dashboard exploration. Sessions were transcribed and analyzed using hybrid deductive-inductive thematic analysis. Participants also completed the System Usability Scale (SUS) and NASA Task Load Index (NASA-TLX).

The mean SUS score was 76.94, indicating good perceived usability, and mean NASA-TLX workload was low (1.98 on a 1-7 scale). Qualitative findings revealed consistent endorsement of LISTS' structured longitudinal tracking and theoretical alignment with well-known frameworks. Usability barriers included terminology density, multi-level unit hierarchy confusion, and ambiguity in modification-tracking workflows. A cross-cutting tension emerged between theoretical fidelity and accessibility for non-expert users. Despite conceptual friction, participants completed tasks successfully and rated the system positively overall.

LISTS demonstrates strong conceptual rigor and functional usability among implementation researchers. However, refinements to terminology scaffolding, hierarchical logic, and modification workflows are needed to enhance accessibility and scalability beyond expert users. Findings highlight broad challenges and considerations for translating implementation science theory and terminology into practical digital infrastructure.