DARTS 2000 online diabetes management system: formative evaluation in clinical practice

Failure to engage in user-informed evaluation of emergent health informatics tools can have negative consequences for future implementation, related both to poor usability or clinical utility and to suboptimal stakeholder buy-in. This paper describes a formative evaluation in primary care of a multifaceted, web-based resource for diabetes management. The primary aims were to assess the usability and utility of the prototype in order to inform system refinements prior to implementation, and to investigate barriers and facilitators to system use so as to aid the development of a tailored implementation plan.

A mixed-method approach involving survey, remote observation, semi-structured interviews and electronic feedback.

One Scottish local health care cooperative comprising five general practice surgeries and their staff.

A survey following temporary exposure to a dummy site revealed high levels of computer familiarity, welcoming attitudes and positive ratings of usability, format and utility. Comments mainly addressed content accuracy, feature suggestions and usability issues. Key barriers and facilitators to use included time and training. Remote observation following access to live clinical data enabled profiling of usage by individual and professional group. In the 3 month observation period administrators were the most frequent users, followed by GPs and nurses, with clinical and data entry screens accessed most often. Semi-structured interviews with key respondents sampled by professional group and usage frequency provided richer qualitative information on barriers and facilitators to use, patterns of system integration into work routines and system usability, content and utility. Content analysis of electronic feedback revealed mainly technical queries and general expressions of satisfaction.

Evaluation informed a number of important and unforeseen improvements to the prototype and helped refine the implementation plan. Engagement in the process of evaluation has led to high levels of stakeholder ownership and widespread implementation.