Developing an instrument for evaluating implementation of clinical practice guidelines: a test-retest study

This study focuses on the development of an instrument for the evaluation of clinical practice guidelines and is one part of a research project about the implementation and use of such guidelines among hospitals in the southern region of Sweden. The aim of the present paper was to investigate the test-retest reliability of a questionnaire.

A questionnaire was designed to gather data about guidelines that have been implemented as well as information about factors, which, according to the Promoting Action on Research Implementation in Health Services (PARIHS)-model, influence the success of implementation. Thirty-nine health professionals at one of the hospitals included in the survey completed the questionnaire on two occasions within a mean time of 5.5 weeks. The test-retest reliability was analysed by means of Cohen's Kappa and percentage concordance.

Eight items had good agreement in terms of strength and high percentage concordance. With regard to the Kappa values, 13 items show moderate and two fair agreement.

The test-retest reliability scores show mainly acceptable results indicating a reasonable stability, thus suggesting the possibility of further developing the instrument. The factors described in the PARIHS-model seem relevant for use in evaluating implementation and use of guidelines. The instrument could benefit from a revision of the language in order to enhance clarity and make it less abstract.