Ability of expert physicians to structure clinical guidelines: reality versus perception

Structuring Textual Clinical Guidelines (GLs) into a formal representation is a necessary prerequisite for supporting their automated application. We had developed a collaborative guideline-structuring methodology that involves expert physicians, clinical editors and knowledge engineers, to produce a machine-comprehensible representation for automated support of evidence-based, guideline-based care. Our goals in the current study were: (1) to investigate the perceptions of the expert physicians and clinical editors as to the relative importance, for the structuring process, of different aspects of the methodology; (2) to assess, for the clinical editors, the inter-correlations among (i) the reported level of understanding of the guideline structuring ontology's (knowledge scheme's) features, (ii) the reported ease of structuring each feature and (iii) the actual objective quality of structuring.

A clinical consensus regarding the contents of three guidelines was prepared by an expert in the domain of each guideline. For each guideline, two clinical editors independently structured the guideline into a semi-formal representation, using the Asbru guideline ontology's features. The quality of the resulting structuring was assessed quantitatively. Each expert physician was asked which aspects were most useful for formation of the consensus. Each clinical editor filled questionnaires relating to: (1) the level of understanding of the ontology's features (before the structuring process); (2) the usefulness of various aspects in the structuring process (after the structuring process); (3) the ease of structuring each ontological feature (after the structuring process). Subjective reports were compared with objective quantitative measures of structuring correctness.

Expert physicians considered having medical expertise and understanding the ontological features as the aspects most useful for creation of a consensus. Clinical editors considered understanding the ontological features and the use of the structuring tools as the aspects most useful for structuring guidelines. There was a positive correlation (R = 0.87, P < 0.001) between the reported ease of understanding ontological features and the reported ease of structuring those features. However, there was no significant correlation between the reported level of understanding the features - or the reported ease of structuring by using those features - and the objective quality of the structuring of these features in actual guidelines.

Aspects considered important for formation of a clinical consensus differ from those for structuring of guidelines. Understanding the features of a structuring ontology is positively correlated with the reported ease of using these features, but neither of these subjective reports correlated with the actual objective quality of the structuring using these features.