Postpublication criticism and the shaping of clinical knowledge

Letters to the editor are an important means for ensuring accountability of authors and editors. They form a part of the postpublication peer review process. I studied the critical footprint made in the medical literature by 3 randomized trials (Hypertension Optimal Treatment [HOT], Captopril Prevention Project [CAPPP], and Swedish Trial in Old Patients with Hypertension 2 [STOP-2]) published in The Lancet and investigated the extent to which that footprint was preserved in shaping clinical knowledge.

Qualitative appraisal of the criticism of each trial, taken from published letters. Agreed weaknesses and unanswered criticisms were identified from the authors' reply. I searched MEDLINE for practice guidelines published after the trial report and sought evidence for incorporation of criticism into these guidelines.

From the time of publication to October 2000, HOT was cited in 9 of 36 practice guidelines; CAPPP, in 6 of 36; and STOP-2, not at all. HOT received 14 published criticisms, 5 comments, and 3 questions, of which 15 were responded to. Only 1 criticism, lack of power, was referred to in 1 guideline. CAPPP received 14 criticisms, 9 comments, and 3 questions, of which 8 were responded to. Only 1 criticism, imbalances between groups, was referred to in 1 guideline. STOP-2 received 12 criticisms, 9 comments, and 3 questions, of which only 6 were responded to.

More than half of all criticism made in correspondence went unanswered by authors. Important weaknesses in trials were ignored in subsequently published practice guidelines. Failure to recognize the critical footprint of primary research weakens the validity of guidelines and distorts clinical knowledge.