Piloting QUADAS-3: a revised tool for the quality assessment of diagnostic accuracy studies

QUADAS-2 is the most widely used tool for evaluating risk of bias and applicability concerns in diagnostic test accuracy studies within systematic reviews. QUADAS-2 has recently been updated to a new version, named QUADAS-3. This paper outlines the piloting process undertaken as part of the development of QUADAS-3.

Multistage piloting: (1) piloting by the QUADAS-3 steering group on a set of five journal papers, (2) piloting workshop at the Global Evidence Summit attended by 16 participants, (3) think aloud interviews with seven researchers who piloted the tool while verbalizing their thoughts, and (4) piloting in five ongoing or completed systematic reviews by seven review authors who provided feedback in an online survey.

Feedback on the tool was generally positive across the four piloting stages. Participants appreciated the structure of the tool, assessment at the estimate level, and the introduction of a framework to define the ideal test accuracy trial. Participants provided suggestions for improvement to the structure and wording of the tool; this led to key changes including the insertion of descriptive prompts within the QUADAS-3 domains, a section at the beginning of the tool to outline the tool's phases and when they should be completed, and clearer wording throughout the tool. Participants also identified areas where further guidance is required for users, including development of worked examples, which will be covered in the associated QUADAS-3 guidance document.

Extensive piloting has ensured that feedback from potential users has been integrated into the development of QUADAS-3.

What is the problem? Doctors often use tests to find out if a person has a certain condition. It is important that these tests can correctly tell people who have the condition from those who do not. This is called test accuracy. Diagnostic reviews are a type of research that bring together results from different studies about the accuracy of a test. In diagnostic reviews, researchers need to check whether the studies they include are reliable. They also need to check whether the studies match the question the review looks at. Researchers can use a tool called QUADAS-2 to do this. QUADAS-2 was made in 2011 and is now out of date. We have made a new version called QUADAS-3. In this paper, we explain how we tested the QUADAS-3 tool before making the final version. What did we do? We tested the tool in four steps. 1. The people making the tool used it on five research papers. 2. It was tested by 16 people at a conference. 3. Seven researchers said their thoughts out loud while using the tool and we gathered their feedback. 4. Seven researchers used the tool in real reviews. They gave feedback in an online survey. We updated the tool after each step. What did we find? Most people liked the QUADAS-3 tool. They also told us ways to make it better. We made changes in response to people's comments. This led to the final version of QUADAS-3.