A general critical appraisal tool: an evaluation of construct validity.

Area of Science:

• Health research methodology
• Psychometrics
• Evidence-based practice

Background:

• Existing critical appraisal tools (CATs) often lack documented development, validity, and reliability testing.
• A novel CAT was developed using existing tools, research theory, and reporting guidelines.
• Further evaluation is needed to confirm the effectiveness of the proposed CAT.

Purpose of the Study:

• To establish a scoring system for a new critical appraisal tool.
• To evaluate the construct validity of the proposed tool prior to reliability testing.

Main Methods:

• Construct validity was assessed using the Standards for educational and psychological testing criteria.
• The proposed CAT was compared against five alternative CATs for internal structure and relations to other variables.
• Sixty research papers across six designs were randomly sampled for analysis.

Main Results:

• The proposed CAT demonstrated significant weak to moderate positive correlations with alternative CATs across most research designs (Kendall's τ 0.33-0.55).
• Stronger correlations were observed in quasi-experimental, descriptive, qualitative, and systematic review designs.
• No significant correlations were found in single system research designs.

Conclusions:

• The proposed critical appraisal tool exhibits sufficient construct validity.
• Inferences drawn from the scores of the proposed CAT are likely sound.
• The tool's theoretical basis and objectives support its validity for appraising research quality.