Validation of algorithms in studies based on routinely collected health data: general principles

Area of Science:

• Real-world data (RWD) analysis
• Pharmacoepidemiology
• Health informatics

Background:

• Real-world data (RWD) from health and administrative databases are increasingly used by decision-makers.
• Algorithms are essential for defining variables in RWD studies, identifying specific health conditions or characteristics.
• Ensuring algorithm validity is critical for generating reliable evidence to support evidence-based healthcare.

Purpose of the Study:

• To systematize terminology, methods, and practical considerations for validating RWD-based algorithms.
• To provide a framework for conducting robust algorithm validation studies.
• To enhance the trustworthiness of evidence derived from RWD.

Main Methods:

• Discussion of algorithm accuracy measures and gold/reference standards.
• Consideration of study size, prioritization of accuracy metrics, and algorithm portability.
• Emphasis on transparency in validation decisions and interpretation of findings.

Main Results:

• Algorithm validity must be assessed within the context of specific data sources.
• Prioritization of validity measures depends on the variable's role in the analysis (e.g., eligibility, exposure, outcome).
• Validation should be an integral part of routine RWD source maintenance.

Conclusions:

• Systematic validation of RWD algorithms is a prerequisite for valid study findings.
• Context-specific evaluation and transparent reporting of algorithm validity are essential.
• Ongoing validation efforts ensure the continued utility and reliability of RWD for healthcare decision-making.