Reliability Analysis of a Newly Developed Questionnaire for Quality Control of Follow-up Visits in PolyIran Study

The PolyIran study is a large-scale pragmatic cluster randomized controlled trial of fixed-dose combination therapy (Polypill) for prevention of cardiovascular diseases (CVD) in Iran. The PolyIran Quality Control Program (PIQCP) including a new questionnaire was developed to assess the quality of data collection during follow-up visits. The aim of this study was to assess the inter-rater reliability of PIQCP questionnaire.

The study was conducted in 26 (11%) randomly selected clusters (from a total of 236 PolyIran clusters). All participants within these 26 clusters were enrolled. The quality scores were measured according to the PIQCP guidelines by two independent raters. The intraclass correlation coefficients (ICC) were measured. In addition, the quality scores were categorized into good (70%) and poor (<70%). The kappa coefficient was used to assess inter-rater agreement for this categorical quality scores.

A total number of 945 PolyIran participants were enrolled of which, 501 (53%) were from intervention arm. In 934 participants (98.8%), the quality score could be successfully identified by both raters. The ICC (95%CI) of the overall quality scores was 0.985 (0.983-0.987). It was 0.976 (0.972-0.980) and 0.988 (0.986-0.990) in intervention and control arms, respectively. We found excellent agreement between the two raters in identifying participants with good and poor quality scores (kappa = 0.988, P < 0.001). The kappa values were 0.972 (P < 0.001) and 1.000 (P < 0.001) in intervention and control arms, respectively.

Our results suggested that the PIQCP questionnaire is a reliable tool for assessing quality of data collection in PolyIran follow-up visits. Using this measure will help us in efficient monitoring of the PolyIran follow-ups and may ensure high quality data.