Demonstrating the Reliability and Structural Validity of Creating Patient-Level and Clinician-Level Scores on the...

The Person Centered Primary Care Measure (PCPCM) was developed to assess "aspects that contribute to patient perceptions regarding the integrating, prioritizing, and personalizing functions of primary care." Several psychometric issues remain unresolved.

We sought to examine the performance of the existing patient-level model, evaluate measurement bias, assess the impact of item-level missingness on reliability, examine the structural validity of creating a clinician-level score, and identify the number of patients needed to achieve a reliable clinician-level score.

We used confirmatory factor analyses (CFA), item response theory, multilevel CFA, and retrospective survey data.

Three thousand one hundred ten patients clustered within 32 clinics and 94 clinicians completed the PCPCM.

CFA supported a single-factor patient-level model with 2 sets of correlated errors (RMSEA=0.06; CFI=0.98; TLI=0.98). Item response theory-based marginal reliability curves demonstrated that reliability drops precipitously if fewer than 6 items are answered. Multilevel CFA supported a single factor at the patient level and a single factor at the clinician level, with 2 sets of patient-level correlated errors (RMSEA=0.07; CFI=0.93; TLI=0.91). Scatter plots of clinician-level model-based and response-based scores showed nonlinearity and larger SEs when clinician scores were based on fewer than 5 patients. Reliability was >0.80 with 5 or more patients and 0.90 with 9 or more.

Our study demonstrates the reliability and structural validity of creating a patient-level PCPCM score as the average of answers to at least 6 PCPCM questions and creating a clinician-level score as an average of the PCPCM scores from at least 5 patients within a clinician.