Establishing the psychometric properties of a scale for evaluating quality in printed education materials

A study establishing the psychometric properties of a quality standard for quantifying the presence (or absence) of instructional design/learning principles contained in printed education materials (PEMs) was conducted in 4 phases. Content validity, a pretest which included creation of a rating key, inter-rater reliability, and intra-rater reliability levels were established for the Bernier Instructional Design Scale (BIDS). Experts in patient education established content validity by identifying 37 instructional design/learning principles as essential to PEM quality from a 90 item domain. An inter-disciplinary group of 89 health professionals applied the BIDS to a test PEM. Group ratings were compared with a rating key which served as a validated standard for the presence of principles in the test PEM. Reliability of the BIDS achieved acceptable levels when rating categories were collapsed from 4 to 3 levels to be consistent with a measurement model for rating checklists. A revised methodological approach and program of psychometric research are described.