Usability, Face Validity, Feasibility, and Reliability of 3 Neurocritical Care Electronic Clinical Quality Measures

The field of neurology lacks neurocritical care (NCC) quality measures because of the complexity of neurologic disorders and challenges in data collection, hindering efforts to assess care quality and improve patient outcomes. This study examined the feasibility, usability, and scientific acceptability of 3 new facility-level neurology electronic clinical quality measures (eCQMs): (1) adult patients with generalized convulsive status epilepticus (GCSE) treated with benzodiazepine within 20 minutes of arrival to the emergency department, (2) dexamethasone administration before or within an hour of the first dose of antibiotics in patients with acute bacterial meningitis (BM), and (3) immunomodulatory therapy [plasma exchange or IV immunoglobulin] for myasthenic crisis (MG).

We evaluated the feasibility, usability, face validity, and reliability of the 3 eCQMs across 3 sites in the United States using a combination of qualitative and quantitative methodologies. The feasibility of reporting was evaluated by the percentage of data elements that do not require manual patient record review. Face validity was determined through semistructured clinician interviews and web survey, and usability was assessed through clinician interviews. Signal-to-noise reliability was calculated based on the testing data obtained from 3 hospitals.

The total number of denominator-eligible patients in the sample was 86 (GCSE), 88 (BM), and 193 (MG). The mean measure rates were as follows: GCSE, 12.3% (11.3%-13.3%); BM, 23.3% (10.8%-35.7%); and MG, 64.2% (46.2%-81.4%). All 3 eCQMs had high face validity (GCSE: 90%, BM: 90%, MG: 94%). The measures had high reliability: GCSE, 0.890 (0.816-0.963); BM, 0.817 (0.669-0.964); and MG, 0.958 (0.948-0.969). We found high feasibility for MG; however, BM and GCSE had feasibility concerns for several key data elements. Clinicians supported the measures' potential to improve care but expressed several usability concerns.

The eCQMs showed potential for being used for quality improvement. However, significant barriers to feasibility include inconsistent recording of diagnosis and procedure codes in electronic health records and underestimation of denominator and/or numerator cases, which hinder reporting in Centers for Medicare & Medicaid Services quality improvement programs. Continued refinement of the eCQM specifications is required before they can be implemented to enhance their impact on NCC quality.