Reliability of EHR-based clinical quality measures: Issues with the comprehension and implementation of eCQMs

Background: It is often assumed that electronic clinical quality measures (eCQMs), expressed in a format suitable for automatic extraction from electronic health records (EHRs), will be reliable by their nature. But several reliability challenges exist for quality measurement across providers, including different EHR systems, and provider variation in how EHRs are implemented. E-CQMs are expressed in formats unfamiliar to many hospitals and clinicians, and can be quite complex and lengthy, potentially leading to difficulty in understanding their intent and logic. Such understanding is needed to ensure that the data requirements are mapped properly to local systems and data capture patterns and workflows in each location. The Hospital eMeasures project, funded by CMS, has conducted extensive testing of measure properties for twelve retooled eCQMs, based on paper-based equivalents, as well as two de novo eCQMs. Experience gained during the testing process highlights key areas that should be addressed to ensure the reliability of quality measurement across hospitals or other providers using eCQMs. Design: Mixed methods were used to test eCQMs, including surveys of hospital IT and quality experts concerning the clarity of the eCQM logic, a survey of hospital IT staff concerning the availability and formatting of specific data elements used in the eCQM, and comparison of data extracted automatically from EHRs with data abstracted by trained nurse reviewers. Findings and Conclusions: The understandability of the eCQMs varied by measure; measures that included nested logic in the population criteria were more likely to be poorly understood. Hospitals varied in how specific data elements were captured or coded, and how data flowed from ancillary IT systems to the inpatient EHR. Robust education of local implementers and EHR vendors will help to improve the reliability of data capture for eCQMs. Thorough testing of both eCQMs and hospital eCQM scores is advisable.