Recorded presentation

Yue Li, Department of Community and Preventive Medicine, University of Rochester, Medical Center, 601 Elmwood Avenue, Box 644, rochester, NY 14642, (585)273-2548, yue_li@urmc.rochester.edu, Andrew Dick, PhD, Community and Preventive Medicine, University of Rochester, Box 644, School of Medicine and Dentistry, Rochester, NY 14642, and Dana B. Mukamel, Ph D, Department of Medicine, Division of General Internal Medicine & Primary Care, University of California, Irvine, Health Policy Research, 100 Theory, Suite 110, Irvine, CA 92697.

Risk adjustment regression models are used to control for patients' preexisting risk factors when comparing quality of care across healthcare providers based on health outcomes. Risk-adjusted quality indicators (QIs) are usually calculated as differences or ratios between the observed and the estimated outcomes (O-to-E differences or O-to-E ratios). Debate on the adequacy of these measures has focused on 1) finite sample sizes (e.g., limited number of patients treated by a physician in a year) and 2) incomplete risk adjustment due to unobservable but important factors. Little work has been done on the choice of the QIs, the specification of the risk adjustment models, and the interrelationship between them. These choices are usually made ad hoc, with no consideration of the underlying, true health care production process (HPP). This study examines the importance of these issues, quantifying the errors in quality assessments using Monte Carlo simulations.

We first defined different HPP functions, and then showed that 1) the O-to-E difference or the O-to-E ratio was consistent with an additive HPP of the outcome rate or a multiplicative one, respectively, and 2) the logistic specifications of the risk adjustment model were inconsistent with either of the two functions.

We obtained the distributions of the patient's estimated outcome and physician quality for the simulation from the empirical risk adjustment scenario ¨C the 1999 New York State coronary artery bypass graft surgery quality report. The error due to misspecification of the measures was quantified by the Kappa statistic between the true quality rankings and the risk-adjusted quality rankings.

We found that, assuming consistent risk adjustment model and perfect model estimation, a misspecified QI (e.g., O-to-E ratio for the additive HPP) introduced significant error (Kappa=0.25 for the additive HPP; Kappa=0.52 for the multiplicative HPP). When the classical logistic model was estimated for the additive HPP, the measured surgeon quality agreed accurately with the true quality (Kappa=0.98 for both QIs). When the logistic model was estimated for the multiplicative HPP, however, the misspecified QI introduced significant error (Kappa=0.50 using O-to-E difference, Kappa=0.99 using O-to-E ratio).

We conclude that a correctly specified QI is more important than consistent model estimation in limiting error of risk adjustment for the two simplest HPPs. Current risk adjustment approach based on logistic regressions is appropriate only if the underlying medical care process is additive, i.e. only if patient risks do not interact with quality of care.

Learning Objectives:

At the end of this presentation particpants will
• understand how outcome-based quality measurement is associated with the underlying medical care process
• understand the importance to estimate consistent risk adjustment model and construct consistent quality indicators with respect to the medical care process.

Keywords: Report Card, Outcome Measures

Presenting author's disclosure statement:
I do not have any significant financial interest/arrangement or affiliation with any organization/institution whose products or services are being discussed in this session.

Recorded presentation