How consistent is the reliability of hospital performance based on simulation and multilevel approaches?

Objective: Many studies have used the hierarchical logistic model to estimate the reliability of provider/insurer performance. The shrinkage factor for the hierarchical model is assumed to be normally distributed with and the assumption may be incorrect. In this study we compare the reliability of individual diabetes mellitus (DM) process measures based on simulation and hierarchical logistic approaches. Data Source: This study utilizes data from Taiwan's National Health Insurance (NHI) database for 2008. Our DM sample size is 1,276,114. Methods: We derive the individual measures from 5 evidence-based process measures. The reliability index is defined as the ratio of the true score variance to the observed score variance. To estimate the reliability of the simulation approach, we compute the two variances above from 1,000 bootstrap samples (replicates). To estimate the reliability of the hierarchical logistic model, we apply the hierarchical logistic model to estimate the true score variance and then use the Spearman Brown Prophecy Formula to estimate the reliability index. Principal Findings: The reliability estimated from the hierarchical logistic model almost has a slightly lower reliability, especially for cases with smaller intraclass correlations (ICC). For example, the Annually Serum Creatine Test has the lower ICC (3%), and thus its reliability estimated by hierarchical model (89%) has more deviance from the reliability (97%) estimated from the simulation methods. The deviance for the Annually Serum Creatine Test is larger than the Annually LDL Test, which has the higher ICC (14%). Conclusions: Our analysis has shown that the reliability derived from the simulation method is almost consistent with the reliability estimated from the hierarchical logistic model. However, when using the hierarchical logistic model to estimate the reliability of provider performance one should be aware of its limitations, especially for providers with smaller ICC.