Case Sampling vs Universal Review for Evaluating Hospital Postoperative Mortality in US Surgical Quality Improvement Programs

IMPORTANCE: Representative surgical case sampling, rather than universal review, is used by US Department of Veterans Affairs (VA) and private-sector national surgical quality improvement (QI) programs to assess program performance and to inform local QI and performance improvement efforts. However, it is unclear whether case sampling is robust for identifying hospitals with safety or quality concerns.

OBJECTIVE: To evaluate whether the sampling strategy used by several national surgical QI programs provides hospitals with data that are representative of their overall quality and safety, as measured by 30-day mortality.

DESIGN, SETTING, AND PARTICIPANTS: This comparative effectiveness study was a national, hospital-level analysis of data from adult patients (aged ≥18 years) who underwent noncardiac surgery at a VA hospital between January 1, 2016, and September 30, 2020. Data were obtained from the VA Surgical Quality Improvement Program (representative sample) and the VA Corporate Data Warehouse surgical domain (100% of surgical cases). Data analysis was performed from July 1 to December 21, 2022.

MAIN OUTCOMES AND MEASURES: The primary outcome was postoperative 30-day mortality. Quarterly, risk-adjusted, 30-day mortality observed-to-expected (O-E) ratios were calculated separately for each hospital using the sample and universal review cohorts. Outlier hospitals (ie, those with higher-than-expected mortality) were identified using an O-E ratio significantly greater than 1.0.

RESULTS: In this study of data from 113 US Department of Veterans Affairs hospitals, the sample cohort comprised 502 953 surgical cases and the universal review cohort comprised 1 703 140. The majority of patients in both the representative sample and the universal sample were men (90.2% vs 91.1%) and were White (74.7% vs 74.5%). Overall, 30-day mortality was 0.8% and 0.6% for the sample and universal review cohorts, respectively (P < .001). Over 2145 quarters of data, hospitals were identified as an outlier in 11.7% of quarters with sampling and in 13.2% with universal review. Average hospital quarterly 30-day mortality rates were 0.4%, 0.8%, and 0.9% for outlier hospitals identified using the sample only, universal review only, and concurrent identification in both data sources, respectively. For nonsampled cases, average hospital quarterly 30-day mortality rates were 1.0% at outlier hospitals and 0.5% at nonoutliers. Among outlier hospital quarters in the sample, 47.4% were concurrently identified with universal review. For those identified with universal review, 42.1% were concurrently identified using the sample.

CONCLUSIONS AND RELEVANCE: In this national, hospital-level study, sampling strategies employed by national surgical QI programs identified less than half of hospitals with higher-than-expected perioperative mortality. These findings suggest that sampling may not adequately represent overall surgical program performance or provide stakeholders with the data necessary to inform QI efforts.