Mortality Measurement: The Veterans Health Affairs Experience in Measu

By Marta L. Render, M.D. and Peter Almenoff, M.D.

Introduction

The Veterans Health Affairs (VA) began a national program to measure and report risk adjusted mortality and length of stay in its intensive care units (ICU) in October of 2004 expanding to all patients admitted for acute medical or surgical conditions in October of 2008. The program, the VA Inpatient Evaluation Center (IPEC), uses a validated risk measure.^1-3 The VA IPEC uses electronic data to measure, people resources to drive change. This brief paper provides an overview of the VA risk methods and lessons learned since implementation.

Risk Adjustment Methodology

The hallmark of the VA risk model is its reliance solely on electronic elements from the VA medical record. Data is directly extracted from the patient treatment file, patient file, laboratory file, and radiology file of each site using customized programming and sent encrypted to the central data repository located at the Cincinnati VA Medical Center. Diagnosis, comorbid disease burden, source of admission (emergency room/outpatient clinic, operating room, nursing home, ward, or outside hospital) are included as categorical variables, while age and the worst measured value of 11 laboratory tests (sodium, blood urea nitrogen, creatinine, bilirubin, albumin, glucose, hematocrit, white blood cell count, pH/paCO2, and PaO2) are treated as cubic splines. Patients are assigned to one mutually exclusive diagnosis using ICD-9-CM coding from the ICU bedsection. Comorbid disease burden is assessed using Elixhauser's approach.^4-5 The risk model in intensive care unit patients has excellent calibration and discrimination when used to predict hospital mortality (validation set of 220,813 cases: c statistic 0.892, Hosmer Lemeshow Chi square 376) or mortality at 30 days (validation set of 193,944: C statistic 0.87, Hosmer Lemeshow goodness of fit statistic chi square 161). Advantages of use of a wholly electronic dataset includes access to reliable laboratory values, improved face validity, reduced cost compared to manual data extraction with the attendant opportunity to extend mortality measurement across an entire system, and ease of updating the models.

Measures Derived from Mortality Data

From the risk model, for each intensive care or hospital, a standardized mortality ratio (SMR, observed/predicted deaths) is determined using a 2-level hierarchical random effects model for outcomes of death at hospital discharge and death at 30 days from admission. The random effects model accounts for nesting of data in each ICU or hospital to improve estimate accuracy. An observed minus predicted length of stay is also determined. Use of the difference between predicted and observed length of stay allows estimation of the cost avoidance or opportunity loss when a unit OMELOS is multiplied by the daily cost of an ICU day and the annual census. We also now track unadjusted mortality at hospital discharge and 30 days for both the acute care and ICU patients and have piloted the unadjusted mortality of patients transferred from the ward to the ICU as an indicator of the ability of a hospital to detect and rescue deteriorating patients. Finally, using the same risk model, we created a physiologic case mix index where the numerator was the predicted mortality for patients in the specific ICU and the denominator the predicted mortality for patients in all VA ICUs. Because the proportion of operative cases (those with surgery in the 24 hours surrounding ICU admission) significantly influenced results when aggregated with non-operative cases, case mix indices were created separately for operative and non-operative and then a weighted measure based on the proportion of each was determined.

Results

The coefficients or weights of the predictors of the model, developed on a pilot dataset from 2002-2004, were fixed to allow tracking of ICU performance overtime, and SMR drifted significantly downward. This drift made interpretation of the SMR more difficult. For example in 2007, the VA SMR nationally was 0.8. Hospitals or ICUs with SMR only slightly above 1 then appeared on face to be "average" (where SMR of 1 = observed/predicted deaths) but in fact might have a 20% difference in risk adjusted mortality. To avoid confusion, the risk models are now recalibrated at the beginning of each year on the prior two years data, and the fixed weights of the predictors then applied to the new year as well as the prior years (to 2002). The reason for the downward drift is unknown, although temporally related to VA initiatives to improve implementation of evidenced based practices. Standardized mortality ratios varied somewhat based on the type of the ICU and level of complexity of the ICU. The ability to stratify by type or level of ICU and create benchmarks that were ICU specific improved the early face validity and acceptance of this measurement approach.

Figure 1. VA SMR30 and SMR Hosp

In some ICUs, there was a dramatic difference between the SMR that predicted death at hospital discharge (SMR_hosp) from the SMR predicting death at 30 days (SMR₃₀). Anecdotal follow-up suggested that variation in discharge practices related to the availability of long term acute care units and palliative care units were important in hospitals with large differences when their SMR_hosp was subtracted from SMR₃₀. Variation in unadjusted mortality of patients transferred from the ward to the ICU also varied significantly (2004 : overall 20%, range 6-36%, 2008: overall 16%, range 4-32%) and has fallen across the VA coincident with implementation of rapid response teams.

Figure 2. Variation in SMR Stratified by Type of ICU and Level of Complexity

Use of multiple mortality measures improves confidence in using the results. For instance, a small ICU with a higher SMR at hospital discharge and a normal SMR at 30 days likely has "normal' performance" and the high SMR_hosp is related to limited resource for long term acute care patients (on vents, severe debilitation). When death occurs in a patient at this hospital even after a year of inpatient care, it counted toward that hospital's mortality; while similar patients in other hospitals were "discharged" when sent to long term acute care hospitals or rehab facilities, and counted as survivors. The signal to noise ratio appeared improved when multiple mortality measures were tracked and concordant (SMR at hospital discharge, unadjusted mortality, mortality at transfer to the ICU from the ward). Similarly, given an imperfect model, when VA case mix was low and SMR elevated again the signal increased. The VA case mix also allowed tracking of the relative severity of illness of hospitalized patients in a system with hospitals with varying services and complexity.

Lessons Learned

Following 4 intense years of building a system that measures and reports risk adjusted mortality in 138 hospitals nationally, we have some lessons regarding structure of a national measurement system outside of the VA that might be valid. First, a risk adjustment model that predicts death at 30-days in addition to a model predicting death at hospital discharge will be important to avoid gaming. Next, resources and expertise to support recalibration of the weights of the model at appropriate time frame, using a large dataset will be needed as part of the infrastructure of the program. Use of laboratory data which provides a surrogate for variation in physiology will likely 1) improve face validity, 2) is probably possible now given the use of computerized systems for laboratory data retrieval in most hospitals, and 3) likely neutralizes the impact of gaming using administrative data. Fourth, regionalization of the results (as has been done with Healthcompare) and/ or stratification by mission or complexity of facility will improve the usability of the results. Finally, because all performance measures inherently will be gamed, thinking about mortality measures as a bundle of indicators rather than a single gold standard might improve the information created by the models.

References

1. Render, M.L., et al., Automated computerized intensive care unit severity of illness measure in the Department of Veterans Affairs: preliminary results. SISVistA Investigators. Scrutiny of ICU Severity Veterans Health Systems Technology Architecture. Crit Care Med 2000. 28(10): p. 3540-6.

2. Render, M.L., et al., Variation in outcomes in Veterans Affairs intensive care units with a computerized severity measure. Crit Care Med 2005. 33(5): p. 930-9.

3. Render, M.L., et al., Veterans Affairs intensive care unit risk adjustment model: validation, updating, recalibration. Crit Care Med 2008. 36(4): p. 1031-42.

4. Elixhauser, A., et al., Comorbidity measures for use with administrative data. Med Care 1998. 36(1): p. 8-27.

5. Johnston, J.A., et al., Impact of different measures of comorbid disease on predicted mortality of intensive care unit patients. Med Care 2002. 40(10): p. 929-40.

Current as of February 2009

Return to Contents
Proceed to Next Section