How Good is Good Enough? A Case Study with Present on Admission (Text Version)

On September 28, 2010, Jeff Geppert made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (940 KB). 

---

Slide 1

 

How good is good enough? A case study with Present on Admission

AHRQ Annual 2010 Annual Conference
September , 2010
Session #45
Jeffrey Geppert, PMP, EdM, JD
Battelle Memorial Institute

Slide 2

 

Acknowledgements

• Support for Quality Indicators II (290-04-0020):
  • Mamatha Pancholi, AHRQ QI Project Officer
• HCUP State Inpatient Databases (SID). Healthcare Cost and Utilization Project (HCUP). 2004-2008. Agency for Healthcare Research and Quality, Rockville, MD. http://www.hcup-us.ahrq.gov/sidoverview.jsp

Slide 3

 

Why Measurement?

• What is the purpose of measurement?
  • Reduce uncertainty in decisionmaking
• What is the decision?
  • Consumers: what provider to select
  • Providers: where to allocate effort
• What is the uncertainty?
  • Consumers: would have selected a different provider
  • Providers: would have allocated effort differently

Slide 4

 

Measuring Uncertainty

• Expected Opportunity Loss (EOL):
  • Amount of loss X Probability of Loss
  • Loss relative to the best available alternative (opportunity)
• Expected Value of Information (EVI):
  • EOL_{After measurement} - EOL_{Before measuremen}^t
  • EVI may be positive or negative
• Risk aversion:
  • If EVI = 0, then the decision is about risk

Slide 5

 

Measuring Uncertainty

• Measure Model:
  • Y = alpha + X*B
• Y - the outcome of interest
  • Numerator, denominator, exclusions
• X*B - patient characteristics
  • Moderators and mediators
• Alpha - provider quality
  • Prior ability (related outcomes, related procedures, structure, context, past performance)
  • Current effort (signal)

Slide 6

 

Iatrogenic Pneumothorax

Sample of a Quality Indicator (QI)

Source: AHRQ PSI Technical Specifications, PSI #6 (Version 4.1b).

Slide 7

 

POA Overview: Approach

• Two sets of algorithms needed to incorporate POA information:
  1. Develop response variables and comorbidity factor covariates in the presence of POA data.
     • Less measurement error thereby more accurate and based on fewer assumptions.
  2. Develop response variables and comorbidity factor covariates in the absence of POA data.
     • Use observed POA data to estimate probability of POA for response and comorbidity factors for patients that do not have POA data.
     • Provide hospital with risk-adjusted rate that would be "most likely" had they collected POA data.
• Present on Admission White Paper:
  • http://qualityindicators.ahrq.gov/Resources/default.aspx

Slide 8

 

Iatrogenic Pneumothorax

• Case study:
  • Select two hospitals (A and B) with POA data (median denominator and rate).
  • Randomly use POA data for X% of discharges and predict the missing
    • None (no POA data)
    • 10%—90%
    • All (full POA data)—"gold standard"
  • Estimate EOL based on events, deaths, inpatient days and charges (NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74).

Slide 9

 

Expected Opportunity Loss

Image: Line graph displays the following information and hospital rates:

SID States:
Events: 8,306
Deaths: 581
Days: 36,547
Charges: $143,697,389

Hospital A:
Events: 3
Deaths: 0
Days: 12
Charges: $48,561

Hospital B:
Events: 4
Deaths: 0
Days: 16
Charges: $63,659

Slide 10

 

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital A (Prior):
Events:  3
Deaths:  0
Days:  12
Charges:   $48,561

Hospital A (Post): ;
Events:  5
Deaths:  0
Days:  24
Charges:  $94,370

Slide 11

 

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital B (Prior):
Events:  4
Deaths:  0
Days:  16
Charges:  $63,659

Hospital B (Post):
Events:  5
Deaths:  0
Days:  22
Charges:  $87,472

Slide 12

 

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital A (Prior):
Events: 5
Deaths: 0
Days: 24
Charges: $94,370

Hospital A (Post):
Events: 6
Deaths: 0
Days: 27
Charges: $107,226
 

Slide 13

 

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital B (Prior):
Events:  5
Deaths:  0
Days:  22
Charges:  $87,472

Hospital B (Post):
Events:  6
Deaths:  0
Days:  26
Charges:  $100,306

Slide 14

 

EVI from Measurement

Image: A tables displays the following information:

Charges—Total for Hospital A and B			Percent
Sample	EOL	EVI
Prior	$112,221
No POA data	$181,843	$69,622	73.05%
10%	$180,688	-$1,155	-1.21%
20	$171,926	-$8,762	-9.19%
30	$172,108	$182	0.19%
40	$188,143	$16,035	16.82%
50	$190,702	$2,559	2.68%
60	$192,917	$2,215	2.32%
70	$205,541	$12,624	13.25%
80	$198,562	-$6,980	-7.32%
90	$212,499	$13,937	14.62%
Full POA Data	$207,532	-$4,967	-5.21%

 

Slide 15

 

Using Imperfect Measures

• The use of any measure (Q_m) that is positively correlated with the true measure (Q_t) results in overall improvement in outcomes.
• This is not true only if:
  • Difference between Q_t and Q_m
  • Variation in Q_t - Q_m among providers
  • Negative correlation between Q_t and Q_m

Slide 16

 

Using Imperfect Measures

Image: A graph shows infections due to Medical Care (PSI#7) for None (No POA Data) / All (Full POA Data). Most incidences (represented by dots) appear within the range of 0.03 None / 0.03 Full.

Slide 17

 

Using Imperfect Measures

The table shows actual reduction in adverse events in moving patients from worse to best providers based on an imperfect measure.

POA Sample	Number of Adverse Events			Percent
	Best 20%	Worse 20%	Difference
None	283	5,169	4,886	91.7%
10%	275	5,213	4,938	92.7%
20%	267	5,245	4,977	93.4%
30%	270	5,287	5,017	94.2%
40%	259	5,323	5,063	95.1%
50%	261	5,374	5,114	96.0%
60%	244	5,377	5,133	96.4%
70%	240	5,427	5,187	97.4%
80%	229	5,469	5,241	98.4%
90%	216	5,516	5,300	99.5%
All	215	5,542	5,327	100.0%

Slide 18

 

Risk Aversion

• A consumer may rationally select a provider with worse measured quality:
  • If the consumer is risk averse and prefers a more certain outcome over the greater risk of a worse outcome.
  • If the consumer has external information about the provider that reduces uncertainty.

Slide 19

 

Risk Aversion

• A consumer may rationally select a provider with worse measured quality:
  • If the consumer is risk averse and prefers a more certain outcome over the greater risk of a worse outcome.
  • If the consumer has other information about the provider that reduces uncertainty.

Slide 20

 

Risk Aversion

Image: Line graph shows Iatrogenic Pneumothorax (PSI #06) rates for Hospital B EOL vs. Hospital A EOL. Hospital A has a sharp spike, while Hospital B is an even bell curve.

Slide 21

 

Conclusions

• Use expected value of information (EVI) to quantify the benefit of adding enhanced data, collecting data over time.
• Largest marginal gain in EVI may come from:
  • Initial measurement
  • Systemic relationships
• Smallest marginal gain in EVI may come
  • Subsequent measurement
  • Provider specific measurement

Slide 22

 

Conclusions

• Target data collection to the measures, consumers and providers with highest marginal EVI.
• Give consumers information about uncertainty to allow them to incorporate preferences about risk and external information.

Slide 23

 

Questions?

Image: An illustration of a microphone is shown.