Overview of the Decision Guide: A Public Reporting Resource for CVEs (Text Version)

On September 27, 2010, Patrick S. Romano made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (13.5 MB). Free PowerPoint® Viewer (Plugin Software Help).

Slide 1

Overview of the Decision Guide: A Public Reporting Resource for CVEs

Patrick S. Romano, MD MPH
Professor of Medicine and Pediatrics
University of California, Davis
June 23, 2010

Slide 2

Available from AHRQ

Image: The cover of "Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives" is shown.

Authors:
Patrick S. Romano, MD MPH
Dominique Ritley, MPH
David Chin, PhD student

With the help of many CVE representatives

Slide 3

Decision Guide Checklist

• Goals:
  • To provide collaboratives with frameworks and tools for selecting measures of quality and resource use (QI 101).
  • To highlight key considerations in selecting measures of quality and resource use, based on a collaborative's evolutionary stage.
• Guide divides answers to 26 questions into five sections:
  • Introduction to Data
  • Introduction to Measures of Quality
  • Introduction to Resource Use/Efficiency Measures
  • Selecting Quality and Resource Use Measures
  • Interpreting Quality and Resource Use Measures
• We focus today on just 6 of those questions

Slide 4

Design of public reporting programs starts with a candid self-assessment (Q25)

Image: Venn diagram showing the intersection of the following three items:

Resources:

• Financial resources
• Analytic capabilities

Environment:

• Available data
• Potential partners
• Stakeholder engagement

Goals:

• P4P
• Public reporting
• Performance improvement

Slide 5

Quality measurement and reporting are important issues across many industries

Image: An egg farm is shown.

Slide 6

In search of a balanced set of quality measures (Q20): Iowa's inspection of Wright County Egg

Image: An Iowa Department of Inspections & Appeals Egg Handler Inspection sheet is shown.

Slide 7

In search of a balanced set of quality measures (Q20): USDA "grader" inspection of shell egg plant

Image: A grader checklist is shown.

Slide 8

What did the USDA miss?

Image: A building with a wall broken is shown.

Slide 9

What else did the USDA miss?

Image: A building wall is shown.

Slide 10

IOM Domains of Quality

Effectiveness

• Providing services based on scientific knowledge (avoiding overuse of inappropriate care, underuse of appropriate care)

Patient Centeredness

• Care that is respectful of and responsive to patient preferences, needs, and values

Timeliness

• Reducing wait times and sometimes harmful delays

Safety

• Avoiding injuries to patients from care that is intended to help

Efficiency

• Avoiding waste of equipment, supplies, ideas, and energy

Equity

• Care does not vary in quality because of personal characteristics

Slide 11

In search of a balanced set of quality measures (Q20): Institute of Medicine, 2010

Image: IOM's quality measures matrix is shown:

Slide 12

What types of measures should be collected and reported (HOW)?

Two images of old farm buildings are shown.

Slide 13

Look inside the structure.

Image: The inside of a farm building is shown.

Slide 14

Look for the outcomes.

Image: A dead chicken is shown.

Slide 15

Classifying types of measures
Donabedian, 2003

• Structure: conditions under which care is provided:
  • Material resources (facilities, equipment)
  • Human resources (ratios, qualifications, experience)
  • Organizational characteristics (size, volume, IT system)
• Process: activities that constitute health care (adherence to guidelines):
  • Screening and diagnosis
  • Treatment and rehabilitation
  • Education and prevention
• Outcome: changes attributable to health care:
  • Mortality, morbidity (complications, readmissions)
  • Knowledge, attitudes, and behaviors
  • Patient experiences/satisfaction

Slide 16

Framework for selecting measures (Q20)

Slide 17

What data sources should we use for quality measurement (Q8/Q9)?

Image: The inside of a chicken farm is shown.

Slide 18

Gold standard = Direct observation

Video recording to identify errors in pediatric trauma resuscitation:

Mean of 5.9 errors per resuscitation, 93% agreement between reviewers.

Mean of 2.2 errors in each seriously injured child, with 20% capture on medical records.

Image: Emergency medical staff are shown treating a child.

Oakley, E. et al. Pediatrics 2006;117:658-664

Slide 19

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

Image: The Hillandale Farms FDA inspection results form is shown.

Slide 20

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

• Observe/record encounters (real or simulated)—$$$$
• Ask patients (CAHPS® surveys):
  • Satisfaction and experiences
  • Morbidity (complications, functional status, quality-of-life)
  • Knowledge, attitudes, and behaviors
• Ask health care providers:
  • Rate others' reputation (US News)
  • Describe material and human resources (Leapfrog survey)
  • Describe safety-related practices (Leapfrog survey)
• Review claims/administrative data sets:
  • Mortality, morbidity (deaths, complications, readmissions)
  • Adherence to guidelines (HEDIS, PQRI)
• Review/abstract medical records (including registries):
  • Mortality, morbidity (deaths, complications, readmissions)
  • Adherence to guidelines (HEDIS, PQRI)

Slide 21

Tremendous growth in NQF-endorsed physician measures (Q8)

• National Voluntary Consensus Standards: Ambulatory Care:
  • 101 measures across 10 priority areas: asthma and respiratory illness; bone and joint conditions; diabetes; heart disease; hypertension; medication management; mental health; obesity; prenatal care; and prevention (including screening).
  • 7 instruments for patient experience.
  • 26 measures of specialty clinician care: bone and joint conditions, eye care, geriatrics, emergency care, skin care (melanoma).
• "Additional Performance Measures 2008":
  • 67 measures for cancer care, infectious disease, perioperative care, and licensed independent practitioners.
• "Using Clinically Enriched Administrative Data":
  • 70 measures across most original priority areas plus child health, chronic kidney disease, gastroesophageal reflux, gynecology, hepatitis, HIV/AIDS, and migraine.

Slide 22

Hospital quality measures are now available "off the shelf" (Q9)

• CMS Medicare's HospitalCompare
• The Joint Commission's QualityCheck
• Commonwealth's "Why not the best?"
• Leapfrog's voluntary survey on CPOE, ICU staffing, evidence-based hospital referral, and NQF Safe Practices implementation
• Specialized state/regional programs (HAIs, AHRQ Quality Indicators, myhealthfinder.com, registries)
• Other (HealthGrades, USNews, etc.)

Slide 23

Image: A screen shot of the "Hospital Compare" Web site is shown.

Slide 24

Image: A screen shot of the "The Joint Commission" Web site is shown.

Slide 25

Image: A screen shot of the "The Commonwealth Fund" Web site is shown.

Slide 26

Image: A screen shot of the "The Leap Frog Group" Web site is shown.

Slide 27

Image: A screen shot of the myHealtheFinder.com Web site is shown.

Slide 28

Hybrid data (Q2)

• Bring together administrative (electronic claims) and medical record data to build on the strengths of each while compensating for weaknesses:
  • Increase the number of data elements for outcome ascertainment or risk adjustment.
  • Reduce the number of records that must be reviewed manually.
  • Reduce the time required to review each record.
• At the physician level, use claims to identify patients with a relevant diagnosis or problem, and use medical records to identify specific clinical findings or treatments.
• At the hospital level, combine ICD-9-CM coded administrative data with laboratory or other clinical data to enhance the performance of risk-adjustment models and to reduce bias in estimates of hospital performance.
  • FL, MN, VA pilot projects
  • Regulatory requirements in CA and PA (Michael Pine's work)

Slide 29

How to select measures for reporting? National Quality Forum criteria (Q22)

• Importance:
  • Leverage point for improving quality.
  • Variation in quality of care or suboptimal performance (overall).
• Scientific acceptability:
  • Well-defined and precisely specified... reliable.
  • Valid, accurately representing the concept being evaluated.
  • Discriminating between real differences in provider performance.
  • Adaptable to patient preferences and variety of settings.
  • Adequate and specified risk-adjustment strategy.
• Usability:
  • Can be used for decision making and implementing change.
  • Differences should be meaningful practically and clinically.
• Feasibility:
  • Benefit should be evaluated against burden.
  • Confidentiality concerns should be addressed.
  • Audit strategy should be available.

Slide 30

Who needs composite scores? (Q10)

Image: the New York City Health Department new restaurant inspection grades notice is shown.

Slide 31

Trends in scores over time in LA County

"A" grade was associated with 5.7% increase in revenue.
"B" grade was associated with 0.7% increase in revenue.
"C" grade was associated with 1% decrease in revenue.
Two studies showed 20% and 13% decreases in hospitalization for food-borne illness in Los Angeles County.

Source: CHOICES 2005; 20(2):97-102 (American Agricultural Economics Association)

Slide 32

Why composite measures for CVEs? (Q10)
(aka summary measures, roll-up measures)

• AHRQ: "condensing multiple quality measures into a single piece of information":
  • Reduces cognitive burden for consumers, providing clearer "signal" and reducing the danger of "cognitive shortcuts".
  • Enhances reliability or ability to discriminate between higher-quality and lower-quality providers.
  • Fits well conceptually with pay-for-performance programs, which explicitly translate multiple quality measures to dollars, allowing providers to prioritize their own efforts.
• But remember two potential concerns:
  • Difficulty achieving consensus on composite construction and scoring, perhaps due to lack of professional consensus.
  • Loss of important information if the composite combines unrelated metrics in a manner that washes out meaningful differences on individual indicators.

Slide 33

Two conceptual approaches (Q10)

• Psychometric or reflective perspective—an underlying, unmeasured factor ("quality") is the cause of what we observe; the observed data reflect this unmeasured factor:
  • Requires a correlation among the measures included in the composite, because different measures can only reflect the same latent factor (i.e., quality) if they are correlated with each other.
• Clinometric or formative perspective—focus on guiding decision-making to optimize welfare instead of measuring an unobserved, latent factor:
  • Use clinical judgment rather than empirical analysis to select component measures.
  • Formed from or defined by specific indicators, so no correlation among component measures is required

Slide 34

Recommended approach for creating a composite score (Q10)

• Identify the purpose... and delineate the quality construct...
• Select the individual measures and/or sub-composite measures to be combined... (may require standardization).
• Ensure that the weighting and scoring of the components supports the goal that is articulated for the measure.
• Combine the component scores, using a specified scoring method...
• Testing for reliability and validity.

Slide 35

Restaurateurs' reaction to the NYC composite score

• Purpose of composite is invalid:
  • "There is no evidence that letter grading increases the identification of risk factors for foodborne illnesses".
  • "...sophomoric, and punitive and demeaning to restaurateurs, as if they are schoolchildren who must be graded".
• Composite is poorly constructed:
  • "How can you possibly justify including non-food safety related items? A leaky faucet, a (missing) sign, a light bulb not covered, an uncovered waste receptacle... mislead the public when it sees a B or C in the window into thinking that the food here is not safe, when the difference between an A or B grade may have nothing to do with food safety."
• Composite is unreliable:
  • "...inconsistency from one inspector to another"
• Composite is invalid due to unmeasured risk:
  • "Most of their buildings (in LA) are not 200 years old, and most of them are not next to empty lots with hundreds of rats. It would be nice if the city would clean up those lots."
• Composite will have unintended consequences (gaming):
  • "...a scarlet letter that will keep people from eating out".
  • "...encourage bribery and corruption. I remember when payoffs were so commonplace that the FBI had to come in and arrest the inspectors.".
  • "...could turn back the clock on New York as the food capital of the world."

Source: New York Times, multiple articles, February-August 2010

Slide 36

New York City's "dirtiest establishment" (now closed): Was it rats or roaches?

Image: A grocery store is shown.

Slide 37

Scoring composite measures (Q10)

Slide 38

Combining quality and resource use measures to highlight high-value care (Q26)

Image: A chart displaying AMI Composite Score vs. Severity Adjusted Charge is shown.

Slide 39

Conclusion: Use available tools from AHRQ!

• CVE Learning Network
  http://www.cvelearningnetwork.org/default.asp
• AHRQ Talking Quality
  https://talkingqualityahrq.gov
• AHRQ Health Care Report Card Compendium
• AHRQ's National Quality Measures Clearinghouse
• My Own Network, powered by AHRQ
  http://www.monahrq.ahrq.gov/
• RWJF's Aligning Forces for Quality
  http://www.forces4quality.org/welcome

Slide 40

AHRQ Decision Guide on Selecting Quality and Resource Use Measures

• Addresses 26 questions community leaders and stakeholders frequently ask about quality and resource use measurement.
• Community quality collaborative leaders informed development.
• Access on-line at:
  https://www.ahrq.gov/qual/perfmeasguide or to order hard copies free of charge:
  • Send an E-mail to AHRQPubs@ahrq.hhs.gov
  • Specify number of copies.
  • Include AHRQ Pub. No. 09(10)-0073