Model Public Report Elements: A Sampler

B. Presentation of Measure Ratings

An effective presentation of measure ratings is based on the following principles:

• Consumers do not define or understand quality in the same way that it is typically measured and reported.
• A framework can help consumers understand the larger concept of quality, as well as better comprehend the individual quality indicators.
• A framework may enable consumers to more effectively participate in the quality discussion.
• If a consistent framework is used in all reports, consumers will expect to see performance ratings in all categories of the framework.

A few existing quality frameworks might be considered as an organizing concept for individual measures within a quality report:

• The Institute of Medicine (IOM) developed a framework consisting of six categories: Effective, safe, patient centered, timely, equitable, and efficient.¹
• The IOM also developed these three categories: overuse, underuse, and misuse.²
• Donabedian developed a framework consisting of three categories: structure, process, and outcome.³

A framework to communicate about health care quality should:

• Indicate clearly what quality of care is.
• Contain no more than three or four categories.
• Be consistent with how quality is measured and conceptualized within the policy and industry arenas (that is, the major categories that are used to define quality by experts and the industry should be reflected in some way in the framework provided to consumers).

The following examples use a framework in communicating what quality of care means.

Tool: Health Matters

Sponsor: Colorado Business Group on Health

URL:  http://www.cbghealth.org/cbgh/index.cfm?LinkServID=E0FEA580-D721-5B0E-DA115CC5582742B9&showMeta=0 PDF File, page 2; Plugin Software Help

Tool: Community Checkup Report

Sponsor: Puget Sound Health Alliance

URL: http://www.wacommunitycheckup.org/editable/files/CommunityCheckup_Nov2008/FAQs_AboutQualityHealthCare_Nov2008.pdf (PDF File; Plugin Software Help)

This next example, which is not from an existing Web report, shows an abbreviated IOM framework for communicating what quality is, which has the advantages of using only three categories and having undergone cognitive testing. With this kind of framework, quality indicators can be displayed within each of the three categories of performance. This approach can make each quality indicator easier to understand while also conveying that "quality" incorporates three different but important dimensions.

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway?: Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

As noted above, the most effective way to use a framework is to actually show the quality indicators arrayed under the main categories of the framework, as in the example below. The framework categories tell the user what the individual indicators mean.

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway? Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

How does one select quality indicators for each element of a framework? The decision to use a framework implies making some early choices about the types of performance measures to include in your report. The table below shows examples of indicators to include if using the Structure, Process, and Outcome framework; the IOM framework of six domains; or the Hibbard-abbreviated IOM framework.

Reference: Romano PS, Hussey P, Ritley D. Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives. Rockville, MD: Agency for Healthcare Research and Quality; May 2010. AHRQ Publication No. 09(10)-0073.

Matrix of Quality Measure Typologies With Examples

1. Displaying measures so that consumers can understand them

Using comparative data to make a choice can be cognitively burdensome. People are more likely to use data if the information is easy to understand and process. Making it easier actually increases the individual's motivation to use the data. There are multiple ways to make the data more evaluable (that is, easier for the user to quickly see better and worse options without much effort). 

Strategies to make comparative data more evaluable and less cognitively burdensome include:

• Use data display approaches that do not require users to hold information in their mind as they look from page to page or at different points on the same page:
  • Make relevant choices visible on one screen.
  • Use symbols instead of numbers (many people have limited numeric skills).
  • Use symbols that do not require a legend to understand what they mean.
  • Use language and terms that consumers understand (avoid language that requires the user to look up the definition).
  • Do not assume people will understand advanced statistical concepts, such as using confidence intervals in data displays.  Consumers do not understand them, and they introduce ambiguity that can result in consumers being less likely to rely on or trust the data.
• Do some of the cognitive work for the user:
  • Summarize information. Providing an overall summary measure of all the other measures helps the end users by doing some of the work for them.
  • Order by performance (or have a function that allows this).  This makes it easy to quickly see top and bottom performers.
  • Use affective labels (e.g., excellent, good, fair, poor) that interpret the information for the user.
  • Highlight high performers.

Here is an example of using symbols instead of numbers. These symbols use color and have words embedded in the symbol to make them easy to use without consulting a legend.

Tool: CalHospitalCompare

Sponsors: California Hospital Assessment and Reporting Taskforce, California HealthCare Foundation, and University of California, San Francisco

URL: http://www.calhospitalcompare.org/comparison.aspx?c=&h=210&h=218&h=155&h=146&h=256  

Here is another example that uses symbols instead of numbers. In this example, the symbols use a color, a shape, and a word embedded in the symbol.  Average scores are faded so that the higher and lower performers "pop" out more. Thus, users can discern a pattern in the data.  The use of colors and shapes helps the user easily see patterns in the data. The embedded words in the symbols mean that the user does not need to rely as much on a legend. The embedded words also help the user by interpreting the information for them (e.g., better, worse). 

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway?  Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

The next example shows a report format that helps the user by providing two overall summary measures: one based on clinical standards and the other based on patient ratings.  This approach greatly reduces the cognitive burden of using multiple data points to compare multiple options.

Tool: Medical Group Ratings

Source: California Office of the Patient Advocate

URL: http://www.opa.ca.gov/report_card/medicalgrouprating.aspx?County=KERN  

Here is an example of a report that uses symbols that have inherent meaning.  Words are embedded in the symbol to make it less necessary to use a legend.

Tool: Partner for Quality Care—Quality of Care Ratings

Sponsor: Oregon Health Care Quality Corp

URL: http://www.partnerforqualitycare.org/selection.php  

The next example uses a summary measure and orders hospitals by performance, with highest quality providers listed first.  Ordering is a powerful way of helping the user quickly discern better and worse options.

Tool: Partner for Quality Care—Quality of Care Ratings

Sponsor: Oregon Health Care Quality Corp

URL: http://www.partnerforqualitycare.org/care.php?region_name=portland_metro  

2. Using consumer-friendly language for measures

Using everyday language and terms familiar to consumers is key to making public reports understandable and usable. Using technical terms and expecting consumers to look them up (even if the definition is only a click away) will discourage use. Writing at a 6^th grade level will make the information accessible to a wider audience. 

Here are some examples of translating technical labels into plain language.

Source: AHRQ Model Quality Reports (available at http://www.qualityindicators.ahrq.gov/Downloads/Modules_Non_Software/QI%20Reporting/Model%20Report%20Health%20Topics.pdf; Plugin Software Help)

In the next example, the original "untranslated" technical version of the ambulatory care quality performance indicators is shown, as well as the plain language translation of those technical labels.

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway? Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

3. Displaying resource use measures

Consumers have had very little access to comparative information on costs and resource use, and until recently, we have known very little about how to present this information effectively. A 2011 AHRQ-funded study led by Judith Hibbard of the University of Oregon and Shoshanna Sofaer of Baruch College used focus groups, cognitive tests, and a randomized laboratory study to identify:

• Cost/resource use measures that resonated with the public.
• Displays of such measures that were most accurately understood, led to "high value" choices, and gave people the most confidence in their choices.

An overarching recommendation of this study is to present cost information in displays that include a strong quality signal (easy to understand and use quality information). In addition, displays work better if they help consumers understand the meaning of the measures by using labels and other strategies that interpret the information for consumers. We present three sample displays with specific recommended features.

Presenting Comparative Costs for Doctor's Office Visit

Recommended features:

• Present cost data in the same display as quality data.
• Present exact cost information.
• Use a "word icon" for quality, which is highly "evaluable" and thus represents a strong quality signal.

Presenting Comparative Hospital Cost, Quality, and Value Information

Recommended features:

• Combine information on quality, cost, and value in the same display.
• Call out value using a familiar signal (checkmark), which is present only for those with high quality and low cost.
• Use a "word icon" for quality, which is highly "evaluable" and thus represents a strong quality signal.
• Use dollar signs to represent comparative costs.

Presenting Comparative Information About Resource Use (Imaging)

Recommended features:

• Use labels and symbols to characterize use of imaging, rather than giving numeric levels.
• Use labels that interpret numeric imaging scores so that consumers do not need to determine for themselves which provider is providing more appropriate care.
• Whenever possible, merge resource use and quality into one highly interpretable score.

4. Encouraging providers to report accurate and complete data

In some cases, significant effort is required by providers to collect, clean, and submit the required data. In these instances, collaboratives may choose to recognize that effort specifically.

In the underlined text and blue box below, Colorado Business Group on Health helps the visitor recognize the commitment made by participating hospitals.

Tool: Health Matters 2009

Sponsor: Colorado Business Group on Health

URL: http://www.cbghealth.org/cbgh/index.cfm?LinkServID=E049392D-D33F-F20A-D2ED70858A72E914&showMeta=0 (PDF File; Plugin Software Help)

In most cases, those providers who choose to participate in public reporting will have to invest significant resources just in collecting and sharing the data and making sure it is accurate; they will have made a significant commitment to transparency. However, if the program is voluntary, other providers may elect not to provide the data. To recognize the additional contributions of those providers who share accurate data about their performance, some community collaboratives have instituted penalties for nonparticipation or incomplete or inaccurate data submission. 

The following examples illustrate ways to denote or call attention to a lack of transparency, effort, or completeness. The report below specifically notes the facility's refusal to sign off on the accuracy of the data submitted.

Tool: Hospital Performance Report

Sponsor:  State of New Jersey Department of Health and Senior Services

URL:  Sample screen shown no longer available online. 

This report describes levels of participation both in terms of the data submitted and actual level of performance.

Tool:  Clinical Outcomes Assessment Program

Sponsor: Foundation for Health Care Quality

URL:  Sample screen shown no longer available online.

In this report, "NR" (measure not reported) is distinguished from nonparticipation as a whole.

Tool: Focus on Hospitals

Sponsor:  Missouri Hospital Association

URL: No longer available online.

In this report, a notation is made when data are presented for only some patients with a given condition, but not all.

Tool: New York State Hospital Profile

Sponsor:  New York State Department of Health

URL: http://hospitals.nyhealth.gov/index.php  

5. Noting in a report that "sample sizes are too small"

Sometimes a provider has too few patients with a particular condition or undergoing a specific procedure to calculate a quality measure. In such cases, a performance label should not be assigned. It is important, however, to let the consumer know that the lack of a measure or performance label does not reflect poorly on the provider's quality of care.

Some sites simply indicate that there are "too few cases."

In the example below, the site provides a warning to the reader, "* Too few patients (<160) with condition of interest to meaningfully report for this clinic."

Tool: Health Alliance Community Checkup

Sponsor: Puget Sound Health Alliance

URL:  http://www.wacommunitycheckup.org/index.php?p=viewreports&orgname=all&county=All+Counties  

6. Using consumer-friendly phrases to replace clinical and technical terms

It is better to use consumer-friendly language (that has undergone cognitive testing), rather than technical terms even if they are linked to a glossary.

The following glossaries may serve as a starting place for report sponsors who seek lay definitions of technical terms. Use of terms that are found in one or more glossaries is not a substitute for cognitive testing.

• http://www.wchq.org/utility/glossary.php
• http://mhcc.maryland.gov/consumerinfo/hospitalguide/patients/consumer_help/glossary.html
• http://info.kyha.com/qualitydata/FAQ.htm#Glossary
• http://web.doh.state.nj.us/apps2/hpr/mortalityfacts.shtml.
• http://www.mihospitalinform.org/Definition.aspx.