You Are Here: AHRQ Archive Home > Oregon Consumer Scorecard Project: Executive Summary

Oregon Consumer Scorecard Project

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Executive Summary

Following is a summary of the Final Report of the Oregon Consumer Scorecard Project, which was funded by the Agency of Health Care Policy and Research (AHCPR) in January 1995 under a contract award to the Oregon Health Policy Institute (OHPI) through the University of Washington's Center for Rural Health Services Research. The purpose of this contract was to provide support to the Oregon Consumer Scorecard Consortium development of a consumer-oriented health plan scorecard model. The AHCPR/OHPI Oregon Consumer Scorecard Project (OCS Project) involved developmental activities and pilot testing of stimulus materials that would subsequently inform the design of a prototype consumer scorecard to be tested throughout the State of Oregon. The OCS Project paid particular attention to the health plan information needs of consumers, as opposed to volume purchasers, as well as to the unique information needs of rural consumers and persons with significant chronic health conditions and disabilities.

Overview
Project Activities
Information Review and Synthesis
Consumer Satisfaction Survey
Performance Measures
Navigational Information
Focus Groups
Project Findings and Challenges
Lessons Learned
Project Staff

Overview

The Oregon Consumer Scorecard Consortium represents a partnership between a group of public and private stakeholders in Oregon who are committed to developing a high quality, consumer-oriented health plan scorecard. Through its committee structure, the Consortium has ensured that all key stakeholders who will benefit from or be affected by the scorecard are involved in the developmental process.

Through the joint efforts of many individuals, the Oregon Consumer Scorecard Consortium was formed, established a work plan, and has produced its first set of scorecard materials. The prototype scorecard materials developed are unique in that they integrate the major types of health plan performance information available into a model consumer-oriented guidebook, "A Consumer Guide to Selecting a Health Plan." The guidebook, which uses some health plan data, was presented to Oregon consumers in a series of focus groups for their review and assessment.

The primary function of the scorecard is to aid consumers in choosing a health plan that best meets their individual needs and preferences for how health care services are delivered. Secondarily, if successful, the Oregon Scorecard will serve as a comparative performance measurement tool that provides feedback to health plans, purchasers, and State policy makers about how well health plans are performing and meeting the expectations and needs of their customers.

The scorecard is intended to aid consumer decision making in two primary ways: 1) by providing comparative health plan information that is based on the expressed information preferences of Oregon consumers; and 2) by serving as an educational tool in those areas of health plan performance where consumers lack understanding about how the various plans "manage" health care. The second function presumes that better informed consumers will make more prudent choices based on objective, reliable measures of technical quality, health plan performance, and consumer satisfaction.

The target population for scorecard prototype development was the OHP-Medicaid population, while fully capitated health plans were the unit of comparison. There were several reasons for focusing at the health-plan level. Health plans are what consumers must choose among when they sign up for the Oregon Health Plan. It is health plans that assume financial risk and clinical responsibility for enrollees, thus it is appropriate to design a performance evaluation tool that monitors quality improvement at the plan level. Finally, Oregon statutes have established explicit policy goals that support the development of capitated managed care plans and delivery systems for all Oregon Health Plan programs.

Project Activities

Several sources of information were used to develop the scorecard stimulus materials that were presented to focus group participants. The AHCPR contract supported the information review and synthesis, the conduct of the focus groups, and the staff effort to synthesize and report on all other Consortium-sponsored data collection activities including the analysis of the consumer satisfaction survey, the pilot testing of Medicaid HEDIS (Health Plan Employer Data and Information Set) measures, and the gathering of descriptive health plan information.

Information Review and Synthesis

To achieve the objectives of the AHCPR contract, staff gathered and synthesized existing information from other scorecard projects throughout the country as well as reviewing the pertinent literature that might inform the following four policy questions:

What information do consumers want and need to make informed choices among plans and providers?
What are the most effective formats for conveying information that is useful to consumers in choosing among health plans?
How do the information needs of rural consumers differ, if at all, from those of urban consumers?
How do the information needs of persons with disabilities or chronic conditions differ from those without such conditions when choosing among health plans?

The value derived from the information review and synthesis was significant. It identified what was currently known about consumer preferences for health care information and what areas needed further development and testing. Additionally, it examined models and measures from other projects that could be useful to the Oregon Consumer Scorecard Consortium and provided a road map for the methodological decisions the project staff faced. Finally, it was suggestive of ways the OCS Project could contribute to the emerging body of knowledge on consumer information needs and preference modeling. The several policy and methodological issues highlighted as a result of this effort are summarized below.

It would be necessary to reconcile the differences between how professionals and consumers think about quality—both in terms of what constitutes technical quality (that is, clinical competence) and how to measure and report it for comparative purposes.

Beyond technical competence, it was clear from our review of the literature that a whole range of clinician attributes are of great importance to consumers, including compassion, communication skills, and respectful listening—each of which ranks high on most consumers' lists of what they look for when selecting a health care provider.

In spite of the wide use of population-based performance measures, such as HEDIS, by both health plans and major purchasers, consumers find most of these condition-specific indicators to be of limited value when making a choice among health plans. For some, the problem was a general lack of understanding of what the indicator represented (for example, whether the rate of low birth weight babies delivered is a good or bad outcome); for others, the indicator had no relevance to their individual circumstance (such as mammography rates to unmarried men).

Finding the balance between population-based performance measures and personalized, consumer-relevant information that represents quality from a consumer perspective was, and will continue to be, a major challenge in developing a consumer-oriented scorecard.

The tension resulting from a focus on health plans as the unit of comparison for scorecard purposes would need to be balanced with the reality that most consumers are interested in the attributes and performance of individual clinicians and the facilities in which they practice.

To deal with this tension, project staff and Consortium members would identify and collect descriptive information about the management practices of health plans that varied, where that variation was judged by consumers to influence the overall quality of care they received. These elements included plan differences in the areas of specialist referral mechanisms, grievance procedures, case management and service coordination for persons with chronic health conditions and disabilities, and other customized features of care management.

The OCS Project was charged with tailoring scorecard choice information to the needs of individual health care consumers, paying particular attention to consumers with special or extraordinary health care needs. Both in terms of measurement and display, tailoring information to special needs consumers was a particularly challenging activity that had not been attempted by other scorecard efforts around the country.

From a measurement perspective, most significant chronic conditions and disabilities are relatively low incidence occurrences within the general population. Even if an efficient way to provide condition or disability-specific information were developed, measuring satisfaction and clinical outcomes within these populations by health plan would be problematic because of the small numbers involved.

The "small numbers" problem would need to be dealt with to provide relevant comparative health plan information to rural consumers.

In many rural areas of the State the population density is low. In these sparsely populated areas, there are small numbers of individuals enrolled in a few health plans, often two or less. To provide comparative consumer satisfaction and health plan performance information that reflects a local health plan experience would require a geographically sensitive sampling strategy for the consumer satisfaction survey. Additionally, there would be special challenges in collecting health plan performance data because of the small denominators involved (the total population from which a measurement is derived).

Consumer Satisfaction Survey

A primary sponsor of the Oregon Consumer Scorecard Consortium, the Office of Medical Assistance Programs (OMAP), is the State Medicaid agency responsible for administering the Oregon Health Plan demonstration. As part of its ongoing quality assurance function, OMAP is required to conduct an annual consumer satisfaction survey of OHP enrollees. This year OMAP designed the survey in consultation with OCS Project staff and the Consortium Technical Committee. The sampling strategy was designed to yield at least 400 responders per health plan to ensure a 95-percent confidence level, thus allowing sufficient numbers to report comparative results by plan.

OMAP provided the OCS Project staff with the raw survey data since a major component of the scorecard stimulus materials was the presentation of comparative consumer satisfaction data. All satisfaction scores presented on the stimulus materials were based on actual data, but presented with fictitious health plan names.

Performance Measures

How health plans monitor, measure, and report population-based performance measures to purchasers has been found to have limited utility to consumers. The ideal measurement of technical quality would involve reporting the outcomes of health care treatments in a language and format that is understandable to a lay audience. However, because of the length of time involved before many clinical outcomes can be measured, and the methodological difficulties involved in outcomes measurement, process measures have been alternatively developed and marketed. These process measures address the technical components of health care delivery and are assumed to reflect efficiency of service delivery and, coincidentally, competency of care delivered. The most widely used health plan performance measurement tool is HEDIS. Included among the most commonly used HEDIS measures are such process indicators as glycohemoglobin screening for diabetics (monitoring how a particular illness is managed) and immunization rates for 2-year-olds (a prevention-oriented measure).

For the educational goal of scorecard development, the OCS Project wanted to include several HEDIS measures on the stimulus materials to probe for their potential utility when presented in various formats. An essential element of the Consortium work plan was to test the feasibility of health plans using common data definitions, standardized measures, and uniform data collection protocols for a subset of Medicaid HEDIS measures since these are the standard in the industry. Through this pilot test it was agreed that a more reliable and valid scorecard could be developed downstream. The pilot nature of the activity offered an opportunity for collaboration in performance monitoring that took into account the needs of all stakeholders.

A primary reason for the pilot was to assess the difficulties and resources associated with collecting and reporting reliable and uniform data. Equally important was to determine how to interpret the data once it has been collected. Unless HEDIS data are collected in a uniform way, there is no way of assessing the methodological challenges involved in both their collection and interpretation.

Navigational Information

A consistent sentiment expressed by consumers was the inadequacy of available health plan-specific information both prior to plan selection and also once enrolled. This theme was particularly strong among consumers with chronic health conditions and disabilities. Plan level "navigational" information was perceived to be inadequate for understanding how a health plan really works—that is, how referrals to specialists occur, how primary care keepers really function, and what happens when disagreements arise with a health care provider or access to a given service.

Responding to this consumer desire for objective, comparative descriptive information on a scorecard was not without its challenges. As straightforward as this type of information might seem, collecting it in a "marketing neutral" fashion was not an easy task. The difficulties were further compounded by the fact that the OHP-Medicaid program offers a uniform benefit package and therefore many service differences relate to organizational characteristics. For example, service delivery may differ based on whether a plan is a closed-panel health maintenance organization (HMO) or a more loosely integrated independent practice association (IPA).

Focus Groups

Through a subcontract to Oregon Health Decisions, Inc., the OCS Project conducted two rounds of focus groups with OHP-Medicaid consumers. Each round consisted of four focus group sessions. The first round sought to understand consumers expressed health plan information needs and preferences for choice decision making purposes. Round II participants were provided a model guidebook and a set of alternative formats and media presentations and were asked to use and critique the materials presented.

Because previous focus group research on consumer information needs had been conducted in urban settings, and involved probing the health care information needs of a general population, the first round of focus groups was targeted at consumers living in rural areas and persons with chronic health care conditions and disabilities. For Round II, the criteria for site selection was whether participating consumers faced a real choice, that is, whether there were two or more competing health plans available in their county of residence.

Findings and Recommendations: Round I

The most appropriate and worthwhile function of a consumer scorecard would be to support the autonomy and self-determination of individual consumers in making choices among health plans. Participants also suggested a scorecard could be a means for improving the quality and efficiency of plans through critical "customer feedback."
The idea of a scorecard only makes sense where there are real choices—this is especially true when factoring in geographic, social, cultural, or other special needs considerations.
Getting information from an inanimate source has serious limitations, even when that source is interactive, such as a computer kiosk. A scorecard should be backed up by an advice counselor, telephone access, or some other mechanism that links the enrollee to a live person.
Specific information that should be included on a consumer scorecard was described hierarchically; that is, there was an overriding need to know which plan ones current physician belongs to, followed by which plan has the highest probability of providing access to a dentist.
Other descriptive or navigational information cited as important was numerous and varied, for example: which providers (clinics and hospitals) participate in which plans; how plans actually work (finding a provider, referrals to specialists, appeals, etc.); how specialty care for acute and chronic conditions is accessed; and how emergencies are handled.
Having information about the technical quality of care delivered by the health plans' affiliated providers was considered important, but most consumers did not feel they had the knowledge to judge clinical competence and therefore would defer to unbiased, expert opinion.
Having comparative information about how health plan enrollees evaluate their plan and its affiliated providers was deemed important, but it should reflect real differences and be reported in a way that reflects "people like me." For example, a person with a chronic illness who is a heavy user of the system is less interested in how healthy, nonusers rate their plans performance.

During the second round of focus groups, participants used a packet of stimulus materials mocked up as "A Consumer Guide to Selecting a Health Plan" to guide them through a hypothetical decisionmaking process. In addition to the core packet, they were shown alternative formats for displaying the same information, such as tables, graphs, and symbols displaying quantitative information and narrative versus yes/no formats for descriptive information. Additionally, alternative media were introduced, including a video, computer kiosk, and a programmed telephone counseling line. The graphics used to present the stimulus materials (e.g., charts, tables, symbols, etc.) are not presented in this electronic summary; however, they are contained in the printed publication available from the National Technical Information Service.

Findings and Recommendations: Round II

The concept of individual preferences was continually reinforced, a poignant reminder of the challenges involved in developing a scorecard that is both individually tailored and has broad-based application.
Participants felt that the "Guide" was an effective choice tool and a significant improvement over the choice materials they currently use.
There was a strong recommendation to control the amount of information presented. It was simultaneously noted that there was too much information and yet not enough detail. This finding highlights the fact that information needs and preferences are highly idiosyncratic. Consumers want to be able to "mine down" into topics they consider to be personally relevant, while skimming the more generic information.
Although there was a general feeling that the information was comprehensive and organized in a logical fashion, some participants felt that the data presented were not explained thoroughly enough. Too much data is confusing—whether graphs, charts, tables, or simple percentages.
A consistent theme that emerged was the need to have a live person available, either by phone, at an orientation meeting, or located elsewhere. This was a particularly important point for persons with low literacy levels.
There were varied reactions to the alternative media presentations, depending largely on personal preference and learning style. The computer kiosk was generally found to be a user friendly and efficient way to present large amounts of information because it could be tailored to individual preferences for level of detail.
The video was seen as an effective and engaging introduction, a "big picture" guide to help people understand managed care and how to make a plan choice. The video had the added advantage of being able to be rewound at any point. One drawback noted was that it was not interactive and therefore could not substitute for a live person.
Participants generally thought that the computer kiosk would help them in making decisions, especially through the ability to customize the level of detail provided. Concerns were raised about its overall availability and cost; also, whether the technology was fully accessible, especially for people with mobility or sensory limitations.
To the extent feasible, participants recommended combining presentation formats and media. It was felt that having multiple options from which to choose would maximize the utility of the available comparative information.
People wanted to see real differences between plans, and how these differences are visually presented is where individual preferences and learning styles surfaced. Focus group participants were split between a preference for numbers, symbols, and graphs as well as whether they preferred narrative or simple "yes" or "no" statements about comparative plan features.

Project Findings and Challenges

The data collection and reporting activities undertaken by the Consortium were designed to test the feasibility of collecting, interpreting, and publishing health plan performance information of interest to Oregon consumers. Additionally, they were to provide insight into the feasibility of reporting comparative health plan information by geographic area and health/functional status. The results were encouraging. Consumers were highly receptive to the materials presented and eager for objective choice materials. The challenges of production were many and are summarized below:

Presentation of data. A major challenge confronting the project was how to present comparative health plan data—both satisfaction and performance, that was meaningful, understandable, and reliable. The popular media is in a frenzy in its critique of managed health care and capitation forms of health care reimbursement. The resulting public mood is one of heightened interest in the subject, but sparse objective information is available on which to judge the adequacy of the health care options in the marketplace. This politically charged environment created a special challenge for the OCS Project to produce comparative information that could stand up to the scrutiny and rigor of legitimate research and analytical methods.

Both consumers and health plans made a strong recommendation to have a scorecard that presented real and meaningful differences between health plans. The concern that differences might be statistically significant but not clinically different continues to be a thorny methodological and political issue.

Differences between how professionals and consumers think about quality. Population-based performance measures currently have minimal relevance to most consumers, yet their potential to shape quality of care and hold health plans accountable is great. This is particularly true in areas of quality monitoring where measures have yet to be developed; for example, the management of secondary conditions among persons with disabilities, outcomes of cancer treatment over time, and growth and development indicators for children in managed care plans.

Finding the balance between professional or expert judgments of health plan performance that are based on reliable clinical standards, and the individualized consumer information preferences identified in focus groups, will continue to challenge future activities of the Consortium.

Tailoring information to the individual needs of consumers. Consumers expressed a strong preference for choice information that was personally relevant, such as information about diabetic care management for persons with diabetes. Equally significant was the issue of reporting comparative information that is geographically sensitive. Rural consumers want to know about the performance of plans doing business in their area, even if the choice is limited—what happens in Portland is not of particular interest to someone living in Enterprise. This issue will challenge the development and reporting of both satisfaction and population-based performance measures in the future.

Costs of providing uniform health plan performance data. Currently, health plans are producing clinical performance data for a variety of audiences including the National Committee for Quality Assurance (NCQA) for accreditation, the state Professional Review Organization (PRO) for quality reviews, internal quality improvement activities, and purchaser requirements (both public and private). Requests for data and quality monitoring will only increase as time passes and our collective ability to measure outcomes increases. At the present time, most of these competing data requests use distinctly different data specifications. A key objective of the Scorecard Consortium is to consolidate and reduce the data burden on health plans through the development of uniform data specifications and sharing of health care information where appropriate to meet public health goals.

Adjusting satisfaction scores. It has been well established in the literature that certain patient characteristics contribute to overall costs and utilization of health services. We also know something about the correlates of satisfaction and that satisfied patients are more active partners in care and treatment regimens, generally resulting in better outcomes. In short, the relationship between satisfaction and outcomes of care is a strong one from both a cost and a quality perspective. Because patient characteristics have been shown to influence many aspects of health care, the OCS Project staff struggled with whether to add them to the satisfaction equation in terms of having demographic variables serve as moderating factors in the reporting of satisfaction scores. Staff analyzed the distribution of a set of respondent demographic characteristics (age, gender, health status, place of residence, and length of time on plan) and found them to be evenly distributed across all health plans, and therefore chose not to use them as control variables. Satisfaction scores were reported for urban and non-urban areas separately as geography was found to exert a significant influence on overall plan ratings for those plans with both urban and rural markets.

Unintended consequences of reporting comparative information. Experience has shown that reporting clinical performance without appropriate severity or risk adjustment can result in misinformation and work against the goal of providing consumers with useful and reliable information on which to judge health care performance. Likewise, reporting health plan differences that are not meaningful can create confusion, give a false sense of difference, and thus penalize or reward health plans inappropriately.

Lessons Learned: Implications for Future Scorecard Development Activities

The OHP-Medicaid population receives a standard benefit package that does not vary from plan to plan, nor are there differences in cost-sharing responsibilities among plans. Because of the uniform nature of the OHP, it is possible to develop a core set of health plan information that is presented to consumers up-front on the scorecard. This type of how to get around in my managed care plan information was consistently voiced as a high priority among focus group participants. Beyond this, scorecard users prefer access to tailored information that is relevant to their individual experience, including how they personally utilize health care services.
Consumers prefer the availability of multiple media presentations when using choice information. Each of the media available (computer kiosks, videos, and telephone back-up) was deemed useful, but a clear message from focus group participants was that written information should always be available because of its lower cost and the ease with which it can be disseminated.
A lot could be accomplished in the production of a consumer scorecard through a voluntary partnership effort. One significant caveat is that in a voluntary effort, the priority given to data collection activities by individual plans is based on their own internal organizational priorities which can ultimately inhibit a statewide effort to produce a reliable and uniform consumer scorecard in a timely fashion.
Although it is easy to assume a health plan scorecard can fulfill many functions, from reporting comparable information to educating consumers about how to be more prudent purchasers and monitor quality, scorecard development is politically and methodologically complex and should proceed in stages with more modest goals.
What the preferred functions of a scorecard should be versus other methods of providing consumers with health plan information needs to be evaluated. Consumers have expressed an overriding need for objective information on which to base choice, judge the quality of care they receive, and better understand how to navigate within a managed care organization. The scorecard may be one of the several vehicles through which to provide consumers with this information. The costs and benefits of various forms of presentation need to be critically evaluated and the message tailored to the medium.
Consumers want more accountability from health plans and the State (or whomever is their health insurance purchaser). They believe that they should be provided with reliable and uniform information that is presented in meaningful ways that display real differences between the health plans available to them. Consumers are savvy about what constitutes marketing and hype and expect something different from a scorecard. A real and meaningful quality feedback loop that extends from the State to the plans to consumers and back to the State should be a top public policy goal.
Given the political nature of presenting comparative health plan information and the potential economic consequences involved, it is critically important to have an objective, disinterested third party responsible for the production of the scorecard. This third party should have the trust and confidence of all vested interests, have the necessary expertise to collect, analyze, and produce comparative information, and be willing to serve as a clearinghouse and negotiator among all interests involved.
There needs to be established uniform standards and data specifications that all health plans and purchasers agree to abide by, and these standards need to be independently "audited" to assure compliance with the data specifications. This audit function does not have to be prohibitively expensive and could be fulfilled by the entity responsible for producing the scorecard.
Because "navigational" descriptive data is deemed so important to consumers, methods needs to be developed to obtain it free of marketing bias so that it can be reported unambiguously, comparatively, and show real differences in how health plans manage care.
The ability to collect Medicaid HEDIS as currently defined needs to be critically examined. The on-again/off-again enrollment patterns of the OHP-Medicaid population work against the current specifications of these measures. The second challenge of HEDIS is its current dubious value to consumers. The Consortium may need to find ways to modify the Medicaid HEDIS data specifications while maintaining the integrity of the individual measures. Additionally, it may want to test creative new ways to collect and report population-based measures that can be made more relevant to individual consumers.
For example, measures other than HEDIS may need to be looked at by which health plans can be held accountable, ones that are more reflective of the health status of Oregon's population. An early activity of the OCS Project staff was to look at Oregon's morbidity and mortality statistics as well as adult risk behaviors. In these sources of data are the basis for accountability measures unique to Oregon.
The Consumer Satisfaction Survey needs to be evaluated in light of this years experience. In spite of its many strengths, which far outnumber its weaknesses, there were several questions that did not yield expected responses and some missed opportunities to better understand the correlates of satisfaction in our Medicaid population. The OCS Project staff may also need to revisit the sampling strategy in light of the strong preference for geographically sensitive data expressed by consumers in the focus groups.
The costs of producing a scorecard are formidable. These costs will be born by both the State and the health plans, therefore they must be justifiable in the short and long run. Cost will be affected by both content and the medium through which the scorecard is delivered to consumers. As the Consortium plans its next steps in scorecard development, a resource assessment should be included in the work plan.