Session J: Using Adverse Event Data to Improve Quality of Care

Moderator: Martin Hatlie, J.D., American Medical Association
Panel members: Brent James, M.D., M. Stat., Intermountain Health Care; and Richard Cook, M.D., University of Chicago

Every Defect a Treasure: Using Defect Data to Reduce Adverse Drug Events—Brent C. James, M.D., M. Stat.

Health care can be regarded as a series of interrelating processes functioning together in systems, and the application of quality improvement (the science of process management) in health care delivery can improve patient care and help to control costs.

Every process consistently produces three classes of outcomes—physical, service, and cost. Physical outcomes include medical outcomes such as complications and therapeutic goals, and functional status measures such as patient perceptions of medical outcomes. Service outcomes include satisfaction and access issues such as waiting times. A causal relationship exists between medical and cost outcomes so that a system designed to control quality will affect cost. To demonstrate the relationship between quality of care and cost, implementing processes in one hospital to improve timing for antibiotic prophylaxis increased optimal timing from 40 percent in 1985 to 96 percent in 1991. The hospital also reduced postoperative deep wound infections from 1.8 to 0.4 percent for an estimated annual savings of $714,000 and 51 fewer cases. In an second example, applying two quality improvement tools—the cause and effect diagram and the tally sheet—revealed six potential causes (drug reaction; failure to adjust for decreased renal function, patient age, patient body mass, or known hematologic factors; error in dosage) that accounted for 80 percent of all adverse drug effects (ADEs) with 66.2 percent of those preventable. When a tool was implemented to calculate renal clearance and to correct for failure to adjust for decreased renal function, the overall ADE rate dropped almost 50 percent and saved the hospital about $500,000 in the cost of care annually. 

A judgment-based approach, asking who to find and eliminate persons not meeting an acceptable quality threshold, results in assumptions about what remains after inspection and elimination of serious failures. A learning-based approach, asking why-what-how, focuses on the process of care to understand the causes of failure. Accordingly, quality improvement finds and eliminates inappropriate variation (process steps) and documents continuous improvement (outcomes). We have the opportunity to redefine the meaning and the nature of medicine (i.e., improved patient care delivery and reduced cost), but first we must understand and manage the underlying processes of health care.

Using Adverse Event Data to Improve Quality of Care—Richard Cook, M.D.

How do we learn from adverse events? We are constantly learning, but the issue is whether what we learn is correct. Associations between things that appear to be correlated may lead us to believe that they are causal, when in fact, they are unrelated; therefore, we need to find ways to go beyond seeing associations.

During the past 15 years, our understanding of how large, complex systems fail has changed. They fail in a particular way, that is, they have certain "signatures." Multiple faults occurring together, rather than a single point failure, cause system failure; there is no root cause. These small, otherwise innocuous, faults that exist in complex systems are termed latent failures. The "Swiss cheese model" of the path of causality of an accident shows a group of defenses or shields such as technical factors (e.g., safety interlocks, automated devices), human characteristics (e.g., training, expertise of operators), and organization and institutional structure (e.g., procedures, team functions, reviews) and "holes" or latent failures within each shield. When many latent failures align, a chain of events, or opportunity, is created that leads to a failure. The implication is that it is hard to defend against a particular accident. Hindsight bias (adding weight retrospectively to factors known to be causally related to the outcome) coupled with the Swiss cheese model of the way complex systems fail, leads to a cycle of error. A shift in the loci of system failures occurs, and, because latent failures are always present in different circumstances, there is a quiet period. Another accident that appears to be unique occurs from a different collection of latent failures, and the cycle of error is repeated.

To learn from medical errors, the presenter proposed the creation of Project Afterwards and Project Aftermath. In Project Afterwards, a national patient safety board would explore obstacles to detection, reporting, investigation, analysis, and dissemination of accident data and explore novel responses to protoaccidents and the dynamic construction of safe operation in a changing sociotechnical world. Initially, the organization could collect data (e.g., narrative stories, chart copies, photographs) from participants in medical accidents and those who respond to them. Project Aftermath would test the principle of using immediate response teams to investigate technical and organizational sources of medical accidents, similar to the National Transportation Safety Board.

The presenter noted that we always have trouble distinguishing between error and failure. Error is a process of social attribution applied to unwanted outcomes. We also need to be careful of the application of technology as the solution. Technology may solve some problems but create others. In medicine today, we tend to be moving from frequent, low-consequence failures to infrequent, high-consequence failures as a consequence of applying technology and, thereby, increasing coupling. 

Return to Contents

Session K: Educating Physicians for Evidence-based Practice 

Moderator: Lynn Kazemekas, Ed.D., AHCPR
Panel members: R. Brian Haynes, M.D., Ph.D., McMaster University; Lawrence G. Smith, M.D., Mt. Sinai Medical Center (NY); and Dave A. Davis, M.D., University of Toronto

Evidence-based Information Resources for Clinical Learning and Practice—R. Brian Haynes, M.D., Ph.D.

Health professionals must maintain awareness of current research in their specialty. Subject-specific services and journals that select and review studies of high quality are becoming more available as evidence-based practice becomes more widespread. These services and journals can serve as the practitioners' main source of current research information.

Electronic databases help us look up several studies on a given topic. MEDLINE is a searchable database for medical research studies across all specialties. New databases are specific to particular areas of practice, such as Best Evidence and ACP Library on Disk. The Cochrane Library, a database of randomized clinical trials, is available on CD-ROM. The Internet is a useful tool for medical information. Sites of interest include MEDLINE (http://www.ncbi.nlm.nih.gov/PubMed/), McMaster University's evidence-based practice site (http://hiru.mcmaster.ca/), SilverPlatter (http://php.silverplatter.com), and a site-specific to diabetes (http://www.diabetes.com).

Practitioners need to be trained to perform competent literature searches on bibliographic databases, and they must know about the tools available via CD-ROM and the Internet. Physicians should subscribe to evidence summary services that are tailored to their particular specialty areas. Clinical settings need to provide computer access for literature searching, CD-ROM use, and Internet connections so that practitioners can access the information immediately in the course of treating patients. 

Evidence-based Medicine: A Cultural Shift in Residency Training—Lawrence G. Smith, M.D.

Evidence-based medicine is difficult to implement for a variety of reasons, including inefficient procedures for accessing relevant evidence, time constraints for busy practitioners, few role models, a lack of sufficient evidence to answer many medical questions, and complicated patients who do not fit into single diagnostic categories. Evidence-based practice has a relative lack of cultural acceptance among senior clinicians, which is a major impediment to new physicians.

Mount Sinai Hospital is attempting to bring about a cultural shift toward evidence-based practice. The teaching methods are changing from the more traditional, passive lecture to a highly interactive, problem-solving model that forces students to build on previous knowledge by using and reinforcing that knowledge and supporting it with evidence. Students are taught to think and to question accepted practices by seeking evidence to support such practices.

Medical students are taught nine key skills:

1. Frame the patient's problem with a declarative statement, collecting, then citing, raw data to support this formulation.
2. Ask the relevant question that evidence needs to answer.
3. Think aloud through the process of clinical judgment to allow correction of mistaken assumptions.
4. Identify their own knowledge gap or any additional information needed to advise the patient.
5. Design a strategy to solve the knowledge gap through their own initiative, CD-ROM searches, literature searches, or consultation of specialists.
6. Search for evidence.
7. Appraise evidence critically.
8. Integrate the evidence into the care of the patient.
9. Teach the evidence they have learned to peers and students.

Medical students use these skills as they interact with peers and teachers in various settings. Care also is taken to ensure Students' ongoing respect for clinicians who choose the more traditional means of practice, such as expert judgment and clinical reasoning. Students ideally incorporate use of evidence with use of logical, clinical reasoning based on seasoned experts' experience in establishing their own style of practice.

Changing Physician Performance: Use of Enabling and Reinforcing Techniques in Maintenance of Learning and Change—Dave A. Davis, M.D.

Continuing medical education (CME) varies in its ability to effect changes in physician performance. Types include educational materials, formal activities (e.g., conferences, which can be didactic or interactive), patient-mediated strategies (e.g., providing educational materials to patients), and use of opinion leaders.

The researchers performed a meta-analysis of more than 100 randomized clinical trials by objectively assessing physician performance on health care outcomes of various types of CME to determine which strategies work best. The majority of studies demonstrated an improvement after educational intervention. The most effective strategies included reminders, patient-mediated interventions, outreach visits, use of opinion leaders, multifaceted intervention (use of more than one CME strategy), and formal activities designed for interactive participation.

CME needs to use practice-based interventions, particularly those focusing on the patient (e.g., providing educational materials to the patient), as these are highly effective. Use of a combination of multiple CME strategies together seems most effective for changing physician performance. 

Return to Contents

Session L: Legislating Clinical Care

Moderator: Edward B. Hirshfeld, J.D., M.B.A., American Medical Association 
Panel members: Alex R. Rodriguez, M.D., National Account Consortium, Inc.; George Isham, M.D., HealthPartners, Minneapolis; Honorable Jane Maroney, Delaware State Representative

Legislating Clinical Care—Edward B. Hirshfeld, J.D., M.B.A.

Historically, legislators have held several roles: regulating health insurance, facilitating entry of new health carriers, financing to expand access, mandating coverage of specific care and products, ensuring standards of care and practice guidelines, and addressing issues of coverage and standards. Legislation has long mandated that health plans cover certain medical services. During the era of health care reform. State health care reform statutes began including legislation that facilitates the adoption of practice guidelines by physicians. The objective was to provide physicians with defense against malpractice actions, if they followed the guidelines properly. 

Recently, legislation has begun to address both coverage of care and methods for provision of care, such as minimum hospital stays for mother and child after birth. Issues under debate concern questions of stake in health care including: What obligation does the legislature have to ensure that all Americans have health care coverage?, If the legislature is not obligated to ensure coverage, how far does its role in consumer protection extend?, Can it legislate coverage mandates for private insurers?, Should it legislate standards of care?, What standards should be used to evaluate the health care that is covered-specific needs of the individual or maximization of health care status in the population?, Who should decide the needs of the individual versus those of the population (voters, judges, the free market)?, and What responsibility does the medical profession have to help decide these issues?

Legislation and Clinical Care—Alex R. Rodriguez, M.D.

Legislation can help provide responsible, accountable managed care or it can interfere with those goals. Doctors working within and for managed care organizations are committed not only to care for the individual patient, but also for the family and the community by addressing such problems as communicable diseases and family violence. Managed care organizations consider themselves to be patient care advocates, and they seek involvement in legislative solutions to barriers to patient care quality.

Legislation of clinical aspects of care is increasing primarily because consumers and purchasers have become more knowledgeable about the processes of care relating to risk, quality, and cost. Thus, there is increasing demand for confidentiality, members' rights and responsibilities, and quality improvement. The activities of consumer advocacy groups have shaped public policy, and the resulting legislation has improved health care. The managed care industry has become concerned, however, about its liability for medical errors that existed before the advent of managed care.

Significant errors in health benefits administration in managed care have led to public distrust. Managed care organizations must strive for more flexibility when dealing with individual patients in order to improve their public image. Government regulation of managed care reassures consumers about accountability of managed care plans and providers. The ability of governments to enforce their regulations is limited, however. Legislation that encourages consistency and equity across plans promotes common standards for all. While improved population data on quality and cost have benefited health care, some negative impacts of regulation may emerge as well, including: (1) interference of market-regulatory activities that add administrative burdens and costs to plans, (2) a tendency to overlook patients' interests in favor of financial interests and political solutions, and (3) legislation of exceptions to maximum lengths of stay that could result in decreased quality monitoring and increased infection potential. The health care system could be improved through the efforts of the executive and legislative branches of the Federal, State, and local governments to develop an accountable and responsible system based on national health care data. 

Return to Contents

Session M: Using Evidence-based Quality Indicators 

Moderator: Irma E. Arispe, Ph.D., Johns Hopkins Bayview Medical Center
Panel members: Ann G. Lawthers, Sc.D., Harvard School of Public Health; and June M. Buckle, Sc.D., Johns Hopkins Bayview Medical Center,

Using Evidence-based Quality Indicators—Irma E. Arispe, Ph.D. 

Quality indicators have been defined as "tools that estimate the extent to which a health care provider delivers clinical services that are appropriate for each patient's condition, provides them safely, competently, and in an appropriate time frame, and achieves desired outcomes in terms of those aspects of patient health and satisfaction that can be affected by clinical services." (Palmer et al., 1996). Quality indicators, or clinical performance measures, are increasingly in demand in this era of managed care. Many are now available, although some organizations choose to create their own. It is crucial, when choosing an already-existing method of quality measurement, to consider the measure's reliability, validity, and appropriateness for the particular purpose. If one is creating a new measure, one should consider development cost, decreased generalizability of tailored measures, and the resulting need for adapting or updating such measures for new purposes.

Cataloging and Evaluating Clinical Performance Measures: CONQUEST and the QMNet Initiative—Ann G. Lawthers, Sc.D.

Clinical performance measures have multiplied to meet the increased demand for assessing quality of care. Potential users face a confusing array from which to choose. AHCPR recognized a need for cataloging the myriad quality indicators and funded development of CONQUEST (Computerized Needs-Oriented Quality Measurement Evaluation System), a database of clinical conditions and clinical performance measures used for managing care of those conditions. Partners in the CONQUEST system development effort include the Harvard School of Public Health, the Center for Health Policy Studies, Mikalix and Company, and the MEDSTAT Group. 

The CONQUEST database provides summary information on 52 conditions addressed in AHCPR evidence-based clinical practice guidelines and relates them to 1,185 performance measures of 53 types. The database allows users to compare measures in order to choose those most suited to their purposes. The system is available free of charge and can be downloaded from the AHCPR Internet site or by calling the AHCPR Publications Clearinghouse at 1-800-358-9295. An updated version is being prepared that features more measures, more conditions, more detail on each measure, and improved user interface. Eventually, CONQUEST will be privatized into QMNET, with a technical support hotline and linking codes between conditions and measures.

Using Evidence-based Quality Indicators—June M. Buckle, Sc.D.

Johns Hopkins Bayview Medical Center recently organized a system for outcome evaluation and care management. The Outcomes Evaluation Team and the Care Management Team collect and analyze data to develop and implement practice guidelines, as well as care for patients' social work needs and manage patients' medical records. The goal is to move patients quickly, safely, and efficiently across the continuum of care toward discharge, emphasizing close consultation with physicians to determine when patients are ready for decreased levels of care. Case Management Team members follow their patients across campus and act as patient advocates with a focus on maximizing optimal outcome. Field studies are conducted, medical records are coded, and persistent readmissions are examined closely to target areas of care in need of improvement. An Extended Care Pathway Model was developed, with identical quality measures employed across levels of care, to allow for comparison of outcomes by setting as well. 

This system has proven to be a valuable tool for guideline development, case management, outcomes evaluation, and risk management. It has enabled treatment teams to focus quality improvement efforts on critical areas, demonstrate the value of new programs in improving outcomes, respond to patients' questions, and determine the best strategies for treating specific patient populations.

Return to Contents

Session N: Using Quality Indicators: Useful Tools for Investigation, Regulation, and Accreditation 

Moderator: Barbara B. Fleming, M.D., Ph.D.
Panel members: Jerod M. Loeb, Ph.D., JCAHO; William E. Golden, M.D., Arkansas Foundation for Medical Care; James G. Sheehan, J.D., U.S. Department of Justice

The Next Wave of Quality Measurement: Key Challenges—Jerod M. Loeb, Ph.D.

The Joint Commission of Accreditation of Healthcare Organizations' (JCAHO) data-driven accreditation process, the ORYX, is part of a new initiative that integrates performance measurements with accreditation. It contains a series of performance requirements that will gradually be implemented for all entities accredited by JCAHO. Data collection will include indicators of health status, satisfaction, and administrative and financial functions. Key challenges to performance measurement are:

1. People's reluctance to be measured.
2. Discordant approaches to measurement:
   1. Selecting a measurement based on available data.
   2. Determining what data are needed.
   3. Determining whether reliable data are available.
3. Benefits of data collection versus cost issues.
4. Lack of data standardization (what to measure and how to measure it).
5. Data integrity (variations in the quality, interpretation, and analysis of data).
6. Risk adjustment (whether to adjust, how to adjust, possibility of producing a false sense of security).
7. Lack of agreement on how data should be reported.
8. Different tiers of measurement (measurement strategies used at different levels of the health care system and authority for certain outcomes, especially physician performance and liability issues).
9. Value of improvements in quality of care not always easy to demonstrate.

Regional Quality Assurance Organizations: Regulator or Disseminator of Guidelines?—William E. Golden, M.D.

The Arkansas Foundation for Medical Care developed, disseminated, and implemented quality improvement projects that translate evidence in guidelines into practice. Although Professional Review Organizations (PROs) have oversight of Medicare delivery to ensure appropriate use of resources, creation of the Health Care Quality Improvement Program enabled the Foundation to take over the role of Quality Improvement Organizations (QIOs) in allowing pattern analysis, authority for chart review, and hypothesis testing. Challenges faced by PROs in creating regional quality improvement projects included designing an efficient project dissemination process, motivating local project adoption, and managing multiple projects at multiple sites.

The projects are based on widely accepted clinical guidelines or on consistent clinical research. Clinically relevant data, essential indicators, and performance measures are collected if needed. Projects are distributed locally as needed. Technical support is provided by outreach nurses and by a database system that monitors each hospital's project status. 

Because the projects do not have a direct stake in enrollments, utilization, or profit, they can address issues of underprovision; conflicts between beneficiaries, providers, and insurers; patient satisfaction; and universal quality improvement. Also, because they focus on capacity building rather than on minimum standards, they have come to be viewed as a community resource.

Quality Improvement Initiatives of the Health Care Financing Administration—Barbara B. Fleming, M.D., Ph.D. 

The Health Care Financing Administration (HCFA), in a shift from the role of payer to that of purchaser, has launched several quality improvement initiatives, including (1) requiring all Medicare risk and cost contractors to report the Health Plan Employer Data and Information Set (HEDIS) measurement set on effectiveness of care and utilization, (2) developing a national performance measurement system in the fee-for-service setting to examine quality of care, (3) developing a claims-based surveillance system for the fee-for-service setting that may raise questions about quality of care for Physician Review Organizations (PROs), (4) collaborating with the National Committee for Quality Assurance (NCQA) to develop a similar surveillance system in the managed care setting, and (5) keeping abreast of the quality of care being delivered and the burden imposed on health care systems by performance measurement systems. As a first step, HCFA is funding the American Diabetes Association (ADA) to work with the NCQA to develop a consensus on common measures for diabetes.

Translating Evidence Into Practice—James G. Sheehan, J.D.

Considerable use and misuse of quality standards and performance measurements occur in the health care system, especially in large health care organizations. This has implications for health care fraud prosecution. Assumptions of good faith and honesty on the part of providers are usually appropriate, but other assumptions can be made as well: (1) performance measurement changes more than just behavior, (2) there are bad people in the world, (3) people respond to economic incentives, (4) powerful people take advantage of the weak unless others get involved, and (5) it is satisfying to see bad people taken away.

Working with these assumptions, there are four major issues.

1. What do we care about in the area of clinical practice guidelines? The most important issue is whether people are denied medically necessary or quality care. Trust is another important issue that often determines whether the doctor's or hospital's decision process is seen as fair and objective.
2. How do we find the problem areas? Problem areas are identified comparatively (e.g., examining evidence from the PROs, HCFA, public sources, and the literature) and anecdotally (e.g., using information from physicians, lawyers).
3. How do we investigate or prosecute if necessary to prove a conscious decision to deny a medically necessary service? The typical guidelines case uses various techniques, such as the data system of the hospital or HMO to see comparable behavior changes and accounts of interactions between primary health care providers and the HMO or hospital system. If the decision process is fair or designed to address whether the patient needs the service under the guidelines, there will be clearly articulated standards and an independent and objective review system. There is also the issue of trust. When an adverse event results from a fair decision process, the interactions between the guidelines, the patient, the treating physician, and the reviewer need to be examined. Protection of patient confidentiality and quality of advocacy must also be examined.
4. How do we change perceptions about how these things work and how do we make sure that they work properly? There are two approaches when misuse is identified: (1) injunctions against and consent decrees with organizations following inadequate approaches and (2) criminal prosecution of organizations engaging in egregious conduct and unfair patient treatment.

Return to Contents
Proceed to Next Section