Optimizing the Format for Presenting Updates of Evidence Reviews: Seeking User Preferences

Select to access the PowerPoint® presentation (215 KB).

Slide 1

Optimizing the Format for Presenting Updates of Evidence Reviews: Seeking User Preferences

Sydne Newberry, PhD
Southern California
Evidence-Based Practice Center
September 11, 2012

Images: Photographs show a man overwhelmed by stacks of paper and manila folders, and a shelf of books covered with dust and cobwebs.

Slide 2

Agenda

• Introduction.
• Examples of executive summaries we created.
• Results of focus group testing.
• Conclusions.
• Next steps.

Slide 3

No one knows the optimal format for updates.

• Do users prefer to see the original and the updated findings side by side?
  • Numerical or narrative?
• Do users care about seeing the original findings at all?
• Does the answer depend on the users?
  • And who are they???
• We tested reactions to a variety of formats among various categories of users.

Slide 4

The Goals for Optimizing the Formats for Update Reports

• To achieve some degree of consistency across Evidence-based Practice Centers (EPC) Comparative Effectiveness Review (CER) updates.
• But be flexible enough to allow for different situations.
  • e.g., the number and kind of new studies.
• No single format may be appropriate for every report.
  • Or every user...SO.
• Accessibility to the various stakeholders is key.

Slide 5

What we did...

Images: The following steps in the process are shown in a circle, connected by arrows:

• Identified  formats  in use (and users).
• Chose a sample report to modify.
• Created 5 test executive summaries.
• Assessed reactions of a focus group (clinical decisionmakers) to the summaries online and via conference call.
• Created 2 new versions of the summaries and discarded 1 based on feedback:
  • Assessed reactions of 2nd focus group (research decisionmakers) to the summaries.
  • Assessed reactions of 3rd group  (community physicians) to the summaries.
• Summarized results.

Image: At the center of the process is a photograph of a woman, crouched as if hiding so that the upper portion of her head and hands show behind the edge of a table on which books are stacked.

Slide 6

We transformed an update report using the formats of 3 other update reports

• A typical update report of a large consortium.
• An update report for a government agency.
• The EPC update review on acute otitis media itself:
  • Executive summary only.
  • Meta-analyses and quantitative results.
• An EPC update review on osteoporosis treatment:
  • Executive summary.
  • Qualitative findings only.

Slide 7

Who Uses Evidence Reports?

• Health plan managers.
• Other payers (private insurers, Centers for Medicare > Medicaid Services [CMS]).
• Clinical guideline developers (professional practice societies, U.S. Preventive Services Task Force [USPSTF], ...).
• Research planners.
• Congress.
• Clinicians at the point of care?
  • EPC's Eisenberg Center focuses on this group.

Slide 8

What we did...

Images: The following steps in the process are shown in a circle, connected by arrows:

• Identified formats in use (and users).
• Chose a sample report to modify.
• Created 5 test executive summaries.
• Assessed reactions of a focus group (clinical decisionmakers) to the summaries online and via conference call.
• Created 2 new versions of the summaries and discarded 1 based on feedback:
  • Assessed reactions of 2nd focus group (research decisionmakers) to the summaries.
  • Assessed reactions of 3rd group (community physicians) to the summaries.
• Summarized results.

Image: At the center of the process is a photograph of a woman, crouched as if hiding so that the upper portion of her head and hands show behind the edge of a table on which books are stacked.

Slide 9

Version 1...

• Shows new publication date.
• Shows new search end date.
• Lists new included studies, if any, in evidence table.
• Provides a table of changes to staff, key questions, methods at the end of the report.

In other words, previous and updated pooled results are not explicitly compared.

Slide 10

Version 1 replaces old with new

Overview

The objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM)...

The Technical Expert Panel and project staff developed a literature search strategy. The initial strategy was developed for MEDLINE® and was customized for other databases. Project staff searched MEDLINE® (January 1998-July 2010), the Cochrane Library (January 1998-July 2010), and the Web of Science. Additional articles were identified by review of reference lists in proceedings, published articles, reports, and guidelines.

Slide 11

Focus Group 1 Included 7 Participants

• Health plan pediatrician who serves on guidelines committees and has been on 2 TEPs.
• Academic family practitioner who has been on 2 TEPS.
• Insurance company research director, former EPC project leader.
• Academic physician/health services researcher/ guidelines developer.
• Health plan physician/manager and guideline developer.
• MS-level coordinator for guidelines committees.
• External affairs officer (former congressional staffer).

Slide 12

Focus group 1's reactions to Version 1 were uniformly negative

• BUT thought this format might be preferred by the average clinician:
  • i.e., one who is less interested in what is old vs. new, and simply needs to know what is up-to-date.
• The health plan guidelines writer shared that his organization's guidelines for practitioners are formatted exactly this way.

Slide 13

Version 2 "Highlights" Changes

The objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM) in children. AOM represents the most common childhood infection for which antibiotics are prescribed in the United States.  Data from the National Ambulatory Medical Care Surveys (NAMCS), which did not differentiate between AOM and otitis media with effusion, indicated that the number of office visits for AOM increased more than two-fold from 1975 to 1990. Gates (1996a) believed that the majority of these cases represented AOM. A diagnosis of AOM requires 1) a history of acute onset of signs and symptoms, 2) the presence of middle ear effusion (MEE), and 3) signs and symptoms of middle-ear inflammation. (Marcy, Takata, Shekelle, et al., 2001).

Slide 14

Several participants slightly preferred Version 2

• Enthusiasm increased if the update report had identical key questions and only quantitative findings changed.

Slide 15

Version 3 Used Revision Marks to Show Changes to Text

• In general we could not make definitive conclusions regarding differences in adverse event rates among antibiotics when taking into account a MCID of 5%. Adverse events were generally more frequent for amoxicillin-clavulanate than for cefdinir, ceftriaxone, or azithromycin.
• Meta-analysis demonstrated that children treated with 7 to 10 day amoxicillin-clavulanate had an 12.9 18 percent (95 percent confidence intervals, 4.5 8 percent and 21.2 28 percent) greater rate of gastrointestinal adverse effects than children treated with 5-day azithromycin (although not reported in the studies, the clavulanate concentration was most likely 31.25 mg per 125 mg of amoxicillin, i.e., original formulation.)

Slide 16

No one in Focus Group 1 liked this version!!!

Slide 17

Version 4 represented entirely new text but featured comparison tables

Slide 18

Version 5 Features a Narrative Summary Table with Bolding, and Shading .

* A text balloon over this statement reads "Some SRs have only narrative findings."

Slide 19

Focus group 1 participants infinitely preferred Versions 4 and 5!!

They shared that:

• Version 4 would be preferable for a report with quantitative findings (i.e., MA and pooled effect sizes), whereas.
• Version 5 would be preferable for a report with qualitative findings.
• Tables like these might even make version 1 more useful to a broad range of users....

Slide 20

More Suggestions to Emphasize Important Changes in an Update Report

• Show updated findings in context of old:
  • Most readers won't have the earlier report on hand.
  • Compare results in text.
• Include both qualitative AND quantitative tables:
  • Or at least include strength of evidence in table.
  • Show changes to key findings in a bulleted list.
• For decisionmakers, provide the figures and tables with the actual data in addition to the summary tables:
  • One of our focus group participants shared the format used by his health plan...

Based on the feedback, we tossed out Version 3, combined Versions 4 and 5, and created a new version.

Slide 21

What we did...

Images: The following steps in the process are shown in a circle, connected by arrows:

• Identified formats  in use (and users).
• Chose a sample report to modify.
• Created 5 test executive summaries.
• Assessed reactions of a focus group (clinical decisionmakers) to the summaries online and via conference call.
• Created 2 new versions of the summaries and discarded 1 based on feedback:
  • Assessed reactions of 2nd focus group (research decisionmakers) to the summaries.
  • Assessed reactions of 3rd group  (community physicians) to the summaries.
• Summarized results.

Image: At the center of the process is a photograph of a woman, crouched as if hiding so that the upper portion of her head and hands show behind the edge of a table on which books are stacked.

Slide 22

The new Version 3 simply included both quantitative and narrative tables...

Slide 23

The new version 4 contained no narrative text

Table S-1. Scope of the Report and Definitions

Results:

• Quantitative tables.
• Narrative tables.

Table S-2. Search Criteria

Image: A sample table heading shows the following items:

• Key Question.
• Databases.
• Article Types.
• Timeframe.
• Number of Studies Reviewed.
• Number of Studies Included.

Image: A flowchart shows the conceptual framework for the Management of Acute Otitis Media Update from the Southern California-RAND EPC.

• Conclusions.
• Limitations.
• Future Research Needs.

Slide 24

What we did...

Images: The following steps in the process are shown in a circle, connected by arrows:

• Identified  formats  in use (and users).
• Chose a sample report to modify.
• Created 5 test executive summaries.
• Assessed reactions of a focus group (clinical decisionmakers) to the summaries online and via conference call.
• Created 2 new versions of the summaries and discarded 1 based on feedback:
  • Assessed reactions of 2nd focus group (research decisionmakers) to the summaries.
  • Assessed reactions of 3rd group  (community physicians) to the summaries.
• Summarized results.

Image: At the center of the process is a photograph of a woman, crouched as if hiding so that the upper portion of her head and hands show behind the edge of a table on which books are stacked.

Slide 25

Focus Group 2 Included 5 Participants

• A director of an office at FDA.
• A director of a federally funded academic research center.
• An FDA microbiologist.
• An administrator at CMS.
• A former congressional staffer with health policy experience.

Slide 26

Focus group 2 participants preferred Versions 3 and 4

• Versions 1 and 2 were not helpful.
  • Want to see what changed and why.
• New version 3 (both sets of tables) was better, but—
  • show how the new findings contributed to changing the conclusions.
  • use color coding or different fonts to increase the salience of what changed.
  • include some wording that would put the changes in context.
  • Show why conclusions changed.
• New version 4 was even better but not a substitute for full report.

Slide 27

What we did...

Images: The following steps in the process are shown in a circle, connected by arrows:

• Identified formats in use (and users).
• Chose a sample report to modify.
• Created 5 test executive summaries.
• Assessed reactions of a focus group (clinical decisionmakers) to the summaries online and via conference call.
• Created 2 new versions of the summaries and discarded 1 based on feedback:
  • Assessed reactions of 2nd focus group (research decisionmakers) to the summaries.
  • Assessed reactions of 3rd group (community physicians) to the summaries.
• Summarized results.

Image: At the center of the process is a photograph of a woman, crouched as if hiding so that the upper portion of her head and hands show behind the edge of a table on which books are stacked.

Slide 28

Community physicians...

• May be the busiest of all groups we included!
• Disliked all but version 4!
  • Preferred graphic display and lack of narrative text.

Slide 29

Our study had a number of limitations...

• Tested only 1 report.
• Included only the executive summary, not the full report, but.
  • Equivalent to format used by large HMO.
• Original report included (almost) only quantitative results.
• Small number of focus groups and participants.
  • Possible sample bias.

Slide 30

Conclusions

• Update reports need to show at least what changed.
• A stand-alone executive summary without narrative text may suffice for point-of-care clinicians.
• Policymakers need full reports.
  • But stand-alone summary serves as useful map to full report.
• Optimum method of presenting conclusions depends on whether meta-analytic or narrative....

Slide 31

Next Steps

• We have submitted draft report to AHRQ.
• Followup testing with more reports and focus groups would be helpful.
  • We propose submitting an update report with a "Version 4" style executive summary for peer review and public comment.

Slide 32

Image: The RAND Health logo is shown.