Panning for Research Questions: Using Systematic Reviews and Stakeholder Input to Identify Needed Research in Comparative Effectiveness

Select to access the PowerPoint® presentation (3.1 MB).

Slide 1

Panning for Research Questions: Using Systematic Reviews and Stakeholder Input to Identify Needed

Tim Carey, MD
RTI-UNC Evidence-based Practice Center (EPC)

Elisabeth Kato, MD, MRP
Effective Health Care Program, AHRQ

Jeanne-Marie Guise, MD, MPH
Oregon Evidence-based Practice Center (OHSU)

Image: The logos for UNC's Cecil G. Sheps Center for Health Services Research, AHRQ's EPC program, and OHSU are shown.

Slide 2

Overview

• The Future Research Needs Program.
• Working with Stakeholders.
• Prioritization Strategies.
• Exercise.
• Discussion and Future Directions.

Slide 3

The Effective Health Care Program

The 2003 Medicare Prescription Drug, Improvement, and Modernization Act authorized the Agency for Healthcare Research and Quality (AHRQ) to improve the quality, effectiveness, and efficiency of health care delivered through Medicare, Medicaid, and State Children's Health Insurance programs.

The Effective Healthcare Program carries out this goals through:

• Evidence Synthesis.
• Evidence Generation.
• Dissemination and Translation.

Slide 4

How Do We Decide Which Comparative Effectiveness Questions to Research?

Traditional research paradigm: unplanned, researcher driven, inconsistent public input.

• Decentralized generation of questions.
• Peer review by researchers.
• Closed-door funder decisions.

New CER/PCOR [comparative effectiveness and patient-centered outcomes research] paradigm: inclusive, rapid, and targeted process.

• Systematic identification of gaps.
• Involvement of a range of stakeholders, including patients.
• Open process.

Slide 5

AHRQ's Position with Stakeholders

"The Agency for Healthcare Research and Quality (AHRQ) firmly believes that involving all stakeholders in the research enterprise from the beginning improves the end product and facilitates the diffusion and implementation of the findings by getting early buy-in from users. Involving stakeholders also helps to ensure that the research reflects the various needs of all diverse users."

Operating Plan for American Recovery and Reinvestment Act Funds for Comparative Effectiveness Research as the Agency for Healthcare Research and Quality (2009)

Slide 6

PCORI's Mission Statement

"PCORI helps people make informed healthcare decisions, and improves healthcare delivery and outcomes, by producing and promoting high-integrity, evidenced-based information, that comes from research guided by patients, caregivers, and the broader health care community."

http://www.pcori.org/about/mission-and-vision/

Slide 7

Great Idea But.... 

• Where do gaps come from?
• Who should be involved?
• What is the best way to prioritize needs?
• How much time is required?
• How much money does it take?

Carey et al. (2012)

Slide 8

Systematic Reviews = Evidence Gaps

Image: The word EVIDENCE is shown in black block letters on a white background. Below it, EVIDENCE appears as white block letters on a black background.

Evidence Gaps = Future Research Needs?

Slide 9

Not Necessarily.....

• The review team may not identify all gaps.
• Not every evidence gap is equally important.
• Different goals.

"More, bigger, longer, better randomized controlled trials are needed".

Slide 10

AHRQ Effective Healthcare Program Future Research Needs Projects

• 2010: AHRQ piloted 8 Future Research Needs (FRNs) Projects to extract research gaps from a systematic review, transforming them into prioritized research questions aided by diverse stakeholders.

  Goal: alert researchers and funders to important evidence gaps affecting healthcare decisions.

• 2011: pilot program reviewed, resulting in a common 7 step framework for AHRQ FRN projects.
• 2012: 29 "second generation" FRNs have been or are being completed, 8 FRN methods papers published to date.

1. Systematic review is published with researcher-determined research gaps.
2. Orientation of stakeholders to CER question, FRN process, and prioritization criteria.
3. Elaboration and consolidation of research gaps through iterative process with stakeholders.
4. Priority ranking of the research gaps.
5. Transformation of research gaps into needs.
6. Refinement and re-ranking of priorities by stakeholders.
7. Addition of study design considerations.

Slide 11

Working With Stakeholders

How do you ensure meaningful and representative engagement?

Image: A cartoon shows a man making a presentation at a podium in front of a crowd; another man at the back of the crowd shouts, "You can engage us better down here!"

Slide 12

What Is Meaningful Engagement?

PCORI:

• "Extending well beyond the simple notion of securing input from a variety of stakeholders."
• "Projects must include stakeholders as partners with significant involvement in all appropriate stages of the research project."
• "True partnership of shared accountability."

Slide 13

Who Do You Need at the Table?

Image: King Arthur and his knights are shown standing around the Round Table.

Slide 14

Stakeholder Groups Defined

Slide 15

Stakeholder Involvement Across Research Phases

Image: A double-headed arrow captioned "Research Continuum" extends above the following items:

Topic Generation →

• Clinicians.
• Consumers.
• Funders.
• Insurers.
• Researchers.
• Policymakers.

Research Prioritization →

• Clinicians.
• Consumers.
• Funders.
• Insurers.
• Manufacturers.
• Researchers.
• Payers.
• Policymakers.

Study Design →

• Clinicians.
• Manufacturers.
• Researchers.
• Policymakers.

Implementation →

• Clinicians.
• Consumers.
• Researchers.
• Policymakers.

Analysis/Interpretation →

• Clinicians.
• Researchers.

Dissemination →

• Clinicians.
• Consumers.
• Funders.
• Insurers.
• Manufacturers.
• Researchers.
• Payers.
• Policymakers.

A second double-headed arrow extends below these lists, with the text "EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER."

Slide 16

Things to Consider in Selecting Stakeholders

Make sure your expectations and those of stakeholders are aligned:

• Stakeholders can play multiple roles.
• Categories are broad, need to consider this carefully.
• Do you want patients, general public, or patient advocates?
• When working with organizations make sure you understand their interests.
• Individual vs. broader interests.
• Preparedness to understand topic & research.

Image: Two construction teams are shown, each standing on one side of a bridge they have been working on; the two sides do not align and will not be able to connect.

Slide 17

How Do You Ensure Meaningful Engagement?

Image: A group of blue figures stand in a circle around the pieces of a partially finished jigsaw puzzle.

Slide 18

Preparation of Stakeholders

• Assume that all stakeholders need orientation to the systematic review and evidence grading processes.
• Familiarity with research methods likely to be very variable.
• Conflicts of interest are to be expected with stakeholder groups, but must be disclosed.
• Patient/consumer/caregivers may need significant orientation and training regarding research the research process.

Image: A cartoon shows two painters working on a large sign that reads "ALWAYS BE PREPARED." One painter says to the other, "I thought you were bringing the paint."

Slide 19

Provide Background Materials in Lay Terms

Brief Description of Project:

    Purpose
    Process
    Timeline
    Background
    Expectations
    Final Product

  Potential Formats:

    Hard Copy
    Web Conference
    In-person

Consider offering personal sessions.

Slide 20

Example: Purpose

• Goals: To establish an agenda to guide the Agency for Healthcare Research and Quality's (AHRQ's) comparative effectiveness review and research activities regarding prevention in women's health.
• Identify topics stakeholders struggle with in clinical decisionmaking and health policy, where answers will improve the quality, effectiveness, and efficiency of United States health care.

Slide 21

Example: The Process

Slide 22

Example: Timeline

• Three meetings:
  • May 11 – Framework.
  • June 8 – Topic Generation.
  • July 13 – Final Prioritization.
• Extranet and email topic generation and participant discussion.

Slide 23

Example: Background

Comparative Effectiveness Reviews

Definition: Comparative Effectiveness Reviews (CERs) systematically and critically appraise existing research comparing the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care.

Purpose: To assist consumers, clinicians, purchasers and policymakers to make informed decisions that will improve health care at both the individual and population levels.

Sox, HC & Greenfield, S (2009).

Slide 24

Example: Expectations

Topic Generation:

• Nominate topics via the Web. [This text is circled in red, with a red text box noting "You are here."]

Topic Development:

• Clarify nomination intent.
• Frame clinical context of nomination.

Topic Refinement:

• Provide "real world" input from provider, patient, and policy perspectives.
• Provide input on clinical logic.
• Public comment via Web.

Review:

• Provide methodological & clinical input as a TEP member.
• Provide input via peer review.
• Provide public comment via Web.

Research Needs Development:

• Participate in discussions to describe research gaps & prioritize research needs.

Report Translation & Dissemination:

• Participate in focus groups.
• Participate in individual interviews.
• Provide expert review of summary guides.

Technical Expert (TEP) Panel

The TEP is constituted to inform the scientific content of the research review and provide guidance to the EPCs throughout the evidence review process. Technical Expert Panels may assist the EPCs by providing input to:

• Focus the literature search.
• Identify inclusion/exclusion criteria.
• Answer specific technical or clinical questions.
• Identify specific subgroups for which there may be heterogeneity of treatment effect.

*Note: The role of the EPC is to synthesize a report of the existing body of evidence. The final EPC report will be published by AHRQ and may be used by other agencies to inform policy or determine clinical guidelines.

Slide 25

Recommendations

• Key Informants: Optional.
• Stakeholder Number: 2-3 stakeholders from each SH category (One stakeholder is not representative; Different stakeholders of the same "type" have different perspectives)—Office of Management and Budget (OMB).
• Stakeholder Constituency: Depends on research phase:
  • Most times clinicians, researchers, funders, consumer advocacy representatives.
• Methods of Engagement: Mixed:
  • In person (telephone or face to face with or without Web)—orientation clarification and expectations delineation; also can be used to generate topics clarify questions.
  • Materials—summarizing overview for stakeholders.

Slide 26

Current Dilemmas

• Different stakeholders need different levels of training.
• What extent can stakeholders be reused from project to project?
• Preparation and experience may change the stakeholder over time to be more like us.

Image: A white figure stands at the center for a circle of red arrows pointing outward in all directions. Above the figure's head is a question mark.

Slide 27

Prioritization

• When to do it.
• How to do it.
• What method is best?
• How can we do it better?

Slide 28

AHRQ Effective Healthcare Program Future Research Needs Projects

• 2010: AHRQ piloted 8 Future Research Needs (FRNs) Projects to extract research gaps from a systematic review, transforming them into prioritized research questions aided by diverse stakeholders.

  Goal: alert researchers and funders to important evidence gaps affecting healthcare decisions.

• 2011: pilot program reviewed, resulting in a common 7 step framework for AHRQ FRN projects.
• 2012: 29 "second generation" FRNs have been or are being completed, 8 FRN methods papers published to date.

1. Systematic review is published with researcher-determined research gaps.
2. Orientation of stakeholders to CER question, FRN process, and prioritization criteria.
3. Elaboration and consolidation of research gaps through iterative process with stakeholders.
4. Priority ranking of the research gaps.
5. Transformation of research gaps into needs.
6. Refinement and re-ranking of priorities by stakeholders.
7. Addition of study design considerations.

Slide 29

Priority Ranking

• Reviews may generate many gaps, need for prioritization.
• Some organizations use broad internet data gathering:
  • Will the participants understand all of the issues?
• Multiple methods currently used:
  • Ranking 1-xx.
  • Likert scale 1-7.
  • Multi-voting, multiple (but limited) votes per choice.
  • Pair-wise comparisons.
  • Delphi methods.
  • Nominal group process.
  • Consensus conference.

Slide 30

Transformation of Research Gaps into Needs

• Gaps are generally in the form of a declarative sentence.
• Needs are questions similar to research questions in a grant proposal.
• Most organizations use PICOTS framework:
  • Population, Intervention, Comparator, Outcome, Timeframe, Setting.
• Methods questions may be important, but may not be a fit for PICOTS.

Slide 31

Refinements and Re-Ranking

• When number of gaps is large, multiple rounds of prioritization may be needed.
  • Many groups use 2-3 rounds.
• Gaps may be combined or split depending on the level of granularity that is useful.
• Is consensus needed?

Slide 32

Images: Screenshots show pages from the Stars Selection Web site concerning prioritization of research needs for ADHD.

Slide 33

Example: AHRQ Future research needs on ADHD

Key Questions from Comparative Effectiveness Review

20 research gaps from the review mapped to the key questions, presented to a group of 12 stakeholders, including funders, advocates, clinicians, regulators, researchers, and policymakers.

After stakeholder input, 29 research gaps. 8 gaps emerged as the top future research needs after two rounds of prioritization using an online prioritization tool.

The next two slides show the presentation of one gap from identification to study design.

Slide 34

Identify Research Gap

For children less than 6 years of age with disruptive behavior disorder or ADHD, limited data are available about the efficacy and effectiveness of psychosocial treatment programs (e.g., parent training and summer behavior treatment programs), alone or in combination with pharmacological interventions, compared with other psychosocial treatment programs, alone or in combination with pharmacological interventions. (KQ 1).

After One Round of Prioritization Apply PICOTS and Develop Research Question:

Research Question: For children less than 6 years of age with disruptive behavior disorder or ADHD, what is the comparative efficacy and effectiveness of specific psychosocial treatments alone compared with pharmacological treatments alone or in combination with psychosocial treatments for patient outcomes?

Slide 35

After Second Round of Prioritization Develop Study Design Considerations:

• Randomized controlled trials
  Randomized trials could be designed to test various components in a 2x2 matrix of psychosocial treatment variants (parent training, school-based intervention, combination, or pharmacological).
  • Advantages of study design for producing a valid result
    Allows isolation of causal inferences related to the intervention being tested. Multiple-armed trials would allow testing of several hypotheses regarding relative efficacy of singular or combination treatment components.
  • Ability to recruit/availability of data
    Common condition in this age group with uncertainty regarding treatment choice; all arms receive some treatment.
  • Resource use, size, and duration
    Large sample size (N = 840; n = 210 per treatment arm) needed. Key outcomes such as school achievement will require follow-up of several years.
  • Ethical, legal, and social issues
    Vulnerable population, careful informed consent will need to occur.

Slide 36

State of the Science

• Multiple groups are currently conducting work in this area.
• Tendency to emphasize cross-cutting issues across diverse stakeholder groups.
• Sufficient common aspects to serve as a consensus:
  • Criteria for gaps identification.
  • Broad aspects of stakeholder panel composition.
  • Need to train stakeholders in PCOR.
  • Explicit prioritization method—but multiple methods currently used, no empiric guidance as to optimal method.
  • Decisions regarding study design considerations.
• Organizations can use existing methods now while refining the approaches

Slide 37

Timelines

• Work can expand to fill a substantial amount of time.
• Rapid turnaround important for:
  • Working from the current systematic review.
  • Dissemination.
• 4-7 month time frame:
  • Step 1: Identification/Recruitment (1-2 months).
  • Step 2: Engagement & Prioritization (2-4 months).
  • Step 3: Analysis/Drafting Report (1-2 months).
  • After publication: Dissemination—targeted vs broad, role of Web vs peer review (may not be the EPC conducting this step)

Slide 38

Recommendations

• Evaluate different stakeholder panel sizes and compositions in prioritization.
• Evaluate the reliability of stakeholder prioritization through replication studies.
• Test different methods of prioritization to assess for transparency, reproducibility and efficiency.
• Clarify role of gap identification and prioritization with other methods such as value of information modeling.
• Collaborate across PCOR programs in improving methods, developing consistent terminology.

Slide 39

Current Dilemmas

• What prioritization works best?
• General versus specific research topics or questions.
• Do you propose study designs?
• How many rounds are required?

Image: A white figure stands at the center for a circle of red arrows pointing outward in all directions. Above the figure's head is a question mark.

Slide 40

Exercise

Future research needs in Attention Deficit Hyperactivity Disorder (ADHD)

Slide 41

Key Questions from Comparative Effectiveness Review

Slide 42

Analytic Framework

Image: The analytic framework for the ADHD Comparative Effectiveness Review is shown.

Slide 43

Evidence Gaps

1. How does the use of different data capturing methods (such as child self-rating scales, parent scales, semistructured interviews) affect apparent treatment efficacy in people ≥6 yrs with ADHD.
2. What are the prescriptive tx or optimal circumstances for adding specific psychosocial and/or pharmacological tx components based on the needs of people ≥6 yrs with ADHD.
3. What are the prescriptive tx or optimal circumstances for adding specific psychosocial and/or pharmacological tx components based on the needs of preschoolers with disruptive behavior disorder or ADHD.
4. What is the amount of variation in case identification and prevalence across geographic areas, age groups, settings and cultures.
5. What is the comparative effectiveness of different preschool interventions on long term outcomes.

Slide 44

Nominal Group Exercise

1) Individual responses.
2) Clarification and consolidation.
3) Ranking responses.

Images: Three photographs show paper stickers with topic ideas written on them, first as individual responses, then grouped for clarification and consolidation of ideas, then ranked.

Slide 45

Summing It All Up

• Systematic reviews provide a feasible basis for inclusive, rapid, and targeted identification of future research needs.
• Meaningful stakeholder involvement requires careful selection and preparation.
• There are many ways to involve stakeholders in prioritization but the optimal approach is not yet clear.

Slide 46

Future Directions for Future Research Needs

• How should research needs identified through topic specific systematic reviews fit into broader topic identification efforts?
• Will the gaps and prioritization resonate with funders and policymakers?
• What are the best ways to communicate with the public and with funders?
• Who should be involved in refining this area?

Slide 47

Questions? Comments?

elisabeth.kato@ahrq.hhs.gov

guisej@ohsu.edu

timothy_carey@med.unc.edu

Slide 48

References

• Carey T, Yon A, Beadles C, Wines R. Prioritizing Future Research through Examination of Research Gaps in Systematic Reviews. Prepared for the Patient-centered Outcomes Research Institute. Washington DC: March 15, 2012.
• Chang SM, Carey T, Kato EU, Guise JM, Sanders GD. Ann Intern Med 2012 Jul 31. doi
• Kane RL, Guise JM, Hartman K, Rothenberg B, Trikalinos T, Wilt T. Presentation of Future Research Needs. Methods Future Research Needs Report No.9. AHRQ Publication No. 12-EHC053-EF. Rockville, MD: Agency for Healthcare Research and Quality. April 2012. http://www.effectivehealthcare.ahrq.gov/reports/final.cfm.
• Sox, HC & Greenfield, S. Comparative effectiveness research: a report from the Institute of Medicine. Annals of Internal Medicine 2009; 151(3):203-205.