Selecting Evidence for Comparative Effectiveness Reviews (Text Version)

On September 16, 2009, Melissa McPheeters made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (913 KB).

---

Slide 1

Selecting Evidence for Comparative Effectiveness Reviews

Melissa McPheeters, PhD., MPH
Associate Director,
Vanderbilt University Evidence-based Practice Center

 

Slide 2

Overview

• Why use selection criteria?
• What do they look like in the context of a systematic review?
• When should they be set?
• How do they affect the conduct and interpretation of the review?

 

Slide 3

 

CER Process Overview

Prepare topic

• Refine key questions
• Develop analytic frameworks

Search for and select studies:

• Identify eligibility criteria
• Search for relevant studies
• Select evidence for inclusion

Abstract data:

• Extract evidence for studies
• Construct evidence tables

Analyze and synthesize data:

• Assess quality of studies
• Assess applicability of studies
• Apply qualitative methods
• Apply qualitative methods (meta-analyses)
• Rate the strength of a body of evidence

Present findings

 

Slide 4

Study Selection Criteria

• Function the same in systematic reviews as in primary research
• Should reflect the analytic framework and key questions
• Are powerful tools for widening or narrowing the scope of a review
• Provide information to determine whether reviews can be compared or combined

 

Slide 5

Types of Criteria

• Population descriptors
• Intervention descriptors
• Outcome descriptors
• Study design
• Study size
• Source of literature (including grey lit)

 

Slide 6

Some Example Criteria

• Adult, community dwelling females
• Study of a screening tool for depression
• United States only
• Hospital-based studies only
• N>200
• Randomized controlled trials
• Articles published in English or German

 

Slide 7

Selecting Criteria

• Review study goals
• Assess analytic framework and key questions
• Tie criteria to PICO(TS)
• Set criteria before beginning abstract review

 

Slide 8

Using broad criteria

• Can be as broad as all related studies
• Helpful for exploring "what is known"
• May result in too much literature to feasibly review, or disparate literature that can't be compared.

 

Slide 9

Using narrow criteria

• May return too little literature
• Can result in inability to answer the intended question
• Helpful in culling homogenous literature
• Can reduce size of the literature to a manageable scope

 

Slide 10

Bias in this context

• Distortion of the estimate of effect that comes from how studies are selected to be included
• Affects the applicability or "external validity" of the review itself

 

Slide 11

Bias in this context II

• For example, included studies may not have been conducted in the patient population whose care the review is intended to affect
  • E.g., the use of studies of twin pregnancies in a review of preterm labor management for low risk women
• Or - selection criteria may be set to include more of a certain study type that either over or under estimates effectiveness

 

Slide 12

Basic Questions

• What is the relevant population?
• What is the intervention of interest?
• To what exposure is the intervention being compared?
• What outcomes are relevant?
• Should time to outcome be limited?
• In what setting should the results be applicable?

 

Slide 13

Considerations

• Grey literature: New technologies may have substantial grey literature, including govt. reports and trade journals
• Non-English articles may be useful for certain types of interventions (e.g. CAM)
• Study size limitations may depend on what is available
  • Easier to set for observational studies
  • Large bodies of efficacy/effectiveness lit may be made up of small trials

 

Slide 14

Exercise 1

• What would you do...
  • If you were asked to review the literature on transition support for adolescents with autism entering adulthood?
  • Before seeing the key questions, consider the categories of criteria that we will want to apply.

 

Slide 15

Apply PICOTS

• Population - condition, disease severity and stage, co-morbidities, patient demographics
• Intervention - dosage, frequency, and method of administration
• Comparator - placebo, usual care, or active control
• Outcome - health outcomes, morbidity, mortality, QoL
• Timing - Duration of follow-up
• Setting - Primary, specialty, in-patient, co-interventions

 

Slide 16

PICOTS

• Population
• Intervention
• Comparators
• Outcomes:
• Timing
• Setting

• What constitutes an adolescent? What constitutes a diagnosis of ASD?
• How is transition support defined?
• Do we compare to no transition support or directly compare types of support?
• What are the goals for adolescents with ASD as they transition to adulthood? Should they be individually focused?
• How quickly should the outcomes be apparent?
• Is transition support provided in multiple settings such as schools, clinics and the community?

 

Slide 17

What would you do with .

• A paper that was about "individuals over age 10?"
• A paper that was about an intervention for individuals with a range of developmental disabilities?
• Or conversely, about children specifically with Asperger syndrome but not other ASDs?

 

Slide 18

Example of a narrow scope

• What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?

 

Slide 19

Selection Criteria

• Efficacy: RCTs only
• Low risk: No prior preterm birth
• No. of studies: 11 studies

 

Slide 20

OAB Study

• Management of Overactive Bladder among Women
• Considerations:
  • OAB is a fairly difficult condition to define
  • Treatments include pharmacologic, behavioral, CAM and procedural - and each area includes very different types of studies
  • Its study is often combined with other types of urological conditions such as stress incontinence or prostate issues

 

Slide 21

Exercise 2

• Set two criteria and consider the potential implications:
  • Minimum study size
  • Gender of study participants

 

Slide 22

Study Size

• 50 at study start
• Implications:
  • Excluded for size only: 79
  • Excluded for N < 20: 36
  • Excluded for N 20-29: 23
  • Excluded for N 30-39: 8
  • Excluded for N 40-49: 12

 

Slide 23

Gender

• Studies had to include at least 75% women
• This decision was based on expert opinion and the size and scope of the literature
• 40 studies were excluded with less than 75% women
• 27 additional studies would have been excluded had the review been limited to studies of only women

 

Slide 24

Other Considerations

• What study designs should be included?
• Include foreign studies? Other languages? Studies conducted in specific parts of the world?
• Is the "grey" or "fugitive" literature included?

 

Slide 25

Types of Studies?

• Limit to RCTs?
• Include observational studies?
• If so, which kinds?
• What is the value of case series?
• What counts as a case series?

 

Slide 26

Example 3

• Maternal-fetal Surgery Technical Brief
  • Included case series with N>=2
  • Only 3 of 169 studies were RCTs and 122 were case series
  • Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups

 

Slide 27

Observational Studies

• Types
  • Cohorts (with comparisons)
  • Case controls
  • Case series
  • Registries/databases

 

Slide 28

Observational Studies

• Well-done observational studies can address issues of applicability and need for longer-term outcomes if they:
  • Include more representative patient populations
  • Have relevant comparators
  • Report more meaningful clinical outcomes over longer timeframes
  • May be a better source of information about harms

 

Slide 29

Foreign Literature

• Positive findings may be more likely to be published in high-profile journals published in English
• Therefore, to include only English may overestimate the positive effect of an intervention
• Empirically, the bias associated with limiting one's review to English only has been shown to be small
• Some content areas may be more heavily published in the foreign literature (e.g., some CAM interventions)

 

Slide 30

Cesarean Delivery

• NIH-OMAR State-of-the-Science conference on outcomes of cesarean delivery on maternal request

 

Slide 31

Exercise 3

• Define the appropriate population group and comparator.
• What other limitations would you put on included literature?

 

Slide 32

The challenge

• No evidence on outcomes of CDMR vs other modes of delivery
• Urgent need for actionable evidence
• Need to recognize and account for confounders

 

Slide 33

Solution

• Expand search to include proxies, weight rungs of evidence to account for confounding
  • Highest rung: Trials of breech delivery, but only for maternal outcomes
  • Next rung: Planned cesarean versus planned vaginal
  • Lowest rung: Comparisons of maternal and neonatal outcomes from actual modes of delivery

 

Slide 34

Summary

• Selection criteria (aka inclusion/exclusion) are an essential tool for setting the scope of the review
• They should be tied to the analytic framework, key questions and PICOTS
• When properly applied, selection criteria can reduce bias and support the applicability of the review