Updating Surveillance System: Assessing the Need for Updating Comparative Effectiveness Reviews

Select to access the PowerPoint® presentation (195 KB).

Slide 1

Updating Surveillance System: Assessing the Need for Updating Comparative Effectiveness Reviews

The University of Ottawa Evidence–based Practice Center The RAND Southern California Evidence-based Practice Center

Alexander Tsertsvadze MD, MSc
Date: September 11, 2012

Image: The Ottawa Hospital Research Institute logo is shown.

Slide 2

Outline

• Background:
  • Why update systematic reviews (SRs)? What is an update of a SR?
  • When/how to update a SR?
  • AHRQ updating surveillance project.
• Methods:
  • Overview of the process.
  • The Ottawa/RAND EPC methodology.
• Findings and implications:
  • Brief overview.

Slide 3

Background

Why update SRs?

SRs play a central role in comparative effectiveness research.

With the emergence of new evidence over time, conclusion(s) of any given SR may become out of date and sometimes misleading.

Changes in the evidence can have significant implications for clinical practice guidelines and for clinical and consumer decisionmaking.

Slide 4

Background

What is an update of SR?

"A discrete event with the aim to search for and identify new evidence to incorporate into a previously completed systematic review".

An update must involve a search for new studies.

Moher and Tsertsvadze. Lancet 2006; 367:881-3
Cochrane Hand Book section 3.2.2

Slide 5

Background

Shojania 2007

Sets of quantitative/qualitative signals indicating need for updating tested on 00 SRs; within 2 yrs, 23% of SRs had signals indicating the need for updating; the median time to the emergence of signal indicating the need for updating was 5.5 years.

French 2005

Of updated SRs (n=254), 9% (n=23) had changes in conclusions within 4 yrs.

Garritty 2010

53/103 of respondents estimated >50% of their respective SRs were likely out-of-date.

Slide 6

Background

One SR found very few methods and strategies used for updating SRs.

The authors identified the need for standardized reliable/efficient method(s) for updating .

(Moher and Tsertsvadze 2007; Tsertsvadze 2011).

Slide 7

Background

In 2007-09, Ottawa and RAND Evidence-based Practice Centers (EPCs) within the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program developed a methodology for determining the need of updating Comparative Effectiveness Reviews (CERs)

• Abbreviated searches for new evidence.
• Identification of signals indicating the need of updating by applying a priori set criteria to the new evidence.
• Expert opinion on validity of CER conclusions.
• Topic-specific safety alerts from FDA and Health Canada.
• Rules/guidance for determining the updating status of conclusion(s) for each key question (KQ) within CER and updating priority for each CER (low, medium, high).

(Shojania 2007; Shekelle 2009).

Slide 8

Background

Objectives

• Recognizing the importance of keeping CERs up-to-date, the AHRQ commissioned the RAND and Ottawa EPCs in collaboration with ECRI Institute to implement an updating surveillance system project.
• This collaborative project would use the previously developed methodology to assess already published CERs (n=42) for the need of updating.

- CERs unlike SRs contain several key questions/conclusions.
- Updating vs. assessing the need for updating.

Slide 9

Updating surveillance process

Image: A flowchart shows the surveillance process:

Start of Review.

• Original CER →
  • ECRI surveillance → FDA/Health Canada alerts →
    • Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER →
      • Low priority & Medium priority → Goes to 2nd cycle of assessment 6 months later (Return to Start of Review)
      • High priority → Refereed for updating.
  • Abbreviated searches for new evidence using the same strategy as in the CER → Screening and extracting data from relevant studies → Identifying qualitative/quantitative signals → Signal detection completed →
    • Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER →
      • Low priority & Medium priority → Goes to 2nd cycle of assessment 6 months later (Return to Start of Review)
      • High priority → Refereed for updating.
  • Contacting experts → Expert opinion →
    • Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER →
      • Low priority & Medium priority → Goes to 2nd cycle of assessment 6 months later (Return to Start of Review).
      • High priority → Refereed for updating.

Third month: End of Review.

Slide 10

Methods

Abbreviated search for new evidence

Image: A box contains the following text, "Search strategies—the same as used in the original CER but limited to...".  Below this box are two arrows each pointing to another text box. The first reads "5 general medicine journals (Ann Intern Med, BMJ, JAMA, Lancet and NEJM)"; the second  reads "5 specialty journals (specific to a topic of CER; most frequently cited journals in CER) ."

Slide 11

Image: The flowchart of the surveillance process shown on Slide 9 is repeated.

Slide 12

Methods

Contacting content experts

Images:  An "Ask The Expert" icon is shown. A cartoon shows a man seated between two computers; the first computer’s screen says "Expert" and the second says "Second Opinion."  The following text in boxes connected by arrows is superimposed over a table of Key Questions for a CER:

Expert Opinion:

• CER-specific clinical experts (content experts, technical expert panel members, expert peer reviewer).
• Other clinical experts (external, local).
• Feeds into the decision on updating status of KQ specific conclusion(s) in each CER.

Slide 13

Image: The flowchart of the surveillance process shown on Slide 9 is repeated.

Slide 14

Methods

Safety surveillance alerts

Image: A box contains the following text, "FDA/Health Canada alerts on CER topics (monthly)".  Below this box are two arrows each pointing to another text box. The first reads "On CERs assigned to RAND EPC)"; the second reads "On CERs assigned to Ottawa EPC." Arrows point from both of these boxes to another box beneath them which contains the text "Feeds into the decision on updating status of KQ specific conclusion(s) in each CER."

Slide 15

Image: The flowchart of the surveillance process shown on Slide 9 is repeated.

Slide 16

Methods

Image: A box contains the following text, "Identifying signals for updating".  Below this box, an arrow points downward to a second box containing the text, "Qualitative signals". Below "Qualitative signals" are two arrows each pointing to another text box. The first contains the following text:

Potentially invalidating change in evidence

[pivotal trial/MA]

• Opposing findings in effectiveness.
• Substantial harm.
• A superior new treatment.

The second box contains the following text:

Major change in evidence

[pivotal trial/MA]

• Important changes in effectiveness short of opposing findings.
• Clinically important expansion of treatment.
• Clinically important caveat.

[non-pivotal trial/MA]

• Opposing findings in effectiveness/harms.

Slide 17

Methods

Image: A box contains the following text, "Identifying signals for updating".  Below this box, an arrow points downward to a second box containing the text, "Quantitative". Below "Quantitative" are two arrows each pointing to another text box. The first contains the following text:

Change in statistical significance

• Significant vs. non-significant.
• Non-significant vs. significant.

The second box contains the text: "At least 50% change in effect size."

Slide 18

Image: The flowchart of the surveillance process shown on Slide 9 is repeated

Slide 19

Methods

Rules for assessing updating status of conclusion(s) for each Key Question

Image: A table shows the following criteria:

Up to date:

• No new evidence.
• Only confirmatory evidence.
• All responding experts: conclusions as still valid.

Possibly out of date:

• Some new evidence.
• A minority of responding experts: having new evidence that might change the conclusion.

Probably out of date:

• Substantial new evidence.
• Majority of responding experts: having new evidence that might change the conclusion.

Out of date:

• New evidence that rendered the CER conclusion out of date or no longer applicable.
  • e.g., withdrawal of a drug or surgical device from the market, a black box warning from FDA, etc.

Slide 20

Methods

Rules for assigning the updating priority to a CER
high, medium, or low

Image: A chart shows the following rules:

How many conclusions?

• Out of date.
• Possibly out of date.
• Probably out of date.

How out of date are conclusions?

• Magnitude/direction of changes in estimates.
• Potential changes in practice or therapy.
• Drugs withdrawn from the market/black box warning.
• Availability of a new treatment.

Slide 21

Methods

• A brief report summarizing the findings from the surveillance process.
  • Title page lists the final priority for updating ("low," "medium," or "high").
  • Description of methods and results.
  • Summary table by conclusion:
    • Findings of the literature search.
    • Findings from FDA and Health Canada.
    • Expert opinion.
    • Assessment of the degree to which the conclusion may be out-of-date.
  • Evidence table.

Slide 22

Findings

1st year of surveillance

• 14 CERs were assessed for updating status and were assigned updating priorities as follows:
  • 2 (14.2%) = 'high priority'.
  • 3 (21.4%) = 'medium priority'.
  • 9 (64.4%) = 'low priority'.

Slide 23

Implications

• We have established a Surveillance Program that has evaluated 14 CERs over the course of 12 months.
• We have shown that a program for regular and active surveillance of CERs is feasible.

Slide 24

Implications

• To our knowledge, the surveillance assessment project is the first large scale effort that has applied a set of methods to assess the currency of evidence-based reports (or systematic reviews) in a structured and standardized manner.
• The application of this methodology proves to be a relatively feasible, efficient, and at the same time, comprehensive approach for assessing the need for updating individual CERs across a wide range of health interventions.

Slide 25

References

• Moher D, Tsertsvadze A. Systematic Reviews: When is an Update an Update? Lancet 2006; 367: 881-883
• Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey. PloS one 2010; 5(4):e9914.
• French SD, McDonald S, McKenzie JE, Green S. Investing in updating: how do conclusions change when Cochrane systematic reviews are updated? BMC Med Res Methodol 2005;5:33.
• Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M et al. A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol 2007; 60(11):1095-10e1.
• Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 2007;147:224e33.
• Shekelle P, Newberry S, Maglione M et al. Assessment of the need to update comparative effectiveness reviews: Report of an initial rapid program assessment (2005-2009) [Internet]. 2009 Sep 10.
• Shekelle PG, Newberry SJ, Wu H, Suttorp M, Motala A, Lim Y-W, Balk EM, Chung M, Yu WW, Lee J, Gaylor JM, Moher D, Ansari MT, Skidmore R, Garritty C. Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods. Rockville (MD): Agency for Healthcare Research and Quality. June 2011. Methods Research Report. AHRQ Publication No. 11-EHC042-EF.
• Tsertsvadze A, Maglione M, Chou R, Garritty C, Coleman C, Lux L et al. Updating comparative effectiveness reviews: current efforts in AHRQ's Effective Health Care Program. J Clin Epidemiol 2011; 64(11):1208-1215.

Slide 26

Thank You!

Image: The Ottawa Hospital Research Institute logo is shown.