Meeting the Challenge of Maintaining Cochrane Reviews

Select to access the PowerPoint® presentation (480 KB).

Slide 1

Meeting the Challenge of Maintaining Cochrane Reviews

Author: Dr David Tovey FRCGP
Editor in Chief, The Cochrane Library

Image: The Cochran Collaboration logo is shown.

Slide 2

Plan

• Introduction.
• Previous Cochrane project.
• "Fit for purpose" project.
• Future challenges and opportunities for co-operation.
• Questions.

Slide 3

Acknowledgements

• Sally Hopewell, Yemisi Takwoingi, Alex Sutton, Rachel Marshall and Bazian Ltd.
• This project was funded by the National Institute for Health Research (NIHR) Cochrane—National Health Service (NHS) Engagement Award Scheme (project number 10/4000/01). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health.

Slide 4

Acknowledgements

• Cochrane Musculoskeletal Group.
• Cochrane Pain, Palliative and Supportive Care Group.
• Cochrane Infectious Disease Group.
• Cochrane Wounds Group.
• Cochrane Neonatal Group.
• Cochrane Airways Group.
• Karla Soares-Weiser.

Slide 5

Introduction: Protecting Archie Cochrane's vision

"It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials."

Image: A photograph of Archie Cochrane is shown.

Slide 6

Introduction: Protecting Archie Cochrane's vision

"It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials."

Image: The photograph of Archie Cochrane is shown again.

Slide 7

Introduction: the size of the challenge

Image: A table shows the number of new reviews and updates added to the Cochrane database in 2009, 2010, and 2011. The row of data showing new reviews for each year is highlighted by a red arrow:

• 2009 - 402.
• 2010 - 449.
• 2011 - 416.

Increase 2010-2009: 11.7%. Increase 2011-2010: -7.3%.

Slide 8

Introduction: the size of the challenge

Image: A table shows the number of new reviews and updates added to the Cochrane database in 2009, 2010, and 2011. The row of data showing total updates for each year is highlighted by a red arrow:

• 2009 - 479.
• 2010 - 524.
• 2011 - 468.

Increase 2010-2009: 9.4%. Increase 2011-2010: -10.7%.

Slide 9

Introduction: the size of the challenge

Image: A table shows the number of new reviews and updates added to the Cochrane database in 2009, 2010, and 2011. The row of data showing total active reviews for each year is highlighted by a red arrow:

• 2009 - 3958.
• 2010 - 4329.
• 2011 - 4713.

Increase 2010-2009: 9.4%. Increase 2011-2010: 8.9%.

Slide 10

Introduction: the size of the challenge

Image: A table shows the number of new reviews and updates added to the Cochrane database in 2009, 2010, and 2011. The row of data showing the percent of active reviews that are up to date (within 2 years) is highlighted by a red arrow:

• 2009 - 39.8%.
• 2010 - 36.3%.
• 2011 - 36.5%.

Increase 2010-2009: -3.5%. Increase 2011-2010: 0.6%.

"The boulder in my rucksack."

Slide 11

Previous Cochrane projects: the updating officer project

• 8 reviews selected.
• On average these reviews each took 6.4 months (range 3-11 months) to update from receipt of the search strategy to submission for editorial review.
• The main challenges:
  • Lack of familiarity with individual Review Groups' methods.
  • Out of date methodology within the existing reviews.
  • Lack of subsequent author commitment.
  • Lack of wish for ongoing commitment.
• Would imply the need for a small army of updating officers!!

Slide 12

"Fit for purpose" project

Three objectives of the project:

• To work with external stakeholders to develop a customisable prioritisation tool—prioritisation by topic.
• To develop a decision tool for determining whether and when to update Cochrane Reviews—prioritisation by status.
• To explore whether targeted consultancy could facilitate updating and identify process efficiencies.

Slide 13

"Fit for purpose" project

Three objectives of the project:

• To work with external stakeholders to develop a customisable prioritisation tool—prioritisation by topic.
• To develop a decision tool for determining whether and when to update Cochrane Reviews—prioritisation by status.
• To explore whether targeted consultancy could facilitate updating and identify process efficiencies.

Slide 14

Methods

• Project conducted by Bazian, co-applicants in the NHS engagement award.
• Stakeholders in the NHS (clinicians, consumers, commissioners etc.) formed a panel.
• The panel had two meetings, and communicated via email before and after meetings.

Slide 15

Results: prioritisation criteria in the tool

Image: A checklist form is shown, requiring a Yes (-1)/No(-0) answer for certain criteria such as Strategic Importance, Patient Importance/Impact, and National Spend.

Slide 16

Results: testing the tool

• The tool was tested on 19 Cochrane Reviews, which found:
  • There is a level of judgement when assigning scores.
  • A certain level of knowledge of the field is helpful when assigning scores.
  • Needs to be customised (notable absentee parameters..).
• Recommended that the results of using the tool are discussed with external stakeholders including patient and carer representatives.

Slide 17

"Fit for purpose" project

Three objectives of the project:

• To work with external stakeholders to develop a customisable prioritisation tool—prioritisation by topic.
• To develop a decision tool for determining whether and when to update Cochrane Reviews—prioritisation by status.
• To explore whether targeted consultancy could facilitate updating and identify process efficiencies.

Slide 18

Methods

• We refined and amalgamated two complementary methodologies for prioritising systematic review updates:
  • A qualitative tool based on a broad range of updating signals (Loudon 2008).
  • Formal statistical methods which assess when the inclusion of new studies is likely to change a review's conclusions (Sutton 2009).

Slide 19

Output: Decision tool

Image: The decision tool flowchart is shown. The key questions to be answered Yes or No are "Is the clinical question answered or no longer relevant?" "Are there any new factors to consider?" and "Are there new studies?"

Slide 20

Step 1: Is the clinical question answered or no longer relevant?

Image: The decision tool flowchart is shown. The first question, "Is the clinical question answered or no longer relevant?" and its Yes response, "Don't update," are circled in red.

Slide 21

Step 2: Are there any new factors to consider?

Image: The decision tool flowchart is shown. The second question, "Are there any new factors to consider?" and its Yes response, leading to further analyses or judgment, are circled in red.

Slide 22

Steps 3 & 4: Are there new studies? Are the conclusions likely to change?

Image: The decision tool flowchart is shown. The third question, "Are there new studies?" and its Yes and No responses, "Apply statistical prediction tool" and "Don't update yet," are circled in red.

Slide 23

Statistical prediction tool

Images: A screenshot shows the statistical prediction tool. The UB logo for METARANK, "a software tool for prioritizing the updating of systematic reviews" is also shown.

Slide 24

About metarank

• Based on minimal information on the new evidence:
  • Assumes an update strategy is in place such that number of new studies and their sample sizes are known.
• 'Signals' of the need to update implemented as a STATA user-written function.
• Performs simulation of several meta-analyses, each with one or more new studies of different sizes.

Slide 25

Metarank output

• Summary of results for each meta-analysis includes details of the original meta-analysis and the signals detected.
• Table with all reviews in the dataset ranked in order of priority:
  • By a given signal.
  • Or the total of all signals triggered.
• Includes descriptive information for the collection e.g.:
  • Average number of trials and participants in the collection of meta-analyses.
  • Number of trials in the largest meta-analysis.
  • Largest number of participants in a meta-analysis.

Slide 26

Images: A screenshot shows a report on Systematic Review Update Signals.

Slide 27

Advantages of metarank

• Assessment of the likelihood of any definable criteria changing in a meta-analysis.
• Ongoing studies can be taken into account:
  • Aid in predicting potential 'shelf life' of a review in the light of new or accruing evidence.
• To provide information with respect to further research needed such as the number of new trials and the number of participants:
  • Useful information to end users of reviews in assessing the stability or validity of a review.

Slide 28

Limitations of metarank

• Relies on availability of some information on new trials.
• Simplistic approach used in deriving the predictive distribution for new studies.
• Other issues external to a review may need to be taken into account such as:
  • The rate at which new evidence on a topic evolves.
  • Public heath significance etc.
• No criteria developed to establish when enough evidence has accrued on a given topic and review that it is deemed decisive and not worth conducting further primary research.

Slide 29

Decision tool: summary

• The decision tool provides a set of criteria that can be used to assess whether to update a Cochrane Review.
• The tool can be applied to a single Cochrane Review or can be used to prioritise a suite of reviews (e.g. those from an individual Cochrane Review Group).

Slide 30

Results: testing the tool

• One Cochrane Review Group (CRG) so far:
  • Cautiously positive—
    • "Helpful structure to assess each review's eligibility for updating."
    • "A transparent way to explain decisions around updating to stakeholders."
  • Time consuming but might improve with experience.
  • Some assumptions around the statistical tool "too crude."
  • Need to have wider experience and use.

Slide 31

"Fit for purpose" project

Three objectives of the project:

• To work with external stakeholders to develop a customisable prioritisation tool—prioritisation by topic.
• To develop a decision tool for determining whether and when to update Cochrane Reviews—prioritisation by status.
• To explore whether targeted consultancy could facilitate updating and identify process efficiencies.

Slide 32

Aims

• We aimed to provide short and focused periods of assistance, to incentivise authors, and to address barriers to updating.
• The intended approach was to retain both the responsibility for producing the update within the authors, and editorial support for within the CRGs.

Slide 33

Methods

• One-year project (July 2010 to July 2011).
• Involved 6 CRGs.
• Each CRG offered 37 days' of assistance for updating 1 to 6 Cochrane Reviews (between November 2010 and April 2011).
• Tasks performed by reviewers from inside and outside Cochrane without specific topic knowledge.

Slide 34

Tasks offered to CRGs and authors

Slide 35

Results: assessment of tasks

Slide 36

Results: CRG and author feedback

• Evaluation form: all CRGs responded, most authors responded (1 on holiday, 1 no response).
• Scale 1-5 (1 greatly improved, 5 greatly worsened).
  
  

  Criteria	Authors	CRGs
  Author motivation	1.6	1.8
  Speed of update	1.6	2.5
  Quality of update	1.8	2.75
  Overall impression of assistance	1.44	1.5

  
  
  
• All authors and managing editors who responded to the question on rolling out an updating service for The Cochrane Collaboration were in favour of an updating service (2 CRGs did not respond).

Slide 37

Results: status of updates

• We worked on 14 Cochrane Review updates.
• Mean time spent: 35.5 hours (median 26.71 hours, range 4.5 to 109.75 hours).
• Lots of activity but...
• In October 2011, no updates yet published.

Slide 38

Future challenges:

• Whose priorities?
• Expectations and the increasing professionalisation of systematic reviews.
• The rise and rise of methods advances.
• How to balance updates versus new reviews?
• Authorship of updates.

Slide 39

Conclusion

• Updating is critical to patient safety and credibility.
• Increasing recognition of need to prioritise (but this isn't as easy as it seems).
• Transparency to user/reader is critical.
• Different approaches but no "one size fits all" solution.

Slide 40

Conclusion: How can we work more effectively together?

• Sharing intelligence:
  • "Keep up" initiative.
  • Shared surveillance.
• Sharing the workload:
  • Sharing data.
  • "Wiki" approaches.
• Novel approaches:
  • Use of data mining and semantic technologies.
  • Limited search and "Summary updates."

Slide 41

Thank you for listening
dtovey@cochrane.org

Slide 42

Authorship of updates

• Scenario:
  A review is updated that originally included 11 studies and now has 17. The text includes 35% the same text as the previous version, across all sections. In addition a summary of findings table has been added. The author team has changed completely and none of the previous authors have contributed to the update.
• Questions:
  How should the work of the previous authors be credited?

Slide 43

International Committee of Medical Journal Editors (ICMJE) guidance on contributorship

Authorship credit should be based on:

1. Substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data.
2. Drafting the article or revising it critically for important intellectual content.
3. Final approval of the version to be published.

Authors should meet conditions 1, 2, and 3.

Slide 44

Authorship of updates

1. As authors of the updated review.
2. Lead author included on the update.
3. Should be negotiated on an individual basis.
4. Original authors not included as authors but acknowledged within the review.
5. Some other approach.