A Tool for the Classification of Study Designs in Systematic Reviews of Interventions and Exposures (Text Version)

---

On September 15, 2009, Meera Viswanathan made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (432 KB).

---

Slide 1

A Tool for the Classification of Study Designs in Systematic Reviews of Interventions and Exposures

Meera Viswanathan, PhD
for the University of Alberta EPC
 

 

Slide 2

Steering Committee

• Ken Bond, UAEPC
• Donna Dryden, UAEPC
• Lisa Hartling, UAEPC
• Krystal Harvey, UAEPC
• P. Lina Santaguida, McMaster EPC
• Karen Siegel, AHRQ
• Meera Viswanathan, RTI-UNC EPC

 

Slide 3

Background

• EPC reports, particularly comparative effectiveness reviews, are increasingly including evidence from nonrandomized and observational designs
• In systematic reviews, study design classification is essential for study selection, risk of bias assessment, approach to data analysis (e.g., pooling), interpretation of results, grading body of evidence
• Assignment of study designs is often given inadequate attention

 

Slide 4

Objectives

• Identify tools for classification of studies by design
• Select a classification tool for evaluation
• Develop guidelines for application of the tool
• Test the tool for accuracy and inter-rater reliability

 

Slide 5

Objective 1: Identification of tools

31 organizations/individuals contacted
11 organizations/individuals responded
23 classification tools received
10 tools selected for closer evaluation
1 tool selected for modification and testing
 

 

Slide 6

Objective 2: Tool selection

• Steering Committee ranked tools based on:
  • Ease of use
  • Unique classification for each study design
  • Unambiguous nomenclature and decision rules/definitions
  • Comprehensiveness
  • Potentially allows for identification of threats to validity and provides a guide to strength of inference
  • Developed by a well-established organization

 

Slide 7

Objective 3: Tool development

• Three top-ranked tools:
  • Cochrane Non-Randomised Studies Methods Group
  • American Dietetic Association
  • RTI-UNC
• Incorporated positive elements of other tools
• Developed glossary

 

Slide 8

Objective 4: Testing round 1

Overall agreement (30 studies, 6 testers)	κ=0.26 (fair)
Graduate level training complete (3 testers)	κ=0.38 (fair)
Graduate level training in progress (3 testers)	κ=0.17 (slight)
Item agreement:
6/6 testers agreed	0
5/6 testers agreed	7 (23%)
4/6 testers agreed	5 (17%)
3/6 testers agreed	9 (30%)
2/6 testers agreed	8 (27%)
No agreement	1 (3%)

 

Slide 9

Objective 4: Testing round 1

• No clear patterns in disagreements
• Disagreements occurred at all decision points
• Tool vs. studies
• Variations in application of the tool

 

Slide 10

Objective 4: Reference standard

Overall agreement (30 studies, 3 raters)	κ=0.33 (fair)
Item agreement:
3/3 raters agreed	7 (23%)
2/3 raters agreed	14 (47%)
No agreement	9 (30%)

 

Slide 11

Objective 4: Testing round 2

Overall agreement (15 studies, 6 testers)	κ=0.45 (moderate)
Graduate level training complete (3 testers)	κ=0.45 (moderate)
Graduate level training in progress (3 testers)	κ=0.39 (fair)
Item agreement:
6/6 testers agreed	3 (20%)
5/6 testers agreed	2 (13%)
4/6 testers agreed	6 (40%)
3/6 testers agreed	2 (13%)
2/6 testers agreed	2 (13%)
No agreement	0

 

Slide 12

Discussion

• Moderate reliability, low agreement with reference standard
• Studies vs. tool as source of disagreement
  • Tool not comprehensive, e.g., quasi-experimental designs
  • Studies challenging, e.g., sample of difficult studies, poor study reporting
• To optimize agreement and reliability:
  • Training in research methods
  • Training in use of tool
  • Pilot testing
  • Decision rules

 

Slide 13

Next Steps

• Test within a real systematic review
• Further testing for specific study designs
• Further evaluation of differences in reliability by education, training, and experience

 

Slide 14

Acknowledgments

• Ahmed Abou-Setta
• Liza Bialy
• Michele Hamm
• Nicola Hooton
• David Jones
• Andrea Milne
• Kelly Russell
• Jennifer Seida
• Kai Wong
• Ben Vandermeer (statistical analysis)

 

Slide 15

Questions?

University of Alberta EPC.Edmonton, Alberta, Canada