Grading Evidence and Recommendations: Starting with GRADE Basics Vs. Utilizing the Full Framework (Text Version)

On September 28, 2010, Yngve Falck-Ytter made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (760 KB). 

---

Slide 1

Grading Evidence and Recommendations: Starting with GRADE Basics vs. Utilizing the Full Framework

AHRQ Annual Meeting 2010:
“Better Care, Better Health: Delivering on Quality for All Americans"

September 28, 2010

Yngve Falck-Ytter, M.D.
Associate Professor of Medicine
Case Western Reserve University, Cleveland, Ohio

Holger Schünemann, M.D., Ph.D.
Chair, Department of Clinical Epidemiology & Biostatistics
Michael Gent Chair in Healthcare Research
McMaster University, Hamilton, Canada

Slide 2

Disclosures

In the past 5 years, Dr. Falck-Ytter received no personal payments for services from industry. His research group received research grants from Three Rivers, Valeant and Roche that were deposited into non-profit research accounts. He is a member of the GRADE working group which has received funding from various governmental entities in the US and Europe, such as the AHRQ. Some of the GRADE work he has done is supported in part by grant # 1 R13 HS016880-01 from the Agency for Healthcare Research and Quality (AHRQ).

Slide 3

Content

Part 1

• A 7 minute version of GRADE.

Part 2

• Rapid interactive exchange contrasting GRADE basic vs. the full GRADE approach.
  • Advantages of a structured approach.
  • Asking good clinical questions.
  • Systematic review vs. ad hoc approaches.
  • Grading the quality of evidence.
  • How to determine the strength of recommendations.

Slide 4

Question to the audience

Decisions in your medical practice are based on:

1. Training, experience and knowledge of respected colleagues.
2. Patient preferences.
3. Convincing evidence (non experimental) from case reports, case series, disease mechanism.
4. RCTs, systematic reviews of RCTs and meta-analyses.
5. All of the above.

Slide 5

Evidence-based clinical decisions

Venn diagram showing the intersections of Clinical circumstances, Patient values and preferences, and Research evidence combining to bring Expertise.

Haynes et al. 2002

Slide 6

A real world example...

P: In patients with acute hepatitis C ...
I: Should anti-viral treatment be used ...
C: Compared to no treatment ...
O: To achieve viral clearance?

Evidence	Recommendation	Organization
B	Class I	AASLD (2009)
II-1	“Should be initiated...”	VA (2006);
1+	A	SIGN (2006)
-/-	“Most authorities...”	AGA (2006)
-/-	B “It works...”	AWMF(2004)

Slide 7

Question to the audience

By now...

1. ...you are thoroughly confused
2. ...you send her to a doctor because treatment is recommended
3. ...you send her to a doctor but she can expect that, according to guidelines, she will not be treated
4. ...you look at the evidence yourself because past experience tells you that guidelines don't help

Slide 8

GRADE is outcome-centric

An image of the old system transitioning into the new GRADE system is shown.
 

Slide 9

Guideline development

The top half of the slide is diagram entitled "Systematic review". It show the steps of PICO:

1. Formulate question (PICO)
2. Select outcomes
3. Rate importance
4. Outcomes across studies
5. Create evidence profile with GRADEpro (Summary of findings & estimate of effect for each outcome)
6. Rate quality of evidence for each outcome (High, Moderate, Low, Very low)
7. Rate overall quality of evidence across outcomes based on lowest quality of critical outcomes.

Rate after Step 6 (Rate quality of evidence for each outcome) is set of steps entitled "RCT start high, obs. data start low".

Grade down

1. Risk of bias
2. Inconsistency
3. Indirectness
4. Imprecision
5. Publication bias

Grade up

1. Large effect
2. Dose response
3. Confounders

Guideline development

Formulate recommendations:

• For or against (direction)
• Strong or weak (strength)

By considering:

• Quality of evidence
• Balance benefits/harms
• Values and preferences

Revise if necessary by considering:

• Resource use (cost)

• “We recommend using...”
• “We suggest using...”
• “We recommend against using...”
• “We suggest against using...”

Slide 10

Question to the audience

Which question follows a well structured clinical PICO format:

1. What is the evidence that food allergens cause eosinophilic esophagitis?
2. Is it known what the evidence is that aspirin can prevent progression of dysplasia to cancer in Barrett's esophagus?
3. In patients undergoing hip replacement, does warfarin compared to aspirin reduce venous thromboembolism, pulmonary embolism and mortality?

Slide 11

That's an excellent question

• Translating informal clinical questions into specific PICO questions = central to GRADE.
• Even if an organization has limited resources, taking care of this step actually saves resources:
  • Helps limiting your scope.
  • Specifies the search strategy more clearly.
  • Guides data extraction.
  • Helps with formulating recommendations.

Slide 12

Taking it to the next level

Informal Question	PICO Question				Method
	Population	Intervention(s)	Comparator(s)	Outcome(s)
Whether to use thrombo-prophylaxis for VTE prophylaxis (drugs)	Patients under-going THR	Any drug (ASA, LDUH, LMWH, fonda-parinux, direct thrombin inhibitors)	No anti-coagulation	Asymptomatic DVT (surrogate for symptomatic VTE); symptomatic DVT; non-fatal PE; fatal PE; bleeding (operative site vs. non-operative site); readmission; re-operation; total mortality	RCT, obs. studies

Slide 13

Importance of outcomes

Deciding on the importance of outcomes on decision making:

Less important

• 1
• 2
• 3

Important

• 4
• 5
• 6

Critically important

• 7
• 8
• 9

P: In patients after hip replacement...
I: Should warfarin rather than...
C: Aspirin be given...
O: To reduce symptomatic venous thromboembolism and mortality?

Slide 14

Question to the audience

Deciding on the importance of outcomes on decision making:

Less important

• 1
• 2
• 3

Important

• 4
• 5
• 6

Critically important

• 7
• 8
• 9

Please rate outcome: Dying from pulmonary embolism

1. (1, 2, 3): Less important for decision making
2. (4, 5, 6): Important for decision making
3. (7, 8, 9): Critically important for decision making

Slide 15

Question to the audience

Deciding on the importance of outcomes on decision making:

Less important

• 1
• 2
• 3

Important

• 4
• 5
• 6

Critically important

• 7
• 8
• 9

Asymptomatic deep vein thrombosis in the calf (e.g., as seen on mandatory venography at end of study)

1. (1, 2, 3): Less important for decision making
2. (4, 5, 6): Important for decision making
3. (7, 8, 9): Critically important for decision making

Slide 16

Deciding on the importance of outcomes on decision making:

Less important

• 1
• 2
• 3

Important

• 4
• 5
• 6

Critically important

• 7
• 8
• 9

Stomach ulcer bleeding requiring endoscopy

1. (1, 2, 3): Less important for decision making
2. (4, 5, 6): Important for decision making
3. (7, 8, 9): Critically important for decision making

Slide 17

Deciding on the importance of outcomes on decision making:

Less important

• 1
• 2
• 3

Important

• 4
• 5
• 6

Critically important

• 7
• 8
• 9

Regular blood work and dose adjustments

1. (1, 2, 3): Less important for decision making
2. (4, 5, 6): Important for decision making
3. (7, 8, 9): Critically important for decision making

Slide 18

Rating the importance of outcomes

• Train the content expert to understand that outcomes that are critical for decision making are identified.
• Rating is done before, during and after the evidence review.
• The rating may change in light of new information.

Slide 19

This slide is duplicate of Slide 9 above.

Slide 20

Taking it to the next level

• Early involvement of consumers in the guideline development process.
• Selecting systematic reviews that are known to make an effort to include consumer views (e.g., Cochrane etc.).
• Can be used to identify research gaps.

Slide 21

Evidence review stage

Image of a flowchart which starts at "What format of evidence do you use?"

Slide 22

Question to the audience

Select the best answer: You can find high quality systematic reviews for "free" here:

1. AHRQ
2. The Cochrane Library
3. Canadian Agency for Drugs and Technologies in Health (CADTH)
4. National Institute for Clinical Excellence (NICE), UK
5. All of the above

Slide 23

Taking it to the next level

• What to look for when selecting evidence review centers.
• Commissioning systematic reviews: Making sure the center understands GRADE requirements.
  • What SR methodology they use.
  • What databases they can search.
  • What software they use.
  • How they document their work.

Slide 24

Question to the audience

GRADE rating evidence: The quality of evidence may need downgrading if:

1. The outcome is reduction of elevated pressure in the eye (IOP) instead of loss of vision.
2. There are large losses to follow-up.
3. Some trials showing benefits, others reporting harms.
4. The confidence interval is wide and there are few events.
5. All of the above.

Slide 25

Quality of evidence: beyond risk of bias

Definition: The extent to which our confidence in an estimate of the treatment effect is adequate to support a particular recommendation

Methodological limitations

Risk of bias:

Allocation concealment
Blinding
Intention-to-treat
Follow-up
Stopped early

Inconsistency of results

Underneath this category is an Odds Ratio graph.

Indirectness of evidence

Sources of indirectness:

Indirect comparisons
Patients
Interventions
Comparators
Outcomes

Imprecision of results

Underneath this category is an Odds Ratio graph.

Publication bias

Underneath this category is an Odds Ratio graph.

Slide 26

Quality assessment criteria

Study design	Quality of evidence	Lower if ...	Higher if...
Randomized trials	High	Study limitations (design and execution)	What can raise the quality of evidence?
	Moderate	Inconsistency
Observational studies	Low	Indirectness
	Very low	Imprecision
		Publication bias

Slide 27

Question to the audience

A systematic review of observational studies showed a relationship between front sleeping position (versus back position) and sudden infant death syndrome (SIDS): OR 2.93 (1.15, 7.47). Rate the quality of evidence for the outcome SIDS:

1. High
2. Moderate
3. Low
4. Very low

Slide 28

Question to the audience

You review all colonoscopies for average risk screening in your health system and document a percentage of patient who developed a perforation after the procedure (evidence of free air on imaging). No comparison group without colonoscopy available. Rate the quality of evidence for the outcome perforation:

1. High
2. Moderate
3. Low
4. Very low

Slide 29

Question to the audience

Several RCTs have shown the effectiveness of natalizumab to induce remission in Crohn's disease. Study/post-marketing data showed 31 cases of potentially lethal progressive multifocal leukoencephalopathy (PML, JC virus related). Rate the quality of evidence for PML:

1. High
2. Moderate
3. Low
4. Very low

Slide 30

Quality assessment criteria

Study design	Quality of evidence	Lower if ...	Higher if...
Randomized trials	High	Study limitations (design and execution)	Large effect (e.g., RR 0.5) Very large effect (e.g., RR 0.2)
	Moderate	Inconsistency	Evidence of dose-response gradient
Observational studies	Low	Indirectness	All plausible confounding would reduce a demonstrated effect
	Very low	Imprecision
		Publication bias

Slide 31

"Categories" of quality (1)

High: Further research is very unlikely to change our confidence in the estimate of effect

Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

Very low: Any estimate of effect is very uncertain

Slide 32

Conceptualizing quality (2)

High: We are very confident that the true effect lies close to that of the estimate of the effect.

Moderate: We are moderately confident in the estimate of effect: The true effect is likely to be close to the estimate of effect, but possibility to be substantially different.

Low: Our confidence in the effect is limited: The true effect may be substantially different from the estimate of the effect.

Very low: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect.

Slide 33

Taking it to the next level

• Advantages of systematically assessing quality of evidence
• Downgrading and upgrading "on-the-fly" can introduce errors

Study / year	Treatment	Allocation concealment	Blinding	No outcome (%)	Analysis	Comments
REMOBILIZE 2009	dabigatran 220 mg QD dabigatran 150 mg QD enoxaparin 30 mg BID	Yes (IVRS) (blocks of 6)	Patients: Y Caregivers: Y Data coll: PY Adjudic: Y Data analysts: ?	269/862 (31.2%) 232/877 (26.5%) 239/876 (27.3%)	ITT: no	Low dose ASA and stocking allowed, but not pneumatic devices

Slide 34

GRADE evidence profile

An image of a GRADE evidence profile table is shown for the question "Should antibiotics vs. no antibiotics be used for children with otitis media?".

Slide 35

Question to the audience

PICO: Should children with otitis media be treated with antibiotics?
Rate the overall quality of evidence for this clinical question by evaluating all critical outcomes (use the evidence profile):

1. High
2. Moderate
3. Low
4. Very low

Slide 36

Image of the steps involved in the GRADE process.

Slide 37

Question to the audience

PICO: Should children with otitis media be treated with antibiotics?

Rate the overall strength or recommendations:

1. "We recommend early antibiotics in children with acute otitis media."
2. "We suggest early antibiotics."
3. "We suggest against using antibiotics initially."
4. "We recommend against using antibiotics initially."

Slide 38

Strength of recommendation

"The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects."

Slide 39

4 determinants of the strength of recommendation

Factors that can weaken the strength of a recommendation	Explanation
Lower quality evidence	The higher the quality of evidence, the more likely is a strong recommendation.
Uncertainty about the balance of benefits versus harms and burdens	The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely is a weak recommendation warranted.
Uncertainty or differences in patients' values	The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Uncertainty about whether the net benefits are worth the costs	The higher the costs of an intervention—that is, the more resources consumed—the less likely is a strong recommendation warranted.

Slide 40

Implications of a strong recommendation

• Patients: Most people in this situation would want the recommended course of action and only a small proportion would not.
• Clinicians: Most patients should receive the recommended course of action.
• Policy makers: The recommendation can be adapted as a policy in most situations.

Slide 41

Implications of a weak recommendation

• Patients: The majority of people in this situation would want the recommended course of action, but many would not.
• Clinicians: Be prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making.
• Policy makers: There is a need for substantial debate and involvement of stakeholders.

Slide 42

Taking it to the next level

• Explicit separation of quality of evidence from making recommendations.
• Correctly balancing the benefits against the undesirable effects.
• Special challenges: resource use.
• Increasing transparency in the process of making recommendations.

Slide 43

Question to the audience

Should patients with chronic hepatitis C be treated with interferon/ribavirin combination? There is high quality evidence for benefits and high quality evidence for harms.
Rate the overall strength or recommendations:

1. "We recommend treatment of chronic hepatitis C."
2. "We suggest treatment..."
3. "We suggest against treating patients..."
4. "We recommend against treating patients..."

Slide 44

Patient values & preferences

• In the absence of evidence, guideline panels have to function as surrogates to estimate values and preferences (V&P).
• Consumer involvement can help.
• Attaching V&P statements to guideline recommendations increases transparency.

Slide 45

Taking it to the next level

• Systematically searching the literature for studies of values and preferences.
• Systematic reviews of V&P.
• Querying the guideline panel to rate health utilities of outcomes using case scenarios.

Slide 46

Question to the audience

Please select the most appropriate answer. The reason you attended this session:

1. Just interested in the topic.
2. Have been involved in narrative evidence reviews, but have not used any formal grading system.
3. Have used a grading system but not GRADE.
4. Using or considered using GRADE.

Slide 47

Question to the audience

Please select the most appropriate answer. Selecting a system to rate the quality of evidence and strength of recommendations, such as GRADE:

1. Appears too expensive to implement.
2. Appears valuable, but still requires substantial upfront expense.
3. Appears to have some upfront cost but long-term savings.
4. I use GRADE—it has been paying off for me.

Slide 48

Basic dimensions

Guideline work aligns along 3 basic dimensions:

• High quality vs. low quality
• Fast vs. slow
• Expensive vs. cheap

Slide 49

Ideal vs. practical ad hoc GRADE approaches

Stage	Elements	Advantage	Comment
Ideal	Systematic review GRADE eTables Qual. of evidence Strength of recommendation	Follows highest standards Methodologically most rigorous Easily maintainable Fully transparent process	Access to methodologist Access to evidence centers Initially more resource intensive, long-term savings
Intermediary	Ad hoc review GRADE eTables Qual. of evidence Strength of recommendation.	Still retaining major advantages of the “ideal approach”	Risk of bias higher Access methodologist rec. Only minimal addl. cost
Initiation	Ad hoc review GRADE eTables Qual. of evidence Strength of recommendation	Option to fully “upgrade” to an “ideal approach” Foundation of a methodologically sound system	Risk of bias higher Access methodologist prn No additional cost

Slide 50

Sources of funding

• Funders may have an agenda.
• Industry—tricky.
• Foundations
• Public—AHRQ, criteria
  • EHC program fit (3: available, relevance for public payer, priority condition).
  • Importance (7: e.g., public interest etc.).
  • No duplication.
  • Feasibility.
  • Impact (6: e.g., addresses inequity).

Slide 51

Taking it to the next level

• Long term planning.
• Create a high quality guideline product.
• Attract high quality guideline panel.
  • Unconflicted methodologist (editor).
  • Content expert (deputy editor).
  • Content expert authors.
  • Health economists.

Slide 52

Taking it to the next level

• GRADE evidence profiles:
  • Condensed and standardized summary of evidence.
  • Are increasingly already created as part of a systematic review (e.g., Cochrane reviews).
  • Flexible presentation (e.g., as summary of findings tables).
  • Initial investment.
  • Long-term value.
  • GRADEpro software (tie-in with RevMan).
  • Avoids duplication of efforts across the globe.

Slide 53

Vision

1. Globalize the evidence, localize recommendations.
2. Focus on questions that are important to patients and clinicians.
3. Undertake collaborative evidence reviews.
4. Use a common metric to assess the quality of evidence and strength of recommendations.
5. Examined collaborative models for funding.

Slide 54

GRADE uptake

A page with 20+ government health agency and publishing company logos is shown.

Slide 55

Conclusion

Gaining acceptance as international standard because GRADE adds value:

1. Criteria for evidence assessment across a range of questions and outcomes.
2. Sensible, systematic, fostering transparency.
3. Balance between simplicity and methodological rigor.