Grantsmanship Session (Text Version)

Slide Presentation from the AHRQ 2008 Annual Conference

---

On September 9, 2008, Francis Chesley, Debbie Rothstein, Kishena Wadhwani and Patrick McNeilly, made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (3 MB).

---

Slide 1

Office of Extramural Research, Education and Priority Populations (OEREP) Grantsmanship Session

• Overview: Francis Chesley, MD—OEREP Director.
• Prepare a Grant Application: Debbie Rothstein, PhD—Senior Advisor to the Director, OEREP.
• AHRQ Peer Review Process: Kishena Wadhwani, PhD, MPH—Div. Scientific Review Director, OEREP.
• Human Subjects Protections and Inclusions: Patrick McNeilly, PhD—AHRQ Institutional Review Board (IRB) Official, OEREP.

Slide 2

Preparing a Grant Application: Steps to Success

• Debbie Rothstein, Ph.D.
  Senior Advisor, Extramural Research.
  Office of Extramural Research,
  Education and Priority Populations
  Agency for Healthcare Research and Quality (AHRQ)
  September 9, 2008.

Slide 3

Getting Started

• Internet is funding gateway.
• AHRQ and National Institutes of Health (NIH) use essentially the same application processes and grant mechanisms.
• Keep abreast of open solicitations and Agency research priorities.

Slide 4

AHRQ's Web site—http://www.ahrq.gov

• Funding Opportunities
  • AHRQ Research Agenda.
  • Funding Announcements.
  • Research Policies.
  • Grants Process.
  • Grant Application Basics.
  • Training and Education.
  • Contracts.

Slide 5

AHRQ and NIH: Grant Application Similarities

• Application forms: Standard Form 424 (Research and Related) [SF 424(R&R)], PHS 398.
• Application submission dates.
• Use of internet as the funding gateway to keep abreast of research priorities and open solicitations, staff can provide technical assistance.
• Announcement publication: NIH Guide for Grants: http://grants1.nih.gov/grants/guide/index.html and Grants.Gov: http://www.grants.gov

Slide 6

AHRQ and NIH: Grant Application Similarities (continued)

• Receipt and referral of grant applications.
• Application review procedures.
• Grant mechanisms—e.g., R01, R03, R13, R36, K01, K02, K08.
• Transition to electronic submission of applications.
• Some common Policy Notices and Funding Opportunity Announcements.

Slide 7

AHRQ and NIH: Grant Application Differences

• Detailed budget vs. modular budget.
• Conference grants—AHRQ has small and large opportunities.
• AHRQ doesn't allow multiple PI's.
• Budget limitations
  • Large grants (R01, R18)—$300K vs. $500K.
  • Small grants (R03)—$100K vs. $50K.
• Fewer grant mechanisms used at AHRQ.

Slide 8

AHRQ Grant Opportunities

• AHRQ announces availability of grants in a Funding Opportunity Announcement (FOA).
• FOAs are published in the NIH Guide for Grants (http://grants.nih.gov/grants/guide/index.html) and posted on Grants.gov (http://www.grants.gov/)
• Program Announcement (PA):
  • Describes broad research interests.
  • Open over a period of time.
• Request for Applications (RFA):
  • More well defined area, specific program objectives.
  • Single application receipt date and amount of funds.

Slide 9

Common AHRQ Grant Opportunities

• R01: large research grants:
  • Feb 5, June 5, Oct 5 application receipt dates.
  • Up to $300K/yr.
• R18: research demonstration grants:
  • Jan 25, May 25, Sept 25 application receipt dates.
  • Up to $300K/yr.
• R03: small research projects:
  • Feb 16, June 16, Oct 16 application receipt dates.
  • Up to $100K in total costs.
• Large and small conference grants (R13):
  • Large: Apr 12, Aug 12, Dec 12 receipt; up to $100K/yr in total costs.
  • Small: Feb 20, Apr 20, Jun 20, Aug 20, Oct 20, Dec 20 receipt; up to $50K.

Slide 10

AHRQ Grant Opportunities: Training/Career Development

• Pre and Postdoctoral Training:
  • Institutional Training Programs (T32).
  • Individual Pre-doctoral Fellowships to Promote Diversity (F31).
  • Individual Post-doctoral Fellowships (F32).
  • Dissertation Grants (R36).
• Career Development Awards:
  • Mentored Scientist Awards (K01).
  • Independent Scientist Awards (K02).
  • Mentored Clinical Scientist Awards (K08).
• Detailed information at:
  
• Send questions to: training@ahrq.hhs.gov

Slide 11

AHRQ Staff Involved in Grants Process

• Referral Officer.
• Review Staff—Scientific Review Officer and Grants Management Specialist.
• Program Staff—Project Officer (PO).
• Grants Management Staff.

Slide 12

Electronic Application and Submission

• Transition to Electronic Receipt began Dec. 2005.
• Vast majority of all competing applications to AHRQ are now submitted electronically using SF 424 (R&R) application.
• All major mechanisms used by AHRQ have transitioned except for the training grants.

Slide 13

What is Grants.gov?

• Federal government's single, online portal to electronically:
  • Find Grant Opportunities.
  • Apply for Grants.
• A cross-agency initiative spanning 900 grant programs from the 26 grant-making agencies, and over $350 billion in annual awards.

Slide 14

Electronic Submission of Grant Applications

• With this new process, it's critical to read and follow all instructions in the announcement and application guide.
• Applicant must prepare for e-submission by completing mandatory registrations.
  • Grants.gov: registration for organization.
  • ERA Commons: registration for organization and the PI.

Slide 15

SF424 (R&R) Application Form

• Provides a consistent electronic submission process through Grants.gov.
• Consists of common data elements, arranged in components.
  • Not all components will be used for every Funding Opportunity Announcement (FOA).
• Each FOA will have the appropriate application package attached.
• From data files, application image is generated.
  • Applicant should view and verify the application image.

Slide 16

Electronic Application and Submission—General Information and Help Links

• SF424 (R&R) Application and Electronic Submission Information:
  http://grants.nih.gov/grants/funding/424/index.htm
• General information on Electronic Submission of Grant Applications:
  http://era.nih.gov/ElectronicReceipt/
• Finding Help:
  http://era.nih.gov/ElectronicReceipt/support.htm

Slide 17

Paper No More, Use 424 (R&R)

The slide is divided in half by a dotted line. Under the title, "Old," are two photographs: one of tree trunks and one of a room full of bundled paper. Arrows point to the right side, where under the title, "New," are two photographs: one of a CD and one of an empty room.

Slide 18

What Determines Which Awards Are Made?

• Scientific merit
  • Significance and originality.
  • Methods.
• Program / Agency considerations
  • What is uniquely AHRQ.
  • Existing research portfolio balance.
  • Anticipated IMPACT of research.
• Availability of funds.

Slide 19

Ingredients of a Successful Grant Application

• Good Idea
• Good Science
• Good Application
• Fits Agency Research Priorities

Slide 20

AHRQ Peer Review Process

• Kishena C. Wadhwani, Ph.D., M.P.H.
  Director, Division of Scientific Review (DSR)
  Kishena.wadhwani@ahrq.hhs.gov
  Phone: (301) 427-1556
• Office of Extramural Research, Education and Priority Population (OEREP)
  Agency for HealthCare Research and Quality (AHRQ)
  Department of Health and Human Services (HHS)

Slide 21

Organizational Chart

Office of Extramural Research, Education and Priority Populations (OEREP).
Division of Scientific Review (DSR).
Current Study Review Group (SRG) Clusters and the Corresponding SRO/GMS.

• OEREP Director: Dr. Francis Chesley
  • Receipts and Referrals.
  • Committee Management.
• DSR Director: Dr. Kishena Wadhwani.
  • Study Section Clusters
    • Health Care System Research (HSR).
      • SRO: Dr. Carl Ohata.
      • GMS: Maureen Gallagher.
    • Health Care Quality & Effectiveness Research (HCQER).
      • SRO: TBD.
      • GMS: Ms. Carmen Colston.
    • Health Care Technology and Decision Sciences (HCTDS).
      • SRO: Dr. Boris Aponte.
      • GMS: Ms. Rebecca Trocki.
    • Health Care Research Training (HCRT).
      • SRO: Ali Azadegan.
      • GMS: Ms. Diane Manuel.

Slide 22

AHRQ Health Services Research SRG and its Primary Research Foci

• Health Care Systems Research (HSR)
  • Translational Research.
  • Implementation Research.
  • Access and Utilization.
  • Cost/Finance/Markets.
  • Qualitative & Quantitative Methods.
  • Delivery Systems.
  • Organizational Studies.
  • Infrastructure Building Research.
  • Provider Workforce.
• Health Care Technology and Decision Sciences (HCTDS)
  • Translational Research.
  • Implementation Research.
  • Technology/HIT Assessment.
  • Information Sciences; Dissemination.
  • Qualitative & Quantitative Methods.
  • Clinical/Translational Decision-making.
  • Patient Utilities, Morbidity & Function.
  • Outcome Research.
  • Healthcare Quality & Cost Improvement.
• Health Care Quality & Effectiveness Research (HCQER)
  • Translational Research.
  • Implementation Research.
  • Healthcare Quality & Effectiveness.
  • Cost-effectiveness Research.
  • Qualitative & Quantitative Methods.
  • Evidence-based Medicine.
  • Quality of Care Research.
  • Access to & Utilization of Care.
  • Clinical Outcome Research.
• Health Care Research Training (HCRT)
  • Translational Research.
  • Implementation Research.
  • Training & Career Development.
  • Training-related Conferences/Workshops.
  • Qualitative & Quantitative Methods.
  • Institutional Training & Career Development.
  • Demonstrations & Evaluations.
  • Outcome Research.
  • Healthcare-related IT Training.

Slide 23

Schedule of Study Section Meetings, FY08-FY09 (TENTATIVE) (Updated: 09/01/2008)

The table presents the tentative October 2008, March 2009, and June 2009 meeting dates for the following Study Sections:

• HSR: Ohata/Gallagher
  • October 24, 2008
  • March 5-6, 2009
  • June 24-25, 2009
• HCQER: TBD/Colston
  • October 30, 2008
  • February 23-24, 2009
  • June 18-19, 2009
• HCTDS: Aponte/Trocki
  • October 29, 2008
  • February 24, 2009
  • June 18-19, 2009
• HCRT: Azadegan/Manuel
  • October 23-24, 2008
  • March 5-6, 2009
  • June 24-25, 2009

Slide 24

Process of Review for New Individual Research Project Grant Applications

• Three overlapping cycles per year:
  • Submit in February: (June, October)
  • Review in June—July: (Oct - Nov, February - March)
  • SLT in July—August: (December, May - June)
  • Earliest award in December: (April, August - September)

Slide 25

Peer Review at NIH/AHRQ: Types of Committees (Study Sections)

• Standing Committees:
  • Chartered; multi-year commitment.
  • Temporary members added as needed.
  • Roster posted on NIH/AHRQ Web sites.
• Special Emphasis Panels (SEPs):
  • All Temporary (Ad-hoc) members.

Slide 26

Purpose of Peer Review

• To evaluate the scientific and technical merit of grant applications, providing information (Recommendations) used by the Agencies/Institutes and Centers to make funding decisions.

Slide 27

Reviewers consider

• Scientific and technical merit of proposed research/training.
• Qualifications of Principal Investigator(s) and research team.
• Availability of resources (Labs, Institutions).

Slide 28

Reviewers also consider (Continued)

• Reasonableness of requested budget for work proposed.
• Other factors (e.g., human subjects, animal welfare, inclusion policies/plans).

Slide 29

Reviewers do not consider

• Program relevance (Except if stated in FOA).
• Policy issues.
• Funding levels.
• Anticipated budget reductions.
• Comparisons with other applications.

Slide 30

Review Criteria

• For all R01, R03, R15, R21, and P01 subprojects:
  • Significance and Originality.
  • Approach—Methods and Data.
  • Innovation.
  • Organization of the Project.
  • Investigators.
  • Environment—Facilities and Resources.

Slide 31

Sole Basis of Review: Review Criteria

(The NIH 5—as an example)

• Significance: Does this study address an important problem? If the aims of the application are achieved, how will scientific knowledge be advanced? What will be the effect of these studies on the concepts or methods that drive this field?
• Approach: Are the conceptual framework, design, methods, and analyses adequately developed, well-integrated, and appropriate to the aims of the project? Does the applicant acknowledge potential problem areas and consider alternative tactics?
• Innovation: Does the project employ novel concepts, approaches or methods? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies?
• Investigator: Is the investigator appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the Principal Investigator and other researchers (if any)?
• Environment: Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of the unique features of the scientific environment or employ useful collaborative arrangements? Is there evidence of institutional support?

Slide 32

Also...

• Reviewers evaluate:
  • Protection of human subjects, the environment, and animal welfare.
  • Inclusion of women, minorities, and children.
  • Inclusion of AHRQ Priority Populations.
  • Budget.

Slide 33

Research Involving Human Subjects

Important Considerations that must be addressed in the application because they impact on priority score—considered to be part of the Approach.

• Are there any risks* to the human subjects?
• Are the protections adequate?
• Are there potential benefits to the subjects and to others?
• What is the importance of the knowledge to be gained?
• Are the plans for inclusion of minorities, both genders and children adequately addressed?
• Is the proposed study exempt from human subject review?
• No page limits.

• Note: * "Risks" include the possibility of physical, psychological, social injury or safety resulting from research.

Slide 34

AHRQ Requirement: Inclusion of Priority Populations

• Inclusion of Children?—Rationale?
• Inclusion of Elderly?—Rationale?
• Inclusion of Rural?—Rationale?
• Inclusion of Inner City?—Rationale?
• Inclusion of Low Income?—Rationale?
• Inclusion of Disabled?—Rationale?
• Inclusion of Chronic Care?—Rationale?
• Inclusion of End of Life?—Rationale?

Slide 35

Review Criteria (Continued)

• For FOAs (RFAs or PAs), modified or additional criteria may be specified.
• For other mechanisms, specific criteria apply.

Slide 36

Peer Review Process

• Each application assigned to 3 reviewers for written comments.
• Streamlined Review may be used.
• Group discussion of each application, including budget recommendation.
• Recommended score range.
• Reviewers privately assign priority score.

Slide 37

Peer Review Group Actions

• Unscored (Streamlined Review).
• Scientific Merit Rating (Priority Score).

Slide 38

Priority Scores/ Percentiles

• Priority scores range from:
  100 (Best) to 500 (Worst)
  [100 x (1.0 - 5.0)]
• In Streamlined Review, those in lower half are eliminated (unscored).
• Percentiles calculated to normalize scoring behavior across review groups.

Slide 39

Priority Scores Assigned by Reviewers

• SCORED (stronger): 1.0 - 1.4 (Outstanding)
                                   1.5 - 2.0 (Excellent) (25%)
                                   2.0 - 3.0                  (25%)
• UNSCORED (weaker ):   3.0 - 4.0
                                           4.0 - 5.0           (50%)

Slide 40

Hints: Preparing an Application

• Start early.
• Read and follow instructions.
• Clarify any confusing instructions early.
• Do not assume staff or reviewers "will know what you mean."
• Include well designed tables and figures.
• Format—consider the reviewers.
• Proofread and check before sending.

Slide 41

Preparing an Application (Continued)

• Explicitly state the purpose of the proposed work.
• Refer to the literature thoroughly but thoughtfully.
• Present an organized, lucid write-up.
• Be mindful of the review criteria to be used.
• Get advice from colleagues.

Slide 42

Art of Persuasion: Assumptions

• That the reviewers are knowledgeable and committed to doing a thorough job of evaluating each application.
• That the reviewers have less time to complete the task than desirable.
• That good formatting will assist reviewers to remember your organization and the thrust of your arguments.

Slide 43

Strategies to Strengthen Applications: The Short List

• Ask scientist with AHRQ support to critically review your application prior to submission—more than once.
• Talk with an Agency (AHRQ, etc.) program officer(s).

Slide 44

Strengthening Applications—The Long List:

• Significance of problem: Best if transparent yet not insultingly obvious.
• Scientific Rationale: New &/or original ideas clearly identified.
• Include a plan for acquiring needed expertise, if needed.
• Discuss the limitations of the chosen approach.
• Include abundant detail about the methodology.
• Scope of work midway on the trivial—grandiose continuum.
• Describe the participants & their involvement in the study in detail.
• Include pilot or preliminary data.
• Describe future research directions.

Slide 45

Help Reviewers See the Merits

• Think like a reviewer—
• Learn as much as possible about The System (e.g., figure out the likely review group).
• Preempt criticism.
• Include collaborators who can compensate for your deficiencies.
• Learn as much as possible about what research projects in your area that have been funded by the NIH/AHRQ/Health Resources and Services Administration (HRSA)/Centers for Disease Control and Prevention (CDC)/etc.

• Note: CRISP Database: (http://crisp.oit.nih.gov)

Slide 46

http://crisp.oit.nih.gov

The screen shot shows the homepage from CRISP's Web site.

Slide 47

HCRT Grant Applications Reviewed and Funded in Fiscal Year (FY) 08

The table presents:

• Types: Reviewed; Funded; Percent Successful
  • F31: 2; 0; 0%
  • F32: 8; 2; 25%
  • K02: 5; 3; 60%
  • K08: 12; 6; 50%
  • R36: 39; 11; 28%
  • R01: 1; 1; 100%
  • R13: 1; 0; 0%
  • Total: 68; 23; 34%

Slide 48

"In God We Trust..All Others Must Bring Data."

Olivia Bartlett, Ph.D.
Chief, Research Programs Review Branch
Division of Extramural Activities, National Cancer Institute (NCI).

Slide 49

• Best wishes to your education and professional endeavors!
• Any questions?

Slide 50

Human Subjects Protections Human Subjects Protections

Overview for Applicants
Patrick McNeilly, Ph.D.
September 2008

Slide 51

Background

• Belmont Report
  • Respect for Persons.
  • Beneficence.
  • Justice.
• Common Rule (45 CFR 46).

• http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm

Slide 52

Common Rule Requires

• IRB Review.
• Informed Consent.
• Evaluations of all applications and proposals involving human subjects (45 CFR 46.120).

Slide 53

Peer Review vs IRB

• Peer review:
  • Evaluate the application.
  • Identify human subjects concerns in any portion of the application.
  • Justification of exemption.
• IRB:
  • Initial and continuing review of research.
  • Approve modifications to research.

Slide 54

Human Subjects Concern

• Any actual or potential unacceptable risk, or inadequate protection against risk, to human subjects as described in any portion of the application.
  • Captured in Summary Statement.
  • Administratively coded.

Slide 55

Common Peer Review Issues

• No human subject protection plan.
• Unidentified subjects.
• Unidentified risks.
• Informed consent issues.

Slide 56

Resolution of Issues

• Written response from Principal Investigator to all human subjects concerns.
• AHRQ Program Official will consult with human protections administrator on adequacy of response.

Slide 57

Exemption from HHS Regulations

• Six categories of exemption.
• Not all health services research is exempt.
• Is there an appropriate justification of exemption?

Slide 58

Inclusion of Women & Minorities

• Women and members of minority groups must be included.
• Unless a clear and compelling rationale to exclude.

• http://grants.nih.gov/grants/funding/women_min/women_min.htm

Slide 59

Priority Populations

• Inner-city; rural; low income; minority; women; children; elderly; special health care needs.
• Studies should consider including one or more AHRQ priority populations.
• AHRQ specific requirement.

• http://grants.nih.gov/grants/guide/notice-files/NOT-HS-03-010.html

Slide 60

Example 1

• The proposed study will provide a screening tool in the form of a Bruising Clinical Decision Rule (BCDR) for discriminating bruises caused by physical child abuse vs. accidental trauma. Success will result in a BCDR to function as a screening tool to identify children and infants with bruising who are at high risk for physical abuse and require further evaluation.

Slide 61

Example 1 (Continued)

• The proposed study is a prospective observational study of bruising characteristics in children over 4 years of age. Data on bruising characteristics will be collected by pediatric emergency medicine physicians and child abuse experts on 1,000 children with bruising.

Slide 62

Example 2

• In an application focusing on the differences in treatment of prostate cancer for whites compared to non-whites.

Slide 63

Example 2 (Continued)

• Reviewers Commented:
  • "Given the application's focus, prostate cancer, the exclusion of women is appropriate. However, the low minority representation and poor recruitment strategies for minorities (85% White, 7% Hispanic, 8% African American) would make it difficult for comparisons of treatment methods by racial and ethnic categories to be made."

Slide 64

• Questions?