Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces: Overview of Project Findings (Text Versi

On September 28, 2010, Marsha Gold made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (1.1 MB).

---

Slide 1

 

Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces:
Overview of Project Findings

by

Marsha Gold, Sc.D., Project Director
Timothy Lake, Ph.D., Deputy Director
Kate Stewart, Ph.D., Researcher
Tara Krissik, M.P.P., Researcher

December 2008

This project was funded by the Agency for Healthcare Research and Quality.

Note: On the bottom of every slide is the logo for Mathematica Policy Research, Inc.

Slide 2

 

Rationale for Project

• While AHRQ's statutory mission explicitly includes support for grant funded research on healthcare costs, productivity, organization, and market forces, the agency does not systematically track the work it funds in this area and what has been accomplished through it.
• Purpose of this project is to address this gap and identify how AHRQ may improve the visibility and impact of this research.

Slide 3

 

Project Details

• MPR evaluation began in October 2006; the two phased project was completed in December 2008.
• Work was guided by an Expert Panel: Sharon Arnold (AcademyHealth), John Christianson (University of Minnesota), Paul Ginsburg (Center for Studying Health Systems Change), Robert Helms (AEI), and Gail Wilensky (Project Hope).
• Michael Hagan was the AHRQ project officer.

Slide 4

 

Key Questions of Interest

1. Since the late 1990s, what grant research has AHRQ funded that relates to healthcare costs, productivity, organization, and market forces?
2. How are the research findings disseminated to public and private decision makers and what factors contribute to their use?
3. What is AHRQ's role in supporting research in this area and how does it compare with that of others, such as NIH and private funders?
4. What actions, if any, could enhance AHRQ's efforts to track, disseminate and encourage use of these research findings?

Slide 5

 

Components of the Study—I

• Descriptive Analysis: Reviewed AHRQ administrative data on all grants, and conducted indepth analysis of a stratified random sample of nine grants.
• Conceptual Framework: Using relevant social science literature, identified pathways for translation and key variables that are likely to influence use of research.
• Case Studies: In depth description and analysis of 7 grants that illustrate diverse ways in which study findings reach target audiences and influence policy debates; based on PI and user interviews.

Slide 6

 

Components of the Study—II

• Comparative Funder Analysis: Comparison of AHRQ's infrastructure and funding for grants in this area to that of NIH (government) and HCFO (private sector).
• Federal Research Translator/User Interviews: Identify how and when this type of research gets used, and awareness of AHRQ research. (Included CBO, GAO, MedPAC, FTC, DOJ, and selected others).
• PI Survey: Web based survey in July-August 2008 to all 149 grantees (70% response). Detail on research focus and key findings, dissemination modes and preferences, uses made of results, and PIs interaction with policymakers. Also obtained feedback on AHRQ's role and performance and how AHRQ compares to other funders.

Slide 7

 

Components of the Study—III

Key Research Questions	Descriptive Analysis of Grants	Framework on Research Use	Case Studies of Use	Comparative Funder Analysis	Federal Research Translator Interviews	Principal Investigator Survey
1. What has AHRQ funded in these areas and how is it used?	X		X			X
2. How are the findings disseminated and what contributes to their use?		X	X		X	X
3. What is AHRQ's role in this research area and how does its performance compare with other funders?				X		X
4. What actions, if any, could enhance AHRQ's efforts to track, disseminate, and encourage use of these research findings?	X	X	X	X	X	X

Slide 8

 

Key Findings: What has AHRQ Funded in this Area between 1998-2006?

Slide 9

 

Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and Market Forces Funded, 1998-2006

Image: A bar chart shows the following data:

• 1998: 1
• 1999: 2
• 2000: 31
• 2001: 41
• 2002: 34
• 2003: 17
• 2004: 5
• 2005: 9
• 2006: 10

Source: MPR analysis of AHRQ Administrative Data.

Slide 10

 

AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and Market Forces, 1998-2006

Image: A bar chart shows the following data:

• 1998: $393,559
• 1999: $887,243
• 2000: $9,822,272
• 2001: $15,864,342
• 2002: $19,292,051
• 2003: $14,177,886
• 2004: $10,752,562
• 2005: $5,103,536
• 2006: $5,096,191

Source: MPR analysis of AHRQ Administrative data.

Note: Includes funding for new grants and continuing grants that were funded in 1998 or later. The combined spending over the period was $81.4 million, of which $12.6 million was for three PO1 grants. 47 RO3 grants under $100,00 accounted for about $4 million in spending.

Slide 11

 

Characteristics of Funded Studies

• Of the 149 studies:
  • 97 studies had organizations as the unit of analysis. Studies of hospitals and health plans were particularly common.
  • 37 studies examined consumer behavior.
  • 15 studies examined markets or purchaser behavior.
• 75 percent of the 149 were national in scope.
• Multiple outcomes studied including quality (53%), use (44%), cost (38%) and access (19%), among others.

Slide 12

 

Nature of Research

PI survey indicates that findings commonly examine how specific outcomes of care are influenced by:

• Economic factors (e.g., provider payment, insurance coverage)
• Organizational characteristics (e.g., nurse leadership, volume)
• Systems and markets (e.g., HMO penetration, capacity constraints)
• Patient characteristics and preferences (e.g., percent minority, DNR orders)

Findings address questions of substantial policy interest today about influences on the performance of health care system.

Slide 13

 

Key Findings: How are Findings Disseminated and What Factors Contribute to Use

Slide 14

 

Dissemination May Occur Through Diverse Pathways—I

• We identified 10 pathways that differ in means used to communicate findings and the role researchers, intermediaries and users play in the process. Pathways include:
  • Traditional: "Big Bang," Gradual Accumulation and Diffusion, Gradual Communication and Formal Syntheses
  • Effective Use of Intermediaries: Researcher as Messenger/Expert, Formal Intermediary Broker, Press Publicizes
  • User Guided: User Directed Syntheses, User Partner Grant Review, User Commissioned Studies, Researcher as User

Slide 15

 

Dissemination May Occur Through Diverse Pathways—II

• Some pathways involve researchers directly engaging with users and others involve intermediaries that help synthesize, apply or communicate the findings.
• Diverse pathways will be effective in different circumstances; effectively reaching diverse audiences for the work enhances the use of well targeted high quality research.

Slide 16

 

PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes

Dissemination Strategy	Percentage of Respondents Reporting
	Major Focus	Minor Focus
Publications (any) Journal article(s) Research report/working paper User-focused research brief/issue paper Chapter Book	97 91 29 12 6 3	2 5 27 19 18 0
Conference Presentations (any) Paper or poster at research conference Presentation at policy-/user-focused meeting	70 57 34	28 28 32
Briefings (any) Policymaker briefings Managerial briefings Interest group briefings	27 18 11 11	26 19 17 18
Other Forms of Dissemination (any) Mass media Federal or state testimony Expert witness Other	9 5 2 0 3	27 23 8 3 4

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Responses based on 97 responding PIs (70% of grantees surveyed).

Slide 17

 

Support from Host Institution for Dissemination Limited

Type of Assistance	Amount of Assistance (Reported %)
	Substantial	Moderate	Limited	None/Did Not Use Resource
Press office for interaction with media	9%	28%	24%	40%
Newsletters reporting on key findings from research	10%	17%	20%	54%
Established working paper series	3%	11%	7%	79%
Established series of research/issue briefs	6%	7%	14%	74%
Funds available to develop user-oriented materials	0%	2%	4%	93%
Information or training on how to understand and interact with potential users of research	1%	3%	13%	83%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Notes: N = 91. These analyses exclude 6 respondents who did not respond to any questions about dissemination support.

Slide 18

 

Mean Percentage of Time PIs Report Interacting with Policymakers

Image: A bar chart shows the following data:

• Actual: 10%
• Desire: 15%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Slide 19

 

The Impact of Research on Policy—I

• Research is one of several influences on decision-making. Importance of research likely to vary with the topic, user, environment and other factors.
• Some research may have an immediate impact but most accumulates and gets applied to relevant topics as issues arise.
• Accumulation of research contributes to a "research reservoir"—use more likely if findings readily available to potential users and applicability of findings clear.

Slide 20

 

The Impact of Research on Policy—II

• Accumulation of findings across multiple studies adds weight to findings.
• Some forms of communication are likely to be stronger in reaching users than others.
• Researchers can enhance usefulness by thinking carefully about the key findings and identify the main "message" of the study ("elevator test").

Slide 21

 

PI Perception of Outcomes Relevant to Their Grants

Image: Bar chart shows percentages of outcomes investigators say are relevant to their goals:

• Add to general knowledge/wisdom: 100%
• Inform other research: 100%
• Contribute to policy debate: 92%
• Target areas for future research: 87%
• New research methods: 73%
• Help organizations improve: 69%
• New models: care/policy design: 67%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: N = 85.

Slide 22

 

Lessons from Case Studies on Ways to Enhance Use

• Develop relationships with potential users and involve them early.
• Be aware of how results may be relevant to different policy decisions and the timing of those decisions.
• Identify where research "fits" in stream of research.
• Develop expertise—and reputation for it.

Slide 23

 

What Federal Research Translators Told Us

• They make extensive use of this type of research.
• They are familiar with ongoing work and consult the literature as time allows.
• They value timeliness but also place a high value on quality and objectivity.
• Regulatory entities find general studies of markets useful in providing context for case specific analysis.
• They see critical gaps in the existing body of research in this area.

Slide 24

 

Key Findings: What Is AHRQ's Role in Context of NIH, HCFO, and Other Funders?

Slide 25

 

General Findings Across the Project

• Researchers view AHRQ as a major source of funding for this research but have turned elsewhere as AHRQ funds have become less available. Low funding levels for this kind of research at AHRQ was the most common problem cited by PIs.
• PIs view AHRQ's oversight over grants more positively than the agency's work on dissemination and communication.
• While NIH processes mirror those of AHRQ, HCFO has more emphasis on dissemination, and researchers score HCFO higher than other funders on technical assistance
• Federal translators perceive AHRQ as more a primary resource for databases and clinical research than for this type of research.

Slide 26

 

Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted)

Organizations	Major Source of Funding	Moderate Funding	Little Funding	No Funding	No Opinion/ Refused
Agency for Healthcare Research and Quality (AHRQ)	40%	30	27	3	0
National Institutes of Health (NIH)	40%	30	24	3	3
Robert Wood Johnson Foundation's Healthcare Financing and Organization (HCFO) program	39%	34	19	1	7
Centers for Medicare & Medicaid Services (CMS)	7%	13	62	6	13
Robert Wood Johnson Foundation, and other programs	10%	43	25	4	9
Other federal or state government	4%	15	13	5	64
Other private foundations	4%	24	16	3	53

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: N = 96.

Slide 27

 

Ratings of AHRQ Technical Support

All Respondents	Ratings Among Those Who Consider Task Part of AHRQ's Job
	Consider Part of AHRQ's Job (Number)	Mean Score	Percent Who Say Excellent or Very Good
Pre-award guidance	82	2.1	68%
Grant award	86	2.0	65%
Grant management	80	2.3	59%
Research methods	40	2.5	53%
Dissemination and communication of findings	73	2.7	40%
Linkages with others interested in the topic of your research	62	2.9	44%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Mean scores calculated based on respondent ratings of AHRQ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor

Slide 28

 

Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others

	Number Reported Funding Source and Provided Rating	Mean Score (SD)
Robert Wood Johnson Foundation's Healthcare Financing and Organization (HCFO) program	35	1.8 (0.9)
Agency for Healthcare Research and Quality (AHRQ)	88	2.3 (1.1)
Robert Wood Johnson Foundation, other programs	45	2.3 (0.9)
Other private foundations	28	2.4 (0.8)
National Institutes of Health (NIH)	60	2.5 (1.1)
Other federal or state government	23	2.6 (1.2)
Centers for Medicare & Medicaid Services (CMS)	16	3.4 (1.0)

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Mean scores calculated based on respondent ratings of funders' technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor.

Slide 29

 

Key Findings: Actions That Could Enhance Use of Research Findings

Slide 30

 

Conclusions From the Project

• AHRQ currently is not highly visible as a source of research on health care costs, productivity, organization and market forces despite a legislative mandate for it and $81 million spent on grants for this kind of research from 1998 through 2006.
• Our findings show that the research AHRQ has funded addresses critical aspects of health system performance that are important to decision-makers.
• Research AHRQ has funded has had an impact but mainly project by project at the initiative of the investigator.
• AHRQ can better track its work and leverage its investments to make its research more visible, make findings more accessible, and enhance their use.

Slide 31

 

Recommendations for AHRQ—I

• Use our study as basis for ongoing tracking of research.
• Create a visible place on the Agency Web site to communicate AHRQ's efforts in this area.
• Create internal capacity for monitoring grant progress to identify emerging outcomes that may be of interest; reward staff for support in this area.
• Communicate with PIs why AHRQ would value ongoing reports of dissemination and use after grants end and develop systems to support such monitoring.

Slide 32

 

Recommendations for AHRQ—II

• Use our project to develop an annotated bibliography with abstracts of publications from AHRQ funded research in this area.
• Work with others on targeted syntheses of findings on topics which AHRQ has invested.
• Help others contact experts on given research topics through online contact information on the Web site or other means.
• Offer researchers a "webinar" on ways of thinking about, developing, and communicating "messages" from the study.

Slide 33

 

Recommendations for AHRQ (and Others)—III

• Elicit more feedback from users on their needs to identify areas that can be supported by indepth, high quality research.
• Work with others to better articulate what this type of research is and its relevance to understanding health system performance and how to enhance outcomes.