Chapter V. Contribution of AHRQ and Program-Wide Infrastructure (continued)

C. AHRQ Council of Partners

1. Council Structure

With the goal of creating a program-wide focus to encourage cross-fertilization of ideas, PFQ required meetings twice a year of grantees organized into the AHRQ Council of Partners (AHRQCoPs). AHRQ staff indicated that the requirement to come to these semi-annual meetings was not typical of all grant contracts, but the agency felt that the meetings were a necessary component of the program to give people face-to-face interaction, time to exchange ideas, and learn from each other.  Moreover, AHRQ saw the cross-project work grantees were asked to do during these meetings as fulfillment of cooperative agreements signed with the agency.

The intent was that grantees would "own" these meetings, create their agendas, and run them. However, AHRQ appears to have been the driver behind both AHRQCoPs and its structure. The RFA required grantees to budget travel funds to meet annually.  AHRQ's general meeting budget was tapped to fund the hotel rental and meals, and other indirect costs of the meetings, all of which were convened in the Washington, D.C. area to make it easier for AHRQ staff to attend.

AHRQ used the first few AHRQCoPs meetings to familiarize grantees with each other's work, and PIs presented their individual projects.  However, at the first meeting, AHRQ staffers proposed the infrastructure for the Council, developed by the PFQ program director in consultation with individual grantees.  They proposed that the Council ratify a charter, elect a chair and vice chair, and organize itself into subcommittees.  The chair turned over several times over the course of the program, more rapidly than originally intended for a variety of reasons (death, change of employment). Four different PIs took on the position of chair over the four years of the program.  

AHRQ proposed subcommittees on Implementation, Dissemination, Partnerships, Evaluation, and Sustainability, since these were all areas important to each of the projects. Earlier, AHRQ staff had discussed an alternative that involved forming subgroups by focus areas.  However, this was rejected in the interest of working on common concerns related to partnerships.  The diversity among grantees was a source of on-going tension within AHRQCoPs as it made finding areas of mutual interest challenging.

By the second meeting, AHRQCoPs had elected a chair. Each of the principal investigators and each of the AHRQ project officers chose one of the subcommittees to participate in. Subsequent COP meetings were convened by the subcommittees and included time for both general sessions and subcommittee work.  Each subcommittee organized content for one of the meetings, and often invited an outside speaker to address a topic consistent with the theme. AHRQ staff reported that grantees initially objected to AHRQ's requirement to collaborate on work outside of their individual projects but acquiesced once it was clear the agency was adamant.

Over the course of the PFQ program, there were seven AHRQCoPs meetings.  Why and how the schedule shifted from an annual to a semi-annual focus is not clear.  Later, meetings— which lasted two days—focused more on the collaborative work the grantees were doing in the subcommittees, and jointly as AHRQCoPs.

2. Perceptions of the AHRQCoPs Meetings

PI Perceptions.  According to several PIs, the greatest benefit of the grantee interaction facilitated by the meetings was the opportunity to network and collaborate.  The AHRQ Council meetings helped grantees form relationships, learn from each other, help each other, and initiate some independent cross-grantee work.  Not surprisingly, the magnitude of this benefit varied among PIs, with some indicating that they benefited a great deal from this interaction and others finding less benefit, believing that the diversity in funded grants hindered grantee-to-grantee learning. 

Some grantees found the meetings useful, some did not.  Some grantees found meetings to be "important," "very useful," and "helpful" because they provided learning opportunities (such as outside speakers) that "added depth to grantee insight and expertise," which informed decisions about their individual projects.  By contrast, some grantees found the meetings to be "unfocused," "not useful," and "painful," requiring time investments they did not have for activities that did not benefit their individual projects.  The grantees that were enthusiastic or interested in the meetings attended regularly and participated; others who found the meetings unhelpful and time-consuming attended infrequently.  Some grantees attended regularly simply because they felt they had to, but in some cases they delegated attendance to more junior staff. Over the course of PFQ's history, most principal investigators continued to attend at least a portion of most meetings and some brought several staff.  The predominant view appeared to be that the meetings were interesting for general learning but not particularly germane to their project work.

Some grantees believed strongly that there was misalignment between AHRQ's expectations and what grantees thought they had to do at the start of the program.  They pointed to the budgetary implications of twice a-year meetings, when they had been asked to budget for one. They also were concerned about the resources they perceived AHRQ expected them to spend on these activities, particularly via subcommittees.  They felt these demands competed for attention with what they were supposed to be doing under the grant. Some also expressed concern about the lack of clear guidance on the desired outcome from collective action.  Others, typically leaders in the process, strongly disagreed and saw substantial value to cross-grantee work. Additionally, the high turnover in AHRQ Council leadership only amplified this perceived lack of structure.

AHRQ Project Officer Perceptions.  The PFQ program director encouraged project officers and other program-related AHRQ staff to attend AHRQCoPs meetings.  Some did so regularly, whereas others participated less often.  Those who did not said it was because their schedules did not allow it; they had more pressing demands, or had attended but did not find the meetings all that interesting.

Because our evaluation started late, we had limited opportunity to observe the AHRQCoPs meetings.  However, based on the two meetings we attended, we concur with those grantees who thought more attention could have been given to setting clearer goals, structuring a tighter agenda, and ensuring a better balance between presentation and discussion time. 

3. Subcommittee Work

Nature of Work. A part of each AHRQCoPs meeting, after the first two, was devoted to subcommittee work. Each of the subcommittees also led one of the semi-annual meetings to inform other grantees about their topic, and some chose to bring in guest speakers.  The PIs and POs in subcommittees also communicated outside semi-annual meetings through E-mails and scheduled (sometimes monthly) phone calls.  Table V.1 provides a summary of who participated in each subcommittee and what the subcommittee produced.

While there appears to be consensus that some subcommittees were more productive than others, PFQ grantees disagreed substantially on the value of the subcommittees and their work. Most, though not all, chairs were enthusiastic about their subcommittees. Subcommittees that were productive seemed to have a higher proportion of positive members; however, the subcommittee also had to function collaboratively to achieve this effect.  Thus, while one subcommittee was very well regarded by AHRQ and AHRQCoPs leadership, its members were much more mixed about the process.

Outcomes.  Grantees most positive about the subcommittees cited two main accomplishments. First, the selected topics helped "crystallize" the five components of translational work in the context of partnerships.  Second, the subcommittees created resources that grantees could use in current and future projects.  For example, one grantee said that participation "prompted groups to repetitively think about the five areas [of partnership, implementation, evaluation, sustainability, and dissemination] in terms of their own projects and gave groups the opportunity to see how those areas played out in real-world contexts."  Some PIs suggested that the subcommittees gave grantees learning that would inform current and future projects.

In contrast, other grantees found the subcommittee work "painful," believed the five topics were an artificial way to link grantees together, and did not benefit individual projects.  "[The subcommittee experience] was like [throwing] a physiologist, a biochemist, and a urologist into the same room and saying work together," said one PI.  While several grantees suggested that grouping grants by content, rather than the five selected topics, would have worked better, others believed that the diversity in projects made it impossible to group grantees in any meaningful way.

Early on, many of the subcommittees created tools and surveys, which were intended to be useful to grantees.  The implementation subcommittee, for example, developed a survey on barriers to implementation that they fielded and shared with AHRQCoPs (Table V.1). However, since subcommittee work and individual grantee projects progressed simultaneously, it was difficult for most projects to incorporate resources as they were produced.  Some subcommittees produced tools that their members used, but few of the other grantees used them. For example, the evaluation subcommittee created an evaluation tool it had hoped all PIs would apply to their projects, but many of the grantees chose not to use it because they had already planned and budgeted an evaluation component of their own design.  However, some grantees believe that the tools and resources produced by the subcommittees will be useful in future work.  

Later in the program, AHRQ and the subcommittee chairs decided that each subcommittee would write an article on its respective topic that would be published together in a journal supplement. We believe their interest was spurred first by a paper on partnerships that the chair of one subcommittee developed, by some of their own interests, and by the desire to leave some program legacy both to their former deceased chair (Mark Young) and to the program as a whole, which they perceived to be under-recognized. The journal supplement would be a way to disseminate grantee experiences and learning under PFQ.  The articles have been an important focus of AHRQCoPs' last two meetings.  Though many PIs consider the supplement to be a worthy effort, several grantees have not completed their data collection and have found the push to develop the journal supplement and the seemingly unrealistic time frame frustrating.  Another criticism has been that while they may be successful grantees, they are not necessarily experts on each of the areas of knowledge that were the focus of their subcommittees.

Perceptions on Subcommittees. The primary frustration expressed by grantees about the subcommittees was that they were not aware at the outset that the subcommittee work was part of AHRQ's expectations.  As one said, "To some extent, this was seen as an unbudgeted, un­reimbursed mandate." Many PIs, including the ones that found the subcommittees beneficial, saw the activities as an unexpected add-on to their grant work.  If the subcommittees had been envisioned in advance and budgeted for by the grantees, maybe the PIs could have done more with them, they said.  Grantees also were frustrated by the lack of initial focus.  One grantee indicated that because AHRQ did not clearly state their goals early on, the PIs "spent a lot of time muddling through the whole process."  She continued, "Had it been clear from the outset [what the agency wanted], it would have released a lot of angst."  However, even without a coherent framework explaining how these subcommittees fit together and what they were supposed to accomplish, some grantees thought the subcommittees managed to create some interesting resources. 

A substantive concern we heard from several grantees was that the focus on the subcommittee work took a lot of time and effort that, according to some grantees, may have been better spent becoming familiar with each other's work and helping each other on individual projects.  Several PIs and POs indicated that the downside of focusing on subcommittee work was that people never developed a sense of where the individual projects were going and what they were doing. One PI indicated that the AHRQCoPs meetings would have been more helpful had they included more feedback and problem-solving from AHRQ on individual projects.

Because  AHRQCoPs was the most visible part of PFQ to AHRQ PFQ staff and leadership, we believe that for some of them AHRQCoPs and its work became the PFQ rather than merely an adjunct, however important, to the grantees' own work.  To the extent this is true, it is unfortunate because PFQ's resources were mainly devoted to the work funded through grants and, as we have described before, grantees typically worked on their projects, some achieving notable successes.

Return to Contents

D. Cross-Grantee Collaboration

An important goal behind regular meetings of PFQ grantees was the hope that such meetings would encourage grantees to learn from one another and build collaborations and partnerships independent of AHRQ.  In general, such collaboration did not develop on a widespread basis. However, there were some notable successes as PFQ grantees were able to form collaborations with each other that were useful for their individual projects.  

For example, Texas A&M and Altarum (two bio-terrorism grantees) formed a working partnership; researchers at Texas A&M provided information from the field that was used to provide input data to Altarum's simulation model, and Altarum gave Texas A&M contacts in Michigan to assist in its surveillance work on the Canadian border.  The two organizations have had regular face-to-face meetings outside PFQ activities and were very positive about the collaboration based on shared interests.

Another example of cross-grantee collaboration is reflected in the help Catholic Healthcare Partners gave to other grantees in connecting them with people or organizations within or affiliated with the CHP system that were relevant to their work. Two CHP long-term care facilities participated in the ISIS project, and CHP identified a cardiology group to collaborate with the AMA for a project named Cardio-HIT, which builds on PFQ work and is funded by AHRQ.

PFQ generated other efforts by grantees with common interests to explore issues of mutual concern. For example, two major national provider organization grantees talked to a provider group grantee about pursing a common initiative, but the endeavor failed to proceed when one withdrew because of lack of funds.  Two grantees focused on pediatric care talked  with each other to see what they might learn.  While most grantees did not build formal collaborations with each other, several PIs indicated that the meetings and subcommittee work led to informal conversations that were useful for exchanging ideas, brain-storming on how to handle various situations, and providing feedback on individual project work.  

While PFQ led to increases in communication, these typically were relatively limited in scope and appear to be similar to what one might expect from any meeting opportunities for networking.  Even when collaboration occurred, it is difficult to determine how many go beyond what would normally have happened in any environment where people come together to discuss research versus what was made possible because of the PFQ structure and its emphasis on partnerships. Some grantees expressed disappointment that PFQ did not include more grantees with similar foci to their own. 

Whether a different structure for AHRQCoPs and its subcommittee work might have facilitated great sharing is unclear.  Some grantees indicated that they might have collaborated more with others had there not already been a huge time commitment to work on subcommittees and produce tools and papers.  Others, however, indicated that the projects were so different that cross-fertilization and collaboration were not possible, and that this "artificial sense of community" did not make it any more possible.  

Return to Contents
Proceed to Next Section