Chapter 8. Description of Ideal Evaluation Methods: Describing Patient Safety Practices

One of the key issues in patient safety practice (PSP) research and literature is adequately defining and describing PSPs. The key goal in standardizing PSP descriptions is to provide sufficient detail on the PSP and its implementation to assess: (a) whether it improves safety, (b) the risks that random or systematic error influences the results, and (c) the applicability to other situations for others to replicate or adapt it. Other goals for standard descriptions include the ability to make biases transparent, evaluate inclusion criteria and determine heterogeneity in literature reviews, and improve the ability of journal editors and reviewers to assess the quality of the information. These goals require some basic information on the PSP itself and on key elements of what was done.

Describing the intervention in sufficient detail so that it can be replicated is a requirement included in reporting guidelines such as SQUIRE for quality improvement studies, and it also was endorsed by our technical expert panel (TEP). The key challenge in PSP studies—in contrast to pharmaceutical or surgical interventions that are concrete or highly standardized and can be precisely defined, described, and delivered across settings—PSPs generally lack sharp boundaries between the practice, implementation process, and context. Indeed, as the science regarding effective implementation strategies and the understanding of context increase, implementation and context often merge together to become part of the intervention. PSP interventions are inherently dependent on context and vice versa—some PSPs include influencing context as part of the intervention. These interventions are often fluid, since some PSPs incorporate into "the intervention" unforeseen necessary adaptations or lessons learned during the implementation process. While this could be a major problem for clinical outcome studies (for example, if physicians were allowed to vary the dose of the study drug or add additional co-interventions at their discretion), the TEP felt that such fluidity was welcome in PSP studies as long as it was measured and described well. The intervention is often iterative, evolving in response to outcome feedback and changing as context changes through the implementation process. Finally, fully developed PSPs are often complex and include multiple components. While the knowledge regarding how to effectively implement a PSP and the effect of context on implementing a PSP is growing rapidly, it is still a relatively immature science. 

Existing descriptions of PSPs in the published literature vary widely, are often incomplete, and criteria to guide researchers on how to describe PSPs, the implementation strategies, or the contexts are lacking. Some organizations have produced criteria for describing a quality improvement intervention, which may have relevance to describing a PSP. For example, the SQUIRE guidelines are an important step in describing quality improvement studies generally aimed at directly changing provider behaviors.¹ Table 2 contains an excerpt of the SQUIRE guidelines (the full set of guidelines is in Appendix M; see also http://squire-statement.org/). 

The NQF also lists requirements for describing quality improvement practices (Table 3). These guidelines and requirements are generic for quality improvement interventions.

No criteria exist for how to describe the key components of a PSP intervention. Thus, we developed general principles of a PSP description based on theory and existing frameworks. We also created two specific examples based on expert opinion from the project team. Key concepts for describing a PSP and additional elements that can be helpful are included in Box 1. In describing these concepts, we recognize that the borders between implementation and context are not sharp, and that the delineations may evolve as our understanding of important contextual factors grows. Currently, the line between what is context and what is PSP is often not clear. Likewise, the PSP and its implementation are usually too intertwined to tease them apart. The criteria we present here are limited by the lack of evidence to state what should be described about a PSP, so they should be considered as general concepts that appear to be important based on principles of behavior change, other guidelines, and the overall finding of this project. 

To supplement these generic criteria for describing the PSP, we provide here more specific criteria for describing two of our five target PSPs: catheter-related bloodstream infection prevention and CPOE. We solicited input from two members of the project, Peter Pronovost and David Bates, who are international experts in designing, implementing, and evaluating these two PSPs. As such, these criteria are based mostly on expert opinion. Nevertheless, in the absence of strong empirical evidence, the opinions of recognized experts can be a valuable source of guidance for implementers and evaluators. Criteria for describing interventions to prevent catheter-related bloodstream infections are presented in Box 2, and criteria for describing CPOE interventions are presented in Box 3.

Our Proposed Framework for a PSP Classification System 

Through a process of synthesizing existing conceptual frameworks and an expert panel consensus process, we developed a conceptual framework for describing the dimensions of PSPs. The framework includes 11 dimensions, as shown in Table 4. The process of how this was developed is described in Appendix 2.

Reference for Chapter 8

1. Davidoff F, Batalden P , Stevens D, et al. SQUIRE Development Group. Publication guidelines for improvement studies in health care: Evolution of the SQUIRE Project. Ann Intern Med 2008; 149(9):670-6.