Evaluation of Health IT Tools and Resources Available at the AHRQ NRC for Health IT Web Site: Final Report

Given the importance of health literacy and the generally favorable view of the Guide by focus group participants, the evaluation team recommends updating the Guide to make it more current and developing a dissemination strategy for the Guide to reach a broader audience of developers, purchasers, and stakeholders of health IT products.

Thirteen recommendations are described in detail below, including their importance (high, medium, or low), level of effort (LOE) (high, medium, or low), and type of subject matter expertise (SME) required. The importance rating reflects the anticipated contribution the item makes to a user's understanding of the Guide or ability to use the Guide. The LOE rating reflects the anticipated relative work effort of completing the recommendation using one full-time equivalent (FTE), from low (a few days) to medium (a few weeks) to high (a few months or more).

Types of expertise are included in Table 14, such as content expertise (for many areas of the Guide) and Web tool design expertise (for the checklist). Several kinds of content expertise are desirable since the Guide presents content for practical use by purchasers, developers, and business stakeholders. Content areas include literacy, health literacy, usability, software design, software development, consumer health IT, and human computer interaction. Web tool design expertise includes skills in Web page prototyping, design, development, heuristic evaluation, and user testing.

Although several dimensions could be used to rank-order these recommendations, those classified as having high importance and low effort offer the highest value. Next to be considered should be recommendations of high importance and high effort followed by the remaining recommendations deemed of medium importance and with low effort. 

4.1 Recommendations for Improving the Health IT Literacy Guide

The Guide should be improved and updated to be more timely, relevant, and useful to developers and purchasers in designing health IT applications that are accessible to adults with varying levels of health literacy. The organization of the sections below is based on the structure of the Health IT Literacy Guide.

4.1.1 Recommendations for Changes to the Introduction and Overview of the Guide

To improve and update Section I (Introduction) and Section II (Overview) of the Health IT Literacy Guide, several changes should be made to make it more timely, relevant, and useful to developers in designing and purchasers in selecting health IT applications that are accessible to adults with different levels of health literacy.

The Guide should include a clear definition of health literacy in the introduction. Based on the evaluation, further discussion of the concepts of usability and accessibility, and how they relate to literacy and health literacy, would be helpful. Defining health literacy and related concepts at the beginning of the Guide will help to avoid any confusion on the focus of the Guide. The definition of literacy used in Healthy People 2020 (HealthyPeople.gov, 2012) is suggested. This recommendation is categorized as being of high importance and of low effort.

The Guide should clarify in the introduction not only who the intended audience is for the Guide, but also the role of this audience in addressing health literacy among consumers and patients. It would be beneficial to include in the introduction of the Guide a more detailed description of the Guide's intended audience and their roles for ensuring that health IT products are accessible for populations with limited literacy. The introduction or overview should identify activities and processes during system purchase, design, and development that impact user understanding of health concepts downstream, especially if the impact is indirect or complex. This recommendation is seen as being of high importance and of low effort. 

The Guide should include some information on the business case and financial aspects (cost savings, product marketability, regulatory compliance, or other benefits) that may accrue from developing or purchasing health IT products that are accessible to limited literacy patients. The intended audience of the Guide should be expanded to include business or financial personnel who may contribute to the decision to develop or purchase health IT products. Since purchase of health IT products is largely influenced by cost and business drivers, addressing the business case and financial aspects for following recommendations in the Guide would be more likely to influence decisionmakers since the cost of developing and providing systems is typically a major factor when choosing to develop or provide health IT products to patients. This work would likely entail conducting a literature review and possibly interviewing subject matter experts in order to describe or develop the business case for supporting limited health literacy among health IT users. This recommendation is deemed as being of high importance and of moderate effort. 

The Guide should describe how a socio-technical framework can be used to understand how patients/consumers with limited literacy benefit from accessible health IT. Research has highlighted the role of health IT in supporting patient understanding not only through technology accessibility, but also by supporting improved cognitive reinforcement, communication, social interaction, and learning. As consumer use of technology continues to grow, the role of health IT in supporting patient learning and understanding is likely to expand. Assessing and improving the effectiveness of health IT to support limited literacy users is facilitated by consideration of the interactions among users, activities, and technology within physical, social, community, and policy contexts (National Academies, 2011). Applying a socio-technical perspective can be valuable in helping developers, purchasers, and other stakeholders consider the many factors that singly and jointly influence the user experience and the value a technology brings. This recommendation is categorized as being of medium importance and of medium effort. 

4.1.2 Recommendations for Changes to Principles of Accessible and Usable Health IT

Changes to Section III (Principles of Accessible and Usable Health IT) of the Guide will make it more timely, relevant, and useful to developers and purchasers.

The content of the Guide should be updated with more current examples. Updated (1) tools and resources that can aid developers and purchasers of health IT, (2) best practices from the literature pertaining to developing accessible health IT products (including universal design principles), and (3) examples of recent health IT products that have been found to be effective with populations with limited literacy skills or limited experience with technology are discussed in detail in Appendix G. These were identified during the environmental scan.

The Guide should include information on technologies that have emerged and expanded since the publication of the Guide in 2007. Health IT platforms such as tablet computers and smartphones, applications such as patient portals on a desktop or mobile device, and devices for mobile tracking and remote sensing are being used more commonly. It is recommended to include information about tablets because medical practices and hospitals are using tablets more often in office settings and with patients. Smartphones are another expanding technology that should be addressed in the Guide, especially given that 45% of Americans now own a smartphone with the capability to access health information through the Internet (Rainie, 2012). Guidelines for smartphones should contain information on apps that consider unique data, interface, and memory issues. It is also recommended to add information on patient portals because they are becoming more commonly available through medical practices striving to meet advancing meaningful use requirements. Lastly, it is recommended to avoid the use of the term "personal digital assistant" or PDA since this term is no longer favored. Detailed information that can assist AHRQ in implementing this recommendation obtained from the environmental scan is reported in Appendix G. This recommendation is seen as being of high importance and of low effort. 

The Guide should include information about significant features that impact the ease of use of health IT for patients and members of the care team. Specifically, additional guidelines are needed for user controls that allow or restrict access to health information; appropriate navigation and linking within PHRs and patient portals; and inclusion of multidirectional communication capabilities. The Guide should also address the role of consumer health vocabularies in developing consumer health informatics products and the importance of integrating customized health information to personalize the user experience. In addition, the universal design section of the Guide should be expanded and integrate key terminology and principles throughout. This information may resonate with health IT developers, aid future dissemination of the Guide, help establish a common language between health literacy advocates and IT developers, and promote important development practices that improve usability for the patient. This recommendation is deemed as being of high importance and of low effort. 

The section in the Guide on home-monitoring devices should be updated because patients' use of these devices has increased, and this will undoubtedly continue. This update should include information on how project vendors need to be flexible in integrating these devices based on consumer needs. This recommendation is seen as being of medium importance and of low effort. 

The section in the Guide on Web sites should be updated to include clear guidance about the use of personal information collected from users. The environmental scan of best practices from published research suggests that limiting and simplifying information requests as well as ensuring that privacy and information use policies are clear will both minimize user confusion and allay privacy concerns. This recommendation is considered to be of low importance and of low effort. 

The Guide should reference additional information on user testing for developing and evaluating health IT products. It is recommended that the Guide address how to involve patients/users in the health IT development and evaluation processes so their feedback can be used to improve health IT products. In addition, it is recommended to reference information on user testing in which developers of health IT products present the user with alternative design options to gauge which content and formats are preferred (Christian, 2012). A useful reference focusing on consumer health IT design, Designing Consumer Health IT: A Guide for Developers and Systems Designers (2012) includes a number of good references that highlight the role of user testing. Culturally specific user testing is particularly useful when developing or providing health IT products for different populations or geographic areas. This recommendation is categorized as being of medium importance and of low effort. 

4.1.3 Recommendations for Changes to the Checklist

The Checklist (in the appendix) of the Guide should be updated and moved into the body of the Guide.

Make the checklist a more prominent focus of the Guide and adding a scoring system, if possible, to make it more useful for developers in designing and testing products and for purchasers in evaluating products. It is also recommended to update the checklist to reflect improvements included in the body of the Guide, such as information on tablets and smartphones, gathering feedback from patients or users of the health IT products through focus groups and usability testing. Updates to the Guide should consider whether separate checklists for developers and purchasers are warranted, whether scenarios should be included in the Guide for ease of use, and whether external references should be included in the checklist for quick access. This recommendation is considered being of high importance and requiring a high level of effort. 

4.1.4 Recommendations for Dissemination of the Guide

Develop a dissemination strategy so the updated Guide can reach more developers and purchasers of health IT products than are currently being reached. It is believed this dissemination strategy should include distributing the updated Guide to several professional organizations related to health informatics and health care provider organizations. HIMSS is one specific professional organization that was found to be the most influential for both developers and purchasers of health IT products. Not only does it seem beneficial to distribute the Guide to HIMSS to disseminate to its members, but the HIMSS Annual Conference could also be used to showcase the Guide and disseminate it directly to both developers and purchasers of health IT products. It is also recommended to include other organizations in the dissemination plan, such as CDC and ONC, as well as certification organizations and grant-sponsoring organizations.

It is also recommended that the Guide should be promoted through various health IT newsletters and listservs^TM to make it more accessible to the intended audience. Some promotion strategies include improving the search result optimization of this Guide to make it easier to find on the Internet using common Web search engines like Google and employing search terms like "development design." The Guide might also be renamed to make it easier to find via a Web search, thereby reaching more developers and purchasers of health IT products.

In addition, it is recommended to use Webinars and other presentation venues to teach about using the Guide to support health IT selection, design, and development. The same approach could be used to solicit ideas for ongoing improvement of the checklists and the Guide. Additional recommendations related to the dissemination of the Guide include the following:

• Ask other Web sites/agencies to add the Guide as a key reference link from their Web site or resources.
• Consider search engine optimization (SEO) approaches to making the Guide easier to locate.
• Consider promoting the Guide at conferences, gatherings, and on the AHRQ Web site.
• Invite experts—from NIST, CDC, Institute of Healthcare Improvement, ONC, HIMSS, AMIA, vendors, other industry groups—to periodically review and update the Guide.
• Observe the use of checklists during purchase, design, and development activities to determine their utility.

Lastly, as part of the dissemination plan, it is recommended that the Guide be used in educational settings such as public health, medical, and nursing schools in order to teach future medical professionals of the importance of health literacy considerations in health IT products. Similarly, it is recommended to incorporate the Guide into developer education as part of health IT design and usability training to give designers a more in-depth understanding of health literacy principles and how best to design products with these principles in mind. The dissemination plan recommendations are categorized as being of high importance and of medium effort. 

Return to Contents

4.2 Recommendations for How Often the Guide Should Be Updated, Where the Guide Should Be Located, and How the Recommendations Should Be Implemented

To serve the needs of developers, purchasers, and business decisionmakers, it is believed that content areas of the Guide will need updates at different frequencies. For example, specific technologies that are commonplace today may be outdated in a few years. Also, new resources providing guidance in the design of health IT for limited literacy users are likely to evolve more quickly than in the past, given intensified interest in this area. Static documents in formats such as PDF are more likely to become outdated than Internet-based Web pages that can be updated more easily, especially when, like a dynamic Web site, changes can be introduced easily or automatically (according to established policies). Perhaps future updates can be scheduled like maintenance activities on a Web site, providing a chance to check and update links to resources, add autogenerated content from webcrawlers, and provide an archive and history of tracked changes, such as a Wiki.

Wherever the Guide is located, it must be easily found to have an impact. Its searchability and ease of retrieval should be tested and adjusted periodically (using common search engines and common search terms) to optimize dissemination. Cross-links that leverage key health IT Web sites sponsored by AHRQ, ONC, CMS, VHA, and many others can improve retrieval and dissemination. Locating the Guide on the health IT tools and resources page seems appropriate.

When implementing the recommendations for updating the Health IT Literacy Guide, a number of areas should be addressed.

• Content updates for information provided in the Guide should be performed approximately every 2 years to keep information from being out of date. A Web page referenced in the Guide but separate from it, containing lists of resources, would permit simple and more frequent updates as needed.
• The checklist should be developed as an interactive tool that is separate from the content portion of the Guide. Creating an effective tool for stakeholders who make or influence development or purchase decisions for health IT will require the development of user requirements, product design concepts, a working product prototype, usability testing and user feedback on the prototype, and development of the tool. Once available in production, tracking of interactive tool use and routine collection of user feedback will provide important input for future improvements to the tool.
• Dissemination-related activities should be planned in advance to position the updated Guide and interactive tool for maximal impact. Even if dissemination is anticipated to be a later effort or part of another project, early dissemination planning could help to refine the overall product concept and design. Dissemination activities might include Webinars, blogs, news stories, SEO, case studies, technology, or health IT conferences, and activities sponsored by other federal agencies or nongovernmental organizations. 

Return to Contents

4.3 General Guidance on How to Develop Similar Tools or Resources

A number of findings and recommendations from this project may have general relevance to future tools or resources developed or guided by AHRQ in a future project.

• Future tools or resources should specifically identify the target audience or audiences for the tool or resource, since understanding and meeting the specific needs of each audience will help to identify the requirements of the tool or resource, impacting the downstream work.
• The future project should identify the functional requirements of the tool or resource in development, anticipating the possibility that the end-product might be a static document, an interactive tool, a video, or in some other type of format.
• The project should address user testing and monitoring of product use to maximize its impact; for an interactive tool or static document, usability testing appropriate to the tool or resource should be planned and performed.
• Coordination of new or updated tools or resources with preexisting related tools or resources should be addressed in the project.
• The project should encourage consideration of a broad set of stakeholders, including business stakeholders, since they often influence or mediate decisionmaking and resources for a downstream project.
• A dissemination strategy should be developed from the beginning of the project, even if it is not carried out until a later project.
• If appropriate, the project should consider a socio-technical approach to planning, designing, and assessing the tool or resource being considered, since many influences play a role in adoption and use of tools and resources. The socio-technical approach draws attention to the people, technologies, and actions that interact when a tool or resource is used in a real-world setting, and how those interactions may be influenced by the physical, social, community, or policy environment. This perspective is also introduced in Section 4.1 of this report.

Return to Contents

4.4 Recommendations Summary

Overall, based on findings from this evaluation, four high-level recommendations emerged: (1) keep the Guide; (2) expand dissemination of the Guide; (3) enhance the utility of the Guide and the audience it addresses; and (4) update the Guide's content.

Keeping the Guide is important because this evaluation found sustained interest in the content of the Guide among purchasers and developers of health IT, as well as a significant gap in knowledge about how health IT can support limited literacy patients and consumers. Increased dissemination of the Guide is critical for gaps in knowledge to be addressed, not only because limited dissemination leads to limited impact, but because broader use will increase the likelihood that improvements in the Guide will be surfaced. Participants in the focus groups, in particular, voiced strong interest in having a checklist tool they could access when making software purchasing decisions, for example. They also thought that business decisionmakers—especially those responsible for system selection, maintenance, and design decisions—were important stakeholders in ensuring that health IT would meet the needs of limited literacy users, and should be explicitly identified as part of the intended audience. Finally, the Guide's content, while still providing some useful information, has numerous areas that are outdated or incomplete. Updates to the Guide's content, leveraging findings from this report and the use of content experts, are anticipated to increase the value brought by the Guide to its intended audience and align it with advances in the industry.

Table 14. Summary of recommendations by Guide section

Return to Contents
Proceed to Next Section