Evaluation of Health IT Tools and Resources Available at the AHRQ NRC for Health IT Web Site: Final Report

3.1 Findings that Address Research Question 1: To what extent does the Health IT Literacy Guide aid developers in designing health IT applications that are accessible to adults with different levels of health literacy?

Familiarity with Limited Health Literacy. Information gathered from the focus groups highlighted a lack of emphasis on the concept of health literacy among developers, or with the concept that limited health literacy users needed support from health IT. Through interview probes, developers reported inconsistent sources of information about health literacy; inconsistent ideas about who was ultimately responsible for ensuring that a product supported limited health literacy users; little knowledge of tools for measuring product performance for limited health literacy users; and that designing a product that supported limited health literacy users was a low priority given the weak market demand for this.

Focus group participants were asked to define health literacy as it applies to their work as developers. Overall, participants had many descriptions and definitions of health literacy and pointed out that there is no universally accepted definition of health literacy. Developers were somewhat familiar with the concept of health literacy. The following are some examples of developers' descriptions of health literacy:

"The ability to say ‘I can have education materials that are understandable from someone that has little education'; It has to be about the language you use, concept, and different languages and culture."
"Presenting information in a way that the education level, culture, and experience in health care settings of the audience do not need to be considered."
"The ability for bidirectional communication with comprehension of all aspects related to health."
"Being able to understand medication labels, physician's advice, and health care issues."
"The motivation to access and to take responsibility of one's own health and family health with health promotion and safety and community health; the ability to interact with health care providers."

Awareness of the Health IT Literacy Guide. A majority of developer focus group participants stated that they were not aware of the Health IT Literacy Guide nor did they believe their peers were familiar with it. Of the 30 developer focus group participants, only 1 developer said they were already aware that the Guide existed.

Main Focus of the Health IT Literacy Guide. Focus group participants were first asked their thoughts on the main point of the Guide. Developers responded that the Guide provided guidelines and principles to apply when designing health IT products and contained information that was designed to aid developers in making better design decisions. They also felt the Guide provided a centralized source of resources on health literacy and health IT design that may be beneficial to developers. A number of developers described the Guide as serving more as an introduction to health IT design by covering only the basics; they suggested that perhaps it would be more useful for developers who are just starting out and are not already familiar with health literacy concepts.

Usefulness of the Health IT Literacy Guide. Focus group participants were next asked how useful they thought the Guide was for helping developers and purchasers to evaluate health IT products.

Developers thought the Guide was a good starting point for beginner developers and contained some good, simple design points and helpful additional health literacy resources. However, developers also felt the Guide's design points were vague and needed more detail. One developer commented that the Guide was too basic and contained information they already knew. Another developer noted that if developers did not already know the information included in this Guide, they probably would not be successful in developing health IT products. Another common theme from developers was that the Guide, which was published in 2007, was already outdated. Developers felt there had been a lot of changes in health IT since 2007, although they did acknowledge the difficulty in producing resources that stay current over time despite rapid changes in technology and continual upgrading of health IT products.

The findings from the focus groups echoed those from the environmental scan. The scan identified 9 tools and resources, 22 best practices manuscripts, and 30 examples of health IT products (refer to Appendix G) that support the notion that the Guide is out of date.

Accessible Health IT for Limited-Literacy Populations Checklist. When focus group participants were asked specifically about the checklist located in the appendix of the Health IT Literacy Guide, a majority felt it was the most useful part of the Guide. Many developers stated that, realistically, they would most likely use only the checklist and would not take the time to read the information in the body of the Guide. Developers thought the checklist provided a quick executive summary and was a great resource to have as an evaluation tool for health IT products. Some developers stated that if they were given this Guide, they would be interested only in the checklist and thought most developers would not be concerned with the background information on health literacy in the body of the Guide. A few developers commented on the checkboxes included in the checklist, observing that not all items can be answered with a simple "yes" or "no" and that there was no opportunity for a respondent to provide an explanation for the answer they choose. These developers felt the checklist was not useful without some explanation of the reasons for marking "yes" or "no" for each item on the checklist.

AHRQ as the Source of the Health IT Literacy Guide and Literacy Information. Some focus group participants, developers in particular, stated that although giving guidelines for developing health IT products and receiving literacy information could be useful, AHRQ (or other governmental entities) should not force developers down a path that might stifle innovation and quell creativity when developing innovative products. Developers did not want guidelines to be seen as requirements but more as recommendations for designing health IT products. These comments were similar to developers' statements when they were asked about responsibility for ensuring that products considered health literacy.

Use of the Health IT Literacy Guide. Focus group participants were asked if other developers would use the Guide for its intended purpose. Most developers agreed that they would use the checklist located in the appendix of the Guide most frequently. When asked when in the software design process they might anticipate using this Guide, most developers said they would anticipate using it during beta-testing or when checking if the content met the guidelines set forth in the Guide for the populations served by their products. One developer commented that he would share the Guide with community health centers and work with their staff to make sure they were following the recommendations laid out in the Guide related to computer kiosks, which are often used in their waiting rooms. One developer mentioned that he anticipated using the Guide at the beginning of the software design process by reviewing the information in the Guide with project members as a way to start thinking about product direction and how to expand on what has already been developed in similar health IT products.

Audiences. Focus group participants were asked for whom they thought the Guide was written, and responses from developers varied to some degree. Some participants thought the Guide was written to provide guidance for purchasers such as health providers and practices when evaluating health IT products for purchase. They felt it gave purchasers information on what they should be looking for when buying health IT products. Participants also felt that the Guide was written for health providers and practices including community health clinics that may be developing their own materials and Web sites in-house to provide to patients. They thought the Guide was intended for practices to use to ensure they were providing materials that followed the recommended guidelines

Developers thought some additional audiences might benefit from using the Guide, such as insurance companies that tailor e-mails and other information to send to consumers.

Sources of Information on Health Literacy. When asked where they learned about health literacy, focus group participants mentioned various sources. A few said that they had taken part in trainings; plain language trainings were mentioned the most frequently mentioned. Others said that they learned about health literacy from experts, studies, journal articles, professional colleagues (including physicians and social workers), through work on grant applications, and from the Institute of Medicine's report, Health Literacy: A Prescription to End Confusion.

Developers were asked about resources they have used and found to be helpful that address health literacy specifically. Overall, very few used resources specifically dealing with health literacy in their development. One developer mentioned Plainlanguage.gov, stating "The principles carry over…clean design, simple interface, and finite info" (plainlanguage.gov, 2011). Another mentioned http://www.readabilityformulas.com, which he uses for testing (ReadabilityFormulas.com, 2013). One developer recommended the Microsoft Office® Readability Toolkit, a built-in part of Microsoft Word© that displays a readability score of text. The majority of developers did not name any specific resources they had used.

Developers and Health Literacy. Almost one-third of developers (9 of 30) said that they conduct some sort of testing of their products for health literacy. Testing included using a Web site like http://www.readabilityformulas.com, using a company identified as "literacy experts," and conducting usability testing with the intended audience. However, the majority of developers said that they do not conduct any health literacy testing of their tools and resources. Nonetheless, many developers stated that health literacy should be addressed throughout the design process. One developer stated that health literacy should be considered "from the very beginning of design. [From the] start of design, from design review and then circle back to testing and user testing. Every step along the way." The concept of considering health literacy at the beginning of the design process was echoed by several developers. For some, this consideration went beyond product development: "it's a cultural element in the organization." Despite this belief that health literacy must be addressed early and often, one developer stated, "The customer is the physician who writes the check…. not the patient (right now). When the patient is spending money, the incentives will change; if we had evidence that we could sell more systems with more literacy-conscious buyers, we'd adjust."

Developers as a group felt that it was important to take responsibility for health literacy during the design process, although many were unsure who currently was responsible in their organizations. Participants were split on who specifically should be responsible for health literacy. Although many felt that they were responsible, some participants who work in the for-profit industry said that the market should drive consideration of health literacy. Comments from participants on the responsibility of health literacy and the necessity of compliance standards, regulations, or certifications concerning health literacy included:

"That is a huge gray area. I don't think anyone is [responsible]."
"I have no idea [who is responsible]; in the for-profit world, this is hard to implement."
"We need national standards that abide by national regulations. Before products come on the market, they should meet standards. The best way is to have these standards in place."
"With having a requirement of the vendor to have their patient-facing technologies be certified, my fear would be you would stifle development."

Barriers to Consideration of Health Literacy in Design Decisions. Developers mentioned the lack of commonly accepted guidelines or sources of expertise for addressing health literacy. They also said that the lack of a specific, recognized, component of the cost to ensure that products address health literacy was a barrier, with one developer stating "It is hard to monetize."

Developers stated the following barriers to considering health literacy in the design of consumer health IT tools and resources:

There is no universally accepted systematic approach to considering and addressing health literacy.
Designers are not familiar with the cultural and language needs of the patient population being targeted.
There is a celebration of new thinking and new priorities even when they are not necessarily better. This "innovation bias," as one developer called it, keeps developers from being able to address issues of literacy more generally.
"Developing for limited health literacy audiences is seen as expensive, detailed, and not enjoyable or interesting."
There is concern about legal issues that may arise when attempting to synthesize information if it is interpreted differently than intended.
Determining the health literacy level of patients who developers are trying to target is difficult, even initially.
Time is a barrier: developers are often working under quick timelines and do not want to delay getting a product to market by having to address health literacy.
Developing for both low and high literacy levels is challenging and takes time.

Return to Contents

3.2 Findings that Address Research Question 2: To what extent does the Health IT Literacy Guide aid purchasers in selecting health IT applications that are accessible to adults with different levels of health literacy?

Familiarity with Health Literacy. Purchasers reported limited familiarity with the concept of health literacy and how health IT should support it, inconsistent sources of information about it, mixed ideas about who was ultimately responsible for ensuring that products supported it, few tools for measuring product performance, and the belief that market demand was weak for products that served those with limited health literacy.

Participants were asked to define health literacy as it applies to their work as developers or purchasers. Overall, participants had many descriptions and definitions of health literacy and pointed out that there is no universally accepted definition of health literacy. Purchasers were less familiar with the concept of health literacy than developers. The following are examples of purchasers' descriptions of health literacy:

"Less emphasis on education and communication and more on goal-oriented action."
"The degree to which patients have the capacity to understand and have the information they need to make medical decisions for themselves and their family."
"To provide or identify someone in public health that a person can ask or have that information provided. To know who they can ask."
"Isn't it just the ability to read?"

Awareness of the Health IT Literacy Guide. A majority of focus group participants stated that they were not aware of the Health Literacy Guide nor did they believe other purchasers were familiar with it. Of the 26 participants in the purchaser focus groups, only 2 were aware that the Guide existed. As for participants' familiarity with AHRQ itself, one purchaser commented that researchers may be more aware of AHRQ, but many medical providers are less familiar with AHRQ and the resources they provide. Another purchaser commented that most purchasers of health IT products do not look to the government for health literacy resources, but rather rely heavily on vendors of health IT products to incorporate any health literacy guidelines into their products.

Main Focus of the Health IT Literacy Guide. Focus group participants were first asked their thoughts on the main point of the Guide. Purchasers agreed with developers that the Guide provided resources and guidelines that providers should use when creating health IT materials or when evaluating health IT products from vendors.

Usefulness of the Health IT Literacy Guide. Focus group participants were next asked how useful they thought the Guide was for helping developers and purchasers to evaluate health IT products.

Purchasers felt the Guide "covered the basics" of health IT design and health literacy principles. They agreed that the Guide would be useful for evaluating health IT products to ensure that they are accessible to all the populations they serve or to use when updating products in house, such as updating their Web site. Purchasers tended to agree with developers that the Guide would be most useful as a starting point or for those who are new to health IT and felt it would be more beneficial for users starting from scratch or not very far along in adopting EHR systems.

The findings from the focus groups echoed those from the environmental scan. The scan identified 9 tools and resources, 22 best practices manuscripts, and 30 examples of health IT products (go to Appendix G) that support the notion that the Guide is out of date.

Health IT Literacy Guide Appendix Checklist. When participants were asked specifically about the checklist located in the appendix of the Health IT Literacy Guide, a majority felt it was the most useful part of the Guide.

Although purchasers mostly agreed with developers that the checklist would be useful for evaluating health IT products, some purchasers felt that the checklist would be more useful for developers and found some of the questions difficult to answer as a purchaser of health IT products. Another purchaser mentioned that a short checklist is a great resource but might be useful only for a limited time because of rapid changes in technologies.

AHRQ as the Source of the Health IT Literacy Guide and Literacy Information. When asked their thoughts on whether the information in the Guide was believable, purchasers felt the information was straightforward and easy to use. As the source of the Guide, AHRQ was seen as more credible and less biased than a guide created by a vendor. Purchasers felt this information was needed when buying systems and products and when creating materials for consumers and patients. Although most purchasers felt the information in this Guide was believable, some purchasers mentioned that deciding what health IT products to buy is more of a financial decision and less about whether patients understand patient-related materials. Other purchasers mentioned that if an individual physician practice is already contractually committed to purchasing specific health IT products, addressing concerns about health literacy will not be a priority until a new contract is considered.

Use of the Health IT Literacy Guide. Similar to developers, purchasers felt that the checklist seemed to be the most useful part of the Guide. Purchasers anticipated that they would use the Guide or checklist to evaluate health IT products and vendors to ensure that they meet the recommended health literacy guidelines. Most purchasers agreed that they could use the Guide as a way to know what they should be looking for when purchasing certain technologies and health IT products. The Guide could prove helpful in the beginning of the process to help narrow down vendors by evaluating their products. Some purchasers also commented that the Guide could be used again toward the middle of the process once the materials have been created or products have been bought; they could use the Guide to check that the materials or products they are receiving and giving to patients are staying true to the guidelines presented in the Guide. A few purchasers noted that buying health IT products is largely a financial decision and if they are already tied to a contract for health IT systems or products, purchasers most likely will not evaluate potential products for health literacy.

Audiences. When asked for whom the Guide was written, purchaser focus group participants agreed with developers that the Guide was written to provide guidance for purchasers, such as health providers and practices, when evaluating health IT products for purchase. They also felt it gave purchasers information on what they should be searching for when buying health IT products.

Some purchasers felt that the Guide would not be very useful to them because it was up to the vendors, not the purchasers, to know this information and to make sure their products meet these guidelines. On the other hand, some purchasers felt that most health IT vendors and programmers do not have an adequate understanding of health literacy and could benefit the most from this Guide.

Focus group participants felt that medical practice personnel, including health care executives, physicians, and nurses, would also benefit from knowing this information.

Sources of Information on Health Literacy. When asked where they learned about health literacy, focus group participants mentioned various sources. Some purchasers said that they learned about health literacy from experts, studies, journal articles, or professional colleagues (clinicians). Some purchasers who were clinicians, social workers, or public health workers said that they learned about health literacy in medical or graduate school. Some purchasers learned about health literacy from colleagues who helped them to select products and from committees set up at their institutions to ensure that products and materials can be read and understood by those with varying levels of literacy. Two purchasers pointed out that they leave issues of health literacy to the vendors who supply their health education materials; one purchaser said, "We are hoping that the content is at the right level, but we don't always have control of it."

Purchasers were asked for resources they have used and found helpful that address health literacy specifically. Purchasers had more experience than developers with tools and resources that address health literacy in selecting products for their institutions. Although some purchasers said that they are directed to specific resources by their vendors or health IT groups, others mentioned the following sources of information on health literacy:

CDC (Centers for Disease Control and Prevention).
AHRQ.
Healthit.gov.
The Health Resources and Services Administration.
Patient-centered care groups.

Purchasers and Health Literacy. Only 2 of 25 purchaser participants reported that they evaluate potential products for health literacy before purchasing. Others said that they rely on the third-party vendors who provide the products to ensure that health literacy is addressed. One participant reported that he had never had a request from his organization to address health literacy, but if he did receive a request, he would work with the vendor to address it.

Several purchasers said that the responsibility for ensuring that purchased tools and resources address health literacy lies with the head administrators or chief executives in their organizations. Those participants who use third-party vendors stated that they often write into an RFP or ask that vendors consider issues of literacy and health literacy. Some purchasers felt that they were not able to ensure that products they buy could be used by those with varied levels of health literacy. One purchaser said, "As a purchaser, I don't really know where to go to say [i.e., determine] if these vendors have even looked at this or if this has been tested."

Barriers to Consideration of Health Literacy in Design and Purchase Decisions. Both developers and purchasers mentioned the lack of commonly accepted guidelines or sources of expertise for addressing health literacy.

Purchasers of consumer health IT products identified a number of barriers to consideration of health literacy in purchase decisions, including language and culture differences among expected users as well as legal concerns (leading to very long documents not likely to be read and understood). Other barriers identified by purchasers included the following:

The number of products to check for health literacy can be a barrier if a company has a large number of education materials.
Poorly trained developers who do not understand that addressing limited health literacy is not the same as simply rewriting for the lowest level reader.
Primary care practices are limited by having an EHR that limits or defines the educational materials and patient portal. Even if purchasers want to address health literacy, they may not be able to do so.
Most providers do not think that health literacy is an issue. They assume that patients can read and understand because they ask questions. There can be a lack of understanding of the value of considering health literacy and evaluating health literacy of already existing materials.

Return to Contents

3.3 Findings that Address Research Questions 3 and 4: In what ways can the Health IT Literacy Guide be improved or updated to be more timely, relevant, and useful to developers in designing health IT applications and purchasers in selecting health IT applications that are accessible to adults with different levels of health literacy?

Definition of Health Literacy. Through discussions with both developers and purchasers, it was found that no single, universally accepted definition of health literacy seems to exist, and both groups have varied thoughts on health literacy. Definitions of health literacy from both groups were not always accurate, and some found it difficult to differentiate health literacy from general literacy. Both developers and purchasers felt that the Guide could use some improvement in the introduction of the Guide (Section I) by providing a proper definition of limited literacy.

Intended Audience of the Guide. Developer and purchaser focus group participants often had conflicting views of who is accountable if health IT products do not address the needs of users with limited health literacy. Purchasers often felt that the responsibility lies with the vendors of the products and assumed it was being addressed, whereas developers themselves had mixed opinions as to who is accountable, with many saying the market should drive consideration of health literacy. Both groups recommended making it clear in the introduction of the Guide (Section 1) who the Guide's intended audience is and who should be using the Guide.

Update on Guidelines for Specific Health IT. A specific area of the Guide that both developers and purchasers agreed needed to be updated was the section on guidelines for specific health IT (Section III). Developers recommended adding specific guidelines for tablets because they are being used in medical practices and hospitals to obtain health information. Both groups also recommended revising the section on Mobile Phone, BlackBerry, and personal digital assistants (PDAs) by adding guidelines for smartphones and tablets, and deleting the information on PDAs. Developers also thought it would be helpful to have information on patient portals included in the Guide and checklist because they are becoming more prominent in medical practice and Meaningful Use requirements. Purchasers also felt the use of home-monitoring devices by patients will increase in the near future and suggested that this section be updated and improved.

Findings from Part 2 of the environmental scan support the thoughts of focus group participants on updated specific health IT and indicate that the evolution of hardware and software capabilities since the Guide was released necessitates updated guidelines for specific forms of health IT products. For example, although guidelines for computer kiosks in the original Guide continue to apply whenever kiosks are used, this particular type of technology may be less widely used than newer portable touchscreen tablets or mobile devices such as smartphones. Similarly, the rise in telehealth technologies requires new guidelines to be introduced. This section provides examples of emerging best practices as newer technologies become more commonplace. Most of these best practices identified in the environmental scan follow what are increasingly becoming normalized design standards when it comes to the user interface of IT products. Overall, the shifts in technology access and online activities have increased expectations about what type of characteristics make consumer IT products helpful for users.

Environmental scan findings also support updated on smartphones and other mobile devices. As mentioned in the introduction of this report, mobile device adoption has greatly increased since the release of the Guide, with mobile devices providing primary Internet access to certain populations affected by the digital divide (Smith, 2011). Moreover, the introduction of smartphones allows users to interact with various types of applications, or "apps," that can serve as data collection and data retrieval mechanisms. As such, findings from the environmental scan indicate that mobile applications need to consider unique data, interface, and memory issues.

Best practices from published research identified in Part 2 of the environmental scan include the following:

Ensure minimal memory usage to keep the application quick enough for a variety of resource-restricted devices (Kailas et al., 2010).
Account for limited battery availability when incorporating biosensor data collection mechanisms (Kailas et al., 2010).
Ensure the security and reliability of data collection mechanisms (Kailas et al., 2010).

As seen from the focus group findings, home monitoring devices were seen as another type of health IT where updated information was needed in the Guide. Findings from the environmental scan on health IT product evaluations include more specific guidance to support the fact that health IT vendors need to be flexible in integrating home monitoring devices based on consumer needs.

Best practices from published research identified in Part 2 of the environmental scan include the following:

Find and match usable devices for specific population needs (e.g., a spirometer, which is easy to use for those with arthritis, and touch screen technologies, which have been shown to work well with limited-literacy populations) (Zayas-Cabán and Dixon, 2010)

Business Case for Accessible Health IT. Developer focus group participants recommended including some information in the Guide on the financial benefits that can occur from developing or purchasing health IT products that are accessible to limited literacy patients. By building a business case for using these guidelines, developers felt it might get more attention because the cost of developing and providing materials is typically a major factor when choosing to develop or provide health IT products to patients. A number of focus group participants felt that since purchase of health IT products is largely influenced by cost and business drivers. Purchaser focus group participants felt the scope of the Guide should be expanded to include users and business or financial personnel who may contribute to the decision to develop or purchase health IT products.

Use of User Testing for Evaluation of Health IT products. Adding information on user testing for developing and evaluating health IT products was also mentioned by both developers and purchasers focus group participants. Both groups emphasized the importance of getting feedback from patients/users continuously throughout the development and evaluation process because they are the targeted audience for many of the health IT products. Participants agreed that the Guide should address how to involve patients/users in the health IT development or evaluation process so their feedback on how they would use the health IT products would be obtained. Purchaser focus group participants felt this Guide should also include more information or guidelines to consider when developing or providing health IT products for different cultural groups, and how to tailor the system/materials around specific cultural populations of relevance to certain geographic areas.

Health IT Literacy Guide Appendix Checklist. Both developer and purchaser focus group participants perceived the checklist to be the most useful part of the Guide and felt it could serve as a valuable resource for evaluating health IT products. To improve the checklist, both developers and purchasers recommended that the checklist should be expanded to include additional IT platforms, such as tablets and smartphones. Another recommendation from both developers and purchasers was to make the checklist more interactive by adding a ranking or scoring system to the checklist may be beneficial to purchasers so they can assess how they are doing when creating materials or when evaluating different products in making purchasing decisions. One purchaser also recommended that, in the section on iterative testing and revision, it would be beneficial to add information on gathering feedback from patients or users of the health IT products, including focus groups or usability testing.

As Web sites become more interactive, as recommended for the Guide by focus group participants, additional guidelines need to be considered, including providing clear directions for how users can enter personal information.

Best practices from published research identified in Part 2 of the environmental scan include the following:

Place limited information-gathering requests at one time on-screen (Peters et al., 2009).
Provide confirmation of data entry to minimize confusion for users (Peters et al., 2009).
Use branching logic to minimize user cognitive overload (Peters et al., 2009).
Allow users to enter detailed information immediately after entering initial data related to details (to minimize navigation overload) (Peters et al., 2009).
Color-code user-selected responses (Wolpin and Stewart, 2011).
Implement portability for future access to PHI from multiple access points (Hernandez, 2009).
Make privacy and information use policies clear and easy to understand (Grossman et al., 2009).
Provide attribution so the source of information is made visible (Grossman et al., 2009).

In addition, as more users have access to the Internet via mobile devices, Web site developers need to ensure sites load quickly on mobile devices and are easy to use on smaller screens (Kailas et al., 2010).

Dissemination of the Health IT Literacy Guide. Focus group participants were asked where they might expect to see this Guide. Both developers and purchasers indicated that although AHRQ was a good resource for researchers, few clinicians or developers of health IT products are aware of AHRQ or would ever think to look to AHRQ for this type of Guide. Making this Guide available through professional organizations was the most frequent response from both groups (Table 12).

Table 12. Professional organizations mentioned by developers and purchasers

Section Objective	Key Findings
HIMSS AMIA American Health Information Management Association American Hospital Association American Medical Management Association American Telehealth Association American Nurses Association Association of Medical Directors of Information Systems	American Academy of Family Physicians Society for Participatory Medicine American Academy of Pediatrics National Association of Community Health Clinics American College of Healthcare Executives American Society for Training and Development

Focus group participants also said they might expect to see this Guide published on other government agency Web sites and certification organizations, including the CDC, Substance Abuse and Mental Health Services Administration (SAMHSA), ONC or HealthIT.gov, NIST, The Joint Commission, CCHIT, and the Health Information Technology Regional Extension Centers. Participants also named grant-sponsoring organizations, such as Robert Wood Johnson Foundation and the Henry J. Kaiser Family Foundation, and public health organizations for places they might expect to find the Guide.

Lastly, a few participants recommended renaming the Guide to make it easier to find via a Web search. Developers mentioned that they would expect to locate the Guide using any common Web search engine, such as Google, employing search terms like "development design."

Participants were asked how AHRQ should try to reach developers and purchasers with the Guide. Developers and purchasers agreed that promotion of this Guide through various health IT newsletters and listservs™, and improving search result optimization, would make it more accessible to the intended audience. Developers noted that the Guide should be incorporated into developer education as part of health IT design training. Purchasers noted that the Guide might be useful in educational settings, such as public health, medical, and nursing schools, in teaching the importance of these health literacy guidelines to future medical professionals.

Again, most developers agreed that sending the Guide to various professional organizations, such as those listed previously, would be most beneficial to reach developers and purchasers and publicize the Guide. They mentioned HIMSS most often as a useful way to disseminate the Guide. Several participants suggested using the HIMSS Annual Conference to showcase the Guide and to help disseminate the Guide to both developers and purchasers.

Return to Contents

3.4 Summary of Findings Related to the Health IT Literacy Guide

The table below summarizes the main findings from the focus groups with developers and purchasers described above in detail. These findings are organized according to the Guide sections for which they apply.

Table 13. Summarizing findings related to the Health IT Literacy Guide

Section Objective	Key Findings
Section I: Introduction To provide definitions of "literacy," examples of health IT applications used by populations with limited literacy, and benefits of limited literacy accessible, health IT design.	Participants from both developer and purchaser groups thought the Guide was written to provide guidance for purchasers, such as health providers and practices, when evaluating health IT products for purchase. Participants had many descriptions and definitions of health literacy and pointed out that there is no universally accepted definition of health literacy. Developers were more familiar with the concept of health literacy than purchasers. Both developers and purchasers reported limited familiarity with the concept of health literacy and how health IT should support it, inconsistent sources of information about health literacy, and few tools for measuring product performance. Purchasers had more experience than developers with tools and resources that address health literacy because they select products for their institutions. Often the tools and resources are suggested and chosen by outside vendors.
Section II: Overview of Health IT for Limited Literacy Populations To provide advantages offered by health IT for limited literacy users and examples of predominant health IT applications used by consumers	Both developers and purchasers reported that market demand was weak for products that served those with limited health literacy. Both developers and purchasers mentioned the lack of commonly accepted guidelines or sources of expertise for addressing health literacy. They also said that the lack of a well-defined cost of making sure products address health literacy was a barrier to consideration.
Section III: Principles of Accessible and Usable Health IT To provide the importance of universal design and a description of universal design principles and to provide accessibility guidelines for general health IT and recommendations for specific health IT	When asked about best practices for developing applications designed for those with limited health literacy, participants pointed to user-centered design or usability testing with the intended audience as best practices. Participants felt that the acceleration of health IT acquisition and adoption by providers has had an uneven impact on provider awareness of consumer use of health IT systems and issues related to health literacy. Participants were more aware of general guidelines for all health IT, such as plain language, relevant content, and cultural awareness. They were less aware of basic universal design principles.
Section IV: Additional Resources To provide articles, Web sites, and other resources on the topics covered in the Guide	Participants were unfamiliar with the Guide and had not seen many of the additional resources presented. They pointed out that they were out of date and should be updated.
Appendix: Checklist To provide accessibility guidelines for general health IT and specific health IT in the form of a checklist	Most participants found the checklist to be the most useful part of the Guide. They would use the checklist, especially if it were updated (to include mobile technology such as tablets) and interactive (such as giving a "score").

3.4.1 Findings from the Expert Interviews Related to Tools and Resources that can Aid Purchasers and Developers of Health IT

Tools and Resources for Developers. Expert Interview participants were asked which tools or resources they found most useful in helping developers to design health IT applications that are accessible to consumers with low health literacy. Although some participants had no specific input (because they did not use tools and resources), others named the following as useful tools that should be considered for inclusion in Section IV of the Guide:

The Plain Language Widget and Application. One health literacy expert described this tool as a dictionary that translates technical terms to more layperson terms.
The Harvard School of Public Health Literacy Web site.This Web site was cited as helpful because it contains several tools and resources to use to ensure that language used is "plain" and written at an appropriate grade level.
The Suitability Assessment of Materials (SAM) Instrument (PDF File). This instrument was originally developed for print materials but was modified for Web tools by the health literacy participant, who uses it at all phases of design.
The Usability.gov Web site. A health literacy expert said, "We've used it to assess different parameters of Web sites…it gives us very tangible items to work on." An HCI developer said "It's a good resource for some basics."
Don't Make Me Think, by Steve Krug. This book "introduces usability concepts in a common sense way for Web design," and was mentioned by an HCI expert.
Jakob Nielsen's Web site. This was described as a resource on accessibility that contains good "general principles for designing usable software that can be extrapolated to those with limited health literacy."

Tools and Applications for Purchasers. Experts were asked for specific tools and resources they have found to be useful in helping purchasers of health IT applications to ensure that the tools they buy are accessible to consumers with low health literacy. Seven of nine experts (three health literacy, two consumer health IT, two HCI) felt that they did not have the expertise to answer this series of questions. The two usability experts who did answer these questions recommended usability testing as an important and useful tool for purchasers. One said, "If it's a bid purchase at an institution, when the decision is narrowed down to the final three, run a usability test with the [end] user. Use that data to help in the final decision." In addition, one usability expert mentioned using questionnaires as a way to determine whether a product is accessible for those with limited health literacy, specifically The Questionnaire for User Interaction Satisfaction (QUIS) (Word File) developed by Ben Shneiderman and Kent Norman at the University of Maryland (Chin et al., 1988). A number of usability-related questionnaires are found at this Web site and should be considered for inclusion as additional resources in Section IV of the Guide.

Related Findings from Experts. When asked about tools and resources that were not helpful to developers, two experts said that readability formulas are not helpful. As one usability expert said, "[Readability formulas] are the wrong way to go. I don't think that writing to a grade level is the right way to go." One health literacy expert said, "Any basic readability stuff is the kiss of death because designers then do not consider how to represent the data that is written at a certain grade level…They just stop designing and turn it over to the readability police." When asked what resources or tools are not helpful for purchasers of health IT applications—to ensure that the tools they buy are accessible to consumers with low health literacy—one usability testing expert said, "I think that just watching a demo from a salesman is a terrible way to decide."

According to one health literacy and one HCI participant, many health IT developers lack basic IT development and design skills in general, including user interface design and user testing experience. Creating linkages between health IT developers and "really good" general IT developers and creative IT teams is another way to ensure that accessibility for low-health literacy populations is addressed. One health literacy expert said, "Most of the health IT people I've come across don't have the background in the broader, more innovative fundamental IT design."

Two usability experts and one consumer health IT expert suggested that usability testing is a critical step in the development process. Conducting this testing with the specific intended audience (i.e., both individuals with limited literacy and limited health literacy) at several points in the development process is crucial to ensuring that health IT applications are successful and accessible. Including information on usability testing in the Guide was also supported by focus group findings. One HCI expert suggested that developers of health IT applications should be limited to "writing the code for which they've been trained" and be linked up with designers who know how to design accessible software. "History has shown that you'll never succeed in…making people understand two different disciplines," he said.

3.4.2 Findings from Expert Interviews and Environmental Scan Related to Designing Accessible Health IT Products

Since the Guide was released, evaluations of health IT products have led to new best practices in health IT. Many of these practices operationalize the principles that were introduced in the Guide. This section provides examples of best practices and examples from experts that can provide specific guidance for health IT developers and purchasers. These best practices have been shown to increase consumer interest in health IT products by (1) providing a user experience that is tailored more specifically to user needs and (2) allowing users to interact with the content easily and to the extent they are interested in more in-depth information.

Trends and Best Practices in Health IT Development. When asked about recent trends in health IT development for those with limited health literacy, most experts suggested that trends for those with limited health literacy would also apply for those with limited literacy and consumers in general. As one usability expert said, "There is a substantial amount of research that shows when you help low literacy people you help the high literacy people too…When you help [people with] low literacy, you help everyone."

A total of four experts from all segments mentioned mobile health as a recent trend that affects health IT development for those with low health literacy. They pointed out that although not everyone has access to a computer, almost everyone, regardless of literacy level (health or otherwise), has a cellphone or smartphone. This technology has encouraged the design and development of texting and smartphone apps. Updating guidelines included in the Guide on mobile technologies was also supported by focus group findings.

A usability expert also mentioned to the importance of considering all methods that learners use, such as text, video, podcasts, gaming, question-and-answer quizzes, avatars, and talking to virtual health care providers.

Increased use of video on health topics by consumers was mentioned as an important trend. Using videos ensures that consumers do not have to read and only have to click on a video to get the information they need. Two important considerations are making sure that videos are short and to the point and that the consumer is informed at the beginning how long the video will last. A usability expert mentioned that one challenge with video is the lack of privacy at work. One expert stated that "people who don't want [other] people at work to know that they're pregnant but [they're on a Web site] and everyone is wondering why you're listening to pregnancy videos."

When asked about important considerations and factors that should be kept in mind when developing tools and resources for those with limited health literacy and should be considered for updating the Guide, experts pointed out that design methods for limited literacy consumers were often similar to design approaches for any consumer:

Get to know the audience. As one HCI expert said, "Developers have to go out and spend time with the people they're designing for. That would improve the development immensely. There is no substitute for having people in direct observation and participation."
Create "personas" as part of the predevelopment process to understand who will be using the site or resource. This persona is a composite figure with individual characteristics that match a targeted end-user.
Organize directions correctly. Directions should be placed throughout the site or application, not just at the beginning. One health literacy expert said, "On some Web pages they give you all the instructions at once and you have to go back and refer to them to figure out what you need to do. People need a step-by-step process."
Pay attention to specific design factors: Organize sites correctly with no scrolling; use "breadcrumbs" to ensure that the user can easily get back to previous pages; design a search function to have automatic fill-in capability in case consumers do not know the spelling of a term; remove text heaviness; and use appropriate font, size, rollovers, etc.

When asked about best practices for development of applications designed for those with limited health literacy, seven of eight experts pointed to user-centered design or usability testing with the intended audience as best practices. A health literacy expert said, "One of the best practices is [to] pull in the group you are going to target with that application. If you are going to target American Indians, you should pull them in." Experts felt strongly that users should be involved in all phases, beginning before development and continuing through design. Iterative usability testing at all phases with the end-user and validation testing were also seen as crucial.

Best Practices for Purchasers. Both usability testing experts agreed that a best practice in helping purchasers is to conduct usability testing. One said, "Have all the people who are going to use it to try it out. In large organizations, the people who buy software are different than the people who use it. Consider the entire product in its context—how patients would actually use it; not just how it is shown in its demo use." Information on usability testing has previously been recommended for inclusion in Section III of the Guide.

Advancements in Consumer Health IT. Experts were asked how rapid advancements in technology and the Internet use have changed the development of consumer health IT and how these advancements have affected consumers. Overall, although experts agreed that some rapid advancement has occurred, especially in the increased availability of smartphones and other mobile technologies, they disagreed on whether these advancements have had much effect on consumers.

According to one health literacy expert, rapid technological advancements have brought the importance of health literacy to the forefront: "When I first started, limited health literacy skills were not addressed. Hospitals and drug companies (anyone who gives patient education) are now having to think about it. That's the way things are moving."

A health literacy expert said that these rapid advancements have been positive because they make consumers "feel more empowered that they can manage their health conditions. [It improves their] self-efficacy." However, a usability expert said that the effect these rapid advancements have had is negative: "[It's] opened up some consumers to greater stress and worry because they don't understand information because it's for health care providers, like the files that come to the EMR from labs—that data is in ‘doctor speak' and consumers have to interpret that data themselves."

Experts felt that the acceleration of health IT acquisition and adoption by providers has had an uneven impact on provider awareness of consumer use of health IT systems and issues related to health literacy. Some providers have been slow to adapt to use of health IT systems. An HCI expert said, "Adoption by providers is uneven and they are uneasy. The majority are still struggling with getting the right script to the pharmacist, not getting the information to the patient." One usability expert who reported going on site visits to provider offices said that "clinicians are getting more opportunity to push educational materials to consumers and [are] more sensitive to language, but I don't see health care providers looking for low health literacy information."

Some providers have adopted new technology and are aware of health literacy but have a difficult time understanding that patients have also adopted new technologies and are often eager for change. A health literacy expert explained, "I think that providers are becoming aware that consumers are a little more savvy about their health. In some situations I have seen some providers feel intimidated." Another health literacy expert said, "There is still a misperception that the public hasn't adopted the same technology. In big urban centers there is still a bias against the great unwashed and poor underserved."

The incorporation of consumer health vocabularies into health IT applications was seen as very important. As one health literacy expert put it, "We need to be more concerned with the words and terms that patients use. [This can get done by getting] rid of the lawyers and head scientists. It's both elitism and very ingrained tradition. The common language has never been considered the scientific one."

Some suggestions from experts that should be considered when updating the Guide included the following:

Incorporate the common misspellings used by consumers on search engines with auto-fill options.
Incorporate the common terms that consumers use such as "sugar" instead of "diabetes." "This also takes into account different cultures and how they use words" said one health literacy expert. Have "interest and understanding" of popular language and discourse.
Ensure that consumers are able to verbally pronounce the condition if they do not know how to spell it. Have an audio button on the Web page that pronounces more technical terms.

In addition to the content included in Section III of the Guide, health IT product evaluations have shown that it is important to account for consumer health vocabularies in developing consumer health informatics products.

Best practices from published research identified in the environmental scan that should be considered for updating Section III of the Guide include the following:

Use simple medical language instead of medical jargon (e.g., "flu" vs. "influenza") (Peters et al., 2009; Sox et al., 2010).
Use consumer health vocabulary tools in designing consumer health informatics applications (e.g., the PlaneTree Classification System) (Keselman et al., 2008).
Test various ways of presenting complex health information (e.g., graphical representation of information) and offer multiple and tailored presentations when possible (Keselman et al., 2008).

There are additional general best practices (Kramer-Jackman and Popkess-Vawter, 2011) that have surfaced since the Health IT Literacy Guide was released that address broad consumer health IT issues related to the increased interactivity of consumer health IT products and increased availability of products such as PHRs (Peters et al., 2009). The implementation of these best practices has been shown to improve the consumer experience by providing users with an increased sense of trust in the products they are using through user control of their information, increased user communication capabilities, external educational and clinician support systems, and general interface accessibility (Cruickshank et al., 2012).

According to findings from the environmental scan, consumer health IT users want health IT products to allow them to communicate with their providers, share health information with them, and connect with others that share their health-related experiences.

Communication best practices from published research identified in Part 2 of the environmental scan that should be considered for updating Section III of the Guide include the following:

Allow patients to prepare for appointments by sending information to providers electronically ahead of time (Weitzman et al., 2009).
Allow user feedback, flagging, or reciprocal notification to amend information (Weitzman et al., 2009).
Include timely message checks by providers to ensure that users receive a timely response (Reti et al., 2010; Walker et al., 2009).
Allow users to use their PHI as a conduit to connect to patients with similar conditions (Halamka et al., 2008).

Findings from Part 2 of the environmental scan also suggest that health IT consumers find products most helpful when they improve their health care experience directly (e.g., when they see clinicians using information they added to their PHR or when they have educational assistance and technical support available to them outside of the health IT application).

Best practices from published research identified in Part 2 of the environmental scan that should be considered for updating Section III of the Guide include the following:

Take advantage of ancillary staff positions to support consumer education, including health coaches and technology/information system navigators (Keselman et al., 2008; Weitzman et al., 2009).
Provide an easy set of audio/video instructions (Whitten et al., 2011).
Partner with libraries and community-based organizations to increase users' computer skills (Hernandez, 2009; Whitten et al., 2011).
Have clinicians reassert instructions for consumer health IT products during appointments or explicitly refer to data collection uses (Hernandez, 2009).
Integrate monitoring device data with patient–provider communication workflow (Ball et al., 2007).

The Importance of Tailoring. All participants thought that personalized and tailored content for low-health literacy consumers was crucial. One health literacy expert said, "I don't think generic information helps people with low health literacy. The more tailored, the more likelihood of the individual being more treatment compliant."

Participants were asked their thoughts on ways to further personalize the user experience by customizing health information while addressing health literacy. Many experts said that the same best practices for health literacy and good design and development should still apply, even with more personalization. One health literacy expert suggested gearing information toward a consumer's particular condition. Another health literacy expert recommended addressing someone in the first person during login, using the first name of the user, and putting content in the second person (e.g., "you need…") to "speak directly to the reader." A usability expert suggested, "One of the things is to build in flexibility—the ability to show or hide information. Show me only the first two things I need to look at; not everything at once." One HCI expert suggested that the "industry needs to borrow from social networking approaches [around] engagement. Let users personalize their portals, network with others with their condition, have a live news feed, [so that the site can be] adopted as a daily part of their lives." Another HCI expert suggested that it was essential to have a home page that allows a user to select who he or she is (patient, caregiver) and tailor information accordingly.

Since the release of the Guide, the ability to provide users with personalized information online has greatly increased, and published research on health IT products often identifies integrating customized health information to personalize the user experience as a best practice.

Best practices from published research identified in the environmental scan that should be considered for updating Section III of the Guide include the following:

Use a patient's medical conditions and PHI to present information about new drugs, treatments, clinical trials, and other resources that may be relevant (Wolpin and Stewart, 2011).
Use patient medication lists from PHRs to trigger drug interaction warnings (Peters et al., 2009).
Incorporating tailored health dashboards to increase medical situational awareness of users based on the conditions they are managing, or their individual risk factors (Ball et al., 2007)
Suggest ways to help patients manage their own care by integrating PHI and biomedical technologies (Ball et al., 2007).

Access for Nonpatients. Although all participants agreed that health IT applications (such as a patient portal) should explicitly provide access for nonpatients—such as providers, family members, and caregivers—most participants mentioned that access should be limited to those who "need" the information. Providing this access was seen as an important way to ensure coordination of care for patients who have caregivers, such as the elderly and others who may have low literacy and receive assistance from others. However, the number one concern and barrier to access expressed by participants was privacy. There was a sense that existing HIPAA laws were not adequately addressing the new technologies, such as patient portals, and that further consideration and agreement nationwide (instead of state by state) are needed regarding privacy and access issues.

In expressing concern about access to nonpatients, one health literacy expert said, "It can be very scary for someone who feels vulnerable. There are caregivers who take advantage of people who are very sick." A usability expert said, "[It is] absolutely important to provide access to everyone [who needs it]. This doesn't mean everyone should [have access]."

Findings from the environmental scan indicate that users want to be able to allow or restrict access to health information stored in the health IT products they use based on their preferences.

Best practices from published research identified in Part 2 of the environmental scan that should be considered for updating Section III of the Guide include the following:

Allow users to share access to their PHR or PHI via proxies using clear and easy-to-use consent mechanisms (Ball et al., 2007; Halamka et al., 2008; Weitzman et al., 2009).
Allow users to store health information in a PHR for more than one individual (Peters et al., 2009; Reti et al., 2010).
Assess needs of family members, caregivers, and other alternate users in product design (Keselman et al., 2008).
Allow users to restrict access to providers and entities they trust and limit the type of information each type of user can see (e.g., by tethering provider-based medical records with a PHR) (Moreno et al., 2007).
Develop clear and understandable privacy policies that specify who can access patient data and how data will and will not be used (Grossman et al., 2009).

Learning Supports. Experts in instructional design describe certain kinds of instructor guidance provided to learners as "scaffolding." Participants were asked their thoughts about information scaffolding after being read the following definition: "Information scaffolding provides support that allows users to move from one concept or screen to another, building on previous knowledge. For IT, it is used for multiple navigation opportunities for users—so they can go back and read about something they are not sure of or skip a screen/information that they already know. It could also be used to help with word pronunciations and definitions."

Most participants agreed that information scaffolding was important, but several pointed out that it was already a standard part of good design and therefore not a new concept or one important only for low literacy or low health literacy audiences.

One health literacy expert said, "As you begin to do research on an issue, you get taken in different directions. You need to be taken back to your original screen and that is true for more than just low health literacy people." Another health literacy expert pointed out that "good information is not linear."

Findings from the environmental scan show that information representation has evolved since the release of the Guide, with an increased integration and visualization of datasets across disciplines made easier by using open-source software and accessible data sources. In addition, health IT product evaluations have shown that users prefer multiple ways of obtaining information on an interface, so purchasers and developers should provide easy-to-use information scaffolding opportunities in the user experience (Keselman et al., 2008; Peters et al., 2011; Weitzman et al., 2009). Information scaffolding refers to support that allows users to move from one concept or screen to another, building on previous knowledge.

Best practices to increase reading and comprehension from published research that should be considered for updating Section III of the Guide include the following:

Provide "mouse-over" explanations of medications and definitions of medical conditions (Ball et al., 2007; Weitzman et al., 2009).
Provide "drill-down" capabilities for treatments and summaries of research findings using hyperlinks (Weitzman et al., 2009).
Provide multiple ways to search for information, including alphabetical lists, and autocomplete enabled search boxes (Weitzman et al., 2009).
Use innovative visualizations to present data to allow data mining by users (Goldberg et al., 2011).
Organize content to optimize behavior change (e.g., content organization should be informed by user needs and context, determinants of health, and behavior change theory) (Goldberg et al., 2011).
Use systems designed to support localization—different languages, different literacy or numeracy levels, different visual approaches—for presenting the same health information (go to http://www.pgsi.com/ for a product example).

Existing Accessibility Guidelines (Section 508 Compliance). Overall, participants felt that the current Section 508 Web site accessibility regulations are, as one health literacy expert said, "a start." Two participants (health literacy and usability) felt that, because of expense and time, most developers and designers were not following the current regulations: "This is the very least anyone can do. On the other hand, research shows people are not doing even this."

One usability expert suggested that current Section 508 regulations should be used as a proxy for yet-to-be-developed guidelines for audiences with low health literacy. Developing new regulations that take into account low health literacy will, according to one health literacy expert, take time and rigorous testing.

Information on Section 508 regulations should be considered when updating Section III of the Guide.

Return to Contents

3.5 Discussion

This evaluation of the Health IT Literacy Guide surfaced a noticeable gap between the focus group findings in which the collective experience of purchasers and developers of health IT showed little attention on limited literacy user requirements when designing or purchasing health IT, and the findings from expert interviews and the environmental scan, which identified evidence that literacy barriers in the health context are creating real difficulties for patients and providers, and can be addressed through resources such as Web sites, best practices guides, and product examples in which limited literacy user needs are a specific focus.

The finding of limited Web site activity for the Guide is consistent with focus group discussions with developers and purchasers, indicating that few were aware that the Guide existed or had accessed the Guide. A number of focus group participants reviewed a copy of the Guide before attending the focus group. Thus, focus group findings are best understood in the context of individuals who recently had (in many cases) their initial exposure to the Guide. In large part due to limited use, the Guide has had a minimal impact on developers who design health IT applications or purchasers who select health IT applications accessible to adults with different levels of health literacy.

Developers and purchasers indicated that there would be value in having a succinct resource, particularly one with a checklist or similar tool, to help familiarize or remind them of the important principles of universal design identified in Section 3 of the Guide, and how to recognize their use. In this way, they endorsed the Guide as being valuable, an affirmative finding for research questions 1 and 2 described in detail in Chapter 3.

A number of improvements to make the Guide more current and more useful to developers and purchasers are suggested in Chapter 4—primarily through updating different areas of the Guide content, enhancing the checklist, and updating helpful resources for further reading.

The request from some participants for the creation of an interactive checklist was intended to help improve and streamline their process for assessing and introducing universal design principles, especially those that address limited literacy users, into the design and purchase of health IT. If an interactive tool is developed, it should (itself) follow universal design principles and leverage techniques such as heuristic analysis, user testing, and prototyping to increase the likelihood that the tool will be valued and useful to its targeted audience.

None of the resources identified in the environmental scan was a direct substitute of the Health IT Literacy Guide, and the scan did not identify an analogue to the checklist found in the Guide. For this reason, the primary recommendations are to: (1) update the Guide, (2) develop the Guide's checklist, (3) disseminate the Guide and expand the intended audience of the Guide to include developers, purchasers, users, and business decisionmakers in management or executive roles.

Return to Contents

3.6 Limitations of the Findings

Given the qualitative methods of research and limited number of participants, results from the expert interviews and focus groups do not represent a statistically valid sample of developers and purchasers of health IT from the broader population. However, since a cross-section of the target audience for the Guide was included, feedback received can be used as input from these key audiences.

Return to Contents
Proceed to Next Section