Evaluation of Health IT Tools and Resources Available at the AHRQ NRC for Health IT Web Site: Final Report

The evaluation of the Health IT Literacy Guide was composed of an environmental scan, expert interviews, and focus groups with developers and purchasers of health IT. Office of Management and Budget (OMB) clearance was secured for the focus group research by AHRQ. RTI Institutional Review Board (IRB) clearance was secured for the focus groups and interviews. Table 8 provides a summary of the methods, and the following sections describe each method in greater detail.

Table 8. Methods summary

Method	Strengths	Why Chosen?	Limitations
Environmental scan	Summarize and categorize available resources	Efficient and effective method to identify relevant guides, best practices, and "model" health IT products released since 2007	It is difficult to capture all resources. This scan was not intended to be an exhaustive literature review.
Expert interviews	Hear directly from experts about resources, best practices, and their impressions of trends in health IT literacy	Excellent way to gather rich data that go beyond what can be listed in a survey and to collect information to inform the focus groups	Small number of interviews (n = 9) means that not all resources and best practices were discussed. Qualitative research findings may not be generalizable; these were the experiences of 9 experts only.
Focus groups with developers and purchasers	Hear directly from intended audiences regarding whether the Guide is reaching them and meeting their needs	Ability to gather rich data from a large number of intended audience members	Intended users may not have experience using the Guide.

2.1 Environmental Scan

2.1.1 Introduction and Purpose

From August to November 2011, the RTI project team conducted an environmental scan to identify examples of tools, resources, products, and best practices published since the release of AHRQ's Health IT Literacy Guide.

The purpose of the scan was twofold: (1) to help answer research questions 1 and 2 (To what extent does the Health IT Literacy Guide aid developers in designing and purchasers in selecting health IT applications that are accessible to adults with different levels of health literacy?) by determining if the information provided in the Health IT Literacy Guide was up to date, and (2) to help answer research questions 3 and 4 (In what ways can the Health IT Literacy Guide be improved or updated to be more timely, relevant, and useful?) by determining what, if any, guidance documents and best practices have emerged since 2007 and might be suitable for a future version of the Guide. The environmental scan built on the recently released AHRQ report, Improving Consumer Health IT Application Development: Lessons Learned from Other Industries.

The environmental scan included three parts:

1. Identify tools and resources released since 2007 that can aid purchasers and developers of health IT, including guidance documents, how-to's, and checklists.
2. Identify examples of recent best practices from the literature (published since 2007) pertaining to developing accessible health IT products (including universal design principles).  
3. Identify examples of recent (since 2007) health IT products that have been found to be effective with populations with limited literacy skills.

The following sections describe the environmental scan methods used to accomplish each part of the scan, including:

• Search criteria.
• Parameters for sources reviewed.
• Databases and other sources searched.
• Data collection and analysis.

2.1.2 Part 1: Identify Tools and Resources that Can Aid Purchasers and Developers of Health IT

Environmental Scan Part 1 Activities Review of relevant materials from AHRQ's Health IT Bibliography. Review of resources provided by and linked to on health IT–related organization Web sites. Web search using approved search terms. Review of IT forums and discussion groups for suggested tools and resources. Inclusion Criteria for Environmental Scan Resources Retained Published later than 2007. Provides specific guidance, recommendations, how-to's, and/or checklists. Addresses literacy and/or health literacy accessibility issues such as physical, cognitive, and technology challenges, or guidance for consumers in making health IT product choices.

 

The following is a detailed description of each scan activity for Part 1.

AHRQ Health IT Bibliography Review. The AHRQ Health IT Bibliography (AHRQ, 2011) consists of approximately 140 resources, including both peer-reviewed articles from professional journals and Web-based resources from highly respected health care and IT organizations. The AHRQ Health IT Bibliography is organized under the following five umbrella categories:

• Organizational Strategy.
• Technology.
• Evaluation.
• Operations.
• Populations and Perspectives.

To identify relevant resources from the AHRQ Health IT Bibliography, the RTI project team reviewed abstracts and summaries of all articles and resources in the bibliography based on the inclusion criteria. Based on this review, 0 resources were retained.

Health IT Organizations and Web site Resources Review. The RTI project team conducted a Web search using the approved search terms listed in Appendix B to identify organizations that provide health IT–related information and tools. Resources were reviewed that were provided by and linked to on the following organizations' Web sites:

• Agency for Healthcare Research and Quality (AHRQ) (http://www.ahrq.gov).
• American National Standards Institute (ANSI) (http://www.ansi.org).
• American Medical Informatics Association (AMIA) (http://www.amia.org).
• American Society of Health Informatics Managers (https://ashim.com/).
• Center for Strategic and International Studies (http://csis.org).
• Certification Commission for Healthcare Information Technology (CCHIT) (http://csis.org).
• European Institute for Health Records (http://www.eurorec.org).
• Health Information Technology Standards Panel (http://www.hitsp.org).
• National Association of Station Chief Information Officers (NASCIO) (http://www.nascio.org).
• National Institute of Standards and Technology (NIST) (http://www.nist.gov/index.html).
• Office of the National Coordinator for Health Information Technology at the U.S. Department of Health and Human Services (ONC) (http://www.healthit.gov).
• Partners in Information Access for the Public Health Workforce (http://phpartners.org/).

The RTI project team reviewed the summaries and introductions of the eligible resources using the inclusion criteria to determine which resources would be retained.

Web Search. The RTI project team conducted a Web search using the set of approved search terms listed in Appendix B. The project team used a Boolean search combining general search terms and Part 1–specific search terms. For example, the following search was performed:

• (health IT OR electronic medical record OR health Web site) AND (accessibility OR literacy) AND (checklist OR guidelines OR design principles).

The RTI project team reviewed the summaries and introductions of each of the eligible resources using the inclusion criteria to determine which resources would be retained.

IT Forums and Discussion Groups Review. To identify IT forums and discussion groups, the RTI project team conducted a Web search of the most active IT developer discussion groups. Based on that search, the team examined the following forums for suggested tools and resources:

• Android (http://developer.android.com/support.html).
• Apple (https://developer.apple.com/devforums).
• Chrome/Google (https://groups.google.com/forum/?fromgroups#!forum/google-chrome-developer-tools).
• CodeGuru (http://forums.codeguru.com).
• Dev Shed (http://forums.devshed.com).
• Google (https://developers.google.com/speed/public-dns/groups).
• Mozilla (https://developer.mozilla.org/en-US/docs/Developer_Guide).
• XDA (http://forum.xda-developers.com).

The RTI project team scanned each forum for relevant discussion threads based on the approved search terms in Appendix B. The RTI project team then reviewed resources and tools suggested in relevant discussions using the inclusion criteria.

2.1.3 Part 2: Identify Examples of Best Practices from the Literature Pertaining to Developing Accessible Health IT Products (Including Universal Design Principles)

Environmental Scan Part 2 Activities Review of relevant materials from AHRQ's Health IT Bibliography. Literature scan using approved search terms and parameters. Inclusion Criteria for Environmental Scan Resources Retained Published later than 2007. Addresses value of clear communication principles in consumer health informatics products to improve user experience; needs of populations with limited health literacy or accessibility issues such as older adults, minorities, and individuals with disabilities; patient-centered or consumer-focused design and development process; or implementation and evaluation of consumer health informatics products.

 

The following is a detailed description of each scan activity for Part 2.

AHRQ Health IT Bibliography Review. To identify relevant resources from the Health IT Bibliography, the RTI project team reviewed abstracts and summaries all articles and resources in the bibliography using the inclusion criteria for Part 2.

Literature Scan. The RTI project team conducted a literature scan using a set of approved search terms and parameters. The search included research in peer-reviewed academic journals, research reports, dissertations, and symposia. The RTI project team used a Boolean search combining general search terms and Part 2–specific search terms. For example, the following searched was performed:

• (health IT OR electronic medical record OR health website) AND (accessibility OR literacy OR universal design OR patient-centered) AND (best practices OR design considerations OR evaluation OR implementation).

Go to Appendix B for a full list of search terms.

The project team scanned articles from the following databases:

• Academic Search Premier.
• PubMed.
• Association for Computing Machinery (ACM) Digital Library.
• Computer Database.
• ArticleFirst.
• PsycINFO.

The project team reviewed each of the article abstracts using the inclusion criteria for Part 2.

2.1.4 Part 3: Identify Examples of Recent Health IT Products that Have Been Found to be Effective with Populations with Limited Literacy Skills

Environmental Scan Part 3 Activities Web search using approved search terms. Reviewed resources provided by and linked to from health IT–related organizations and Web sites. Literature scan using approved search terms and parameters. Inclusion Criteria for Environmental Scan Resources Retained Released later than 2007. Product with a specific name or brand (excluded references to general products such as EHRs or personal health records [PHRs]). Developed specifically to meet the needs of individuals with accessibility challenges such as limited literacy, cognitive delays, hearing impairment, visual impairment, dexterity issues, etc.; tested with individuals with accessibility challenges; or exemplified best practices identified in Part 2.

 

The following is a detailed description of each scan activity for Part 3.

Web Search. The RTI project team conducted a Web search using a set of approved search terms listed in Appendix B. The RTI project team used a Boolean search combining general search terms (e.g., health literacy, usability, accessibility, health) and Part 3–specific search terms (e.g., application, decision-aid, personal health record). For example, the following search was performed:

• (health IT OR personal health record OR health application OR personal health tool) AND (accessibility OR literacy OR patient-centered OR easy to use).

Go to Appendix B for a full list of search terms.

The RTI project team reviewed descriptions of each of the products using the inclusion criteria.

Health IT Organizations and Web Sites Resource Review. The project team reviewed the organizations' Web sites listed in Part 1 for examples of products that met the inclusion criteria for Part 3.

Literature Scan. The RTI project team reviewed the 36 articles yielded in the Part 2 literature scan for product examples. Each article was reviewed for products that met the inclusion criteria for Part 3.

2.1.5 Analysis

The results from the environmental scan were analyzed and synthesized to focus on identifying patterns and common themes both within and across resources. Analysis of the environmental scan results involved the organization of findings into an Excel spreadsheet, with the results of each part included on a separate tab. A systematic review of all items that met the inclusion criteria was conducted independently by two team members, with an eye toward identifying themes and trends across resources. The results of the environmental scan were compared to the Guide recommendations to assess areas of ongoing consistency as well as technology-related updates to the recommendations (e.g., exponential increase in mobile device usage). Finally, conclusions and recommendations were developed that flowed logically from the findings.

Return to Contents

2.2 Expert Interviews

2.2.1 Introduction and Purpose

This section summarizes the methods from nine telephone interviews with experts in the areas of health literacy, consumer health IT, usability, and HCI. The purpose of the interviews was to

1. Ensure that all of the best tools and resources published after January 1, 2007 related to developing and purchasing health IT tools and applications accessible to individuals with low health literacy have been captured.
2. Ensure appropriate examples of emerging best practices in developing health IT tools and applications accessible to individuals with low health literacy and relevant gaps in the Health IT Literacy Guide have been identified.
3. Understand participants' perceptions of the types of tools and key issues in health literacy that need to be considered if developers and purchasers of health IT are to improve how they address health literacy and, in turn, ensure that consumer health IT is accessible and understandable regardless of literacy level.

2.2.2 Recruitment

Participants were recruited from a master list compiled by AHRQ and the RTI project team. This compilation drew on existing relationships with groups such as the Health and Human Services Health Literacy Working Group, the Literacy Information and Communication System Health Literacy Discussion List (http://lincs.ed.gov/lincs/discussions/healthliteracy/about_health.html), and patient advocacy groups.

Twenty-one prospective participants were contacted by email. Two attempts were made to contact participants by email, followed by one telephone call from a project staff member. Ten participants agreed to participate. One of these participants was unable to schedule an interview time.

2.2.3 Segmentation Strategy and Affiliations of Participants

Four areas of participant expertise were identified (health literacy, usability, consumer health IT, and HCI), with two or three individuals representing each area for a total of nine interviews. The segmentation strategy for the nine interviews is shown in Table 9.

Table 9. Segmentation and description of participants for expert interviews (n = 9)

Area of Expertise	Description	Number of Interviews
Health literacy	Professionals with expertise in the development of messages, materials, and IT applications for individuals with limited health literacy skills	3
Usability	Professionals with expertise in the evaluation and study of user interfaces, especially those specific to health	2
Consumer health IT	Professionals with expertise in the development and design of consumer health IT tools and applications, such as electronic personal health records and health Web sites	2
Human-computer interaction (HCI)	Professionals with expertise in the design, evaluation, and implementation of interactive computing systems for human use, especially those specific to health	2

 

2.2.4 Data Collection

After recruitment and scheduling, each participant was sent an email invitation, which included a toll-free number to call at the scheduled time, a consent form (Appendix C), and a list of the questions that would be asked during the hour-long interview. Participants were instructed to read the consent prior to the time of the interview. The interviewer began by confirming that participants had read the consent form and answering any questions they had.

The interviewer used a semistructured interview guide (Appendix D) that was developed by the RTI project team and AHRQ. Each interview lasted approximately 60 minutes, and each participant was offered an honorarium to thank them for participation. Two participants, including one who was a Federal government employee, declined the honorarium.

2.2.5 Analysis

Analysis of the expert interview data was used to identify key areas of strengths, weaknesses, and opportunities for improving the Health IT Literacy Guide, including content, usability, and dissemination. The analysis involved (1) a systematic review of all recordings and observer notes to identify, where possible, trends across segments; (2) an examination of findings to detect differences among segments; (3) an examination of findings, topic by topic; and (4) development of conclusions and recommendations that flowed logically from the findings.

A qualitative thematic and grounded theory analysis was used to identify thematic consistencies across participants and groups (Miles and Huberman, 1994). Using this approach, a matrix was created with each interview question in one row and the responses from each participant in subsequent columns to make comparisons across the groups and look for themes and patterns in the findings.

Return to Contents

2.3 Focus Groups

After recruitment and scheduling, each participant was sent an invitation via email, which included a toll-free number to call at the scheduled time, a consent form (Appendix E), and an electronic copy of the Health IT Literacy Guide. Participants were instructed to read the consent form and Guide prior to the time of the focus group. The moderator began by confirming that each participant had read the consent form and answering any questions from the participants.

The moderator used a semistructured interview guide (Appendix F) that was developed by the RTI project team and AHRQ. Each focus group lasted approximately 90 minutes, and each participant was mailed an honorarium afterwards to thank them for their participation. Two participants, including one Federal government employee, declined the honorarium.

2.3.1 Introduction and Purpose

The focus groups were the third step in the overall evaluation and were completed in June 2012. This series of focus groups with developers and purchasers of consumer health IT was intended to determine the following:

1. How developers and purchasers consider/define health literacy in health IT tools;
2. What sources of information they may use to assess health literacy in products or to have conversations about health literacy with vendors/developers they may be working with;
3. What they believe are the key issues or have experienced as issues regarding health literacy when developing or implementing health IT tools for consumers.
4. At what point in the process of developing or purchasing health IT they consider issues such as patient accessibility and health literacy.

2.3.2 Recruitment

Recruitment involved working in collaboration with individuals, Federal purchasers, and professional organizations and societies representing the target audiences (e.g., Association of Medical Directors of Information Systems; Healthcare Information and Management Systems Society [HIMSS], AMIA) to electronically distribute information about the focus group opportunities through their listservs or membership lists.

2.3.3 Segmentation Strategy and Affiliations of Participants

Eligibility criteria for focus group participation included the following:

• Involved in development or selection of consumer health IT tools at their organization;
• Able to participate in a 60- to 90-minute discussion.
• English-speaking.

Because the intended audience for the Health IT Literacy Guide included both developers and purchasers of health IT, RTI created a segmentation strategy for the focus groups, including audience descriptions and participant targets, as shown in Table 10. Within the main segments of developers and purchasers, participants came from many types of companies and organizations, including hospitals, health care facilities, universities, insurers, and health industry associations. More information on the affiliations of the focus group participants is presented in Table 11.

Up to three attempts were made to contact each participant. The first two attempts were made by email, followed by one telephone call from a project staff member. The focus groups were announced on several email lists, which generated hundreds of inquiries from prospective participants. A staff member corresponded with prospective participants and screened them for suitability based on the criteria above.

Table 10. Segmentation strategy for 10 focus groups

Target Audience	Description	Number of Focus Groups	Number of Participants
Developers	Those working for health IT vendors or organizations that create health IT tools providers use to convey information to patients; this may also include researchers, many of whom are also developing or evaluating tools aimed at health care consumers.	1 (in person) 5 (telephone)	30
Purchasers	Those working in organizations involved in the selection of health IT tools or products for use by consumers, often in coordination with health care providers (e.g., hospitals, physician practices, and community health centers); this may also include health plans, pharmaceutical companies, foundations, other nonprofit organizations, and government purchasers at the Federal, State, and local levels.	4 (telephone)	26

Table 11. Affiliations of focus group participants

Organization Type	Purchaser Affiliations (n=26)*	Developer Affiliations (n=30)
Research/academic	8% (n = 2)	27% (n = 8)
Health care provider	58% (n = 15)	13% (n = 4)
Local/State/Federal government	19% (n = 5)	3% (n = 1)
Vendor	12% (n = 3)	57% (n = 17)
Insurer/health plan	12% (n = 3)	—
Association/nonprofit	8% (n = 2)	—

* Note: Some participants matched more than one organization type.

2.3.4 Data Collection

The majority of focus groups took place by telephone. One focus group was held in person at the HIMSS Annual Conference in February 2012 (Las Vegas, Nevada).

For the telephone focus groups, after recruitment and scheduling, each participant was e‑mailed a toll-free number to call at the scheduled time, a consent form (Appendix E), and an electronic copy of the Health IT Literacy Guide. Participants were instructed to read the consent and review the Guide prior to the time of the focus group. The moderator began by asking participants if they had read the consent form and answering any questions from the participants.

The moderator used a semistructured interview guide (Appendix F) that was developed by the RTI project team and AHRQ. Each focus group lasted approximately 90 minutes, and each participant who accepted an honorarium was mailed one afterwards to thank them for their participation.

2.3.5 Analysis

Similar to the analysis for the expert interviews, the focus group analysis involved (1) a systematic review of all recordings and observer notes to identify, where possible, trends across segments; (2) an examination of findings to detect differences among segments; (3) an examination of findings, topic by topic; and (4) development of conclusions and recommendations that flowed logically from the findings.

To analyze the focus groups, a data matrix was created with each focus group question in one row and the responses from each focus group in subsequent columns to make comparisons across the groups and look for themes and patterns in the findings

Return to Contents
Proceed to Next Section