Consumers' Appraisal of Anecdotal Accounts of Patient Experience

Select to access the PowerPoint® presentation (2 MB).

Slide 1

Consumers' Appraisal of Anecdotal Accounts of Patient Experience

AHRQ Annual Conference
Steven C. Martino, PhD
September 10, 2012

Image: The RAND HEALTH logo is shown.

Slide 2

Customer Reviews of Products and Services are Omnipresent Online

• Amazon.com first began allowing customers to post reviews of products in 1995.
• Now almost all retail Web sites include them 78% of internet users (58% of adults) have researched a product or service online, and 32% have posted a comment or review (Pew Internet & American Life Survey, 2011).
• Reviews are a trusted source of information on product or service quality.
• Current retail industry focus is on increasing the relevance of online customer review.

Slide 3

Image: A screenshot of the Tripadvisor Web site shows information and ratings for a restaurant called Casbah.

Slide 4

Image: A screenshot of the Tripadvisor Web site shows a bad review of the Casbah.

Slide 5

Image: A screenshot of the Tripadvisor Web site shows a good review of the Casbah.

Slide 6

Narrative Accounts of Patient Experience are Increasingly Available Online

• Dozens of commercial Web sites provide narrative accounts of patients' experiences with their doctors (sometimes accompanied by ratings):
  • e.g., Vitals.com, HealthGrades, Yelp, Angie's List, and Consumer Reports.
• RateMDs currently hosts narrative reviews of at least 1 in 6 practicing doctors in the U.S. (Gao et al., 2012).
• Currently, the only equivalent in the public sector is NHS Choices (British National Health Service).

Slide 7

Currently, the Audience for these Sites is Small

• 80% of internet users (59% of adults) have looked online for health information.
• 16% of internet users have consulted online rankings or reviews of doctors or other providers (4% have posted such reviews).
• 15% have consulted rankings or reviews of hospitals or other medical facilities (3% have posted such reviews).

Slide 8

Image: A screenshot of the RateMDs.com Web site shows information and ratings for a sample doctor.

Slide 9

Image: A screenshot of the RateMDs.com Web site shows reviews for the sample doctor.

Slide 10

Promises and Limitations of Patient Narratives

• Patient narratives often cover the same domains as standardized surveys of patient experience but in ways that can be more vivid, concrete, and engaging.
• Yet, the sample of respondents is often not representative (biased) and small (poor signal, low reliability):
  • On RateMDs, the average number of reviews per physician is three (Lagu et al., 2010).
• If presented with standardized data, patient narratives could make unfamiliar, difficult-to-understand quality measures more salient and evaluable.
• Yet, anecdotal data may overwhelm more formal statistical information.

Slide 11

Extracting Meaning from Narratives is a Complex Cognitive Task

Image: Three boxes are shown, respectively containing the following text:

Enduring Characteristics of the Doctor:

• Technical ability.
• Interpersonal skill.

Enduring Characteristics of the Environment:

• Health system guidelines.
• Case load and mix.

Unique Aspects of the Situation:

• Individual patient.
• Circumstantial context.

Arrows point from these three text boxes to a fourth box, which contains the text "Doctor's Behavior."

Slide 12

Extracting Meaning from Narratives is a Complex Cognitive Task

Image: The text boxes in Slide 11 are repeated, but the first box and "Doctor's Behavior" are highlighted while the other two boxes are grayed out. A text box in green reads "Consumers try to infer this (arrow points to 'Enduring Characteristics of the Doctor') from this (arrow points to 'Doctor's Behavior')."

Slide 13

Cognitive Strategies that People Use to Discern Among Multiple Possible Causes of Behavior...

• Typically, people look for:
  • Consistency in behavior across time and situations: Has the doctor behaved a certain way repeatedly across time and situations?
  • Consensus in people's experiences with the person: Do other patients report getting similar treatment from the doctor?
  • Distinctiveness of behavior: Is this doctor the only doctor to have behaved this way toward the patient?

Slide 14

. . . Are Not Necessarily Applicable in the Case of Patient Narratives

• Consumers often do not have the information needed to answer questions about consistency, consensus, and distinctiveness:
  • Usually limited to a small number of narratives.
  • Do not get to "observe" the doctor's behavior over multiple occasions with the same patient or get the patient's view of other doctors.
  • Narratives rarely mention situational factors that may have facilitated or constrained behavior; instead, they strongly imply dispositional causes.

Slide 15

Consumers' Values, Disposition, and Past Experiences May Affect Their Appraisal

• Consumers differ in how much they value data on patient experience and the importance they ascribe to various facets of patient experience and doctor quality.
• Consumers with different decisionmaking styles may use narrative accounts differently.
• Consumers have different levels of experience with healthcare quality data.

Slide 16

Overview of Research Questions

• What cues do people use to judge the value and authenticity of patient narratives?
• How salient (memorable) are patient narratives relative to standardized measures of healthcare quality?
• How trustworthy and useful do consumers perceive narratives to be?
• What are bases for skepticism?
• Do people's preferred decisionmaking styles, health status, or past exposure to healthcare quality data affect their attention to and appraisal of patient narratives?

Slide 17

The SelectMD Web site: Basic Design

• A fictitious public reporting Web site custom built for this and related studies.
• Designed to be consistent with current public reports in terms of content, format, and functionality.
• Presents three types of data:
  • Standardized patient experience measures (CAHPS®).
  • Clinical process and outcome measures (HEDIS).
  • Patient narratives (reviews).
• Hidden tracking system to monitor:
  • Click patterns.
  • Time spent on each page.

Slide 18

The SelectMD Web site: Creating Realistic Patient Narratives

• Modeled on actual narratives collected from RateMDs.
• Matched to actual narratives based on length, cognitive complexity, and clarity of writing.
• Pilot testing to compare actual (RateMD) narratives to a large set of fictitious ones constructed for this study.
• Matched to reported CAHPS scores:
  • Focused on areas of experience covered by CAHPS.
  • Higher scores associated with more positive comments (but imperfectly).
  • Comments varied in emotional tone for all doctors.

Slide 19

Image: A screenshot of the Select MD Web site shows a page which allows users to search for a doctor by type of doctor and proximity to the patient's location.

Slide 20

Image: A screenshot of the Select MD Web site shows the results of a search, with sample doctors.

Slide 21

Image: A screenshot of the Select MD Web site shows the sample doctors rated according to Service Quality.

Slide 22

Image: A screenshot of the Select MD Web site shows the sample doctors rated according Treatment Quality.

Slide 23

Image: A screenshot of the Select MD Web site shows patient reviews of a sample doctor.

Slide 24

Study Participants

Pretesting of Patient Narratives:

• Three rounds of testing, (n = 8-9 per round.
• Convenience sample of internet users with recent healthcare experience, mix of demographic characteristics.

Online Experiment:

• Random sample drawn from Knowledge Networks online research panel:
  • Representative national sample (>60,000 households).
  • Limited to households with computer-based internet access (89% of total panel), aged 25-64 years.
  • Of 1,757 invitations, over 48% accepted (n = 849).

Slide 25

Experimental Design: Six Study Arms

Slide 26

Experimental Design: Six Study Arms

Slide 27

Study Measures

Questions asked of pretest participants:

• Emotional valence of narratives.
• Perceived informativeness & authenticity of narratives.

Questions asked prior to MD Choice:

• Prior exposure to public reports on plans, hospitals, and doctors.
• Health status.

Questions asked after MD Choice:

• Recall of measures.
• Perceived usefulness and trustworthiness of measures.
• Decisionmaking styles.

Slide 28

Qualities of Narratives Associated with Their Perceived Usefulness and Authenticity

• Strongly worded narratives were perceived as less authentic (p = .001) but more informative (p <.001) than mildly worded narratives.
• Narratives about negative experiences were perceived as less authentic (p <.001) than narratives about positive experiences.
• Length of narratives was positively related to perceived authenticity, r (129) = 0.25, p = .005, and positively related to perceived informativeness, r (129) = 0.30, p = .001.

Slide 29

Recall of Patient Narratives

• Of Ss who were shown patient narratives on the SelectMD site, 86% recalled seeing them.
• Of Ss shown narratives plus CAHPS measures:
  • 58% recalled both.
  • 32% recalled narratives only.
  • 4% recalled CAHPS measures only.
  • 7% recalled neither.
• Of Ss shown narratives plus HEDIS measures:
  • 29% recalled both.
  • 53% recalled narratives only.
  • 3% recalled HEDIS measures only.
  • 15% recalled neither.

Slide 30

Recall of Patient Narratives (cont.)

• Ss low in avoidant decisionmaking were more likely to recall patient narratives than Ss high in avoidant decisionmaking (89% vs. 84%; p = .07).
• Ss high in intuitive decisionmaking were more likely to recall patient narratives than Ss low in intuitive decisionmaking (88% vs. 83%; p = .10).
• Ss high in intuitive decisionmaking and low on rational decisionmaking were especially likely (94%) to recall patient narratives.

Slide 31

Recall of Patient Narratives (cont.)

• 81% of Ss who recalled the narrative data provided open-ended data on what they remembered specifically.

Slide 32

Perceived Usefulness of Patient Narratives

• Among Ss who recalled seeing the narratives, 44% said that they were very useful for decisionmaking, 44% said somewhat useful, 9% said only a little useful, and 3% said not at all useful.
• Unexpectedly, Ss high in intuitive decisionmaking judged the narratives as less useful than did Ss low in intuitive decisionmaking (p = .06).
• Ss with recent serious or chronic health conditions saw the narratives as less useful than Ss without those conditions (p = .01).
• Past exposure to healthcare quality data was not related to perceived usefulness of the narratives (p = .34).

Slide 33

Trustworthiness of Patient Narratives

• Ss perceptions of the usefulness and trustworthiness of narratives were strongly related, r = 0.59, N = 588, p <.001.
• Among Ss who recalled seeing the narratives, 18% said that they were very trustworthy, 65% said somewhat trustworthy, 15% said only a little trustworthy, and 2% said not at all trustworthy.
• Decisionmaking style and prior exposure to healthcare quality data were unrelated to how trustworthy narratives were perceived to be.
• Ss with recent serious or chronic health conditions trusted narratives less than Ss without those conditions (p = .04).

Slide 34

Reasons Given for Mistrust of Patient Narratives

^*Given by 101 subjects who judged narratives to be "only a little" or "not at all" trustworthy.

Slide 35

Attention Given to Patient Narratives

• Overall, Ss (N = 593) read reviews of 17.1% (SD = 24.4) doctors (includes those who did not drill down to reviews).
• Ss who drilled down to the reviews (N = 157) read the reviews of 30.2% (SD = 25.6) of doctors.
• Ss high in avoidant decisionmaking read reviews of a smaller percentage (11.2%) of doctors than Ss low in avoidant decisionmaking (14.9%; p = 0.10).

Slide 36

Attention Given to Patient Narratives (cont.)

• Ss low in analytic decisionmaking and low in intuitive decisionmaking read reviews of a smaller percentage (11.7%) of doctors than Ss with other combinations of these two styles (who read the reviews of 17-20% of doctors; F = 3.2, p = .02).
• Ss with prior exposure to healthcare quality information read reviews of a larger percentage of (20.3%) of doctors than Ss without prior exposure (15.1%; p = .01).
• Ss with recent serious or chronic health conditions read reviews of a larger percentage of doctors (19%) than did Ss without such conditions (16%; p =.09).

Slide 37

Conclusions

• People use cues in narratives to determine their value and relevance.
• When presented together, patient narratives appear to be more memorable (salient) than standardized data on quality.
• Most people trust patient narratives and find them useful for decisionmaking.
• A minority recognizes the limitation of patient narratives as a source of data on data on doctor quality.
• Positive and negative accounts of patient experience are recalled equally; no domain of patient experience is more salient than others.
• Attention to and evaluation of patient narratives depends on people's decisionmaking tendencies, health status, and prior experience with healthcare quality data.

Slide 38

Implications

• It may be possible to increase healthcare consumers' engagement in and use of reports of standardized quality data by incorporating patient narratives.
• Need to elicit patient narratives in a way that makes them most useful to consumers (requires attention to issues of reliability and validity).
• Need to present narratives in a way that clarifies their value (as illustrative rather than representative), relevance (by building trust in the methods of elicitation and reporting) and limitations.