This paper is in the following e-collections/theme issues:
Evaluating User Experiences of the Secure Messaging Tool on the Veterans Affairs’ Patient Portal System
Jolie N Haun1,2, EdS, PhD; Jason D Lind1,3, MPH, PhD; Stephanie L Shimada4,5,6, PhD; Tracey L Martin7, RN, MSN; Robert M Gosline8; Nicole Antinori1, MBA; Max Stewart9; Steven R Simon9, MD, MPH
1Department of Veterans Affairs, HSR&D/RR&D Center of Innovation on Disability and Rehabilitation Research, James A Haley VA Medical Center, Tampa, FL, United States
2Department of Community & Family Health, Unversity of South Florida, College of Public Health, Tampa, FL, United States
3Department of Anthropology, University of South Florida, Tampa, FL, United States
4Department of Veterans Affairs, Center for Healthcare Organization and Implementation Research (CHOIR) and National eHealth Quality Enhancement Research Initiative (QUERI) Coordinating Center, Edith Nourse Rogers Memorial VA Hospital, Bedford, MA, United States
5Department of Health Policy and Management, Boston University School of Public Health, Boston, MA, United States
6Division of Health Informatics and Implementation Science, Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA, United States
7Department of Veterans Affairs, VA New England Health Care System, Bedford, MA, United States
8Department of Veterans Affairs, James A Haley VA Medical Center, Tampa, FL, United States
9Department of Veterans Affairs, Center for Healthcare Organization and Implementation Research, Section of General Internal Medicine, VA Boston Healthcare System, Boston, MA, United States
Department of Veterans Affairs
HSR&D/RR&D Center of Innovation on Disability and Rehabilitation Research
James A Haley VA Medical Center
8900 Grand Oak Cir (151R)
Tampa, FL, 33637-1022
Phone: 1 813 558 7622
Fax: 1 813 558 7616
Background: The United States Department of Veterans Affairs has implemented an electronic asynchronous “Secure Messaging” tool within a Web-based patient portal (ie, My HealtheVet) to support patient-provider communication. This electronic resource promotes continuous and coordinated patient-centered care, but to date little research has evaluated patients’ experiences and preferences for using Secure Messaging.
Objective: The objectives of this mixed-methods study were to (1) characterize veterans’ experiences using Secure Messaging in the My HealtheVet portal over a 3-month period, including system usability, (2) identify barriers to and facilitators of use, and (3) describe strategies to support veterans’ use of Secure Messaging.
Methods: We recruited 33 veterans who had access to and had previously used the portal’s Secure Messaging tool. We used a combination of in-depth interviews, face-to-face user-testing, review of transmitted secure messages between veterans and staff, and telephone interviews three months following initial contact. We assessed participants’ computer and health literacy during initial and follow-up interviews. We used a content-analysis approach to identify dominant themes in the qualitative data. We compared inferences from each of the data sources (interviews, user-testing, and message review) to identify convergent and divergent data trends.
Results: The majority of veterans (27/33, 82%) reported being satisfied with Secure Messaging at initial interview; satisfaction ratings increased to 97% (31/32, 1 missing) during follow-up interviews. Veterans noted Secure Messaging to be useful for communicating with their primary care team to manage health care needs (eg, health-related questions, test requests and results, medication refills and questions, managing appointments). Four domains emerged from interviews: (1) perceived benefits of using Secure Messaging, (2) barriers to using Secure Messaging, (3) facilitators for using Secure Messaging, and (4) suggestions for improving Secure Messaging. Veterans identified and demonstrated impediments to successful system usage that can be addressed with education, skill building, and system modifications. Analysis of secure message content data provided insights to reasons for use that were not disclosed by participants during interviews, specifically sensitive health topics such as erectile dysfunction and sexually transmitted disease inquiries.
Conclusions: Veterans perceive Secure Messaging in the My HealtheVet patient portal as a useful tool for communicating with health care teams. However, to maximize sustained utilization of Secure Messaging, marketing, education, skill building, and system modifications are needed. Data from this study can inform a large-scale quantitative assessment of Secure Messaging users’ experiences in a representative sample to validate qualitative findings.
(J Med Internet Res 2014;16(3):e75)
veterans; secure messaging; patient-provider communication; Department of Veterans Affairs; usability testing; mixed methods; patient-centered care
The Institute of Medicine (IOM) has identified patient-provider communication as a central component to improving quality of care and patient outcomes . My HealtheVet is the Department of Veterans Affairs’ (VA) online patient portal and personal health record designed for veterans, active duty service members, and their dependents and caregivers. My HealtheVet provides veterans with tools (eg, Blue Button, VA immunization records, laboratory test results, prescription refills, VA appointments) to make informed decisions and manage their health care. “Secure Messaging” is an email-like electronic resource within My HealtheVet designed to promote continuity of patient-provider communication [2-4]. As VA further implements Patient Aligned Care Teams (PACT) as a model of the patient-centered medical home, secure messaging is emerging as a key mechanism of communication between veterans and their health care team members. Successful implementation of secure messaging is therefore a priority not only for VA but also for other health care systems in the United States that strive to adopt principles of the patient-centered medical home. Moreover, outside VA, providers are being incentivized via Stage 2 Meaningful Use requirements (Medicare Electronic Health Records (EHR) Incentive Program) to use secure messaging among at least 5% of their patients to communicate relevant health information .
Previous work has demonstrated the utility and value of providing patients access to their electronic health record [6-8]. Patients also value secure messaging to communicate electronically with their providers [2-4]. Effective use of secure messaging can improve patient self-care management, patient engagement, and utilization of health services. In addition to allowing an option for self-care management, this electronic tool holds potential for supporting clinical tasks including medication reconciliation . Secure messaging supports system utilization benefits in addition to perceived benefits by patient and clinical team users. A recent study showed a 7-10% decrease in outpatient visits and a 14% reduction in telephone contacts as a result of secure messaging [10,11]. Houston et al reported that 95% of respondents felt email was a more efficient means of communication with their physicians than the telephone, and 77% noted being able to communicate adequately via email without a face-to-face appointment . Patient use of secure messaging has been associated with improved outcomes for chronic conditions [10,12]. Zhou et al reported in a recent study that within a two-month period there were improvements in care as measured by the Healthcare Effectiveness Data and Information Set (HEDIS) . Patients with diabetes using secure messaging improved on all measures recommended for testing and control of glucose, cholesterol, and blood pressure levels by an average of 2.4-6.5% compared with patients not using secure messaging. In the same study, rates of received health services improved in the secure messaging group compared to the control group . These findings suggest that successful implementation of secure messaging may provide a viable cost-efficient means of patient-provider communication.
Implementing health information technology, such as secure messaging, requires systematic inquiry grounded in implementation science to identify barriers to and facilitators of user adoption and utilization. The Technology Acceptance Model (TAM)  and the Theory of Planned Behavior (TPB)  have been found to be useful in predicting adoption of technology. While secure messaging has been shown to promote continuous and coordinated patient-centered care, little research has evaluated patients’ experiences with and preferences for using secure messaging. In order to maximize sustained utilization of secure messaging, marketing, education, skill building, and minor system modifications may be needed. Evaluation of secure messaging users’ experiences using the TAM and TPB frameworks can increase our understanding of issues related to access, continuity, and coordination of care for veterans that will support adoption and long-term utilization of Secure Messaging in My HealtheVet.
Findings from the Secure Messaging evaluation research will inform efforts to transform care delivery both within and beyond the VA system. Thus, the aims of this study were to (1) characterize veterans’ beliefs, attitudes, and perceptions toward using the Secure Messaging tool, (2) describe the patterns of veterans’ use of Secure Messaging, (3) identify the barriers to and facilitators of using Secure Messaging, and (4) describe strategies for promoting facilitators and overcoming barriers to using Secure Messaging.
This prospective descriptive qualitative study used mixed-methods to describe veterans’ experiences using Secure Messaging in the My HealtheVet portal. As an implementation study, the underlying objective was to understand veterans’ needs to promote increased access to and sustained utilization of the Secure Messaging tool. A combination of in-depth interviews, user-testing, a 3-month review of transmitted secure messages between veterans and staff, and 3-month follow-up phone interviews was used to characterize veteran Secure Messaging utilization. Demographic data as well as computer and health literacy measurements were collected through survey and in-depth interviews at baseline and 3-month follow-up.
Setting and Participants
The two-site study was conducted at two large VA Medical Centers (VAMCs): the James A. Haley Veterans’ Hospital (Tampa, Florida) and the Veterans Affairs Boston Healthcare System (Boston, Massachusetts). We used administrative data to identify veterans at both VAMCs who had registered for My HealtheVet, completed the in-person process of authenticating their identity, and accessed the system to “opt-in” to use Secure Messaging. This approach identified 3926 potential participants at Tampa and 924 at Boston. Next, randomization was used to create contact lists of 120 potential participants from each site list. All 240 potential participants were contacted and screened to be purposively sampled based on their self-reported previous use of Secure Messaging. Participants were recruited for study participation until domain and theme saturation was reached.
Inclusion criteria included veterans who were independent Secure Messaging users, without cognitive impairment that prevented use of a personal computer or the ability to provide informed consent. Based on qualitative sampling methods [15,16], saturation was anticipated to occur between 12 to 15 interviews; an over-recruitment strategy was used at each site to allow for attrition, resulting in 33 total participants. One participant was lost to follow-up for unknown reasons, resulting in a complete dataset of 32 participants. Veterans received up to US$50 for their participation: US$20 for participation in the initial interview and user-testing and an additional US$30 for allowing the researchers unrestricted access to review the content of their secure messages and participation in the 3-month follow-up telephone interview. Participants provided informed consent upon their arrival for the initial face-to-face interview and user-testing. This study was approved and regulated by the VA Central Institutional Review Board.
Data Collection Instruments
Data were collected using demographic and health literacy surveys, in-depth face-to-face interviews, Secure Messaging usability testing, prospective collection of the content of secure messages, and 3-month follow-up telephone interviews. All data, with exception of the Secure Messaging data, were collected at two time points: during a baseline in-person meeting and during a 3-month follow-up phone interview. Prospective Secure Messaging data were collected between the baseline and 3-month follow-up time points.
Participant Surveys and Assessments
During the initial research visit, veterans completed a 13-item demographic survey to ascertain age, gender, race/ethnicity, education level, income level, marital status, computer use, Internet use, My HealtheVet use, and Secure Messaging use. Health literacy was assessed using two validated instruments: (1) the Brief Health Literacy Screening Tool (BRIEF), and (2) the Rapid Estimate of Adult Literacy in Medicine (REALM) survey. The BRIEF is a 4-item self-report screening tool to assess health literacy skills . The REALM assesses health literacy by having respondents verbally articulate three columns of 22 health-related terms .
Electronic health literacy was also assessed using two instruments: (1) the eHealth Literacy Scale (eHEALS), and (2) the Computer-Email-Web (CEW) Fluency Scale. The eHEALS is a 10-item measure of eHealth literacy developed to measure consumers’ knowledge, comfort, and perceived skills at finding, evaluating, and applying electronic health information to health problems . The CEW Fluency Scale is a 21-item measure of common computer skills .
Face-to-face semi-structured interviews with participants were conducted by an experienced interviewer trained in the social sciences. Interviews focused on participants’ experiences using Secure Messaging. The interview guide was created following the Theory of Planned Behavior (TPB) framework to elicit beliefs and attitudes, subjective norms, perceived behavioral control, and behavioral intention toward Secure Messaging use. Other interview questions were developed based on the Technology Acceptance Model (TAM) and addressed usefulness and ease of use of Secure Messaging. Interviews followed the guide but were open-ended in nature, allowing the interviewer flexibility to ask probing questions and to follow up on interesting topics and user experiences related to Secure Messaging.
Based on the initial interviews, a brief phone interview guide was developed to address Secure Messaging use during the 3-month period after the first interview. These interviews were conducted to assess recent Secure Messaging use: usefulness, expectations, barriers and facilitators, satisfaction, and suggestions for improvement.
Secure Messaging User-Testing
In-person Secure Messaging user-testing was conducted to prompt participants to complete a series of tasks they would normally encounter while using Secure Messaging. User tasks included navigating to the My HealtheVet site, logging in to Secure Messaging, setting user preferences, checking the Inbox, opening a secure message, opening and reading an attachment, and sending a secure message. Task completion, obstacles, and facilitators were recorded using a checklist, which directly corresponded to the user-testing tasks. Usability testing with each participant was conducted using Morae software [21,22], and allowed for the live, remote observation and video-recording of the user being tested (eg, recording of clicks, keystrokes, and other events) . Participants were asked to “think aloud” and vocalize their thoughts, experiences, feelings, and opinions while interacting with the program as they used the Secure Messaging feature [24,25].
Secure Messaging Content
Secure messages were collected, both outgoing and incoming secure messages were collected for each participant over a 3-month period following their provision of informed consent. Data included sender and recipient identification, date and time of delivery, subject header, category of message subject (eg, test, appointment, medication, general), and verbatim content of the secure message text. We examined the quantity of messages, message content, exchange patterns, and timing of inbound and outbound messages between participants and their health care teams. This approach allowed for analysis of authentic user content and patterns to further inform research findings.
Data Management and Analysis
All data, including interviews and paper-based surveys gathered in this study were stored on a secure VA network. Audio recordings of all interviews were transcribed and subsequently analyzed using ATLAS.ti , qualitative data analysis software. Descriptive statistics from veteran surveys were managed using the statistical software suite SPSS version 21 (SPSS IBM, New York, USA). Data from Secure Messaging usability testing were captured using Morae recording software.
We used content analysis methods to analyze all interview data to identify domains and taxonomies related to participants’ experiences using Secure Messaging . We used the semi-structured interview guide to organize and code interview text to develop thematic categories. Categories were grouped into taxonomic relationships and then compared and contrasted across coded categories. Coding schemas were developed by two research team members to create domains and taxonomies and evaluated for inter-rater reliability and credibility. Data were then categorized and interpreted, and barriers and facilitators were identified. Quantitative data were summarized with descriptive statistics to describe sample characteristics. Frequency counts and proportions provided a descriptive overview of the user-testing findings.
A total of 33 participants were recruited, of whom 32 provided complete data. One participant provided initial interview, user-testing, and secure message content data, but could not be reached for the follow-up phone interview.
Survey and Assessment Findings
The majority of participants were older white males (26/33, 79%) and ranged in age from 27 to 77 years, mean age 59.5 (SD 11.9). All participants had at least a high school education, and 64% (21/33) had an annual income of US$35,001 or more. Demographic characteristics are reported in Table 1.
Though skills varied, the majority of participants had adequate health literacy and eHealth competency skills. Study participants had higher levels of health literacy than the general veteran population . Though comparative studies are not available for this population using these tools, the electronic health literacy scores on the eHEALS and the CEW produced similar findings. Instrument range, sample range, mean, and SDs are illustrated in Table 2.
At baseline, all participants (n=33, 100%) reported using a computer and the Internet more than once a week. Most participants (22/33, 67%) reported using Secure Messaging for at least the past six months (10/33, 30%) or longer (12/33, 36%), while the remaining participants reported using Secure Messaging for less than six months (11/33, 33%). The majority of participants (28/33, 85%) reported using Secure Messaging “at least once a month” (12/33, 36%) or “a few times a year” (16/33, 49%). Most veterans (27/33, 82%) reported being satisfied with Secure Messaging.