Project Overview:
Project Type:
Public Sector | Custom Software Delivery | Product & Technical Advisory
Project Duration:
April 2022 - November 2022
My Role:
I served as the Senior Experience Designer responsible for leading all research initiatives and creating all the designs and interactive prototypes responsible for driving the team’s production
Agile Design | Concept Generation | Prototyping | Usability Testing | User Research | Visual Communications | Wireframing
Tools Used:
Adobe XD | Mural | Zenhub
Background Overview
The Virtual Agent is an unauthenticated chatbot that provides veterans, caretakers, and veteran beneficiaries with a self-service path to understanding their VA benefits and empowering them with correct information to action on. The chatbot was released in beta in March 2022 to 25% traffic and released to 50% availability in May 2022.
The team's focus was to become the first layer of communication, acting as a file clerk or librarian by ingesting user questions/needs and surfacing relevant va.gov content. Our aim for the following year was to drive trust in the VA as the source of truth by expanding the chatbot content and personalized offerings and creating a centralized robust self-service tool.
The Challenge
According to previously collected analytics, roughly ten thousand people viewed the chatbot page and spent an average of three minutes on the page. There were also 7,000 conversations that were analyzed which indicated that approximately 69% of those users were asking highly specific questions across five themes. The majority of these questions required users to be signed in which the chatbot was unable to assist with in its present state. Additional pre-existing high level findings also revealed other user pain points. One of these indicated that the chatbot’s failure rate was high because users asked highly personalized questions and the chatbot could only answer with generic information from VA.gov. Lastly, the chatbot also could not make attempts to disambiguate the user’s intent, present a user with other associated resources, and connect users with a human just to name a few things.
As a result of these findings the team decided to pursue efforts that would take the chatbot from a general bot to a feature bot, conceptualize language and personality, relocate the chatbot to the lower right-hand corner of the page where users anticipated it, and prioritize content to surface from one of the derived themes.
User Problem:
Veterans and their families want the VA.gov chatbot to provide them with specific and personalized answers befitting of their situations. However, this can only be accomplished when users are logged in to their personal accounts which the chatbot could not assist them with. Due to this, the website's site traffic and overall perceived usefulness were impacted.
Design Question:
How might we further enhance the VA.gov Chatbot to be able to assist Veterans and their families with personalized information and general sign in assistance?
_________________________________________________________________
Approaching the Solution
Given the overwhelming desire for the Chatbot to address sign in capabilities as seen in the analytics, the team conducted an internal discovery to identify which areas of the sign in process to focus on. Simultaneously, the team decided to conduct a round of usability testing. This round of research concentrated on gathering feedback on the current state of the Virtual Chatbot, testing potential flows for resetting a password and general sign in with users, and prioritizing future Chatbot features.
I served as Lead Researcher and contributed in the following ways:
1. Research & Insight Gathering
+ Designed research assets (i.e. journey maps, mind maps, and market research) to encourage team ideation and assist in the discovery
+ Conducted 6 remote user interviews with Veterans to learn of their pain points, current feelings toward the Chatbot, and gage their priorities for sign in assistance
+ Developed research documentation (i.e. conversation guide, research guide, low-fidelity wireframes)
+ Synthesized research findings into actionable items that the team could implement
From the research insights we learned that the clinical features category resonated most with users. This category would focus on the chatbot’s ability to be able to set appointments, handle prescriptions, and locate nearby health physicians. The results also validated the previous research insights that highlighted that participants wanted to utilize the Chatbot for specific and personal tasks. Finally, we learned that the proposed password reset and general sign in paths were acceptable with participants.
Research assets that were utilized to arrive at our initial design concepts
Prioritizing the Right Thing
2. Concept Development & Prototyping
After our initial round of research our team conducted a discovery with our internal team and stakeholders. Five possible tasks were identified that the team felt would align with the Chatbot being able to assist with more personalized tasks and build upon our previous research. These tasks would address general sign in assistance, problem escalation, account education, resetting a password, and account creation. These concepts would also be the focal point of our next round of research.
Regarding the ideation and development of these concepts, I assisted the progression of the project in the following ways:
+ Creating low-fidelity wireframes for 5 design concepts/features that demonstrated potential workflows for each task
+ Utilizing the VA design system to streamline the design process while increasing design outputs during sprints
+ Collaborating with the program manager, project manager, developers, and stakeholders to gather internal feedback, promote clarity in our design direction, and and to quickly reiterate on the designs
3. Prototyping & Storytelling
After gaining consensus on our wireframes I created clickable prototypes that we could use for usability testing. Unlike our previous round of research, this round of testing would require a larger testing pool and proved unique in the way I had to ensure each of the five flows were properly tested. In order to adequately test each of the five tasks, I presented participants with a menu of options that would trigger various paths based upon their initial selection. The menu would only show after participants experienced the initial Chatbot conversation. I felt doing this would be best to address the linearity of the chatbot’s conversation. After gaining buy in from the team, I created interactive prototypes in Adobe XD that showed the refined flows at higher fidelity.
As I acted as both the designer and researcher for this round of testing, I contributed in the following ways with our prototypes:
+ Creating high-fidelity, clickable prototypes for the 5 design concepts/features that demonstrated potential workflows for each task
+ Conducting 10 unmoderated interviews with Veterans to simulate real world scenarios within the Chatbot with each of our scenarios
+ Partnering with developers to ensure the integration of the VA design system promoting familiarity with the Chatbot
We learned several insights from the round of usability testing. For example, we found that participants valued the problem escalation flow the most followed by password reset. Users also desired for the Chatbot to do more advanced tasks like prefill forms, retain password information, and retain previous conversations within the Chatbot. The participants also appreciated the presence of an “I’m not sure” option in the task menu. Finally, the majority of users found that the task menu offered good variety despite the vagueness of some of the titles. These insights would go on to inform our next set of design decisions for the chatbot.
Screenshots showing design intent from low to high fidelity
The Results
Ultimately the findings from both rounds of research led to several successes. The team proceeded to develop and deliver pieces of the proposed flows that were tested with more participants. Some notable takeaways that were incorporated were the task menu and a “Do I need an account?” button that was a deviation of the “I’m not sure” button shown in the testing. The Chatbot is currently active on VA.gov and utilized by Veterans everyday.
Screenshots of the VA Chatbot at present day