Project Overview:
Project Type:
Public Sector | Custom Software Delivery | Product & Technical Advisory
Project Duration:
April 2022 - November 2022
My Role:
I served as the Senior Experience Designer responsible for leading all research initiatives and creating all the designs and interactive prototypes responsible for driving the team’s production
Agile Design | Concept Generation | Prototyping | Usability Testing | User Research | Visual Communications | Wireframing
Tools Used:
Adobe XD | Mural | Zenhub
Background Overview
The Virtual Agent is an unauthenticated chatbot that provides veterans, caretakers, and veteran beneficiaries with a self-service path to understanding their VA benefits and empowering them with correct information to action on. The chatbot was released in beta in March 2022 to 25% traffic and released to 50% availability in May 2022.
The team's focus was to become the first layer of communication, acting as a file clerk or librarian by ingesting user questions/needs and surfacing relevant va.gov content. Our aim for the following year was to drive trust in the VA as the source of truth by expanding the chatbot content and personalized offerings and creating a centralized robust self-service tool.
The Challenge
According to previously collected analytics, roughly ten thousand people viewed the chatbot page and spent an average of three minutes on the page. There were also 7,000 conversations that were analyzed which indicated that approximately 69% of those users were asking highly specific questions across five themes. The majority of these questions required users to be signed in which the chatbot was unable to assist with in its present state. Additional pre-existing high level findings also revealed other user pain points. One of these indicated that the chatbot’s failure rate was high because users asked highly personalized questions and the chatbot could only answer with generic information from VA.gov. Lastly, the chatbot also could not make attempts to disambiguate the user’s intent, present a user with other associated resources, and connect users with a human just to name a few things.
As a result of these findings the team decided to pursue efforts that would take the chatbot from a general bot to a feature bot, conceptualize language and personality, relocate the chatbot to the lower right-hand corner of the page where users anticipated it, and prioritize content to surface from one of the derived themes.
Approaching the Solution
The team decided to conduct two rounds of usability testing to determine what path we ought to pursue. The first round of research concentrated on gathering feedback on the current state of the Virtual Chatbot, testing potential flows for resetting a password and general sign in with users, and prioritizing future Chatbot features. As the Lead Researcher on the account, I conducted 6 remote user interviews with Veterans. I created a research guide and a conversation guide that explained the extent of the research, outlined the criteria for potential users, and listed in detail what I would cover. I also created low fidelity wireframes showing the current and proposed flows for users to provide their feedback. Finally after the tests were completed, I synthesized all of our findings and shared back the insights with the team.
From the research insights we learned that the clinical features category resonated most with users. This category would focus on the chatbot’s ability to be able to set appointments, handle prescriptions, and locate nearby health physicians. The results also validated the previous research insights that highlighted that participants wanted to utilize the Chatbot for specific and personal tasks. Finally, we learned that the proposed password reset and general sign in paths were acceptable with participants.
Low fidelity wireframe examples
Test screens that were utilized during the usability sessions
A screenshot reflecting some of the research synthesis process
Prioritizing the Right Thing
After our initial round of research our team conducted a discovery with our internal team and stakeholders. This discovery would determine what exact task we wanted to pursue next for the Chatbot. Five possible tasks were identified that the team felt would align with the Chatbot being able to assist with more personalized tasks and build upon our previous research. These tasks would address general sign in assistance, problem escalation, account education, resetting a password, and account creation. Our next round of research would focus on testing the selected tasks and prioritizing them with participants.
For this round of research, I acted as both designer and researcher. I used Mural to create low fidelity wireframes that illustrated the potential flows and chatbot conversations with participants. In order to adequately test each of the five tasks, I presented participants with a menu of options that would trigger various paths based upon their initial selection. The menu would only show after participants experienced the initial Chatbot conversation. I felt doing this would be best to address the linearity of the chatbot’s conversation. After gaining buy in from the team, I created interactive prototypes in Adobe XD that showed the refined flows at higher fidelity.
Following the completion of the prototypes, I conducted 10 moderated interviews where we were able to test each of the flows with participants. We learned several insights from the round of usability testing. For example, we found that participants valued the problem escalation flow the most with the password reset task following in importance. Users also desired for the Chatbot to do more advanced tasks like prefill forms, retain password information, and retain previous conversations within the Chatbot. The participants also appreciated the presence of an “I’m not sure” option in the task menu. Finally, the majority of users found that the task menu offered good variety despite the vagueness of some of the titles. These insights would go on to inform our next set of design decisions for the chatbot.
Sample screenshots that were tested with users
The Results
Ultimately the findings from both rounds of research led to several successes. The team proceeded to develop and deliver pieces of the proposed flows that were tested with more participants. Some notable takeaways that were incorporated were the task menu and a “Do I need an account?” button that was a deviation of the “I’m not sure” button shown in the testing. The Chatbot is currently active on VA.gov and utilized by Veterans everyday.
Screenshots of the VA Chatbot at present day