Appendix 1:
User Research Checklist
Before any design or implementation, work with potential end-users and domain experts to reduce uncertainty in design. Be prepared to provide the answers to these questions to the team members who will make design decisions for the chatbot.
| ID | Recommendation | Research Task | Check |
|---|---|---|---|
| 1 | 4.1.1 Recognize platform needs | Ask users about their past experiences completing the tasks that the chatbot is intended to support. What accessibility barriers have they encountered? Build a list of features or functions that need to be accessible. | |
| 2 | 4.1.2 Accessibility certifications | Ask development teams of other chatbots in the same domain (e.g., government services) what chatbot platform they used and what accessibility testing they have performed. | |
| 3 | 4.1.3 Choose an accessible platform | Ask users about accessible chatbots they have interacted with, especially in the same domain (e.g., government services). Try to contact those developers and ask about the platform and tools they used. | |
| 4 | 4.2.3 Use emotional intelligence | Learn about the user’s emotional state while completing this task. Is the task content innately distressing, like death or debt? Observe people completing the task with human representatives. Do the representatives exhibit emotional intelligence when working with users? Ask the representatives about how they recognize emotional distress and how they change the task process or communication style in response to detected distress. | |
| 5 | 4.2.5 Use simple messages | Determine the language level of your users. Does the intended demographic skew young or old, have a relatively high or low educational attainment, or tend to speak English as a second language? Review the task language with users. Which words are most commonly not understood? | |
| 6 | 4.2.7 Build simple conversations | How do users prefer to complete the task or tasks that the chatbot is going to support? Ask users to break the task down into 3-5 steps. If a step is complex, ask them to break it down again into 3-5 sub-steps. Look for patterns across users. Once a conversation pattern is designed, test is with users in a simple mockup. Can the user finish the task intuitively? Where do they get stuck? | |
| 7 | 4.2.8 Clarify complex topics | List out the topics that the chatbot is supporting. Ask users whether they think the topic is cconfusing or unclear. Ask objective questions (i.e., questions with correct answers) about the topics to learn about average knowledge. Alternatively, talk to government employees who may have a finger on the pulse of public knowledge. | |
| 8 | 4.2.13 Highlight important info | Determine the most common answer(s) to a question. Is there data from the current way of accomplishing this task in a database that you can query? Can the customer service representatives that perform this task offer insights on common responses to the question? Ask users to rank the most common or important answer in a set. After initial launch of the chatbot, collect data on response frequency and change the order if needed. | |
| 9 | 4.2.14 Limit choices | Hold a user feedback session to evaluate and refine the options that a chatbot offers. Do users struggle to decide between two options? Is an option picked infrequently? What do users select if the option they need isn’t present? | |
| 10 | 4.2.15 List commands | Learn about the essential workflows that users want to follow and design the list of commands to support that. Ask members of the target user group to sort chatbot capabilities into “critical”, “useful”, and “other”. Do the commands use language the users understand? | |
| 11 | 4.2.16 Set expectations | What do users expect to happen after the conversation? What language communicates these next steps accurately? Do users have the information and resources to complete next steps? | |
| 12 | 4.2.17 Communicate privacy policies | Work with policy experts and users to understand what data must be saved to complete the task, what privacy policies protect that data, and what degree of privacy users would expect with the data. | |
| 13 | 4.2.18 Recognize errors | Observe users completing this task in person. What phrasing do they use? Do the users understand the way the chatbot communicates, evewn if it doesn’t match the way they communicate? | |
| 14 | 4.2.19 Reduce panic triggers | Ask users to describe how they feel about the topic that the chatbot is intended to address. Take note of the emotions the users mention and the specific part of the task that prompts the emotional response. You may discover something about the process in general that could be improved, with or without a chatbot! | |
| 15 | 4.3.6 Offer color contrast | It’s difficult to select one color contrast that works well for all users. Users with low vision may benefit from high contrast, while users with cognitive or attention disabilities may prefer interfaces with less stimulating contrast. Offering multiple color palettes may require additional development. Work with a diverse set of target users to determine an appropriate color palette for your chatbot. Pay close attention to the non-neutral colors in the website’s color palette - black, white, and grey are well-researched, but a specific purple may not be. View the colors on a range of user devices, as colors are rendered differently on different screens. | |
| 16 | 4.3.14 Accept varied language | If you have access to historical user data like forms, see if there are any common typos, mis-spellings, or other mistakes. | |
| 17 | 4.3.14 Accept varied language | Learn about the communication styles of the target user groups. What vocabulary to do users employ? How do the current customer service representatives communicate policy and information to users? | |
| 18 | 4.4.2 Make user aware of the chatbot | Find out where your users would look for a chatbot on your website so that you can provide access to the chatbot where they expect it. Ask them where the activation button should be and if they would look for a skip link or help menu to navigate to the chatbot. Does the user go directly to the “Search” function in their browser to look for the chatbot? | |
| 19 | 4.4.5 Consider accessibility in transitions | Find out what human resources are available and the gaps in accessibility to these resources. If possible, make the transition to these existing resources as seamless as possible. If there is not an accessible human resource, consider addressing that unmet need. | |
| 20 | 4.4.6 Time out gracefully | Are users typically completing the task while doing other things like seeking out documentation? Does the task take a long time, which might require multiple phases of effort? How long does the task take when a user is completely focused? How long does it take when distracted? | |
| 21 | 4.5.3 Test with devices | Ask users what types of devices and browsers they use to access web content. Be sure to pay attention to software versions. Ask the user how familiar they are with the device. | |
| 22 | 4.5.5 Include diverse users | Who is your target population? What is the target task that this chatbot helps with? Who needs or wants to do this task? How does the target population complete the task today? What disabilities are more prevalent in this population than in the general public? |