Appendix 3:
User Questionnaire Items
These questionnaire items assess the chatbot’s accessibility for users. Incorporate these questions into user questionnaires throughout the design process to evaluate the accessibility of the chatbot. For questions that are subjective, use a 1 to 5 or 1 to 7 Likert scale. For questions that are objective, compare the user’s response to an ideal value.
| ID | Recommendation | Question | When to ask | How to ask | Answer |
|---|---|---|---|---|---|
| 1 | 4.2.1 Identify the chatbot | Did the chatbot clearly communicate that it is not a human? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 2 | 4.2.2 State chatbot purpose | What is the purpose of this chatbot? | after task | objective | _________ |
| 3 | 4.2.3 Use emotional intelligence | Did the chatbot respond appropriately to the emotions you felt? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 4 | 4.2.4 Keep messages short | Were any messages too long? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 5 | 4.2.5 Use simple messages | Are the chatbot’s messages easy to understand? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 6 | 4.2.6 Limit the number of messages | Were there more steps than necessary to accomplish the task? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 7 | 4.2.7 Build simple conversations | Are you able to steer the conversation to the topic of your choice? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 8 | 4.2.7 Build simple conversations | Were you able to accomplish your task with a reasonable number of commands? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 9 | 4.2.7 Build simple conversations | Did the task take too many steps? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 10 | 4.2.8 Clarify complex topics | Did the chatbot complex topics so that you could understand them? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 11 | 4.2.9 Support context-aware conversations | Did you you have to give the chatbot information it should already know? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 12 | 4.2.10 Supplement icons with text | Do you understand all visual communication? If you used AT, did you understand the device’s interpretation of visual communication? | during or after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 13 | 4.2.11 Media accessibility | Did you understand how the rich media contributed to the conversation when interpreted through your AT device? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 14 | 4.2.12 Supplement text with icons | Do you understand all written communication? | during or after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 15 | 4.2.12 Supplement text with icons | If the chatbot used symbols, were you able to understand their meaning? | during or after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 16 | 4.2.14 Limit choices | When you’re offered choices, is there at least one that meets your needs? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 17 | 4.2.14 Limit choices | When you’re offered choices, do you feel overwhelmed by the number of options? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 18 | 4.2.14 Limit choices | Was it easy to find an input suggestion from the options offered? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 19 | 4.2.15 List commands | Did you know how to start the task you wanted to accomplish? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 20 | 4.2.15 List commands | Does the chatbot offer helpful conversation suggestions when you don’t know what to say? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 21 | 4.2.16 Set expectations | Do you know what your next steps are after the conversation with the chatbot? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 22 | 4.2.17 Communicate privacy policies | Do you know what the chatbot is going to do with your data? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 23 | 4.2.17 Communicate privacy policies | Are you comfortable with the chatbot’s treatment of your data? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 24 | 4.2.18 Recognize errors | Did the chatbot acknowledge when it misunderstood you? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 25 | 4.2.19 Reduce panic triggers | Did you feel calm throughout the conversation? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 26 | 4.3.1 Offer multi-modal interaction | Were you able to complete your task the way you prefer to communicate using technology such as keyboard and mouse, speech-to-text, screen reader, etc.? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 27 | 4.3.1 Offer multi-modal interaction | Can you interact with all interactive elements in the chatbot with your input modality of choice? | during or after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 28 | 4.3.2 Support STT and TTS | Is the chatbot experience and capability similar whether you’re using your own TTS/STT device or a built-in functionality? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 29 | 4.3.3 Support keyboard navigation | Can you easily access everything you need on the chatbot with the keyboard? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 30 | 4.3.5 Support zoom | Can you comfortably interact with the chatbot when it is magnified to 200%? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 31 | 4.3.6 Offer color contrast | Does the color palette offer enough contrast to see the content easily? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 32 | 4.3.6 Offer color contrast | Is the color palette jarring or over-stimulating? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 33 | 4.3.8 Consider font size | Is the default font size easy to read? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 34 | 4.3.8 Consider font size | Are you easily able to adjust the font size to something convenient for you? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 35 | 4.3.9 Highlight visual focus | Can you identify where the focus is at all times? | during or after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 36 | 4.3.10 Consider chatbot size | Is the amount of scrolling you need to do acceptable? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 37 | 4.3.14 Accept varied language | Does the chatbot allow you to communicate with your preferred word choice? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 38 | 4.3.14 Accept varied language | Does the chatbot understand your messages when they contain typos, English mistakes, or flawed speech-to-text? | during or after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 39 | 4.3.15 Allow user customization | Can you adjust the chatbot’s visual and audio settings to your needs and preferences? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 40 | 4.3.15 Allow user customization | Can you adjust the chatbot’s conversation settings (including volume and speech rate control) to your needs and preferences? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 41 | 4.3.16 Manage pacing | How did you feel about the pace of the conversation? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 42 | 4.4.1 Support navigation to the chatbot | Does the tab order make sense on this website? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 43 | 4.4.1 Support navigation to the chatbot | Can you consistently navigate to the chatbot? | during or after task | objective | _________ |
| 44 | 4.4.2 Make user aware of the chatbot | Which of the following chatbot interactions were available to you? Text, voice, voice-to-text, touch screen, [insert other modalities]. | after task | objective | _________ |
| 45 | 4.4.2 Make user aware of the chatbot | Did you know there was a chatbot? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 46 | 4.4.3 Offer accessible activation | Was it easy to activate the chatbot? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 47 | 4.4.3 Offer accessible activation | Was it easy to navigate to the chatbot button? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 48 | 4.4.3 Offer accessible activation | Was it easy to find the chatbot button? | during or after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 49 | 4.4.6 Time out gracefully | Does the chatbot allow you to communicate at your preferred pacing without timing out? | after task | subjective or objective. | 1 - 2 - 3 - 4 - 5 or _________ |
| 50 | 4.4.8 Notify for new messages | Were the notifications effective at getting your attention? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 51 | 4.4.9 Use appropriate notifications | Did audible chatbot notifications sound appropriate? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 52 | 4.4.9 Use appropriate notifications | Were the notifications too intrusive? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 53 | 4.4.9 Use appropriate notifications | Were you able to focus on the task when you needed to and not be distracted by the chatbot? | after task | subjective | 1 - 2 - 3 - 4 - 5 |
| 54 | 4.5.3 Test with devices | Are you satisfied with the chatbot experience when using your AT device? | after task | subjective | 1 - 2 - 3 - 4 - 5 |