The 7 Dos and Don’ts of Usability Testing

Usability testing often sounds intimidating to UX designers and digital marketers. It can be hard enough for managers to build internal consensus on a communication strategy. Deciding how the company website implements this through its content and design features is another hurdle. But observing how customers actually use the product – and being willing to listen, learn and change design features accordingly – adds a whole extra layer of complexity that many simply prefer not to deal with.

Managers justify this negative attitude toward UX testing with any number of excuses:

• “We already know what our users think.”
• “We don’t need to test a site we paid a design agency a lot of money to develop.”
• “We’ve tested within the company, and everyone loved it.”
• “I don’t feel comfortable bothering our users – they’re too busy.”
• “Steve Jobs didn’t care what users thought, so why should we?”
• “It is too expensive and takes too long.”

Here at Mouseflow would like to disagree. Just as we think that conducting user research is an extremely important first step in any design process, we’re convinced that concluding this process with solid usability testing is crucial. Furthermore, it is equally important to keep testing digital designs as they evolve over time.

We’ve seen many instances of how usability testing has resulted in significant design improvements that have a direct impact on the bottom line. We have learned a few things about usability testing along the way, and offer these 7 dos and don’ts for anyone working with or looking to get into usability testing.

1. Do listen to your audience

This may sound obvious, but the importance of audience feedback is too often neglected —especially when the information received does not match preconceived notions or past findings.  Whether you’re dealing with subjective opinions or empirical data, you may not immediately get the answers you want. Sometimes, you will even get answers that directly contradict previous insights. However, those answers may still be correct and, in some cases, could reveal flaws or gaps in the data you have.

What you get from your users is meant to be listened to, then analyzed, studied, and considered before it can be utilized. Any manager that treats raw feedback from usability tests as instantly actionable is sure to encounter problems.

So how do you avoid reaching an incorrect conclusion?

Being deliberately open in your inquiry is key. The first step of any design process is to recognize your own biases and suppositions—some of which may be flawed. Articulate them carefully – we find it helpful to frame them as hypotheses to be tested – and be sure to write them down. Throughout the whole design process, be sure to test your biases continuously and let them provide a set of reference points that will enable you to stay open to learning something new rather than confirming what you expected or hoped for.

2. Don’t treat your audience as a monolithic entity

Whether your test group involves a handful or dozens of participants, it’s crucial to see them as the heterogeneous bunch of individuals they are. Understand that while they must be representative of your wider target audience, they will never be able to embody it completely.  Even more importantly, understand that they have their own goals – their own reasons for being on your site.

Developing a basic understanding of each user’s goals – and their journey to realize these goals on your site – is crucial. We’re pretty fond of the Jobs To Be Done approach, which provides a framework and mindset that encourages you to comprehend user intent and the type of solutions they are seeking. We like to combine the Jobs To Be Done approach with analytics from Mouseflow to help us see how users negotiate our site as they reach for their goals and try to get their own jobs done. This allows us to gain insight into how and where the various groups and sub-groups that make up our audience at large either glide through the web experience or run into barriers that prevent them from completing their jobs.

As you proceed with the test, you may realize that there are even more different groups than you expected, or that pre-existing groups aren’t quite what you thought they were. This is all part of the process: you are gathering new insight, and it is better to go back to the drawing board than to charge ahead in the wrong direction.

3. Don’t assume that a small (or large) sample size is better by default

The “tried and true” method of only relying on five users is over 25 years old. Similarly, the current fascination with “big data” makes it tempting to gather as much information as possible. Both approaches can work depending on your methods and goals. Larger sample sizes can be costly, time-consuming, take longer to analyze, and may not be necessary at all. Small sample sizes benefit from being more “practical” when it comes to deployment but can only address a certain number of problems.

Ritch Macefield’s research on the matter led him to design this chart:

(Source: Macefield, Ritch. “How to Specify the Participant Group Size for Usability Studies: A Practitioner’s Guide.” Journal of Usability Studies, Vol. 5, No. 1, 2009.)

The whole article is worth a read, but to briefly summarize its findings for those who want to do reliable usability tests without depending on experts:

• If you’re looking to uncover usability problems in your product interface, a medium sample size (say, 10-15 participants) is generally accurate.
  • The sample size should increase depending on 1) the test’s complexity (e.g. trying to address multiple, hard-to-pin-down problems) or 2) the level of criticality (e.g. are we trying to solve an issue that, if unaddressed, would bring operations to a halt?)
• Tests that revolve around novel design changes should be kept fairly small. The more experimental or “untested” the interface or design moves, the smaller the sample group should be
• Macefield suggests that punctuated studies, although trickier to plan, have numerous benefits. These studies involve several stages, potentially with different sample sizes. Between each stage, the testers analyze their findings and potentially fix issues before the next round of testing—allowing you to solve problems in a progressive manner, reduce or eliminate biases, uncover new issues, and engage in an iterative process without having to plan new tests in the future.

And most importantly, any form of usability testing, regardless of the number of participants, is better than not testing at all.

4. Do think of usability testing as a multi-step process, and act accordingly

Usability testing is, in essence a form of user research, and as such is often a part of a larger design process. But where user research is the first step in the design process, usability testing is usually the last. Most often it is meant to test either a prototype or a nearly finished design. It is therefore not a one-size fits all process, and each test is meant to address a specific issue. As such, consider the following steps as the bare minimum when it comes to planning:

• Define your objective(s) and metrics clearly and early.
• Plan the whole process ahead of time, and make sure to establish a schedule with key milestones. This should include time for the test itself, but also time to analyze the data and to devise implementation strategies.
• As part of the planning process, choose usability testing methods that align with the plan and your objectives.

Just as important: remain flexible. Planning your process is required, but you may need to make changes as you go along. Should that happen, go back to basics: what are we trying to achieve? Who is our audience? What jobs do they hope to complete, and what’s stopping them from completing these?

5. Do use a variety of usability testing methods and tools if needed and when possible

Choosing a usability testing method can be daunting. How do you choose the one that is best suited for the job. Each have their benefits and drawbacks, and whether they’re worth using will depend on what you’re trying to achieve.

To make this choice easier we adhere to Jacob Nielsen’s and Donald Norman’s basic rules of usability:

1. Watch what people actually do.
2. Do not believe what people say they do.
3. Definitely, don’t believe what people predict they may do in the future.

Here are some of the main testing methods and tools you should consider:

• Moderated vs. unmoderated remote usability testing (URUT): The former happens with a real-time exchange and, as the name implies, in the form of moderated sessions. Your chosen facilitator(s) will be there to answer questions, guide the participants, and engage directly with their feedback. URUT, on the other hand, is a hands-off approach where users complete the test themselves—generally in front of a computer. URUT can be cheaper, more straightforward to deploy, and may involve dozens of users. The main drawback of URUT (and session replay) is that it will tell you what the user does and how – but not why.
• Diagnostic tools, such as eye tracking and card sorting, can be helpful to follow what users actually do, as can session replay, heatmaps, funnels, and surveys.
• Usability benchmarking: Tightly scripted usability studies are performed with several participants, using precise and predetermined measures of performance.
• A/B or multivariate testing: A method of testing different designs on a site by randomly assigning groups of users to interact with each of the different designs and measuring the effect of these assignments on user behavior.

6. Don’t use a scope that’s too broad or too vague

The point of usability testing is to obtain actionable feedback about what works, what doesn’t, and how to move forward. The answers you get from your participants are only as good as your test’s scope and the questions you ask.

Start out with an aim that’s as clear as the results you hope for. For example, if your goal is to address specific problems, begin by establishing a list of things you believe are problematic. Then, zero-in on each issue, and frame your scope accordingly. If you are working on a new project, consider breaking it down into smaller components before tackling the whole.

Whether you already have an idea of what you want to address or not, this part of the process should include the feedback you’ve previously gathered. After all, your users have already given you answers to questions you might not even have formulated yet. Their thoughts will confirm critical issues, or help you adjust your focus and list of priorities.

7. Don’t use test data in a vacuum

Just like our first point about listening to the user, this one should be obvious too, right?

Unfortunately, it can be way too tempting to assume that the data gathered in your most recent tests (or the test that includes the most users, or the one whose results match previous findings…) supersedes your previous inquiries. I freely admit to having done this myself though – and more than once. However, by giving one data set primacy over the others, you risk missing the forest for the trees.

Instead, mix and match what you already have—including previous usability tests, analytics, unprompted user feedback, and so much more. A mountain of data isn’t the best answer by default, but it might contain the data you need. See it as building a structure, with each brick contributing in its own way. Your past findings and your latest ones should always work in unison: leverage one to understand the other, and vice-versa.

So, what do you think? Do our 7 dos and don’ts correlate to your experience in usability testing? Ping me on Twitter: @Caspar_Reece or LinkedIn: https://www.linkedin.com/in/casparreece/  to discuss!

​