Measuring Usability Without a Lab: Units of Effort

Units of Effort (UE) was born several months ago a friend sent me an article about measuring usability by the amount of energy spent on a task, not the amount of time spent. I loved it so much that I decided to explore it as a substitute for usability testing. Being in a situation without budget, we were always trying different ways to guerilla-test for usability.

What Is It?

In a nutshell, the idea challenges the notion that time spent is an accurate measurement for success or failure in a usability test. Taking the concept a step further, I tweaked it, attempting to make the math easier and account for some common psychological specifics that apply to user interfaces.

Units of Effort (EU) is a way to estimate usability by calculating the amount of energy it takes to complete a task.

It tested well, but my team agreed it was specific to usability. They wondered if we could expand it in a way to measure someones experience. The solution: multipliers. By doing this, we were able to maintain the spirit of the project, measuring energy, while adding an optional way to estimate the interface’s impact on a user.

The first set of multipliers focuses on a user’s emotional state. How are they feeling when they interact? How does this change their potential reaction to the interface? By using different multipliers, it’s easy to calculate a range from frustration to pleasure, based on a user’s mood.

The second set of multipliers focuses on brand awareness. If they have goodwill built up towards your users, you can get away with a little more (not that you should try…) than you would otherwise. And if you haven’t built up goodwill, you’ll have a better idea exactly how perfect your usability needs to be, so you can overcome previous bad experiences.

It’s Good, But Not Perfect

Here’s what’s really nice about EU – it’s fast. You estimate usability, and the impact of mood and awareness on it in minutes, rather than hours, days, or weeks. It gives designers a way to take ownership of their work and add some substance to it, without reaching into the bank of psychology terms, complaining about lack of budget, or relying strictly on best practices. You can take something that’s typically regarded as subjective, a design, and explain it in terms that anyone will understand – numbers.

As you probably expect, it doesn’t work as a 1-for-1 substitute for usability testing. The numbers you calculate using this are predicated on an ideal path to a user’s task completion. It could be used to measure multiple paths, but there is no way of knowing which paths are common without making assumptions. In those types of situations you should run a tree test to determine how users are navigating through the site or application.

Here’s What We Tried It On

That’s not to say that Units of Effort are not useful. My teams have tested it, using it to successfully:

• Optimize a mobile app registration with no testing budget
• Update the interface of a brokerage application with no testing budget
• Calculate the relative usability of different webpages
• Estimate the effectiveness of potential interface solutions
• Quickly point out spots in interfaces that are troublesome
• Add data to low-and-no budget situations where relying on best practices is challenged
• Explain design decisions to stakeholders
• Gain resources to conduct more thorough usability testing (!)

Shout out to @joshmacfarlane for sending me the original article. Check it out, it’s really thought-provoking.

Download the Units of Effort PDF, try it out, and let me know what you think about it.