Heuristic Evaluation

There is already a myriad of information available on the web about Heuristic Evaluation (HE). The purpose of this post is not to simply re-iterate that information.  As a usability practitioner I have performed many heuristic evaluations on websites and software applications, and have helped other people with less experience perform heuristic evaluations.  In this post you will find a brief summary of HE, and answers to some of the common questions I am asked by new heuristic evaluators.

Overview of Heuristic Evaluation

What is heuristic evaluation? Heuristic evaluation is a quick and dirty method of identifying problems in a user interface (ie/ a website, or a software application).  The method was first published in 1990 by Jakob Nielsen and Rolf Molich.  They had distilled their experiences with hundreds of software interfaces into a set of 10 usability principles that the most usable interfaces adhered to.  This list of “heuristics” is what is most commonly used today (for better or worse) when performing heuristic evaluations of websites and software.

“How do I perform a heuristic evaluation?”

The first thing that you need to know about performing a heuristic evaluation is this: it’s easy and you can do it. Don’t get overwhelmed by the unknown.  By the end of this article (in 5 minutes) you will know enough to do a heuristic evaluation.

Traditional heuristic evaluation involves having a few people independently examine how well a user interface adheres to a set of usability heuristics.  The evaluation is usually done in one of two ways:

1. Task-based: Each user is given a set of tasks that they are to complete using the interface.  They are armed with the list of heuristics to evaluate the interface against, and as they perform the tasks they make a list of the violations they encounter.
2. Element-based: Each user is asked to go through the interface a few times.  After they are familiar with the interface, they are asked to evaluate the different elements that they interact with against the set of heuristics, noting any problems they have had.

I have found the task-based heuristic evaluation to be very effective, though the information gained from each type of evaluation is slightly different.  Task-based HE tends to reveal more issues related to the interactions required while performing the specific tasks.  For this reason, it’s important to carefully select which tasks you give to the evaluators, which depends on your goals.  Element-based HE tends to reveal more aesthetic problems, and problems with navigation.  Since people don’t have specific tasks they are attempting to complete, they tend to focus more on the things they notice when “surfing” around a website.

Regardless of which method you choose, the results of the individual evaluators must still be aggregated.  Each evaluator will give you their findings, at which point you must group the similar problems and remove the duplicates.  Make sure that you have a clear understanding of each of the problems by asking the evaluators questions to clear up any ambiguity or misunderstanding.  At the end of the exercise, you end up with a list of unique problems with the interface.

It’s important to ask your evaluators to be as specific as possible when doing the evaluation.  Comments like “I don’t like this” are not valuable.  Encourage them to link the issues they find to the heuristics.  This ensures that they will separate the issues instead of grouping them into one mega-issue, and makes it easier to track what issues have been dealt with when making fixes.  It also allows you to make partial fixes to problems, instead of having to tackle a mega-problem (which you might not want to do).

“Who should I get to evaluate the interface?”

This seems to be a major sticking point for companies and individuals new to performing heuristic evaluation.  Don’t let this stop you. Sure, usability professionals (“expert evaluators”) are preferred as they are more skilled at finding usability issues than non-experts, but they are not required.  If you don’t have the budget to hire usability gurus to evaluate your site, it’s not the end of the world.  You can do this with non-experts and still gain a lot of insight into the problems with your interface.  Qualities to look for in potential evaluators (ranked in order of importance):

• a heartbeat
• they have not used the UI before
• they are potential users of your UI

If a user only meets the first criteria, they are still a valid test user and will give you valuable insight into problems with your UI.  The other criteria are nice to have in an evaluator, but not necessary.  Not having used the UI before is a bonus, because they have no expectations.  Getting potential users of your UI to evaluate it is a good way to see how your target demographic responds to it.

“What will I gain by doing a heuristic evaluation?”

The results are pretty incredible: with only 3 to 5 non-expert users evaluating an interface, most of the major usability issues are discovered.  It’s wise to do a new round of heuristic evaluation after each re-design: the benefits you gain from it outweigh the time required to do the evaluation.  When this is no longer true, then your design is most likely relatively stable .

“What are the problems with heuristic evaluation?”

Actually, people generally don’t ask this question.  But they should :).

A heuristic evaluation is not a golden bullet:  it is a useful tool and will help to identify usability issues with an interface, but it’s got pitfalls too.

An obvious problem with heuristic evaluation is that it identifies problems, not solutions.  You and your team still need to come up with the solutions and ways to correct the identified issues.  You can solicit feedback from the users that identified the problems, as they usually are happy to give their ideas (after all, they did find the problems).

Another problem with HE is that the issues identified by the users depend on the list of heuristics that you use to evaluate the interface.  People will look for problems that fit into the heuristics, and this can limit the scope of the problems that are identified.  Nielson’s heuristics are a very useful collection of usability principles, but are they all that you need?  Heuristics specific to the domain of your application may be of more value.  For example, if you’re evaluating an online store creating trust is essential, but there’s nothing in the heuristics about trust.

“Is heuristic evaluation the only usability method I need to use?”

I wish I could say yes, but the truth is, no, it’s not.  It’s a great place to start, and if you have a limited usability budget and are on tight deadlines, then heuristic evaluation is a good way to identify the major issues with your software.  It can be easily integrated into design cycles at little cost.  However, heuristic evaluation is not a substitution for user testing.  Different usability inspection methods give different benefits.  The types of problems you identify with HE are going to be different than what you identify with user testing, and vice versa.

And now…

Go and perform a heuristic evaluation!  If you’ve never done one yourself, get a list of heuristics (Nielson’s are the most famous and widely accepted) and perform a heuristic evaluation.  Do it on your own website if you have one, and if not, any website that you read/use regularly.  Next, get a few friends to perform an evaluation on the site, and see what you come up with.  You might be pleasantly surprised.