June/July 2007

Volume 47, Number 6

Report on “Conducting a User-Centered Expert Review” Session at 2007 STC Annual Conference

by Kristy L. Astry

You’re accosted in the hall by an anxious coworker who needs to know – right now – what you think of his product, which could be a Web site, form, or quick-start guide. “Can you please take a quick look at this?” carries the expectation, “Give me your verdict now because I don’t have time, money, or resources to do proper usability testing! After all, you’re the expert, right?” It’s a tall order, and you want to do right by consumers who will use that product. So, can you do a quick review and still give constructive feedback? Yes, according to usability experts Whitney Quesenbery and Caroline Jarrett.

The 30-Minute Review

In their “Conducting a User-Centered Expert Review” STC conference session, Quesenbery and Jarrett presented how to give an honest evaluation by conducting an effective review (and add to your credibility as an expert) in only 30 minutes or two days. Here’s how:

1. Don’t look at it. At the very least, try to avoid more than a glance. Tell your colleague you’ll get back to him or her in a half-hour.
   • You have only one chance to see it for the first time and to think about it like a user. Reviewing is different from using.
   • If you look at it before you try to use it, you’ll see it differently.

(Time elapsed:  1 minute for discussing the point with your colleague.)

2. Write a story. Develop a persona within a targeted demographic that might use the product (for example, a website). Ask:
   • Who is using the product?
   • Why are they using it?
   • How do they feel about it?
   • What do they expect to happen?
   • Are the users different from you? If so, how different?
   • Do you really know these people?

(Time elapsed:  5 minutes.)

3. Try to use the product as your persona.

• If looking for information, what’s the first step you would take?
• If trying to do something, can you do it?
• What else would you need to know?

(Time elapsed:  15 minutes.)

4. Now review the product
   • Look for relationship problems. A relationship is the interaction between the user and the product.
     • What are the user’s goals?
     • What are the business goals? Are these aligned?
   • Look for conversation problems. A conversation is the appearance of the product.
     • Are the headings, text, images meaningful?
     • Can the user find good “first clicks,” where the user gets validation that he or she is on the “right track?”
     • Is it tidy and attractive?

(Time elapsed:  25 minutes.)

5. Report your findings.
   With only five minutes left from the budget, you can’t report everything, so you could give a flavor of what needs to be fixed in the next version.
   • Give at least one positive comment. You want to be the usability buddy. You want the colleague to get you involved earlier next time.
   • Pick different types of problems (relationship or conversation).
   • Look for richness of information you can share in your review.
   • Get more time so you can do a more thorough review.

• The user’s story – the persona and the goal – guides the review.
• Your knowledge of usability and good design help you understand the problems the users would encounter.
• Heuristics help you keep critical points in mind.
• A user-centered review is a perspective. It keeps you grounded in the user experience.
• Even if you have almost no information about the persona, you can use your common sense and your own assumptions.
• If you have assumptions, state them clearly.
• This makes it easier to understand the context in which you make your suggestions.
• Acknowledge constraints (business or design) you had to consider.
• Strive to adjust your opinion easily if you learn something new about your users or the product.
• A good review considers more than just a checklist. Put all these points together and you have a user-centered review.

The Two-Day Review

• You could get a second opinion. The more people who can look at the product means:
• You get more insights.
• You have less chance of fixating on a minor issue.
• For best results, get a real user or real potential user to look at the product. This lets you really see the user’s expectations, and how the product supports or fails the user.
• Quesenbery was asked to review an online poker website. She doesn’t play poker, but has a friend who plays poker. This person has never seen an online poker website, but still brings her own expectations to her review: Is it easy to use? Understandable? Fun? Frustrating?
• If you can’t find a real user, consult another expert.
• Look at the product’s competition.
• If their websites, for example, are markedly different, ask why.
• How are their sites organized?
• What questions are they answering?
• Are there conventions that you should be adopting?
• More time means you can do more comprehensive reporting:
• You can cover a wider coverage of tasks and people. You could include more users, especially those whose goals contrast.
• You can cover a wider range of problems:
• You can try to discuss details of every question.
• If your colleague thinks in terms of a severity scale (1 to 10), then use it.
• You can make the report easier to use, including screenshots and callouts. This lets you focus on the most important messages.

The Risks of User-Centered Reviews

• Your colleague reads your review and becomes overconfident. Based on your report, he believes he doesn’t need to do any usability testing.
• You become overconfident and believe that you know more about the user than you really do.
• Sometimes heuristic methods can be poor at predicting actual problems.

• Get real users
• Run proper tests
• Track results

In Summary

• You think about people first.
• You think about what they need to do.
• Then you develop your review guidelines.