ToddElliott – firstname.lastname@example.org
Until fairly recently, when designers wanted to test an idea or
design, they sought out an outside usability agency or, rented a
room, some expensive equipment and recruited users to come into an
artificial environment to participate in a usability study. In the
past few years, technology has become available that has brought
the cost and complexity of user testing down to a level where it is
available for anyone. In addition, it allows designers to be
exposed to users, in their native environment, not a sterile lab
across town. You can run a test in only a week for less than a few
There are numerous reasons remote user testing is valuable:
1. It’s cheap – most tests can be run for little more than the cost
of your time and the incentive to the user.
2. It’s fast – done right, a study can be put together,
executed and summarized in about a week. You also get
immediate feedback that can be rolled directly into your
3. Mediate conflict – User testing can be used to resolve internal
4. Test ideas – A quick user test can often help resolve a
difficult design choice, like labeling or determining whether users
can figure out how to use the new drag and drop system for adding
items to their shopping cart.
5. Reduce release cycles – Instead of putting a new idea out in the
wild and then having to scramble with a follow-up release, running
a quick user test can often yield valuable insight that will save
you time in the long run.
One of the best reasons to do remote research however, is that it
offers you a window into the users environment. In addition to
seeing how they tend to navigate a page you will see what kind of
electronic interruptions occur, such as IM and email. You get to
experience a little bit of their physical environment, is it noisy
or quiet? Do they have kids competing for their attention? This is
the kind of real-life testing environment that can yield unexpected
and extremely valuable results.
One user test we conducted not long ago included a large segment of
users that were relatively new parents, meaning there were often
kids of all ages from in the background while we were testing. The
product had been tested extensively in QA, and the usability tests
were almost an afterthought. During the course of our testing, we
discovered one very important issue: If the users walked away from
the sign-up process for more than a couple of minutes, their
session would expire and they would lose all of the information
they had input. They had to start all over again. This only
came up because one of the users we tested had a small child that
demanded constant partial attention, and caused her to walk away
from the computer for several minutes. It was one of the most
valuable outcomes of the test. More importantly, it was a wholly
unanticipated outcome that was only possible because we were
testing the application.
Whether your client is an external one, or an internal group,
invite them to watch the tests. Clients that listen in on user
tests almost always come away with a fresh perspective on how users
use their site.
There are many different types of remote user testing, the case
above was the result of standard moderated remote usability study.
However there are a number of other ways of collecting feedback
from users without having them come into a lab.
Card Sorting is a valuable tool in its own right to help inform the
top level navigational structure of a site. It’s a common tool used
by Information Architects everywhere, and is a cheap and reliable
way to quickly get feedback on navigation and findability. This is
done by… yes, sorting cards. Each card is labeled with a piece of
information or a function. The cards are given to the users who are
asked to sort them into logical groups. This helps uncover users
expectations, create logical categories and help with naming of
those categories. This type of data is very helpful early in the
project when you’re trying to establish the overall navigation.
The best part is that tools are available online that make it easy
to do this sort of testing remotely, which means you can design a
card sorting study for dozens, even hundreds of users without the
time consuming overhead of managing so many in person
Online tools like Optimalsort and Userzoom make it easier than ever to
run card sorts. Userzoom is newer to the space and appears to offer
a really nice output for easy analysis. I’m looking forward to
using it at the next opportunity.
Remote Moderated Usability testing
In the case of moderated remote user testing, you can test specific
ideas or solutions for a small audience: How long does it take
users to find things? What do users think your site labels mean?
Can users figure out how to get where they want to go? You
can even test multiple designs on the fly, and test improvements as
you go along.
It is also valuable to have your clients observe and listen in on
the sessions using Adobe Connect or other meeting applications.
Meeting applications like these are what allow you to see the users
screen. As I mentioned earlier, the insight into the users
experience can often foster empathy for the user, which is can be
very valuable, if you need their support to improve the
There are a number of tools available for the lab environment that
aren’t viable remotely. It’s sometimes desirable to do eye tracking
or to record the users expressions while running the session.
Though we don’t typically do this, there are more and more tools
becoming available that add this type of functionality for fans of
remote usability studies..
One example for recording users is Silverback which uses the
cameras built into the Macbook and Macbook Pro to record the user
as well as their screen. This is very handy if capturing the users
expression while they are going through the test.
Recruiting For Your Session
It’s important to test your designs and solutions with the intended
audience. For that reason, often the best audience to take
advantage of is the one already coming to you. In order to recruit
them, all you need is a survey and a survey tool. Your survey
should be short and designed to capture a spread of your current
audience. It’s useful to offer an incentive. Amazon gift
certificates make great incentives, and are easy to manage because
they can be emailed in bulk from Amazon.com.
There are a whole host of survey tools available for practically
free out there. They include Survey Monkey, Wufoo.com, even Google Docs has a survey form option
now, and User Experience experts, Bolt Peters, have a very nice,
supported solution for a small fee. Once you create the survey, all
you need to do is identify the page or pages you want to recruit
bottom of the page. Depending on how much traffic your site gets,
you’ll usually want to select the pages that get the most
Once the survey is up and running, you can start scheduling a pool
of candidates, or you can try to recruit them on the fly as they
come to the website, and have them join the online meeting.
Designing A Script
One key to a good user test is a flexible script. If you force the
user through a pre-determined set of tasks, you’ll risk miss out on
those unexpected opportunities. Letting the user direct the
study. I often like to start the study by asking the user what they
came to the site to do originally. This is often referred to as
their “passionate task” or the task that they came to the site to
do. It can often give you surprising insight into users thought
That being said, I recommend taking the time beforehand to map out
a series of questions. A good rule of thumb is to try to limit the
test to 45 minutes. People often start to lose focus after that
long. It doesn’t take many questions to fill up an hour, so it’s
usually a good idea to pick about 3-4 main tasks.
A script should also include all the logistical information up
front. Introduce yourself, and give them a little background on the
study’s subject. You want to ask their consent, give them specific
details on how long the test will last and how the software works.
When you’re done, don’t forget to thank them and confirm the
address you’ll be sending their incentive to.
Screen Sharing & Capturing the Output
The three keys to a successful test are:
o You can talk to the user
o You can see what they’re doing
o (optional) You can record both the audio and the video for later
This is where the field has gotten pretty interesting lately. There
are a number of inexpensive solutions out there for screen
sharing. UserVue by Techsmith is still one of the
best screen sharing and recording solutions out there, but
the rumor is that they will be going away in 2010.
Both iChat and Skype offer screen sharing, and allow you to voice
chat. If quick and dirty is your focus, Skype has a large installed
user base and is already free. Other solutions include Adobe
Connect, which has a little more up front cost, but also a whole
range of uses beyond user testing.
If it’s important to record your session, Camtasia by Techsmith is one of the best
solutions out there, and is now available for the Mac and the PC.
And of course, Silverback, which I mentioned earlier, although it’s
currently for Mac only.
Finally, I’d like to mention a few tips not covered in this
overview. The first is that it’s very handy have someone sit in
with you to take notes. It take a lot of practice, but it is
possible to run a test and take notes at the same time.
Secondly, save at least 15 minutes between each test to summarize
what you heard. This will help you quickly include those insights
in your next design.
There you have it, all the basic tools you need to run your own
user tests. How about it readers, are there tools out there that
you like that I haven’t mentioned? If so, let me know.