A lot of people recently have asked me about the new crop of user testing tools available on the internet. One specific tool that comes up a lot is usertesting.com, and I’d like to talk a little bit about my experience with it.
Frankly, when I first saw the site, my initial reaction was, “Well, that’s not going to be very useful.” I’ve heard a similar gut reaction from several user researchers. Once I’d given it a try, my reaction changed to, “Oh, shit. This could seriously cut into my income.”
Having used it several times now, I can happily say that neither of these reactions was correct.
Having used it several times now, I can happily say that neither of these reactions was correct.
The Cons:
Let’s start with the things that I originally noted about usertesting.com that made me think it wouldn’t be very useful.
There is no moderator.
Not having a moderator means that there isn’t a human being running the test who can ask follow up questions and delve deeply into issues that come up naturally during a session. Good moderators don’t just follow a script; they ask the right questions to really understand why users are doing what they’re doing.
It’s less useful for testing incomplete interactive prototypes.
I have not yet found a good way to test prototypes with usertesting.com. When I design, I create very sketchy, incomplete mockups of products. Typically, these only run in one browser, they have no visual design, and large parts of them won’t really work or will use fake data.
I can deal with all of these issues in a one-on-one session by explaining that the prototypes are not real, giving the participant some help in areas where the prototype isn’t perfect, or gently reminding them not to fixate on the visual design. This isn’t really possible without a human moderator.
Often, when I’m testing, I’ll schedule a break of a few hours or overnight between the first few participants and the last few. This gives me time to make fast changes to the prototype. Let’s say that the first couple of sessions show a really big problem. By quickly fixing that problem, I can get users past the problem and find the next biggest problem. Other times, I’ll actually change the discussion guide on the fly because something interesting comes up, and I want to learn more about it.
If you want to iterate between sessions with usertesting.com, you have to request a few users, wait for them to finish, watch the videos, and then decide whether to get more users or make a change. It’s not impossible, but it’s somehow doesn’t flow as naturally as it does during a standard series of sessions.
The Pros:
There is no moderator.
Hey, wait a minute! That was a con, too! Yep, it was.You see, great moderators are fantastically helpful for user testing. Unfortunately, most untrained people are terrible moderators. They lead, they explain, they won’t shut up…basically they do everything you can imagine to ensure that you’ll have a biased test.
If you don’t have somebody who is great at moderating user sessions, it’s very possible that you’ll be better off without a moderator at all. At least nobody will be actively screwing up your sessions and ruining your results.
It is incredibly fast and cheap.
User testing can be expensive and time consuming. Recruiting alone can cost hundreds of dollars. Often, recruiting firms want at least a week to line up participants, and you always have to schedule extra people to make up for no-shows and unexpectedly crazy people. (Note: My rule of thumb is that out of about every 8 people, you’ll get two no-shows and one crazy person. Your mileage may vary.)
This is simply not true with usertesting.com. Set up your test, pay a ridiculously small amount of money per participant, go out for a longish lunch, and come back to completed videos of people using your product. There’s no such thing as a no-show, and if one participant is crazy, you can give them a bad rating and have another one in a matter of hours.
Seriously. This is the feature that totally sold it for me. It reduced my turn around time on many tests from two weeks to overnight.
It is very easy to share the videos.
You know who should be watching user tests? EVERYBODY IN YOUR ORGANIZATION. You know who typically watches user tests? The moderator. Sometimes you get a product manager or an engineer in one or two sessions, if you’re lucky or you offer bribes.
But sending out a link to online videos of people using the product vastly increases the number of people who will see at least one user test. And yes, you could video tape or screen capture in-person tests, convert then to video, post them on a server and send the information to people, but this is just much easier.
But sending out a link to online videos of people using the product vastly increases the number of people who will see at least one user test. And yes, you could video tape or screen capture in-person tests, convert then to video, post them on a server and send the information to people, but this is just much easier.
When I’d Use It
So, what’s the upshot? Should you use usertesting.com or not? I would say sure. Just use it for very specific purposes.
For example, it’s fantastic when you want to just get an overall “state of the product” idea of how people are using specific parts of your product.
Another great use of it is for spinning up a new employee or making sure that current employees understand how people are using new features. Sitting down and watching three or four videos pretty quickly gets employees to feel their users’ pain.
When I Wouldn't Use It
There are times I wouldn’t use it, too. For example, I wouldn’t use it to try to get feedback on in-progress wireframes. It’s just too hard to get people to understand the limitations of an incomplete prototype without a moderator.
I also wouldn’t use it in any sort of test where I wanted to do a really deep dive into a particular experience or get specific sorts of feedback from a user. Sometimes, especially in early research, I’m not entirely sure what I want to learn, and being able to sit and explore with a participant for an hour or two is tremendously valuable. Usertesting.com doesn’t give you that, and that’s ok. That’s what researchers are for.
I’m not really sure how I’d use it to do a comparison of various different versions of features either. If I wanted to show someone three different possible implementations of a feature to see which they preferred, I can’t think of a way to do that with usertesting.com. I’d be thrilled if somebody had a clever way to do it though, so feel free to share it with me in the comments.
The Final Word
So, is it going to replace great user researchers? Absolutely not. Is it going to be a great tool that researchers can use in specific circumstances to make certain types of testing faster and cheaper? Absolutely!
Perhaps the best argument for using usertesting.com is that you can introduce it into companies that aren't currently doing any user research at all. The next time somebody in your organization says they don't have the time or budget for user testing, laugh maniacally and prove them wrong.
Perhaps the best argument for using usertesting.com is that you can introduce it into companies that aren't currently doing any user research at all. The next time somebody in your organization says they don't have the time or budget for user testing, laugh maniacally and prove them wrong.
Like the post? Follow me on Twitter!