Remote Usability Testing – What, When, and How?

Situation: As a national leader in your field, you understand user experiences may vary based on their roles and functions of using the site or product. For better insight into the needs and pressure points of the site and gather ideas on how to address those concerns, you would like to do some usability testing in California , Texas, and New York but as usual, you have a strict budget and a short time frame to complete a study. What should you do? 
In the past, companies have had very few options that would answer these questions and allow them to coordinate a study of this size quickly or frugally, outside of the standard usability test or web analytics study. However in recent years, usability professionals have utilized web conferencing software and a host of communication devices to develop remote usability testing as an alternative to conducting standard “on location” usability tests.
When is remote testing appropriate?  

Remote usability testing is a methodology that allows moderators to gather feedback about a website, software product, or wire frame via an online meeting space or web conference forum. Remote usability testing is most beneficial to clients who:

  • Are interested in multi-market studies
  • Have a tight deadline
  • Have user groups that are more difficult to schedule
  • Limited budget

It has been our experience that remote usability testing is currently the most cost efficient alternative to standard usability testing as it allows clients more flexibility in project scheduling, access to geographically dispersed users groups, and is recourse to replace or schedule difficult user profiles. Using the scenario above, this article will compare remote usability to a standard domestic usability study that includes one or all of the factors above. Please see International Testing Volume 46 for remote testing considerations specific to international testing.

 

What are the advantages of Remote Testing? 

  • Cost Savings: Typically, in addition to the standard cost of a usability test, multi-market tests require facility rental fees ranging from $1200-$1800 per day and travel expenses of the moderator(s), often including airfare, car rental, food, and hotel expenses. In a remote usability experience, moderators conduct the study in house and eliminate facility rental and travel expenses.
  • Shortened project timeline: When conducting a multi city study, a large percentage of time is spent coordinating facility reservations and traveling between locations, increasing the standard 3 week life cycle of a project upwards of 4 or 5 weeks (partly dependent upon the number of locations tested). By conducting the sessions remotely, there is no need to coordinate with facility schedules or allocate additional time to travel between testing sites. Moderators will usually host and conduct the sessions from their own usability labs, and again, eliminate the need to rent a space or make travel arrangements.
  • Geographically dispersed user groups: There are a number of companies who have developed websites and products for a smaller population of people who are usually not centrally located. As previously discussed, trying to conduct usability testing in all locations is expensive, time consuming and not often possible.  Through the use of web conferencing software, moderators are able to circumvent the need to get all the participants in one location. So as in the case of the previous example, instead of scheduling facilities in 3 states over the course of two weeks to assess the needs of 3 different markets, 12 participants can be tested in the span of a week, remotely to gather a fuller representation of the each user group.

 

Disadvantages of remote testing:

  • When viewing a video of participant reaction is important:  While remote viewing software will capture the desktop activity, clients are not able to see a capture of the participants.  To alleviate this as an issue, consider requiring participants to own and have the ability to operate a webcam to use during the session.  However, keep in mind this may require more setup/prep time with each user and slightly longer sessions are in order.
  • When secure content is of high concern:  Though remote sessions are conducted in private online meeting spaces, moderators are unable to guarantee users are viewing the session without observers in the room.

Decision: Remote usability testing, when executed correctly, provides companies a cost-effective and time-efficient alternative to get valid user feedback without the added burden of travelling to multiple locations, renting additional facilities or dealing with recruiting challenges. For these reasons, an increasing number of companies in recent months are opting to conduct remote usability testing in lieu of the standard usability test.
For more information about usability testing and other solutions, please visit http://www.usabilitysciences.com/services.

Advertisements

3 comments on “Remote Usability Testing – What, When, and How?

  1. Malgorzata says:

    Not sure how crrvnoteosial this is. Most of your observations relate to clear market trends that started a couple of years back, particularly with regard to tools and techniques.I have to take slight exception to your ROI remarks as ROI is always a backward-looking metric. As such, there is no such thing as an “ROI guarantee”. And without a baseline and sensible metrics, you cannot even do a proper calculation after the fact. I have to agree with you that our industry has done more to screw up the definition of ROI than any other folks on earth. (for several years, if you searched for ROI, Jakob Nielsen hit the 1 and 3 spots on Google. DJ addresses some of the basic metric problems well in his/her comment.As usability becomes commoditised, we will see fewer charlatans. But certification will not eliminate them. Just look at all the mediocre information architects who flaunt library school degrees but have no grasp of basic business concepts. And look at all the lousy drivers who have valid drivers licenses. If there is money to be made in usability, it will be in analysis, not in observation. And the cheap tools don’t provide anything other than neutral observation.As to user experience, I agree entirely that the phrase is misused; it is actually an umbrella term for a wide range of disciplines, not a job title. But I think the industry is already starting to sort this out and the various conferences (UX London, UX Lisbon, etc.) reflect this.DJ brings up service design, which has been one of my pet peeves for years now, having cut my teeth in service design back in the 80s. There is no question that the usability folks will be affected. But that’s not because they should be doing service design, but because they are basically trained observers who are rarely qualified to make strategic recommendations regarding the stuff they test – which is why the price is going to fall as the field becomes commoditized, including the price of the tools used.

  2. […] Remote Usability Testing – What, When, and How? […]

  3. Danuze says:

    Having wkroed with David on a number of projects I can certainly say that great ROI has been achieved, especially in areas of personalisation……I am already seeing the use of MVT tools coming to the forefront where I am currently working, and whilst they are very good, they are also being used as tactical solutions for fixing issues, which does not work, and just papers over cracks, until someone releases new code, screws up the 100% MVT test, and ultimately ruins any good work/customer experience that has been live for a period of time under a guise!The prominence of remote testing software is alos coming to the forefront where I am, and again I certainly agree with the fact that this just makes everyone gain the ability to become an expert, and makes it more accessible for people to execute UE testing badly.I am expecting more and more people to be going down the personalisation route in 2010, but the interesting thing for me is how people without unique identifiers and “known” customer will execute it….some are doing it well at present, most do it poorly and actually it turns more people off……

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s