Sizing Up the Competition, Getting the Most Out of Comparative Testing

A new year is upon us, and perhaps you’re thinking it’s time to size up the competition and dominate your space. At Usability Sciences, one of the services we offer is the Comparative Test. We use this methodology anytime our clients want to find out where they stand among the competition, and more importantly, formulate a strategy for their next move. This article explains methodology, how to get the most value out of it, and when it’s appropriate to use.

How it works and what you’ll get:

The Comparative Test typically places your website/product against two to three competitors. Test participants are asked to perform the same core tasks on each product. As they work through the products, they are asked to provide qualitative feedback as well as complete a variety of surveys, including satisfaction and preference surveys.

During this process, the following occurs – A) Users identify likes and dislikes of each product, and ultimately identify a preference and B) through observation, the analyst team uncovers strengths and weaknesses of each product. When these are coupled with one another, the analyst team is then able to create and recommend a Best-of-Breed model. In other words, if the best attributes from all competitors were merged into one product, the end result would be the best of breed.

This will equip your team with the knowledge to answer important questions such as “What new features and functions should we incorporate into our design?”, “What’s working well with our product and what needs improvement?”, and “What ideas can we borrow from our competition and what ideas should we avoid?”

What we recommend:

We recommend comparing three products during the study (ideally, your product and two competitor products). This allows participants to spend a reasonable amount of time on each product and get a feel for how the products differ from one another. Occasionally, clients want to compare a greater number of products. This can be done, but the richness of insight into any given product may be compromised. Basically, more time spent per product equates to a deeper user experience and thus, a more meaningful comparison.

We’ll work with you to create tasks that expose users to the products’ core features and functions. The goal is to ensure an apples-to-apples experience so that users can make valid comparisons. And don’t worry; if the products have unique features, we have ways to incorporate them into the study as well.

Typically, we recommend recruiting around 12 users in a study of this nature. Trends will begin to emerge between users 4-8, allowing results to comfortably solidify between users 8-12.

What you won’t get:

A common misconception is that a Comparative Test will yield recommendations to resolve usability issues. These typically won’t surface in this type of methodology due to the fact that users are not afforded the amount of time needed to truly explore trouble spots and provide in-depth suggestions. More importantly, moderator questioning is limited so as to avoid artificially amplifying both positive and negative issues and coloring user impressions. Our moderators provide the playing field and the users are left to their own experience to draw conclusions.

You also won’t get metrics such as Time on Task, Success/Failure rates, etc. These metrics are derived in our more clinical and rigidly structured methodology known as the Competitive Test. That’s a topic for another discussion.

When it’s appropriate:

A Comparative Test is a valuable tool that can be used at a variety of different stages of a product’s life. If you have yet to design a website/product or have one in an early stage, this methodology can help identify what features and attributes are most valued by users. If you’re preparing a redesign for an existing website/product, the methodology can help steer you in the right direction, and more importantly, steer you away from bad decisions. And finally, if you have a mature product, the Comparative methodology is an excellent way to gauge where you stand against newer competitors and help identify enhancements and revisions for your own product.

Contact us to discuss how we might use this methodology to serve your needs and help put you ahead of your competition.

— Jason Vasilas, Senior User Experience Specialist


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s