How Many Respondents are Too Many in Online UX Research?

When planning studies for the usability lab, sooner or later, the question gets asked, “How many users to do we need to test?” Depending on the goals of the study, and whom you ask, you’ll get answers ranging from 5 to 30. Most experts agree that testing more than that is not the best use of your limited usability budget, since each additional test participant costs money to recruit, test, and compensate.

In the world of online user experience research, a similar question comes up: “How many respondents do we need to consider the survey complete?” In the online realm, additional survey respondents are expensive not so much in terms of money, but in terms of time. How long can you wait for more and more people to complete the survey?

How many responses you’ll need really depends on what you plan to do with the data. What are the main goals of the project? Do you intend to use the data to inform design decisions (e.g., for a site re-design effort) or do you intend to use the data as benchmarks/metrics and compare it to some other data set (e.g., data collected in previous rounds or future rounds of research)? Related to the overarching goals of the research project are the analyses you’d like to have done on the data. Are you interested in analyzing click-stream data or are you strictly interested in survey responses? Will you want to “slice and dice” the data numerous ways to see how different demographic groups respond or how certain survey responses relate to other survey responses?

If you desire complex analyses, analysis of click-stream data, or multiple cross-tabulations of the various survey questions, then we recommend a minimum of 3,000 responses. This number of responses makes allowances for the large amount of variation that we see in site visitor behavior and helps to prevent any particular sub-group of respondents (e.g., first-time visitors) from being too small for meaningful analysis.

If you know at the outset that you are not interested in analyzing click-stream data and that your desired analysis of the survey responses does not involve complex or multiple cross-tabs, then answering the question of “How many responses do I need?” really boils down to two different scenarios:

  1. You need to compare this data set to another data set (e.g., from past or future rounds of research). If you will eventually have more than one data set AND you want to answer questions such as, “Did success increase from Round 1 to Round 2?” then we’d recommend gathering as much data as time would allow. If you intend to compare the data across data sets OR if you just don’t know whether or not you’ll need to compare the data at some point in the future, then we usually recommend a minimum of 1000 responses. Gathering 1000 or more sessions for a survey gives you greater flexibility in terms of how you might use the data in the future. One thousand sessions provide a sufficiently narrow margin of error (± 2.6% at a 90% confidence level) that you can draw conclusions about apparent differences between the two data sets and trust that those conclusions are reliable.
  2. You primarily are running the survey for qualitative purposes (for example, in order to inform design decisions, gather verbatim feedback, discover usability issues, etc.). If you know that you are not going to need to make numerical comparisons between two data sets, then you can feel reasonably comfortable with fewer sessions. The fewer sessions you gather, the wider your margin of error becomes. For example, at 90% confidence, here’s how the margin of error looks for various sample sizes less than 1000:
Survey sample size response table

As you can see, with 400 survey responses, your margin of error is somewhere close to ±4%. You can also see from the table that the relationship between number of sessions and margin of error is not linear, and there is a point of diminishing returns. If your research goals fall in this second category, you need to consider how wide a margin of error you feel comfortable with and balance that with how long you have available to let the survey run.

So, just as with lab-based studies, the answer to “How many responses do we need?” varies depending on the goals of your study. In general, though, it should fall between a few hundred and a few thousand.


How to Use the ‘Quick View’ Feature to Enhance the Browsing Experience

It has become commonplace for e-commerce sites to provide a Quick View feature to enhance the shopping experience for their users. This post addresses the “Do’s and Don’ts” of the Quick View feature.

Let’s start with a definition. The Quick View is a site feature that allows users to quickly view detailed product information without navigating away from the product listing page. Upon mouse-over of a product image, the user simply clicks a button, typically labeled ‘Quick View’, and an overlaid window appears, presenting the product detail. This minimizes time spent browsing products since the user no longer has to wait for the site to refresh and display a product detail page every time he/she wants to review product details.

Consider the screenshot examples below. Figure 1 displays a Quick View button for the product of interest (in this case, labeled ‘QuickLook’). Figure 2 displays the resulting Quick View window.

Figure 1

Figure 2

Creating User Awareness 

To alert users to the presence of the Quick View feature, follow the guidelines below:

  • Ensure the Quick View button appears on mouse-over of the product image, as opposed to appearing at all times. This presents a cleaner look and feel. Additionally, ensure the button appears instantly upon mouse-over.
  • Ensure the Quick View button is centrally located within the product image and thus in the users’ direct line of sight.
  • Ensure visual treatments of the button are such that the button commands attention. In the example screenshot above, notice the button’s contrasting colors.

Optimal Quick View Windows

For an optimal Quick View window, consider the following guidelines:

  • Provide dynamic content within the window.  Ensure relevant product features, such as product size and color are dynamically available via mouse-over or click of an icon.
  • Balance conciseness with rich content through the use of bulleted text and/or tabbed information, if needed.
  • Provide the ability to navigate to the actual product details page so that users can obtain additional information, if desired.
  • Provide the ability to add to cart from the Quick View window, as well as make all required selections such as size, color, quantity, etc.
  • Include an easy to locate ‘close’ icon in the top right of the window.
  • When adding to cart, present relevant error messages that dynamically appear within the window if required selections have been overlooked or if there is insufficient inventory.
  • Upon clicking the ‘Add to Cart’ button, provide clear notification that the item has been added to cart.  Ensure the window closes automatically, so the user is not required to take the extra step to close it.

Potential Pitfalls

When implementing a Quick View feature, avoid these potential pitfalls:

  • Do not strip out all product detail from the product listing page just because you have placed the details within the Quick View window.  This is particularly important in the case of technical or electronic products.  Stripping out all product detail from the detail page will likely frustrate users as they will be forced to rely on the Quick View window to access product details.
  • Do not do away with the product detail page.  Though the Quick View feature is now more common on e-commerce sites, there are still many users who want to navigate to and view a product detail page before making a purchase decision.  Ensure the Quick View is not the sole option for providing product detail.

By following the recommendations above, the Quick View can become a powerful tool that enhances the user experience by cutting down time spent browsing.

Improving the User Experience of ‘Live Chat’ Can Improve Sales

Many online retail sites have decided to offer their visitors an online chat option, a clear signal that online retailers and shoppers alike are beginning to embrace live chat as an effective alternative to telephone and email customer service support.

Our focus here is to examine the live chat feature in the context of usability best practices as it relates to the following factors:

  • Look and feel of the live chat icon
  • Availability and placement of live chat option
  • Content and user interface design of chat screen

Look and Feel of Live Chat Option

Examples below give an insight into the look and feel of some typical chat icons.

Live chat icons found on retail sites vary greatly and include some of the following:



Overstock_revised.jpg    Need_Help.jpg

Deciding what type of visual representation to adopt should depend on the demographic segmentation of a particular site’s target visitors. If a site caters to a narrow demographic segment, then an icon with a picture representing that demographic could be viewed in a positive light. However, if a site draws visitors from a broader demographic segment in terms of gender, culture and geography (both national and international), then a prudent choice may be to use a gender- and culture-neutral icon such as a stylized thought bubble or even a text link.

Availability and Placement of Live Chat Option

Some practices you may want to avoid or modify in regard to the placement of the live chat option:

  • Some sites attempt to aggressively establish live interaction with their site visitors by presenting visitors with a chat window within a few minutes of visitors arriving at the site. This could become a distraction which may cause visitors to leave the site altogether. Our usability test studies have shown visitors prefer to seek out live chat when they perceive a need for it. To that end, discretion should be used when considering whether to push unsolicited live chat onto visitors.
  • Many retail sites choose to place the chat option link in the text links area at the bottom of site pages and the homepage, as illustrated below:nordstrom_revised 2.jpg

    Though placing the chat link at the bottom of the page gives visitors a way to access the option if they so desire, more often than not, users tend to ignore information placed below the page fold, especially if they have to scroll extensively to access it.

Some practices you will want to learn more about and employ when it comes to placement of the live chat option:

  • The emerging trend of placing the chat feature in the general area of the top navigation bar could serve visitors better by making the chat icon fairly inconspicuous, but at the same time giving it more visibility than a text link located within the page footer. For one such illustration, see the graphic below:Tool_Bar_containing_chat_icon_revised_Dell.jpg
  • Many retail sites also place the chat option strategically in the vicinity of the ‘Add to Cart’ button on the product detail page, as illustrated in the example below:Prod_pg_revised.jpg
  • Additionally, the chat feature can be placed at all the possible points where users go to seek answers to questions they may have, such as throughout the checkout process. Other likely points would be the help section, the contact us section, the customer service area and the tech support area.
  • Placing a chat icon beside the search entry field may also increase the visibility of the chat option as the search entry field typically garners a significant amount of visitor traffic.

Content and User Interface Design of Live Chat Screen

Once visitors access the chat functionality, the challenge now shifts to providing them with a user-friendly and seamless user experience. To that end, the following guidelines should be kept in mind:

  • Avoid requiring users to enter personal information such as their telephone number and email address, as this may discourage users from initiating the chat.
  • When a visitor initiates a chat, ideally, a representative should be available to respond immediately. If that is not possible, the visitor should be shown a message displaying the estimated wait time.
  • In situations where representatives have to leave the chat momentarily to check records or obtain additional information, ensure the representative informs the visitor of this by saying something to the effect of “Give me a moment and I will check that for you.”
  • When representatives are typing their response, display a message on the chat screen that reads, “Representative is typing a message.” This will keep visitors informed and they will be less likely to question a delay in response time due to a lengthy message.

When live chat is implemented with usability in mind, it helps online retailers offer site visitors a measure of hand-holding, which in turn, could pave the way for minimizing cart abandonment and maximizing sales. Following these best practices can help make a good feature even more effective.

How to Build a Usability Lab – Part 1

This is the first in a series of ongoing blog posts aimed at giving you a blow-by-blow account of our upcoming move to our new offices, and the ensuing adventure of building out our new state-of-the-art usability labs.

We at Usability Sciences are in the process of moving to a new office building and a ton of work is going on to make it a successful effort. We thought it would be great to share with you our experiences related to building out our new usability labs, as well as a focus group facility.


We have been at our current facility since 1999 and have expanded twice during that time, building out our usability labs, client rooms and offices each time to our current 17,000 sq. ft. space. We have been perfectly happy here but feel the time has come to move to a new office space with state-of-the art technology and upgraded interiors. The new building we have scoped out is in Las Colinas, the main business district of Irving, TX, and is located close to a new DART Light Rail Station, making getting to downtown Dallas or the DFW Airport a cinch. Our new offices will take up  the entire 19,000 sq. ft. of the 16th floor.


When thinking about building out the new facility, we focused on these primary goals:

  1. Keep the labs, focus group and client areas separate from our employee offices.
  2. Build out four state-of-the-art usability labs to accommodate all the various usability testing methodologies that we do.
  3. Improve our audio/visual setup for all labs and client areas.
  4. Build out one fully-equipped focus group room.
  5. Provide our clients, who come to watch and participate in running our studies, spacious work areas and a better working environment.

Separating Labs and Client Areas from Employee Offices

Shown below is the almost complete floor plan, with the labs, client rooms and focus group facility outlined in red.

Key Items in the Floor Plan:

  1. Four usability testing labs of varying sizes. Some have large user rooms for conducting tests related to kiosks, gaming, and product package scenarios. Some of the labs have smaller user rooms with larger control rooms for situations where 3-4 clients would like to be in the control room.
  2. A very large focus group room with attached control room.
  3. Three client observation rooms. The focus group room doubles as a fourth observation room.
  4.  A separate user waiting-room, centrally located within easy access to all labs.
  5. Large, well-equipped client kitchen.
  6. Client phone room off of the kitchen.
  7. All access to employee offices controlled via keyed doors.

In future posts, we will cover in more detail how we plan to achieve the goals laid out above, and also provide updates as we go through the bidding and build-out process, and the final acceptance of our new space.

We would love to hear from you about any experiences or insights you may have had with building out a usability lab. Please share with us your own adventures!