If you’re not doing user testing, you should be. Otherwise, you have no idea how your customers actually behave.
That’s according to Amy Chowdhry, who, in recent months, has become all too familiar with remote user testing as CEO of AnswerLab, an independent research and service agency that helps clients like Facebook, Google and Amazon with the process.
Chowdhry said the company, which until early March conducted 75 percent of its testing in person in locations like the Empire State Building and on Market Street in San Francisco, went completely remote over the course of five business days, shifting 90 projects to online settings.
I think it’s important right now to recognize that remote research might be our only option for a really long time.”
“I’ve seen a lot of discussion happening among the UX community about the pros and cons between remote testing and in-person testing. And I think it’s important right now to recognize that remote research might be our only option for a really long time,” she said. “So I’m asking people to reframe their mindset from focusing so much on, ‘Oh my gosh, what are we going to lose?’ to instead, ‘What can remote do for me?’ It’s the responsible question to ask.”
So there’s step one.
Choose the Right Platform
For Chowdhry, selecting the right platform is a critical next move. Videoconferencing platforms like Zoom are an option for firms on a tight budget, but for large companies leading research involving multiple stakeholders, she recommends third-party services specifically designed for UX research, such as UserTesting, dscout, Userlytics or Validately.
Before conducting the actual testing, she told me, it’s a good idea to go through a dry run.
“You want to do walk-ins with the participants prior to the session, so they have comfort with the platform you’re using. It could take 15 minutes, but you both log in to make sure the person doesn’t have any problem getting access, they understand how to use the controls and they’re set up in an environment that will be conducive to the research.”
Empathy for research participants is also important — particularly in the current moment.
“So many people have been in shelter in place for two months now. People who live alone have been very socially isolated, and sometimes a research moderator could be their only connection in a couple of weeks,” Chowdhry said. “So we need to be recognizing that and open to being extremely empathetic in that moment.”
Set a Clearly Defined Goal
AnswerLab’s 130-person team works primarily with mature startups and Fortune 500 companies through annual retainer agreements, but companies of virtually any size, stage of development and means can conduct remote user testing.
The first and foremost thing is having a clearly defined goal. What are you trying to answer?”
One popular third-party service is actually called UserTesting. Chief Insights Officer Janelle Estes said the company has 1,500 enterprise-tier subscription customers, from travel companies like Expedia to financial services firms like Capital One, and also works with startups and mid-size firms. On an ad hoc basis, firms can conduct up to 15 tests for $49 per test participant, with tests typically involving 5-8 people.
“The first and foremost thing is having a clearly defined goal,” Estes said. “What are you trying to answer? And if possible, tying that to a business objective. How will we define success or what metric am I trying to impact? Am I trying to impact conversion? Am I trying to impact customer satisfaction? Am I trying to impact the deflection of support calls?”
Work in Stages
UserTesting, like other third-party sites, helps firms gather usability feedback once they’ve identified a focus area. Sample users from a proprietary panel of 1.3 million paid, on-demand participants, answer questions or perform software tasks in live video tests or asynchronous pre-recorded activities.
This data, Estes told me, is then used by companies like CarMax, Zillow, Evernote and Carta, to understand how customers interact with their products, so they can make better business decisions, optimize software performance and, hopefully, boost conversion rates. Companies can return to the cloud-based site multiple times to view stored videos, isolate “moments of interest,” and analyze responses and user pain points saved in a dashboard.
User testing works best, Estes said, when it flows from a large question to smaller ones. This process often occurs in stages, beginning with discovery. “How do I solve the right problem and how do I surface the right solution?” as Estes described it.
The second stage, Chowdhry said, involves a key choice and “occurs [when] the client has a series of concepts in mind. We think we want to go these two or three different directions. We have these ideas for what the value proposition should be. Which of these would be the best way to go?”
Should all these clothes be organized men’s versus women’s first or should they be organized jackets first and then men’s versus women’s?”
Once that’s narrowed down, testing can explore the information architecture of an app or website, categorizing information and creating a mental model for the experience. This can occur in several ways, Chowdhry said. Card sorting involves organizing conceptual ideas into categories with tools like OptimalSort. In participatory design, teams sit down together to talk through a flow. Tree jacking, another popular framework, is a method for determining how fast people can find specific information.
“A great way to describe this might be, imagine you’re thinking of how to create a hierarchy of content categories on [an e-commerce site]: should all these clothes be organized men’s versus women’s first or should they be organized jackets first and then men’s versus women’s?” Chowdhry said.
When all that’s done, you’re ready for the iterative product design and development stage.
“So now you’ve got a functional prototype in some form,” Chowdhry explained. “It could be wireframes; it could have some design to it and we iteratively test it. So you test it with five to six people. The client makes a bunch of changes based on recommendations from that. You test it again and you go through a series of processes until you get to this end design that is going to be the best for launch.”
Mimic Testing Practices You’ve Done in Person
For those trying out remote testing for the first time, Estes recommends starting with face-to-face video conversations that mimic in-person testing scenarios.
“It doesn’t require a ton of prep and set-up,” she said. “You go in with a set of questions, and there’s not a lot of programming or coding required before you can start to get feedback.”
More mature firms can study the experiences of users on competitors’ platforms to gain insight into how their software stacks up.
“Let’s say I’m a financial services provider, and I’m trying to increase the number of people who apply for mortgage applications,” Estes said. “What I’m going to do is go out and recruit people who are in the market for purchasing a new home. I’m going to have them go through that process using my experience, but then I’m also going to have them go through my top two competitors.”
Firms can also compare their online experiences against those of best-in-class companies, such as Target, Apple and USAA, “the poster children of stellar customer experience,” Estes said.
One UserTesting study, for example, identified brands that appeared on two or more popular consumer experience indices, such as Forrester’s, the CX Index and the MPS Index, and then assessed them on various criteria, such as their mobile experiences.
Sharing her screen to show me a thread of videos of users interacting with Starbucks Coffee’s mobile app, she walked me through a sample test in which participants are prompted to perform a series of tasks, talk through the process and rate the level of difficulty.
“Would you shop on this website?” she asked, moving her cursor to the instructions tab. “I see the coffee I want, that’s good.”
By analyzing the data, she said, companies can make the user experience better, which can lead to faster end-to-end transactions, increased customer satisfaction, and, hopefully, increased engagement and sales.
Solve the Right Problems
One of the advantages remote has over in-person testing, Chowdhry points out, is it allows you to select from a wider pool of testers, who might otherwise be excluded due to scheduling conflicts, geographic distance, or childcare and travel challenges.
Participants can be screened for all sorts of considerations: age, income, region, brand experience, online spending and so on. At UserTesting, panelists are rated on the quality of their feedback and paid $10 for 15-minute sessions.
So you have access to people, but you also have the ability to find the right people or the people that are relevant to your business.”
“What people can do within our platform is essentially filter who they want to talk to,” Estes said. “So if I know I need to talk to Millennials, I can use certain demographic filters to target just those people. If I know I need to connect with parents with children between the ages of 13 and 18, I can do that. So you have access to people, but you also have the ability to find the right people or the people that are relevant to your business.”
Testing can lead to some counterintuitive insights. For example, when Walmart Canada was experiencing a drop-off in visitors to its buy pages, the design team was puzzled. Users were going to the site and finding products, according to Estes, but they weren’t adding them to their shopping carts.
Internal hypotheses were being floated about what the issue was. Were the prices too high? Was there not enough detail in the product descriptions? Was it because the store didn’t offer free shipping?
Actually, none of the above: “What they learned was that people actually wanted to see higher-quality and different product imagery, but the product imagery just wasn’t meeting their needs,” Estes said. “Our platform helped them identify the right change to make because otherwise that team would have just been shooting in the dark.”
User testing can also go in the other direction, revealing when something is working particularly well. A Coca-Cola billboard on display in Times Square reads “Staying apart is the best way to stay united,” with a large separation between the brand letters. The creative agency Mercado McCann, in partnership with the soda company, wanted to see how it would play with the public, so they used UserTesting’s remote testing service.
“We actually sent people an image of the billboard, and we were like, OK, what do you think?” Estes said. “If you look at the results that we have here, it was clear how it made people feel. Things like supported, safe, caring, warm, hopeful, intellectual. What resonated with them? The phrase, the simple message, the quote and the red color board. And then, what to improve. Nothing. I wouldn’t change anything.”
A rare result, Estes said, but valuable nonetheless.
Benchmark Your Results
User testing rarely ends when a release goes live.
“Usually, you want to benchmark it,” Chowdhry said. “So you want to make sure it’s delivering a great experience over time. With that approach, 50 to 200 people are testing something and you’re gathering metrics. And you may be looking at yourself and your competitors and making sure your product is comparatively adding the highest value to users and the best experience.”
Estes said these metrics are often collected with web analytics software, and vary from company to company. “One of our large software customers wanted to get people signed up in five seconds and then give them five minutes to have an ‘aha!’ moment. Maybe today their sign-up flow takes 20 seconds and time to wow is more like 15 minutes.”
“Another customer that comes to mind,” Estes told me, “was trying to figure out how to get fewer people to call their call center. They have a certain percentage of people that call the customer support line and they want to reduce that. The way that they did that was by understanding why people call, particularly those who are already on the website.”
Remote User Testing Is Thriving
Since the start of the coronavirus pandemic, Estes said UserTesting has seen a dramatic spike in new users: 1,300 COVID-19 related customer studies have been launched since April 14, and 125 new customer accounts are related to the virus.
Travel companies are looking at the flows for canceling and rebooking reservations. Newspapers are looking at registration walls and how to streamline those.”
“Travel companies,” she said, “are looking at the flows for canceling and rebooking reservations, newspapers are looking at registration walls and how to streamline those. We even have a public school that was testing the findability of instructional videos and activities. It’s really across the board.”
Chowdhry has witnessed a similar trend. “Obviously, tech giants are seeing more engagement right now,” she said. “Financial services firms are really doubling down on their digital experiences even more than they were before. These clients are growing a great deal even through what’s going on right now.”
Remote testing may be winning new adherents, but for all the enthusiasm behind it, the process is rarely seamless.
“The challenge you have is that, because you’re not sitting in the same room with the person, sometimes it can be harder to build that connection and rapport,” Estes said.
If a team is testing a prototype, not a live website, Chowdhry added, a process for sharing it securely needs to be considered, and that can require outside consultation. If a team is testing hardware, they need to determine if they will ship the product to testers or try to recreate the experience online.
Companies like AnswerLab are pivoting to find solutions for these emerging obstacles, and many of the ideas hold promise. “We’ve figured out a way to stream what’s happening inside a VR headset into a private group so that we and our clients can see what the participant is experiencing,” Chowdhry said.
Figuring out how to fully account them will take time, but for advocates, it’s not so much a question of whether you should conduct remote user testing, as when you should start.
“There’s so much changing on a daily basis, not only in the world around us, but also in how people are reacting and managing the situation on a personal and professional level. If you’re not talking to your customers as you shift in this new environment, you will fail,” Estes said.
The better way forward, experts say, is simply to do it. Learn a little. Then do it better.
