The Brightwork Remote UX Research Approach - Brightwork Research & Analysis

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

Remote UX research is our preferred approach.
In this article, we will cover the benefits of doing remote UX research.

Introduction

UX research is often presented as same location-based to get the most feedback from the users. We challenge that assertion in this article and discuss our approach for remote UX research.

Our References for This Article

See this link if you want to see our references for this article and other related Brightwork articles at this link.

The Common Impression of Remote UX Research

This impression is well explained in the following quotation.

Too often, remote user research is considered “settle for it” research. It’s research when the team lacks the budget to travel, the time to be in the field, or the resources to bring users into the lab. – Dscout

Dscout proposes the following way to look at remote UX research.

Believe it or not, when done right, remote research can actually get you better data—and more often than you’d expect, it should strategically be your first choice.

Now Dscout is not a neutral source on this topic, as they make a remote UX research platform. However, we have found the same thing as Dscout, and we don’t use Dscout as our UX research platform. We are also not associated with Dscout and have never spoken or otherwise corresponded with them.

There are several remote UX testing platforms or tools, and we have tested a number of them and used several in our own UX testing, but we can’t recommend one over the other as they tend to do different things a little better than other options in the market. For example, some UX testing software is designed for unmoderated UX testing (which is generally much more experience), (unmoderated UX testing is not something we do, but many companies do). (We cover the topic of moderated versus unmoderated UX testing in the article Moderated Versus Unmoderated Remote UX Research Testing.

Secondly, the tool used depends upon the research question that is required to be asked. The software vendors tend to present their solution as being the single solution for all UX testing, but we have yet to encounter such a tool.

In a future article, we may delve into a comparison of these tools.

Remote UX Research and Covid

It is no secret that the covid shutdowns greatly increase the acceptance of remote work of all types. Curiously, in the consulting area, even after the tools for remote interaction evolved (such as videoconferencing, video calls, screen sharing, Google Sheets, etc) the need for consultants to be one site to bill hours caused people to have to degrade the environment by flying on a weekly basis to their client sites. The one positive aspect of covid was it forced the issue, and once forced, companies and workers alike realized that much of the effort involved in moving people around to do non-physical work was unnecessary and wasteful in terms of time and resources.

The change in the work and personal culture is explained in the following quotation.

Although most of us have been shopping online for years, we’re now shopping for everything online. All schools have had to adapt to some clumsy form of online education. Live performances have transitioned to streaming online. Doctors have had to embrace telemedicine. Everyone is now using online-meeting software to communicate with friends and family. With all of these formerly in-person activities becoming online activities, it’s more important than ever to conduct user research to improve the user experience of such services. – UXMatters

However, the benefits of remote UX research were apparent to us before the covid shutdowns. This applies to how we manage Brightwork as a company as well. For instance, we can’t imaging limiting employees or contractors to just whoever is in a particular geography.

The Benefits of Remote UX Research

If we could use one word to describe the benefits of remote UX research it would be “flexibility.” Remote UX research allows the following.

To reach anyone in the world and at a time convenient for them.
To broaden out the user test subjects.
To record sessions as part of the videoconferencing application’s standard functionality.
To access users, not at the main headquarters, which often means being able to broaden the net of user feedback.
If the user subject is not within the company (which is the case for most web or app testing) and which means the user subject must be paid, the cost of user testing is much lower when performed remotely.

The benefits of remote UX research are so great, and the costs and other inefficiencies of on-site UX research are so high, I have wondered if the reason for emphasizing on-site UX research by so many UX researchers and UX researching firms has been to simply create an event or a type of performative experience for the client.

Furthermore, all of these benefits must be balanced against the costs of on-site UX research, which includes the following:

The time taken from the test subject’s scheduled to travel to a specific location.
The time of the travel to the on-site location (either by the UX tester and or the people not based at the on-site location).
The expense of the travel, accommodations etc..

This very obvious advantage is expounded upon in the following quotation.

Since a typical remote research session requires only about an hour of each participant’s time, it’s much easier for people to participate in remote research. Joining an online meeting for an hour is much less effort than taking half a day off to travel to a research facility, participate in a session, then travel back. – UXMatters

All of these issues and even more limitations and costs of on-site UI research have undermined testing since UI research first began to be a thing companies did.

Expanding the Test Subject Group

Unless the company is small, on-site UX testing has historically meant shrinking the test group to what was cost-effective.

This is explained by the following quotation.

Because remote sessions are much easier and less time consuming for participants, it can be easier, quicker, and less expensive to recruit participants for remote UX research. It’s also easier to recruit people who would be too busy to participate in in-person sessions. Doctors, lawyers, and other highly paid professionals are often unable or unwilling to take several hours off to participate in in-person sessions. It’s much easier for them to join an hour-long remote session. Because participating in remote UX research is much easier, you’re likely to recruit more representative participants than you would get if you included only the usual suspects—those professional research participants who are always willing to come in for an in-person session. – UXMatters

While researching and writing on the topic it immediately occurred to me that this broadens out far more than just UI research.

For example, at Brightwork Research & Analysis, we have researched many software topics. One of these has been software implementations. In the area of software implementations, it has not only been the UX testing that is normally headquarters centric but the business logic of applications that has had this strong headquarters bias. I was a consultant on one project where the IT department at headquarters, which was in the UK, lied to the division based in the US about what the software could do so that the US division would choose the configuration selected by the UK IT department. I knew the software worked in more ways than the UK let on, but the US division would have never found out if I had not told them. I was then pressured by the UK IT department to keep the US division in the dark, as they did not need to know.

As I covered in Global Versus Single Software Instances for SAP, this results in systems that meet the needs of headquarters but are often a poor fit for the other areas of the company. This has been the bugaboo of system implementation and has retarded implementation uptake for as long as systems have been implemented by companies.

Eliminating the Lab Environment for UX Testing

One of the biggest oversights in research is performing testing in a lab environment, which is necessary in many cases for controlling intruding factors, but then assuming that the lab environment applies to the real world. For UX testing, the closer the user subject can be kept to their natural environment the better. Users don’t work in a teaching environment or classroom environment. They are typically going doing tasks on the computer alone (they may be in a cubicle next to many people, but are still normally solitary when using the computer). The remote environment, where we are just watching and interacting with them by phone keeps the test subject in a more natural environment. We keep a congenial interaction style so that the pressure of judgment does not impact the performance of the test subject. It is extremely important that the test subject does not think they are being scored or ranked against something which might increase their anxiety and reduce performance — altering it from how they would perform under normal circumstances.

These benefits are explained in the following quotation.

For those of us who conduct UX research regularly, it’s easy to forget how strange and unnatural it can feel to participate in UX research. Sitting in a usability lab or market-research facility, in the midst of cameras and microphones, with an intimidating one-way mirror, and having a UX researcher question and observe you while taking notes can feel awkward and intimidating. Similarly, even in your natural environment, having a few researchers visit you in your home or workplace to interview and observe you while recording video of what you’re doing can also feel very uncomfortable and unnatural. In contrast, participants are often more comfortable and less intimidated by remote UX research. They remain in the familiar environment of their home or workplace, using their own devices, being observed only via a single Webcam, at a distance from the researcher. Remote sessions are more realistic than taking people out of their natural surroundings to the artificial environment of a lab or market-research facility. – UXMatters

What the Future of Remote Testing and Remote Work Holds

In our view, companies are still in the early stages of understanding how to really leverage all of the great collaborative remote tools that are at our disposal. Covid was the wake-up call, but there is still much more transformation and adaptation to thinking processes that are still to come.

Secondly, every year that passes the collaborative tools get better, while the physical transportation infrastructure gets more overwhelmed. This continually puts the ball in the court of remote work.

The Brightwork Remote UX Approach

We are very focused on the “lower common denominator” UX testing. And remote UX testing is perfect for this.

We won’t lay out everything in this article but here are a few characteristics of our remote UX approach.

Approach Step #1: Record all Sessions: Focus on Analysis After the Session

We focus on teasing out information and commentary during the moderated session from the user rather than focusing on what they are doing. (We don’t do unmoderated sessions for UX testing)

There is plenty of time for doing the observing and measuring the activities when we rewatch the session later. But interacting with the user allows for the next part of our approach.

NNg provides a good explanation of how to perform moderated testing.

Approach Step #2: Write Down the User Commentary into the UI Test Sheet Field

Every step in the pathway has a field for the user commentary on that step.

Not every step necessarily will have user commentary, but there is an available field for the commentary. By placing tags on the commentary, we can then report on the percentage of users that found the step “Frustrating” or “Confusing” or “Unclear” and so on.

After many users have been recorded, we can observe patterns in the comments per field.

Approach Step #3: Recording the Timing of Different Pathways From the Video Recording

We will record all of the durations of each activity for each user into a Google Sheet. We then average the timings, and take the high and low points in elapsed time and use this in our report.

Approach Step #4: Integration and Documenting the Observations

It is important not to comingle or confuse the data recording part of the process with the next step, which is gleaning and documenting insights from the test videos. The ability to pause and rewind video files makes the viewing of the videos (rather than when moderating the session) the best time to perform analysis. Because of our long history of research, this is our favorite part of the process, because it is where we get to figure out the UI issues and synthesize the observations of others into a coherent whole.

In this step, we write up the analysis in a long-form document and then produce a synopsis in a presentation form.

The long-form document is more for us and allows us to think through the test observations.
The presentation is for the developers and executives.

These are the broad outlines, there is more to it, of course, but it suffices to say, we have never felt that we could not get something in a remote recorded session that we would have gotten by being in the same room with a test subject. And when all of the benefits of remote UI testing is added together, for us the remote UI testing was the clear winner.

The Influence of Don Norman and Jakob Nielsen’s NNg on Our Approach

NNg does not specify the exact tools to use — but as a research basis and philosophy, we are very much consistent with the material published at NNg. When we started doing UX research, we read as much as we could, and continually found NNg to have the most research-oriented information on UX design and research. There are really not that many true or pure UX research companies out there — and NNg is one of the very few that produced original research, rather than just selling UX testing services to companies.

Conclusion

We can provide a better value to our clients with remote UX testing than with on-site UX testing.