How We Use Remote UX Testing Software - Brightwork Research & Analysis

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

Remote UX testing has many software tools that one can use to record video and store UX test data.
We cover how we accomplish this.

Introduction

There are many options for recording UX data. It can be confusing as to the right tool to use.

Our References for This Article

See this link if you want to see our references for this article and other related Brightwork articles at this link.

Specialized UX Testing Software

There are many options in choosing UX remote testing and recording software.

Loop11 is one of these. These tools or “platforms” (a term we disagree with) are in the cloud, and they are necessary to run unmoderated or automated testing. These tools require the researcher to follow their workflow, which we find limiting.

Some UX testing tools like Lookback make a big deal on their websites of seeing the user’s face (and for them to see ours) during the session. Here is the first question that came to mind when watching this video. Why would we want to do this? The primary value that comes from the shared session is the shared screen and the voice interaction.

Notice the video thumbnail in the lower left. This man has the camera angled up — and it looks like he is wearing his ceiling fan as a hat. People do this because their background is messy or otherwise embarrassing. However, it makes the image look like the view a dentist might find familiar.

We have been using Zoom for our user testing, and when we do, we disable video sharing. This has the following benefits.

Benefit #1: Lower Bandwidth Consumption

It reduces the bandwidth that the call consumes and leaves more for the capture of the screen. This is important when the WiFi signal is not as strong as one would like.

Benefit #2: Less Distraction

Seeing a face is distracting unless you are just having a conversation and not attempting to perform a task. We question how much including video into a call adds to a conversation, as the camera does not replicate a face-to-face conversation. On the video calls that we have, it seems like the experience of seeing the person’s face wears off after a few minutes. The experience is too intense. Normal human interaction does not consist of staring into each other’s eyes intently for the whole conversation.

As UI testing is about performing a task, having a user switch between viewing our face to working on the screen only serves to reduce the reality of the test as in real life, when a user accesses the website or app, they don’t interact with our face in a small square on the screen.

Benefit #3: Reducing Superficial Prep

Not using the video capability of the conferencing software also means less inadequate preparation for the call. This is because neither the test subject nor we have to worry about what we are wearing, their backgrounds, or focus on anything but the screen and voices.

From our perspective, the user could do the test naked in a bathtub. As long as they are comfortable and the WiFi signal is strong, they can see and control the screen.

Preparation for the UX testing is not something we are diminishing, and this is quite the contrary, very important. However, preparing one’s appearance — or surroundings, either for the test subject or the moderator, is not a good use of time.

Benefit #4: Enhanced Flexibility

It is possible to perform our UX testing from a phone. This may be surprising, but most of our involvement is our voice guidance and listening to the test subject. The call is recorded and then transcribed (using a tool like Otter). So we don’t have to take notes by hand during the session and can place notes into the call orally stating the note or observation.

How Well Does Moderation Work from a Phone?

By turning the phone sideways, we can get a good view of the screen. See the following screenshot.

This is our view on our phone of the Amazon website and not maximized. Surprisingly we can even see the screen when it is held in lengthways. And, of course, if the user is testing an app, they are using a phone. Also, their screen is the same size as ours.

This was surprising to us when we first started performing UI testing this way. Of course, recall that we will moderate tens or even hundreds of UI tests, so we know the screen very well by the time we begin the testing.

The Surprising Effectiveness of UI Testing From a Phone

If you had asked us before trying it, we would not have believed there would be enough screen real estate. But it works well. And without using the video, we can reduce the bandwidth so we can participate in UI tests using just a cell signal, which further increases our flexibility.

This is just another example of how the phone continues to infringe on what used to be done exclusively on computers.

Zoom for Mobile UX Testing Moderation

Zoom’s mainstay UX testing application does not do mobile conferencing where the moderator is the host. This is because the mobile app for Zoom has most of its functionality stripped out from it. The screen you viewed above was taken using Google Meet. We have not extensively tested the various offerings to see which is best at mobile, so we can’t say which offerers are good at this and which are not.

The Downsides of Using a Phone for Moderated UX Testing

There are a few downsides to using a phone for moderated UX testing. One is that many of the tools don’t allow recording from the phone — this is true of both Zoom and Google Workspaces.

A second issue is that the recorded video is of lower quality as it is recording not the screen of the user but the moderator. Of course, if it is a phone app being tested, the video size is the same and the quality is not higher recording from a computer screen as the session host.

Recording The UX Test from a Computer

Specialized UI testing tools have a built-in screen and audio recording, making the generic screen/videoconferencing applications like Zoom.

Each of these offerings promotes this recording functionality. I tested several well-known applications, and I found the video quality to be either naturally lower or lower than I normally record when I use a specialized screen recording application.

There are several excellent quality video and audio screen recording applications for the PC, Mac, and Linux OSs.

The advantage of stand-alone video and audio recording apps with the advantage of having their own editors. This allows callouts and other additions to be made when preparing the video for presentation to larger groups.

Storage for Session Video

Many UX testing tools emphasize how they provide storage, but storage is not a big deal and is quite generic at this point. These vendors probably buy their storage from the same providers could, like an AWS bucket.

We use Google Docs for our storage, but Dropbox works equally well, and we have been very impressed with the storage offering from Digital Ocean, but have not gotten around to using that for our primary video storage. See sample.

Recording the Data of the UX Testing

Recording the testing session is, of course, the capture step of the process. What follows is the creation of the data from the session.

This is done automatically in the UX testing tools, but not to the specificity that we want.

This is a sample step timing measuring sheet.

Any spreadsheet software can be used to record timings from the video session.

The comments can follow the same table format.

These two sheets are duplicated for as many observers as are part of the test. They review the video and provide their feedback on the sheet. Typically we just create a new tab for each observer and call the tab the name of the person. So “Tom – Comments”, “Marie Comments” etc.. Everyone works from the same collaborative workbook.

Any number of statistics can be reported off of the raw data. Here we hone in on areas that are taking longer than average than we think they should complete.

This shows the total time per user in a pie chart.

Documentation and Final Conclusions

So far we have discussed the accumulation of the raw data. However, that is not the end of the process. In step 4, which is explained in our article The Brightwork Remote UX Research Approach we write up the analysis in a long form document and then produce a synopsis in a presentation form.

The long-form document is more for us and allows us to think through the test observations.
The presentation is for the developers and executives.

Conclusion

Comprehensive solutions for remote UX testing recording are a bit like the claims made by ERP vendors. ERP vendors promise a single solution, but then the individual functionalities are normally mediocre. The reason is that it is difficult and unlikely for a “single solution” to do any of the individual items the best. And as with all cloud applications, they are designed to lock you into their application. If you want to move to a different application, there is a data migration step. In most cases, you will just end up keeping the old subscription until the project is over, and then migrate to the new tool for a new project, and then cancel the subscription when the old project is over.

Rather than using a single specialized and comprehensive tool we have been more comfortable combining more narrow applications to make a total UX test recording system than with one of the established specialized UX testing providers. But it is important to note that this is not the norm. Most companies that do this type of testing tend to prefer an application like Look11 or UserZoom.