Articles /

Screen Recording, knowing the opportunities and the limitations

Learn from others - but be careful on what you conclude

Among the suite of analytical tools we use, screen recording can provide some of the most effective qualitative feedback.

On paper, seeing (and making conclusions) from the actions of a few users can seem a little precarious - and certainly not statistically significant. But that really is not the point. The truth is that video can provide an empathic feeling for the user, something which no quantitative analytics can offer. And that is a good thing.

There are several types of screen recording, each with their own advantages and disadvantages. They are generally made up of a user being either on-premises or remote, being aware or unaware that they are being recorded, and finally tasked or untasked with an objective. The key takeaway is to understand the limitations of each one of these approaches, and to watch out for any false conclusions that may be drawn.

A great example of flawed thinking was highlighted in the Malcolm Gladwell book, Blink. This covered users taking the Pepsi Challenge which was a real world task-led study where consumers, when asked to taste both Pepsi and Coca Cola, declared by a significant margin their undying love for Pepsi.

The result? Coca Cola reformulated and lost millions. The flaw? Consumers did not really prefer Pepsi after all.

So how does this apply to interpreting your screen recordings? Coca Cola made a fundamental mistake of ignoring the environment these tasters were placed in. By assuming that their preferences constituted entirely of taste, they ignored other factors such as location, time of day, brand halo, thirst, etc.

We can apply the same logic flaws for each of the types of screen recordings, of which these are a few:

Task driven: If a user has been set a task to complete, their focus will not be representative of someone needing to achieve this task in the wild. They will be more driven, less distracted, and not exposed to environmental issues such as interruptions, connectivity issues and tangential thinking. Task specific testing is mostly useful when testing interfaces (e.g. booking tools) rather than general site usage.

Moderated testing: Sitting next to a user looking into the whites of their eyes (or cunningly hiding behind a one-way mirror) will make them behave differently. If you want proof, just try typing a sentence whilst someone is watching you - you will, almost certainly, misspell a word, or mishit a key! With this testing approach, look to gain insight from a qualitative perspective. Be focused on what you are trying to learn, and prepare a common question-set which you will engage the user with.

Anonymous user screen recording: These individuals will be visiting the site for multiple reasons and will vary significantly in demographic and capability. The key is to define the specifics on what you are trying to learn, to build up a set of hypotheses to inform these learnings, and then remove any noise (unrelated user journeys) which do not relate. Avoid drawing too many conclusions from individual recordings. Build up a set of results based on multiple recordings instead, and then combine the results further by using other user feedback methodologies.

Generally, make sure that your approach to screen recordings does not focus just on the user interface. Although valid, there is so much more to learn; including how the user feels, their capability to understand the messaging, and how they feel your site compares to the competition. If necessary, introduce additional design elements on the page to provide supplementary insights.

Once you have drawn conclusions from the recordings, do not simply implement knee-jerk changes. Validate your learnings through analytics (where possible), through user feedback sessions and, most importantly, split variant testing. This will help you make sure that you do not end up running in circles.

By Drew Griffiths

Managing Director

Share this

Say hello

For an informal chat about your digital requirements, call us on 020 8050 0007, email hello@coherence.co.uk