Norman Nielsen Usability week in London – Research beyond user testing

Christian Rohrer that holds an impressive CV with experience from leading user experience research teams at Yahoo and eBay, and a current position at Mcfee, talked during the Norman Nielsen Usability week about methods that complement usability studies.

Field studies
During a full-day training course, a lot of emphasis was put on field studies.  Christian stressed, “Understanding the real product and user experience is not possible in a lab”. As an example, Christian shared a story from eBay’s field studies where it was revealed that uploading a photo on eBay caused an error for 1 out of 9 observed users. One might think that this is not a high number, but after consulting the web analytics team of eBay it was discovered through analysis of web statistics that 2600 unique users had this error every day!

The above example well illustrates, how two methods complement each other. However Christian also warns for giving your listeners numbers (e.g. 1 out of 9) from your qualitative studies since the listeners will start to expect numbers and take it for the only truth. I wonder what would have happened in this case if the web statistics were not there to support the findings from the field studies? Field studies are not about numbers, but about describing and understanding the users and the user needs, and about finding the gap between what the users say they do and what users actually do (this is where many business opportunities lie!).

Group exercise –the dining experience
During lunchtime we practiced field studies in the hotel restaurant, our mission was to dig into the dining experience. We practiced both interviews and observations, and the use of note guidelines. Afterwards we did the data analysis together, and finally we got to use Post-its notes (no UX conference without colourful Post-its, right?) by using Affinity diagram.

Affinity diagram
Affinity diagram is a method to sort data into logical groups, basically, you write down one finding from the field (and if it is a observation or quote) per Post-it. Finally, you together as group, agree on how to group and subgroup all notes.

During the analysing of the notes it was clear that all groups had a hard time not to jump to conclusions and solutions, and here the practiced note guideline came to help, since it made it easy to see if the note was an observation, a quote or an interpretation. I really like the way we used the affinity diagram and I also believe that it is a good tool to use when communicating field study findings to various stakeholders.

Tips for field studies
– Always bring along non UX-colleagues to field studies
– Create note-taking guidelines that help the team to differ between observation and interpretation.
– When users generalize, they make it look nicer, therefore start broad “What do you usually do during a working day” and then go into specific details.
– Important to have a post-session debrief together, we quickly forget information!
– Do the data analysis together as a group

The truth is not only in the numbers
To sum up the day I agree that the quantitative research can be a good complement to user studies. Yet one need to watch out for creating a company culture where the numbers are the only source of information relied upon when it comes to design decisions.

About

Mia Ajanki works as usability engineer at the communication department of Visma. She has previous experience of user-centered design from her role as project manager, developing a Manufacturing Execution System for factories. M.Sc. In Technical Design with specialization in HMI.
Connect with Mia: