Valid research is essential for successful product development, but it can easily become tainted in a variety of ways. The Hawthorne effect is a well-known and controversial concept that has implications for user research. This post will describe the Hawthorne effect, the issues it may cause during user research, and suggestions for mitigating the phenomena.
What is the Hawthorne Effect?
In the 1920’s and 30’s Elton Mayo ran a group of studies exploring how the work environment influenced productivity at the Western Electric factory located in Hawthorne Illinois. The series of studies found that no matter how the work environment was altered, including lighting changes, working hours, and breaks, productivity improved. Even when the lights were dimmed, workers were more productive.
The researchers believed that allowing workers to discuss changes to their environment and employees feeling that the management cared about the working conditions led to higher productivity. In other words the results implied that the workers were being more productive because they knew that the researchers were present.
There have been numerous follow-up studies about the Hawthorne effect and many have produced conflicting conclusions. Regardless of its controversial nature, the Hawthorne effect has important implications for user experience research.
Why does it matter?
Usability tests and user research are only valid when conducted with proper experimental designs. If a user changes his or her behavior during testing, demonstrating the Hawthorne effect, the results will not be realistic. Those faulty findings may be used to drive design decisions leading to a product that tries to solve the wrong problems.
Avoiding the Hawthorne effect
Since research plays a big role in successful product development it is important to mitigate the Hawthorne effect. There are a few simple ways to minimize the phenomena.
Telling participants that they will be taking part of a ‘usability session’ or ‘usability review’ can change their expectations. If participants feel calm and are not pressed to meet a certain goal there is less of a chance they will demonstrate the Hawthorne effect.
Another strategy is to keep the expected results of the study hidden from the user. This will reduce the chance that a user will feel pressured to perform in a certain way to align with the tester’s expectations.
Finally, using analytics to find a baseline measurement for a task before running in-person usability tests can help researchers determine how much the testing environment is influencing the results of the study. For example, if a researcher wanted to study form completions on a specific webpage he or she could use web analytics to find the average time current users take to complete a similar task. This would serve as a baseline rate. During in-person usability testing the researcher could have participants complete the same task used to calculate the baseline rate. This would allow the researcher to find the variance between in-person testing performance and the online users’ baseline rate. That variance would be considered while analyzing the rest of the usability test results to confirm that the in-person study results are not overtly influenced by the Hawthorne effect.
Conducting user experience research requires a commitment to mitigating research design flaws. With a few simple actions, the validity of a study can be improved by decreasing the Hawthorne effect.
More to Read: