web analytics at UR BOF

Yesterday evening was the 7th meeting of CHI Netherlands User Research BOF at the Creative Learning Lab in Amsterdam, with two presentations on web analytics.

Adam Cox en Martijn Klompenhouwer of User Intelligence shared some interesting examples and lessons learned of combining web analytics and user research.

While web analytics can provide a lot of data on where users are coming from (e.g. referrals, search engines, etc), what they are doing on a site, and where & when they leave a site, they do lack an accurate interpretation of the reasons behind these actions (i.e. "why").

User research on the other hand provides good insights and qualitative data, but often lacks sufficient quantity of data to easily verify results, and is less suitable for capturing users' behavior over longer periods of time or catch infrequent user behavior.

Both approaches complement each other very well, and can be very powerful in combination, for example by using statistically significant findings of web analytics to support and confirm hunches gained through user research, as well as applying user research to further investigate the rational for specific issues uncovered through web analytics' continuous monitoring.

Adam and Martijn shared several examples where they had successfully combined web analytics and user research. For example one customer had in the past evaluated their site by starting from the homepage or internal search engine. However, web analytics showed that most visitors of the site actually arrived at the detail pages through external search engines. Thus usability test scripts were adjusted accordingly, which emphasized several issues that usability tests had previously not encountered.

In another case, an expert review indicated that the homepage was confusing to users, because it offered too many options. Web analytics showed that 90% of all visitors continued on to the same secondary page. Thus the other options could be removed or relocated to less prominent positions.

Another customer discovered through web analytics that their sales funnel had a disproportionately high rate of customers abandoning the shopping process at a specific step. During usability testing, particular attention was given to this page and the user's reaction. Whereas during the test scenarios, users continued through checkout, further in depth questioning revealed why they would otherwise have abandoned their purchase at this point.

Web analytics can also be useful to define the target audience for user research, for example based on location (local, national, international), time of day (home vs. office users), screen resolution, operating systems, etc.

For further information, Adam and Martijn recommended Web Analytics Demystified by Eric Peterson and Web Analytics: An Hour A Day by Avinash Kaushik, as well as Web Analytics Wednesdays and upcoming seminar by WAA Netherlands.

Eelco van Kuik of Conversion Company calls himself a 'hardcore web analyst' and has so far mostly been oriented toward online marketing and online channel management. Eelco provided a short introduction to web analytics 2.0, according to Kaushik's model, with different methods of increasing depth and complexity:

1. clickstream analysis; most widely know approach using tools such as Google analytics
2. multiple outcome analysis; comparing data from different measurement systems or tools to validate results and establish common trends
3. online experiments; A/B testing to compare effectiveness of different design or marketing propositions.
4. targeting; e.g. multivariate testing example presented by Martin McNulty, where significantly different test results occurred at different times of the weeks due to varying user groups, and thus the decision to display different designs at different times of the week.
5. voice of customer; adding some detail through online surveys and video analysis (i.e. tracking user's mouse movements and selections on a page)
6. competitive intelligence; comparative data and tests through 3rd parties such as Alexa

Eelco pointed out the importance of focusing on a particular area or priority initially, and extending the analysis incrementally rather than taking on too much in one go. First the basics need to be well established and set up correct, before moving on to more advanced approaches. Also the total cost of ownership has to be considered, including ongoing maintenance of tags etc and setting up a process for ongoing evaluation and taking according actions based on the accumulated data .

The presentation raised a lot of ethical questions and heated discussions amongst the audience. As Eelco pointed out, it is important to pay close attention to the process and regulations regarding where and by whom information is captured, reviewed and stored (e.g. internally or via external facilities, in which country etc.). It seems advisable, particularly for organizations handling sensitive information, to consult with experienced Web analytics experts aware of these issues and implications.

Personally, I don't believe self regulation of the industry can be trusted to prevent any unethical business practices and abuse of data. Certainly considering (video) analysis of unwitting (and possibly unwilling) test participants, and the enormous amount of data accumulated by Google through their analytics tool, I cannot shake a distinct paranoid feeling, and consider blocking any such clandestine activities from happening in my browser. It was reassuring, though, to hear that there are some (publicly funded) efforts to review sites concerning appropriate use of web analytics systems.

However, I am very intrigued by the opportunities offered through web analytics to validate, and refine usability research and thus improve the user experience. Definitely worth while investigating further!

For more impressions see also Tess Doucet' blog.