Reflection for Reading Seminar 2 - Evaluation [Pavithren]

Chapter 13 introduces us to the different ways designers can evaluate their design. It also emphasizes the importance of conducting evaluation through the Why, What, Where, and When of evaluation. Evaluation allows us to collect information on how the end user will use our product. We can conduct studies in a controlled or uncontrolled environment. This could depend on whether we want to test a hypothesis and perform statistical analysis or determine how the product is going to be used.  With the emergence of big data, now we can collect massive amounts of data and perform data analytics to aid us in the design process. For example, in android smartphones or software on computers, prompts may be shown to the user asking about collecting anonymous data for improving the software. Some of the data collected could be tracking crashes, analyzing user behavior etc. One importance aspect of conducting evaluation is to respect user privacy and confidentiality. When researchers might have to obtain a CITI certification and permission from universities’ institutional review board(IRB) prior to conducting the study.

Chapter 14 delves into the mechanics of evaluation studies such as laboratory studies and field studies. For example, a researcher might want to evaluate how kids learn using a computer or a tablet. To achieve this, they might observe their touch patterns or gaze using eye tracking equipment. The data collected from these techniques could then be used to further improve the way the kids are taught using a computer/tablet. Tools such as IBM SPSS helps researchers analyze data and reach conclusions much easily. Another interesting thing highlighted in this chapter is the possibility of discovering surprising findings during a field study. During a controlled study, there may be certain restrictions on the user’s behavior or interaction with the product, but when maybe its tested in the WILD, researchers might come across interesting findings they wouldn’t have anticipated.

Chapter 15 introduces heuristic evaluation which is based on a set of guidelines/checklist created by some researchers to test UI and so on conform to well-known principles. This is a much cheaper and quicker option as compared to hiring users to testing the product. Walkthroughs are another method of conducting testing without the involvement of users. It involves going through a task in the product and identifying problematic features. Data collected from software or websites such as user behaviors and errors encountered could be useful in improving future versions of the software and could also be used for customizing the product per user’s preferences.

All in all, evaluation is an important aspect of user-centric design which helps to improve user experience.

Questions

How do I identify users who might not be acting normally during a field study after they realize that the researchers are observing them? The Dillema: If I inform the user that he is participating, he might not behave naturally. In the other hand if I record data without informing the user, its an ethical issue as I have not gotten their consent

With the increased usage of Big Data to perform analytics, will researchers be missing out as they are merely interpreting data and there might be misinterpretation of data?

Given our time constraint, can we look at previous research work done in this field and support our design ideas using them?