Seminar 2 -- Mengdi

Chapter 13 An Evaluation Framework

Evaluation is conducted to promote user experience by negotiating between designers and users. But real world constraints play an important role on shaping the evaluations. First of all, goals need to be determined. By setting the goal properly, we will naturally know the scope of this evaluation.Then questions need to be explored carefully, by partition a general one into several fine-grained ones, which enables us gain the intrinsic reason of a certain phenomenon.Next, evaluation method is to be chosen, depending on what data is needed to answer the questions and which theories or frameworks are appropriate to the context.While conducting the evaluation, practical issues such as budgets, schedule, appropriate participants needs to be taken into account. Ethical issues cannot be ignored either. We should always remember respecting participants' privacy. Do not reveal their information while reporting unless they consent, do not feel trouble explaining the aim of this study etc.The last step is evaluating, analyzing and interpreting the data. Reliability, validity, ecological validity, biases and scope are the key words of preventing us from a bad evaluation.

Chapter 14
This chapter extends the topic of environment settings in last chapter. In usability testing, it mentions that time and number are important measurements of users' performance, such as  time to complete a task, number of errors a user made for per task.
Usability lab is a traditional way to conduct evaluation. The settings of the lab mimics the real environment where a certain product is to be used. Also, distractions are eliminated. But maintaining such a lab costs lots of money and labor. Then mobile usability testing equipment is a trend since mid-90s. Video cameras, laptops etc. are temporarily set up in an office. Another trend is conducting remote usability testing. Users interaction with a product is logged for evaluators to analysis.
The experiments are conducted to verify a set of hypothesis, which are proposed based on theory or previous research. Normally, the hypothesis is to find the influence of some factor to the usability. The factors are independent variables which can be manipulated by evaluators, the others are dependent variables. Those irrelevant variables will are set to be a constant to avoid potential influence on the experiment result. 
Participants are also a concern of experiment design. Same-participant design is letting all participants performing all the condition settings. This decrease individual difference to different settings, but participants may be trained by previous condition, which may effect the behavior in latter experiments. Designers should also pay attention to use counterbalancing to decrease the bias of order-effect, which means the order of condition may effect the performance of the participant. Different-participant design is allocate different condition to a participant randomly, each participant only experience one experiment. This design method has no train-effect or order-effect but requires a lot more participants than same-participant design. Matched-participant design pairs participants by their characteristics like gender, expertise. But matching may neglect some important variables.

Chapter 15 Analytical Evaluation

In this chapter, it mainly talks about two inspection methods, heuristic evaluation and walkthroughs, and a few predictive models such as GOMS and Fitt's Law.

Experts instead of users are involved in inspections. Heuristic evaluation is guided by a list of usability principles, such as visibility of system status, match between system and the real world etc.  But depending on different products, not all heuristics can be applied, normally 5~10 of which is enough.

In cognitive walkthroughs the designer and several experts conduct the evaluation together. The evaluators walk through the action sequences for each task, placing the context of a typical scenario and imagine will users know what to do, how to do, and understand from feedback whether the action was correct or not. Then evaluators and designer discuss together and revise the design.

In pluralistic walkthroughs, each expert pretend to be a typical user in a certain prototype screen, which focus on user's tasks at a detailed level.

GOMS, stands for goals, operators, methods, and selection rules. Goals refers to the particular state the user wants to achieve. Operators refer to the cognitive processes and physical actions that need to be performed in order to attain those goals. Methods are learned procedures for accomplishing the goals. Selection rules is to determine which solution to use if multiple solutions exist.
KLM provides numerical predictions of user performance so that different features of systems and applications can be easily compared to see which might be optimal for a specific kinds of task.
Fitt's Law is used to help designers decide where to locate buttons, what size they should be, and how close together they should be on a screen display.

Question:

How to ensure ecological validity?

In pluralistic walkthroughs, how to choose typical users from a various range?
Is there any principle of choosing different participant-setting design?