Seminar Two: Reading Reflection by Hui Pang

Chapter 13: Introducing Evaluation

Evaluation focuses on both the usability of the system and on the users’ experience when interacting with it. It also enables us to check that the design is appropriate and acceptable for the wider user population. It can either involve participants or not. Some of the same data gathering methods are also used in evaluation. Which evaluation methods to use depend on the goals. 

Formative evaluations are done during design to check that a product continues to meet users’ needs, while summative evaluations are done to assess the success of a finished product. 

There are 3 broad categories of evaluations. These are controlled settings involving users(e.g. usability testing and experiments), natural settings involving users(e.g. field studies), any settings not involving users(e.g. inspections, heuristics, walkthroughs, models, and analytics). A key concern for deciding is how much control is needed. Also combinations of methods are used to obtain a richer understanding. 

Evaluation can also be opportunistic, done early in the design process to obtain feedback quickly and confirms whether it is worth proceeding to develop an idea into a prototype. They are informal and do not require many resources.  

It is important to ask for participants' informed consent and not to over-generalize findings from an evaluation. 

Chapter 15: Inspections, Analytics, and Models

Evaluation can also be done without users to be present. Instead, experts, who are knowledgeable about both interaction design and needs and typical behaviour of users, will provide feedback. 

Inspection methods typically involve an expert role-playing the users for whom the product is designed, analyzing aspects of an interface, and identifying any potential usability problems by using a set of guidelines. 

The most well known are heuristic evaluation and walkthroughs. In heuristic evaluation, experts, guided by a set of usability high-level design principles known as heuristics, evaluate whether user-interface elements conform to tried and tested principles. Walkthroughs involve walking through a task with the product and noting problematic usability features. 

Analytics involves user interaction logging, which is usually done remotely. 

Predictive models involve analyzing various physical and mental operations needed to perform certain tasks and operationalizing them as quantitive measures. 

One of the most commonly used is Fitt’s Law. It predicts the time it takes to point at a target. 

Questions: 

1. Which evaluation method will better serve our design purpose? 
2. How many evaluators are needed to carry out a thorough heuristic evaluation?