Reading Seminar 2 Notes: Agneev Guin

Chapter 10: Introducing Evaluation

The example on the HutchWorld Usability Study clearly states that any number of usability tests cannot give a real scenario perspective. No system is ever perfect. The integration of the HutchWorld with the real clinical environment gave some insights on the natural setting. Also, one important aspect is "if users don't like a system, it doesn't matter how successful the usability testing is: they probably won't use it."

Chapter 11: An Evaluation Framework

The evaluation needs to address all aspects and can be captured as a data in the design process. Thus the evaluations are classified as Objective and Subjective, Quantitative and Qualitative and Laboratory and field or naturalistic studies.

The DECIDE framework focuses on the goals and applies the logical and practical implementation which are to be dealt while planning an evaluation. The acronym DECIDE expands to the steps carried in the process.
1. Determine the overall goals that the evaluation addresses.
2. Explore the specific questions to be answered.
3. Choose the evaluation paradigm and techniques to answer the questions.
4. Identify the practical issues that must be addressed, such as selecting participants.
5. Decide how to deal with the ethical issues.
6. Evaluate, interpret, and present the data.

The focus of the framework on the practical issues constitutes of users, facilities, schedule, budget and expertise while the ethical issues focuses on the code of ethics, informed consent and deceptive experiment. The reliability, validity, bias and scope can only be observed using the data captured.

Question:
Is it necessary to tell the users that they are being logged? Should double blinding be a general aspect of evaluation?
What if the ethical aspects cannot be met?