Evaluation


John Alexis Guerra Gómez| ja.guerrag[at]uniandes.edu.co| @duto_guerra
Jose Tiberio Hernández | jhernand[at]uniandes.edu.co
Universidad de los Andes


http://johnguerra.co/lectures/visualAnalytics_fall2019/10_HCI_Evaluation/

Based onslides from Tamara Munzner

Evaluation

  • Controlled Experiments
  • Natural settings
  • Any setting not involving users

Expert reviews

  • Design Experts
  • Visualization Experts
  • Usability Experts
  • Domain Experts

Types of expert reviews

  • Heuristic evaluation (golden rules)
  • Guidelines review
  • Consistency inspection
  • Cognitive walkthrough
  • Metaphors of human thinking
  • Formal usability inspection (courtroom style)
  • Accesibility inspection

8 golden rules of design

  • Strive for consistency
  • Cater for universal usability
  • Offer informative feedback
  • Design dialogs to yield closure
  • Prevent errors
  • Permit easy reversal of actions
  • Support internal locus of control
  • Reduce short-term memory load

Controlled Experiments

  • Experiments in the lab
  • Controlled confounding variables
  • Measure one or more quantitative variables
    • Usability testing
    • Living labs

What to measure?

  • Time to learn
  • Speed of performance
  • Rate of errors
  • Retention over time
  • Subject satisfaction

Usability Testing

Brighton Uni Usability Lab

Natural settings involving users

  • Observation
  • Interviews
  • Logging

Triangulation

Different researchers observe the same effect

Interviews

  • Unstructured
  • Structured
  • Semi-structured
  • Focus group
  • Telephone/Online interviews

Questionnaire

Like interviews but without the researcher present

Likert Scale

What do you think?

  • Strongly disagree
  • disagree
  • ok
  • agree
  • Strongly agree

More about likert scales

  • Can be 3, 5, 7 or more responses
  • Continuos or Discrete
  • Middle response is the balance

Likert Scales D3

https://blockbuilder.org/john-guerra/57d31bae7685139cecd731bba9f48090

Likert Scales Vega-lite

https://vega.github.io/vega-lite/examples/layer_likert.html

Likert Scales Vega-lite

https://vega.github.io/vega-lite/examples/concat_layer_voyager_result.html

Observation

  • User's setting
  • Can be direct or indirect

Direct Observation in the field

Ethnography

Direct Observation in controlled environments

  • Think aloud techniques

Direct Observation: tracking users

  • Diaries
  • Interaction Logs and web analytics

MILCS

  • Multi-dimensional
  • In-depth
  • Long-term
  • Case studies

TreeVersity MILCS

Focus groups

One researcher, many attendees

Prototyping

  • Low vs high fidelity?
  • Read data
  • Build scenarios, tell a story

Quatitative evaluation

http://yatani.jp/teaching/doku.php?id=hcistats:anova

References