What We Are Going to Learn

• Human Computer Interaction
• Techniques for evaluation
  • Controlled experiments
  • Interviews
  • Surveys
  • Case studies
• Usability studies
  • Running the usability study
  • Choosing tasks
  • Prioritization
  • Likert scales

Evaluation

• Controlled experiments
• Natural settings
• Any setting not involving users (expert reviews)

Expert Reviews

• Design experts
• Visualization experts
• Usability experts
• Domain experts

Types of Expert Reviews

• Heuristic evaluation (golden rules)
• Guidelines review
• Consistency inspection
• Cognitive walkthrough
• Metaphors of human thinking
• Formal usability inspection (courtroom style)
• Accesibility inspection

Eight Golden Rules of Design

• Strive for consistency
• Cater for universal usability
• Offer informative feedback
• Design dialogs to yield closure
• Prevent errors
• Permit easy reversal of actions
• Support internal locus of control
• Reduce short-term memory load

Controlled Experiments

• Experiments in the lab
• Controlled confounding variables
• Measure one or more quantitative variables
  • Usability testing
  • Living labs

What to Measure?

• Time to learn
• Speed of performance
• Rate of errors
• Retention over time
• Subject satisfaction

Variables

• Independent variables (causes)
• Dependent variables (effect)
• Extraneous variables (that can affect the experiment)

Controlled Experiment example

Usability Testing

Usability Testing

Natural Settings Involving Users

• Observation
• Interviews
• Logging

Triangulation

Different researchers observe the same effect.

Interviews

• Unstructured
• Structured
• Semi-structured
• Focus group
• Telephone/online interviews

Questionnaire

Like interviews but without the researcher present

Likert Scales

Likert Scale

What do you think?

• Strongly disagree
• Disagree
• Okay
• Agree
• Strongly agree

More About Likert Scales

• Can be 3, 5, 7, or more responses
• Continuous or discrete
• Middle response is the balance

Likert Scales d3

Likert Scales Vega-Lite

Likert Scales Vega-L ite (cont.)

Other Methods

Observation

• User's setting
• Can be direct or indirect

Direct Observation in the Field

Ethnography

Direct Observation in Controlled Environments

• Think aloud techniques

Direct Observation: Tracking Users

• Diaries
• Interaction logs and web analytics

MILCS

• Multi-dimensional
• In-depth
• Long-term
• Case studies

TreeVersity MILCS

• Thirteen different case studies with nine agencies
• TreeVersity 2: UMD Budget 2010 - 2012

Focus groups

One researcher, many attendees

Prototyping

• Low vs. high fidelity?
• Read data
• Build scenarios, tell a story

Quatitative evaluation

Running a Usability Study

Validity Checks

• Earlier stages:
  • Observe and interview target users (needs assessment)
  • Design data abstraction/operation (data types, transformation, operations)
  • Justify encoding/interaction design (design heuristics, perception research)
  • Informal analysis/qualitative analysis of prototypes (task-based)
  • Algorithm complexity analysis/evaluation
• Mid- and later stages:
  • Qualitative analysis of system (task-based)
  • Algorithm performance analysis
  • Lab or crowdsourced user study
  • Field study of the deployed system

Formal Usability Study

Goal: Does the visualization allow the user/analyst to perform key tasks?

Task-Oriented Visual Insights

• Basic insights:
  • Read a value
  • Identify extrema
  • Characterize distribution
  • Describe correlation
• Comparative insights:
  • Compare values
  • Compare extrema
  • Compare distribution
  • Compare correlation

Usability Study: Logistics

• You will need:
  • Visualization with test data loaded
  • Consent form (if required)
  • Task list
  • Protocol (study procedures and debrief questions)
  • Surveys/interviews and any additional data-collection instruments
  • Audio or video recorder, notepad

How Many People Do You Need?

"Lab" Doesn’t Need to Mean a Formal Lab

Software for Collecting Audio/Video

• Video of user
• Screencapture of user actions
• Audio of entire session

Online Tools

• Surveys
• Mouse tracking/navigation tracking

Prioritization

You’ve Collected Data

• Task completion
• Time on task
• Notes
• Interview responses
• Survey responses
• ...Then what?

Table source: Stasko, J., Catrambone, R., Guzdial, M., & McDonald, K. (2000). An evaluation of space-filling information visualizations for depicting hierarchical structures.

What is the Analyst’s Information Scent?

MoSCoW Prioritization

• Must
• Should
• Could
• Won't

Severity Ratings

1. Not a real problem
2. Cosmetic
3. Minor usability issue
4. Major usability issue
5. Critical issue

Limitations

1. Ecological validity
2. Are performance-oriented tasks the complete story?

Usability Study Demo

References

• Shneiderman, B. and Plaisant, C., 1987. Designing the user interface: Strategies for effective human-computer interaction
• Rogers, Y., Sharp, H., Preece, J. and Tepper, M., 2007. Interaction design: beyond human-computer interaction.
• Martin, D.W., 2007. Doing psychology experiments. Cengage Learning.

What We Learned

• Human Computer Interaction
• Techniques for evaluation
  • Controlled experiments
  • Interviews
  • Surveys
  • Case studies
• Usability studies
  • Running the usability study
  • Choosing tasks
  • Prioritization
  • Likert scales