Authors: Rene F. Kizilcec
CHI 2016, San Jose, CA, USA
What is the high-level research question.
How transparency in the design of algorothmic interfaces can affect users’ trust towards the system.
Specific Research Question and Hypothesis
- H1: Expectation violation reduces users trust towards the system.
- H2: Changes in interface transparency affect trust depending on whether expectations are violated.
- H3: If expectations are violated, procedural transparency increases trust,
but additional information about outcomes erodes this trust.
Key Findings
- In the low and high transparency conditions, expectation violation was negatively correlated with
trust
- Trust was uncorrelated with expectation violation in the medium transparency condition
Data Collection Methodologies
- Study was conducted on 103 participants enrolled in a MOOC on Coursera.
- Participants submitted an essay and peer reviewed the essays sbmitted by others.
- Once a learner and her peers had graded her essay, she would receive her combined and adjusted
peer grade accompanied by different amounts of information about the grading process depending
on the transparency condition.
- Transparency was manipulated in this study by providing different types of explanations.
- Low Transparency: only one sentence was shown: “Your computed grade is X, which is the grade you received from your peers.”
- Medium Transparency: more information about the computation of the final grade was provided: “Your computed grade is X, which is based on the grades you received from your peers and adjusted for their bias and accuracy in grading. The accuracy and bias are estimated using a statistical procedure that employs an expectation maximization algorithm with a prior for class grades. This adjusts your grade for easy/harsh graders and grader proficiency.
- High Transparency: in addition to the explanation of the grading process, participants saw the raw individual peer grades they received and how these were adjusted to arrive at their final grade.
- Immediately following the transparency manipulation, participants answered questions to measure their trust
in the peer assessment system.
- Four items assessed facets of trust
- “To what extent do you understand how your grade is computed in peer grading?”
- “How fair or unfair was the peer grading process?”
- “How accurate or inaccurate was the peer grading process?”, and
- “How much did you trust or distrust your peers to grade you fairly?”
- Participants responded on construct-specific and fully labeled response scales with
- 5 points for the unipolar item about understanding (‘No un- derstanding at all’ to ‘Excellent understanding’), and
- 7 points for all other items (e.g., ‘Definitely fair’ to ‘Definitely unfair’)
Data Analysis Methodologies
A 2 (expectations violated vs. not violated) by 3 (transparency: low, medium, high) ANOVA
was conducted to test the first two hypotheses.
Did the paper draw convincing conclusions using the methodology?
Yes. However, the study included only a small sample size of participants and focused on self-report outcomes.
Describe a few ways in which the data collection and analysis methodology can be improved to better answer the research questions
We can do qualitative interviews to gain deeper insights into the user experience and
replicate the results in different contexts to better answer the research questions.