Toward a Learning Science for Complex Crowdsourcing Tasks
Complex Tasks and Learning in Crowdsourcing
/
Doroudi, Shayan
/
Kamar, Ece
/
Brunskill, Emma
/
Horvitz, Eric
Proceedings of the ACM CHI'16 Conference on Human Factors in Computing
Systems
2016-05-07
v.1
p.2623-2634
© Copyright 2016 ACM
Summary: We explore how crowdworkers can be trained to tackle complex crowdsourcing
tasks. We are particularly interested in training novice workers to perform
well on solving tasks in situations where the space of strategies is large and
workers need to discover and try different strategies to be successful. In a
first experiment, we perform a comparison of five different training
strategies. For complex web search challenges, we show that providing expert
examples is an effective form of training, surpassing other forms of training
in nearly all measures of interest. However, such training relies on access to
domain expertise, which may be expensive or lacking. Therefore, in a second
experiment we study the feasibility of training workers in the absence of
domain expertise. We show that having workers validate the work of their peer
workers can be even more effective than having them review expert examples if
we only present solutions filtered by a threshold length. The results suggest
that crowdsourced solutions of peer workers may be harnessed in an automated
training pipeline.
Interface Design Optimization as a Multi-Armed Bandit Problem
Making Interfaces Work for Each Individual
/
Lomas, J. Derek
/
Forlizzi, Jodi
/
Poonwala, Nikhil
/
Patel, Nirmal
/
Shodhan, Sharan
/
Patel, Kishan
/
Koedinger, Ken
/
Brunskill, Emma
Proceedings of the ACM CHI'16 Conference on Human Factors in Computing
Systems
2016-05-07
v.1
p.4142-4153
© Copyright 2016 ACM
Summary: "Multi-armed bandits" offer a new paradigm for the AI-assisted design of
user interfaces. To help designers understand the potential, we present the
results of two experimental comparisons between bandit algorithms and random
assignment. Our studies are intended to show designers how bandits algorithms
are able to rapidly explore an experimental design space and automatically
select the optimal design configuration. Our present focus is on the
optimization of a game design space. The results of our experiments show that
bandits can make data-driven design more efficient and accessible to interface
designers, but that human participation is essential to ensure that AI systems
optimize for the right metric. Based on our results, we introduce several
design lessons that help keep human design judgment in the loop. We also
consider the future of human-technology teamwork in AI-assisted design and
scientific inquiry. Finally, as bandits deploy fewer low-performing conditions
than typical experiments, we discuss ethical implications for bandits in
large-scale experiments in education.
Towards automatic experimentation of educational knowledge
Games and education
/
Liu, Yun-En
/
Mandel, Travis
/
Brunskill, Emma
/
Popovic, Zoran
Proceedings of ACM CHI 2014 Conference on Human Factors in Computing Systems
2014-04-26
v.1
p.3349-3358
© Copyright 2014 ACM
Summary: We present a general automatic experimentation and hypothesis generation
framework that utilizes a large set of users to explore the effects of
different parts of an intervention parameter space on any objective function.
We also incorporate importance sampling, allowing us to run these automatic
experiments even if we cannot give out the exact intervention distributions
that we want. To show the utility of this framework, we present an
implementation in the domain of fractions and numberlines, using an online
educational game as the source of players. Our system is able to automatically
explore the parameter space and generate hypotheses about what types of
numberlines lead to maximal short-term transfer; testing on a separate dataset
shows the most promising hypotheses are valid. We briefly discuss our results
in the context of the wider educational literature, showing that one of our
results is not explained by current research on multiple fraction
representations, thus proving our ability to generate potentially interesting
hypotheses to test.
Designing mobile interfaces for novice and low-literacy users
/
Medhi, Indrani
/
Patnaik, Somani
/
Brunskill, Emma
/
Gautama, S. N. Nagasena
/
Thies, William
/
Toyama, Kentaro
ACM Transactions on Computer-Human Interaction
2011-04
v.18
n.1
p.2
© Copyright 2011 ACM
Summary: While mobile phones have found broad application in bringing health,
financial, and other services to the developing world, usability remains a
major hurdle for novice and low-literacy populations. In this article, we take
two steps to evaluate and improve the usability of mobile interfaces for such
users. First, we offer an ethnographic study of the usability barriers facing
90 low-literacy subjects in India, Kenya, the Philippines, and South Africa.
Then, via two studies involving over 70 subjects in India, we quantitatively
compare the usability of different points in the mobile design space. In
addition to text interfaces such as electronic forms, SMS, and USSD, we
consider three text-free interfaces: a spoken dialog system, a graphical
interface, and a live operator.
Our results confirm that textual interfaces are unusable by first-time
low-literacy users, and error prone for literate but novice users. In the
context of healthcare, we find that a live operator is up to ten times more
accurate than text-based interfaces, and can also be cost effective in
countries such as India. In the context of mobile banking, we find that task
completion is highest with a graphical interface, but those who understand the
spoken dialog system can use it more quickly due to their comfort and
familiarity with speech. We synthesize our findings into a set of design
recommendations.
