I invited Mike Mozer (co-author of this paper) to talk about this work to our research group. It was such a cool and relevant study to the type of evaluation we care about today, which is starting to lean more on human behavior (versus the conventional “lowering of the loss.”) This paper discusses an experiment framework for modeling “human engagement” in game-like tasks on Mechanical Turk.
Given a game like Flappy Bird, how do factors (like difficulty) impact human engagement? With these characteristics, the researchers aimed to fit a model to predict how engagement varied.
For the methodology, human subjects:
Played for a mandatory time of 2:30 minutes
Played voluntarily afterwards
Indicated how long they expected other mTurkers to play.
Answered a survey of 6 true/false questions (true = engaging game)
We use Bayesian optimization methods to design games that maximize user engagement. Participants are paid to try a game for several minutes, at which point they can quit or continue to play voluntarily with no further compensation. Engagement is measured by player persistence, projections of how long others will play, and a post-game survey. Using Gaussian process surrogate-based optimization, we conduct efficient experiments to identify game design characteristics---specifically those influencing difficulty---that lead to maximal engagement. We study two games requiring trajectory planning, the difficulty of each is determined by a three-dimensional continuous design space. Two of the design dimensions manipulate the game in user-transparent manner (e.g., the spacing of obstacles), the third in a subtle and possibly covert manner (incremental trajectory corrections). Converging results indicate that overt difficulty manipulations are effective in modulating engagement only when combined with the covert manipulation, suggesting the critical role of a user's self-perception of competence.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.