The following schedule is tentative and subject to change.

Date Topic Readings Assignments & Project
Aug 20 Introduction & Course overview

Lecture (slides)
Aug 22 Crowdsourcing:
Background and applications

Lecture (slides)

Required

Howe. The Rise of Crowdsourcing. WIRED, June 2006.

Howe. Crowdsourcing: A Definition. June 2006.

Optional

von Ahn et al. reCaptcha: Human-based Character Recognition via Web Security Measures. Science, September 2008

von Ahn and Dabbish. Labeling Images with a Computer Game. CHI'04

Cooper et al. Predicting Protein Structures with a Multiplayer Online Game. Nature, August 2010.

Assignment 1: Due before next class (11:59pm, Aug 26)
Aug 27 Crowd workers

Lecture (slides)

Required

Martin et al. Being a Turker. CSCW'14

Yin et al. The Communication Network Within the Crowd. WWW'16

Optional

Gray et al. The Crowd is a Collaborative Network. CSCW'16

Difallah et al. Demographics and Dynamics of Mechanical Turk Workers. WSDM'18

Sign-up for presentations!
Aug 29 Requesters in crowdsourcing markets

Lecture (slides)

Creating a HIT on MTurk
No required readings. Bring laptops to class.
Be an MTurk Requester (Part 1)
Be an MTurk Requester (Part 2)

Assignment 2: Due on 11:59pm, Sep 14

Sep 3 No class (Labor day)
Sep 5 Crowdsourcing platform:
Tasks and dynamics

Lecture (slides)

Required

Difallah et al. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. WWW'15

Optional

Gadiraju et al. A Taxonomy of Microtasks on the Web. HT'14

Vakharia and Lease. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. iConference'15

Jain et al. Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace. VLDB Endowment, March 2017

Sep 10 Crowdsourcing:
Opportunities and challenges

Lecture (slides)

Required

Kittur et al. The Future of Crowd Work. CSCW'13

Optional

Quinn and Bederson. Human Computation: A Survey and Taxonomy of a Growing Field. CHI'11

Gadiraju et al. Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing. IEEE Intelligent Systems, July 2015

Sep 12 Incentive design:
Financial incentives

Lecture

Required

Ho et al. Incentivizing High Quality Crowdwork. WWW'15

Optional

Mason and Watts. Financial Incentives and the "Performance of Crowds". HCOMP'09

Yin et al. The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. AAAI'13

Harris. The Effects of Pay-to-Quit Incentives on Crowdworker Task Quality. CSCW'15

Sep 17 Incentive design:
Other incentives

Presenter: Tianai Zhao

Required

Rogstadius et al. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. ICWSM'11

Shaw et al. Designing Incentives for Inexpert Human Raters. CSCW'11

Optional

Nov et al. Scientists@Home: What Drives the Quantity and Quality of Online Citizen Science Participation?. PLOS ONE, April 2014

Feyisetan and Simperl. Social Incentives in Paid Collaborative Crowdsourcing. TIST, September 2017

Sep 19 Incentive design:
Intelligent management

Presenter: Altug Gemalmaz

Required

Yin and Chen. Bonus or Not? Learn to Reward in Crowdsourcing. IJCAI'15

Optional

Gao and Parameswaran. Finish Them!: Pricing Algorithms for Human Computation. VLDB Endowment, October 2014

Feyisetan et al. Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives. WWW'15

Final project: Guideline out; proposal due on Oct 3
Sep 24 Task assignment and recommendation

Presenter: Md Nasim

Required

Ho and Vaughan. Online Task Assignment in Crowdsourcing Markets. AAAI'12

Difallah et al. Pick-a-Crowd: Tell Me What You Like, and I'll Tell You What To Do. WWW'13

Optional

Lin et al. Signals in the Silence: Models of Implicit Feedback in a Recommendation System for Crowdsourcing. AAAI'14

Mavridis et al. Using Hierarchical Skills for Optimized Task Assignment in Knowledge-Intensive Crowdsourcing. WWW'16

Sep 26 Quality assurance:
Empirical methods

Presenter: Naman Patwari

Required

Dow et al. Shephearding the Crowd Yields Better Work. CSCW'12

Optional

Huang and Fu. Enhancing Reliability Using Peer Consistency Evaluation in Human Computation. CSCW'13

Sampath et al. Cognitively Inspired Task Design to Improve User Performance on Crowdsourcing Platforms. CHI'14

Doroudi et al. Toward a Learning Science for Complex Crowdsourcing Tasks. CHI'16

Oct 1 Quality assurance:
Algorithmic approaches

Presenter: Aldo Porco

Required

Whitehill et al. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. NIPS'09

Optional

Oyama et al. Accurate Integration of Crowdsourced Labels Using Workers' Self-Reported Confidence Scores. IJCAI'13

Sunahase et al. Pairwise HITS: Quality Estimation from Pairwise Comparisons in Creator-Evaluator Crowdsourcing Process. AAAI'17

Wang et al. Obtaining High-Quality Label by Distinguishing between Easy and Hard Items in Crowdsourcing. IJCAI'17

Oct 3 Quality assurance:
Intelligent management

Presenter: Amy Rechkemmer

Required

Kamar et al. Combining Human and Machine Intelligence in Large-Scale Crowdsourcing. AAMAS'12

Optional

Bragg et al. Optimal Testing for Crowd Workers. AAMAS'16

Gurari and Grauman. CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question. CHI'17

Oct 8 No class (October break)
Oct 10 Final project: Q&A Class
Oct 15 Final project: Pitch
Oct 17 Engagement control:
Empirical methods

Presenter: KMA Solaiman

Required

Yu et al. A Comparison of Social, Learning, and Financial Strategies on Crowd Engagement and Output Quality. CSCW'14

Law et al. Curiosity Killed the Cat, but Makes Crowdwork Better. CHI'16

Optional

Preist et al. Competing or Aiming to be Average?: Normification as a Means of Engaging Digital Volunteers. CSCW'14

Dai et al. And Now for Something Completely Different: Improving Crowdsourcing Workflows with Micro-Diversions. CSCW'15

Oct 22 Engagement control:
Intelligent management

Presenter: Altug Gemalmaz

Required

Segal et al. Intervention Strategies for Increasing Engagement in Crowdsourcing: Platform, Predictions, and Experiments. IJCAI'16

Optional

Segal et al. Optimizing Interventions via Offline Policy Evaluation: Studies in Citizen Science. AAAI'18

Oct 24 Workflow design:
Use-case specific workflows

Presenter: Md Nasim

Required

Bernstein et al. Soylent: A Word Processor with a Crowd Inside. UIST'10

Chilton et al. Cascade: Crowdsourcing Taxonomy Creation. CHI'13

Optional

Noronha et al. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. UIST'11

Kim et al. Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. CHI'14

Oct 29 Workflow design:
General workflows

Presenter: Tianai Zhao

Required

Little et al. TurKit: Human Computation Algorithms on Mechanical Turk. UIST'10

Optional

Kittur et al. CrowdForge: Crowdsourcing Complex Work. UIST'11

Kulkarni et al. Collaboratively Crowdsourcing Workflows with Turkomatic. CSCW'12

Oct 31 Workflow design:
Intelligent management

Presenter: Aldo Porco

Required

Dai et al. Decision-theoretic Control of Crowd-sourced Workflows. AAAI'10

Optional

Lin et al. Dynamically Switching between Synergistic Workflows for Crowdsourcing. AAAI'12

Bragg et al. Crowdsourcing Multi-Label Classification for Taxonomy Creation. HCOMP'13

Tran-Thanh et al. Crowdsourcing Complex Workflows under Budget Constraints. AAAI'15

Nov 5 Beyond independence:
Cooperative work examples

Presenter: Amy Rechkemmer

Required

Drapeau et al. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. HCOMP'16

Optional

Suzuki et al. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-Internships. CHI'16

Chang et al. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. CHI'17

Nov 7 Final project: Peer feedback
Nov 12 Cooperative work:
Complex task

Presenter: Naman Patwari

Required

Retelny et al. Expert Crowdsourcing with Flash Teams. UIST'14

Optional

Hahn et al. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. CHI'16

Valentine et al. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. CHI'17

Nov 14 Cooperative work:
Intelligent management

Presenter: KMA Solaiman

Required

Salehi et al. Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability. CSCW'17

Optional

Singla et al. Learning to Hire Teams. HCOMP'15

Zhou et al. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. CHI'18

Nov 19 Crowd-Powered Systems

Lecture

Required

Lasecki et al. Real-Time Captioning by Groups of Non-Experts. UIST'12

Kokkalis et al. MyriadHub: Efficiently Scaling Personalized Email Conversations with Valet Crowdsourcing. CHI'17

Optional

Vashistha et al. Respeak: A Voice-based, Crowd-powered Speech Transcription System. CHI'17

Nguyen et al. An Interpretable Joint Graphical Model for Fact-Checking from Crowds. AAAI'18

Nov 21 No class (Thanksgiving)
Nov 26 Crowdsourcing: Future Ideas

Lecture

Required

Whiting et al. Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms. CSCW'17

Optional

Morris et al. Subcontracting Microwork. CHI'17

Vaish et al. Crowd Research: Open and Scalable University Laboratories. UIST'17

Nov 28 Final project presentation (Session 1)
Dec 3 Final project presentation (Session 2)