The following schedule is tentative and subject to change.

Date Topic Readings Assignments & Project
Jan 13 Introduction & Course overview Assignment 1: Due on 11:59pm, Jan 19
Jan 15 Crowdsourcing:
Applications and crowd workers



Howe. The Rise of Crowdsourcing. WIRED, June 2006.

Martin et al. Being a Turker. CSCW'14


von Ahn and Dabbish. Labeling Images with a Computer Game. CHI'04

von Ahn et al. reCaptcha: Human-based Character Recognition via Web Security Measures. Science, September 2008

Cooper et al. Predicting Protein Structures with a Multiplayer Online Game. Nature, August 2010.

Yin et al. The Communication Network Within the Crowd. WWW'16

Difallah et al. Demographics and Dynamics of Mechanical Turk Workers. WSDM'18

Jan 20 No class (Martin Luther King Day)
Jan 22 No class (Instructor travel)
Jan 27 Crowdsourcing requesters


Creating a HIT on MTurk
No required readings. Bring laptops to class.
Be an MTurk Requester (Part 1)

Assignment 2: Due on 11:59pm, Feb 4

Jan 29 Crowdsourcing platform:
Tasks and dynamics



Difallah et al. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. WWW'15


Gadiraju et al. A Taxonomy of Microtasks on the Web. HT'14

Vakharia and Lease. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. iConference'15

Jain et al. Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace. VLDB Endowment, March 2017

Feb 3 Interpretable ML: Definitions and Methods



Ribeiro et al. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD'16

Lipton. The Mythos of Model Interpretability. Communications of ACM, September 2018


Wang and Rudin. Falling Rule Lists. AISTATS'15

Caruana et al. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. KDD'15

Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML'18

Guidotti et al. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys, August 2018

Feb 5 Fairness in ML: Definitions and Methods



Angwin et al. Machine Bias. 2016

Zafar et al. Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment. WWW'17


Caliskan et al. Semantics Derived Automatically from Language Corpora Contain Human-Like Biases. Science. April 2017

Verma and Rubin. Fairness Definitions Explained. FairWare'18

Ustun et al. Fairness without Harm: Decoupled Classifiers with Preference Guarantees. ICML'19

Feb 10 Human-AI Interaction



Doshi-Velez and Kim. Towards a Rigorous Science of Interpretable Machine Learning. 2017

Amershi et al. Guidelines for Human-AI Interaction. CHI'19


Holstein et al. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. CHI'19

Final project: Guideline out; proposal due on Feb 23
Feb 12 Incentive design:
Empirical studies


Ho et al. Incentivizing High Quality Crowdwork. WWW'15

Rogstadius et al. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. ICWSM'11


Shaw et al. Designing Incentives for Inexpert Human Raters. CSCW'11

Yin et al. The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. AAAI'13

Chandler and Kapelner. Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets. Journal of Economic Behavior & Organization, 2013

Harris. The Effects of Pay-to-Quit Incentives on Crowdworker Task Quality. CSCW'15

Feb 17 Incentive design:
Intelligent management


Yin and Chen. Bonus or Not? Learn to Reward in Crowdsourcing. IJCAI'15


Gao and Parameswaran. Finish Them!: Pricing Algorithms for Human Computation. VLDB Endowment, October 2014

Feyisetan et al. Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives. WWW'15

Feb 19 Task assignment and recommendation


Ho and Vaughan. Online Task Assignment in Crowdsourcing Markets. AAAI'12

Difallah et al. Pick-a-Crowd: Tell Me What You Like, and I'll Tell You What To Do. WWW'13


Lin et al. Signals in the Silence: Models of Implicit Feedback in a Recommendation System for Crowdsourcing. AAAI'14

Mavridis et al. Using Hierarchical Skills for Optimized Task Assignment in Knowledge-Intensive Crowdsourcing. WWW'16

Feb 24 Final project: Pitch
Feb 26 Quality assurance:
Empirical methods


Dow et al. Shephearding the Crowd Yields Better Work. CSCW'12

Zhu et al. Reviewing versus Doing: Learning and Performance in Crowd Assessment. CSCW'14


Huang and Fu. Enhancing Reliability Using Peer Consistency Evaluation in Human Computation. CSCW'13

Sampath et al. Cognitively Inspired Task Design to Improve User Performance on Crowdsourcing Platforms. CHI'14

Doroudi et al. Toward a Learning Science for Complex Crowdsourcing Tasks. CHI'16

Mar 2 Quality assurance:
Intelligent management


Whitehill et al. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. NIPS'09

Bragg et al. Optimal Testing for Crowd Workers. AAMAS'16


Kamar et al. Combining Human and Machine Intelligence in Large-Scale Crowdsourcing. AAMAS'12

Oyama et al. Accurate Integration of Crowdsourced Labels Using Workers' Self-Reported Confidence Scores. IJCAI'13

Sunahase et al. Pairwise HITS: Quality Estimation from Pairwise Comparisons in Creator-Evaluator Crowdsourcing Process. AAAI'17

Wang et al. Obtaining High-Quality Label by Distinguishing between Easy and Hard Items in Crowdsourcing. IJCAI'17

Gurari and Grauman. CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question. CHI'17

Mar 4 Engagement control


Law et al. Curiosity Killed the Cat, but Makes Crowdwork Better. CHI'16

Segal et al. Intervention Strategies for Increasing Engagement in Crowdsourcing: Platform, Predictions, and Experiments. IJCAI'16


Yu et al. A Comparison of Social, Learning, and Financial Strategies on Crowd Engagement and Output Quality. CSCW'14

Preist et al. Competing or Aiming to be Average?: Normification as a Means of Engaging Digital Volunteers. CSCW'14

Dai et al. And Now for Something Completely Different: Improving Crowdsourcing Workflows with Micro-Diversions. CSCW'15

Segal et al. Optimizing Interventions via Offline Policy Evaluation: Studies in Citizen Science. AAAI'18

Mar 9 Workflow design:
Specific and general workflows


Bernstein et al. Soylent: A Word Processor with a Crowd Inside. UIST'10

Little et al. TurKit: Human Computation Algorithms on Mechanical Turk. UIST'10


Kittur et al. CrowdForge: Crowdsourcing Complex Work. UIST'11

Noronha et al. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. UIST'11

Kulkarni et al. Collaboratively Crowdsourcing Workflows with Turkomatic. CSCW'12

Chilton et al. Cascade: Crowdsourcing Taxonomy Creation. CHI'13

Kim et al. Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. CHI'14

Mar 11Workflow design:
Intelligent management


Dai et al. Decision-theoretic Control of Crowd-sourced Workflows. AAAI'10


Lin et al. Dynamically Switching between Synergistic Workflows for Crowdsourcing. AAAI'12

Bragg et al. Crowdsourcing Multi-Label Classification for Taxonomy Creation. HCOMP'13

Tran-Thanh et al. Crowdsourcing Complex Workflows under Budget Constraints. AAAI'15

Mar 16 No class (Spring break)
Mar 18 No class (Spring break)
Mar 23 Coopeartive work:


Retelny et al. Expert Crowdsourcing with Flash Teams. UIST'14

Drapeau et al. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. HCOMP'16


Suzuki et al. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-Internships. CHI'16

Chang et al. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. CHI'17

Valentine et al. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. CHI'17

Mar 25 Cooperative work:
Intelligent management


Zhou et al. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. CHI'18

Tang et al. Leveraging Peer Communication to Enhance Crowdsourcing. WWW'19


Singla et al. Learning to Hire Teams. HCOMP'15

Salehi et al. Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability. CSCW'17

Final project: Midterm report due on Mar 29
Mar 30 Trust in AI/ML


Yin et al. Understanding the Effect of Accuracy on Trust in Machine Learning Models. CHI'19

Kocielnik et al. Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems. CHI'19


Lim and Dey. Investigating Intelligibility for Uncertain Context-Aware Applications. UbiComp'11

Yu et al. Trust and Reliance Based on System Accuracy. UMAP'16

Yu et al. User Trust Dynamics: An Investigation Driven by Differences in System Performance. IUI'17

Apr 1 Mental Models of ML


Bansal et al. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. HCOMP'19


Bansal et al. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. AAAI'19

Apr 6 Interpretable ML: User studies


Lai and Tan. On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. ACM FAT*'19

Lage et al. Human Evaluation of Models Built for Interpretability. HCOMP'19


Poursabzi-Sangdeh et al. Manipulating and Measuring Model Interpretability. 2018

Nourani et al. The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems. HCOMP'19

Apr 8 Interpretable ML: Interface


Hohman et al. Gamut: A Design Prob to Understand How Data Scientists Understand Machine Learning Models. CHI'19

Cheng et al. Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. CHI'19


Cai et al. The Effects of Example-based Explanations in a Machine Learning Interface. IUI'19

Apr 13 Bias in Crowdsourced Data


Hube et al. Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. CHI'19

Otterbacher et al. How Do We Talk about Other People? Group (Un)Fairness in Natural Language Image Descriptions. HCOMP'19


Eickhoff. Cognitive Biases in Crowdsourcing. WSDM'18

Apr 15 Fairness in ML: Empirical Evaluations


Saxena et al. How Do Fairness Definitions Fare?: Exmaining Publich Attitudes Towards Algorithmic Definitions of Fairness. AIES'19

Dodge et al. Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgement. IUI'19


Binns et al. 'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions. CHI'18

Apr 20 Human-AI Team


Nushi et al. Towards Accoutable AI: Hybrid Human-Machine Analyses for Characterizing System Failure. HCOMP'18

Green and Chen. The Principles and Limits of Algorithm-in-the-Loop Decision Making. CSCW'19


Green and Chen. Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. ACM FAT*'19

Vandenhof. A Hybrid Approach to Identifying Unknown Unknowns of Predictive Models. HCOMP'19

Ray et al. Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval. HCOMP'19

Apr 22 No class (Project day)
Apr 27 Final project presentation (Session 1)
Apr 29 Final project presentation (Session 2)