| Date |
Topic |
Readings |
Assignments & Project |
| Jan 13 |
Introduction & Course overview |
|
Assignment 1:
Due on 11:59pm, Jan 19 |
| Jan 15 |
Crowdsourcing:
Applications and crowd workers
Lecture |
Required
Howe. The Rise of Crowdsourcing. WIRED, June 2006.
Martin et al. Being a Turker. CSCW'14
Optional
von Ahn and Dabbish. Labeling Images with a Computer Game. CHI'04
von Ahn et al. reCaptcha: Human-based Character Recognition via Web Security Measures. Science, September 2008
Cooper et al. Predicting Protein Structures with a Multiplayer Online Game. Nature, August 2010.
Yin et al. The Communication Network Within the Crowd. WWW'16
Difallah et al. Demographics and Dynamics of Mechanical Turk Workers. WSDM'18
|
|
| Jan 20 |
No class (Martin Luther King Day) |
|
|
| Jan 22 |
No class (Instructor travel) |
|
|
| Jan 27 |
Crowdsourcing requesters
Lecture |
Creating a HIT on MTurk
No required readings. Bring laptops to class.
Be an MTurk Requester (Part 1)
Be an MTurk Requester (Part 2)
|
Assignment 2: Due on 11:59pm, Feb 9
|
| Jan 29 |
Crowdsourcing platform:
Tasks and dynamics
Lecture |
Required
Difallah et al. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. WWW'15
Optional
Gadiraju et al. A Taxonomy of Microtasks on the Web. HT'14
Vakharia and Lease. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. iConference'15
Jain et al. Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace. VLDB Endowment, March 2017
|
|
| Feb 3 |
Interpretable ML: Definitions and Methods
Lecture |
Required
Ribeiro et al. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD'16
Lipton. The Mythos of Model Interpretability. Communications of ACM, September 2018
Optional
Wang and Rudin. Falling Rule Lists. AISTATS'15
Caruana et al. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. KDD'15
Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML'18
Guidotti et al. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys, August 2018
|
|
| Feb 5 |
Fairness in ML: Definitions and Methods
Lecture |
Required
Angwin et al. Machine Bias. 2016
Zafar et al. Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment. WWW'17
Optional
Caliskan et al. Semantics Derived Automatically from Language Corpora Contain Human-Like Biases. Science. April 2017
Verma and Rubin. Fairness Definitions Explained. FairWare'18
Ustun et al. Fairness without Harm: Decoupled Classifiers with Preference Guarantees. ICML'19
|
|
| Feb 10 |
Human-AI Interaction
Lecture |
Required
Doshi-Velez and Kim. Towards a Rigorous Science of Interpretable Machine Learning. 2017
Amershi et al. Guidelines for Human-AI Interaction. CHI'19
Optional
Holstein et al. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. CHI'19
| Final project: Guideline out; proposal due on Feb 23 |
| Feb 12 |
Incentive design:
Empirical studies
Presenter: Xiao |
Required
Ho et al. Incentivizing High Quality Crowdwork. WWW'15
Rogstadius et al. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. ICWSM'11
Optional
Shaw et al. Designing Incentives for Inexpert Human Raters. CSCW'11
Yin et al. The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. AAAI'13
Chandler and Kapelner. Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets. Journal of Economic Behavior & Organization, 2013
Harris. The Effects of Pay-to-Quit Incentives on Crowdworker Task Quality. CSCW'15
|
|
| Feb 17 |
Incentive design:
Intelligent management
Presenter: Weilong |
Required
Yin and Chen. Bonus or Not? Learn to Reward in Crowdsourcing. IJCAI'15
Optional
Gao and Parameswaran. Finish Them!: Pricing Algorithms for Human Computation. VLDB Endowment, October 2014
Feyisetan et al. Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives. WWW'15
|
|
| Feb 19 |
Task assignment and recommendation
Presenter: Zheng |
Required
Ho and Vaughan. Online Task Assignment in Crowdsourcing Markets. AAAI'12
Difallah et al. Pick-a-Crowd: Tell Me What You Like, and I'll Tell You What To Do. WWW'13
Optional
Lin et al. Signals in the Silence: Models of Implicit Feedback in a Recommendation System for Crowdsourcing. AAAI'14
Mavridis et al. Using Hierarchical Skills for Optimized Task Assignment in Knowledge-Intensive Crowdsourcing. WWW'16
|
|
| Feb 24 |
Final project: Pitch |
|
|
| Feb 26 |
Quality assurance:
Empirical methods
Presenter: Chun-Wei |
Required
Dow et al. Shephearding the Crowd Yields Better Work. CSCW'12
Zhu et al. Reviewing versus Doing: Learning and Performance in Crowd Assessment. CSCW'14
Optional
Huang and Fu. Enhancing Reliability Using Peer Consistency Evaluation in Human Computation. CSCW'13
Sampath et al. Cognitively Inspired Task Design to Improve User Performance on Crowdsourcing Platforms. CHI'14
Doroudi et al. Toward a Learning Science for Complex Crowdsourcing Tasks. CHI'16
|
|
| Mar 2 |
Quality assurance:
Intelligent management
Presenter: Guoyang |
Required
Whitehill et al. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. NIPS'09
Bragg et al. Optimal Testing for Crowd Workers. AAMAS'16
Optional
Kamar et al. Combining Human and Machine Intelligence in Large-Scale Crowdsourcing. AAMAS'12
Oyama et al. Accurate Integration of Crowdsourced Labels Using Workers' Self-Reported Confidence Scores. IJCAI'13
Sunahase et al. Pairwise HITS: Quality Estimation from Pairwise Comparisons in Creator-Evaluator Crowdsourcing Process. AAAI'17
Wang et al. Obtaining High-Quality Label by Distinguishing between Easy and Hard Items in Crowdsourcing. IJCAI'17
Gurari and Grauman.
CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question. CHI'17 |
|
| Mar 4 |
Engagement control
Presenter: Xinru |
Required
Law et al. Curiosity Killed the Cat, but Makes Crowdwork Better. CHI'16
Segal et al. Intervention Strategies for Increasing Engagement in Crowdsourcing: Platform, Predictions, and Experiments. IJCAI'16
Optional
Yu et al. A Comparison of Social, Learning, and Financial Strategies on Crowd Engagement and Output Quality. CSCW'14
Preist et al. Competing or Aiming to be Average?: Normification as a Means of Engaging Digital Volunteers. CSCW'14
Dai et al. And Now for Something Completely Different: Improving Crowdsourcing Workflows with Micro-Diversions. CSCW'15
Segal et al. Optimizing Interventions via Offline Policy Evaluation: Studies in Citizen Science. AAAI'18
|
|
| Mar 9 |
Workflow design:
Specific and general workflows
Presenter: Haobo |
Required
Bernstein et al. Soylent: A Word Processor with a Crowd Inside. UIST'10
Little et al. TurKit: Human Computation Algorithms on Mechanical Turk. UIST'10
Optional
Kittur et al. CrowdForge: Crowdsourcing Complex Work. UIST'11
Noronha et al. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. UIST'11
Kulkarni et al. Collaboratively Crowdsourcing Workflows with Turkomatic. CSCW'12
Chilton et al. Cascade: Crowdsourcing Taxonomy Creation. CHI'13
Kim et al. Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. CHI'14
|
|
| Mar 11 | Workflow design:
Intelligent management
Presenter: Guizhen |
Required
Dai et al. Decision-theoretic Control of Crowd-sourced Workflows. AAAI'10
Optional
Lin et al. Dynamically Switching between Synergistic Workflows for Crowdsourcing. AAAI'12
Bragg et al. Crowdsourcing Multi-Label Classification for Taxonomy Creation. HCOMP'13
Tran-Thanh et al. Crowdsourcing Complex Workflows under Budget Constraints. AAAI'15
|
|
| Mar 16 |
No class (Spring break) |
|
|
| Mar 18 |
No class (Spring break) |
|
|
| Mar 23 |
Coopeartive work:
Applications
Presenter: Zhuoran |
Required
Retelny et al. Expert Crowdsourcing with Flash Teams. UIST'14
Drapeau et al. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. HCOMP'16
Optional
Suzuki et al. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-Internships. CHI'16
Chang et al. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. CHI'17
Valentine et al. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. CHI'17 |
|
| Mar 25 |
Cooperative work:
Intelligent management
Presenter: Xiaoni |
Required
Zhou et al. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. CHI'18
Tang et al. Leveraging Peer Communication to Enhance Crowdsourcing. WWW'19
Optional
Singla et al. Learning to Hire Teams. HCOMP'15
Salehi et al. Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability. CSCW'17
|
Final project: Midterm report due on Mar 29 |
| Mar 30 |
Trust in AI/ML
Presenter: Guoyang |
Required
Yin et al. Understanding the Effect of Accuracy on Trust in Machine Learning Models. CHI'19
Kocielnik et al. Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems. CHI'19
Optional
Lim and Dey. Investigating Intelligibility for Uncertain Context-Aware Applications. UbiComp'11
Yu et al. Trust and Reliance Based on System Accuracy. UMAP'16
Yu et al. User Trust Dynamics: An Investigation Driven by Differences in System Performance. IUI'17
|
|
| Apr 1 |
Mental Models of ML
Presenter: Chun-Wei |
Required
Bansal et al. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. HCOMP'19
Optional
Bansal et al. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. AAAI'19
|
|
| Apr 6 |
Interpretable ML: User studies
Presenter: Xiao & Haobo |
Required
Lai and Tan. On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. ACM FAT*'19
Lage et al. Human Evaluation of Models Built for Interpretability. HCOMP'19
Optional
Poursabzi-Sangdeh et al. Manipulating and Measuring Model Interpretability. 2018
Nourani et al. The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems. HCOMP'19
|
|
| Apr 8 |
Interpretable ML: Interface
Presenter: Xiaoni |
Required
Hohman et al. Gamut: A Design Prob to Understand How Data Scientists Understand Machine Learning Models. CHI'19
Cheng et al. Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. CHI'19
Optional
Cai et al. The Effects of Example-based Explanations in a Machine Learning Interface. IUI'19
|
|
| Apr 13 |
Bias in Crowdsourced Data
Presenter: Zheng & Weilong |
Required
Hube et al. Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. CHI'19
Otterbacher et al. How Do We Talk about Other People? Group (Un)Fairness in Natural Language Image Descriptions. HCOMP'19
Optional
Eickhoff. Cognitive Biases in Crowdsourcing. WSDM'18
|
|
| Apr 15 |
Fairness in ML: Empirical Evaluations
Presenter: Zhuoran |
Required
Saxena et al. How Do Fairness Definitions Fare?: Exmaining Publich Attitudes Towards Algorithmic Definitions of Fairness. AIES'19
Dodge et al. Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgement. IUI'19
Optional
Binns et al. 'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions. CHI'18
|
|
| Apr 20 |
Human-AI Team
Presenter: Xinru |
Required
Nushi et al. Towards Accoutable AI: Hybrid Human-Machine Analyses for Characterizing System Failure. HCOMP'18
Green and Chen. The Principles and Limits of Algorithm-in-the-Loop Decision Making. CSCW'19
Optional
Green and Chen. Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. ACM FAT*'19
Vandenhof. A Hybrid Approach to Identifying Unknown Unknowns of Predictive Models. HCOMP'19
Ray et al. Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval. HCOMP'19
|
|
| Apr 22 |
No class (Project day) |
|
|
| Apr 27 |
Final project presentation (Session 1) |
|
|
| Apr 29 |
Final project presentation (Session 2) |
|
|