{"id":33216605,"url":"https://github.com/BoulderDS/human-centered-machine-learning","last_synced_at":"2025-11-21T07:02:29.331Z","repository":{"id":106258580,"uuid":"232194493","full_name":"BoulderDS/human-centered-machine-learning","owner":"BoulderDS","description":null,"archived":false,"fork":false,"pushed_at":"2020-02-18T19:16:00.000Z","size":38,"stargazers_count":53,"open_issues_count":0,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-07T06:34:12.630Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BoulderDS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-01-06T22:10:34.000Z","updated_at":"2023-11-29T05:25:09.000Z","dependencies_parsed_at":"2024-01-21T06:39:34.365Z","dependency_job_id":"0bb0b9fd-3258-4f14-872d-29877a65c0be","html_url":"https://github.com/BoulderDS/human-centered-machine-learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BoulderDS/human-centered-machine-learning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoulderDS%2Fhuman-centered-machine-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoulderDS%2Fhuman-centered-machine-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoulderDS%2Fhuman-centered-machine-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoulderDS%2Fhuman-centered-machine-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BoulderDS","download_url":"https://codeload.github.com/BoulderDS/human-centered-machine-learning/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BoulderDS%2Fhuman-centered-machine-learning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285573085,"owners_count":27194648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-21T02:00:06.175Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T14:00:16.532Z","updated_at":"2025-11-21T07:02:29.319Z","avatar_url":"https://github.com/BoulderDS.png","language":null,"funding_links":[],"categories":["6. Useful Resources"],"sub_categories":["Courses \u0026 Talks"],"readme":"Human-centered Machine Learning (Spring 2020)\n============================\n\n## Course staff\nInsturctor: Chenhao Tan [contact](mailto:Chenhao.Tan@colorado.edu)      \nOffice hours: 2:15-3:00pm on Monday, 12:00-1:00pm on Wednesday, or by appointment (ECES 118A)\n\n\n## Logistics\n\n* Location and time: ECES 112, 1:00-2:15pm on Mondays and Wednesdays\n* [Syllabus](syllabus.md) (Must READ if you are taking the course)\n* [Piazza](https://piazza.com/class/k52x8u1t89s10e)\n\n\nSchedule\n===========================\n\n* Week 1: Introduction\n\t* Jan 13, Introduction  \n\t* Jan 15, [Ask not what AI can do, but what AI should do: Towards a Framework of Task Delegability](https://arxiv.org/abs/1902.03245). Brian Lubars and Chenhao Tan. NeurIPS 2019.  \n\t_Additional reading_:\n\t\t* [Human-centered Machine Learning: a Machine-in-the-loop Approach](https://medium.com/@ChenhaoTan/human-centered-machine-learning-a-machine-in-the-loop-approach-ed024db34fe7), Chenhao Tan.\n\n* Week 2: You are not so Smart\n\t* Jan 20, MLK.\n\t* Jan 22, [Human Decisions and Machine Predictions](https://academic.oup.com/qje/article/doi/10.1093/qje/qjx032/4095198/Human-Decisions-and-Machine-Predictions#).\n\tJon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan. \nQuarterly Journal of Economics, 2018.   \n\t_Additional reading_:\n\t\t* [Judgment under Uncertainty: Heuristics and Biases. Amos Tversky and Daniel Kahneman](http://science.sciencemag.org/content/185/4157/1124), Science, 1974.\n\t\t* [Assessing Human Error Against a Benchmark of Perfection](https://arxiv.org/abs/1606.04956). Ashton Anderson, Jon Kleinberg, Sendhil Mullainathan. KDD 2016.\n\t\t* [Algorithm Aversion: People Erroneously Avoid Algorithms\nAfter Seeing Them Err](http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf). Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey, Journal of Experimental Psychology: General, 2014.\n\t\t* Podcast: [You are not so smart](https://youarenotsosmart.com/podcast/)\n\t\t* [The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter](https://chenhaot.com/papers/wording-for-propagation.html). Chenhao Tan, Lillian Lee, Bo Pang. ACL 2014.   \n\t\t* Thinking, Fast and Slow. Daniel Kahneman. 2011. \n\n\t\t\n* Week 3: Machine-in-the-loop Interactions\n\t* Jan 27, [Principles of Mixed-Initiative User Interfaces](http://erichorvitz.com/chi99horvitz.pdf). Eric Horvitz. In Proceedings of CHI, 1999.\n\t* Jan 29, [Towards A Rigorous Science of Interpretable Machine Learning](https://arxiv.org/abs/1702.08608). Finale Doshi-Velez and Been Kim.   \n\t_Additional reading_:  \n\t\t* [Guidelines for Human-AI Interaction](https://www.microsoft.com/en-us/research/uploads/prod/2019/01/Guidelines-for-Human-AI-Interaction-camera-ready.pdf). Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson,\nJina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. CHI 2019.\n\t\t* [Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda](https://dl.acm.org/doi/10.1145/3173574.3174156).  Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, Mohan Kankanhalli. CHI 2018.\n\t\t* [Interpretable machine learning: definitions, methods, and applications](https://arxiv.org/abs/1901.04592). W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, Bin Yu. PNAS 2019.\n\t\t* [Natural Language Translation at the Intersection of AI and HCI](https://dl.acm.org/citation.cfm?id=2798086). Spence Gree, Jeffrey Heer, Christopher D. Manning. Queue, 2015.    \n\t\t* [A Review of User Interface Design for Interactive Machine Learning](https://www.repository.cam.ac.uk/bitstream/handle/1810/274032/TIIS_Special_Issue_IML_Survey.pdf). John J. Dudley and Per Ola Kristensson. ACM Transactions on Interactive Intelligent Systems. 2018.\n\t\t* [Beyond binary choices: Integrating individual and social creativity](https://www.sciencedirect.com/science/article/pii/S1071581905000479). Gerhard Fischer, Elisa Giaccardi, Hal Eden, Masanori Sugimoto, Yunwen Ye. International Journal of Human-Computer Studies, 2005.  \n\t\t* [The Mythos of Model Interpretability](https://arxiv.org/abs/1606.03490), Zachary C. Lipton.  \n\t\t* [A Human-Centered Agenda for Intelligible Machine Learning](http://www.jennwv.com/papers/intel-chapter.pdf). Jennifer Wortman Vaughan and Hanna Wallach.  \n\t\t* [Who is the “Human” in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media](http://steviechancellor.com/wp-content/uploads/2019/09/HCML-CSCW-2019.pdf). Stevie Chancellor, Eric P.S Baumer, and Munmun De Choudhury. CSCW 2019.\n\t\t* [Human Centered Systems in the Perspective of Organizational and Social Informatics](https://scholarworks.iu.edu/dspace/bitstream/handle/2022/1798/wp97-04B.html). Rob Kling and Leigh Star. 1997.   \n\t\u003cspan style=\"color:red\"\u003e**Replication playground/paper critique due on Jan 31; although I accept this homework one week late.**\u003c/span\u003e\n\n\n* Week 4: Feature-based Explanations\n\t* Feb 3, [Why should I trust you?: Explaining the Predictions of Any Classifier](https://arxiv.org/abs/1602.04938). Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. KDD 2016.\n\t* Feb 5, [Attention is not not explanations](https://arxiv.org/abs/1908.04626). Sarah Wiegreffe, Yuval Pinter. EMNLP 2019.    \n\t_Additional reading_:\n\t\t* [A Unified Approach to Interpreting Model Predictions](https://arxiv.org/abs/1705.07874). Scott Lundberg, Su-In Lee. NeurIPS 2017.\n\t\t* [Attention is not explanations](https://arxiv.org/abs/1902.10186) Sarthak Jain, Byron C. Wallace. NAACL 2019.\n\t\t* [Is Attention Interpretable?](https://arxiv.org/abs/1906.03731). Sofia Serrano, Noah A. Smith. ACL 2019.\n\t\t* [Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification](https://arxiv.org/abs/1910.08534). Vivian Lai, Jon Z. Cai, Chenhao Tan. EMNLP 2019.\n\t\t* [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://arxiv.org/abs/1705.06640), Pei, Cao, Yang and Jana. In Proceedings of SOSP, 2017.\n\t\t* [Rationalizing Neural Predictions](https://people.csail.mit.edu/taolei/papers/emnlp16_rationale.pdf), Tao Lei, Regina Barzilay and Tommi Jaakkola. In Proceedings of EMNLP, 2016.\n\t\t* [Learning Explanatory Rules from Noisy Data](https://arxiv.org/abs/1711.04574). Richard Evans, Edward Grefenstette. Journal of Artificial Intelligence Research, 2018.\n\t\t* [Network Dissection: Quantifying Interpretability of Deep Visual Representations](http://netdissect.csail.mit.edu/final-network-dissection.pdf). David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. In Proceedings of CVPR 2017.    \n\t\u003cspan style=\"color:red\"\u003e**First proposal due on Feb 7**\u003c/span\u003e\n\n* Week 5: First Proposal\n\t* Feb 10, presentation \u0026 discussion\n\t* Feb 12, presentation \u0026 discussion    \n\t\u003cspan style=\"color:red\"\u003e**First proposal peer feedback due on Feb 14.**\u003c/span\u003e\n\t\n* Week 6: Example-based explanations\n\t* Feb 17, [Examples are not Enough, Learn to Criticize!\nCriticism for Interpretability](http://papers.nips.cc/paper/6300-examples-are-not-enough-learn-to-criticize-criticism-for-interpretability.pdf). Been Kim, Rajiv Khanna, Oluwasanmi Koyejo. NeurIPS 2016.\n\t* Feb 19, [Deep Weighted Averaging Classifiers](https://arxiv.org/abs/1811.02579). Dallas Card, Michael Zhang, Noah A. Smith. FAT* 2019.    \n\t_Additional reading_:\n\t\t* [Case-based explanation of non-case-based learning methods](https://www.ncbi.nlm.nih.gov/pubmed/10566351). Rich Caruana, Hooshang Kangarloo, John David\nN. Dionisio, Usha Sinha, David Johnson. AMIA 1999.\n\t\t* [How Case Based Reasoning Explained Neural Networks: An XAI Survey of Post-Hoc Explanation-by-Example in ANN-CBR Twins](https://arxiv.org/abs/1905.07186). Mark T Keane, Eoin M Kenny. ICCBR 2019.\n\t\t* [Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning](https://arxiv.org/abs/1803.04765). Nicolas Papernot, Patrick McDaniel. \n\t\t* [Interactive and Interpretable Machine Learning Models for Human Machine Collaboration](http://people.csail.mit.edu/beenkim/papers/BKimPhDThesis.pdf), Been Kim, PhD thesis.    \n\t\u003cspan style=\"color:red\"\u003e**Second proposal due on Feb 21.**\u003c/span\u003e\n\n* Week 7: Second proposal\n\t* Feb 24, presentation \u0026 discussion\n\t* Feb 26, presentation \u0026 discussion  \n\t\u003cspan style=\"color:red\"\u003e**Second proposal peer feedback due on Feb 28.**\u003c/span\u003e   \n\n* Week 8: Counterfactual explanations\n\t* Mar 2, [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://arxiv.org/abs/1711.00399). Sandra Wachter, Brent Mittelstadt, Chris Russell.\n\t* Mar 4, [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://arxiv.org/abs/1905.07697). Ramaravind Kommiya Mothilal, Amit Sharma, Chenhao Tan. FAT* 2020.    \n\tAdditional reading   \n\t\t* [The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons](https://arxiv.org/abs/1912.04930). Solon Barocas, Andrew D. Selbst, Manish Raghavan. FAT* 2020.\n\t\t* [Efficient Search for Diverse Coherent Explanations](https://arxiv.org/abs/1901.04909). Chris Russell. FAT* 2019.\n\t\u003cspan style=\"color:red\"\u003e**Team formation due on Mar 7.**\u003c/span\u003e\n\n* Week 9: Adversarial attacks\n\t* Mar 9, [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://arxiv.org/abs/1908.07125). Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh. EMNLP 2019.\n\t* Mar 11, [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://arxiv.org/abs/1911.02508). Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, Himabindu Lakkaraju. AIES 2020.    \n\t_Additional reading_:\n\t\t* [Misleading Failures of Partial-input Baselines](https://arxiv.org/abs/1905.05778). Eric Wallace, Shi Feng, and Jordan Boyd-Graber. ACL 2019. \n\n* Week 10: Human-AI interaction --- Decision Making\n\t* Mar 16, [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://arxiv.org/abs/1811.07901). Vivian Lai, Chenhao Tan. FAT* 2019.\n\t* Mar 18, [The Principles and Limits of Algorithm-in-the-Loop Decision Making](https://www.benzevgreen.com/wp-content/uploads/2019/09/19-cscw.pdf). Ben Green, Yiling Chen. CSCW 2019.        \n\t_Additional reading_: \n\t\t* [Prediction Policy Problems](https://www.aeaweb.org/articles?id=10.1257/aer.p20151023). Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, Ziad Obermeyer. American Economic Review, 2015.  \n\t\t* [The Tyranny of Data? The Bright and Dark Sides of Data-Driven Decision-Making for Social Good](https://link.springer.com/chapter/10.1007/978-3-319-54024-5_1). Bruno Lepri, Jacopo Staiano, David Sangokoya, Emmanuel Letouzé, and Nuria Oliver. Transparent Data Mining for Big and Small Data, 2017.  \n\t\t* [Predicting the knowledge–recklessness distinction in the human brain](http://www.pnas.org/content/114/12/3222.full). Iris Vilares, Michael J. Wesley, Woo-Young Ahn, Richard J. Bonnie, Morris Hoffman, Owen D. Jones, Stephen J. Morse, Gideon Yaffe, Terry Lohrenz, and P. Read Montague. PNAS, 2016. \n\n* Week 11: Spring break\n\n\n* Week 12: Human-AI interaction --- Creative writing\n\t* Mar 30, [Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories](https://chenhaot.com/papers/creative-writing-with-a-machine-in-the-loop.html). In Proceedings of IUI, 2017.\n\t* Apr 1, [Creative Help: A Story Writing Assistant](http://people.ict.usc.edu/~gordon/publications/ICIDS15.PDF). Melissa Roemmele, Andrew S. Gordon. In Proceedings of ICIDS, 2015.   \n\t_Additional reading_:    \n\t\t* [Inside Jokes: Identifying Humorous Cartoon Captions](http://www.cs.huji.ac.il/~dshahaf/pHumor.pdf). Dafna Shahaf, Eric Horvitz, Robert Mankoff. In Proceedings of KDD, 2015.\n\t\t* [Hafez: an Interactive Poetry Generation System](http://xingshi.me/data/pdf/ACL2017demo.pdf). Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. In Proceedings of ACL, 2017 (Demo Track).\n\n* Week 13: Project midpoint presentation\n\t* Apr 6, free time to work on projects\n\t* Apr 8, presentation   \n\t\u003cspan style=\"color:red\"\u003e**Project peer feedback due on Apr 10.**\u003c/span\u003e \n\n* Week 14: Human-AI interaction --- Trust\n\t* Apr 13, [Trust in automation: Designing for appropriate reliance](https://journals.sagepub.com/doi/10.1518/hfes.46.1.50_30392). John Lee and Katrina See. Human factors, 2004.\n\t* Apr 15, [Understanding the Effect of Accuracy on Trust in\nMachine Learning Models](http://mingyin.org/CHI-19/accuracy.pdf). Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. CHI 2019.     \n\t_Additional reading_:\n\t\t* [Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems](https://dl.acm.org/doi/10.1145/3290605.3300717). Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. CHI 2019.\n\n\n* Week 15: Fairness, Accountability, and Transparency\n\t* Apr 20, [European Union regulations on algorithmic decision-making and a \"right to explanation\"](https://arxiv.org/pdf/1606.08813.pdf). Bryce Goodman and Seth Flaxman.\n\t* Apr 22, [Roles for Computing in Social Change](https://arxiv.org/abs/1912.04883). Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David G. Robinson. FAT* 2020.       \n\t_Additional reading_:  \n\t\t* [Equality of opportunity in supervised learning](https://arxiv.org/abs/1610.02413). Moritz Hardt, Eric Price, Nathan Srebro. NeurIPS 2016.\n\t\t* [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://arxiv.org/abs/1609.05807). Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan. ITCS 2017.   \n\t\t* [Algorithmic decision making and the cost of fairness](https://arxiv.org/abs/1701.08230). Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Huq. KDD 2017. \n\t\t* [Fairness and Abstraction in Sociotechnical Systems](https://dl.acm.org/doi/10.1145/3287560.3287598). Andrew D. Selbst, danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, Janet Vertesi. FAT* 2019.\n\t\n* Week 16: Final Project Presentation  \n\t* TBD, presentations (could be different from normal class time).   \n\u003cspan style=\"color:red\"\u003e**Final project report due on May 1.**\u003c/span\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBoulderDS%2Fhuman-centered-machine-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBoulderDS%2Fhuman-centered-machine-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBoulderDS%2Fhuman-centered-machine-learning/lists"}