Participatory Approaches to Machine Learning

ICML 2020 Workshop (July 17)

@misc{paml,
    author={Kulynych, Bogdan and
            Madras, David and
            Milli, Smitha and
            Raji, Inioluwa Deborah and
            Zhou, Angela, and
            Zemel, Richard},
    title={Participatory Approaches to Machine Learning},
    howpublished={International Conference on Machine Learning Workshop},
    month=July,
    year=2020
}

Overview

The designers of a machine learning (ML) system typically have far more power over the system than the individuals who are ultimately impacted by the system and its decisions. Recommender platforms shape the users’ preferences; the individuals classified by a model often do not have means to contest a decision; and the data required by supervised ML systems necessitates that the privacy and labour of many yield to the design choices of a few.

The fields of algorithmic fairness and human-centered ML often focus on centralized solutions, lending increasing power to system designers and operators, and less to users and affected populations. In response to the growing social-science critique of the power imbalance present in the research, design, and deployment of ML systems, we wish to consider a new set of technical formulations for the ML community on the subject of more democratic, cooperative, and participatory ML systems.

Our workshop aims to explore methods that, by design, enable and encourage the perspectives of those impacted by an ML system to shape the system and its decisions. By involving affected populations in shaping the goals of the overall system, we hope to move beyond just tools for enabling human participation and progress towards a redesign of power dynamics in ML systems.

Topics

We invite work that makes progress on the workshop themes, including but not limited to:

• Recourse and reachability: Creating systems that ensure the user has the potential to adapt their predictions or recommendations (Ustun, Spangher, and Liu 2019; Dean, Rich, and Recht 2020).

• Richer interactive ML: Incorporating richer user feedback into ML systems. Limitations of current methods in fully capturing a user’s ‘preferences’ (Schnabel, Bennett, and Joachims 2018; Yang et al. 2019)

• Mechanism design or computational social choice and learning: designs that incorporate preference elicitation in allocative ML systems, such as adaptive experimentation or learning-based algorithmic governance (Narita 2019; M. K. Lee et al. 2019; Kahng et al. 2019)

• Collective and participatory design applied to community involvement in ML: Qualitative and quantitative methods and frameworks aimed at giving a voice to communities affected by ML systems. Tools for collective training processes and collective design of ML systems (Katell et al. 2020; Brown et al. 2019; Halfaker and Geiger 2019; Patton et al. 2020).

• Deferral and/or abstention: Classification frameworks which incorporate deferral to humans, or at least a non-classification outcome. (Madras, Pitassi, and Zemel 2018; El-Yaniv and Wiener 2010).

• Documentation & audit methods: Work which is intended to inform and engage users in model design, development or deployment processes and its limitations (Raji et al. 2020; Mitchell et al. 2019).

• Contestation: Technological methods and tools for analyzing/protesting/contesting the outcomes of ML systems in the absence of centralized cooperation  (Kulynych & Overdorf et al. 2020).

• Analysis or audits of amplifiers of systemic injustice in decision systems (Arnold et al. 2020; Pierson et al. 2020; Obermeyer et al. 2019; Ali & Sapieżyński et al. 2019; Raji and Buolamwini 2019; Buolamwini and Gebru 2018).

• Tools that support community and worker organization (Matias and Mou 2018; Irani and Silberman 2013; Salehi et al. 2015).

Schedule

The central part of our workshop is the livestream on the ICML virtual site. The livestream will broadcast invited talks and panels, as well as the latest announcements and possible schedule changes. The interactive discussions with other participants—poster and breakout sessions—will happen on our Discord server.

Poster Sessions

Each contributed paper has its own audio/video channel on our Discord server. During the “poster” session, participants can discuss the paper with its authors in the respective channel. Before joining a channel as a participant, please make sure to familiarize yourself with the paper and watch the paper’s short introduction video.

Poster Session 1 (2:45 PM - 3:30 PM UTC)

• #15 More than a label: machine-assisted data interpretation
• #17 Participation is Not a Design Fix for Machine Learning
• #22 Metric-Free Individual Fairness in Online Learning
• #29 Heuristic-Based Weak Learning for Automated Decision-Making
• #32 Contestable City Algorithms
• #33 Preference Elicitation and Aggregation to Aid with Patient Triage during the COVID-19 Pandemic
• #39 Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
• #41 Iterative Interactive Reward Learning
• #45 Can Algorithms Help Support Participatory Housing?
• #49 Recourse for Humans
• #5 The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
• #7 Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics
• #10 Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems
• #13 Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics

Poster Session 2 (7:15 PM - 8:00 PM UTC)

• #12 A Review of the UK-ICO’s Draft Guidance on the AI Auditing Framework
• #14 Is Machine Learning Speaking my Language? A Critical Look at the NLP-Pipeline Across 8 Human Languages
• #19 Interpretable Privacy for Deep Learning Inference
• #27 Designing Recommender Systems with Reachability in Mind
• #31 Deconstructing the Filter Bubble: User Decision-Making and Recommender Systems
• #37 What If I Don't Like Any Of The Choices? The Limits of Preference Elicitation for Participatory Algorithm Design
• #38 Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems
• #40 Fairness, Equality, and Power in Algorithmic Decision-Making
• #42 What are you optimizing for? Aligning Recommender Systems with Human Values
• #3 Adapting a kidney exchange algorithm to align with human values
• #6 Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity
• #11 Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives
• #30 Designing Fairly Fair Classifiers Via Economic Fairness Notions

Breakout Sessions

On our Discord server, you can chat with other workshop participants at any time in thematic text or audio/video channels; we will add channels during the day if topics come up. In addition to this, as an experiment, we will have thematic discussion sessions guided by facilitators:

Breakout Session 1 (3:30 PM - 4:15 PM UTC)

• Community Organization and ML Systems. Facilitated by Seda Gürses (TU Delft)
• Stakeholder Engagement in System Design. Facilitated by Haiyi Zhu (CMU)
• Recourse. Facilitated by Kristen Vaccaro (UIUC)

Breakout Session 2 (8:00 PM - 8:45 PM UTC)

• Community Organization and ML Systems. Facilitated by David Robinson (Cornell)
• Richer Interactive ML. Facilitated by Sarah Dean (Berkeley)

Contributed Papers

The contributed papers include both novel research papers as well as papers that have already been published in other venues (marked as “Encore”).

Qualitative frameworks, methods, and analyses

Algorithmic approaches and quantitative methods

Applications

Critical examinations

Policy

Invited Speakers

• Nicolas Papernot, University of Toronto/Vector Institute
• Alexandra Chouldechova, Carnegie Mellon University
• Tawana Petty, Detroit Community Technology Project
• Berk Ustun, Harvard University
• Aleksandra Korolova, University of Southern California
• Piotr Sapieżyński, Northeastern University
• Jamelle Watson-Daniels, Data for Black Lives

Call for Participation

We would like to experiment with breakout discussions to encourage group discussions around shared topics of interest. Please fill out our this form if you are interested in joining or organizing such a discussion.

Call for Papers

The workshop will include contributed papers. All accepted papers will be allocated either a virtual poster presentation, or a virtual talk slot. We will not publish proceedings, but will optionally link the papers and talk recordings on the workshop website.

We invite submissions in two tracks:

• Research Track. Full papers, works-in-progress, position papers, and case studies. We expect that these submissions introduce novel ideas or results.

  The papers should have up to 4 pages (excluding references, acknowledgements, or appendices), and be formatted using the ICML submission template. Papers should be anonymized.

• Encore Track. Papers that have already been accepted at other venues.

  There are no format requirements for this track. The papers should be accepted at another recognized archival conference or journal, and be submitted by one of the paper’s authors.

Submissions are currently closed.

For any questions, please send us an email to participatory-ml-organizers at lists.mayfirst.org

Timeline

• Submission deadline: June 22, 2020, AoE
• Notification: July 1, 2020
• Workshop: July 17, 2020

Organizing Committee

• Angela Zhou, Cornell University
• David Madras, University of Toronto
• Inioluwa Deborah Raji, AI Now Institute
• Bogdan Kulynych, EPFL
• Smitha Milli, UC Berkeley
• Richard Zemel, University of Toronto

Funding assistance

We are grateful to Open Philanthropy for providing funding assistance for the workshop. If you want to participate in the workshop and require funding, please write us an email to participatory-ml-organizers at lists.mayfirst.org