NeurIPS 2022 Tutorials (2024)

The Role of Meta-learning for Few-shot Learning

Foundational Robustness of Foundation Models

Foundation models adopting the methodology of deep learning with pre-training on large-scale unlabeled data and finetuning with task-specific supervision are becoming a mainstream technique in machine learning. Although foundation models hold many promises in learning general representations and few-shot/zero-shot generalization across domains and data modalities, at the same time they raise unprecedented challenges and considerable risks in robustness and privacy due to the use of the excessive volume of data and complex neural network architectures. This tutorial aims to deliver a Coursera-like online tutorial containing comprehensive lectures, a hands-on and interactive Jupyter/Colab live coding demo, and a panel discussion on different aspects of trustworthiness in foundation models.

Lifelong Learning Machines

Neurosymbolic Programming

This tutorial will provide an overview of recent advances in Neurosymbolic Programming. The objective in this area is to learn neurosymbolic programs, which combine elements of both neural networks and classical symbolic programs with the aim of inheriting the benefits of both. A key advantage of the neurosymbolic programmiing approach is that here, one learns models that look more like the models that domain experts write by hand in code, but that are also more expressive than classical interpretable models in machine learning. Also, neurosymbolic programs can more easily incorporate prior knowledge and are easier to analyze and verify. From the point of view of techniques, neurosymbolic programming combines ideas from machine learning and program synthesis and represents an exciting new contact point between the two communities. This tutorial will cover a broad range of basic concepts in the area, including neurosymbolic architectures, domain-specific languages, architecture/program search algorithms, meta-learning algorithms such as library learning, and applications to science and autonomy. Our panel will discuss open challenges in the field and ways in which machine learning and programming languages researchers can come together to address them. The tutorial is an abridged version of the tutorial at the Neurosymbolic Programming summer school …

Advances in NLP and their Applications to Healthcare

Recent advances in Natural Language Processing (NLP) have propelled the state of the art to new highs. One such advance is the use of external memory to support reasoning in deep learning models such as Transformers.
Without external memory to store sufficient background knowledge, reasoning in NLP systems must be performed based on limited information leading to poor performance on knowledge-rich tasks. Conversely, NLP systems with access to external memory have resulted in significant performance gains on many important tasks including question answering (QA) and other tasks associated with QA such as fact verification, and entity linking. The tutorial will present : 1) an overview of state of the art approaches for representing background knowledge in addressable memory, and 2) applications in the healthcare domain.

Probabilistic Circuits: Representations, Inference, Learning and Applications

In several real-world scenarios, decision making involves complex reasoning, i.e., the ability to answer complex probabilistic queries. Moreover, in many sensitive domains like health- care and economical decision making, the result of these queries is required to be exact as approximations without guarantees would make the decision making process brittle. In all these scenarios, tractable probabilistic inference and learning are becoming more and more mandatory. In this tutorial, we will introduce the framework of probabilistic circuits (PCs) under which one can learn deep generative models that guarantee exact inference in polynomial (often linear) time. After certain recent algorithmic and theoretical results, which we will discuss in this tutorial, PCs have achieved impressive results in probabilistic modeling, sometimes outperforming intractable models such as variational autoencoders. We will show the syntax and semantics of PCs and show how several commonly used ML models -- from Gaussian mixture models to HMMs and decision trees -- can be understood as computational graphs within the PC framework. We will discuss how PCs are special cases of neural networks, when restricting network with certain structural properties enables different tractability scenarios. This unified view of probabilistic ML models opens up a range of ways to learn PCs …

Algorithmic fairness: at the intersections

As machine learning models permeate every aspect of decision making systems in consequential areas such as healthcare, banking, hiring and education, it has become critical for these models to satisfy trustworthiness desiderata such as fairness, privacy, robustness and interpretability. Initially studied in isolation, recent work has emerged at the intersection of these different fields of research, leading to interesting questions on how fairness can be achieved under privacy, interpretability and robustness constraints. Given the interesting questions that emerge at the intersection of these different fields, this tutorial aims to investigate how these different topics relate, and how they can augment each other to provide better or more suited definitions and mitigation strategies for algorithmic fairness. We are particularly interested in addressing open questions in the field, such as: how algorithmic fairness is compatible with privacy constraints? What are the trade-offs when we consider algorithmic fairness at the intersection of robustness? Can we develop fair and explainable models? We will also articulate some limitations of technical approaches to algorithmic fairness, and discuss critiques that are coming from outside of computer science.

Advances in Bayesian Optimization

Incentive-Aware Machine Learning: A Tale of Robustness, Fairness, Improvement, and Performativity

When an algorithm can make consequential decisions for people's lives, people have an incentive to respond to the algorithm strategically in order to obtain a more desirable decision. This means that unless the algorithm adapts to this strategizing, it may end up creating policy decisions that are incompatible with the original policy's goal. This has been the mantra of the rapidly growing research area of incentive-aware Machine Learning (ML). In this tutorial, we introduce this area to the broader ML community. After a primer on the basic background needed, we introduce the audience to the four perspectives that have been studied so far: the robustness perspective (where the decision-maker tries to create algorithms that are robust to strategizing), the fairness perspective (where we study the inequalities that arise or are reinforced as a result of strategizing), the improvement perspective (where the learner tries to incentivize effort exertion towards actually improving their points), and the performativity perspective (where the decision-maker wishes to achieve a notion of stability in these settings).

Data Compression with Machine Learning

The efficient communication of information has enormous societal and environmental impact, and stands to benefit from the machine learning revolution seen in other fields. Through this tutorial, we hope to disseminate the ideas of information theory and compression to a broad audience, overview the core methodologies in learning-based compression (i.e., neural compression), and present the relevant technical challenges and open problems defining a new frontier of probabilistic machine learning. Besides covering the technical grounds, we will also explore the broader underlying themes and future research in our panel discussion, focusing on the interplay between computation and communication, the role of machine learning, and societal considerations such as fairness, privacy, and energy footprint as we strive to make our learning and information processing systems more efficient.

Creative Culture and Machine Learning

Fair and Socially Responsible ML for Recommendations: Challenges and Perspectives

Algorithmic rankings have become increasingly common in the online world. From social media to streaming services and from e-commerce to hiring, ranking has become the primary way online users ingest information. In many cases, recommendations have implications for both the users and items (or creators) being ranked. These systems are also increasingly personalized to viewers, relying on imperfect information to learn and respond to user preferences. In recent years, the machine learning community has become increasingly aware of potential harms to consumers (e.g. echo chambers, addictive design, virality of harmful content) and creators (e.g. access to opportunity, misattribution and appropriation). In this tutorial we will explore the current state of research on responsible recommendations and the primary challenges with understanding, evaluating and training these systems for user and content providers. This tutorial additionally presents the primary challenges in applying this research in practice. The perspectives and methods presented in this tutorial will apply to recommendation systems generally and will not include any specific information regarding actual recommendation products. The tutorial is designed to be accessible to a broad audience of machine learning practitioners, some background in predictive systems and ranking is beneficial but not required.

Theory and Practice of Efficient and Accurate Dataset Construction

Data is one of the key drivers of progress in machine learning. Modern datasets require scale far beyond the ability of individual domain experts to produce. To overcome this limitation, a wide variety of techniques have been developed to build large datasets efficiently, including crowdsourcing, automated labeling, weak supervision, and many more. This tutorial describes classical and modern methods for building datasets beyond manual hand-labeling. It covers both theoretical and practical aspects of dataset construction. Theoretically, we discuss guarantees for a variety of crowdsourcing, active learning-based, and weak supervision techniques, with a particular focus on generalization properties of downstream models trained on the resulting datasets. Practically, we describe several popular systems implementing such techniques and their use in industry and beyond. We cover both the promise and potential pitfalls of using such methods. Finally, we offer a comparison of automated dataset construction versus other popular approaches to dealing with a lack of large amounts of labeled data, including few- and zero-shot methods enabled by foundation models.