Recently, large language models (LM) have been shown to sample mostly coherent long-form text. This astonishing level of fluency has driven an increasing interest to understand how these models work and, in particular, how to interpret and evaluate them. Additionally, the growing use of sophisticated LM frameworks has lowered the threshold for users to train newmodels or to fine-tune existing models for transfer learning. However, selecting the best LM from the expanding selection of pre-trained deep LM architectures is challenging, as there are few tools available to qualitatively compare models for specialized use-cases, e.g. to answer questions like: "What parts of a domain specific text can the fine-tuned model capture better than the general model?"

We introduce LMdiff: an interactive visual analysis tool for comparing LMs by qualitatively inspecting concrete samples generated by another model or drawn from a reference corpus. We provide an offline method to search for interesting samples, a live demo, and source code for the demo session that supports multiple models and allows users to upload their own example text.

Human-in-the-loop data labeling is generally considered a tedious, error-prone and expensive activity. Automation of the labeling task is desirable, but current approaches can conflict with principles of trust and human agency. We are developing a data labeling experience where the human labeler transparently interacts with an AI assistant to reach automation readiness, at which point the remainder of the labeling task can be delegated to a virtual assistant. Our approach combines semi-supervised learning, active learning, and human-machine decision tracking to reduce labeling effort and support reliable automation. The demo takes participant through an online end-to-end AI assisted data labeling experience, starting with manual labeling, then assisted labeling and ultimately transitioning to automated labeling via a system of readiness checkpoints.

Automated extraction and analysis of key information from unstructured documents is a central problem in information retrieval. Businesses are often inundated with large volumes of documents like financial statements, contracts and agreements, invoices and customer lists, which are generally meant for human comprehension and consumption, and hence automation becomes non-trivial.

Currently, the information from such documents is extracted by some combination of manual work and proprietary scripts that break often as something changes, leading to low efficiency, high labor cost, and inconsistencies in the output. Investment banks, fund managers, marketing agencies, and investors spend millions to either buy the data or outsource the whole process, while the data is available publicly for free. We describe a capability for automated extraction and real-time analysis of datasets from a large corpus of documents containing running text and tables. Current version of our product works with millions of HTML documents from Securities and Exchange Commission (SEC) filings. These filings contain mandatory disclosures like financial information, executive compensation, mergers and acquisitions and key management changes from US corporations.

Our algorithm extracts information from millions of documents, normalizes and stores it in an efficient queryable format, interprets input queries and looks up relevant documents to compose an answer.

We demonstrate GNOME (Generating Novelty in Open-world Multi-agent Environments), an experimental platform that is designed to test the effectiveness of multi-agent AI systems when faced with novelty. GNOME separates the development of AI gameplaying agents with the simulator, allowing unanticipated novelty (in essence, novelty that is not subject to model-selection bias). Through the demonstration, we also hope to foster an open discussion on AI robustness and the nature of novelty in real-world environments. GNOME will employ a creative audience-interaction methodology well-suited to a virtual conference, as we will expose the facilities of the simulator (including live simulation) through a Web GUI.

Even with the rise of unsupervised learning and weak supervision techniques, human-labeled data is still a necessary part of machine learning pipelines in many real-world contexts and applications. This often involves using crowdworkers for the laborious task of labeling large amounts of data. This is a largely asynchronous process and can lead to conflict among the workers, where individual labelers potentially submit labels in disagreement from each other for a given data item. When such noisy data is fed to a machine learning model, the accuracy and performance (on test data) of the overall system can suffer. One popular workaround is to entirely discard the data items with conflict. This however, leads to wastage of expensive, human-supplied data. Moreover, the data points with conflicting labels often are the data points which are crucial in determining the decision boundaries for the model itself. Another possibility is to automate conflict resolution. Here however, given humans themselves are in disagreement, state-of-the-art models can not be expected to reliably solve the problem. In practice therefore, it becomes imperative for a human to step in and resolve the conflict. Given conflict resolution is a non-trivial task, assistance of expensive subject matter experts (SMEs) is required. To help manage the SME’s time more efficiently, we propose an intelligent approach to resolve label conflicts by automatically re-ranking the conflicts in such an order that the conflicts with the most missing information useful to the model are displayed first, complete with ML assistance to auto-resolve easy conflicts, and explanations for justifying decisions and improving explainability.

We propose a demonstration of our novel DeepRacing framework at NeurIPS 2020 as a platform for training and evaluating high-speed autonomous race cars. DeepRacing uses the immensely popular and photo-realistic Formula One racing game and converts it into a simulation environment for autonomous racing.

We will demo both the ability to autonomously race the F1 car in the game using control inputs predicted by machine learned driving policies as well as tag images of the driver's point-of-view with various state information (such as the position, velocity, and control values for the racing agents) to enable generation of labelled datasets for supervised machine learning. We will demonstrate this technology in a real-time web broadcast with interactive inputs from the NeurIPS audience.

Collaborative intelligence is a technique for using more than one computing device to perform a computational task. A possible application of this technique is to assist mobile client edge devices in performing inference of deep learning models by sharing the workload with a server. In one typical setup, the mobile device performs a partial inference of the model, up to an intermediate layer. The output tensor of this intermediate layer is then transmitted over a network (e.g. WiFi, LTE, 3G) to a server, which completes the remaining inference, and then transmits the result back to the client. Such a strategy can reduce network usage, resulting in reduced bandwidth costs, lower energy consumption, faster inference, and provide better privacy guarantees. A working implementation of this was shown in our demo at NeurIPS 2019. This year, we present a library that will enable researchers and developers to create collaborative intelligence systems themselves quickly and easily.This demo presents a new library for developing and deploying collaborative intelligence systems. Computational and communication subprocesses are expressed as a directed acyclic graph. Expressing the entire process as a computational graph provides several advantages including modularity, graph serializability and transmission, and easier scheduling and optimization. Library features include: graph definition via a functional API inspired by Keras and PyTorch, over-the-network execution of graphs that span across multiple devices, API for Android (Kotlin/Java) edge clients and servers (Python), integration with Reactive Extensions (Rx), optimal scheduling for low latency and high throughput, asynchronous execution and multi-threading support, backpressure handling, and modules for network transmission of compressed feature tensor data.

In this demo we propose a compositional tool that generates musical sequences based on prosody of speech recorded by the user. The tool allows any user–-regardless of musical training--to use their own speech to generate musical melodies, while hearing the direct connection between their recorded speech and resulting music. This is achieved with a pipeline combining speech-based signal processing [1,2], musical heuristics, and a set of transformer models [3,4] trained for new musical tasks. Importantly, the pipeline is designed to work with any kind of speech input and does not require a paired dataset for the training of the said transformer model.

Our approach consists of the following steps:

1. Estimate the F0 values and loudness envelope of the speech signal.
2. Convert this into a sequence of musical constraints derived from the speech signal.
3. Apply one or more transformer models—each trained on different musical tasks or datasets—to this constraint sequence to produce musical sequences that follow or accompany the speech patterns in a variety of ways.

The demo is self-explanatory: the audience can interact with the system by either providing a live-recording using a web-based recording interface or by uploading a pre-recorded speech sample. The system then provides a visualization of the formant contours extracted from the provided speech sample, the set of note constraints obtained from the speech, and the sequence of musical notes as generated by the transformers. The audience can also listen to—and interactively mix the levels (volume) of—the input speech sample, initial note sequences, and the musical sequences as generated by the transformer models.

https://jasondeon.github.io/musicalSpeech/

[1] Rabiner & Huang. Fundamentals of speech recognition. [2] Dumpala et al. Sine-wave speech as pre-processing for downstream tasks. Symp. FRSM 2020 [3] Vaswani et al. Attention is all you need. NeurIPS 2017 [4] Huang et al, Music Transformer ICLR 2018

Explaining neural model predictions to users requires creativity. Especially in enterprise applications, where there are costs associated with users' time, and their trust in the model predictions is critical for adoption. For link prediction in master data management, we have built a number of explainability solutions drawing from research in interpretability, fact verification, path ranking, neuro-symbolic reasoning and self-explaining AI. In this demo, we present explanations for link prediction in a creative way, to allow users to choose explanations they are more comfortable with.

Gender Neutralization is an important task in text anonymization and generatively producing gender-free descriptions of people and objects. We demonstrate a web tool that utilizes Coreference Resolution and a heuristic to neutralize long gendered texts.

Project Website: https://projects.parthraghav.com/pronoun-pro/