Our exhibit shows an interactive video game with AI-generated music and visuals. Players can choose a collection of emotions that affect the game’s generated music. The game’s protagonist is an aspiring musician seeking to make their mark in the bustling New Orleans music scene. They navigate the city’s rich cultural tapestry, encountering iconic locations, hidden gems, and electrifying performances along the way.
The novelty of our game comes primarily from our music generation system, which consists of two major components: a sentiment-based harmonic progression generator and a music theory-informed generative framework for melody composition. The harmonic content, as well as orchestration, tempo, and dynamics, is determined using a novel model called SentimentComposer that is based on a mixture of emotions. This sentiment mixture is derived from the current game state. From a corpus of labeled music, the model learned a set of Markovian transition matrices, one for each emotion, and weighted each one by the proportions of the sentiment mixture. Sampling harmonies from the model is easy and highly interpretable.
Using the sentiment-driven harmonic progression, a second novel model, SchenkComposer, generates a melody probabilistically to play atop the harmony. SchenkComposer uses a novel probabilistic context-free grammar informed by musical form theory and Schenkerian analysis in order to generate music with a cohesive, long-term structure.
Beyond the music, the visual backdrops of the game are generated using Microsoft Bing’s image generator powered by DALLE 3. The text narrative was written in collaboration with ChatGPT.
New Orleans: An Adventure in Music celebrates New Orleans’ diverse cultural legacy through its music and iconic scenery. New Orleans is known for its musical fusion, blending jazz with various genres such as blues, gospel, classical, and more – to pay homage to this multitude of influences, our model is trained music from sultry blues to energetic ragtime. Players are able to create their own songs with a cohesive long-term structure influenced by New Orlean’s musical history through our musical model. The game’s intuitive mechanics make playing and creating music a joyous and rewarding experience for anyone. We believe that our project is an innovative, interpretable, and interactive love letter to the city’s diverse musical heritage that anyone can enjoy.