Generate music using Meta’s MusicGen on Colab

Introduction

In the vast realm of artificial intelligence, deep learning has revolutionized numerous domains, including natural language processing, computer vision, and speech recognition. However, one fascinating area that has captivated researchers and music enthusiasts alike is the generation of music using artificial intelligence algorithms. MusicGen, a state-of-the-art, controllable text-to-music model that seamlessly translates textual prompts into captivating musical compositions.

What is MusicGen?

MusicGen is a remarkable model designed for music generation that offers simplicity and controllability. Unlike existing methods such as MusicLM, MusicGen stands out by eliminating the need for a self-supervised semantic representation. The model employs a single-stage auto-regressive Transformer architecture and is trained using a 32kHz EnCodec tokenizer. Notably, MusicGen generates all four codebooks in a single pass, setting it apart from conventional approaches. By introducing a slight delay between the codebooks, the model demonstrates the ability to predict them in parallel, resulting in a mere 50 auto-regressive steps per second of audio. This innovative approach optimizes the efficiency and speed of the music generation process.

MusicGen is trained on 20k hours of licensed music. They also trained it on the internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.

Pre-requisites:

As per the official MusicGen GitHub repo https://github.com/facebookresearch/audiocraft/tree/main.

Available MusicGen models

There are 4 pre-trained models available and they are as follows:

• small: 300M model, Text to music only
• medium: 1.5B model, Text to music only
• melody: 1.5B model, Text to music and text+melody to music
• large: 3.3B model, Text to music only

Experiments

Below is the output of the Conditional music generation using the MusicGen large model.

Text Input: Jingle bell tune with violin and piano

Output: (Using MusicGen "large" model)

Note: As the audio file is not supported on Medium, to check the output go to the original site where the blog is published.

Below is the output of the MusicGen “melody” model. We used the above audio and text input to generate the following audio.

Text Input: Add heavy drums drums and only drums

Output: (Using MusicGen "melody" model)

Note: As the audio file is not supported on Medium, to check the output, go to the original site where the blog is published.

How to setup the MusicGen on Colab

Make sure you are using a GPU for faster inference. It took ~9 minutes to generate 10 seconds of audio using the CPU whereas using GPU(T4) it took just 35 seconds.

• Before starting make sure torch and torchaudio are installed in the colab.

Install the audiocraft library from Facebook.

!python3 -m pip install -U git+https://github.com/facebookresearch/audiocraft#egg=audiocraftImport necessary libraries.

Import necessary libraries.

from audiocraft.models import musicgen
from audiocraft.utils.notebook import display_audio
import torchfrom audiocraft.data.audio import audio_write

Load the model
The list of models is as follows:

# | model types are => small, medium, melody, large |
# | size of models are => 300M, 1.5B, 1.5B, 3.3B |
model = musicgen.MusicGen.get_pretrained('large', device='cuda')

Set the parameters (Optional)

model.set_generation_params(duration=60) # this will generate 60 seconds of audio.

Conditional Music Generation ( Generate the music by providing text. )

res = model.generate_unconditional( num_samples=1, progress=True)
# this will show the music controls on the screendisplay_audio(res, 16000)

To generate unconditional music

res = model.generate_unconditional( num_samples=1, progress=True)
# this will show the music controls on the screendisplay_audio(res, 16000)

To generate music continuation

To create a music continuation, we will need an audio file. We will feed that file to the model, and the model will generate and add more music to it.

from audiocraft.utils.notebook import display_audio
import torchaudio


path_to_audio = "path-to-audio-file.wav"
description = "Jazz jazz and only jazz"


# Load audio from a file. Make sure to trim the file if it is too long!
prompt_waveform, prompt_sr = torchaudio.load( path_to_audio )
prompt_duration = 15
prompt_waveform = prompt_waveform[..., :int(prompt_duration * prompt_sr)]
output = model.generate_continuation(prompt_waveform, prompt_sample_rate=prompt_sr,
descriptions=[ description ], progress=True)
display_audio(output, sample_rate=32000)

To generate melody conditional generation

model = musicgen.MusicGen.get_pretrained('melody', device='cuda')


model.set_generation_params(duration=20)


melody_waveform, sr = torchaudio.load("path-to-audio-file.wav")
melody_waveform = melody_waveform.unsqueeze(0).repeat(2, 1, 1)
output = model.generate_with_chroma(
descriptions=['Add heavy drums'], melody_wavs=melody_waveform, melody_sample_rate=sr,progress=True)
display_audio(output, sample_rate=32000)

Write the audio file to the disk.

If you want to download the file from the colab then you will need to write the wav file on the disk. Here is the function that will write a wav file onto the disk. It will take the model output as a first input and the filename as a second input.

def write_wav(output, file_initials):
    try:
        for idx, one_wav in enumerate(output):
        audio_write(f'{file_initials}_{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
        return True
    except Exception as e:
        print("error while writing the file ", e)
        return None


# this will write a file that starts with bollywood
write_wav(res, "audio-file")

Full Implementation (Google colab file link)

Full implementation of the Meta’s MusicGen library by Pragnakalp Techlabs is given in the colab file. Feel free to explore and create music using it.
Pragnakalp Techlabs | Meta’s MusicGen Implementation

Conclusion

In conclusion, Audiocraft’s MusicGen is a powerful and controllable music generation model. Looking ahead, Audiocraft holds exciting future potential for advancements in AI-generated music. Whether you’re a musician or an AI enthusiast, Audiocraft’s MusicGen opens up a world of creative possibilities.

Originally published at Generate Music Using Meta’s MusicGen On Colab on June 16, 2023.