Cog Implementation of MusicGen-Remixer

MusicGen Remixer is an app based on MusicGen Chord, the modified version of Meta's MusicGen Melody model, which can generate music based on audio-based chord conditions or text-based chord conditions.

You can demo this model or learn how to use it with Replicate's API here.

Run with Cog

Cog is an open-source tool that packages machine learning models in a standard, production-ready container. You can deploy your packaged model to your own infrastructure, or to Replicate, where users can interact with it via web interface or API.

Prerequisites

Cog. Follow these instructions to install Cog, or just run:

sudo curl -o /usr/local/bin/cog -L "https://github.com/replicate/cog/releases/latest/download/cog_$(uname -s)_$(uname -m)"
sudo chmod +x /usr/local/bin/cog

Note, to use Cog, you'll also need an installation of Docker.

GPU machine. You'll need a Linux machine with an NVIDIA GPU attached and the NVIDIA Container Toolkit installed. If you don't already have access to a machine with a GPU, check out our guide to getting a GPU machine.

Step 1. Clone this repository

git clone https://github.com/sakemin/cog-musicgen-chord

Step 2. Run the model

To run the model, you need a local copy of the model's Docker image. You can satisfy this requirement by specifying the image ID in your call to predict like:

cog predict r8.im/sakemin/musicgen-remixer@sha256:a31601435459035e7c3b0b8e56db374ebf5ae483f2e9c7c25e41c6ac3761fe52 -i prompt="bossa nova" -i music_input=@/your/path/to/input/music.wav

For more information, see the Cog section here

Alternatively, you can build the image yourself, either by running cog build or by letting cog predict trigger the build process implicitly. For example, the following will trigger the build process and then execute prediction:

cog predict -i prompt="bossa nova" -i music_input=@/your/path/to/input/music.wav

Note, the first time you run cog predict, model weights and other requisite assets will be downloaded if they're not available locally. This download only needs to be executed once.

Run on replicate

Step 1. Ensure that all assets are available locally

If you haven't already, you should ensure that your model runs locally with cog predict. This will guarantee that all assets are accessible. E.g., run:

cog predict -i prompt="bossa nova" -i music_input=@/your/path/to/input/music.wav

Step 2. Create a model on Replicate.

Go to replicate.com/create to create a Replicate model. If you want to keep the model private, make sure to specify "private".

Step 3. Configure the model's hardware

Replicate supports running models on variety of CPU and GPU configurations. For the best performance, you'll want to run this model on an A100 instance.

Click on the "Settings" tab on your model page, scroll down to "GPU hardware", and select "A100". Then click "Save".

Step 4: Push the model to Replicate

cog login

Push the contents of your current directory to Replicate, using the model name you specified in step 1:

cog push r8.im/username/modelname

Learn more about pushing models to Replicate.

Prediction

Prediction Parameters

model_version: Model type. Computations take longer when using large or stereo models.
prompt: A description of the music you want to generate.
music_input: An audio file input for the remix.
multi_band_diffusion: If True, the EnCodec tokens will be decoded with MultiBand Diffusion. Not compatible with stereo models.
normalization_strategy: Strategy for normalizing audio.
beat_sync_threshold: When beat syncing, if the gap between generated downbeat timing and input audio downbeat timing is larger than beat_sync_threshold, consider the beats are not corresponding.
chroma_coefficient: Coefficient value multiplied to multi-hot chord chroma.
top_k: Reduces sampling to the k most likely tokens.
top_p: Reduces sampling to tokens with cumulative probability of p. When set to 0 (default), top_k sampling is used.
temperature: Controls the 'conservativeness' of the sampling process. Higher temperature means more diversity.
classifier_free_guidance: Increases the influence of inputs on the output. Higher values produce lower-varience outputs that adhere more closely to inputs.
output_format: str = Output format for generated audio. "wav", "mp3"
seed: Seed for random number generator. If None or -1, a random seed will be used.

Multi-Band Diffusion

Multi-Band Diffusion(MBD) is used for decoding the EnCodec tokens.
If the tokens are decoded with MBD, than the output audio quality is better.
Using MBD takes more calculation time, since it has its own prediction sequence.

References

Chord recognition from audio file is performed using BTC model, by Jonggwon Park.
- Paper : A Bi-Directional Transformer for Musical Chord Recognition
Vocal dropping is implemented using Meta's demucs.
Downbeat tracking and BPM retrieval is perfromed using All-In-One Music Structure Analyzer by Taejun Kim.
Beat-syncing is performed with PyTSMod by MAC Lab @KAIST

Licenses

All code in this repository is licensed under the Apache License 2.0 license.
The weights in this repository repository are released under the CC-BY-NC 4.0 license as found in the LICENSE_weights file.
The code in the Audiocraft repository is released under the MIT license (see LICENSE file).
The weights in the Audiocraft repository are released under the CC-BY-NC 4.0 license (see LICENSE_weights file).

tomoyukiorita / cog-musicgen-remixer Goto Github PK

cog-musicgen-remixer's Introduction