slseanwu/jazz_transformer

Transformer-XL for Jazz music composition. Paper: "The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-Composed Music through Quantitative Measures", ISMIR 2020

The Jazz Transformer

An adapted Transformer-XL deep learning model that composes Jazz music (lead sheets—chord progression & melody).

Tensorflow implementation of the automatic music composition model presented in our paper:

• The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures
  Shih-Lun Wu and Yi-Hsuan Yang
  The 21st International Society for Music Information Retrieval Conference (ISMIR), 2020.

Want to listen to some compositions by the Jazz Transformer first? Click here!

Usage Notes

Prerequisites

• Python 3.6 (install)
• Recommended: a working GPU with ≥2GB of memory
• Install dependencies (pip or pip3, depending on your sytem)

pip3 install -r requirements.txt

Compose Some Songs Right Away

• Download pretrained model

./download_model.sh

• Inference (compose)

python3 inference.py [--model MODEL] [--temp TEMP] [--struct_csv CSV] [--n_bars N_BARS] output_midi

• output_midi: path to the output MIDI file
• --model MODEL: path to the trained model checkpoint (default: the downloaded checkpoint)
• --temp TEMP: sampling temperature for generation (default: 1.2)
• --n_bars N_BARS: # of bars to generate (default: 32)
• --struct_csv CSV : path to the output csv file that records generated structure-related events (optional)

Train from Scratch

• Preprocess dataset

./data_preprocess.sh

• Train the model

python3 train.py ckpt_dir log_file

• ckpt_dir: directory to save checkpoints
• log_file: path to the log file

Likewise, you may compose music with the model trained by yourself using inference.py (see above for instructions)

Directory Structure

├── data_preprocess.sh      (executes python scripts to build vocab and prepare data) 
├── inference.py            (generates Jazz music)
├── requirements.txt        (python dependencies)
├── train.py                (trains Transformer-XL from scratch)
├── data                    (.pkl files for training)
├── mcsv_beat               (Jazzomat dataset content---beats+chords)
├── mcsv_melody             (Jazzomat dataset content---solo melody)
├── output                  (sample generated piece)
│   ├── demo.csv
│   ├── demo.midi
├── pickles                 (houses required metadata for training)
├── remi_encs_struct        (contains training data in readable REMI event sequences)
├── src
│   ├── build_chord_profile.py   (reads and stores key templates for different chord types defined in ``chord_profile.txt``)
│   ├── build_vocab.py           (builds the vocabulary for the Jazz Transformer)
│   ├── chord_processor.py       (the class and methods for converting notes to chords and vice versa)
│   ├── chord_profile.txt        (hand-crafted key templates for each chord type)
│   ├── containers.py            (container classes for events in mcsv files)
│   ├── convert_to_remi.py       (converts Jazzomat dataset to REMI events for training)
│   ├── explore_mcsv.py          (utilities for reading events from dataset .csv files)
│   ├── mcsv_to_midi.py          (converts mcsv file to midi format)
│   ├── midi_decoder.py          (the class and methods for conversion from REMI to midi)
│   ├── mlus_events.txt          (the mlu events used by the Jazz Transformer)
│   ├── mlu_processor.py         (the class and methods for defining and parsing Mid-level Unit (MLU) events)
│   ├── prepare_data.py          (splits data into training and validation sets before training the Jazz transformer)
│   ├── remi_containers.py       (container classes for REMI events)
│   ├── utils.py                 (miscellaneous utilities)
├── transformer_xl
│   ├── model_aug.py             (Jazz Transformer model)
│   ├── modules.py               (functions for constructing Transformer-XL)

Acknowledgements

The Jazz Transformer is trained on the Weimar Jazz Database (WJazzD), a dataset meticulously annotated by the Jazzomat Research Project (@ University of Music FRANZ LISZT Weimar). Many thanks to them for the great work and making it publicly accessible!

• URL for WJazzD: https://jazzomat.hfm-weimar.de/dbformat/dboverview.html

Also, we would like to thank Yi-Jen Shih (@ NTUEE, personal GitHub) for the help he provided in arranging the codes of this repository.

See Also

• The repository of evaluation metrics (also proposed in our paper) for machine-composed music:
  https://github.com/slSeanWU/MusDr

jazz_transformer's Issues

Something wrong with inference.py, the output file can't be played.

As I follow the instructions to compose music

I got this log output:

Start model inference...
2020-08-05 05:55:32.732437: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-08-05 05:55:33.762416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
failed iterations: 1
failed iterations: 1
failed iterations: 2
failed iterations: 3
failed iterations: 4
failed iterations: 5
failed iterations: 6
failed iterations: 7
failed iterations: 8
failed iterations: 9
failed iterations: 10
failed iterations: 1
failed iterations: 1
failed iterations: 1
generated 6460 events
First 20 events: ['Part-Start_I', 'Rep-Start_1', 'Bar', 'Position_0/64', 'Tempo-Class_0', 'Tempo_67.50', 'Position_16/64', 'Tempo-Class_0', 'Tempo_65.00', 'Position_32/64', 'Tempo-Class_0', 'Tempo_67.50', 'Position_48/64', 'Tempo-Class_0', 'Tempo_62.50', 'Position_55/64', 'Phrase', 'MLU-Type_lick', 'MLU-Subtype_lick-general', 'Backref-Referred']

now converting events to MIDI: ./output/test.mid ...

It did generate midi file in output dir, but the file can't be played in normal player. It seems like the writing file stage makes something wrong ...

How did you process the Weimer jazz data on melody spy

Could you provide the steps in which you used to extract the beat and melody csv files ?

Thank you

Hey guys,

Thank you for sharing your work. Looks very nice and promising.

I have a question: Is it possible to use your code/can you provide code to train on my own MIDI dataset. It seems that you are using proprietary format of the Jazz dataset so I can't try it on my own to evaluate your work and results.

I will really appreciate your help here.

Thank you.

Uniqueness of the generated samples

First and foremost, @slSeanWU , thank you very much for releasing this project under the MIT license! 👍

To be honest, I am quite convinced that the output midi files that I'm getting from this project (using the pretrained model) are too quality+long to be true (considering all of my other attempts to generate decent jazz by the use of AI that were all a failure so far).
What I really mean, is that, due to the quality of the output, I suspect that there is some overfitting going on (or maybe I even completely omitted some important point/catch from the readme :D).

Having said that, my questions are:
1. Is it really spilling out the unique melodies?
2. Is there a relatively simple way to check for the uniqueness of the output melody against the dataset that was used in the training?

I would really appreciate your (or anyone else's) answer very much!