jacbz / lofi Goto Github PK

View Code? Open in Web Editor NEW

343.0 8.0 97.0 13.86 MB

ML-supported lo-fi music generator

Home Page: https://lofi.jacobzhang.de?default

License: Apache License 2.0

HTML 3.93% TypeScript 43.78% JavaScript 0.95% SCSS 12.11% Python 39.10% Dockerfile 0.13%

music machine-learning vae tonejs lofi-music pytorch flask typescript lofi

lofi's Introduction

Lofi

A ML-supported lo-fi music generator.
Explore »

About

We trained a VAE model in PyTorch to represent a lo-fi track as a vector of 100 features. A lo-fi track consists of chords, melodies, and other musical parameters. The web client uses Tone.js to make a dusty lo-fi track out of these parameters.

Architecture

Client: The client is written in TypeScript and built with Webpack. It uses Tone.js to generate music.
Model: The model is implemented in PyTorch. We synthesized various datasets, including Hooktheory and Spotify.
Server: The server is a basic Flask instance that deploys the trained model checkpoint. The client communicates with the server using a REST API.

Setup

If you only want to tinker around with the client, you will only need the client folder. This will use the project's server as the backend.

If you want to deploy your own model, you can either train your own model (see the instructions in the model) or download the pre-trained checkpoint from here. Once you have deployed the server, change the server address inside client\src\api.ts.

Set up the client

Install node.js LTS.
Navigate to the client folder and run npm install to install the dependencies.
Run npm run serve to develop or npm run build to build a distributable.

By default, this uses the project's server as the backend. You can also train your own model and deploy your own server.

Train your own model

See the model folder for details. Once you have trained your model, put the checkpoint in the checkpoints folder.

Deploy your own server

See the server folder for details. You can use the provided Dockerfile. Don't forget to change the API url in the client.

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

Acknowledgments

Big thanks to ZOOPRA UG for hosting the server!

lofi's People

Stargazers

Watchers

Forkers

catsvilles cssquirrel mhackiori ngngsonan windsorel gaby hassnian tkuece-cilab enigmaticelectro alexin-tech ra101 xxn11 sartijo8181 huichuanliu jlcarvalho tdmusicdt pandarawen pranav2207 fachastorm celestincordos mathiaszero preetampvp codenut hongyoon rodrigozrusso raresportan niteshldd miziodel holasoymalva orlgln elliotwutingfeng nishanc foreversaga zachos-tech blackbeanca erohin bonesoul limnic pashkatrick zwerg4 microml rononchen leokite69 jv3ga osbarcelos79 fuleinist mmmdbot xing1p anan474 milkliker miravideo tbergman hunglichou dbhurley heinetl alankrantas scorchchamp marcosrochedo spencer-damiano codebybowen overvoidjs moseti1 jmftrindade jijjy alidjanka sanskarsakya bit2map spinnyspiwal beezisback younase ben015 jimmyhwang emanuelfromflorence kolonist ciganova battlestarship chrisvfabio haha4github cmcintosh progo111 rohanrusta21 syasj r0hnx ahaduk jsuwmc axwaizee7 jv-root pauloduarte43 fivetechvietnam mmogang scott-park soniqvision rebel-internet james-bltg nguyenvanduocit eric2018 shyam-sivasubramanian

lofi's Issues

Continue training from checkpoint

First, congratulations for the amazing project!!

I would like to know, How can I continue training from the available checkpoint?

Thanks for the project!

may you create a docker compose deploy?

I wanna try it but i know almost nothing of python xD

Thanks in advance.

ModuleNotFoundError: No module named 'model'

ModuleNotFoundError: No module named 'model' in docker and running outside of docker cant seem to get it to see the models even when they are in the same directory as the server or manually pointing to the location by changing it in app.py not sure what I am doing wrong. works fine with default settings but I would like to run this locally

Way to generate more melodies sequences

Thanks for the previous answer. I have tried and it effectively lengthens the duration of the theme, but repeating the same thing over and over again. Is there a way to generate more sequences of different melodies to have a more extensive musical track but not for that reason repetitive?

Problems with training the model

Hi, I finally got everything ready and it was my first time training a model. However I noticed some problems with lofi2lofi_train.py:

Unable to continue training with pre-trained / previous results
Everytime I run the script, it trains from epoch 0 and doesn't load any files. How make the script to load previous results and continue training?
High CPU / disk usage
I notice that the RAM usage is always rising and extreme high. It reached MemoryError at 211 epochs.
Is there anyway to improve the training script?

I'm not sure when it will start overfitting but for now, I want to have as much epochs as possible.

Really love this project and I want to make it better.

Thank you!

Continuous Playback

I know this hasn't been updated in a while, but could there be a feature where it reaches the end of the playlist, and generates another song? Continuous playback?

Docker fails to start

I'm unsure why I'm getting this error when trying to run the Lofi docker image.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/site-packages/flask/__main__.py", line 3, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 1050, in main
    cli.main()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 911, in run_command
    raise e from None
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 897, in run_command
    app = info.load_app()
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 312, in load_app
    app = locate_app(import_name, None, raise_if_not_found=False)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 218, in locate_app
    __import__(module_name)
  File "/app/app.py", line 15, in <module>
    limiter = Limiter(
TypeError: __init__() got multiple values for argument 'key_func'

The commands I used in Docker are as follows. Using your Dockerfile

docker build -t lofi .
docker run -p 8080:5000 -v /Lofi/model:/model -v /Lofi/server:/server lofi

"Error!"

Since some days, I have an Error when I click on generate, Even when I use your website, Is it still an active project? Thanks :)

Requirements for the model part?

Hello, thanks for creating this project. I was wondering if there's a requirements.txt for the model part? I didn't really see one in the project.

Auto Play

Hello, how do we make it autoplay when the site is opened?

What are the terms of use and licensing of the generated songs?

When using the live version of the app, all the songs generated are copyrighted? What about when forking the project and tuning the model?

Unable to train

Hey There!

I'm running into this error "Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead." when trying with CPU tensor. Below is the error output I get. I have also tried with CUDA and get other various errors. So I'll stick with CPU for now.

python3.8 lofi2lofi_train.py Using cpu device Starting training: lofi2lofi == Epoch 0 == Scheduled sampling rate: C 0, M 0 Batch 0: Loss 5.507 (C: 2.230 + KL: 0.021 + M: 2.710 + T: 0.059 + K: 0.081 + Mo: 0.193 + V: 0.090 + E: 0.123) Validation Batch 0: Loss 5.470 (C: 2.192 + KL: 0.020 + M: 2.705 + T: 0.059 + K: 0.082 + Mo: 0.188 + V: 0.052 + E: 0.172) Epoch chord loss: 2.230, melody loss: 2.710, KL: 0.021, chord accuracy: 1.765, melody accuracy: 0.624 VALIDATION: epoch chord loss: 2.192, melody loss: 2.705, KL: 0.020, chord accuracy: 14.286, melody accuracy: 0.311 Traceback (most recent call last): File "lofi2lofi_train.py", line 14, in <module> train(dataset, model, "lofi2lofi") File "/mnt/Storage//Lofi/model/model/train.py", line 190, in train axs[0, 0].plot(epochs, train_losses_chords, label='Train', color='royalblue') File "/home/ubuntu/.local/lib/python3.8/site-packages/matplotlib/axes/_axes.py", line 1688, in plot lines = [*self._get_lines(*args, data=data, **kwargs)] File "/home/ubuntu/.local/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 311, in __call__ yield from self._plot_args( File "/home/ubuntu/.local/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 494, in _plot_args y = _check_1d(xy[1]) File "/home/ubuntu/.local/lib/python3.8/site-packages/matplotlib/cbook/__init__.py", line 1348, in _check_1d return np.atleast_1d(x) File "<__array_function__ internals>", line 200, in atleast_1d File "/home/ubuntu/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 65, in atleast_1d ary = asanyarray(ary) File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor.py", line 643, in __array__ return self.numpy() RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

Command I run is python3.8 lofi2lofi_train.py

instruments

Hi, thanks for this amazing repository, I am converting this project to python code...
And I want to know that the folder "https://github.com/jacbz/Lofi/tree/main/client/assets/samples/instruments" has full instruments and the music is made only on these instruments?

Error when generating a new Play List

The error is in the web version https://lofi.jacobzhang.de.

The same error in my virtual machine with a fork made to try to improve and add a new collection of lead sheets and be able to improve the project.

Recomendation to generate .wav or .mp3 files

Do you recommend any way to, instead of playing the song in tone.js, generate a .wav or .mp3 file?

Change average length of song

Impressive project! My congratulations! The generated music is very good. I have a question, is it possible that the average duration is 3 minutes instead of 1:30? I have downloaded the code but I can't figure out where to modify to achieve that. I clarify I do not program in python. Thank you!