Musical creation and transcription assistance via generative AI

[walterbender]

Musical creation and transcription assistance via generative AI

[walterbender]

@falgun143 the project won't be offered under DMP this year.
While I am not wedded to any particular implementation details, I suspect it will leverage LLMs with some measure of training data.

[falgun143]

Hello @walterbender Sir ,Will this project be heavily based on ML algorithms ,Or is it based on training of the base model(Like llama2 ,gemma etc...) using the custom data. And will this project be under DMP?

[falgun143]

@walterbender Sir any thoughts?

[gitjeet]

@walterbender For the conversion of recorded music to MB, instead of relying on server-side processing LLM, I believe we could integrate this functionality directly into MB itself. This would involve converting the recorded sound to MIDI format and then to MB.

Once done i think we could then send any format abc or midi to LLM server to improvise

[falgun143]

@walterbender For the conversion of recorded music to MB, instead of relying on server-side processing LLM, I believe we could integrate this functionality directly into MB itself. This would involve converting the recorded sound to MIDI format and then to MB.

Once done i think we could then send any format abc or midi to LLM server to improvise

@gitjeet ,I think as MB is already a huge and old project ,Integrating this llm project directly to the MB codebase may further make MB work slower I feel.

[falgun143]

@walterbender For the conversion of recorded music to MB, instead of relying on server-side processing LLM, I believe we could integrate this functionality directly into MB itself. This would involve converting the recorded sound to MIDI format and then to MB.
Once done i think we could then send any format abc or midi to LLM server to improvise

@gitjeet ,I think as MB is already a huge and old project ,Integrating this llm project directly to the MB codebase may further make MB work slower I feel.

Nope, I think it wouldn't have any issue with the time complexity, and it would be easy and cost-effective. To be clear, what I meant is not integrating the LLM to MB (client-side). What I meant is only the conversion of mp3 -> midi -> MB, which does not require LLM. It is a pure graph algorithm. MB already has conversion for MB -> Midi or MB -> ABC. Send this midi or ABC to LLM (server-side, not client-side). PS: i don't think its old project

Good insight,Then there will not be any frontend right?.But tell me how can we upload an external audio file and extract musical features and then send it to llm? . And also if you see the description of this project it has been clearly written that we have to create an in app interface.