petersalomonsen/javascriptmusic

A.K.A. WebAssembly Music. Live coding music and synthesis in Javascript / AssemblyScript (WebAssembly)

javascriptmusic's Issues

mod, xm, s3m, it, MusicXML & verovio integration?

1. https://github.com/deskjet/chiptune2.js is used as the web player at https://modarchive.org/index.php?request=view_by_moduleid&query=166351 Play all tracker formats supported by libopenmpt (including mod, xm, s3m, it) similar to https://github.com/martincameron/micromod a player library for the ProTracker MOD music format
   for Javascript (HTML5 Web Audio), Java, ANSI C (SDL) and Pascal (SDL).
2. https://github.com/w3c/musicxml/releases/tag/v4.0 MusicXML 4 is the standard for import and export https://www.npmjs.com/package/musicjson is a MusicXML to MusicJSON bi-directional converter
3. https://github.com/rism-digital/verovio/tree/develop-humdrum Verovio is a fast, portable and lightweight library for engraving Music Encoding Initiative (MEI) digital scores into SVG images. Verovio also contains on-the-fly converters to render Plaine & Easie Code, Humdrum, Musedata, MusicXML, EsAC, and ABC digital scores. see https://verovio.humdrum.org/

Licensing

Do you intend to make this open source at any point? I've seen a few of your videos and love the idea of a live coded performance.

Great experiment!

Let me give some tips & tricks how you improve AS code more if you don't mind)

You don't need own min/max/abs for f32 here because AS has builtin intrinsics for that which translates to single wasm opcodes like f32.min/f32.max/f32.abs. Just use min<f32>(a, b), max<f32>(a, b) and abs<f32>(x)

Questions! (Btw, this project is awesome!!)

Hi, this project really looks amazing and after I played in the web version for a while, looked through the code here in the repo, my head started spinning around, and I got very curios about the possibilities of making music with JS the way you did it with node.js and wasm. Now I have tons of questions and I really hope you will have time to answer all of them, thank you! :)

1. I would love to try out and also do some more music with node.js, I suppose I would have to install ZynAddSubFX or Yoshimi first (I just saw Yoshimi mentioned in the code so I just suppose it works too) can I also use another VST instruments following your examples? I'm on Ubuntu/Linux if something but will I be right to guess that eventually it's possible even to use "premium" windows VSTs via wine or something?
2. How can we adjust the knobs, parameters of the VST plugin in node.js without their GUI, making it into custom instrument or something we can use in the code? Is it even possible via midi? Or maybe every VST synths have their own project files that can be parsed/changed?
3. How can I render the audio data to .wav when using node.js? I saw that is pretty easy with Sox when working with 4klang or Webassembly. But here we have Recorder class, does that mean it records in real time only? How could I go around this and just render the audio data without waiting for play-through?
4. Could you advise, share links/info/resources on where I could learn more about how we can manipulate VST and other audio plugins with just a code and no GUI, DAW etc? I am really fascinated by the idea of making high quality music with code, no GUI, but I cannot wrap my head around it and understand how could it actually work, for example, how can we actually adjust all the knobs and parameters of the plugin to make custom instrument/sound from it. I tried to search more info about such things but currently there is not much about it. Are there similar projects/packages that work with VSTs (or similar high quality plugins) you know of?
5. I may have misunderstood something but I noticed that for WebAssembly version you are writing custom instruments with Typescript, could we actually use VST instruments compiled to WASM, I know there are WAMs (https://www.webaudiomodules.org/wamsynths/) could we use them without their GUIs?

Thanks again for reading through all of this! :)

Feedback, features propositions, more questions (time, duration, seeking, audio samples, etc)

Hi, I know, I know, last time I was after node.js version but it was before I realized that actually WebAssembly does everything I would ever need, and we can render and play audio server-side in node.js+sox same as in browser, WASM should be truly pronounced as AWESOME! :) Last few days I spent creating and live coding music in the browser and I got few ideas and feature requests, help offers:

1. Tracking current time while playing the song, calculating full song duration, seeking:
   Now, for the first one I know there are already few attempts to track time with logCurrentSongTime() in pattern_tools.js, but unfortunately I could not hack it to work good. I found another way that works good - tracking currentTime of global.audioworkletnode.context. Now, in my experience developers use setInterval() function a lot, when making music players with JS, etc but I'm sure there are better ways to dynamically update and log current time while playing. I would dig more into logCurrentSongTime() function and check how Tone.js does things, they have Transport class with .scheduleRepeat ( ) function, from what I know they use time precise clock of WebAudio, not JS one. More about this: here
   In any case, one way or another getting current time while playing should be fairly easy to implement, but getting full duration and seeking, this is where I'm limited in ideas, I just believe that it would be a cool feature and good user experience to get full song duration after we evaluate the code, then we could have a simple horizontal slider which scrolls while song is playing, and we can update the current position. This would be a cool feature, so we don't have to wait one part playing through when we actually wanna heart next patters. What do you think? With your blessings and guidance I'm ready to get on this one and submit PR as soon as I hack it together. So, you could just give me your thoughts and theory behind it, I will do the coding if something. :)

2. Audio Samples. Okay, this one is straight feature request, I remember you mentioned something about this in our previous issue discussion, I'm sure you are already planning those in any case, just wanna mention that it would be cool to have this freedom of adding custom audio samples, for cool kicks, drums, atmos, percussion, etc. And also would be cool if this would work same as whole AssemblyScript synth, both in browser and node.js+sox for quick audio rendering, so I guess we would have to implement custom AudioBuffer/AudioBuffersource in typescript, without using browser's AudioContext? Once we have AudioBuffersource it should be fairly easy to implement something like Sampler , allowing the user to play multiple samples with predefined pitches. I'm ready to help with this one too, let me know what you think!

3. Now, a bit of a feedback of a user who tried to create some music in the browser... Mixes... oh, I fairly know JS, never touched TypeScript before, and overall doing DSP stuff, making instruments so you can later code patterns with them, oh, this is quite exhausting for unprepared user :D I guess the current system holds quite high entry threshold for newcomers. Maybe it would be possible to implement some kind of presets of ready instruments/effects like other live coding environments do (read Sonic Pi)? I know, I'm probably over my head here, sorry, just sharing my thoughts. :)

4. I decided to forget about other ways of live coding music, like on server with node.js, and stick with AssemblyScript implementation you proposed but still I'm thinking that it would be cool to have some freedom of using other synthesizers with current sequencer and interface. For example, there is WASM version of Yoshimi I believe we could use it too, for composing in browser and rendering the audio on server with sox. Just, from what I understand the current sequencer is coming after your 4klang experiences, would it works with other synths, I mean, would it be possible overall? What do you think?

Thanks for reading, sorry, for that much of text, I hope I'm not being too much annoying here! One more thing, you should finally name your project, so people can be like:

insertnamehere is Awesome!!! Please continue developing and improving it!

Fantastic project ... and a few questions

Hi Peter and thanks for a fantastic project, posts and presentations.

I'm a JS/TS/3D dev but currently learning AssemblyScript, in part to do some retro C64, VIC-II & SID emulation with WASM. I'm a graphics guy and have rarely touched on audio, so am seeking advice and simple AS audio examples, the latter being hard to come by.

1. I'd like to emulate SID digi playback but I understand that ideally requires cycle accurate emulation, which may not be possible if calling my WASM code or accessing shared memory on requestAnimationFrame. If I iterate faster than that in JS, even the high-resolution timers aren't guaranteed to be that accurate, plus I risk locking up the browser. What's a good approach here?
2. Is it possible to generate audio directly in AS/WASM or can I only manipulate buffers on the WASM side then output audio on the JS side with the Web Audio API?
3. Is there a simple AS audio example that you know of that generates a simple sine or square waveform at a high sample rate?

Writing VST and native Audio plugins in AssemblyScript?

Hello Peter, yes, this maybe is not completely related to this repo but I think you are the best person to ask about this. :)

A while ago I read your article about converting WASM to C. And this got me thinking if writing a native audio application would be possible with AssemblyScript at all?
While doing a bit of research (not having practical experiments though), here are my thoughts:
Maybe we could somehow use the converted C code to build the plugins?
For example, this compiles PureData patches to C. "The C source can then be used to build VST plugins or integrate with games engines like Unity, Unreal or Wvise".

Another theory of mine, we could write our audio app/synth in AssemblyScript and use the WASM binary files with node.js, or other server environments (Python) to i/o the audio. I'm sure there are also ways to build independent UIs either with Web technologies or some native toolkits. Also, the easiest and most obvious way is using Electrons.js but from what I understand bringing the whole Chromium engine will influence the performance drastically.

These are just theories and I would love to hear your theoretical opinions on this too, pretty much sure some other people will also find this useful. Would be cool to be able to write cross-platform apps in Typescript for both Web and Native, Desktop. :)

new related project

Hi @petersalomonsen just wanted to first say a big thank you for your awesome work on this project.

I was especially impressed with the asssemblyscript based synth work you have been doing, so was hoping to incorporate it in part in a project I'm working on for a RPI based "groovebox" connected to a dedicated midi controller (Akai Fire initially).

As I've been building my project with Dart, I started an initial prototyping project to see if I could even make this work reasonably so I wanted to let you know I've got a little bit of the way now.

I've looked a little bit through the repo heree, but f you would have an pointers on how I could setup compiling your AS synth code that would be most appreciated.

Also I couldn't see which license you are using for your AS code such as https://github.com/petersalomonsen/javascriptmusic/blob/master/wasmaudioworklet/synth1/assembly/synth/sineoscillator.class.ts ?

possible different or additional license?

Continuing on from discussion in #58 ...

I'd like to be able to make use of your great existing assemblyscript synth work in my project, but my code is usually under a BSD or MIT license which I think may not be compatible with the GPL currently used for this project so I wanted to ask if it would be possible to have at least that part of this project licensed under a license other than GPL or perhaps dual licensed under GPL and another license?