windfisch/loopfisch

A loop machine written in Rust (uses Jack)

loopfisch

An audio and MIDI loop machine written in Rust.

This is still in active development, but the basic functionality already works.

Features

• JACK Audio output
• Audio and MIDI support
• Simultaneous recording of both Audio and MIDI at a time
• As seamless as possible switching between both
• Multiple audio chains (implemented by having multiple JACK ports)
• separate "Main speakers" and "Monitoring headphones" output chains (not yet)
• Browser-based user interface
• Fully (PC-)keyboard-controllable (not yet)
• MIDI clock master
• MIDI transport slave (not yet)
• Close-to-full unit test coverage for the engine

Build instructions

Currently, you need to use the unstable Rust toolchain, version 1.50 or later. You can add a per-directory override using

rustup override set nightly
rustup update # might or might not be needed

You might need to update rust, for more information, you can refer to https://rocket.rs/v0.4/guide/getting-started/.

Usage

Start a jack server, then launch the engine using cargo run. You should now be able to make REST requests like curl localhost:8000 /api/synths.

In order to be able to use the GUI, change to the web directory and run yarn serve. (You might need to yarn install before). Then access http://localhost:8080 in your web browser.

Test suite

Run the tests with cargo test.

Note that the outsourced_allocation_buffer tests are inherently racy and it might be needed to increase sleep time in fn wait(). There is nothing we can do about this, as this is the intended use case for outsourced_allocation_buffer. In normal situations, the allocator thread has between 0.2 to 2 seconds to react, but this would slow the test suite down heavily.

loopfisch's Issues

midi and audio recording should "preroll" a bit

when playing a loop, the very first note might be lost, because it was played a tiny bit before the actual loop start

not sure how to deal with this, but "pre-rolling and cross-fading" could solve this.

allow adding midi-only takes

Implement chain-pass-through

• engine: chains should pass through audio-in to audio-out.
• REST-api
• GUI: should automatically set/unset chain-passthrough when at least one midi-take of the chain is unmuted
• GUI: should disallow midi-takes of different chains in the same synths to be unmuted
• maybe not unset passthrough when the last midi-take is muted again (wait until another chain is unmuted. or wait a certain time.)

test cases for takes

test cases for the metronome

destruction of old devices

collector thread that runs the destructor

blocks #12

Gracefully handle BPM changes when takes exist

Currently, the GUI allows this and the engine will crash.

• disallow changing the BPM in the GUI
• disallow changing the BPM in the engine frontend (so that the backend won't crash)
• communicate the error in the rest api
• write test case

delete synths,chains,takes (rest api + engine)

handle changes of audio/midi latency

send time with GET requests

so that the GUI has a good timestamp directly after connecting.

remove dummy data

send "time zero" signal to the gui

the gui's reference timepoint should be reset whenever the engine signals a loop point, so that the pie charts are properly synchronizes.

also, the gui should send the loop length.

auto-connect ports

at least for debugging

rename engine::GuiAudioTake etc to frontend::AudioTake, fix dependency situation

takes.rs, backend.rs, jack_driver.rs seem all quite backend-ish.

there should be backend::AudioTake and frontend::AudioTake (the latter being exposed, the former not)

unit tests for realtime_send_queue

actually record takes

• record them
• communicate this to the GUI

Improve MidiRegistry implementation

MidiRegistry is currently a 2kB array that is searched completely and copied when takes begin or finish recording.

One might want to have a linear map or a tree map flattened to an array where less data needs to be copied around.

Start/stop midi transport on next transition

send F0 or F7 on a transition (or now.)

Offer this per synth.

do not spawn one helper thread per OutsourcedAllocationBuffer

maybe just use async tasks

unit tests for outsourced_allocation_buffer

write unit tests for engine

implement a dummy driver for testing purposes. especially: proper latency handling for midi and audio

blocked by: #22

rename jack ports when renaming synths/chains

do not use thread::park/unpark

thread::park/unpark briefly lock a mutex and thus is not appropriate for an realtime audio thread.
better use futex or eventfd instead.

Lost MIDI note-on and -off events close to loop boundaries

When a take is recorded and the notes are released after the end of the recording, then the note-off event is not recorded. This makes the note "hang".

Possible solutions: keep track of all active notes and release all notes automatically at the end of a take.

Similarly, early note-ons might be lost, if received before the start of a loop. Maybe we should delay those a bit until the actual loop start?

allow takes with different lengths (multiple of the loop length)

easy way: let the user pre-select the length of the recording.

the hard way: let the user stop the recording and then "round" to the closest loop length multiple

the difficulty is that "stop recording" shortly after the looppoint shall discard some samples. Looping would need to start in the middle of the take then. This has two implications:

• for audio, we need to skip forward a few samples
• for midi, we additionally need to handle those midi events accordingly. What do we do with a delayed event? discard it (and then have possibly dangling note-offs or miss a controller event)? or delay them?

Possible solution: We always (at each looppoint, or actually shortly before) rewind playback as usual, but consider the takes muted until recording has been finished. Advantage: we can share the "delayed event" problem with muting/unmuting.

UX design considerations

The hierarchy as seen by the user is as follows:

toplevel
 |--- 1 main output (loudspeakers)
 |--- 1 aux output (monitoring headphones)
 |--- N devices (e.g. "DeepMind", "Korg Monologue" and "E-Guitar")
 |     |--- 0 or 1 associated audio port pair
 |     |--- 0 or 1 associated MIDI port pair (0 for the "E-Guitar case")
 |     |--- N tracks (e.g. "bass line", "lead1", "lead2", "pad")
 |     |     |--- N audio takes
 |     |     |--- M midi takes
 |     |     `--- O groups
 |     |           `--- reference to 0 or more toggleables
 |     `--- (M groups; see above)
 `-- (M groups; see above)

A toggleable is a midi/audio take or a group (non-cyclic).

A group can be put into 3 modes on both outputs: "muted", "audio playing" or "midi playing". Its associated takes can be in a state that do not fit into any of the three modes. (I.e. a group can be "audio playing on live, but midi playing on monitor)

A midi take can be muted or playing.

An audio take can be muted, playing on main or playing on monitor.

A device that has a MIDI input can be muted, passed through to main or to monitor (or both)

Recording an audio take automatically creates a group containing the audio take just recorded, all midi takes playing on the same device, and the midi take just recorded on that device, if any.

TODO: what do we do with "Putting a group into midi playing mode, where another group is already in midi playing mode, or where the device is outputting to a different output than the audio track?"?

• refuse
• warn
• just do it

TODO: how do we handle "Putting a group from audio playing into midi playing mode (queued)"?

• just do it
• start playing MIDI instantly, but mute the audio-through of that device; then cross-fade the looping audio-take(s) and the audio-through of the device at the loop point.

TODO: how do we handle "Putting a group from midi playing into audio playing mode"?

• just do it?
• crossfade, i.e. turn of the device's audio-through until the device has become silent? (Then reenable to allow the user to jam around)

All toggling actions can either be instant or queued to happen at the next loop point.

set the correct value for associated_miditakes

currently, only the newly generated audio/midi takes are associated. all other unmuted miditakes should be, too!

communicate take state (waiting, recording, playing, waiting-for-mute) to the GUI

multiple audio routes

• two outputs (live and monitoring/headphones)
• each track should be mute-able on any output
• copy monitoring mute state to live state / vice versa
• presets (can be applied to either live or monitor or both)

midi mute must turn off all notes

(generally, a panic button would be nice)

implement "mute on next transition"

click removal

Loops currently click on their loop point.
Use LSAR (Least Squares Autoregression) like in Audacity's repair effect (which uses InterpolateAudio) to remove the click.

unit tests for midi registry

GUI elements for tempo selection

• BPM input field + number-of-measures (fixed to 4?) with "set" button
• tempo tap
• disable setting when takes exist.

see #15

act as MIDI clock master

REST: support take patch (esp. mute)

crash when setting the loop length when takes exist

the frontendThreadState does not check yet whether there are takes, but the backend asserts on this.

error[E0554]: `#![feature]` may not be used on the stable release channel

Trying a cargo build but getting this atm;

error[E0554]: `#![feature]` may not be used on the stable release channel
  --> src/main.rs:16:1
   |
16 | #![feature(proc_macro_hygiene)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

use newtypes for various IDs

abstract from jack dependency

use more traits, less direct knowledge of Jack.

blocks #23.

allow renaming in the GUI

test frontend/backend synchronization

tempo selection

The easy way: add a "tap tempo" button (and a beat number field, defaults to 8)

The hard way: let the first loop length decide. Problem: we need to skip latency frames upon the first loop.

GUI: support takes

• "new take" button
• display record state
• display playback indicator
• associated takes don't work
• unmuting an audio take group's midi does not work.

use sample rate / loop length in the GUI

• transmit the sampling rate via GET requests
• transmit loop length via GET requests
• do not hardcode loop length in take.vue
• do not hardcode sampling rate in rest_api

fix GUI support for take playback position with over-long takes

soft mute and unmute to prevent clicks

mute/unmute should quickly fade in/out the buffers to avoid phase clicks.

Metronome

send all pending note-offs prior to shutdown

chop / reduce loop length

• for the song as a whole
• for single tracks