Audio + MIDI

[spatialaudio]

Audio + MIDI

[mgeier]

Very good, we are getting closer ...

My next question is: Do you really need if __name__ == "__main__"?

If not, you can remove it.
If yes, you'll have to move the definition of the process callback into the if __name__ == "__main__" block.

Also: using client.activate() and then deactivate() and close() looks very much like it could need a with statement, right?
I guess the disconnect() and unregister() would happen automatically (right?), but it's of course OK to keep it there if you want to bring attention to the cleanup.

Also: be careful when using pass while handling those exceptions (see #101 (comment)). You might want to consider using raise instead.
Or you remove the callback separation for now, to keep it simple.
You can split the callback in two parts later, if needed.

in your example there is a massive use of threading and queue, so it results very difficult for me...

I know, but the whole point why I asked whether you can load the whole file into memory was to be able to get rid of this difficult stuff.

You can load the entire audio file somewhere outside the callback function:

audio_data, samplerate = sf.read(audio_file)

The variable audio_data should then be visible in the callback function, where you can use NumPy indexing to get the right slice of it to play back.

I probably should make an example for this at some point, but for now you can probably start with something like:

outdata[:] = audio_data[current_frame:current_frame+blocksize]

You'll have to calculate the current_frame, of course.

Do you know the number of channels beforehand?
If not, you might have to be a bit more careful with indexing to make it work for a single-channel file.

Also, you'll have to consider what happens when the audio file is over.
After that, you'll have to stop playback or you'll have to fill the output with zeros.
And you should consider the situation where the very last block of the file is smaller than the blocksize used in the callback.
In this case you'll have to copy the last partial block and then fill the rest of the frames with zeros.

[mgeier]

I'm glad that it works now, that's great!

I don't think that time.sleep() is the right tool for the job, given that the audio/MIDI playback runs asynchronously.
If you call time.sleep() in the main thread, it will not change anything and playback will continue, if you call it in the audio thread, audio playback will break and you'll get xruns.

adjustable delay for compensating the delay between audio and lights

That very much depends on what you mean by "adjustable".
Is it enough to have a constant value (that you find by trial and error), or do you need to change the delay dynamically during playback?

The latter case could become quite complicated, so I hope the former is sufficient.
If that's the case, I think there is a quite simple way:

• if audio is too early, insert a few zeros in the beginning of the array.
• if MIDI is too early, start with an offset greater than zero. The value is in seconds, so this should be easy.

read a file containing the list of projects; how to run a new project when the playback is finished?

There are many ways to do this, here are just a few ideas:

• Add command line parsing to your script and re-start your script with different parameters once it is finished. You could do that e.g. with a shell script, or with another Python script or whatever.
• Refactor your code into a reusable function that accepts the the project parameters as function parameters. Then you can create a new Python script that calls your function repeatedly.
• Interweave the playlist logic directly into your existing code

In the first one you'll have to create a new JACK client and connect to JACK each time, but I think with the other options you could reuse the same JACK client and keep it connected all the time.

I don't know what's best here, I guess it is mostly a matter of taste. And of course it depends on how exactly you are planning to run this whole thing.

[mgeier]

Very good, it's getting simpler!

Do you know the number of channels beforehand?

Yes, I do.

OK, that's good, then you don't have to worry about channel indexing, because your arrays are all one-dimensional.

There is something wrong: I have no errors, but the process terminates almost immediately.

Well look at your code and see what it does: it creates a bunch of stuff, defines a few functions and finally enters a with statement where it connects two ports.

And then?

Nothing more to do, the script is finished and the Python interpreter terminates.

You will somehow have to wait ... probably a threading.Event helps with that?
I think I've used that in some of the examples ...

But that's not the only problem. Another one is the call to get_array().
The docs (https://jackclient-python.readthedocs.io/en/0.5.3/api.html#jack.OwnPort.get_array) say:

This method shall only be called from within the process callback

[mgeier]

start with an offset greater than zero. The value is in seconds, so this should be easy.

Unfortunately it doesn't work. "Normal" values are expressed in milliseconds; for example for a distance of 100 m, delay is (more or less) 300 ms, but I can't put offset = 0.3 because this value must be an integer.

Oh, sorry, I was wrong. The offset value is in samples.
So if I'm not mistaken again, you can use offset = round(0.3 * samplerate) to get a delay of 300 ms.

BTW, you should make sure that JACK uses the same sample rate as your audio files!

[Stemby]

What do you think about replacing this def process(frames): with def midi_process(frames):, and this def process(frames): with def audio_process(frames):?

@client.set_process_callback
def midi_process(frames):
    [...]


@client.set_process_callback
def audio_process(frames):
     [...]

How can client.set_process_callback understand what kind of work (processing audio or MIDI) has to be done?

[mgeier]

@C0nsultant has already answered your question about multiple callbacks: #101 (comment)

Coming back to your original question, I would say it very much depends on how exactly you want to read your audio file and what requirements you have for that.

Would it be OK to read the whole audio file into memory at once?

This would make the callback much simpler, but it of course requires that the whole audio file fits into memory.

[Stemby]

@mgeier,
thank you.

Would it be OK to read the whole audio file into memory at once?

Yes, no problem.

Here is my current (not working) code:

https://github.com/Plantarium-Seregno/Luces/blob/main/luces.py

[mgeier]

Would it be OK to read the whole audio file into memory at once?

Yes, no problem.

In that case, you don't need threading.Thread and you shouldn't use it.

If both the audio and the MIDI file are loaded into memory in their entirety, you can access them directly in the process callback. This means you don't need the queue either.

In the process callback you can simply get the current audio and MIDI data and send it to the hardware.
It's really much simpler than your code suggests!

Here's my suggestion what you should do to make this work:

• remove everything that mentions threading.Thread
• remove everything that mentions queue in any form
• concentrate on either audio or MIDI first, comment the other one out
• once one of them works (and only then), try to add the other one back in

Once you've reached the third point (but not earlier), feel free to share your code here, then I can have a look.

[Stemby]

Thank you very much, @mgeier!

I'm concentrating on the audio part; I'm trying to get a very minimal script from scratch.

#!/usr/bin/env python3
"""Controller for Luces.

This plays a MIDI file and an audio file at the same time.

"""

import jack
import soundfile as sf
#from mido import MidiFile

project_dir = 'Media'
project = 'test'
audio_file = '{}/{}.wav'.format(project_dir, project)
midi_file = '{}/{}.mid'.format(project_dir, project)

client = jack.Client('Luces')


def audio_process(frames):
    pass


def midi_process(frames):
    pass


@client.set_process_callback
def process(frames):
    try:
        audio_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass
    try:
        midi_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass


if __name__ == "__main__":
    audio_out = client.outports.register('audio_out') # Mono
    target_audio_port = client.get_ports(is_physical=True, is_input=True,
            is_audio=True)[0] # Audio Output 1 (L)

    client.activate()

    client.connect(audio_out, target_audio_port)

    with sf.SoundFile(audio_file) as f:
        pass

    audio_out.disconnect()

    audio_out.unregister()

    client.deactivate()
    client.close()

Now I can't understand how to use soundfile and audio_process(frames) in order to send audio data to the sound card: in your example there is a massive use of threading' and queue`, so it results very difficult for me...

[Stemby]

Thank you, @mgeier, for your kind reply.

Here is my current attempt:

#!/usr/bin/env python3
"""Controller for Luces.

This plays a MIDI file and an audio file at the same time.

"""

import jack
import soundfile as sf
#from mido import MidiFile


project_dir = 'Media'
project = 'test'

audio_file = f'{project_dir}/{project}.wav'
midi_file = f'{project_dir}/{project}.mid'

audio_data, samplerate = sf.read(audio_file)

client = jack.Client('Luces')

audio_out_port = client.outports.register('audio_out') # Mono
target_audio_port = client.get_ports(is_physical=True, is_input=True,
        is_audio=True)[0] # Soundcard Audio Output 1 (L)

current_frame = 0
out_audio_data = audio_data[current_frame:current_frame+client.blocksize]
audio_buffer = audio_out_port.get_array()


@client.set_process_callback # TODO: to be removed
def audio_process(frames):
    # TODO: How can I send out_audio_data to audio_out_port? 
    #       The following code doesn't work (or it is uncomplete)
    if len(out_audio_data) < client.blocksize:                          # Last block
        out_audio_data += [0]*(client.blocksize-len(out_audio_data))    # Fill with zeros
    audio_buffer[:] = out_audio_data

    current_frame += client.blocksize
    out_audio_data[:] = audio_data[current_frame:current_frame+client.blocksize]


def midi_process(frames):
    pass


#@client.set_process_callback # TODO: to be uncommented
def process(frames):
    try:
        audio_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass
    try:
        midi_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass


with client:
    client.connect(audio_out_port, target_audio_port)

# TODO: I have to stop the playback when the audio file is over.

There is something wrong: I have no errors, but the process terminates almost immediately.

Do you know the number of channels beforehand?

Yes, I do.

The workflow is: Ardour → Export as mono audio file → Python client → Soundcard Audio Output 1 (L)

[Stemby]

Thank you, @mgeier!

https://github.com/Plantarium-Seregno/Luces/blob/main/luces.py

Now, after having fixed the remaining TODOs, I'd like to implement 2 new features:

1. adjustable delay for compensating the delay between audio and lights (controlled via MIDI) caused by the speed of sound
2. playlist: a list of projects the script must automatically execute

What should pay attention to in order not to break the script?

My hypotheses:

1. using time.sleep (where and how?)
2. read a file containing the list of projects; how to run a new project when the playback is finished?

Thank you!

[Stemby]

Hello, @mgeier,
thank you for your replay and sorry for the delay.

* if MIDI is too early

This is my case: speed of light is much, much faster than speed of sound; so, at 50-100 m (or more) from the source you see the lights immediately, but the sound arrives after some time (depending on the distance).

start with an offset greater than zero. The value is in seconds, so this should be easy.

Unfortunately it doesn't work. "Normal" values are expressed in milliseconds; for example for a distance of 100 m, delay is (more or less) 300 ms, but I can't put offset = 0.3 because this value must be an integer.

Is there a solution?

---

* Add command line parsing to your script and re-start your script with different parameters once it is finished. You could do that e.g. with a shell script, or with another Python script or whatever.

I don't like this way, but I found it simpler, so at the moment I'm using a second Python script.

* Interweave the playlist logic directly into your existing code

In TODO list :)

Thank you again!

[Stemby]

So if I'm not mistaken again, you can use offset = round(0.3 * samplerate) to get a delay of 300 ms.

Yes, that works!

Thank you very much.

BTW, you should make sure that JACK uses the same sample rate as your audio files!

Good to know, thanks.