solarliner/bevy-kira-components

Experimental Bevy <-> Kira integration crate

bevy-kira-components's Issues

Missing .toggle() functionality

Like in this example: https://github.com/bevyengine/bevy/blob/abddbf2d95ab3b0c37acfb11b34a9f323772fad1/examples/audio/audio_control.rs#L38

API for enumerating and selecting an audio output

Summary

Currently, if users want to output to a specific audio device, they need to manually give it in the backend settings. It's a bit too low level for general usage, and we should have a way to make it so that the audio device can be easily enumerated (to allow building settings UIs) and set.

Note that Kira does not support changing the audio device after having created the audio engine.

Expose playback_rate to AudioFileSettings

Summary

playback_rate exists in:

https://github.com/tesselode/kira/blob/ed69848baf290c2def2b58412b4b0a035c435fd7/crates/kira/src/sound/static_sound/settings.rs#L24

And in:

https://github.com/tesselode/kira/blob/ed69848baf290c2def2b58412b4b0a035c435fd7/crates/kira/src/sound/streaming/settings.rs#L22

But there's no way to set it in bevy-kira-components.

Implementation proposal

Expose it in AudioFileSettings:

/// Settings available to the user when instantiating an audio file.
#[derive(Debug, Component, Deserialize, Serialize)]
pub struct AudioFileSettings {
    /// By default, sounds will start playing right away when inserted. Setting this to `true`
    /// prevents that.
    pub start_paused: bool,
    /// Volume at which the audio will play at.
    pub volume: f64,
    /// The playback rate of the sound.
    ///
    /// Changing the playback rate changes both the speed and the pitch of the
    /// sound.
    pub playback_rate: f64,
    /// Panning (in 0..=1) for the sound, where 0 is hard left, and 1 is hard right.
    pub panning: f64,
    /// Optionally loop a region of the sound (given in seconds)
    pub loop_region: Option<Region>,
    /// Only play a specific region of the file
    pub play_region: Region,
    /// Play the file in reverse (not available for streaming sound files)
    pub reverse: bool,
    // pub start_time: StartTime, // TODO: Implement with serializable types
}

As with volume, just support fixed and so expose it as an f64/Fixed.

Callers could then do this:

commands.spawn(
    AudioFileBundle {
        source,
        settings: AudioFileSettings {
            loop_region,
            volume,
            playback_rate: 0.25, // play at 25% of the usual speed
            ..default()
        },
        ..default()
    },
);

Looks like a pretty tiny PR unless I'm missing something, let me know if you're happy with this approach and I can submit it.

Ability to increment/decrement volume

One of the audio examples in the bevy repo allows the user to adjust volume with button presses: https://github.com/bevyengine/bevy/blob/aa80e2d658ed1b84787fd512839a8c84238979ef/examples/audio/audio_control.rs#L49

We're currently gated on tesselode/kira#92

Align error handling conventions on Bevy

Summary

Align the API to Bevy conventions, and remove most Result<(), E> returns types, instead logging on error.

It goes against my personal preferences, as I would prefer Bevy's error uility function be used, but it's more important to be aligned with the rest of Bevy's API instead of doing our own thing.

Loading AudioFile from another asset appears to be the cause of broken spatialization

Summary

rustc 1.78.0 (9b00956e5 2024-04-29)
Bevy version: 0.13.0
`bevy-kira-components` version: main

I created a slightly modified version of the spatial example here: https://github.com/SolarLiner/bevy-kira-components/compare/main...mgi388:bevy-kira-components:spatial-asset-example?expand=1

However, the difference is that instead of source coming from AssetServer.load directly, I have another asset drums.custom file, and that asset embeds the audio file name. This is the contents of drums.custom:

drums.ogg

Then, I have a custom asset loader which loads .custom files and simply takes the contents of it to be a path to the audio file to load, and I keep a handle to it. Then, I use that handle as source once it's loaded. These lines show where the entity is created.

However, also note that if you try and use the same asset name embedded in drums.custom, the issue remains. See my comment in the code, also repeated here:

Passing a direct source to the emitter works as well, but only if it
is a different asset path to the one embedded in custom asset. Try
changing this to just drums.ogg and notice that spatialization does
not work either. I.e. it seems that when something else "owns" the
handle spatialization does not work, but if source gets its own
"unique" handle, it works.

Expected behavior

Spatialization should still work. In the branch, you can uncomment the direct source and see how it originally works.

Actual behavior

Spatialization does not work, but the audio still plays.

I can't see anything obviously wrong with my use of Bevy Assets, but that's not an impossibility. FYI: I originally discovered this because I had hard-coded audio paths in my WIP project, and when I changed my project to start loading audio files from game asset files, I noticed that spatialization stopped working.

Reproduction steps

• Change into the new example and cargo run
• Just listen to the sound not change at all as the sphere goes around
• Note, I've tested this on main which was 81b696a at the time.

Pushing effect commands has no effect

In the interactive example update_track_panning does not seem to work. Is this a known issue?

Apologies if I'm jumping the gun here, I know you haven't really published this crate yet. But having used only kira before and not bevy_kira_audio this crate seems a lot more intuitive to me. My goal is to use it to demonstrate some custom kira effects.

[Tracking] Re-implement existing `bevy_audio` examples

Summary

Re-implementing examples is a good smoke test for feature-parity with the existing bevy_audio plugin.

• audio.rs #17
• audio_control.rs #9
• decodable.rs -> rename to "custom_sound" #17
• pitch.rs
• soundtrack.rs
• spatial_audio_2d.rs
• spatial_audio_3d.rs

Should we trigger an event when audio ends or let users reach into PlaybackState in their own systems?

Summary

Callers can add a AudioFileEndBehavior component to their entity to indicate to bevy-kira-components what it should do when audio ends. This allows callers to have some convenience around entity/audio cleanup without having to poll themselves.

I suggested on Discord that it could be a good idea to additionally fire an event when audio ends, and let callers handle it themselves. I'm currently using Bevy's RemovedComponents to watch for audio ending and then inserting a new AudioFileBundle onto my entity to make it play the next sound in my SFX sequence. Despite bugs such as #23, I think RemovedComponents is working fine for my use case and so I don't actually have a need for such an event right now.

Triggering an event could be a good future escape hatch for cases where bevy-kira-components is too opinionated about ways the user wants to handle audio ended, but I also realised that in the audio_control example it's relatively easy to check the playback state yourself and do what you want. Plus, you can handle any playback state, not just Stopped. So it's not entirely clear to me that we should trigger an event here and so I've created this issue as a way to track that the idea exists, but not that it should necessarily be done (and to be clear that means at some stage this issue could just be closed as wontfix).

I know that in Bevy 0.14 observers are going to be available and there's also a question of whether that feature is appropriate here. It also looks like RemovedComponents is planned to be replaced with observers.

Integrate Kira tracks

Summary

Integrate tracks as entities in Bevy.

Implementation proposal

Tracks

Similarly to how audio sources are currently integrated, tracks can be added in two steps:

1. A first struct which details the track setup is added to the ECS by the user through a TrackBundle.
2. A ECS system reacts to additions of this component to create the Kira track with the specified properties.
   Due to a limitation with Kira, tracks cannot be updated after they're created, so the original struct cannot be used to update the track live (needs Kira issue to change this)

Track handles should be wrapped in a newtype component to let users change the track after it has been created.

Effects

Tracks in Kira have an effects rack, a list of effects that process the sound coming into the track sequentially. Support for effects is a highly requested feature, so the API for it should be as intuitive as possible.

Effects should ideally be thought of as entities as well, which simplifies access from systems. They need to be attached to a track, which eventually can become a relationship once those land in future versions of Bevy; however for now, the only feature we can reliably use is the parenting relationship system; so for now, effects can be implemented as being children entities of the track entity they're on.

Integration of effects into the ECS is done is much of the same way as audio sources. Since there are multiple types of effects, each strongly typed, there should be a generic plugin that takes care of the integration.

The user would specify a settings component through an EffectBundle, which would then apply the effect to the parented track. An EffectControl component is then made available once the effect is added to the track, making it available to be controlled by other systems.

Another downside of the approach is that effect ordering is not explicitly given; insertion order can be recovered from the entity ID, which should be monotonically increasing in terms of its Ord implementation. This is fine, as there are no ways to either reorder effects or any way to insert any post-creation (see Limitations below).

Limitations

Tracks and effects in Kira cannot be "structurally changed" after being created, ie. you cannot add or remove tracks, change their routing, or add or reorder effects after the track has been created. This means that all track-related setup needs to happened within the same tick, which is unfortunate and the path forward most likely warrants asking for more runtime control from Kira.

Possible alternatives

"Render graph"-like setup

Another possibility is to provide a plugin build-time API for creating tracks and effects. This would disallow adding new tracks after the fact, but lends itself to Kira's way of working better. After build, the mixer structure is frozen, and one can only interact with tracks from a Mixer resource which owns all track and effect handles. This also means that audio systems modifying tracks and effects essentially cannot run in parallel, but also simplifies the implementation.

Effects as part of the track

The track can "own" the effects, which simplifies integration and setup, but means all queries changing effects need to have exclusive access over audio track components, which limits the parallelization the scheduler can do over audio-centric systems. It allows a more explicit ordering of effects though.