Comments (3)
Thanks for the additional report. This is actually something we have been aware of recently but we haven't put out a mitigation yet because we figured no one would use sample rates that low. See 8688906
I will release a fix for it today
from web-audio-api-rs.
Thanks for the report. This is very interesting since you are combining the MediaDevices API and the MediaRecorder API, without an AudioContext in between. This is of course fine on the web, but not something we have really accounted for in our implementation - which main focus is the Web Audio API.
Tacking an AudioContext in between makes it work:
use std::sync::{Arc, Mutex};
use web_audio_api::context::{AudioContext, AudioContextLatencyCategory, AudioContextOptions};
use web_audio_api::media_devices;
use web_audio_api::media_devices::MediaStreamConstraints;
use web_audio_api::media_recorder::MediaRecorder;
use web_audio_api::node::AudioNode;
// If you are on Linux and use ALSA as audio backend backend, you might want to run
// the example with the `WEB_AUDIO_LATENCY=playback ` env variable which will
// increase the buffer size to 1024
//
// `WEB_AUDIO_LATENCY=playback cargo run --release --example recorder
fn main() {
let latency_hint = match std::env::var("WEB_AUDIO_LATENCY").as_deref() {
Ok("playback") => AudioContextLatencyCategory::Playback,
_ => AudioContextLatencyCategory::default(),
};
let context = AudioContext::new(AudioContextOptions {
latency_hint,
..AudioContextOptions::default()
});
// connect the mic via an AudioContext media_stream_source to an media_stream_destination
let mic = media_devices::get_user_media_sync(MediaStreamConstraints::Audio);
let stream_source = context.create_media_stream_source(&mic);
let dest = context.create_media_stream_destination();
stream_source.connect(&dest);
// record the media_stream_destination
let recorder = MediaRecorder::new(dest.stream());
let recording: Arc<Mutex<Vec<u8>>> = Default::default();
let arc = Arc::clone(&recording);
recorder.set_ondataavailable(move |mut event| {
let mut r = arc.lock().unwrap();
r.append(&mut event.blob);
});
recorder.start();
std::thread::sleep(std::time::Duration::from_secs(4));
recorder.stop();
std::thread::sleep(std::time::Duration::from_secs(1));
let data = recording.lock().unwrap();
let mut file = std::fs::OpenOptions::new()
.create(true)
.write(true)
.open("unprocessed.wav")
.unwrap();
file.write_all(&data).unwrap();
}
You might wonder what's happening in your example and why it produces a large wav with garbage. The AudioContext in our library is responsible for timing of the render quantums to produce e.g. 48000 samples per second. We have taken some shortcuts in implementing the MediaStream API that results in them supplying samples on demand, instead of having an internal clock themselves. When you hook the mic directly to the recorder, the recorders polls for samples continuously. The stream is trying to provide them ASAP, and inserting silent frames whenever the underlying source (mic in this case) cannot provide them. Since there is no AudioContext clock to tame the recorder polling, the result is a giant wav file (the faster your computer, the larger the file). The noise is because it is mixing true signal with silent gaps.
In any case this is something we need to address. For now remember to actually use the Web Audio API when using our library :)
NB: the examples/recorder.rs
file has a somewhat better implementation than your 1000ms delay to flush the recorder.
NB2: some audio players (e.g. iTunes) don't display the length of the wav files correctly in case of 'streaming wavs'. Don't let this confuse you, it plays just fine
from web-audio-api-rs.
Thank you so much for the thorough response!
the examples/recorder.rs file has a somewhat better implementation than your 1000ms delay to flush the recorder.
Thank you for the hint. I plan to send the blobs directly into the preprocessing and then into the neural network over channels. The start/stop function will just be controlled with a GUI.
One last question if you don't mind. Part of my preprocessing is, that the audio needs to be resampled to 15kHz.
The example code for resampling just uses the default audio context sample rate.
However, if I adjust the sample rate of the audio context in your example above like this:
let context = AudioContext::new(AudioContextOptions {
latency_hint,
sample_rate: Some(15_000.0),
..AudioContextOptions::default()
});
Construction of the audio context panics with this stack trace:
thread 'main' panicked at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/common.rs:25:5:
assertion `left == right` failed: Input FFT buffer must be a multiple of FFT length. Expected multiple of 639, got len = 1124
left: 485
right: 0
stack backtrace:
0: rust_begin_unwind
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
1: core::panicking::panic_fmt
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:270:5
4: rustfft::common::fft_error_inplace
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/common.rs:25:5
5: <rustfft::avx::avx_mixed_radix::MixedRadix9xnAvx<A,T> as rustfft::Fft<T>>::process_with_scratch
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/avx/mod.rs:206:21
6: rustfft::Fft::process
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustfft-6.1.0/src/lib.rs:188:9
7: hrtf::make_hrtf
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:256:5
8: hrtf::HrtfSphere::new::{{closure}}
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:636:33
9: core::iter::adapters::map::map_try_fold::{{closure}}
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/adapters/map.rs:91:28
10: core::iter::traits::iterator::Iterator::try_fold
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/traits/iterator.rs:2461:21
11: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/adapters/map.rs:117:9
12: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/in_place_collect.rs:258:13
13: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/in_place_collect.rs:182:28
14: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/mod.rs:2749:9
15: core::iter::traits::iterator::Iterator::collect
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/iter/traits/iterator.rs:2053:9
16: hrtf::HrtfSphere::new
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:632:22
17: hrtf::HrtfProcessor::new
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hrtf-0.8.1/src/lib.rs:871:27
18: web_audio_api::node::panner::load_hrtf_processor::{{closure}}
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/node/panner.rs:36:29
19: std::collections::hash::map::Entry<K,V>::or_insert_with
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/collections/hash/map.rs:2560:43
20: web_audio_api::node::panner::load_hrtf_processor
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/node/panner.rs:27:5
21: web_audio_api::context::concrete_base::ConcreteBaseAudioContext::new
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/context/concrete_base.rs:245:13
22: web_audio_api::context::online::AudioContext::new
at /home/david/.cargo/registry/src/index.crates.io-6f17d22bba15001f/web-audio-api-0.37.0/src/context/online.rs:181:20
23: mic::main
at ./examples/mic.rs:20:19
24: core::ops::function::FnOnce::call_once
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Is there anything I can do about this?
I already tried to just change the sample rate in pipewire, but the audio context keeps using 44.1kHz as a default.
from web-audio-api-rs.
Related Issues (20)
- Free audio processor resources when the node will never start processing HOT 2
- Should the AudioContext continue playing when it is dropped?
- Avoid O(nĀ²) graph ordering when adding nodes 'organically' HOT 2
- Issue with NodeCollection HOT 3
- Review AudioNode connect/disconnect methods
- Rename ChannelConfigOptions to AudioNodeOptions
- Fix `showcase/mic_playback` example - use Worklet instead of bare AudioNode trait
- Implement dynamic lifetime of AudioContext
- OS X Only: Looping Sounds Break Audio HOT 18
- ScriptProcessorNode: avoid (de)allocations of input/output AudioBuffer for realtime safety
- Implement onended events for more cases HOT 1
- disconnect -> connect does not work on PannerNode HOT 2
- ScriptProcessorNode: setting numberOfOutputs to more than 1 panics HOT 1
- Suspended + Sound put windows apps into "Not Responding" state HOT 6
- AudioContext::resume not `Send` anymore? HOT 1
- Difficulty Specifying Media Types and Streaming in Rust MediaRecorder Implementation HOT 5
- MediaRecorder spec compliance: add options and isTypeSupported
- MediaRecorder: support more mimeTypes than audio/wav
- Disable all dependencies for compilation-only test? HOT 2
- Audio is delayed on Android HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from web-audio-api-rs.