mtg/essentia.js

JavaScript library for music/audio analysis and processing powered by Essentia WebAssembly

Essentia.js is a JavaScript (JS) library for music/audio signal analysis and processing developed at the Music Technology Group, UPF, Barcelona. The core of library is powered by Essentia C++ library back-end using WebAssembly built via Emscripten along with a high-level JS and TypeScript API and add-on utility modules. ie, You can run an extensive collection of music/audio processing and analysis algorithms/models on your web browser or Node.js runtime applications. It supports both real-time and offline audio analysis use-cases.

The packaged add-on modules includes configurable feature extractors for some selected features, interface for feature extraction and inference of a collection of pre-trained audio ML models using Tensorflow.js, some helper classes for visualizing common music processing features directly into your HTML web page using Plotly.js library.

• License: https://essentia.upf.edu/licensing_information.html

• Repository: https://github.com/MTG/essentia.js

• Releases: https://github.com/MTG/essentia.js/releases

• Live demos: https://mtg.github.io/essentia.js/examples

• Documentation: https://mtg.github.io/essentia.js/docs/api

• Tutorials: https://mtg.github.io/essentia.js/docs/api/list_tutorial.html

You are also most welcome to contribute to essentia.js.

NOTE: Essentia.js is currently under rapid development. This means that APIs and features will evolve. It is recommended that devs who adopt essentia.js today upgrade their installation as new releases become available, as backwards compatibility is not yet guaranteed. Some of the algorithms are not yet manually tested on the JavaScript front. Please submit the issues at https://github.com/MTG/essentia.js/issues.

Citing Essentia.js

If you want to cite Essentia.js in a scholarly work, please use the following references.

Albin Correya, Jorge Marcos-Fernández, Luis Joglar-Ongay, Pablo Alonso-Jiménez, Xavier Serra, Dmitry Bogdanov. Audio and Music Analysis on the Web using Essentia.js, Transactions of the International Society for Music Information Retrieval (TISMIR). 4(1), pp. 167–181. 2021.

Albin Correya, Dmitry Bogdanov, Luis Joglar-Ongay, Xavier Serra. Essentia.js: A JavaScript Library for Music and Audio Analysis on the Web, 21st International Society for Music Information Retrieval Conference (ISMIR 2020), pp. 605-612. 2020.

essentia.js's Issues

problem downsampling and computing NNLS chroma

HI!! This library is awesome compared with other alternatives as meyda.

I have a problem downsampling audio before calculating the nnls chroma.

ERROR

Uncaught (in promise) 6094632 - Exception catching is disabled, this exception cannot be caught. Compile with -s DISABLE_EXCEPTION_CATCHING=0 or DISABLE_EXCEPTION_CATCHING=2 to catch.

Error is propaged after async function onClickFeatureExtractor() function.

CODE

function handleFiles(event) {
  var files = event.target.files;
  $("#src").attr("src", URL.createObjectURL(files[0]));
  document.getElementById("audio").load();
}

document.getElementById("upload").addEventListener("change", handleFiles, false);


let essentia

/* "https://freesound.org/data/previews/328/328857_230356-lq.mp3"; */

let audioData;
// fallback for cross-browser Web Audio API BaseAudioContext
const AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext();
let plotChroma;
let plotContainerId = "plotDiv";

let isComputed = false;


// callback function which compute the frame-wise HPCP chroma of input audioURL on a button.onclick event
async function onClickFeatureExtractor() {
  let audioURL = document.getElementById("audio").currentSrc;
  console.log(audioURL);
  let chromagram = [];
  // load audio file from an url

  audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
  audioData = essentia.arrayToVector(audioData);
  audioData = essentia.Resample(audioData, 44100, 8000).signal; // sample rate
  console.log(audioData);
}

$(document).ready(function() {
  
  // create EssentaPlot instance
  plotChroma = new EssentiaPlot.PlotHeatmap(
    Plotly, // Plotly.js global 
    plotContainerId, // HTML container id
    "chroma", // type of plot
    EssentiaPlot.LayoutChromaPlot // layout settings
  );

  // Now let's load the essentia wasm back-end, if so create UI elements for computing features
  EssentiaModule().then(async function(WasmModule) {

    essentia = new Essentia(WasmModule);

    // essentia version log to html div
    $("#logDiv").html(
      "<h5> essentia-" + essentia.version + " wasm backend loaded ... </h5>"
    );

    $("#logDiv").append(
      '<button id="btn" class="ui white inverted button">Compute HPCP Chroma </button>'
    );

    var button = document.getElementById("btn");

    // add onclick event handler to comoute button
    button.addEventListener("click", () => onClickFeatureExtractor(), false);
  });
});

Thank you so much!!!!

Add TypeScript type definition file for index.js and add a TS project example

Currently, we distribute the TypeScript type definition (.d.ts) files separately for each module through our releases. Though devs can use these builds for using essentia.js on a TypeScript project like this example, adding an index.d.ts type definition file for index.js can make it easy for developers to integrate into a TS project as similar as in JS. This was mentioned in #46

It would be also nice to have a TypeScript project as an example in the repository for reference.

How?

• Add index.d.ts file and specify it inside the package.json file
• Update tsconfg.json with the respective settings
• Publishing types along with the newest release of the package.
• Add an example to the repository using this typescript entry point.

SpectralPeaks

It says in the documentation

It gives best results with dB input, a blackman-harris 92dB window and interpolation set to true. According to [1], spectral peak frequencies tend to be about twice as accurate when dB magnitude is used rather than just linear magnitude.

Is there a way to compute a spectrum with dB magnitude using Essentia?

Update Dockerfile for compiling the latest upstream of Essentia C++ library

Update the docker image with the latest static builds of the Essentia library along with the newly added Eigen3 dependencies.

EnergyBand and EnergyBandRatio

Both throw errors if trying to pass values for startCutoffFrequency and stopCutoffFrequency

Write integration tests

Description

Add integration tests for the library's algorithms. Assume that the upstream Essentia, from which Essentia.js is built, is already tested. Thus, these integration tests should check that the JS part (wrapper) is working correctly.

However, this could include unit tests as a prior step, to ensure that the Emscripten transpilation process (c++ –> WASM) doesn't negatively affect algorithms' output or render some of them unusable.

How?

Consider creating basic unit tests as a prior step to compare JS algorithm outputs to Python algorithm outputs as a kind of regression testing. These tests could be generated automatically from code_generator.py, then the results saved to file for comparison with another smaller script.

Do this first set of tests on Node.js
E. g. https://github.com/avajs/ava

Switch to object-oriented interface

Description

Change library's interface to object-oriented, where each algorithm consists of a class with a .compute() method.

// E.g.
const framecutter = new essentia.FrameCutter( {frameSize: 2048, hopSize: 512} );
const output = framecutter.compute(audioVector);

Why?

With the current interface, where each algorithm is a function, each call to a given algorithm involves creating an AlgorithmFactory, an instance of the algorithm, and deleting said instance at the end of the compute cycle (see here, for example).

An OO interface (class + compute method) would avoid such drawback, potentially making the library use resources more efficiently. Additionally, it would match the upstream Essentia Python bindings' interface that so many existing users are already be familiar with.

How?

This requires editing /src/python/code_generator.py to change how it generates the c++ wrapper around the upstream library's algorithms.

Trying to use essentia.SuperFluxExtractor with Max/MSP Api

Hello,

I'm trying to use essentia.SuperFluxExtractor with the Max/MSP Api :

const max = require('max-api');
const essentia = require('essentia.js');
const load = require('audio-loader');

max.addHandler("path", (pathname) => {	
load(pathname).then(function(buffer){
let channel = buffer.getChannelData(0);
let inputSignalVector = essentia.arrayToVector(channel);
let onset = essentia.SuperFluxExtractor(inputSignalVector, 20., 4096., 1024., 16., 44100., 0.02);
let ticks = essentia.vectorToArray(onset.onsets);

The script print this error to the console but I still have some values of onsets back. Can you help me ?

stderr �[0;32m[ INFO ] �[0mOn connection TriangularBands::bands → SuperFluxNovelty::bands:
stderr �[0;32m[ INFO ] �[0mBUFFER SIZE MISMATCH: max=0 - asked for read size 3
stderr �[0;32m[ INFO ] �[0mresizing buffer to 24/3
stderr �[0;32m[ INFO ] �[0mOn connection SuperFluxNovelty::differences → SuperFluxPeaks::novelty:
stderr �[0;32m[ INFO ] �[0mBUFFER SIZE MISMATCH: max=0 - asked for read size 4096
stderr �[0;32m[ INFO ] �[0mresizing buffer to 36040/4505
stderr �[0;32m[ INFO ] �[0mTriangularBands: input spectrum size (2049) does not correspond to the "inputSize" parameter (1025). Recomputing the filter bank.

Rename global namespace title of Essentia WASM builds and JS wrappers

Rename the Essentia WASM builds' global namespace title from EssentiaModule to EssentiaWASM in the JS wrappers for better coherence with the terminologies used in the Essentia.js ISMIR paper.

• An example of importing essentia-wasm.module.js

import Essentia from 'essentia.js-core.es.js';
// import essentia-wasm backend
import { EssentiaWASM } from 'essentia-wasm.module.js';

const essentia = new Essentia(EssentiaWASM);

• An example of importing essentia-wasm.web.js

<html lang="en">
  <head>
    <script src="essentia-wasm.web.js"></script>
    <script src="essentia.js-core.js"></script>
    <script>
      let essentia;

      EssentiaWASM().then( function(essentiaWasm) {
        essentia = new Essentia(essentiaWasm);
        // prints version of the essentia wasm backend
        console.log(essentia.version)
        // prints all the available algorithms in essentia.js
        console.log(essentia.algorithmNames);

        // add your custom audio feature extraction callbacks here
      });
    </script>
  </head>
</html>

DynamicComplexity

The dynamicComplexity output of the DynamicComplexity algorithm is always 0 regardless of input or parameters

Custom wrapper and JS binding for MonoMixer algorithm

Custom JS binding for downmixing stereo signal to mono using the MonoMixer algorithm with a wrapper around StereoMuxer algorithm in essentia.

As mentioned in #27, currently there are no JS bindings for algorithms that expect vector_stereosample. This proposed wrapper could expose the following interface for JS bindings instead of expecting a vector_stereosample type.

MonoMixer(std::vector<float>& left_channel, std::vector<float>& right_channel);

Add wrappers for TempoCNN

Write wrapper JS code as part of essentia.js-model package for using TempoCNN.

• Implement correct model input representation and shape
• Inference with Tensorflow.js

Support for ES6 export and imports in essentia wasm builds

Current wasm builds of essentia don't support ES6 style imports.

• Emscripten has added support for exporting wasm builds as ES6 modules. See
  emscripten-core/emscripten#6284

• A workaround to this issue has been suggested in the below given link.
  https://github.com/GoogleChromeLabs/web-audio-samples/blob/master/audio-worklet/design-pattern/lib/em-es6-module.js

Compile using docker

I realize this is a new project so the documentation is not done yet but I thought I would give it a go. When I try to compile in docker using the below command I get a bunch of unresolved symbol errors. I tried adding #include <essentia/debugging.h> to essentiamin.cpp but no luck. I am not familiar with emscripten so any help would be appreciated.

root@c5f24f99030b:/essentia.js# emconfigure sh -c './build-essentia-bindings.sh'

  emconfigure is a helper for configure, setting various environment
  variables so that emcc etc. are used. Typical usage:

    emconfigure ./configure [FLAGS]

  (but you can run any command instead of configure)



Compiling the bindings to bitcode ...

error: unresolved symbol: _ZN8essentia10nameOfTypeERKSt9type_info
error: unresolved symbol: _ZN8essentia12ParameterMap3addERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS_9ParameterE
error: unresolved symbol: _ZN8essentia13setDebugLevelEi
error: unresolved symbol: _ZN8essentia15unsetDebugLevelEi
error: unresolved symbol: _ZN8essentia4Pool3addERKNSt3__212basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_6vectorIfNS5_IfEEEEb
error: unresolved symbol: _ZN8essentia4initEv
error: unresolved symbol: _ZN8essentia6Logger5debugENS_15DebuggingModuleERKNSt3__212basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEb
error: unresolved symbol: _ZN8essentia8shutdownEv
error: unresolved symbol: _ZN8essentia8standard9Algorithm5inputERKNSt3__212basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE
error: unresolved symbol: _ZN8essentia8standard9Algorithm6outputERKNSt3__212basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE
error: unresolved symbol: _ZN8essentia9ParameterD1Ev
error: unresolved symbol: _ZN8essentia14loggerInstanceE
error: unresolved symbol: _ZN8essentia15EssentiaFactoryINS_8standard9AlgorithmEE9_instanceE
error: unresolved symbol: _ZN8essentia15infoLevelActiveE
error: unresolved symbol: _ZN8essentia16debugIndentLevelE
error: unresolved symbol: _ZN8essentia16errorLevelActiveE
error: unresolved symbol: _ZN8essentia18warningLevelActiveE
error: unresolved symbol: _ZN8essentia20activatedDebugLevelsE
error: unresolved symbol: _ZN8essentia8standard9Algorithm14processingModeE
Aborting compilation due to previous errors | undefined
Traceback (most recent call last):
  File "/emscripten/emcc.py", line 3042, in <module>
    sys.exit(run())
  File "/emscripten/emcc.py", line 1759, in run
    final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
  File "/emscripten/tools/shared.py", line 2274, in emscripten
    emscripten._main(cmdline)
  File "/emscripten/emscripten.py", line 2222, in _main
    return temp_files.run_and_clean(lambda: main(
  File "/emscripten/tools/tempfiles.py", line 93, in run_and_clean
    return func()
  File "/emscripten/emscripten.py", line 2227, in <lambda>
    DEBUG=DEBUG,
  File "/emscripten/emscripten.py", line 2153, in main
    temp_files=temp_files, DEBUG=DEBUG)
  File "/emscripten/emscripten.py", line 86, in emscript
    glue, forwarded_data = compiler_glue(metadata, libraries, compiler_engine, temp_files, DEBUG)
  File "/emscripten/emscripten.py", line 218, in compiler_glue
    glue, forwarded_data = compile_settings(compiler_engine, libraries, temp_files)
  File "/emscripten/emscripten.py", line 530, in compile_settings
    cwd=path_from_root('src'), error_limit=300)
  File "/emscripten/tools/jsrun.py", line 132, in run_js
    raise Exception('Expected the command ' + str(command) + ' to finish with return code ' + str(assert_returncode) + ', but it returned with code ' + str(proc.returncode) + ' instead! Output: ' + str(ret)[:error_limit])
Exception: Expected the command ['nodejs', '/emscripten/src/compiler.js', '/tmp/tmpDdWe__.txt', '/emscripten/src/embind/emval.js', '/emscripten/src/embind/embind.js', '/emscripten/src/library_pthread_stub.js'] to finish with return code 0, but it returned with code 1 instead! Output: // The Module object: Our interface to the outside world. We import
// and export values on it. There are various ways Module can be used:
// 1. Not defined. We create it here
// 2. A function parameter, function(Module) { ..generated code.. }
// 3. pre-run appended it, var Module = {}; ..generated
ERROR:root:Configure step failed with non-zero return code: 1.  Command line: sh -c ./build-essentia-bindings.sh at /essentia.js

Feature Request: MultiPitchKlapuri or MultiPitchMelodia

There are already many single pitch estimation algorithms on the web but none for multi-pitch detection. If you are thinking about adding any other algorithms soon these two would be good options.

Emscripten bindings for the all set of available essentia algorithms

Javascript bindings for the whole set of essentia C++ algorithms as similar to the python bindings.

'getAudioChannelDataFromURL' method fails on Safari

Issue

getAudioChannelDataFromURL method in essentia JS API fails with the following error while using on Safari web browser.

Unhandled Promise Rejection: TypeError: Not enough arguments

This is because Safari doesn't support the promise-based decodeAudioData method of Web Audio API AudioContext instances.

Proposed fix

Use callbacks instead of promise-based method in getAudioChannelDataFromURL.
Also, add a fallback to resume the webAudioCtx if it's not running.

Or

use https://github.com/chrisguttandin/standardized-audio-context for dealing cross-browser Web Audio API calls.

PitchContoursMultiMelody

Hello,

I'm trying to use the PitchContoursMultiMelody algorithm to implement real-time multi pitch detection. I'm following the recommended processing chain mentioned in the description of the PitchContours algorithm, which seems to work fine, but after I pass the outputs to the PitchContoursMultiMelody the audio worklet seems to freeze either immediately or after a couple of seconds (no errors are thrown). I'm unsure why this happens and would appreciate any help.

Note that I do try to explicitly delete the C++ objects every time the worklet process runs, as suggested by the Emscripten documentation (in case that's a memory issue), but this doesn't seem to make any difference (I don't have much experience with Emscripten so I might have got that wrong).

See my code below:

import { EssentiaModule } from '/essentia-wasm.module.js'
import Essentia from '/essentia.js-core.es.js'

class AnalyserProcessor extends AudioWorkletProcessor {
  constructor() {
    super()
    this.essentia = new Essentia(EssentiaModule)
    this.initFrames()
    this.initAggregate()
  }

  initFrames() {
    this.frames = []
    for (let i = 0; i < 16; i++) {
      this.frames[i] = {
        array: new Float32Array(2048),
        offset: -i - 1,
      }
    }
  }

  initAggregate() {
    this.aggregate = {
      bins: new this.essentia.module.VectorVectorFloat(),
      saliences: new this.essentia.module.VectorVectorFloat(),
      length: 0,
    }
  }

  process(inputs, outputs, parameters) {
    const essentia = this.essentia

    // assume mono
    const input = inputs[0][0]

    this.frames.forEach(frame => {
      frame.offset += 1

      if (frame.offset >= 0) {
        frame.offset %= 16
        frame.array.set(input, frame.offset * 128)

        if (frame.offset == 15) {
          // start processing chain
          const signal = essentia.arrayToVector(frame.array)
          const eqloud = essentia.EqualLoudness(signal).signal
          const window = essentia.Windowing(
            eqloud, true, 2048, 'hann', 4, true
          ).frame
          const spectrum = essentia.Spectrum(window).spectrum
          const spectralPeaks = essentia.SpectralPeaks(spectrum)

          // check sizes before converting to arrays
          if (
            spectralPeaks.frequencies.size()
            && spectralPeaks.magnitudes.size()
          ) {

            // check peaks before salience function
            const freq = essentia.vectorToArray(spectralPeaks.frequencies)
            const mag = essentia.vectorToArray(spectralPeaks.magnitudes)
            if (!freq.some(f => f <= 0) && !mag.some(m => m < 0)) {
            
              // pitch salience
              const salienceFunction = essentia.PitchSalienceFunction(
                ...Object.values(spectralPeaks)
              ).salienceFunction
              const salienceFunctionPeaks =
                essentia.PitchSalienceFunctionPeaks(salienceFunction)
              this.aggregate.bins.push_back(
                salienceFunctionPeaks.salienceBins
              )
              this.aggregate.saliences.push_back(
                salienceFunctionPeaks.salienceValues
              )
              this.aggregate.length++

              if (this.aggregate.length == 128) {
                const pitchContours = this.essentia.PitchContours(
                  this.aggregate.bins, this.aggregate.saliences
                )

                console.log('PitchContours', pitchContours)

                const pitchContoursMultiMelody =
                  this
                  .essentia
                  .PitchContoursMultiMelody(...Object.values(pitchContours))
                  .pitch

                console.log('MultiMelody', pitchContoursMultiMelody)

                // delete objects ?
                pitchContoursMultiMelody.delete()
                pitchContours.contoursStartTimes.delete()
                pitchContours.contoursSaliences.delete()
                pitchContours.contoursBins.delete()
                this.aggregate.bins.delete()
                this.aggregate.saliences.delete()
                this.initAggregate()
              }

              salienceFunctionPeaks.salienceValues.delete()
              salienceFunctionPeaks.salienceBins.delete()
              salienceFunction.delete()
            }
          }

          spectralPeaks.frequencies.delete()
          spectralPeaks.magnitudes.delete()
          spectrum.delete()
          window.delete()
          eqloud.delete()
          signal.delete()
        }
      }
    })

    return true
  }
}

registerProcessor('AnalyserProcessor', AnalyserProcessor)

Output of EssentiaWorkletProcessor in rms-rt example

Hey all!

I am working through the rms-rt example, and am unsure what is different between the output values below in essentia-worklet-processor.js:

  process(inputs, outputs, parameters) {
    ...
    let output = outputs[0];
    ...
    let vectorInput = this.essentia.arrayToVector(input[0]);
    ...
    let rmsFrame = this.essentia.RMS(vectorInput) // input audio frame
    output[0][0] = rmsFrame.rms;
    return true;
  }

and those retrieved in index.html by:

        analyserNode.getFloatTimeDomainData(analyserData);
        let rms = analyserData[0];

There doesn't seem to be a one-to-one relationship? So, if I want the exact value of rmsFrame.rms in my top-level drawing function etc, how would I access it? Sorry if I am missing something obvious!

Example: https://github.com/MTG/essentia.js/tree/dev/examples/rms-rt

Thanks!

Chords

Hello, lates examples does not include anything about chords...do you have an example or an app whicch dedects chords from mic input?

Pass all algorithm parameters to JS API as a single JS object

Description

Modify JS interface to accept algorithm parameters as a single JS object instead of passing it as different function argument variables.

Why?

Currently, our JS interface allows passing algorithm parameters as function arguments which is similar to the Python bindings of the upstream library. All the algorithms in Essentia have default values for each parameter unless the user specified any custom one.

Since the order of function argument matters in JS. ie. If you want to modify the 5th function argument, you need to provide values for the first 4 function arguments. This cannot be optimal for developer experience since Essentia.js has algorithms that have more than 15 parameters.

Modifying the Typescript wrapper of Essentia.js to pass algorithm parameters as a single JS object will mitigate this issue. ie. Developers can specify algorithm parameters with a JS object with only selected parameters they wanted to modify.

How?

This can be potentially achieved by modifying the code_generator.py script to generate the Typescript wrapper code with the newly modified interface. Note that the code generator for the C++ wrapper doesn't need to change.

For example, in the case of LogSpectrum algorithm, some similar logic like below can be applied to achieve the desired change.

  type ParamsLogSpectrum = {
    binsPerSemitone?: number=3,
    frameSize?: number=1025, 
    rollOn?: number=0,
    sampleRate?: number=44100
  }

  LogSpectrum(spectrum: any, params: ParamsLogSpectrum) {

    // TODO: add logic to modify the values of params according to user input
    // otherwise fallback to default
    return this.algorithms.LogSpectrum(
      spectrum,
      params.binsPerSemitone,
      params.frameSize,
      params.rollOn,
      params.sampleRate
    );
  }

in case we switch to the object-oriented interface (see #64) for algorithms in the future. This change can be defined in its configure method.

Onsets and OnsetDetection

The Onsets algorithm expects a 'matrix_real' as its first input. How can use the output of the OnsetDetection algorithm to create such a matrix in JavaScript?
Also, the second input of OnsetDetection algorithm is 'phase'. Which algorithm should I use to compute that?

Many thanks in advance for your help!

RhythmExtractor throws BindingError when passed VectorFloat

const bpm = essentia.RhythmExtractor(vectorSignal, 1024, 1024, 256, 0.1, 208, 40, 1024, 16000, [], 0.24, true, true).bpm; throws BindingError with message: "Cannot pass "" as a VectorFloat"

Fails on Firefox and Chrome. vectorSignal is of type VectorFloat, and contains a full audio file converted from a Float32Array.

using version 0.1.0

Migrate CI/CD to github actions

Change

Switch CI/CD pipeline from Travis CI to GitHub Actions.

Motivation

• Recent changes to Travis CI pricing plans for an unlimited CI/CD pipeline on a public repository.
• Actions are completely free and integrated with the rest of GitHub.

Essentia.js Tonality Analysis

Is it possible to do a Tonality Analysis with essentia.js ?
Like in this example in python: https://essentia.upf.edu/essentia_python_examples.html#tonality-analysis-hpcp-key-and-scale

Also, the benchmarks about KeyExtractor return really different data from the Python one. https://mtg.github.io/essentia.js-benchmarks/

Thanks

Test benefits of OO interface

(Related to #64)

Description

Before committing to an Object-Oriented interface of class + compute method, try out the approach outlined on #64 for only one or two algorithms (e.g. melspectrogram), and compare to the current (functional) interface.

How?

One of the aims stated in #64, besides matching the upstream Essentia interface, is to make the library more performant.

Thus, comparison should be made between the two interface styles in terms of execution speed and/or memory.

For execution speed, something like benchmark.js could be used.
For memory usage, perhaps try Chrome's performance.measureUserAgentSpecificMemory() or the memory profiling options on devtools.

Unified behavior and interface for returning essentia algorithm outputs

Issue:

JavaScript doesn’t support pass-by-reference native JS variables
ie. base types like float, int, string etc

ie, Expected algorithm outputs are not returned while using essentia algorithms with multiple outputs that has these aforementioned data types.

Also, it would be better to have a unified behavior for the essentia algorithm interface despite it has one or more output returns.

Proposed Fix:

Interface JS Object as default return type using emscripten::val for every essentia algorithm.

JS binding for LoudnessEBUR128 algorithm

As mentioned in #27, currently there are no JS bindings for algorithms that expect vector_stereosample type for its inputs or output variables. However, an intermediate wrapper can be made for LoudnessEBUR128 using StereoMuxer algorithm in essentia.

This proposed wrapper could expose the following interface for JS bindings instead of expecting a vector_stereosample type.

LoudnessEBUR128(std::vector<float>& left_channel, std::vector<float>& right_channel, const float hopSize, const float sampleRate, const bool startAtZero);

Create template for issues and PRs

What?

Create basic issue and PR templates for everyone to use.

Motivation

Having a uniform structure should make it easier for maintainers and developers to deal with issues and PRs, understand, categorize, and prioritize them. It is also a way to ensure all necessary info is covered.

References:

• github how-to docs
• repo with examples

Wrong 'zeroPadding' value for windowing in essentia.js extractors

zeroPadding parameter of the Windowing operations in the essentia.js-extractor add-on was wrongly configured to the user given frameSize.

High-level JS API to essentia.js

Issue

In current embind JS bindings for essentia algorithms, the user needs to specify all the required value for all the parameters. This can be irritating since a lot of essentia Algorithms has more than 10 parameters such as PredominantPitchMelodia, MultiPitchKlapurietc. The current way of binding the class EssentiaJS using embind is not binding the default parameter values specified in the class.

Proposed Fix

A possible solution for this issue on CPP side is to overload every method inside the class EssentiaJS as detailed in the embind documentation. But, comes at the cost of redundancy and bad code readability.

On the other hand, on JS side we could automatically generate a JS interface by extending the existing python code generation script. This could also benefit in generating automated documentation for essentia.js using tools such as jsdoc.

Spectrum fails with odd frame sizes, Uncaught WASM error

What is the issue about?

• Bug
• Feature request
• Usage question
• Documentation
• Contributing / Development

What part(s) of Essentia.js is involved?

• essentia.js-core (vanilla algorithms)
• essentia.js-model (machine learning algorithms)
• essentia.js-plot (plotting utility module)
• essentia.js-extractor (typical algorithm combinations utility)

Description

Any odd frame size for the Spectrum algorithm produces an uncaught WASM error. Neither the Essentia.js documentation nor the upstream docs mention that frame size should be even or a power of two.

Steps to reproduce / Code snippets / Screenshots

	const frames = self.essentia.FrameGenerator(msg.data.audio, frameSize, frameSize*0.5);
	for (let i = 0; i < frames.size(); i++) {
		const audioVector = frames.get(i);
		const windowedFrame = self.essentia.Windowing(audioVector, true, frameSize).frame;
		const spectrum = self.essentia.Spectrum(windowedFrame, frameSize).spectrum;
		const mfccs = self.essentia.MFCC(spectrum, 2, 11000, spectrum.size()).mfcc;
		corpusArray.push({
			id: corpusSizeCount,
			frame: self.essentia.vectorToArray(audioVector),
			analysis: self.essentia.vectorToArray(mfccs)
		});
		corpusSizeCount++;
	}

For odd frame sizes, this produces the following error on the dev console:

System info

Hardware: MacBook Pro 15 inch 2018, 16GB RAM, 2,2 GHz Intel Core i7 6 cores
OS: macOS Catalina 10.15.7
Platform: Chrome 96.0.4664.110
Library version: Essentia.js 0.1.3

JS interface for algorithms that expects non-supported I/O types in Embind

Issue

The current JS bindings of Essentia WASM backend which generated using Emscripten Embind doesn't provide a factory template to map STL data types such as std::complex (see this issue)(vector_complex) and custom datatypes used in Essentia such as TNT::Array*(matrix_real)

The following algorithms expect either an input, parameter, or output variable of std::complex type.

['CartesianToPolar', 'PolarToCartesian', 'Magnitude', 'ConstantQ', 'NSGConstantQ', 'NSGIConstantQ', 'FFT', 'IFFT', 'FFTC', 'IFFTC', 'HarmonicMask', 'HarmonicModelAnal', 'SineModelAnal', 'SineModelSynth']

The following algorithms expect either an input, parameter, or output variable of TNT::Array* type.

['BpmHistogram', 'FadeDetection', 'HumDetector', 'Onsets', 'Panning', 'SBic', 'SingleGaussian']

There was a bug in the python code generator which included these algorithms in the current builds. All the above-listed algorithms will be excluded from the future essentia.js API until this issue was addressed and solved.

Currently, the most straight-forward way to use these algorithms in JS end will be to cross-compile a custom-written Essentia CPP extractor which abstracts these types with common JS supported data types such as JS object using emscripten::val.

In addition, the following algorithms expect a vector_stereosample type for either input or output variables.

['FalseStereoDetector', 'LoudnessEBUR128', 'StereoDemuxer', 'StereoMuxer', 'StereoTrimmer'].

OR

We could write some generic custom CPP wrappers which could expose a JS interface to these algorithms. Which requires a bit of work.

Help running example

I'm trying to run the example from the html provided in the readme. Here are the steps I took:

1. Create an index.html in the essentia.js directory.
2. Ranpython -m SimpleHTTPServer in the essentia.js directory
3. Open http://localhost:8000 in chrome

I get the following in the console:

I tried defining an empty audio buffer by adding the following lines before getting the channel data:

var audioCtx = new AudioContext();
var audioBuffer = audioCtx.createBuffer(1, 22050, 22050);

However then I get a complaint that typedFloat32ArrayVec is not defined. I suspect something has not loaded correctly (This is my first venture into using wasm). I'm also not sure about these errors to stdio.html, is there a dependency I'm missing?

MFCC uncaught error: frameSize lower bound

What is the issue about?

• Bug
• Feature request
• Usage question
• Documentation
• Contributing / Development

What part(s) of Essentia.js is involved?

• essentia.js-core (vanilla algorithms)
• essentia.js-model (machine learning algorithms)
• essentia.js-plot (plotting utility module)
• essentia.js-extractor (typical algorithm combinations utility)

Description

When computing MFCCs, any frame size equal to or lower than 426 results in an uncaught WASM error.

Steps to reproduce / Code snippets / Screenshots

	const frames = self.essentia.FrameGenerator(msg.data.audio, frameSize, frameSize*0.5);
	for (let i = 0; i < frames.size(); i++) {
		const audioVector = frames.get(i);
		const windowedFrame = self.essentia.Windowing(audioVector, true, frameSize).frame;
		const spectrum = self.essentia.Spectrum(windowedFrame, frameSize).spectrum;
		const mfccs = self.essentia.MFCC(spectrum, 2, 11000, spectrum.size()).mfcc;
		corpusArray.push({
			id: corpusSizeCount,
			frame: self.essentia.vectorToArray(audioVector),
			analysis: self.essentia.vectorToArray(mfccs)
		});
		corpusSizeCount++;
	}

For the frame size described above, this produces the following error on the dev console:

System info

Hardware: MacBook Pro 15 inch 2018, 16GB RAM, 2,2 GHz Intel Core i7 6 cores
OS: macOS Catalina 10.15.7
Platform: Chrome 96.0.4664.110
Library version: Essentia.js 0.1.3

Migrate to python 3

Modify scripts in /src/python to use Python 3 instead of Python 2.

This mainly involves changing old string formatting style, for example:

return "const %s %s=std::vector<float>%s" % (map_types_to_cpp(param_dict['type'])

...to using the .format() string method or f-strings.

• Official how-to docs
• Consider using something like 2to3

Config option to slide-out x-axis values in realtime charting

While doing a realtime chart, it would be nice to add the functionality to slide-along the x-axis to the left-side with a given offset value.

We could add a wrapper to call the Plotly.relayout method in the Plotly.js to achieve this.

Efficient way to pass JS typed array to std::vector in both C++ and JS side

Issue

Most of the essentia algorithms use std::vector type as array input. This means we might need to convert to and from JS typed array and std::vector types.

Currently, we are achieving this using some helper functions in essentia.tools.js which does the conversion by copying its elements. This might be an overhead for doing efficient computation. Maybe a better way would be is to pass its pointer and directly accessing Emscripten heap.

Proposed fix

There are some proposed fixes in this thread emscripten-core/emscripten#5519 (comment).

Some initial benchmarks in this thread shows its potential for JS typed array to std::vector conversion from the C++ side.

Cannot import node module on Electron.

due to the size of the wasm file, I cannot require the essentia.js module on my electron application. How may I work around this? I do not understand how to use web workers for this purpose.

TypeError: Cannot convert "undefined" to unsigned int

What is the issue about?

• Bug

What part(s) of Essentia.js is involved?

• essentia.js-core (vanilla algorithms)

Description

TypeError: Cannot convert "undefined" to unsigned int
at Object.toWireType
File essentia.js/dist/essentia-wasm.umd.js:19:2523624

Steps to reproduce / Code snippets / Screenshots

var esLib = require('essentia.js');
const essentia = new esLib.Essentia(esLib.EssentiaWASM);
console.log("essentia.js version: ", essentia.version);
let wav = require('node-wav');
let buffer = fs.readFileSync('./recordings/test.wav');
// Decode wav audio file using node-wav package
let audio = wav.decode(buffer);
// Convert audio data into VectorFloat type
const audioLeftChannelData = essentia.arrayToVector(audio.channelData[0]);
const audioRightChannelData = essentia.arrayToVector(audio.channelData[1]);

// Downmix stereo audio to mono
const audioDownMixed = essentia.MonoMixer(audioLeftChannelData, audioRightChannelData).audio;

// Create overlapping frames of the audio with given frameSize and hopSize
let frames = essentia.FrameGenerator(audioDownMixed,
    1024, // frameSize
    512 // hopSize
);


// Iterate through frames and compute danceability feature
for (var i=0; i < frames.size(); i++) {
    let danceability = essentia.Danceability(frames.get(i)).danceability;
    console.log("Danceability: ", danceability);
};
// delete frames from memmory
frames.delete();

// After the use of essentia.js
essentia.delete();

System info

Macos 11.3.1
Nodejs 14.0.0
essentia.js version: 2.1-beta6-dev

Dissonance

It seems that the output ranges from 0 to 0.5, not 0 to 1 as said in the documentation.
Also, the output depends heavily on the frame size. I've found that a size of ~1024 samples works best. If larger frames are used (4096+) the output tends to only fluctuate between 0.49 and 0.5. If smaller frames are used, the output is mostly 0. Is that because "the algorithm estimates total dissonance by summing up the normalized dissonance values for each pair of peaks"?

Node.js example

Any chance of adding a minimal example for using essentia.js in Node? The main README.md says this is possible, but all the examples are written for the browser.

Set Essentia warnings and infos to false while using non-debug mode

The current builds of essentia.js prints WARNINGS and INFO messages from the Essentia WASM backend to the JS console in the non-debug mode. This was only supposed to be enabled when the user has explicitly opted for the debug mode.

Running with Typescript (and Electron)

I'm trying to get Essentia.js working in my Electron app (with Typescipt and webpack) but I'm having trouble with it.

I also can't seem to get it working on this repl.it: https://repl.it/@christilut/essentia-typescript#index.ts

Not sure what I'm doing wrong here. It seems to complain no typings are available, but I thought they should be.
My Electron app says the same but continues anyway and runs into different import errors.

What is the correct way to import Essentia.js in Typescript? Since there are multiple ways, it would help to know the right way and see what happens then.

AudioWorklet support, add usage examples using AudioWorklet design pattern

• Add essentia wasm build recipes that support AudioWorklet usage.

• Add usage examples for using AudioWorklet design pattern. Existing essentia wasm builds doesn't support ES6 imports as mentioned in #7. Once it's fixed, we will be able to use this pattern.

Add cross-browser testing

Description

Extend #66 to cover cross-browser testing. Again, mainly integration tests since the core algorithms are transpiled from upstream Essentia, which is itself tested, but could also include unit tests to check that the transpilation process with Emscripten doesn't introduce any issues.

How?

• Try with browserstack
• Use tfjs testing with browserstack as a guide

Expose add-on modules to npm entrypoint

Currently, the npm index.js only exposes an instance of the core essentia.js. It would be nice to bundle all the essentia.js related modules into a common namespace and let the user create the instances as they prefer.

A node.js example would be ,

let esPkg = require("essentia.js");

// core essentia.js API
esPkg.Essentia
// essentia WASM backend
esPkg.EssentiaWASM
// add-on modules
esPkg.EssentiaModel
esPkg.EssentiaExtractor
esPkg.EssentiaPlot

When to use Extractor vs standard Essentia algorithms

This is a general question about Essentia.js usage.
What is the basic difference between the algorithms in the core Essentia library versus the essentiaExtractor library? Under what conditions should someone use one or the other?

In the real-time examples, pitch detection is done using Essentia.PitchMelodia while in the Mel Spectrum and HPCP Chroma examples EssentiaExtractor.melSpectrumExtractor and EssentiaExtractor.hpcpExtractor.

But the reasoning behind this is hard to grok having been introduced to the javascript bindings only and having no experience with the C++ or Python libraries.

I would very much appreciate an explanation!

And of course, thank you all for this wonderful work.

Broken code links on Netlify-hosted melspectrogram-rt and pitch-melodia-rt demos

What is the issue about?

• Bug
• Feature request
• Usage question
• Documentation
• Contributing / Development

What part(s) of Essentia.js is involved?

• essentia.js-core (vanilla algorithms)
• essentia.js-model (machine learning algorithms)
• essentia.js-plot (plotting utility module)
• essentia.js-extractor (typical algorithm combinations utility)
• demos (essentia.js/examples)

Description

1. Broken code links on Netlify-hosted versions (SharedArrayBuffer enabled) of melspectrogram-rt and pitch-melodia-rt demos, e.g. https://essentiajs-melspectrogram.netlify.app/

2. Also original demos that point to Netlify-hosted versions use window.confirm() dialog which does not show when the page is not the active tab of the front window (e.g. when the user opens demo in new tab from examples index). This is bad user experience, and means many people may not see the working demo.

Steps to reproduce / Code snippets / Screenshots

• go to https://essentiajs-melspectrogram.netlify.app/ and click on the "Code" link.
• Scroll-click or right-click -> open link in new tab, on "Melspectrogram (Real-time)" from https://mtg.github.io/essentia.js/examples/

System info

Problem 1) N/A since it's a URL problem
Problem 2) seen on Chrome 95.0.4638.54, on Ubuntu 21.10

EssentiaTFInputExtractor.audioBufferToMonoSignal throws TypeError

Problem

When using EssentiaTFInputExtractor.audioBufferToMonoSignal with a stereo audio file the following exception is thrown:

Uncaught (in promise) TypeError: vect.size is not a function
    at Object.Module.vectorToArray (essentia-wasm.web.js:27)
    at EssentiaTFInputExtractor.vectorToArray (essentia.js-model.js:154)
    at EssentiaTFInputExtractor.audioBufferToMonoSignal (essentia.js-model.js:201)
    at EssentiaTFInputExtractor.downsampleAudioBuffer (essentia.js-model.js:218)

Suggested Solution

I believe this happens because vectorToArray is receiving a JS object (returned by the MonoMixer algorithm inside of it) instead of the VectorFloat it expects. As per documentation, this VectorFloat is in the audio property of the returned object.